# Multipart uploads and checksums

Multipart uploads start, take parts, list, complete and abort as S3's do, and the request checksums SDKs send by default are verified, stored and echoed.

A multipart upload stores a large object in parts. The simulator keeps S3's lifecycle, part listings and multipart ETag, so the helpers that upload in parts work against it, such as boto3's `upload_file` and `aws s3 cp` above 8 MiB. Start an upload, put two parts, list them and complete the object:

```bash
s3curl() { curl -s --aws-sigv4 'aws:amz:us-east-1:s3' --user 'AKIAIOSFODNN7EXAMPLE:wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY' "$@"; }
uid=$(s3curl -X POST "$RYSTIC_URL/acme-demo-site/backup.tar?uploads" | grep -oE '<UploadId>[^<]*' | cut -c11-)
e1=$(head -c 5242880 /dev/zero | s3curl -X PUT --data-binary @- "$RYSTIC_URL/acme-demo-site/backup.tar?partNumber=1&uploadId=$uid" -D - -o /dev/null | grep -i '^etag:' | cut -d' ' -f2 | tr -d '\r')
e2=$(s3curl -X PUT --data tail "$RYSTIC_URL/acme-demo-site/backup.tar?partNumber=2&uploadId=$uid" -D - -o /dev/null | grep -i '^etag:' | cut -d' ' -f2 | tr -d '\r')
s3curl "$RYSTIC_URL/acme-demo-site/backup.tar?uploadId=$uid" | grep -oE '<PartNumber>[0-9]+|<Size>[0-9]+'
# -> <PartNumber>1
# -> <Size>5242880
# -> <PartNumber>2
# -> <Size>4
s3curl -X POST --data "<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>$e1</ETag></Part><Part><PartNumber>2</PartNumber><ETag>$e2</ETag></Part></CompleteMultipartUpload>" "$RYSTIC_URL/acme-demo-site/backup.tar?uploadId=$uid" | grep -oE '<ETag>[^<]*'
# -> <ETag>&#34;47693b6aafc99e607a33f321da4dc903-2&#34;
```

### Parts and their limits

Every part but the last must be at least 5 MiB, or `CompleteMultipartUpload` returns `400 EntityTooSmall`. `AbortMultipartUpload` drops the upload and its parts, after which `ListParts` returns `404 NoSuchUpload`. `ListMultipartUploads` lists the uploads still open.

**Differs from S3:** an object is held to 64 MiB however it is uploaded ([Limitations](https://www.rystic.ai/docs/simulators/s3.md#limitations)). So an upload completes with up to twelve parts of 5 MiB and a smaller last one.

### Request checksums

The AWS SDKs send a checksum with every upload by default, in a header or an `aws-chunked` trailer. Its header is one of `x-amz-checksum-crc32`, `-crc32c`, `-crc64nvme`, `-sha1` or `-sha256`. The simulator verifies the checksum, stores it and echoes it under `x-amz-checksum-mode: ENABLED`, so the SDK's integrity validation passes. A mismatch returns `400 BadDigest` and stores nothing:

```bash
sum=$(printf hello | openssl dgst -sha256 -binary | base64)
s3curl -X PUT --data hello -H "x-amz-checksum-sha256: $sum" "$RYSTIC_URL/acme-demo-site/hello.txt" -w '%{http_code}\n'
# -> 200
s3curl -I -H 'x-amz-checksum-mode: ENABLED' "$RYSTIC_URL/acme-demo-site/hello.txt" | grep -i '^x-amz-checksum-sha256'
# -> x-amz-checksum-sha256: LPJNul+wow4m6DsqxbninhsWHlwfp0JecwQzYpOLmCQ=
s3curl -X PUT --data hello -H 'x-amz-checksum-sha256: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=' "$RYSTIC_URL/acme-demo-site/hello.txt" | grep -oE '<Code>[^<]*'
# -> <Code>BadDigest
```

A multipart upload created with `x-amz-checksum-algorithm: CRC32` completes with S3's composite checksum, `<base64>-<part count>`. Read an object's checksum type from the `CompleteMultipartUpload` response or from `GetObjectAttributes`, which both return `COMPOSITE`.

**Differs from S3:** `HeadObject` and `GetObject` return `x-amz-checksum-type: FULL_OBJECT` for such an object, where S3 returns `COMPOSITE` ([Limitations](https://www.rystic.ai/docs/simulators/s3.md#limitations)).

### Streamed bodies

The simulator decodes a body in the `aws-chunked` content coding, and the object stores the payload alone. The SDKs stream an upload with a trailing checksum this way, marked by `Content-Encoding: aws-chunked` or `x-amz-content-sha256: STREAMING-…`.

| The streamed body | The simulator returns |
|---|---|
| a payload whose length differs from `x-amz-decoded-content-length` | `400 IncompleteBody` |
| a trailing `x-amz-checksum-*`, named in `x-amz-trailer`, that does not match | `400 BadDigest`, as for the header form |
| framing that does not parse | `400 IncompleteBody`, before anything is stored |

**Differs from S3:** a chunked body sent without `x-amz-decoded-content-length` is still decoded. S3 requires the header.

---

Source: https://www.rystic.ai/docs/simulators/s3/features/multipart-uploads · Markdown: https://www.rystic.ai/docs/simulators/s3/features/multipart-uploads.md