Multipart uploads and checksums

A multipart upload stores a large object in parts. The simulator keeps S3’s lifecycle, part listings and multipart ETag, so the helpers that upload in parts work against it, such as boto3’s upload_file and aws s3 cp above 8 MiB. Start an upload, put two parts, list them and complete the object:

s3curl() { curl -s --aws-sigv4 'aws:amz:us-east-1:s3' --user 'AKIAIOSFODNN7EXAMPLE:wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY' "$@"; }
uid=$(s3curl -X POST "$RYSTIC_URL/acme-demo-site/backup.tar?uploads" | grep -oE '<UploadId>[^<]*' | cut -c11-)
e1=$(head -c 5242880 /dev/zero | s3curl -X PUT --data-binary @- "$RYSTIC_URL/acme-demo-site/backup.tar?partNumber=1&uploadId=$uid" -D - -o /dev/null | grep -i '^etag:' | cut -d' ' -f2 | tr -d '\r')
e2=$(s3curl -X PUT --data tail "$RYSTIC_URL/acme-demo-site/backup.tar?partNumber=2&uploadId=$uid" -D - -o /dev/null | grep -i '^etag:' | cut -d' ' -f2 | tr -d '\r')
s3curl "$RYSTIC_URL/acme-demo-site/backup.tar?uploadId=$uid" | grep -oE '<PartNumber>[0-9]+|<Size>[0-9]+'
# -> <PartNumber>1
# -> <Size>5242880
# -> <PartNumber>2
# -> <Size>4
s3curl -X POST --data "<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>$e1</ETag></Part><Part><PartNumber>2</PartNumber><ETag>$e2</ETag></Part></CompleteMultipartUpload>" "$RYSTIC_URL/acme-demo-site/backup.tar?uploadId=$uid" | grep -oE '<ETag>[^<]*'
# -> <ETag>&#34;47693b6aafc99e607a33f321da4dc903-2&#34;

Parts and their limits

Every part but the last must be at least 5 MiB, or CompleteMultipartUpload returns 400 EntityTooSmall. AbortMultipartUpload drops the upload and its parts, after which ListParts returns 404 NoSuchUpload. ListMultipartUploads lists the uploads still open.

Differs from S3: an object is held to 64 MiB however it is uploaded (Limitations). So an upload completes with up to twelve parts of 5 MiB and a smaller last one.

Request checksums

The AWS SDKs send a checksum with every upload by default, in a header or an aws-chunked trailer. Its header is one of x-amz-checksum-crc32, -crc32c, -crc64nvme, -sha1 or -sha256. The simulator verifies the checksum, stores it and echoes it under x-amz-checksum-mode: ENABLED, so the SDK’s integrity validation passes. A mismatch returns 400 BadDigest and stores nothing:

sum=$(printf hello | openssl dgst -sha256 -binary | base64)
s3curl -X PUT --data hello -H "x-amz-checksum-sha256: $sum" "$RYSTIC_URL/acme-demo-site/hello.txt" -w '%{http_code}\n'
# -> 200
s3curl -I -H 'x-amz-checksum-mode: ENABLED' "$RYSTIC_URL/acme-demo-site/hello.txt" | grep -i '^x-amz-checksum-sha256'
# -> x-amz-checksum-sha256: LPJNul+wow4m6DsqxbninhsWHlwfp0JecwQzYpOLmCQ=
s3curl -X PUT --data hello -H 'x-amz-checksum-sha256: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=' "$RYSTIC_URL/acme-demo-site/hello.txt" | grep -oE '<Code>[^<]*'
# -> <Code>BadDigest

A multipart upload created with x-amz-checksum-algorithm: CRC32 completes with S3’s composite checksum, <base64>-<part count>. Read an object’s checksum type from the CompleteMultipartUpload response or from GetObjectAttributes, which both return COMPOSITE.

Differs from S3: HeadObject and GetObject return x-amz-checksum-type: FULL_OBJECT for such an object, where S3 returns COMPOSITE (Limitations).

Streamed bodies

The simulator decodes a body in the aws-chunked content coding, and the object stores the payload alone. The SDKs stream an upload with a trailing checksum this way, marked by Content-Encoding: aws-chunked or x-amz-content-sha256: STREAMING-….

Differs from S3: a chunked body sent without x-amz-decoded-content-length is still decoded. S3 requires the header.