Search code examples
amazon-web-servicesrestamazon-s3postmanmultipart

Problem in uploading multipart Amazon S3 Rest API using PostMan

I am trying to use the Amazon-S3 REST API to upload large file in chunks.

As per the API documentation here I formed my request with postman as follows.

After Initiating CreateMultipartUpload Post Request, I'm successfully getting UploadId for chunk put requests. enter image description here This is Working Fine.

I understand Multipart order, but when executing the step uploading partNumber & UploadId chunks using POSTMAN , I'm getting SHA256Mismatch error because POSTMAN calculating Content-MD5 for whole file not for chunks.

  1. Header enter image description here

  2. Params enter image description here

  3. Body enter image description here

I have found multiple solutions on various forums but those solutions didn't work. Am I missing something here?

Solution

  • Successfully uploaded to Amazon S3 using PostMan using Amazon's multipart upload and the key (for me) was to add a Content-MD5 header manually, and paste in the Base64 encoded MD5 hash for that part (details below). This may not be the exact problem the OP was having, but still, I wanted to share how to use PostMan, provided you have good working IAM key id and secret to your Amazon S3 bucket.

    First, I split a 9 MB "mytest.pdf" file into two parts to test this (I used Linux/WSL command: split -b 5242880 mytest.pdf) making sure the first part is larger than 5MB (the last part can be smaller than 5 MB).

    Next, set up PostMan with following four requests:

    PostMan Requests screenshot

    1. CreateMultipartUpload (e.g., POST https://{{mybucket}}.s3.{{myregion}}.amazonaws.com/mytest.pdf?uploads
    2. UploadPart1 (e.g., PUT https://{{mybucket}}.s3.{{myregion}}.amazonaws.com/mytest.pdf?partNumber=1&uploadId=297a2XMl9kNDqw1BaKl7jk6uK_Mop0mCV68TmWU2n8xjsrM6sgt0hu.93J92Qgw8yaEHlrlj0MSoc9ljmU3sD3dlQsGJixMq9hugPDRTkikM0KV6rmLdpmHjFcWzDEDO)
    3. UploadPart2 (e.g., PUT https://{{mybucket}}.s3.{{myregion}}.amazonaws.com/mytest.pdf?partNumber=2&uploadId=297a2XMl9kNDqw1BaKl7jk6uK_Mop0mCV68TmWU2n8xjsrM6sgt0hu.93J92Qgw8yaEHlrlj0MSoc9ljmU3sD3dlQsGJixMq9hugPDRTkikM0KV6rmLdpmHjFcWzDEDO)
    4. CompleteMultipartUpload (e.g., POST https://{{mybucket}}.s3.{{myregion}}.amazonaws.com/mytest.pdf?uploadId=297a2XMl9kNDqw1BaKl7jk6uK_Mop0mCV68TmWU2n8xjsrM6sgt0hu.93J92Qgw8yaEHlrlj0MSoc9ljmU3sD3dlQsGJixMq9hugPDRTkikM0KV6rmLdpmHjFcWzDEDO)

    I pasted in my IAM key id and access key in the authorization section of PostMan (separate articles for that)

    Ran the CreateMultipartUpload POST to get the UploadId from Amazon.

    Next, calculated MD5 hash of each of my two file parts (I used 7-zip but use tool of your choice). Now, converted that result into Base64. However, I had to make sure that I ended up with 22 characters followed optionally by two equals signs == . When I converted MD5 as text to Base64, I ended up with a longer string, which ended with a single equal sign, not double (this is a sign that it is not encoded in a way that Amazon expects it and will produce an InvalidDigest error). For example, if you use 7-zip to calculate MD5 hash on file part and get the value 58942651efd0f5886810d04ed9df502f, and then use a tool such as the Bases64 encoder below and choose input at Text you get NTg5NDI2NTFlZmQwZjU4ODY4MTBkMDRlZDlkZjUwMmY= but if you choose "HEX" as the input, you get a smaller string WJQmUe/Q9YhoENBO2d9QLw== that is 24 characters (22 characters + 2 equals signs). That's the one you want.

    (no implied endorsement of this tool - not affiliated https://emn178.github.io/online-tools/base64_encode.html)

    If you get it wrong, Amazon will reply with the InvalidDigest error below

    <?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>InvalidDigest</Code>
    <Message>The Content-MD5 you specified was invalid.</Message>
    <Content-MD5>thisisbad</Content-MD5>
    <RequestId>8274DC9566D4AAA8</RequestId>
    <HostId>H6kSy4cl+54nMon1Hq6AGjmTX/MfTVMQQr8vEVNXUnPlfMtIt8HPdObfusckhBpwpG/CJ6ORWv16c=</HostId>
</Error>

    PostMan UploadPart1 Content-MD5 header

    Ran both UploadPart1 and 2

    Finally ran CompleteMultipartUpload with copied and pasted Etag values from the Headers of the previous 2 requests

    <CompleteMultipartUpload>
    <Part>
    <PartNumber>1</PartNumber>
    <ETag>"c716d98e83db1edb27fc25fd03e0ae32"</ETag>
    </Part>
    <Part>
    <PartNumber>2</PartNumber>
    <ETag>"58942651efd0f5886810d04ed9df502f"</ETag>
    </Part>
</CompleteMultipartUpload>