What I am doing wrong with S3 PUT request using AWS Signature Version 4

Question

I am using the following code to make a simple PUT request to AWS S3 using a version 4 signature:

from collections import OrderedDict
from dateutil import parser
import datetime
import hashlib
import hmac
import requests

key_id = "REDACTED"
secret = "REDACTED"
bucket_name = "REDACTED"

def hashb16(message):
    return hashlib.sha256(message).hexdigest()

def HMAC(key, message):
    return hmac.new(key, message, hashlib.sha256)

current_time = datetime.datetime.utcnow()

url = "https://s3-eu-west-2.amazonaws.com/{}/sometest.txt".format(bucket_name)
payload = "Welcome to Amazon S3."
headers = {
    'date': current_time.strftime("%a, %d %b %Y %H:%m:%S GMT"),
    'host': "s3-eu-west-2.amazonaws.com",
    'x-amz-content-sha256': hashb16(payload),
    'x-amz-date': current_time.strftime('%Y%m%dT%H%M%SZ'),
    'x-amz-storage-class': 'REDUCED_REDUNDANCY'
}
sorted_headers = sorted([k for k in headers])
region = "eu-west-2"
service = "s3"

# step 1
HTTPRequestMethod = "PUT"
CanonicalURI = "/sometest.txt"
CanonicalQueryString = ""
CanonicalHeaders = ""
for key in sorted_headers:
    CanonicalHeaders += "{}:{}\n".format(key.lower(), headers[key])
SignedHeaders = "{}".format(";".join(sorted_headers))
HexEncondeHashRequestPayload = hashb16(payload)
CanonicalRequest = "{}\n{}\n{}\n{}{}\n{}".format(HTTPRequestMethod, CanonicalURI, CanonicalQueryString, CanonicalHeaders, SignedHeaders, HexEncondeHashRequestPayload)

# step 2
Algorithm = "AWS4-HMAC-SHA256"
RequestDateTime = current_time.strftime('%Y%m%dT%H%M%SZ')
CredentialScope = "{}/{}/{}/{}".format(current_time.strftime("%Y%m%d"), region, service, "aws4_request")
HashedCanonicalRequest = hashb16(CanonicalRequest)
StringToSign = "{}\n{}\n{}\n{}".format(Algorithm, RequestDateTime, CredentialScope, HashedCanonicalRequest)

#step 3
kDate = HMAC("AWS4" + secret,     current_time.strftime("%Y%m%d")).digest()
kRegion = HMAC(kDate, region).digest()
kService = HMAC(kRegion, service).digest()
kSigning = HMAC(kService, "aws4_request").digest()
signature = HMAC(kSigning, StringToSign).hexdigest()

#step 4
Authorization = "AWS4-HMAC-SHA256 Credential={}/{},SignedHeaders={},Signature={}".format(key_id, CredentialScope, ";".join(sorted_headers), signature)
headers["Authorization"] = Authorization

response = requests.request("PUT", url, headers=headers)

print response.status_code
print response.text

The steps above are per the AWS documentation. I have tested the hashing functions using the examples from this page and they check out.

Unfortunately, when performing the actual request I get a 403 status code with the usual invalid signature message. What's that I'm missing out in the code above?

Answer 1

Your StringToSign is incorrect. Specifically the HashedCanonicalRequest is incorrect.

The Amazon error response will show you the exact StringToSign. This will help you figure out what is wrong.

[EDIT - I modified your code to work. Note: I removed the payload calculation and changed it to UNSIGNED-PAYLOAD. There were two minor issues:

1) In the StringToSign the bucket name was not specified in the URL (after the line current_time)

2) You were missing a \n in CanonicalRequest.

from collections import OrderedDict
from dateutil import parser
import datetime
import hashlib
import hmac
import requests

key_id = ""
secret = "/N6VFTasQCJic3CqL9tj80UGB6Ba1B"
region = ""
bucket_name = ""
service = "s3"

def hashb16(message):
    return hashlib.sha256(message).hexdigest()

def HMAC(key, message):
    return hmac.new(key, message, hashlib.sha256)

current_time = datetime.datetime.utcnow()

url = "https://s3-us-west-2.amazonaws.com/{}/sometest.txt".format(bucket_name)
payload = "Welcome to Amazon S3."
headers = {
    'date': current_time.strftime("%a, %d %b %Y %H:%m:%S GMT"),
    'host': "s3-us-west-2.amazonaws.com",
    'x-amz-content-sha256': 'UNSIGNED-PAYLOAD',
    'x-amz-date': current_time.strftime('%Y%m%dT%H%M%SZ'),
    'x-amz-storage-class': 'REDUCED_REDUNDANCY'
}
sorted_headers = sorted([k for k in headers])

# step 1
HTTPRequestMethod = "PUT"
CanonicalURI = "/{}/sometest.txt".format(bucket_name)
CanonicalQueryString = ""
CanonicalHeaders = ""
for key in sorted_headers:
    CanonicalHeaders += "{}:{}\n".format(key.lower(), headers[key])
SignedHeaders = "{}".format(";".join(sorted_headers))
HexEncondeHashRequestPayload = hashb16(payload)
CanonicalRequest = "{}\n{}\n{}\n{}\n{}\n{}".format(HTTPRequestMethod, CanonicalURI, CanonicalQueryString, CanonicalHeaders, SignedHeaders, 'UNSIGNED-PAYLOAD')

# step 2
Algorithm = "AWS4-HMAC-SHA256"
RequestDateTime = current_time.strftime('%Y%m%dT%H%M%SZ')
CredentialScope = "{}/{}/{}/{}".format(current_time.strftime("%Y%m%d"), region, service, "aws4_request")
HashedCanonicalRequest = hashb16(CanonicalRequest)
StringToSign = "{}\n{}\n{}\n{}".format(Algorithm, RequestDateTime, CredentialScope, HashedCanonicalRequest)

#step 3
kDate = HMAC("AWS4" + secret,     current_time.strftime("%Y%m%d")).digest()
kRegion = HMAC(kDate, region).digest()
kService = HMAC(kRegion, service).digest()
kSigning = HMAC(kService, "aws4_request").digest()
signature = HMAC(kSigning, StringToSign).hexdigest()

#step 4
Authorization = "AWS4-HMAC-SHA256 Credential={}/{},SignedHeaders={},Signature={}".format(key_id, CredentialScope, ";".join(sorted_headers), signature)
headers["Authorization"] = Authorization

response = requests.request("PUT", url, headers=headers)

print response.status_code
print response.text