I'm using boto3 (version 1.4.4) to talk to Amazon's Kinesis API:
import boto3
kinesis = boto3.client('kinesis')
# write a record with data '\x08' to the test stream
response = kinesis.put_record(StreamName='test', Data=b'\x08', PartitionKey='foobar')
print(response['ResponseMetadata']['HTTPStatusCode']) # 200
# now read from the test stream
shard_it = kinesis.get_shard_iterator(StreamName="test", ShardId='shardId-000000000000', ShardIteratorType="LATEST")["ShardIterator"]
response = kinesis.get_records(ShardIterator=shard_it, Limit=10)
print(response['ResponseMetadata']['HTTPStatusCode']) # 200
print(response['Records']) # []
When I test it with any data without the \x escape I'm able to get back the record as expected. Amazon boto3's doc says that "The data blob can be any type of data; for example, a segment from a log file, geographic/location data, website clickstream data, and so on." then why is the message with \x escaped characters dropped? Am I expected to '\x08'.encode('string_escape') before sending the data to kinesis?
If you are interested, I have characters like \x08 in the message data because I'm trying to write a serialized protocol buffer message to a Kinesis stream.
Okay so I finally figured it out. The reason why it wasn't working was because my botocore was on version 1.4.62. I only realized it because another script that ran fine on my colleague's machine was throwing exceptions on mine. We had the same boto3 version but different botocore versions. After I pip install botocore==1.5.26 both the other script and my kinesis put_record started working.
tldr: botocore 1.4.62 is horribly broken in many ways so upgrade NOW. I can't believe how much of my life is wasted by outdated broken libraries. I wonder if Amazon dev can unpublish broken versions of the client?