Kinesis Agent not sending records to Stream

Question

I have built a Kinesis Firehose stream to push data into redshift and am trying to push data from an EC2 instance using kinesis agent.

Firehose Stream is able to parse the records but not identify the firehose streams am getting the following java error.

com.amazon.kinesis.streaming.agent.tailing.AsyncPublisher 
[ERROR] AsyncPublisher[kinesis:csvtoredshiftstream:/tmp/redshift.log*]:RecordBuffer
(id=2,records=2,bytes=45) Retriable send error (com.amazonaws.services.kinesis.model.ResourceNotFoundException: 
Stream csvtoredshiftstream under account xyz not found. 
(Service: AmazonKinesis; Status Code: 400; 
Error Code: ResourceNotFoundException; 
Request ID: f4a63623-9a15-b2f8-a597-13b478c81bbc)). Will retry.

Request your pointers to identify and resolve the issue.

Regards, Srivignesh KN

Answer 1

Thank you @peter,

1. I was able to overcome the error using by specifying the inputs in the agent.json in the following manner for firehose events. { "cloudwatch.emitMetrics": true, "kinesis.endpoint": "", "firehose.endpoint": "firehose.us-west-2.amazonaws.com", "flows": [ { "filePattern": "/tmp/s3streaming.", "deliveryStream": "S3TestingStream", "partitionKeyOption": "RANDOM" }, { "filePattern": "/tmp/app.log", "deliveryStream": "yourdeliverystream" } ] } –

2. Moreover for Kinesis Streams to work as expected, the S3 bucket also needs to be created in the same region as that of the Streams.

If the Stream is created in West-2 region, the S3 bucket should also be created in the same region.

Thanks & Regards, Srivignesh KN