Search code examples
node.jsamazon-web-servicesamazon-s3streamamazon-kinesis-firehose

what the correct template mapping should be in the Integration Request aws api-getway?

I am working on an event stream system. And my solution is: api-gatway->firehose data stream->data stream delivery->s3.

1.Send a json request(action is PUT) to api-getway
2.Integrate the request with Kinesis service
3.Valid data in the template mapping of the Integration Request
4.Delivery the data to data stream by firehose data delivery
5.Consume the stream and save into S3

It works with below data 1.Request data

{
   "Data" : "this is a test"
}

2.Template mapping code of the Integration Request with firehose

{ 
"StreamName" : "Stream",
"Data" : "$util.base64Encode($input.json('$.Data'))",
"PartitionKey" : "1"

}

Execution log for request 4ad5f3a1-da7d-436f-bd3f-879ba045c622
Fri Jan 15 15:14:19 UTC 2021 : Starting execution for request: 4ad5f3a1-da7d-436f-bd3f-879ba045c622
Fri Jan 15 15:14:19 UTC 2021 : HTTP Method: PUT, Resource Path: /record
Fri Jan 15 15:14:19 UTC 2021 : Method request path: {}
Fri Jan 15 15:14:19 UTC 2021 : Method request query string: {}
Fri Jan 15 15:14:19 UTC 2021 : Method request headers: {}
Fri Jan 15 15:14:19 UTC 2021 : Method request body before transformations: {
      "property":"1",
      "name":"name"
   }

Fri Jan 15 15:14:19 UTC 2021 : Endpoint request URI: https://kinesis.us-east-1.amazonaws.com/?Action=PutRecord
Fri Jan 15 15:14:19 UTC 2021 : Endpoint request headers: {Authorization=***************************************************************************************************************************************************************************************************************************************************************************************9bb292, X-Amz-Date=20210115T151419Z, x-amzn-apigateway-api-id=450acvde3l, Accept=application/json, User-Agent=AmazonAPIGateway_450acvde3l, X-Amz-Security-Token=IQoJb3JpZ2luX2VjEEAaCXVzLWVhc3QtMSJHMEUCIQC7W693nrtuUGrgUvOpn+PUKTRYTGAzRYVSWUPKoAc4eAIgGp/uH4yV/vynaMQUBNEBTwP79m0P9nVfCmw9CMWjY0Mq5QIIGBACGgw1MzUxMTE0MzMzNDAiDKy2z/4lnGJJJAWaOyrCAnANknvMUIbvf9diTZ4qYsdJ3LjETy9hE0f9pUZTBh7aKRjn2y4hbZc/oXGpsmHcgsgJAlIGD51gD0O5LR+N2EKjQeeo5GijkYvZvdbmxVRAZR27EhFzujCk3okePxAzVb3SeCQDWrBnEWn2VwqX4Y75QWwOgL8ZfcuxNKF/wbks0ukPFBo0p68KQ24PIXOYGMs9IAxHYa195+5mUMSi5Av62QKPpBbUAgZ2IJt9EKvRKyLYCs5XzS4lPCgsIbwouVB38f6UQKdQ9XsmD0cX04ODkdj2/0GHne6ufar7nANd3o08PmALd9mXK4z4gt/OxkZUe2AVLXvNQ4IodzW0WOUe1nFvq6YxpveDVfXKDRSK [TRUNCATED]
Fri Jan 15 15:14:19 UTC 2021 : Endpoint request body after transformations: { 
    "StreamName" : "EventStream",
    "Data" : "IiI=",
    "PartitionKey" : "1" 
}

Fri Jan 15 15:14:19 UTC 2021 : Sending request to https://kinesis.us-east-1.amazonaws.com/?Action=PutRecord
Fri Jan 15 15:14:19 UTC 2021 : Received response. Status: 200, Integration latency: 21 ms
Fri Jan 15 15:14:19 UTC 2021 : Endpoint response headers: {x-amzn-RequestId=c028bf47-0059-8b93-9853-0cccfda9a977, x-amz-id-2=L15V+8AS8gN3HxJwiz0qVqi/UJn2xSWueRMnXdhqgjFw6TeuaRlYX62DZK9pa+O1PcKopTP55aHRLdhX0cQwovKefVpiuRtv, Date=Fri, 15 Jan 2021 15:14:19 GMT, Content-Type=application/x-amz-json-1.1, Content-Length=110}
Fri Jan 15 15:14:19 UTC 2021 : Endpoint response body before transformations: {"SequenceNumber":"49614572211779959518343530489315214421429290892684951554","ShardId":"shardId-000000000000"}
Fri Jan 15 15:14:19 UTC 2021 : Method response body after transformations: {"SequenceNumber":"49614572211779959518343530489315214421429290892684951554","ShardId":"shardId-000000000000"}
Fri Jan 15 15:14:19 UTC 2021 : Method response headers: {X-Amzn-Trace-Id=Root=1-6001b14b-bbccae126175224317a10a4e, Content-Type=application/json}
Fri Jan 15 15:14:19 UTC 2021 : Successfully completed execution
Fri Jan 15 15:14:19 UTC 2021 : Method completed with status: 200

But does not work with below: 1.Request data

{
   "Data" : {
     "property":"1",
     "name":"name"       
   }
}

2.Template mapping code of the Integration Request with firehose

{ "StreamName" : "Stream", "Data" : { "property" : $input.json('$.property'), "name" : $input.json('$.name') }, "PartitionKey" : "1" }

Execution log for request 05b707cc-d95c-40bf-8b75-375048697414
Fri Jan 15 15:39:34 UTC 2021 : Starting execution for request: 05b707cc-d95c-40bf-8b75-375048697414
Fri Jan 15 15:39:34 UTC 2021 : HTTP Method: PUT, Resource Path: /record
Fri Jan 15 15:39:34 UTC 2021 : Method request path: {}
Fri Jan 15 15:39:34 UTC 2021 : Method request query string: {}
Fri Jan 15 15:39:34 UTC 2021 : Method request headers: {}
Fri Jan 15 15:39:34 UTC 2021 : Method request body before transformations: {
      "property":"1",
      "name":"name"
   }

Fri Jan 15 15:39:34 UTC 2021 : Endpoint request URI: https://kinesis.us-east-1.amazonaws.com/?Action=PutRecord
Fri Jan 15 15:39:34 UTC 2021 : Endpoint request headers: {Authorization=***************************************************************************************************************************************************************************************************************************************************************************************a43c70, X-Amz-Date=20210115T153934Z, x-amzn-apigateway-api-id=450acvde3l, Accept=application/json, User-Agent=AmazonAPIGateway_450acvde3l, X-Amz-Security-Token=IQoJb3JpZ2luX2VjEEAaCXVzLWVhc3QtMSJFMEMCHw0tTLi/gwtxUjwAQSl/CIY8aie2nmayl+Qsm6/i520CID9iXCFafaQTh4YqE1/tzvKgMO5IlYgFJrcNbAQB2Nc+KuUCCBkQAhoMNTM1MTExNDMzMzQwIgxeE22ch5hdFy3nB+UqwgJyKnQpnLuEY3zpcbRdEO5jks7yfx2+o1xfIz9Kga0S1PojPfzxh5aD/PthhP8D0jutv96ZVe8p52TwfSnv/z3YeDCFzsnw/U9kGFzGVt1pY2JMB4sg1vU7li8pFP/qiUQ3QA8cXbp4nWeE3kQGlPG4pjH0MsOvowTxM8G6yKosvCdD8fVyCJxWIjFnn1+dK9GGV/MnZlnaVqc57z0n0nrHgLjxBzDcDKJ5/xrgcqcYmUETFj8NyDJ9ESzCp0PhKJV9tGF4LgxbAgffe2Yw/3qpQyB6JNqJrZEczADp3gL0rjIBXhbnx5Yizs9MBMtoB9L22mAwEeqJRx4lK12wOqQZ5+0homLCugauYVoy3juNv/zW [TRUNCATED]
Fri Jan 15 15:39:34 UTC 2021 : Endpoint request body after transformations: { 
    "StreamName" : "EventStream",
    "Data" : { 
        "property" : "1",
        "name" : "name"
    },
    
    "PartitionKey" : "1" 
}

Fri Jan 15 15:39:34 UTC 2021 : Sending request to https://kinesis.us-east-1.amazonaws.com/?Action=PutRecord
Fri Jan 15 15:39:34 UTC 2021 : Received response. Status: 400, Integration latency: 2 ms
Fri Jan 15 15:39:34 UTC 2021 : Endpoint response headers: {x-amzn-RequestId=cc937c57-744b-fca2-94e8-c5214b4386dd, x-amz-id-2=fqg8MLIbHOfeeFXp6wpii3l4yl32mI/5RyTwYQyzw9/OqpdLNqCBvBbTp8x7Q4YWAroefbfHb5IUEYeD68SQqh2bq87nL4vp, connection=close, Date=Fri, 15 Jan 2021 15:39:34 GMT, Content-Type=application/x-amz-json-1.1, Content-Length=99}
Fri Jan 15 15:39:34 UTC 2021 : Endpoint response body before transformations: {"__type":"SerializationException","Message":"Start of structure or map found where not expected."}
Fri Jan 15 15:39:34 UTC 2021 : Method response body after transformations: {"__type":"SerializationException","Message":"Start of structure or map found where not expected."}
Fri Jan 15 15:39:34 UTC 2021 : Method response headers: {X-Amzn-Trace-Id=Root=1-6001b736-328415f2db20383946bffd9e, Content-Type=application/json}
Fri Jan 15 15:39:34 UTC 2021 : Successfully completed execution
Fri Jan 15 15:39:34 UTC 2021 : Method completed with status: 200

reference: https://www.youtube.com/watch?v=0UxiV5sUlcA

Solution

  • My bad and I forgot to encode in template mapping. The code should be:

    { 
  "StreamName" : "Stream", 
  "Data" : "$util.base64Encode(
        { 
           "property" : $input.json('$.property'), 
           "name" : $input.json('$.name') 
        }
  ), 
  "PartitionKey" : "1" 
 }