Search code examples
apache-flinkerror-codelocalstack

Getting Service: AmazonKinesis; Status Code: 502 with apache-flink and localstack Kinesis

My local setup consists of local apache-flink (installed via brew) and localstack with the Kinesis service running.

my docker-compose has

  localstack:
    image: localstack/localstack:0.10.7
    environment:
      - SERVICES=kinesis
    ports:
      - "4568:4568"

and my Kinesis Consumer:

kinesisConsumerConfig.setProperty(ConsumerConfigConstants.AWS_ACCESS_KEY_ID, "123");
kinesisConsumerConfig.setProperty(ConsumerConfigConstants.AWS_SECRET_ACCESS_KEY, "123");
kinesisConsumerConfig.setProperty(ConsumerConfigConstants.AWS_ENDPOINT, "http://localhost:4568");

but when I run the Flink program I get this error:

Caused by: org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.model.AmazonKinesisException: null (Service: AmazonKinesis; Status Code: 502; Error Code: null; Request ID: null)

It only happens when using localstack. If I connect to my Kinesis stream on my AWS account it works perfectly.

Solution

  • Turns out we need to disable cbor and cert checking via ENV var and start flink in the same console

    export AWS_CBOR_DISABLE=1
DISABLE_CERT_CHECKING_JAVA_OPTS="-Dorg.apache.flink.kinesis.shaded.com.amazonaws.sdk.disableCertChecking"
export FLINK_ENV_JAVA_OPTS=${DISABLE_CERT_CHECKING_JAVA_OPTS}
/usr/local/Cellar/apache-flink/1.9.1/libexec/bin/start-cluster.sh