I am following the next example to connects to Amazon Kinesis Data Streams and outputs the joined results to Amazon S3 in parquet format: https://docs.aws.amazon.com/glue/latest/dg/glue-etl-scala-example.html
When I configure the connection to AWS and trying to run it, it throws the next error that can't find the credentials and I have assigned in "C:\Users\user.aws\credentials" also I have added environment variables :
AWS_ACCESS_KEY_ID=XXX
AWS_SECRET_ACCESS_KEY=XxXX
But still failing with:
ERROR AWSInstanceProfileCredentialsProviderWithRetries: Got an exception while fetching credentials org.apache.s
park.sql.kinesis.shaded.amazonaws.SdkClientException: Unable to load credentials from service endpoint
The error message is not about your own credentials that you have in "C:\Users\user.aws\credentials", but about missing credentials in an instance profile.
The class AWSInstanceProfileCredentialsProviderWithRetries extends InstanceProfileCredentialsProvider which is:
Credentials provider implementation that loads credentials from the Amazon EC2 Instance Metadata Service
The code you are executing should be running on EC2 instance, therefore it expects to get credentials form instance role. What's more the instance role should have all necessary permissions which your code requires, e.g., using Kinesis.
From the link provided in your question, it is not clear where the example code should be exactly executed. Whether it is a custom ec2 instance, or some glue-related managed instance.