Search code examples
amazon-web-servicesspring-bootkubernetesmicroservicesamazon-eks

Spring boot application can't find aws credentials from any credentials chain


I'm trying to migrate Several spring boot services to EKS and they can't retrieve aws credentials from credentials chain and pods are failing with following error: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain

These are what I've tried so far:

I'm using Web identity token from AWS STS for credentials retrieval.

@Bean
public AWSCredentialsProvider awsCredentialsProvider() {
    if (System.getenv("AWS_WEB_IDENTITY_TOKEN_FILE") != null) {
        return WebIdentityTokenCredentialsProvider.builder().build();
    }
    return new DefaultAWSCredentialsProviderChain();
}

@Bean
public SqsClient sqsClient(AWSCredentialsProvider awsCredentialsProvider) {
    return SqsClient
            .builder()
            .credentialsProvider(() -> (AwsCredentials) awsCredentialsProvider.getCredentials())
            .region(Region.EU_WEST_1).build();
}

@Bean
public SnsClient snsClient(AWSCredentialsProvider awsCredentialsProvider) {
    return SnsClient
            .builder()
            .credentialsProvider(() -> (AwsCredentials) awsCredentialsProvider.getCredentials())
            .region(Region.EU_WEST_1).build();
}

The services also have aws-java-sdk-sts maven dependency packaged.

IAM role for the services is also fine and AWS_WEB_IDENTITY_TOKEN_FILE is a also automatically created within pod after each Jenkins build based on K8s manifest file.

From pod I can make GET and POST request to SNS and SQS without any problem.


Solution

  • Problem was fixed.

    Main issue was conflicting AWS SDK BOM version with individual models. Also previous version of BOM I was using wasn't supporting AWS SDK v2.x .

    These are the main take aways from the issue:

    1. AWS SDK authenticate services using credentials provider chain . The default credential provider chain of the AWS SDK for Java 2.x searches for credentials in your environment using a predefined sequence.

      1.1 As of AWS SDK for Java 2.x Web identity token from AWS STS is within default provider chain.

      1.2 As long as using v2 of the SDK and having the STS dependency makes explicit configuration of Web identity token redundant.

      1.3 Make sure candidate service is using AWS SDK v2 as it’ll reduce the configuration code to minimum.

    If a candidate service using AWS SDK v1 following configuration should be added as Web identity token isn’t in default provider chain for v1.

    @Bean
    public AWSCredentialsProvider awsCredentialsProvider() {
        if (System.getenv("AWS_WEB_IDENTITY_TOKEN_FILE") != null) {
            return WebIdentityTokenCredentialsProvider.builder().build();
        }    
        return new DefaultAWSCredentialsProviderChain();
    }
    

    Last but not least try to use try to use latest AWS SDK BOM dependency . (currently all modules have the same version, but this may not always be the case)