AWS-SDK alignment errors with Spark 2.1.0 on AWS EMR?

Question

I'm running emr-5.3.1 with spark 2.1.0 on AWS.

When I'm submitting spark task with my JAR (FAT JAR) I'm getting the following error:

user class threw exception: java.lang.NoSuchMethodError: com.amazonaws.auth.DefaultAWSCredentialsProviderChain.getInstance()Lcom/amazonaws/auth/DefaultAWSCredentialsProviderChain;

I can only guess it is because I built my jar using a different AWS-SDK version then the one installed in the Spark 2.1.0

1. What is the right AWS_SDK version installed on the EMR's Spark 2.1.0?
2. Is there a way to force my submitted spark task to run with my jars?

Answer 1

...Spark-submit ignores the jars submitted by the user and uses the jars under /usr/share/aws/aws-java-sdk/ which for EMR 5.4 are of version 1.10.75.1. spark-submit has a parameter which can override the server jars with the user jars, however this can cause other issues... (StayerX)

Original post: https://github.com/aws/aws-sdk-java/issues/1094