Getting access token to use in aws eks job submit, comes back with error java.lang.IllegalArgumentException:

Question

to run spark-submit command from EC2 I generate a token for user with

https://awscli.amazonaws.com/v2/documentation/api/latest/reference/eks/get-token.html

aws eks get-token --cluster-name spark-on-eks --region eu-west-2 > token

This comes back with a token

{
    "kind": "ExecCredential",
    "apiVersion": "client.authentication.k8s.io/v1beta1",
    "spec": {},
    "status": {
        "expirationTimestamp": "2023-03-30T10:02:17Z",
        "token": "k8s-aws-v1.aHR0cHM6Ly9zdHMuZXUtd2VzdC0yLmFtYXpvbmF3cy5jb20vP0FjdGlvbj1HZXRDYWxsZXJJZGVudGl0eSZWZXJzaW9uPTIwMTEtMDYtMTUmWC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBUldRUzZVQ0JKTjdWQjZFQiUyRjIwMjMwMzMwJTJGZXUtd2VzdC0yJTJGc3RzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyMzAzMzBUMDk0ODE3WiZYLUFtei1FeHBpcmVzPTYwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCUzQngtazhzLWF3cy1pZCZYLUFtei1TaWduYXR1cmU9NGUxZGE5MzkxYjZmMGFhM2M3ZDQyM2EyMGQyNTRhMWU4NWQ4M2IwNmEzNTRiNmExOWRkMWIxNmMzODZiYTA0YQ"
    }
}

In spark-submit, I pass this as

export K8S_TOKEN=/var/run/secrets/kubernetes.io/serviceaccount/token
--conf spark.kubernetes.authenticate.submission.oauthTokenFile=$K8S_TOKEN 

but it throws this error

Exception in thread "main" java.lang.IllegalArgumentException: Unexpected char 0x0a at 8 in Authorization value: Bearer {
    "kind": "ExecCredential",
    "apiVersion": "client.authentication.k8s.io/v1beta1",
    "spec": {},
    "status": {
        "expirationTimestamp": "2023-03-30T10:02:17Z",
        "token": "k8s-aws-v1.aHR0cHM6Ly9zdHMuZXUtd2VzdC0yLmFtYXpvbmF3cy5jb20vP0FjdGlvbj1HZXRDYWxsZXJJZGVudGl0eSZWZXJzaW9uPTIwMTEtMDYtMTUmWC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBUldRUzZVQ0JKTjdWQjZFQiUyRjIwMjMwMzMwJTJGZXUtd2VzdC0yJTJGc3RzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyMzAzMzBUMDk0ODE3WiZYLUFtei1FeHBpcmVzPTYwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCUzQngtazhzLWF3cy1pZCZYLUFtei1TaWduYXR1cmU9NGUxZGE5MzkxYjZmMGFhM2M3ZDQyM2EyMGQyNTRhMWU4NWQ4M2IwNmEzNTRiNmExOWRkMWIxNmMzODZiYTA0YQ"
    }
}


        at okhttp3.Headers.checkValue(Headers.java:272)
        at okhttp3.Headers$Builder.add(Headers.java:312)
        at okhttp3.Request$Builder.addHeader(Request.java:196)
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:150)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
        at okhttp3.RealCall.execute(RealCall.java:93)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:490)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:451)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:252)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:879)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:341)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:84)
        at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:139)
        at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$3(KubernetesClientApplication.scala:213)
        at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$3$adapted(KubernetesClientApplication.scala:207)
        at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2611)
        at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:207)
        at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:179)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
23/03

Is this an invalid token in this case or there is an issue with syntax of token generated?

Thanks

Answer 1

The solution was to get rid of newline character from the token generated

aws eks get-token --cluster-name $CLUSTER_NAME --region $REGION --output text | base64 | tr -d '\n' > token