When I submit spark-shell on a Kerberised YARN cluster. The YARN container quit immediately, Diagnositc in YARN application history shows that:
Application application_1515782018863_0007 failed 2 times due to AM Container for appattempt_1515782018863_0007_000002 exited with exitCode: -1000
For more detailed output, check application tracking page:http://ip-172-31-11-83.us-west-2.compute.internal:8088/cluster/app/application_1515782018863_0007Then, click on links to logs of each attempt.
Diagnostics: Application application_1515782018863_0007 initialization failed (exitCode=255) with output: **User datapass not found**
Failing this attempt. Failing the application.
The problem is that the user 'datapass' exist and is well-configured in the cluster:
kadmin.local: listprincs
...
datapass/[email protected]
datapass/[email protected]
[email protected]
[email protected]
...
$ hadoop fs -ls /user
Found 9 items
drwxr-xr-x - datapass datapass 0 2018-01-13 02:11 /user/datapass
drwxrwxrwx - hadoop hadoop 0 2018-01-12 18:50 /user/hadoop
drwxr-xr-x - mapred mapred 0 2018-01-12 18:33 /user/history
drwxrwxrwx - hdfs hadoop 0 2018-01-12 18:33 /user/hive
drwxrwxrwx - hue hue 0 2018-01-12 18:33 /user/hue
drwxrwxrwx - livy livy 0 2018-01-12 18:33 /user/livy
drwxrwxrwx - oozie oozie 0 2018-01-12 18:38 /user/oozie
drwxrwxrwx - root hadoop 0 2018-01-12 18:33 /user/root
drwxrwxrwx - spark spark 0 2018-01-12 18:33 /user/spark
So why YARN is giving me this false information?
Found the reason:
The user 'datapass' has to be created on the operating systems of all hadoop nodes, otherwise this weird error will be triggered.