Search code examples
apache-sparkhadoophadoop-yarn

How to end Spark Submit and State Accepted

I'm running data cleaning job using apache griffin : https://griffin.apache.org/docs/quickstart.html

and after submitting the spark job

spark-submit --class org.apache.griffin.measure.Application --master yarn --deploy-mode client --queue default \ --driver-memory 1g --executor-memory 1g --num-executors 2 \ /home/bigdata/apache-hive-2.2.0-bin/measure-0.4.0.jar \ /home/bigdata/apache-hive-2.2.0-bin/env.json /home/bigdata/apache-hive-2.2.0-bin/dq.json

My job is being submitted like the below:

20/04/08 13:18:30 INFO yarn.Client: Application report for application_1586344612496_0247 (state: ACCEPTED) 20/04/08 13:18:31 INFO yarn.Client: Application report for application_1586344612496_0247 (state: ACCEPTED) 20/04/08 13:18:32 INFO yarn.Client: Application report for application_1586344612496_0247 (state: ACCEPTED) 20/04/08 13:18:33 INFO yarn.Client: Application report for application_1586344612496_0247 (state: ACCEPTED) 20/04/08 13:18:34 INFO yarn.Client: Application report for application_1586344612496_0247 (state: ACCEPTED) 20/04/08 13:18:35 INFO yarn.Client: Application report for application_1586344612496_0247 (state: ACCEPTED) 20/04/08 13:18:36 INFO yarn.Client: Application report for application_1586344612496_0247 (state: ACCEPTED) 20/04/08 13:18:37 INFO yarn.Client: Application report for application_1586344612496_0247 (state: ACCEPTED) 20/04/08 13:18:38 INFO yarn.Client: Application report for application_1586344612496_0247 (state: ACCEPTED) 20/04/08 13:18:39 INFO yarn.Client: Application report for application_1586344612496_0247 (state: ACCEPTED) 20/04/08 13:18:40 INFO yarn.Client: Application report for application_1586344612496_0247 (state: ACCEPTED)

And never stops:

and When I check the status of the yarn:

bigdata@dq2:~$ yarn application -status application_1586344612496_0231 20/04/08 13:16:31 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 Application Report : Application-Id : application_1586344612496_0231 Application-Name : batch_accu Application-Type : SPARK User : bigdata Queue : default Start-Time : 1586348775760 Finish-Time : 0 Progress : 0% State : ACCEPTED Final-State : UNDEFINED Tracking-URL : N/A RPC Port : -1 AM Host : N/A Aggregate Resource Allocation : 0 MB-seconds, 0 vcore-seconds Diagnostics :

Job is not moving can anyone pls help....

Solution

  • In my experience, there could be many causes for this issue, but the first checks you should do are the following:

    • Your firewall could be blocking some of the ports between the nodes inside your Hadoop cluster, so the computing never starts. Try to disable temporally the firewall for the private interface, and try again to exclude this problem (if this is the problem, reactivate the firewall and identify the ports you need to open!)
    • Spark might be configured uncorrectly (i.e. resources requirement)