Search code examples
hadoopapache-sparkconnectionhadoop-yarndistributed-computing

Hadoop+Spark - once application will start properly once will display the error


I have a file with 40 000 lines, in every line single word, each other.

I start Hadoop, master, slaves - everything works:

start-dfs.sh;start-yarn.sh;/opt/spark/sbin/start-master.sh;/opt/spark/sbin/start-slaves.sh

I run the sample application, which counts the words:

/opt/spark/bin/spark-submit --class org.apache.spark.examples.JavaWordCount --deploy-mode cluster --master yarn --driver-memory 2g --executor-memory 2g /opt/spark/examples/jars/spark-examples_2.11-2.0.0.jar hdfs://hadoop-master:9000/input/outputData.fcs

and I have problem: once application will start properly and displays the result and once will display the error (and I don't change nothing between the restarts):

ERROR shuffle.RetryingBlockFetcher: Exception while beginning fetch of 1 outstanding blocks 
java.io.IOException: Failed to connect to iws2/172.29.77.40:43688

Link to full log with error

I have no idea what is wrong. Below my configuration file:

yarn-site.xml

All slaves and master are set correctry in files "master", "slaves".


Solution

  • Thanks for reply.

    Files from /opt/hadoop/hadoop/etc/hadoop/ (they are identical on master and slaves):

    core-site.xml

    hdfs-site.xml

    mapred-site.xml

    One file that I have modified in Spark (/opt/spark/conf/; it's identical on master and slaves):

    spark-env.sh

    And hosts file:

    /etc/hosts

    Files "master" and "slaves" are identical on all nodes: slaves, master.

    I don't know where is problem.


    EDIT:

    I changed in capacity-scheduler.xml:

      <property>
         <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
         <value>1</value>
      </property>
    

    and in yarn-site.xml:

      <property>
         <name>yarn.nodemanager.vmem-pmem-ratio</name>
         <value>2.1</value>
      </property>
    

    Now it works!

    Second option: I changed input file.

    Maybe it will be useful to others.