I have a file with 40 000 lines, in every line single word, each other.
I start Hadoop, master, slaves - everything works:
start-dfs.sh;start-yarn.sh;/opt/spark/sbin/start-master.sh;/opt/spark/sbin/start-slaves.sh
I run the sample application, which counts the words:
/opt/spark/bin/spark-submit --class org.apache.spark.examples.JavaWordCount --deploy-mode cluster --master yarn --driver-memory 2g --executor-memory 2g /opt/spark/examples/jars/spark-examples_2.11-2.0.0.jar hdfs://hadoop-master:9000/input/outputData.fcs
and I have problem: once application will start properly and displays the result and once will display the error (and I don't change nothing between the restarts):
ERROR shuffle.RetryingBlockFetcher: Exception while beginning fetch of 1 outstanding blocks
java.io.IOException: Failed to connect to iws2/172.29.77.40:43688
I have no idea what is wrong. Below my configuration file:
All slaves and master are set correctry in files "master", "slaves".
Thanks for reply.
Files from /opt/hadoop/hadoop/etc/hadoop/ (they are identical on master and slaves):
One file that I have modified in Spark (/opt/spark/conf/; it's identical on master and slaves):
And hosts file:
Files "master" and "slaves" are identical on all nodes: slaves, master.
I don't know where is problem.
EDIT:
I changed in capacity-scheduler.xml:
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>1</value>
</property>
and in yarn-site.xml:
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>2.1</value>
</property>
Now it works!
Second option: I changed input file.
Maybe it will be useful to others.