Search code examples
apache-sparkmesosmesosphere

Apache Mesos 0.27 fetcher Error: Address already in use:

After upgrading apache Mesos to 0.27 on Ubuntu 14.04 (3 master - 3 slave setup) I get the following error while trying to connect to the cluster with Spark 1.6 (client mode without using docker images):

 I0219 12:40:47.684662 11484 fetcher.cpp:379] Fetching URI   
 'hdfs://auto-ha/spark/spark.tgz'
 I0219 12:40:47.684691 11484 fetcher.cpp:250] Fetching directly into the sandbox directory
 I0219 12:40:47.684720 11484 fetcher.cpp:187] Fetching URI     
 'hdfs://auto-ha/spark/spark.tgz'
 I0219 12:40:48.139446 11484 fetcher.cpp:109] Downloading resource     
 with Hadoop client from 'hdfs://auto-ha/spark/spark.tgz' to     '/tmp/mesos/slaves/a7907b94-6dc9-437c-b027-b71379a9e0e1-  S3/frameworks/a7907b94-6dc9-437c-b027-b71379a9e0e1- 0006/executors/5/runs/48753051-450a-4043-908d-58f277633cf4/spark.tgz'
 F0219 12:40:48.140929 11484 process.cpp:892] Failed to initialize: Failed   to bind on 172.16.8.166:5051: Address already in use: Address already in   use [98]
*** Check failure stack trace: ***
    @     0x7ff38fe76a3d  google::LogMessage::Fail()
@     0x7ff38fe7887d  google::LogMessage::SendToLog()
@     0x7ff38fe7662c  google::LogMessage::Flush()
@     0x7ff38fe76839  google::LogMessage::~LogMessage()
@     0x7ff38fe777a2  google::ErrnoLogMessage::~ErrnoLogMessage()
@     0x7ff38fe1d149  process::initialize()
@     0x7ff38fe1e7d2  process::ProcessBase::ProcessBase()
@     0x7ff38fe4d631  process::reap()
@     0x7ff38fe56235  process::subprocess()
@     0x7ff38f603059  HDFS::copyToLocal()
@           0x40eecd  download()
@           0x40b8ea  main
@     0x7ff38de81ec5  (unknown)
@           0x40d2c3  (unknown)
Aborted (core dumped)

End fetcher log for container 48753051-450a-4043-908d-58f277633cf4

Seems to be a port conflict on the mesos-fetcher, however using:

sudo lsof -i | grep 5051

nothing appears to be listenning on that port.

Mesos 0.26 worked well on the same setup.

The hdfs filesystem is working and the docker containerizer is working well. I have also checked for fetcher port config but nothing.

Any hints?

Thanks!

Solution

  • After doing a cold reboot with and upgrade of all the nodes of the cluster(the mesos package was upgraded too) everything is running fine.

    Sems to be a bug of mesos after the upgrade to 0.27.

    Thanks anyway!