Search code examples
sshhadoopcygwinlocalhost

hadoop with cygwin: hadoop-daemon.sh: not found

I am installing hadoop on cygwin. I have a windows 7 64-bit machine and i've been following the instrux here: http://alans.se/blog/2010/hadoop-hbase-cygwin-windows-7-x64/#software. I have passwordless SSH setup in cygwin and have installed hadoop 0.20.2.

When I issue bin/start-dfs.sh, I get the following:

starting namenode, logging to /usr/local/hadoop-0.20.2/bin/../logs/hadoop-T3S-namenode-t3s780mon.out
localhost: /usr/local/hadoop-0.20.2/bin/hadoop-daemon.sh: /usr/local/hadoop-0.20.2/bin/hadoop-daemon.sh: not found
localhost: /usr/local/hadoop-0.20.2/bin/hadoop-daemon.sh: /usr/local/hadoop-0.20.2/bin/hadoop-daemon.sh: not found

The file exists in that exact location. The permissions seem fine (it's owned by "Administrators" and i'm running as a user who has "Administrators" privilege. And anyway, it's finding and executing other scripts before getting to this point. This output happens at this line in the bin/slaves.sh script

for slave in `cat "$HOSTLIST"|sed  "s/#.*$//;/^$/d"`; do
 ssh $HADOOP_SSH_OPTS $slave $"${@// /\\ }" \
   2>&1 | sed "s/^/$slave: /" &
 if [ "$HADOOP_SLAVE_SLEEP" != "" ]; then
   echo "@@@ slave.sh is SLEEPING"
   sleep $HADOOP_SLAVE_SLEEP
 fi
done

Specifically at the "ssh..." line. And essentially, what that's calling is (when it all the script variables are resolved):

ssh localhost cd /usr/local/hadoop-0.20.2/bin/.. ; /usr/local/hadoop-0.20.2/bin/hadoop-daemon.sh --config /usr/local/hadoop-0.20.2/bin/../conf start datanode

which, when i run manually (i.e., pasting it into the cygwin console) does not return any such error.

So what's causing this script to fail at this point with this particular message? I assume it's related to how i have ssh set up, but can't puzzle it out, since i feel i've done all in a standard way...

thanks for any help / ideas / clues.

Solution

  • check the user in /etc/passwd. it must have a valid home dir and shell as defined in /etc/shells. next in the slave ssh command line, add the -l ${USER} to the ssh invocations.

    better yet, follow this tutorial: http://en.wikisource.org/wiki/User:Fkorning/Code/Hadoop-on-Cygwin