Search code examples
javahadoopnutchweb-crawler

Nutch Crawl error - Input path does not exist

I have nutch/hadoop setup with 2 datanode server. I tried to crawl some urls but nutch fails with this error:

Fetcher: segment: crawl/segments
Fetcher: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://devcluster01:9000/user/nutch/crawl/segments/crawl_generate
    at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:190)
    at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:44)
    at org.apache.nutch.fetcher.Fetcher$InputFormat.getSplits(Fetcher.java:105)
    at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249)
    at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1107)
    at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:1145)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:1116)

Can someone help me? I don't know how to solve this! Many many Thx!

Solution

  • verify whether nutch/crawl/segments/crawl_generate path is correct.

    Either path is wrong or parse phase is not completed.