On some machines I get the following error when running KMeans:
java.io.FileNotFoundException: File file:/home/daniel/Aimee/agent/big-data/mahout-clusters/target/output/clusters-0-final/_policy does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371) ~[hadoop-core-0.20.204.0.jar:na]
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245) ~[hadoop-core-0.20.204.0.jar:na]
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:720) ~[hadoop-core-0.20.204.0.jar:na]
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417) ~[hadoop-core-0.20.204.0.jar:na]
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412) ~[hadoop-core-0.20.204.0.jar:na]
at org.apache.mahout.clustering.classify.ClusterClassifier.readPolicy(ClusterClassifier.java:223) ~[mahout-core-0.7.jar:0.7]
at org.apache.mahout.clustering.classify.ClusterClassificationMapper.setup(ClusterClassificationMapper.java:75) ~[mahout-core-0.7.jar:0.7]
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142) ~[hadoop-core-0.20.204.0.jar:na]
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) ~[hadoop-core-0.20.204.0.jar:na]
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) ~[hadoop-core-0.20.204.0.jar:na]
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) ~[hadoop-core-0.20.204.0.jar:na]
It seems like a certain file is not being generated for some reason. I tried to google this, but there's nothing documented on the web regarding this kind of problem. I use mahout-core 0.7 which uses hadoop-core 0.20.204.0. Thanks for any clue, this issue is sucking my time ... :-(
OK, this took me some time to solve this, so I leave this answer to anyone who might bump into this. The reason was that since I used canopy to calculate the first centers for KMeans, and canopy was not configured with runClustering=true it did not generate the _policy file. However, this behavior is inconsistent. On some machines the file is being generated regardless of runClustering value.