Search code examples
javaapache-sparkcrc

How to avoid generating .crc file in spark saveAsHadoopFile()?


Is there any configuration to turn off generating .crc file while using saveAsHadoopFile method in java - apache spark?

I using Apache Spark 1.6.2 and running my application in standalone mode.


Solution

  • I didnt found any way to avoid creating .crc files while using saveAsHadoopFile() method. Hence i wrote an utility to delete those unwanted .crc and .SUCCESS files.