Search code examples
apache-sparkhiveapache-spark-sqlesrihive-serde

spark use hive custom serde for JSON but class not found


Following along with https://github.com/Esri/gis-tools-for-hadoop/wiki/Aggregating-CSV-Data-%28Spatial-Binning%29 but on spark the classes for the serde are not found.

ClassNotFoundException: Class com.esri.hadoop.hive.serde.JsonSerde not found

My build definition looks like: https://github.com/geoHeil/spatial-heatmaps/blob/master/esri/build.sbt and includes:

 "com.esri.hadoop" % "spatial-sdk-hive" % esriVersion,
 "com.esri.hadoop" % "spatial-sdk-json" % esriVersion,

and the whole project can be found on GitHub: https://github.com/geoHeil/spatial-heatmaps/tree/master/esri

App is executed by running sbt console and pasting the code (for now)


Solution

  • The best approach would be to upgrade the GIS-Tools-for-Hadoop samples to Spatial-Framework-for-Hadoop v2 and Esri-Geometry-for-Java v2 - https://github.com/Esri/gis-tools-for-hadoop/issues/66.

    Alternate workaround would be to downgrade to v1.2 of Spatial-Framework-for-Hadoop.

    [disclosure: collaborator]