Search code examples
hadoopmapreducehadoop2spring-cloud-streamspring-cloud-dataflow

Run MapReduce Jar in Spring cloud data


I need to run a mapreduce spring boot application in spring cloud data flow. Usually applications registered in scdf is executed using "java -jar jar-name" command. But my program is a mapreduce and it has to be executed using "hadoop jar jar-name". How do I achieve this ? What would be better approach to run mapreduce application in scdf ? Is it possible to directly register mapreduce apps ?

I'm using local data flow server to register the application.


Solution

  • In SCDF the format of the command to run a JAR file is managed by a deployer. For example, there are local deployer. Cloud Foundry etc... There is/was Hadoop/YARN but it was discontinued I believe. Given that the deployer itself is an SPI you can easily implement your own or even fork/extend local-deployer and modify only what's needed.