Search code examples
jmsrmispring-batchhttpinvoker

Spring Batch : Remote Chunking & Partitioning without using jms


I am new to spring batch. I want to run spring batch jobs using remote chunking & partitioning technique on multiple servers without using jms. I want to use HTTP Invoker or RMI rather than using jms. But, All examples of remote chunking & partitioning use jms. I can't find examples that use HTTP Invoker or RMI. I wonder if it is possible..

English is not my mother language.. please excuse any errors on my part


Solution

  • You can use any form of communication you want for remote partitioning. However, remote chunking does require persistent communication which is why JMS is typically used.

    The reason you see JMS for remote partitioning is because it's easier to configure a clustered environment with JMS than it is for HTTP. The reason for that is everyone (master and all the slaves) only need to know where the queue is to talk to. Using HTTP as a communication mechanism requires the master and slaves to know a lot more. The master needs to know how to evenly distribute the partitions over all the slaves and where to send the requests to for each slave. All the slaves also need to know where the master is. JMS's centralized distribution model also allows you to dynamically add new slaves during processing where HTTP would require you to have some way to register a new slave with the master.

    The reason persistent communication is required for remote chunking is that there is nothing in the remote partition model to prevent an item from being processed twice since it's sent over the wire (remote partitioning just sends descriptions of the data across and the job repository prevents data from being processed twice).

    You can read more about the difference between the two in my answer here: Difference between spring batch remote chunking and remote partitioning