How to enrich message in File Route from SFTP to JMS in clustered environment using Apache Camel?

Question

We are going to read a file from SFTP and put employees in database.

Following is XML Structure

 <employees>
     <employee></employee>
     <employee></employee>
     <employee></employee>
 <employees>    

The strategy i think is:

1. Pick XML file from SFTP ->
2. Fetch Employee No. from XML -> (24000 Employees)
3. Fetch Data from "System 1 and System 2" on base of employee no. in xml file all together in memory ->
4. Split Employee from XML ->
5. Assign Data from "System 1 and System 2" to Each employee Exchange ->
6. Put Each employee XML to JMS Queue (Read in clustered envionment)

Is there any other best strategy to handle this scenario in Apache Camel in abve route or route 2 (Jms Queue)?

Note: As it is clustered environment; we can't fetch data from System1 & System2 in servers itself then we need to keep tracking by assigning batch id to one file contents. You can think this scenario in other integrations too

Answer 1

The above steps can be improved by

1. Speed up your process after step 4 by parallel processing (e.g. use threadpool)
2. Fasten SFTP process by release core thread after step 3 (e.g. use wireTap)
   • To prevent data lost, store exchange to another persistent storage (e.g. JMS queue)
   • The application's memory usage will increase

If step 3 (fetch data from "System 1 and System 2") can be done concurrently, launch multiple application instance with idempotent (leverage external cache or DB) might help for lots of files.