Mulesoft Batch Processing Multiple Connectors

Question

I have 3 Salesforce instances that I need to poll data from and then act on that data via watermarked changes. I am trying to design this in a way where it can be dynamic enough to where if I needed to add in another X number of sources, I'm not duplicating a bunch of business logic that all of the batches have in common.

Here is an example of what my current flow looks like with a single Salesforce Org.

Now what I was hoping to do was be able to somehow use a for-each over a config file that allowed me to pull the login credentials from the .properties file and poll each org and then process the records appropriately.

Is there a way this can be structured to have multiple polls/batches, but use different connectors?

The query on all the orgs are the same, it is only the connection that is different.

My .properties file looks something like this:

sf.org1.user=email@domain.com
sf.org1.pass=![encrypted]
sf.org1.token=![encrypted]

sf.org2.user=email@domain.com
sf.org2.pass=![encrypted]
sf.org2.token=![encrypted]

Can I do this in a dynamic fashion, or do I need to add a batch process for every org I add?

Answer 1

If you add a property with a comma separate list of orgs, it can be split into a list. With a <foreach> each element of the list can be used to parametrize the execution of the same batch per org.

I would take the poll from the batch to trigger the flow instead.

Example properties:

organizations=org1,org2

Example flow and batch:

  <flow name="pollFlow">
    <poll doc:name="Poll" >
      <fixed-frequency-scheduler frequency="10" timeUnit="SECONDS"/>
      <logger message="Flow started" level="INFO" doc:name="Logger"/>
    </poll>
    <dw:transform-message doc:name="Transform Message">
      <dw:input-payload mimeType="application/java" />
      <dw:set-payload><![CDATA[%dw 1.0
%output application/java
---
p('organizations') splitBy ',' map {
    user: p('sf.' ++ $ ++ '.user'),
    pass: p('sf.' ++ $ ++ '.pass'),
    token: p('sf.' ++ $ ++ '.token')
}
]]>
      </dw:set-payload>
    </dw:transform-message>
    <logger message="list of orgs credential: #[payload]" level="INFO" doc:name="Logger"/>
    <foreach doc:name="For Each">
      <batch:execute name="so-dynamic-propertiesBatch" doc:name="Batch Execute" />
    </foreach>
  </flow>
  
  <batch:job name="orgBatch">
    <batch:input>
      <logger message="on input phase #[payload]" level="INFO" doc:name="Logger" />
      <!-- here do the request with credentials payload.user, payload.pass and payload.token -->
    </batch:input>
    <batch:process-records>
      <batch:step name="Batch_Step1">
      ...