Search code examples
apache-kafkasalesforce

SalesForce to Kafka using the new Pub/Sub API

I am trying to figure out the best way to write Salesforce Data to a Kafka topic. I want the Kafka topic to be a state store, not containing only the changed fields (delta topic).

SalesForce recommends using their new Pub/Sub API based on gRPC but I see it only supports platform events and CDC:

  • I don't like using CDC as it only contains the changed fields and then a Kafka streams app would have to be built to create a full state object based on all the changes. This is too much effort and also performance might not be that good when joining all the changes
  • I don't like also the platform event solution because you can define a custom schema but then you have to write custom Apex code to publish data to your platform event, too much effort as well

I think it would be best to use pushTopic but unfortunately it's no longer supported by SalesForce and therefore it's not available in their new Pub/Sub API. Do you have any idea what would be the best way by using Pub/Sub API to receive a full state of the object (containing all fields) every time some of the object field changes?

Solution

  • You can enrich the CDC's with extra fields. Those could maybe be enough in your case.

    Enrich Change Events with Extra Fields When Subscribed with CometD or Pub/Sub API