I have an application publishing messages on a Kafka Topic. I've used Confluent's Snow Sync to synchronize the topic with a Snowflake table so when I publish a message to the topic it gets inserted in the table.
Each time a insert/update/delete operation is done from my app I'm inserting into our database but I also need to keep it in sync with Snowflake and we're planning to use Kafka topics for this. (we want to avoid Confluent's connectors to not create dependencies).
To summarize I'm sending a message that holds records information. And it should be inserted/ updated/ deleted. Insert is easy to handle using Snow Sync. What's the best practice to handle the other 2 operations? I'm mentioning that the data volume is going to be very large.
This is the approach I've decided to follow. I'm leaving the url to the article here in case someone else needs to do something similar: Data materialization in Snowflake using Confluent Kafka