Using CosmosDB change feeds to detect deletions in non-timestamped data

Question

I have a data source with roughly 30,000 records. When I poll the data source I receive all the records in one go. There's no timestamp indicating when the data source was last updated.

I'd like to use Azure to transform the data into a series of add, update and delete instructions that are published to a service bus.

I could post each record to CosmosDB and use the change feed to generate the add and update instructions, which I would then publish to the service bus. But this wouldn't handle deletions. In the case of deletions CosmosDB would have the record but the incoming data source wouldn't, so no delete instruction would be created.

In this scenario is there a way to use a CosmosDB change feed to generate the deletions too? If not, is there another "Azurey" way to generate add, update and delete instructions from polling a non-timestamped data source? Any help much appreciated! Thank you

Answer 1

One way to accomplish this today is to insert a tombstone flag "delete": true in your record.

Then when this item appears in Change Feed you can check for this property and if it exists, fire this off to ServiceBus and then call DeleteItemAsync() from your process hosting Change Feed whether a VM, Azure Function etc.