Why reliable collections is empty after micro-service upgrade and not invoking event OnDataLossAsync to restore state from external backup?
We have large scale system based on stateful services
<StatefulServiceType ServiceTypeName="UserServiceType" HasPersistedState="true" />
HasPersistedState is set as true, and data replicated across replicas, in case of VM failure data still valid and recovering with OnDataLossAsync but after upgrade collections is empty.
I have tried all upgrade options (remove, keep, auto ugrade) application, result the same - collections is empty.
For now we decided to replicate data to blob storage and recover it after service update which is not perfect solution, data recovery takes a few minutes and it makes some service unavailable/inconsistent for that time.
So we are looking for solution that allows to save data after upgrade.
Upgrading a stateful service does not remove data in a Reliable Collection by design. The entire Service Fabric platform is built around solving this very problem with rolling in-place upgrades of stateful service.
My guess is you are either not actually performing rolling upgrades, and instead you are deleting and re-creating the service. That's what the default "Remove Application" setting in Visual Studio Application Debug Mode does - it deletes the application when you stop debugging (thus deleting state). The "Auto Upgrade" option will not delete the application and will automatically perform a rolling upgrading for you the next time you press F5. If your state is gone in this case, then I'm willing to bet there is a bug in your service code. Try it with a brand new stateful service project.