I am using Couchbase cache. There is one couchbase server. I have multiple servers to host the app and follow a slight modification of the blue green deployment strategy to deploy app on all the servers.
Deployment strategy - Consider first 50% servers as A and rest as B.
1. Detach A from ELB, B is still serving the requests.
2. Deploy app on A.
3. Reconnect A to ELB.
4. Detach B from ELB but B has some requests to process.
5. Invalidate cache.
6. New requests go to A while B completes requests that it had pending.
7. Post completing pending requests, start deployment on B.
8. Post deployment on B, reconnect B to ELB.
The problem rises with the cache in the Couchbase server. While B has pending requests that run on old code, A is all set to process new requests running on new code. The cache server contains no cached data after step 5.
But in step 6, B will load data into cache according to old code. If A uses that cache, the app will break on A.
If in step 6, A loads data into cache and B uses that cached data, the app will again break on B.
How can I solve this crucial section problem where both A and B are serving requests simultaneously? Any change in deployment strategy is also welcome.
Adding a release tag to the cache key (aka versioning) is probably the most common solution. The release tag differentiates between data cached by different versions of the API. But one should keep the memory demand in mind as the memory usage will be doubled. Also, remember to delete cache for older versions of the API if they are no more receiving requests.