Recommended GCP blue-green deployment pattern?

Question

We are developing/supporting a mobile app for a client that uses React-Native talking to a GCP backend. The database is Firestore, object storage uses GCP Storage, we use GCP app-engine for a few REST calls, and a series of Cloud Functions to handle some Firestore and GCP Storage processing.

Our prod deploys consist of delivering the mobile-app build files to the client (they distribute app internally on their own servers) and for the backend we use GCP CloudBuild to deploy our app-engine and cloud functions from our master/prod branch. However we have had a series of rough prod deploys over the last six months that largely boil down to unexpected environment config issues when we didn't encounter on dev or qa envs (or forgot about), as well as a few db update issues (datamodel changes). This leaves us with egg on our face with the client as the rollout can be delayed by a number of hours as we fix the issues.

To try and resolve this we have been considering a blue-green deployment strategy for the GCP backend. This would consist of creating a new GCP project, deploying latest master/prod branch to it, and allowing us to test fully before delivering updated mobile app to client - no surprises. This sounds great in theory but in practice it's not so simple. The main issue is the Firestore database and GCP Storage - we would also need to transfer over the storage objects and Firestore database to new project. Doable, but yet another complication.

So our Blue-Green idea is this:

1. We create a new "Firestore/GCP storage" project who's only purpose would be to act as the "prod" data-storage location (no app-engine or Cloud Functions in this project), and act as auth provider. This project would be static and not updated between prod deploys
2. We create a new 'prod' (green) GCP project that will host the latest App-Engine and Cloud Functions for the green deployment. This project would be given permissions to access "Firestore/GCP storage" project data and storage objects
3. The mobile-app would connect with the "data-storage" project for communicating with Firestore and uploading files, and would connect with the new "Green" project for App-Engine REST calls

We are hoping that pattern would allow us to do a Blue-Green backend logic deployment without needing to also worry about data/object transfers. My question is this: has anyone ever used this type of deployment pattern in GCP? Is it a recommended pattern or even a shunned anti-pattern?

Any feedback is appreciated. Thanks

Answer 1

I agree that something crucial like a mobile backend must be tested before the launch.

Accessing to the APIs of the Virtual Machines using gcloud, or migrating the traffic accordingly to the approve of the new updates, is something familiar to Cloud Build using the builders provided.

https://cloud.google.com/cloud-build/docs/cloud-builders#supported_builder_images_provided_by

Cloud Build lets you to deploy according to a conceptual staging or phasing that you could define previously (e.g. blue/green strategy). During the definition, you have to keep in mind the type of application to deploy; in this case you have mentioned a Firestore instance and the general suggestion is to replicate in another instance to get all arranged.

https://cloud.google.com/appengine/docs/flexible/nodejs/migrating-traffic

In the other hand, you can have a kubernetes cluster that allows you to manage these features in a more natural way, so maybe you have to evaluate the workload of your application and try Cloud build beyond the patterns.

I hope this point of view was useful for you.