Search code examples
google-cloud-platformcelery

How to deploy a celery app to google cloud?


Google cloud desperately wants you to use Cloud Tasks for distributed task processing. This is a non-starter. They offer no way to reliably emulate tasks execution locally, thus violating a core principle of continuous integration (TDD).

Consequently, we seek to deploy celery in combination with a flask application to solve this problem. The flask application is running on Cloud Run. Easy peasy. But... what about the celery application? The problem seems simple: we wish to deploy a docker image to a container orchestration platform which abstracts away as much work as possible. Cloud Run is great, but it is meant for containers receiving requests. Celery does not qualify.

What google service should we use to solve this problem?


Solution

  • You can deploy your application with GKE as a Kubernetes Deployment or StatefulSet, and expose it as a Kubernetes Service. It will allow your Flask application running on the Cloud run to Communicate with the celery Application through the Kubernetes Service.

    Once the celery application deploy on GKE you can use the Kubernetes features such as Horizontal Pod Autoscaling and Load balancing to ensure the application is Highly available and scalable.