Search code examples
airflowgoogle-cloud-composervcpu

Cloud Composer pricing

I've set up a cloud composer environment and I'm very confused with the pricing. In the pricing page, it's written that any environment would have at least 2 DB vCPU and 2 Web server vCPU. Cloud Composer environment specs

Further in the page, it's written that it's possible to partially run the environment (e.g. 25%/month). Cloud Composer pricing

However, in my case, my environment seems to be running 24/24h (my billing report shows ~48 hours of usage per day, corresponding to 24h per vCPU I guess) Billing report example

The problem is that I've nothing running on it, except for airflow_monitoring DAG, that I can't stop even by switching it off or deleting it (but I read here Why is there an automatic DAG 'airflow_monitoring' generated in GCP Composer? that it was necessary?).

So my question is: why does the doc say I can partially run my environment whereas the latter seems to be running all the time without any DAG scheduled on it?

Solution

  • It's not currently possible to "turn off" an environment aside from deleting it entirely, because there are certain system components that always need to be active (like the managed database). The example is referring to using an environment for development for a week, and then getting rid of it completely.

    Airflow consumes CPU and memory regardless of if DAGs are running/scheduled, so Composer is charging for the GCE instances/GKE cluster it's deployed on.