In Cloud composer1, dag last task fail with incomplete log on daily schedule at 1:00 am on 24th march 2022 but same dag ran successfully on other days
Below is the pic of failed dag with last task incomplete log in google cloud composer version composer-1.17.8-airflow-2.1.4 on 24th march 2022
Next day, Below is the pic of successful dag with last task complete log in google cloud composer version composer-1.17.8-airflow-2.1.4 on 25th march 2022 without any modification in dag code and other dependencies files
Could you please let us know the exact reason why cloud composer 1 behaviour was different on 24th march 2022 because all other days the same dag with same code and dependencies file on scheduled run at 1:00 am was successful?
On the manual run of same failed dags, the result was successful. Also kept the dag with same code and dependencies file for observation/monitoring next few days on scheduled run. The result was successful. Need more clarity on 24th march 2022 failed dag having last task incomplete log
Incomplete logs often means the Airflow worker pod was evicted, which is usually when a node in a Kubernetes cluster is running out of memory or disk, it activates a flag signaling that it is under pressure. If you go to your GKE cluster (the one under Composer's hood) you will probably see that there is indeed an evicted pod (GKE > Workloads > "airflow-worker").
You will probably see in "Tasks Instances" that said tasks have no worker (Hostname) assigned, which, added to incomplete logs, is a proof of the death of the pod
Since this normally happens in highly parallelised DAGs, a way to avoid this is to reduce the worker concurrency or use a better machine. The composer issues only occur either at task instance level or network level. Our composer issue was at the task instance level which we've identified now.
High CPU usage is often the root cause of Worker Pod evictions. If there is very high usage, one should consider scaling out the Composer environment or changing the schedule of DAG runs.
- How to get jobId that was submitted using Dataproc Workflow Template
- Why is BeamRunPythonPipelineOperator unable to track dataflow job status, keeps waiting until job end without returning Dataflow logs?
- Set up Structured logging in cloud composer
- How to upgrade/update minor version of composer via Terraform?
- Reading XCOM and Airflow variables probably slows down Airflow (in Google Cloud Composer)
- What is the correct fields in `time_partitioning` for GCSToBigQueryOperator in Airflow
- curl command BashOpertor in Cloud Composer
- Passing return value from operator to following operator in Airflow
- Airflow instance in GCP Composer Webserver / Scheduler down
- INVALID_ARGUMENT error while updating gcloud container cluster
- Airflow unable to mount a google cloud bucket using gcsfuse
- Is it possible to amend environment config for GCP Cloud Composer?
- Stopping the next execution of an Airflow DAG while the current execution is still running
- Updating a Variable doesn't seem to have any effect
- Create Dynamic task mapping in Airflow from BigQueryGetDataOperator
- How can I dynamically change a bucket name in an Airflow DAG?
- Allocate a static IP to Data Fusion or Cloud composer instance
- is Composer multi regional?
- Airflow Dbt via Kubernetes Pod operator (Cloud Composer 2)
- KubernetesPodOperator is saying forbidden in composer for default namespace?
- Run DAG after TriggerDagRunOperator fails
- Can't create Cloud Composer environment on GCP
- Kubernetespodoperator how to use cmds or cmds and arguments to run multiple commands
- CloudSqlExecuteQueryOperator does not return results to xcom
- Cannot access task instance to get xcom
- How to control the parallelism or concurrency of an Airflow installation?
- set a timeout for Airflow KubernetesPodOperator task
- How to pass a nested dictionaries as env variables to Kubernetes Pod Operator
- SpannerQueryDatabaseInstanceOperator how to perform a parametrized query
- Cloud Composer Kubernetes API timeout issue