Possibly a dumb question, but when I read the Airflow Arch for 2.4.2, there was no mention of Triggerer. The only pieces were Metastore, Webserver, Scheduler(executor is part of this).
Having said this, do we still need Triggerer pod in airflow 2.4.2 if the deployment is on EKS, and executor is KubernetesExecutor ? What does the Triggerer pod do here? Thanks
It’s needed for the deferrable operator, which is a pretty advanced operator if you’re just getting started. That operator frees up worker slots when a worker makes an API call that takes a while to execute, and marks the execution is awaiting a trigger. It’s completion is detected by the triggerer, which then is checked by the scheduler process when reassigning the worker to subsequent tasks.