I have encountered something called LivyBatchOperator but unable to find a very good example for it to submit pyspark applications in airflow. Any info on this would really be appreciated. Thanks in advance.
I come across this blog post which can help you to walk through available options on Airflow + Spark.
Here is an example of LivyBatchOperator and here is on how to install airflow-livy-operators.
I would recommend below options :
Let me know your response !