Can I use Airflow to start/stop spark streaming job

Question

I have two type of job: Spark Batch jobs and and Spark streaming jobs.

I would like to schedule and manage them both with airflow.

Airflow is using for job has stop. But I want to use it for my streaming job. Can anyone give me some idea or other tool for my streaming job

Answer 1

Batch jobs - including spark can be scheduled via airflow scheduler.

Streaming Jobs - technically yes - there is nothing stopping you. However, it would not make sense to do the same for spark streaming jobs as these are long running jobs and it would be better to execute them via other tools.