apache-sparkdatabricksjob-schedulingdatabricks-workflows
How to to trigger a Databricks job from another Databricks job?
I'm currently working on a project where I have two distinct jobs on Databricks. The second job is dependent on the results of the first one.
I am wondering if there is a way to automatically trigger the second job once the first one has completed successfully. Ideally, I would like to accomplish this directly within Databricks without the need for an external scheduling or orchestration tool. Has anyone been able to implement this type of setup or know if it's possible?
Solution
Databricks is now rolling out the new functionality, called "Job as a Task" that allows to trigger another job as a task in a workflow. Documentation isn't updated yet, but you may see it in the UI.
- Select "Run Job" when adding a new task:
- Select specific job to execute as a task:
- Spark dataframe write method writing many small files
- How to calculate the size of dataframe in bytes in Spark?
- Spark Row object instantiated differently from overloaded prototypes?
- How to copy and convert parquet files to csv
- How to tune spark executor number, cores and executor memory?
- Spark/pyspark on same version but "py4j.Py4JException: Constructor org.apache.spark.api.python.PythonFunction does not exist"
- How to disable all logging info to the spark console from .net application
- How to resolve the following AWS Glue error while writing to Redshift using Spark: "ORA-01722: invalid number"?
- how to avoid row number in read_sql output
- Spark getItem shortcut
- Spark: fill spec value between flag values
- How to read Parquet file from S3 without spark? Java
- spark dataframe sum of column based on condition
- Check whether boolean column contains only True values
- reading csv file with header and tail into apache arrow
- Pyspark - how to initialize common DataFrameReader options separately?
- Open, High, Low, Close, Volume in PySpark using tick data
- Dataframe.write() produces csv file on single node jobs cluster, but not on 2+1 nodes cluster
- Internals of worker/executor usage during coalesce/repartition
- How to run Multi threaded jobs in apache spark using scala or python?
- Including null values in an Apache Spark Join
- Why is my PySpark row_number column messed up when applying a schema?
- Split a datafarme column based on another column - Column is not iterable
- Issue with Multiple Spark Structured Streaming Jobs Consuming Same Kafka Topic
- Pass system property to spark-submit and read file from classpath or custom path
- Pandas cannot read parquet files created in PySpark
- conditional split based on list of column
- Explode JSON array into rows
- How to use pyspark regex to correctly break data with pipe delimited with literal pipe inside?
- Meaning of Exchange in Spark Stage