Passing Context information to task in DataBricks Workflows

Question

I need in my task the scheduling date of the job run to run queries based on a timestamp. Is there a way to do it with Parameters?

I am able to set the start-time (https://learn.microsoft.com/en-us/azure/databricks/data-engineering/jobs/jobs#task-parameter-variables). But I need the exact scheduling time.

Answer 1

I guess that you are asking this because your job has several tasks. If that's the case, you can use Dbutils.jobs.taskvalues for doing so, in the first task of your job, you can get the current timestamp as the first command executed and then set a job taskValue:

from datetime import datetime

utc_timestamp = datetime.utcnow()
dbutils.jobs.taskValues.set("job_start_timestamp", utc_timestamp)

And in the other task in which you want to make use of these values:

job_start_timestamp = dbutils.jobs.taskValues.get(taskKey="first_task_name",key="job_start_timestamp")