Search code examples
kubeflowkubeflow-pipelines

How to change parameters for a Kubeflow Pipeline Recurring Run?

I would like to run my Kubeflow pipeline every 30 minutes using a "Recurring Run". For each run I would like to tweak my parameters. For example:

First run : START_MONTH=2019-01,END_MONTH=2010-02

Second run : START_MONTH=2019-02,END_MONTH=2010-03

and so on.

I can to pass a new json parameter to the job to each model this:

{ 
   1: [{"START_MONTH":"2019-01"},{"END_MONTH":"2019-02"}],
   2: [{"START_MONTH":"2019-02"},{"END_MONTH":"2019-03"}],
}

Now I need a way to index into the parameters based on whether the job is running for the first, second, etc. time.

Is there any way to access the job metadata inside of a pipeline to determine which "run" a job is currently on or does the job itself need to externalise this?

Solution

  • I ended up :

    • adding a json variable to model the parameter list
    • adding a string variable and defaulting it to {{workflow.name}} so it can be used to deduce which "run number" is currently being executed (e.g. myjob-xyz-1, myjob-abc-2, etc) within the pipeline.