Search code examples
gitversion-controljupyterkubeflow

How can I version-control a python lightweight kubeflow component?

It's appropriate to generate a Kubeflow component YAML specification from a python function - see e.g.

https://www.kubeflow.org/docs/components/pipelines/sdk/v2/component-development

We are trying to avoid versioning the Jupyter notebooks.

We would like to version our components in git. If we version the YAML, can we easily regenerate the python function (or even the Jupyter notebook) using any off-the-shelf utils?

Or is there a better way?

Thanks!

Solution

  • In the end we have opted to version control a python file from which the yaml specification is auto-generated.

    The notebook can also be auto-generated using jupytext.

    The python file is thus the master.

    I can't see a satisfactory way to generate the spec from a notebook, but I stand to be corrected.