Search code examples
kuberneteskubeflowkubeflow-pipelines

Using kfp.dls.containerOp() to run multiple scripts on Kubeflow Pipelines

I have been using the Kubeflow dsl container op command to run a python script on a custom for my Kubeflow pipeline. My configuration looks something like this :

def test_container_op():
    input_path = '/home/jovyan/'
    return dsl.ContainerOp(
        name='test container',
        image="<image name>",
        command=[
             'python', '/home/jovyan/test.py'
        ],
        file_outputs={
            'modeule-logs' : input_path + 'output.log' 
        }
    )

Now, I also want to run a bash script called deploy.sh within the same container. I haven't seen examples of that. Is there something like

command = [
'/bin/bash', '/home/jovyan/deploy.sh',
'python', '/home/jovyan/test.py'
]

Not sure if it's possible. Would appreciate the help.

Solution

  • Kubeflow job is just a Kubernetes job, thus you are limited with Kubernetes job entrypoint being a single command. However you can still chain multiple commands into a single sh command:

    sh -c "echo 'my first job' && echo 'my second job'"

    So that you kubeflow command can be:

    command = [
'/bin/sh', '-c', '/home/jovyan/deploy.sh && python /home/jovyan/test.py'
]