Search code examples
kubernetesjupyter-notebookipythonkubeflowkubeflow-pipelines

ipython terminates with 'Syntax error: end of file unexpected' inside Kubeflow Pod, not happening in Docker container


I'm trying to create a Kubeflow Pipeline component. My command works fine in a local Docker container, but within a Pod in a Kubeflow Pipeline it terminates with:

ipython /tmp/input-postgresql.ipynb: 1: ipython /tmp/input-postgresql.ipynb: Syntax error: end of file unexpected

To reproduce on Docker:

docker run -it continuumio/anaconda3:2020.07 /bin/sh
curl -o /tmp/input-postgresql.ipynb https://raw.githubusercontent.com/IBM/claimed/master/component-library/input/input-postgresql.ipynb && ipython /tmp/input-postgresql.ipynb

To reproduce on Kubeflow 1.6.0 please use the pipeline.yaml file


Solution

  • Had to change the component.yaml now it works

    Changed syntax for command definition and therefore also needed to find out how to pass parameters:

    implementation:
      container:
        image: continuumio/anaconda3:2020.07
        command:
        - sh
        - -ec
        - |
          host=$0
          database=$1
          user=$2
          password=$3
          port=$4
          sql=$5
          data_dir=$6
          output_data_csv=$7
          mkdir -p $output_data_csv
          wget https://raw.githubusercontent.com/IBM/claimed/master/component-library/input/input-postgresql.ipynb
          ipython ./input-postgresql.ipynb host=$host database=$database user=$user password=$password port=$port sql=$sql data_dir=$data_dir output_data_csv=$output_data_csv
        - {inputValue: host}
        - {inputValue: database}
        - {inputValue: user}
        - {inputValue: password}
        - {inputValue: port}
        - {inputValue: sql}
        - {inputValue: data_dir}
        - {outputPath: output_data_csv}