Sharing secrets in Kubeflow pipeline

Question

I want to share some secrets with my Kubeflow pipeline so I can use them as environment variables in my containers. I've written a pipeline-secrets.yaml that looks like this:

apiVersion: v1
kind: Secret
metadata:
  name: pipeline-secrets
  namespace: kubeflow
type: Opaque
data:
  mysql_db_name: <SECRET>
  mysql_username: <SECRET>
  mysql_password: <SECRET>
  mysql_endpoint: <SECRET>

and a pipeline-pod-defaults.yaml that looks like this:

apiVersion: kubeflow.org/v1alpha1
kind: PodDefault
metadata:
  name: pipeline-pod-defaults
  namespace: kubeflow
specs:
  desc: Configure pipeline secrets as environment variables
  env:
  - name: MYSQL_DB_NAME
    valueFrom:
      secretKeyRef:
        name: pepeline-secrets
        key: mysql_db_name
  - name: MYSQL_USER_NAME
    valueFrom:
      secretKeyRef:
        name: pepeline-secrets
        key: mysql_username
  - name: MYSQL_PASSWORD
    valueFrom:
      secretKeyRef:
        name: pepeline-secrets
        key: mysql_password
  - name: MYSQL_ENDPOINT
    valueFrom:
      secretKeyRef:
        name: pepeline-secrets
        key: mysql_endpoint

And this is how my pipeline looks like:

import kfp
from kfp.dsl import ContainerOp
from kubernetes import client as k8s_client

@kfp.dsl.pipeline(
    name="Training pipeline",
    description=""
)
def train_pipeline():
    get_data = ContainerOp(
        name="Get data",
        image=BASE_IMAGE,
        file_outputs={
            'data': 'data.csv'
        }
    )
    
    kfp.dsl.get_pipeline_conf().set_image_pull_secrets([
        k8s_client.V1ObjectReference(name="regcred"),
        k8s_client.V1ObjectReference(name="pipeline-secrets"),
    ])
    kfp.dsl.ResourceOp(
        name="pipeline-pod-defaults",
        k8s_resource=k8s_client.V1ObjectReference(name="pipeline-pod-defaults"),
        action="apply"
    )

But at the end I'm getting this error:

This step is in Failed state with this message: error: error validating "/tmp/manifest.yaml": error validating data: [apiVersion not set, kind not set]; if you choose to ignore these errors, turn validation off with --validate=false

Is this the correct approach? How can I share my secrets with the rest of the pipeline? Sorry if it's a newbie question, I'm new to both Kubernetes and Kubeflow

Answer 1

has native support for using Kubernetes Secrets now:

https://kfp-kubernetes.readthedocs.io/en/kfp-kubernetes-1.0.0/