Search code examples
kubeflowkubeflow-pipelines

Can i set Kubeflow pipeline storage path?

I'm trying to install Kubeflow with S3.

The artifact repository has been set up.

I would like to know how to specify the s3 prefix of the storage where the pipeline is stored.

Is this possible?

how can i set configmap?

please help me

Solution

  • You can create a ConfigMap in Kubernetes with the S3 storage configuration. The ConfigMap should include the S3 bucket name and the desired prefix. You can create a YAML file like this:

    apiVersion: v1
kind: ConfigMap
metadata:
  name: kubeflow-pipeline-config
data:
  s3-endpoint: <S3_ENDPOINT> # Example: "s3.amazonaws.com"
  s3-use-https: "1" # Use "1" for HTTPS, "0" for HTTP
  s3-bucket: <S3_BUCKET_NAME>
  s3-prefix: <S3_PREFIX> # The prefix under which pipeline artifacts will be stored

    Apply the ConfigMap to your Kubernetes cluster to make it available to Kubeflow Pipelines.

    kubectl apply -f your-configmap.yaml

    You need to configure your Kubeflow Pipelines installation to use the ConfigMap you created

    Once you have set up the S3 storage backend with the prefix in the ConfigMap, when you create and run a pipeline, it should automatically use the S3 prefix you specified in the ConfigMap.