Search code examples
minikubekubeflow-pipelines

issues when setting up kubeflow pipeline on minikube

I have a minikube running on macos. When trying to setup kubeflow pipeline I got the following output:

(base) ~/ml $ export PIPELINE_VERSION=1.7.0
(base) ~/ml $ kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
namespace/kubeflow created
customresourcedefinition.apiextensions.k8s.io/clusterworkflowtemplates.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workfloweventbindings.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io unchanged
serviceaccount/kubeflow-pipelines-cache-deployer-sa created
clusterrole.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-clusterrole unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubeflow-pipelines-cache-deployer-clusterrolebinding unchanged
unable to recognize "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=1.7.0": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=1.7.0": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=1.7.0": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"

(base) ~/ml $ kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "applications.app.k8s.io" not found
(base) ~/ml $  
(base) ~/ml $ kubectl get crd -A
NAME                                   CREATED AT
clusterworkflowtemplates.argoproj.io   2021-12-18T15:28:31Z
cronworkflows.argoproj.io              2021-12-18T15:28:31Z
workfloweventbindings.argoproj.io      2021-12-18T15:28:31Z
workflows.argoproj.io                  2021-12-18T15:28:31Z
workflowtemplates.argoproj.io          2021-12-18T15:28:31Z

In particular, what does it mean:

unable to recognize "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=1.7.0": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"

Is it the root cause for the following error:

Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "applications.app.k8s.io" not found

 (base) ~/ml $ minikube version
minikube version: v1.24.0
commit: 76b94fb3c4e8ac5062daf70d60cf03ddcc0a741b

(base) ~/ml $ kubectl api-resources --api-group=apiextensions.k8s.io -o wide
NAME                        SHORTNAMES   APIVERSION                NAMESPACED   KIND                       VERBS
customresourcedefinitions   crd,crds     apiextensions.k8s.io/v1   false        CustomResourceDefinition   [create delete deletecollection get list patch update watch]
Solution

  • It turns out the kubeflow pipeline 1.7.0 does not work with kubernetes version higher than 1.22. I used kubernetes 1.21.8 with minikube and there is no problem installing kubeflow pipeline 1.7.0.

    Yes, this is correct behaviour.

    You have mentioned:

    In particular, what does it mean:

    unable to recognize "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=1.7.0": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"

    This is connected directly to Kubernetes version 1.22 api changes:

    The v1.22 release will stop serving the API versions we've listed immediately below. These are all beta APIs that were previously deprecated in favor of newer and more stable API versions.

    • Beta versions of the ValidatingWebhookConfiguration and MutatingWebhookConfiguration API (the admissionregistration.k8s.io/v1beta1 API versions)
    • The beta CustomResourceDefinition API (apiextensions.k8s.io/v1beta1)
    • The beta APIService API (apiregistration.k8s.io/v1beta1)
    • The beta TokenReview API (authentication.k8s.io/v1beta1)
    • Beta API versions of SubjectAccessReview, LocalSubjectAccessReview, SelfSubjectAccessReview (API versions from authorization.k8s.io/v1beta1)
    • The beta CertificateSigningRequest API (certificates.k8s.io/v1beta1)
    • The beta Lease API (coordination.k8s.io/v1beta1)
    • All beta Ingress APIs (the extensions/v1beta1 and networking.k8s.io/v1beta1 API versions)

    As of version 1.22 it is not possible to use apiextensions.k8s.io/v1beta1(this API is no longer available) and if you want to install a pipeline using this type of API, you can only use Kubernetes version 1.21.