Got "ERROR:root:Failed to get healthz info attempt 1 of 5." when deploy pipeline in the Kubeflow

I am trying to follow the Kubeflow v2 quickstart.

First, I deployed Kubeflow to a local Kubernetes cluster by

export PIPELINE_VERSION="2.0.0-alpha.4"

kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=$PIPELINE_VERSION"

I port forwarded by

kubectl port-forward service/ml-pipeline-ui --namespace=kubeflow 38620:80

I can see the UI at http://localhost:38620

Next, I installed kfp 2.0.1. And here is my code:

from kfp import client, dsl


@dsl.component
def addition_component(num1: int, num2: int) -> int:
    return num1 + num2


@dsl.pipeline(name="addition-pipeline")
def my_pipeline(a: int, b: int, c: int = 10):
    add_task_1 = addition_component(num1=a, num2=b)
    add_task_2 = addition_component(num1=add_task_1.output, num2=c)


endpoint = "http://localhost:38620"  # <- Not entirely sure if it is correct as it is missing in the quickstart document.
kfp_client = client.Client(host=endpoint)
run = kfp_client.create_run_from_pipeline_func(
    my_pipeline,
    arguments={"a": 1, "b": 2},
)
url = f"{endpoint}/#/runs/details/{run.run_id}"
print(url)

However, I got error

python src/main.py

/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp/client/client.py:158: FutureWarning: This client only works with Kubeflow Pipeline v2.0.0-beta.2 and later versions.
  warnings.warn(
ERROR:root:Failed to get healthz info attempt 1 of 5.
Traceback (most recent call last):
  File "/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp/client/client.py", line 435, in get_kfp_healthz
    return self._healthz_api.get_healthz()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp_server_api/api/healthz_service_api.py", line 63, in get_healthz
    return self.get_healthz_with_http_info(**kwargs)  # noqa: E501
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp_server_api/api/healthz_service_api.py", line 134, in get_healthz_with_http_info
    return self.api_client.call_api(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp_server_api/api_client.py", line 364, in call_api
    return self.__call_api(resource_path, method,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp_server_api/api_client.py", line 188, in __call_api
    raise e
  File "/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp_server_api/api_client.py", line 181, in __call_api
    response_data = self.request(
                    ^^^^^^^^^^^^^
  File "/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp_server_api/api_client.py", line 389, in request
    return self.rest_client.GET(url,
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp_server_api/rest.py", line 230, in GET
    return self.request("GET", url,
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp_server_api/rest.py", line 224, in request
    raise ApiException(http_resp=r)
kfp_server_api.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'X-Powered-By': 'Express', 'Content-Security-Policy': "default-src 'none'", 'X-Content-Type-Options': 'nosniff', 'Content-Type': 'text/html; charset=utf-8', 'Content-Length': '159', 'Date': 'Mon, 03 Jul 2023 03:59:33 GMT', 'Connection': 'keep-alive', 'Keep-Alive': 'timeout=5'})
HTTP response body: <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /apis/v2beta1/healthz</pre>
</body>
</html>

Inside for endpoint, I am using http://localhost:38620, However, I am not entirely sure if I am using correct one as it is missing in quickstart document.

I tried to access

http://localhost:38620/apis/v2beta1/healthz
http://localhost:38620/#/apis/v2beta1/healthz

and they do not exist.

I found a similar issue at https://github.com/kubeflow/kubeflow/issues/5989 but there is not any helpful info inside.

Any guide would be appreciate!

Solution

UPDATE: I have opened a pull request to update the document. Today should be good.

I had a closer look of the Python script log. It has a line:

FutureWarning: This client only works with Kubeflow Pipeline v2.0.0-beta.2 and later versions.

It turns out the Kubeflow v2 quickstart is out-of-date.

After I replaced 2.0.0-alpha.4 to 2.0.0 (You can find latest version at https://github.com/kubeflow/pipelines/releases?q=Version) and redeploy by

export PIPELINE_VERSION="2.0.0"

kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=$PIPELINE_VERSION"

Now the pipeline script succeed:

python src/main.py

/Users/hongbo-miao/Library/Caches/pypoetry/virtualenvs/hm-kubeflow-calculate-PriecqfA-py3.11/lib/python3.11/site-packages/kfp/client/client.py:158: FutureWarning: This client only works with Kubeflow Pipeline v2.0.0-beta.2 and later versions.
  warnings.warn(
Experiment details: http://localhost:38620/#/experiments/details/c9d82bc3-712e-422a-804f-fcbe5d2b8acb
Run details: http://localhost:38620/#/runs/details/87398dae-6488-446b-b650-7bbc580aab56
http://localhost:38620/#/runs/details/87398dae-6488-446b-b650-7bbc580aab56

The health endpoint http://localhost:38620/apis/v2beta1/healthz also returns

{
  "buildDate": "Tue Jun 20 16:56:00 UTC 2023",
  "frontendCommitHash": "e03e31219387b587b700ba3e31a02df486aa364f",
  "frontendTagName": "2.0.0",
  "apiServerReady": true,
  "apiServerCommitHash": "e03e31219387b587b700ba3e31a02df486aa364f",
  "apiServerTagName": "2.0.0",
  "apiServerMultiUser": false,
  "multi_user": false
}