Search code examples
pythongoogle-cloud-vertex-aikfp

Enable web access or interactive shell for PipelineJob tasks for Vertex AI

I am trying to debug a PipelineJob that I launch on Vertex AI. Is there a way to enable web access on the components like you can when you launch Custom Jobs? This way, I could ssh into the running task and do a bunch of debugging.

Here is a simplified version of my pipeline code:

import kfp.v2.dsl as dsl

from google.cloud import aiplatform
from kfp.v2 import compiler
from kfp.v2.dsl import (
    component,
    Input,
    Output,
    Dataset,
    Metrics,
    Model,
    Artifact,
    graph_component
)
from copy import copy
from kfp.v2.google.client import AIPlatformClient
from typing import Optional, Dict, Union, List

@component(
    packages_to_install=['google-cloud-aiplatform']
)
def hello_world():
    import time
    print("Hello world")
    time.sleep(300)

@dsl.pipeline(
    name = "dataprep"
)
def train_model_pipeline(style: int):
    # Set Up Training and Test Data
    hello_op = hello_world()

I expected to be able to set enable_web_access(True) on the task, but that doesn't seem like an option because it's part of the CustomJob spec and not the PipelineTask.

Solution

  • As of now, there is no feature or any workaround to enable web access to kfp components.

    There is already a filed feature request regarding this issue filed by @Optimus.

    Posting the answer as community wiki for the benefit of the community that might encounter this use case in the future.

    Feel free to edit this answer for additional information.