Is it possible from a gcp vertex job to hit an http endpoint in another gcp pod in the same project?

Question

I have a custom container vertex endpoint that is passed a url as input so that the job can call it to get a particular frame of data needed for the job. (gcs:// buckets do work) but I want to specifically use an http request to a server in the same gcp project.

I have tried setting the endpoint up as private using the --networks param on the endpoint but then get the message:

{
  "error": {
    "code": 400,
    "message": "Making request from public OnePlatform API is not allowed on a private Endpoint peered with network (projects/11111111111/global/networks/some-dev-project-vpc).",
    "status": "FAILED_PRECONDITION"
  }
}

when I try to hit that private vertex endpoint. I've tried curling it from within a running pod in the same project, but that didn't work either.

Is there a way to do this? Thanks

Answer 1

The error states that your request is to a public API, which may because you are using the public url schema to make your prediction. The structure of vertex endpoints differ between private and public, so double check that you are using the private endpoint url for your requests.

Public

https://{REGION}-aiplatform.googleapis.com/v1/projects/{PROJECT}/locations/{REGION}/endpoints/{ENDPOINT_ID}:predict

Private

http://{ENDPOINT_ID}.aiplatform.googleapis.com/v1/models/{DEPLOYED_MODEL_ID}:predict

You can generate a private endpoint url using the following gcloud command:

gcloud beta ai endpoints describe {ENDPOINT_ID} \
  --region=us-central1 \
  --format="value(deployedModels.privateEndpoints.predictHttpUri)"

More documentation on private endpoints can be found here.