Vertex AI endpoint 500 Internal Server Error

Question

I tried to deploy a custom container to Vertex AI endpoint using LLM model (PaLM), the container is successfully deployed to the endpoint with the following code and dockerfile. But when I tried to query it with Vertex AI API or gcloud cli, I get a 500 Internal Server Error reply.

May I know what's the cause of this error?

Am I using the right way to deploy the model?

Python Code

import uvicorn

#import tensorflow as tf
import os
import numpy as np
#from enum import Enum
#from typing import List, Optional
#from pydantic import BaseModel

from fastapi import Request, FastAPI, Response
from fastapi.responses import JSONResponse

from langchain.vectorstores.matching_engine import MatchingEngine
from langchain.agents import Tool
from langchain.embeddings import VertexAIEmbeddings
from vertexai.preview.language_models import TextGenerationModel

embeddings = VertexAIEmbeddings()

INDEX_ID = "<index id>"
ENDPOINT_ID = "<index endpoint id>"
PROJECT_ID = '<project name>'
REGION = 'us-central1'
DOCS_BUCKET='<bucket name>'
TEXT_GENERATION_MODEL='text-bison@001'

def matching_engine_search(question):

    vector_store = MatchingEngine.from_components(
                        index_id=INDEX_ID,
                        region=REGION,
                        embedding=embeddings,
                        project_id=PROJECT_ID,
                        endpoint_id=ENDPOINT_ID,
                        gcs_bucket_name=DOCS_BUCKET)

    relevant_documentation=vector_store.similarity_search(question, k=8)
    context = "\n".join([doc.page_content for doc in relevant_documentation])[:10000] #[:10000]
    return str(context)

app = FastAPI(title="Chatbot")

AIP_HEALTH_ROUTE = os.environ.get('AIP_HEALTH_ROUTE', '/health')
AIP_PREDICT_ROUTE = os.environ.get('AIP_PREDICT_ROUTE', '/predict')

#class Prediction(BaseModel):
#  response: str 


@app.get(AIP_HEALTH_ROUTE, status_code=200)
async def health():
    return {'health': 'ok'}

@app.post(AIP_PREDICT_ROUTE)#, 
          #response_model=Predictions,
          #response_model_exclude_unset=True
async def predict(request: Request):
    body = await request.json()
    print(body)

    question = body["question"]

    matching_engine_response=matching_engine_search(question)

    prompt=f"""
    Follow exactly those 3 steps:
    1. Read the context below and aggregrate this data
    Context : {matching_engine_response}
    2. Answer the question using only this context
    3. Show the source for your answers
    User Question: {question}


    If you don't have any context and are unsure of the answer, reply that you don't know about this topic.
    """

    model = TextGenerationModel.from_pretrained(TEXT_GENERATION_MODEL)
    response = model.predict(
            prompt,
            temperature=0.2,
            top_k=40,
            top_p=.8,
            max_output_tokens=1024,
    )

    print(f"Question: \n{question}")
    print(f"Response: \n{response.text}")


    outputs = response.text

    return {"predictions": [{"response": response.text}] }#Prediction(outputs)

if __name__ == "__main__":
  uvicorn.run(app, host="0.0.0.0",port=8080)

Docker file

FROM tiangolo/uvicorn-gunicorn-fastapi:python3.8-slim
RUN pip install --no-cache-dir google-cloud-aiplatform==1.25.0 langchain==0.0.187 xmltodict==0.13.0 unstructured==0.7.0 pdf2image==1.16.3 numpy==1.23.1 pydantic==1.10.8 typing-inspect==0.8.0 typing_extensions==4.5.0
COPY main.py ./main.py

Cloudbuild.yaml

steps:
# Build the container image
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'gcr.io/<project name>/chatbot', '.']
# Push the container image to Container Registry
- name: 'gcr.io/cloud-builders/docker'
  args: ['push', 'gcr.io/<project name>/chatbot']

images:
- gcr.io/<project name>/chatbot

Code to query the model endpoint

from google.cloud import aiplatform

aiplatform.init(project=PROJECT_ID,
                location=REGION)

instances = [{"question": "<Some question>"}]

endpoint = aiplatform.Endpoint("projects/<project id>/locations/us-central1/endpoints/<model endpoint id>")

prediction = endpoint.predict(instances=instances)
print(prediction)

Error message

Answer 1

As mentioned in the document, the internal errors are usually transient and trying to resend the request might resolve the issue. If the error still persists, you can contact support or you can open a new thread on the issue tracker describing your issue.