I am currently using Vertex AI's Multimodal Embedding Model (https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-multimodal-embeddings)
I was able to get the Image and Text examples running just fine using the Python SDK and POST, however, when I try to run the following Video example (from the above link), I get the following error
import vertexai
from vertexai.vision_models import MultiModalEmbeddingModel, Video
project_id = 'project_name'
location = 'us-central1'
vertexai.init(project=project_id, location=location)
# Document metadata
video_path = 'gs://test-public-bucket-123/dog_jumping_short.mp4' # public small 1 MB 7 second video file
description = 'dogs jumping'
model = MultiModalEmbeddingModel.from_pretrained("multimodalembedding@001")
video = Video.load_from_file(video_path)
embeddings = model.get_embeddings(
video=video
)
# Video Embeddings are segmented based on the video_segment_config.
print("Video Embeddings:")
for video_embedding in embeddings.video_embeddings:
print(
f"Video Segment: {video_embedding.start_offset_sec} - {video_embedding.end_offset_sec}"
)
print(f"Embedding: {video_embedding.embedding}")
print(f"Text Embedding: {embeddings.text_embedding}")
The error I get when running this sample code is related to Deadline
Traceback (most recent call last):
File "/Users/me/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 76, in error_remapped_callable
return callable_(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/me/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__
return _end_unary_response_blocking(state, call, False, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/me/Code/multi-modal-test/multi-modal/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAUTHENTICATED
details = "Video embedding failed with the following error: Deadline"
debug_error_string = "UNKNOWN:Error received from peer ipv4:142.250.81.234:443 {grpc_message:"Video embedding failed with the following error: Deadline", grpc_status:16, created_time:"2024-05-23T15:53:15.429704-04:00"}"
Note that I have been able to embed contextual_text
and image
just fine with the same authentication, so I am fairly certain this has nothing to do with authentication even though the error says so.
I have also tried to POST using the following cURL, but I get the same error response Deadline
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://us-central1-aiplatform.googleapis.com/v1/projects/multi/locations/us-central1/publishers/google/models/multimodalembedding@001:predict"
I have the following APIs enabled (note that according to docs, I only need to have Vertex AI API enabled)
These are my IAM permissions
- members:
- user:[email protected]
role: roles/aiplatform.admin
- members:
- user:[email protected]
role: roles/aiplatform.user
- members:
- user:[email protected]
role: roles/ml.admin
- members:
- user:[email protected]
role: roles/owner
- members:
- user:[email protected]
role: roles/storage.admin
- members:
- user:[email protected]
role: roles/visionai.admin
etag: BwYaJsGtzOk=
version: 1
I am not using a Service Account, but a User Account.
Anyone know what I can do here?
This might come from a discrepancy between the "current project" of gcloud, and the "Application Defaults Credentials". This can be confusing because the exact order of these commands matter.
To make sure, could you try in this order:
1.
gcloud config set project MY_PROJECT_ID
(doc)
2.
gcloud auth application-default login
(doc)