Running batch predictions with fine-tuned PALM model in Vertex AI always throws ValueError

Question

For my code:

import vertexai
from vertexai.preview.language_models import TextGenerationModel

model = TextGenerationModel.get_tuned_model("projects/PROJECT_ID/locations/us-central1/models/MODEL_ID")
batch_file=f'gs://my_bucket/test_.jsonl'
batch_file_output=f'gs://my_bucket/response'
batch_prediction_job = model.batch_predict(
  dataset=[batch_file],
  destination_uri_prefix=batch_file_output,,
  model_parameters={
      "maxOutputTokens": "300",
      "temperature": "0.0",
      "topP": "0.95",
      "topK": "1",
  }
)

does not work and throws this error:

ValueError                                Traceback (most recent call last)
/tmp/ipykernel_1/123140971.py in <module>
     12       "temperature": "0.0",
     13       "topP": "0.95",
---> 14       "topK": "1",
     15   },
     16 )

~/.local/lib/python3.7/site-packages/vertexai/language_models/_language_models.py in batch_predict(self, destination_uri_prefix, dataset, model_parameters, **_kwargs)
    430             dataset=dataset,
    431             destination_uri_prefix=destination_uri_prefix,
--> 432             model_parameters=model_parameters,
    433         )
    434 

~/.local/lib/python3.7/site-packages/vertexai/language_models/_language_models.py in batch_predict(self, dataset, destination_uri_prefix, model_parameters)
    381         # TODO(b/284512065): Batch prediction service does not support
    382         # fully qualified publisher model names yet
--> 383         publishers_index = model_name.index("/publishers/")
    384         if publishers_index > 0:
    385             model_name = model_name[publishers_index + 1 :]

ValueError: substring not found

That comment above the error (the TODO) is pretty interesting. Seems to imply it isn't supported.

is there any way to run batch prediction mode with a fine tuned PALM model?

Note this similar code works without any errors.

text_model = TextGenerationModel.from_pretrained("text-bison")
batch_prediction_job = text_model.batch_predict(
  dataset=[batch_file],
  destination_uri_prefix=batch_file_output,
  model_parameters={
      "maxOutputTokens": "300",
      "temperature": "0.0",
      "topP": "0.95",
      "topK": "1",
  },
)

But the big difference is the non-fine-tuned model.

I am on ver 1.30.1 of google-cloud-aiplatform

Answer 1

From my ticket to Google support on aug 29 2023:

I have a confirmation from the product engineers that we don't support fine-tuned model for batch prediction.

When asked if it would be added in the future:

We don't have plan for fine tuned model yet. Hence, unfortunately I don't have any ETA whether this would be considered in the future.

The answer is that this is not currently supported by google and may or may not be in the future.