Can Google's Gemini models do auto function calling with complex parameters?

Question

I am using Google's gemini-1.5-pro-002 model to build a chat bot. The model is deployed on VertexAI. I need the bot to do auto function calling. It works if the function parameters are simple types such as string or integer. I am having issues getting it to work with complex types.

The following request does not work:

curl -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
https://${LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${LOCATION}/publishers/google/models/${MODEL_ID}:generateContent \
-d '{
    "contents": [
        {
            "role": "user",
            "parts": [
                {
                    "text": "Search for German people with the last name of Greene"
                }
            ]
        }
    ],
    "tools": [
        {
            "functionDeclarations": [
                {
                    "name": "SearchPlugin_SearchPerson",
                    "description": "Find a person given a search request",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "search_request": {
                                "type": "object",
                                "properties": {
                                    "first_name": {
                                        "type": "string",
                                        "description": "The person first_name"
                                    },
                                    "last_name": {
                                        "type": "string",
                                        "description": "The person last_name"
                                    },
                                    "nationality": {
                                        "type": "string",
                                        "description": "The person nationality"
                                    }
                                }
                            }
                        },
                        "required": [
                            "search_request"
                        ]
                    }
                }
            ]
        }
    ]
}'

Gemini replies with

{
    "candidates": [
        {
            "content": {
                "role": "model",
                "parts": [
                    {
                        "functionCall": {
                            "name": "SearchPlugin_SearchPerson",
                            "args": {
                                "search_request": "unknown"
                            }
                        }
                    }
                ]
            },
            "finishReason": "STOP"
        }
    ],
    "usageMetadata": {
        "promptTokenCount": 54,
        "candidatesTokenCount": 9,
        "totalTokenCount": 63
    },
    "modelVersion": "gemini-1.5-pro-002"
}

Gemini can use the tool correctly if I define the tool like this:

{
    "functionDeclarations": [
        {
            "name": "SearchPlugin_SearchPerson",
            "description": "Find a person given a search request.",
            "parameters": {
                "type": "object",
                "properties": {
                    "first_name": {
                        "type": "string",
                        "description": "The person first_name."
                    },
                    "last_name": {
                        "type": "string",
                        "description": "The person last_name."
                    },
                    "nationality": {
                        "type": "string",
                        "description": "The person nationality."
                    }
                },
                "required": [
                    "last_name"
                ]
            }
        }
    ]
}

Can Gemini call functions with complex parameters?

Answer 1

About your question Can Gemini call functions with complex parameters?, it's yes. But, in the current stage, the result depends on the model.

When I tested your showing request body with several models, I noticed that the result depends on the model. It seems that in the current stage (September 30, 2024), the following result is obtained. So, as the current workaround, if you want to directly use your showing request body, how about changing the model from gemini-1.5-pro-002 to another?

Result

model	Result
gemini-1.5-pro	failure
gemini-1.5-pro-001	failure
gemini-1.5-pro-latest	failure
gemini-1.5-pro-exp-0827	failure
gemini-1.5-pro-002	failure
gemini-1.5-flash	success
gemini-1.5-flash-001	success
gemini-1.5-flash-latest	success
gemini-1.5-flash-8b-exp-0827	failure
gemini-1.5-flash-002	failure
gemini-1.5-flash-8b-exp-0924	success

failure: "search_request": "unknown" was obtained from your request body. success: "search_request": {"last_name": "Greene","nationality": "German"} was obtained from your request body.

Note:

• Gemini is growing now. So, I believe that this result will be updated in the future update. For example, the current result of success might also be changed in the update. So, please think of this answer as of September 30, 2024.