Call function with Semantic kernel and Ollama and llama3.1

Question

This question is a continuation of Semantic Kernel plugin not working with ollama and phi3

Now that Ollama does support tools I have revisited my experiment.

I have stumbled on something I can't get my head around.

The following code is a compilation of two attempts:

#pragma warning disable SKEXP0010
#pragma warning disable SKEXP0001
async Task Main()
{
    var kernelBuilder = Kernel.CreateBuilder()
            .AddOpenAIChatCompletion(                        // We use Semantic Kernel OpenAI API
                modelId: "llama3.1", //phi3
                apiKey: null,
                endpoint: new Uri("http://localhost:11434")) // With Ollama OpenAI API endpoint
            ;
        
    kernelBuilder.Plugins.AddFromType<LightPlugin>();
        
    var kernel = kernelBuilder.Build();
    
    var history = new ChatHistory();

    var chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();

    OpenAIPromptExecutionSettings openAIPromptExecutionSettings = new()
    {
        ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions
    };

    
    history.AddUserMessage("Turn on the light");

    var result1 = await chatCompletionService.GetChatMessageContentAsync(
            history,
            executionSettings: openAIPromptExecutionSettings,
            kernel: kernel);
            
    Console.WriteLine("Assistant 1 > " + result1.Content);  
    
    StringBuilder builder = new();

    await foreach (var message in chatCompletionService.GetStreamingChatMessageContentsAsync(history, kernel: kernel, executionSettings: openAIPromptExecutionSettings))        
    {
        builder.Append(message.Content);
    }

    var result2 = builder.ToString();
    Console.WriteLine("Assistant 2 > " + result2);
}

public class LightPlugin
{
    public bool IsOn { get; set; } = false;

#pragma warning disable CA1024 // Use properties where appropriate
    [KernelFunction]
    [Description("Gets the state of the light.")]
    public string GetState() => IsOn ? "on" : "off";
#pragma warning restore CA1024 // Use properties where appropriate

    [KernelFunction]
    [Description("Changes the state of the light.'")]
    public string ChangeState(bool newState)
    {
        this.IsOn = newState;
        var state = GetState();

        Console.WriteLine($"[Light is now {state}]");

        return state;
    }
}

Here are two methods that should be usable for the same purpose: GetChatMessageContentAsync and GetStreamingChatMessageContentsAsync.

In most cases, the first one actually calls the function, but the second returns such a JSON structure as a response: {"name": "LightPlugin-ChangeState", "parameters": {"newState": true}}. It seems to me, that it depends on how the response from the underlying service is segmented. If it is received in one chunk, it is executed. If not, then not. I suppose it is a bug, not a feature. Or am I missing something?

Answer 1

You may try adding a system prompt to the settings to tell the model it can actually use plugins.

OpenAIPromptExecutionSettings settings = new OpenAIPromptExecutionSettings()
{
    ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions,
    Temperature = .1,
    ChatSystemPrompt = @"
Assistant is a large language model. 
This assistant uses plugins to interact with the software."
};

Also I needed to turn down the temperature for more consistent results. This seems to be even more important than the system prompt.

To improve the quality of the answers that used the plugins it also helps to specify what system the plugins interact with (e.g. "with the home automation" instead of just "the software").