Chat Playground, View Code Issue With GPT4o and Multiple attachments

Using Azure OpenAI Chat Playground and I went to get the code for my conversation, but the user messages are missing. I uploaded two jpgs and asked for a transcription and a translation, and received the correct output, but I wanted to review the code it generated so that I could get an example of how the attachments are sent.

Why are the user messages missing, and is this a bug, or is there something I can do to see the SDK implementation of the messages that were sent?

// Note: The Azure OpenAI client library for .NET is in preview.
// Install the .NET library via NuGet: dotnet add package Azure.AI.OpenAI --prerelease
using Azure;
using Azure.AI.OpenAI;

OpenAIClient client = new OpenAIClient(
  new Uri("https://[our endpoint].openai.azure.com/"),
  new AzureKeyCredential(Environment.GetEnvironmentVariable("AZURE_OPENAI_API_KEY")));

  Response<ChatCompletions> responseWithoutStream = await client.GetChatCompletionsAsync(
  "GPT4o",
  new ChatCompletionsOptions()
  {
    Messages =
    {
      new ChatMessage(ChatRole.System, @"You are an AI assistant that helps people find information."),
      new ChatMessage(ChatRole.Assistant, @"json
{
  \"TranscribedText\": \"[Transcribed text was all here, and very good.]",
  
  \"TranslatedText\": \"[Translation was here, and was very good as well.]"
}

/* I asked for another translation, just to see if there was a glitch generating the user messages with just one user message but it isn't included either. */
"),      new ChatMessage(ChatRole.Assistant, @"json
{
  \"TranscribedText\": \"[Trascribed text was given here.]",
  
  \"TranslatedText\": \"[New translation was given here.]\"
}
"),
    },
    Temperature = (float)0.7,
    MaxTokens = 800,
    NucleusSamplingFactor = (float)0.95,
    FrequencyPenalty = 0,
    PresencePenalty = 0,
  });


ChatCompletions response = responseWithoutStream.Value;

I am using a deployment of GPT4o.

As an alternative, I opened up the browser developer tools to see what the site was sending, and see that the images are sent as dataurls (I see that in the call), but I was hoping for the C# code using the Azure OpenAI SDK to see how it is done through that route.

Solution

I cannot answer the first part of the question as to why the Azure OpenAI interface is not showing the UserMessages when there are attachments, but I did confirm when I do not use the GPT4o deployment, and use just a GPT4 deployment, I do see the sent user messages:

// Note: The Azure OpenAI client library for .NET is in preview.
// Install the .NET library via NuGet: dotnet add package Azure.AI.OpenAI --prerelease
using Azure;
using Azure.AI.OpenAI;

OpenAIClient client = new OpenAIClient(
  new Uri("https://[our endpoint].openai.azure.com/"),
  new     AzureKeyCredential(Environment.GetEnvironmentVariable("AZURE_OPENAI_API_KEY"))    );

  Response<ChatCompletions> responseWithoutStream = await client.GetChatCompletionsAsync(
  "Assistants_Playground_Test",
  new ChatCompletionsOptions()
  {
    Messages =
    {
      new ChatMessage(ChatRole.System, @"You are an AI assistant that helps people find information."),
      **new ChatMessage(ChatRole.User, @"Please provide a translation for \"Hola\" from Spanish to English"),**      new ChatMessage(ChatRole.Assistant, @"\"Hola\" translates from Spanish to English as \"Hello\"."),
},
Temperature = (float)0.7,
MaxTokens = 800,
NucleusSamplingFactor = (float)0.95,
FrequencyPenalty = 0,
PresencePenalty = 0,
});


ChatCompletions response = responseWithoutStream.Value;

From the following MS Learn document, I was able to answer the overall question, of how do you send images with your User ChatMessage calls:

If you want to use a local image, you can use the following Python code to convert it to base64 so it can be passed to the API

https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/gpt-with-vision?tabs=rest%2Csystem-assigned%2Cresource

Then I found this SO answer showing how to create a dataurl in C#:

return "<img src="data:image/" + Path.GetExtension(imgFile).Replace(".","") + ";base64," + Convert.ToBase64String(File.ReadAllBytes(imgFile)) + "" />";

how to convert Image to Data URI for Html with C#?

The DataUrl ended up being too long though, and as noted here: https://github.com/Azure/azure-sdk-for-net/pull/43830 I was able to use the call:

ChatMessageImageContentItem is augmented with two new constructors: one accepts a BinaryData for the image and a matching string for the image format's MIME type (like image/png);

The C# code that ended up working for me was:

// Note: The Azure OpenAI client library for .NET is in preview.
// Install the .NET library via NuGet: dotnet add package Azure.AI.OpenAI --prerelease
using Azure;
using Azure.AI.OpenAI;
using Azure.Core;

BinaryData binaryData = BinaryData.FromBytes(File.ReadAllBytes(filePath));
var imageContent = new ChatMessageImageContentItem(binaryData, "image/jpg");

OpenAIClient client = new OpenAIClient(
  new Uri("https://[my endpoint].openai.azure.com/"),
  new AzureKeyCredential("[my key]"));

Response<ChatCompletions> responseWithoutStream = await client.GetChatCompletionsAsync(
new ChatCompletionsOptions()
{
    Messages =
  {
      new ChatRequestSystemMessage(@"You are an AI assistant that helps people find information."),
      new ChatRequestUserMessage(new ChatMessageContentItem[]{ imageContent,new ChatMessageTextContentItem("Please transcribe and then translate the transcription from English to Spanish.") })},
    Temperature = (float)0.7,
    MaxTokens = 800,
    DeploymentName = "GPT4o",
    NucleusSamplingFactor = (float)0.95,
    FrequencyPenalty = 0,
    PresencePenalty = 0,
    });


    ChatCompletions response = responseWithoutStream.Value;
    Console.WriteLine(response.Choices[0].Message.Content);