Concurrent Api call to OpenAI is returning empty response

Question

I need to process a batch of video items, format them, and call an asynchronous API to get additional data in a YAML format. However, I'm encountering an issue where the API responses are mostly empty, despite the tasks completing successfully. Below is the relevant code.

var fetchTasks = videos.Select(async item =>
{
    try
    {
        var formattedItem = new VideoItem
        {
            Title = item.Title,
            Subtitle = _extractService.GetCleanSubtitle(item.Subtitle ?? ""),
            Description = _extractService.GetCleanSubtitle(item.Description ?? ""),
        };

        if (string.IsNullOrEmpty(formattedItem.Subtitle) && string.IsNullOrEmpty(formattedItem.Description))
        {
            _logger.LogWarning($"Video {item.Id} has empty subtitle or description. Marking as not processed.");
            item.NotProcess = true;
            return new { item, processStatus = false, request = string.Empty, response = (string)null };
        }
        else
        {
            var builder = new RequestBuilder(formattedItem).WithYamlFormatPath(yamlPath);
            var yamlRequest = builder.Build();

            _logger.LogInformation($"Built YAML request: {yamlRequest}");

            var response = await _apiService.CallApiAsync(yamlRequest);

            _logger.LogInformation($"Response for video {item.Id}: {response}");

            return new { item, processStatus = true, request = yamlRequest, response };
        }
    }
    catch (Exception ex)
    {
        _logger.LogError($"Error fetching details for video {item.Id}: {ex.Message}");
        return null;
    }
});

var fetchedDetails = (await Task.WhenAll(fetchTasks)).Where(result => result != null);

CallApiAsync Method

public async Task<string> CallApiAsync(string response)
{
    try
    {
        var completionResult = await _openAiService.ChatCompletion.CreateCompletion(new ChatCompletionCreateRequest
        {
            Messages = new List<ChatMessage>
            {
                ChatMessage.FromSystem("You are a helpful assistant."),
                ChatMessage.FromUser(response)
            },
            Model = Models.Gpt_3_5_Turbo,
        });
        if (completionResult.Successful)
        {
            Console.WriteLine(completionResult.Choices.First().Message.Content);
            return completionResult.Choices.First().Message.Content;
        }
        return "";
    }
    catch (Exception ex)
    {
        // Log or handle exception as neede
        Console.WriteLine($"Error calling ChatGPT API: {ex.Message}");
        throw;
    }
}

After the task has been completed to process the response.

foreach (var item in fetchedDetails)
{
    var video = item.video;
    var request = item.request;
    var response = item.response;
     var dbResponse = new Response
     {
         Request = request,
         YtVideoId = item .Id,
         Response = response
     };
}

Here while saving the response to the database I am often getting the item.response as empty string ("").

When I put breakpoint and debug in can see the empty string "".

It seems like the call to api has been started but not awaited properly within the batch. The tasks appear to complete without waiting for the response from CallApiAsync.

Does it might be the issue that

var completionResult = await _openAiService.ChatCompletion.CreateCompletion(new ChatCompletionCreateRequest

this code from https://github.com/betalgo/openai/blob/master/OpenAI.SDK/Managers/OpenAIChatCompletions.cs could be the reason of empty response?

How can I refactor this code to wait to the external API call to get the response correctly and process it in parallel?

Answer 1

The reason for the empty value on response was caused by Open Ai itself.

I was getting the following error while making the API call, which I hadn't noticed at first.

Rate limit reached for gpt-3.5-turbo in organization on tokens per min (TPM): Limit 60000, Used 57140, Requested 4647. Please try again in 1.787s. Visit platform.openai.com/account/rate-limits to learn more.

Which means technically It won't allow concurrent operation.