Endpoint on local API not working unless consumed endpoint is accessed first

Question

I am creating a prototype for a news aggregator web application, right now I am creating the backend for it where it processes endpoints from https://newsapi.org/

In Swagger UI I get this error that I've defined

The debugger shows that the response is 400 Bad request which is completely false when I grabbed the exact URL used for the request and pasted it into my browser/postman, which works, and in effect makes my swagger API work as well:

This is the content of await response.Content.ReadAsStringAsync()

"{\"status\":\"error\",\"code\":\"userAgentMissing\",\"message\":
\"Please set your User-Agent header to identify your application. Anonymous requests are not allowed.\"}"

Suddenly my Swagger UI starts working and returns the same response, I've noticed that this is persistent, meaning stopping and starting my WebApi keeps the ''activation'' for my consumed endpoint, but not if I change the parameters e.g from us to ua

This is how I'm accessing the endpoint from newsapi

 public async Task<NewsApiResponse> GetNewsAsync(string country, string category)
 {
     string url = $"https://newsapi.org/v2/top-headlines?country={country}&category={category}&apiKey={_apiKey}";

     var response = await _httpClient.GetAsync(url);
     response.EnsureSuccessStatusCode();

     var content = await response.Content.ReadAsStringAsync();
     var newsData = JsonConvert.DeserializeObject<NewsApiResponse>(content);

     // Check if the response status is 'error'
     if (newsData.Status == "error")
     {
         throw new HttpRequestException($"Error from News API: {newsData.ErrorMessage}");
     }

     return newsData;
 }

And this is my endpoint on my own backend.

[HttpGet("GetNews")]
public async Task<IActionResult> GetNews([FromQuery] string country, [FromQuery] string category)
{
    try
    {
        var news = await _newsService.GetNewsAsync(country, category);
        return Ok(news);
    }
    catch (Exception ex)
    {
        return StatusCode(500, "An error occurred while processing your request");
    }
}

In the beginning I thought it was a race condition issue because I used the debugger and copied the url to paste it in my browser to test it before stepping through further, but that was misleading apparently. Is this typical behavior for SwaggerUI and newsapi?

Answer 1

400 Bad Request could have lots of different reasons, so it can be hard to debug. Fortunately, NewsAPI offers a more detailed error description in their response. Quoting from their documentation

Error codes

When an HTTP error is returned we populate the code and message properties in the response containing more information

So by looking at the content of await response.Content.ReadAsStringAsync(), you will get closer to the root cause.

userAgentMissing

Please set your User-Agent header to identify your application. Anonymous requests are not allowed.

With this information you can find out what to do: You need to set a value for the User-Agent header in your HttpRequest. This header is described in RFC 9110 and generally contains information about the caller. If you send a request via your browser, it will automatically be set. But in SwaggerUI, this does not seem to be the case.

You can set the User-Agent header like this:

httpClient.DefaultRequestHeaders.Add("User-Agent", "My C# App");