I have a task to connect to a third party API and I'm having a bit of a conundrum that I can't seem to solve. The owner of the third party API insists it is an issue on my end because other clients are able to connect to them just fine.
For a brief overview, we connect to them for financial data. The have a single endpoint running in apache that accepts an xml payload. This payload determines the data they send back as an xml file inside a zip.
We are running in a .NET 6 API. Any process that does a single request to their server works just fine. In fact, in a singular request to our API, I can make 2 calls into their system. The third call, however, no matter what it is, always fails.
EDIT: This failure happens both running locally in IIS on my dev box and in Azure where we host the service.
When we were validating the xml using a remote DTD on their server it would get the DTD, perform the request for data using the validated xml, and then on the second request getting the DTD would always return a 400 error. I got around this by pulling the DTD locally and loading it as an embedded resource instead. Now, the first two calls for data will succeed, but any calls after that will not. Our current process needs to make 4 calls for data.
public static async Task<byte[]> PostToABCServiceAsync(this IRestHelper restHelper, RestRequest request, string xml, CancellationToken cancellationToken = default)
{
var content = new CapturedXmlContent(xml, "text/xml");
return await ((ABCRestHelper)restHelper).ExecuteAsync(request,
(flrl) => flrl
.WithHeader("Content-Type", "text/xml")
.PostAsync(content, cancellationToken)
.ReceiveBytes());
}
We have integration tests at multiple levels of the code (only the calls to their service, the code that orchestrates the multiple calls, and the outer endpoint for the process that needs this data) that call into their API and get results without issue. These tests always run green and get valid data from their service. Every time without fail. It's only when a request to their service is made through our api that it fails.
We have verified on multiple occasions that the xml payload we are sending for all requests is valid and returns data when they use internal tools to test the xml. The raw xml generated can be put into postman and sent over with no issue.
If we change the order of the calls it is always the third call that fails, even if it would otherwise succeed. The real kick in the pants here is they have a way to request the data for multiple clients at a time. When we do this it still fails with the same 400 error. It's two calls at this point for the data, not 3, and the second call always fails, no matter the order.
We have tried setting max connections to more than 2 but that did not work. We have tried using a newly instantiated and immediately disposed HttpClient with 100 max connections on the handler. We tried setting 100 max connections in a web.config. We have tried changing our DI scopes/lifetimes. We have tried eliminating our application layer by going from the controller directly to their service.
None of it works and none of it makes sense because the integration test run green and return valid data. It's only when making the call from within the code in our API that it seems to not work.
We enabled http tracing on app service and have discovered that even though the url is https, the underlying http client appears to be forcing us to port 80 - even when we put port 443 in the url. I found this post with the same issue, but their fix did not work for me.
I've also added logging events in flurl to detect if they send us on a redirect and that is not happening either.
The status code we get back from their server is always a 400 error and always html markup from apache.
EDIT:
Here are some samples from our rest library which we use to connect to a half dozen other apis just fine, including connecting within our own ecosystem.
The entry point:
public async Task<HttpResponseMessage> PostAsync(RestRequest request, CancellationToken cancellationToken = default)
{
var content = request.Body as HttpContent;
if (content == null && request.Body != null)
{
throw new ArgumentException("The request Body must inherit from System.Net.Http.HttpContent.");
}
var response = await ExecuteAsync(request,
(flrl) => flrl.PostAsync(content, cancellationToken));
return response.ResponseMessage;
}
The execute method:
public async Task<TResponse> ExecuteAsync<TResponse>(RestRequest request,
Func<IFlurlRequest, Task<TResponse>> flurlRequest)
{
//build the URL first so we can render it how we want as far as encoding goes
var requestUrl = request.BaseUri.AbsoluteUri
.AppendPathSegments(request.Segments);
if (request.QueryParameters?.Any() ?? false)
{
requestUrl = requestUrl.SetQueryParams(request.QueryParameters);
}
if (request.Port.HasValue)
{
requestUrl.Port = request.Port;
}
using (var client = new FlurlClient(requestUrl.ToString(request.EncodeSpaceAsPlus)))
{
if (request.Settings != null)
{
client.Configure(request.Settings);
}
if (request.BeforeCallAsync != null)
{
client.Settings.BeforeCallAsync = request.BeforeCallAsync;
}
if (request.AfterCallAsync != null)
{
client.Settings.AfterCallAsync = request.AfterCallAsync;
}
if (request.OnRedirectAsync != null)
{
client.Settings.OnRedirectAsync = request.OnRedirectAsync;
}
if (request.OnErrorAsync != null)
{
client.Settings.OnErrorAsync = request.OnErrorAsync;
}
client.AllowHttpStatus(string.Join(",", request.AllowHttpStatuses));
client.HttpClient.DefaultRequestHeaders.Clear();
client.Headers.Clear();
if (request.Headers?.Any() ?? false)
{
client.WithHeaders(request.Headers);
}
if (request.Timeout > 0)
{
client.WithTimeout(request.Timeout);
}
try
{
if (_throttle != null)
{
return await _throttle.Enqueue(() => flurlRequest(client.Request()));
}
else
{
return await flurlRequest(client.Request());
}
}
catch (FlurlHttpTimeoutException e)
{
throw new CustomRestTimeoutException(e.Call?.Request?.Url?.ToString(), e);
}
catch (FlurlParsingException e)
{
string responseContent = null;
if (e.Call?.Response != null)
{
try
{
responseContent = await e.Call?.Response?.GetStringAsync();
}
catch (Exception)
{
//eat all errors for pulling the response content
}
}
throw new CustomRestParsingException(e.Call?.Request?.Url?.ToString(), responseContent, e);
}
catch (FlurlHttpException e)
{
int? statusCode = null;
string responseErrorMessage = (e.Call?.Exception?.Message) ?? e.Message;
string responseContent = null;
if (e.Call?.HttpResponseMessage != null)
{
//if the call fails, the response might be null
statusCode = (int?)e.Call.HttpResponseMessage.StatusCode;
try
{
responseContent = await e.Call.HttpResponseMessage.Content.ReadAsStringAsync();
}
catch (Exception)
{
//eat all errors for pulling the response content
}
}
throw new CustomRestException(statusCode.HasValue ? (int)statusCode.Value : 0,
e.Call?.Request?.Url?.ToString(), responseErrorMessage, responseContent, e);
}
}
}
and just to show usage, here's how we build the request:
CapturedXmlContent content = null;
try
{
content = new CapturedXmlContent(_serializer.Serialize(query, false), "text/xml");
}
catch (Exception e)
{
//log writing omitted
throw;
}
var request = _requestFactory.GetRequest()
.WithHeader("Content-Type", "text/xml")
.WithBody(content)
.OnBeforeCallAsync(LogBeforeCallAsync)
.OnAfterCallAsync(LogAfterCallAsync)
.OnRedirectAsync(LogRedirectAsync)
.OnErrorAsync(LogErrorAsync)
.Build();
try
{
var start = DateTime.UtcNow;
using var response = await _restHelper.PostAsync(request, cancellationToken);
var end = DateTime.UtcNow;
return await ProcessZipResponseAsync<DataRequestDto, DataResponseDto>(query, response, attempt, start, end);
}
catch (Exception e)
{
//log writing omitted
throw;
}
UPDATE
We have new findings. If we use an Azure Function app and run the process as a Durable Function, then we can connect to their server just fine. Same with integration test that run in the nunit runner.
If we try to make a function endpoint that calls them as part of a request, we can make one single request to their service before all other requests that are part of the request into our API/Function app begin to fail.
After much trial and error, including rewriting our usage of flurl and other changes to try to make sure no headers were set that we didn't intend to set, we discovered that if we moved into an Azure function app and ran the api call as a Durable Function, it worked, but when we ran as a "synchronous" function it failed with a 400 after the first call.
It appears something with the http context is being passed along to the client causing a 400 error. We could not narrow down exactly what it was, but as a test we decided to run the http call in a new thread to leave behind the context.
This code fixes the issue:
Task<HttpResponseMessage> queryTask;
using (ExecutionContext.SuppressFlow())
{
queryTask = Task.Run<HttpResponseMessage>(() => _restHelper.PostAsync(request, cancellationToken), cancellationToken);
Task.WaitAll(queryTask);
}
using var response = queryTask.Result;
We still are unsure WHY this fixes it or what else could be done to stop the passthrough of headers or cookies or whatever was causing the 400 error (would have to be happening outside flurl, but as part of the API or Function app processing of outgoing http requests).