HttpClient w/HttpMessageHandler data concurrency issue

Question

I have a microservice that integrates with a third-party service via API calls. To handle the integration, I've created a typed HttpClient with a HttpMessageHandler registered as a transient service to set headers based on the incoming request (as headers are specific to the caller of the microservice).

Here is the flow:

• Microservice receives a request in a controller, where headers (identifying the account - system and tenant) are passed
• Controller calls a typed HttpClient
• The HttpClient sends a request to the third-party service.
• The HttpMessageHandler determines the account information based on the received request and sets the headers for the outgoing request.
• All requests and responses, including the account info, are logged in the microservice’s database.

Problem: When we deployed the microservice to production and started handling multiple clients, we encountered a concurrency issue: data for Account X was logged, but the request to the third party was processed under Account Y.

To investigate this, I tried:

1. I logged the headers information (account) in the HttpMessageHandler before and directly after sending the request

2. I am using Serilog as a logger, and so I installed a NuGet package serilog-httpclient and I logged everything.

3. I have used singleton instance of HttpClient directly from the controller, and used it in one of the APIs.

The first and second ways always yielded the same result, i.e. before and after the request sent to the third party, the headers were correct, but the third party responds with an error that the data doesn't exists in the account, while it actually does. However, when I went with the third solution, the issue stopped happening in that particular API, but the other APIs still encounters the same issue.

I’m unsure if the issue is due to concurrency within my code or something else. The logs consistently show the correct headers, but somehow, the request reaches the third-party service with incorrect account information. Could this still be a concurrency issue within my service, or is it possible the problem lies with the third-party service?

Here is my code:

Startup.cs

        services.AddScoped<IMySession, MySession>();

        services.AddScoped<IAccountService, AccountService>();

        services.AddTransient<MyAuthenticationMessageHandler>();

        services.AddHttpClient<MyAPI>(c =>
        {
            c.BaseAddress = new Uri(configuration["MyAPIUrl"]);
        })
        .LogRequestResponse(p =>
        {
            p.LogMode = LogMode.LogAll;
            p.RequestHeaderLogMode = LogMode.LogAll;
            p.RequestBodyLogMode = LogMode.LogAll;
            p.ResponseHeaderLogMode = LogMode.LogAll;
            p.ResponseBodyLogMode = LogMode.LogAll;
            p.MaskedProperties.Clear();
        })           
        .AddHttpMessageHandler<MyAuthenticationMessageHandler>();

MySession.cs

    public class MySession : IMySession
{
    private readonly IHttpContextAccessor _httpContextAccessor;

    public MySession(IHttpContextAccessor httpContextAccessor)
    {
        _httpContextAccessor = httpContextAccessor;
    }

    public int TenantId
    {
        get
        {
            return int.Parse(_httpContextAccessor.HttpContext.User.FindFirst(Consts.ClaimTenantId).Value);
        }
    }

    public int SystemId
    {
        get
        {
            return int.Parse(_httpContextAccessor.HttpContext.User.FindFirst(Consts.ClaimSystemId).Value);
        }
    }
}

MyController.cs

    public class MyController : MyControllerBase
{
    private readonly MyAPI _myAPI;

    public MyController(MyAPI myAPI)
        : base(serviceProvider)
    {
        _myAPI = myAPI;
    }

    [HttpGet("Status")]
    public async Task<CallResult<int?>> GetStatusAsync(int myId)
    {
        try
        {
            var response = await _myAPI.GetContractAsync();

            //rest of the code
        }
        catch (Exception ex)
        {
            return PrepareFailedCallResult<int?>(ex);
        }
    }
}

MyAuthenticationMessagehandler.cs

    public class MyAuthenticationMessageHandler : DelegatingHandler
{
    private readonly IHttpContextAccessor _httpContextAccessor;

    public MyAuthenticationMessageHandler(IHttpContextAccessor httpContextAccessor)
    {
        _httpContextAccessor = httpContextAccessor;
    }

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var accountService = _httpContextAccessor.HttpContext.RequestServices.GetRequiredService<IAccountService>();
        var accountInfo = await accountService.GetAccountInfo();

        request.Headers.Add(APIKeys.APIId, accountInfo.APIId);
        request.Headers.Add(APIKeys.APIKey, accountInfo.APIKey);
        request.Headers.Add(APIKeys.APIAuthorizationToken, accountInfo.APIAuthorizationToken);

        return await base.SendAsync(request, cancellationToken);
    }
}

ApplicationServiceBase.cs

public abstract class ApplicationServiceBase
{
    protected readonly IMySession Session;

    public ApplicationServiceBase(IMapper mapper, IServiceProvider serviceProvider)
    {
        Session = serviceProvider.GetService<ITajeerSession>();
    }
}

AccountService.cs

    public class AccountService : ApplicationServiceBase, IAccountService
{
    private readonly IRepository<Account> _accountRepository;

    public AccountService(IRepository<Account> accountRepository, IMapper mapper, IServiceProvider serviceProvider) : base(mapper, serviceProvider)
    {
        _accountRepository = accountRepository;
    }

    public async Task<AccountInfoDto> GetAccountInfo()
    {
        Expression<Func<Account, bool>> predicate =
            account =>
                account.TenantId == Session.TenantId &&
                account.SystemId == Session.SystemId;

        var account = (await _accountRepository.All(predicate)).Select(a => _mapper.Map<AccountInfoDto>(a)).FirstOrDefault();
        if (account == null)
            throw new Exception($"Account with TenantId:{Session.TenantId}, SystemId:{Session.SystemId} is invalid");

        return account;
    }
}

Answer 1

The code I wrote turned out to be non-erroneous and would never cause a data concurrency issue. The issue was from the service provider that I was integrating with. The admit that they had a concurrency issue, and after a while, they resolved it. Problem solved.