Search code examples
c#httpclientc#-ziparchive

HttpClient SendAsync with ArchiveZip in multi threaded environment

//------ BELOW IS THE ORIGINAL QUESTION BUT THE SOLUTION was connected with usage of the ZipArchive class to get multiple streams to multiple files and use them in multiple threads. Real issue and answer to it in the "Answer" post. //------------

I have quite a complicated case... I am calling my own API from other API. I am doing this 6 times at once. During first call I need to acquire an Access token and cache it so that all other requests can use it. If I will do first call synchronously and then rest 5 asynchronously at once - everything works. But all 6 at once - I receive an error from SendAsync (with access token generated correctly):

The archive entry was compressed using an unsupported compression method.

If I remove SemaphoreSlim from my code and all 6 call will generate token on its own (because all will not be able to get it from cache) then it works too... The code for the token (this class is created for each request separately that is why semaphore is static):

public class AzureCredentialCached : ChainedTokenCredential
{
    private static readonly SemaphoreSlim _semaphore = new SemaphoreSlim(1, 1);

    public override async ValueTask<AccessToken> GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken = default)
    {
        AccessToken? cachedToken = _cacheService.TryGetToken(requestContext, _uri);
        if (cachedToken is null)
        {
            await _semaphore.WaitAsync(cancellationToken);
            try
            {
                cachedToken = _cacheService.TryGetToken(requestContext, _uri);
                if (cachedToken is null)
                {
                    cachedToken = await base.GetTokenAsync(requestContext, cancellationToken);
                    _cacheService.InsertNewToken((AccessToken)cachedToken, requestContext, _uri);
                }
            }
            finally
            {
                _semaphore.Release();
            }
        }

        return (AccessToken)cachedToken;
    }
}

Delegation handler:

public class StorageApiAuthorizeRequestHandler : DelegatingHandler
{
    private readonly IAzureCredentialCacheService _credentialCacheService;
    private readonly GeneralSection _generalConfig;
    private readonly StorageApiSection _storageConfig;

    public StorageApiAuthorizeRequestHandler(IAzureCredentialCacheService credentialCacheService, IOptions<StorageApiSection> storageConfig, IOptions<GeneralSection> generalConfig)
    {
        _credentialCacheService = credentialCacheService;
        _generalConfig = generalConfig.Value;
        _storageConfig = storageConfig.Value;
    }

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var credential = new AzureCredentialCached(_credentialCacheService, new Uri(_storageConfig.Address, UriKind.Absolute), new Azure.Core.TokenCredential[]
        {
            new ManagedIdentityCredential(_generalConfig.ManagedIdentityClientId),
            new VisualStudioCredential()
        });

        var scopes = new[] { _storageConfig.Scope };
        var accessToken = (await credential.GetTokenAsync(new Azure.Core.TokenRequestContext(scopes))).Token;

        request.Headers.Add(HeaderNames.Authorization, $"Bearer {accessToken}");

        try
        {
            return await base.SendAsync(request, cancellationToken);
        }
        catch (Exception ex)
        {
            throw;
        }
    }
}

enter image description here

The archive entry was compressed using an unsupported compression method.
 at System.IO.Compression.Inflater.Inflate(FlushCode flushCode)
   at System.IO.Compression.Inflater.ReadInflateOutput(Byte* bufPtr, Int32 length, FlushCode flushCode, Int32& bytesRead)
   at System.IO.Compression.Inflater.ReadOutput(Byte* bufPtr, Int32 length, Int32& bytesRead)
   at System.IO.Compression.Inflater.InflateVerified(Byte* bufPtr, Int32 length)
   at System.IO.Compression.DeflateStream.CopyToStream.<WriteAsyncCore>d__10.MoveNext()
   at System.IO.Stream.<<CopyToAsync>g__Core|29_0>d.MoveNext()
   at System.IO.Compression.DeflateStream.CopyToStream.<CopyFromSourceToDestinationAsync>d__6.MoveNext()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpContent.<<CopyToAsync>g__WaitAsync|56_0>d.MoveNext()
   at System.Net.Http.MultipartContent.<SerializeToStreamAsyncCore>d__23.MoveNext()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpContent.<<CopyToAsync>g__WaitAsync|56_0>d.MoveNext()
   at System.Net.Http.HttpConnection.<SendRequestContentAsync>d__70.MoveNext()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnection.<SendAsyncCore>d__64.MoveNext()
   at System.Net.Http.HttpConnection.<SendAsyncCore>d__64.MoveNext()
   at System.Net.Http.HttpConnectionPool.<SendWithVersionDetectionAndRetryAsync>d__84.MoveNext()
   at System.Net.Http.DiagnosticsHandler.<SendAsyncCore>d__8.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.RedirectHandler.<SendAsync>d__4.MoveNext()
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendAsync>d__5.MoveNext()
   at MyAPP.ApiAuthentication.StorageApiAuthorizeRequestHandler.<SendAsync>d__4.MoveNext() in C:\MyAPP\ApiAuthentication\StorageApiAuthorizeRequestHandler.cs:line 37
Solution

  • The issue was actually connected with what I was sending. We are using ZipArchive class to extract files from ZIP archive and upload them to another our API. So the flow was like that:

    1. I am opening stream to a file in ZIP

      var entry = _zipArchive.GetEntry(path); entry. Open();

    2. I am doing a SendAsync with this stream in the request

    3. Opening another completely new Stream to another file

    4. I am doing a SendAsync with this stream in the request ...

    5. In the meantime out second API starts and loads configurations. It receives multiple requests but because of Semaphores each one is waiting for the first one to load the APP configuration from Azure

    6. When it is done, it tries to pull streams from the first API at the same time

    7. Looks like even different streams from ZipArchive library cannot be used at the same time. SendAsync throws exceptions because I suppose those streams are decompressed on the fly and intersects somehow with each other.

    The solution was to copy all those streams to a separate MemoryStreams (and thanks to that they are decompressed one by one at first). Now all SendAsyncs work at the same time without any issue....