Search code examples
c#.netdockerazure-table-storageazurite

How do configure azure table storage in local docker container?

I am trying to set up a repository class that will create a table if it doesn't exist in my azure storage database. Here's the constructor:

    public TableStorageRepository(string storageAccountConnectionString, string tableName)
    {

        var options = new TableClientOptions
        {
            Retry =
            {
                MaxRetries = 5,
                Delay = TimeSpan.FromSeconds(2),
                MaxDelay = TimeSpan.FromSeconds(10),
                Mode = RetryMode.Exponential
            }
        };
       var serviceClient = new TableServiceClient(storageAccountConnectionString, options);
        _tableClient = serviceClient.GetTableClient(tableName);
        _tableClient.CreateIfNotExists();
    }

Here's the logic that calls it / set ups DI in Program.cs:

builder.Services.AddScoped<ITableStorageRepository<DeploymentDto>>(provider =>
{
    var connectionString = builder.Configuration["AzureTableStorage:ConnectionString"]; //use appsettings.[Development].json
    var tableName="Deployment";
    return new TableStorageRepository<DeploymentDto>(connectionString, tableName);
});

When I try to trigger a method on my controller the GETs deployments, the repository logic is trigger but it fails when it tries the CreateifNotExists() method. The error is:

System.AggregateException: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (An error occurred while sending the request.) (An error occurred while sending the request.) (An error occurred while sending the request.) (An error occurred while sending the request.) Azure.RequestFailedException: An error occurred while sending the request. System.Net.Http.HttpRequestException: An error occurred while sending the request. System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded) at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Azure.Core.Pipeline.HttpClientTransport.ProcessSyncOrAsync(HttpMessage message, Boolean async) End of inner exception stack trace --- at Azure.Core.Pipeline.HttpClientTransport.ProcessSyncOrAsync(HttpMessage message, Boolean async) at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) End of inner exception stack trace ---

Any suggestions would be appreciated.

I tried to change the code so instead of a connection string, it passes individual parameters. Soething like this:

    public TableStorageRepository(string accountName, string accountKey, string tableName, string endpointUri = null)
    {
        var serviceUri = endpointUri != null ? new Uri(endpointUri) : new Uri($"https://{accountName}.table.core.windows.net");
        var credential = new TableSharedKeyCredential(accountName, accountKey);
        _tableClient = new TableClient(serviceUri, tableName, credential);
                // Ensure the table exists asynchronously
        _tableClient.CreateIfNotExistsAsync().GetAwaiter().GetResult();
    }

I get the same results - which makes sense because I think the connection itself is being made correctly. At least, thats my interpretation .

The next thing i started to check out was to see if there's a way to try to use the api to call database methods directly, just to see if the storage system responds.
I have the following section in docker-compose.yml:

azureTableStorage:
    image: mcr.microsoft.com/azure-storage/azurite:latest #Todo: replace with a real version once everyting is working
    container_name: azureTableStorage
    ports:
      - "10000:10000" # Blob service
      - "10001:10001" # Queue service
      - "10002:10002" # Table service      
    command: "azurite-table --loose --location /data --debug /dev/stdout"
    volumes:
      - .azureTableStorage/data:/azureTableStorage/data
    environment:
      - AZURITE_ACCOUNTS=devstoreaccount1:Eby8vdM02xNOcqFevkbPZRvz1FclJ7t3YvsZ6Eby8vdM02xNOcqFevkbPZRvz1FclJ7t3YvsZ6 #sdk expects base64.  But docker file just expects plain

The container starts up no problems. But ... I'm not able to run any curl commands inside the container (or outside of it) to see the table data.

Here's what I tried inside the container:

</Error>/opt/azurite # curl -X GET \
>   -H "Authorization: SharedKey devstoreaccount1:RWJ5OHZkTTAyeE5PY3FGZXZrYlBaUnZ6MUZjbEo3dDNZdnNaNkVieTh2ZE0wMnhOT2NxRmV2a2JQWlJ2ejFGY2xKN3QzWXZzWjY=" \
>   -H "Accept: application/json;odata=fullmetadata" \
>   -H "x-ms-version: 2019-02-02" \
>   http://127.0.0.1:10002/devstoreaccount1/Tables
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
  <Code>AuthorizationFailure</Code>
  <Message>Server failed to authenticate the request. Make sure the value of the Authorization header is formed correctly including the signature.
RequestId:5376245b-4fd1-4db9-992f-6bb88c1c658c
Time:2025-02-10T19:22:24.822Z</Message>
</Error>/opt/azurite #

The key value i'm sending is the value i get back from this powershell command:
[Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes("Eby8vdM02xNOcqFevkbPZRvz1FclJ7t3YvsZ6Eby8vdM02xNOcqFevkbPZRvz1FclJ7t3YvsZ6"))

EDIT 1

I have followed the instructions.

  1. Updated my docker-compose to include the environment variable. Screenshot below:
  2. My Program.cs looks like this:
    enter image description here
  3. The correct connnection string is being passed from the appsettings.Development.json: enter image description here

But the constructor fails with the error:

Exception has occurred: CLR/System.FormatException An exception of type 'System.FormatException' occurred in System.Private.CoreLib.dll but was not handled in user code: 'The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.'
at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength) at System.Convert.FromBase64String(String s) at Azure.Data.Tables.TableSharedKeyCredential.SetAccountKey(String accountKey) at Azure.Data.Tables.TableSharedKeyCredential..ctor(String accountName, String accountKey) at Azure.Data.Tables.TableConnectionString.GetCredentials(ConnectionString settings) at Azure.Data.Tables.TableConnectionString.ParseInternal(String connectionString, TableConnectionString& accountInformation, String& error) at Azure.Data.Tables.TableConnectionString.Parse(String connectionString) at Azure.Data.Tables.TableClient..ctor(String connectionString, String tableName, TableClientOptions options) at Azure.Data.Tables.TableClient..ctor(String connectionString, String tableName) at Provider.Gateway.Infrastructure.Storage.TableStorage.TableStorageRepository`1..ctor(String connectionString, String tableName) in /Users/me/src/projects/provider-gateway/Provider.Gateway/Infrastructure/Storage/TableStorageRepository.cs:line 46

EDIT 2

I've updated my appsettings.Development.json file to look like this:

 {
     "AzureTableStorage": {
         "ConnectionString": "UseDevelopmentStorage=true"
     }
 }

And now when I run i don't get any errors. I can POST and GET. But the last thing I need to document for the team is a way to "see" the data in the container, outside the context of our web application. So I downloaded azure storage explorer.

And I tried to create a connection to a local storage container. But i think it's connecting to my development box (host) instance of azurite and not the one running in my container.

I've proven this to myself by debugging my app, GETting the list of deployments. Then I do a docker compose down on all my containers and I redo the GET and it still works. I opened the azure extension in vscode and I can see that it was creating it locally on my dev box ("host" machine to the container).

In case it helps, when I do a docker ps I get the following:

CONTAINER ID   IMAGE                                            COMMAND                  CREATED         STATUS                   PORTS                                       NAMES
c13cdb5f263b   ghcr.io/open-webui/open-webui:main               "bash start.sh"          2 minutes ago   Up 2 minutes (healthy)   0.0.0.0:3000->8080/tcp                      apg-open-webui-1
a9d70b8d1600   ollama/ollama:0.5.7                              "/bin/ollama serve"      2 minutes ago   Up 2 minutes             0.0.0.0:11434->11434/tcp                    apg-ollama-1
a61983d56ed4   mcr.microsoft.com/azure-storage/azurite:latest   "docker-entrypoint.s…"   2 minutes ago   Up 2 minutes             10000-10001/tcp, 0.0.0.0:10002->10002/tcp   azureTableStorage

The other artifact I found that proves its not writing to the container is that i have a azurite_db_table.json file in the root of the project folder instead of inside the mapped volume defined in the docker file for this container.

EDIT 3

Your suggestion for the volumes tag in the docker-compose is giving me errors - likely I'm not using the correct syntax:

enter image description here

Solution

  • I think the root cause of the issue was that I had azurite running locally on my "host machine". but I also had it containerized. When I start debugging in vscode, it would automatically start the local instance on the host machine because in appsettings.Development.json I had the following:

    {
    "AzureTableStorage": {
        "ConnectionString": "UseDevelopmentStorage=true"
    }
}

    I replaced that with:

    {
    "AzureTableStorage": {
      "ConnectionString": "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;TableEndpoint=http://localhost:10002/devstoreaccount1"
    }
  }

    Now when I start vscode, it doesn't automatically start the azure table service on my host. I can tell because on the status bar at the bottom of vscode, it doesn't show the service running.

    And now when i create a new "deployment" record in the database using swagger, I can prove that it's updating the container data using the tip from the other answer:

    /opt/azurite # cat /data/__azurite_db_table__.json | grep Deployment
{"filename":"/data/__azurite_db_table__.json","collections":[{"name":"$TABLES_COLLECTION$","data":[{"account":"devstoreaccount1","table":"Deployment","meta":{"revision":0,"created":1739802123425,"version":0},"$loki":1}],"idIndex":null,"binaryIndices":{"account":{"name":"account","dirty":false,"values":[0]},"table":{"name":"table","dirty":false,"values":[0]}},"constraints":null,"uniqueNames":[],"transforms":{},"objType":"$TABLES_COLLECTION$","dirty":false,"cachedIndex":null,"cachedBinaryIndex":null,"cachedData":null,"adaptiveBinaryIndices":true,"transactional":false,"cloneObjects":false,"cloneMethod":"parse-stringify","asyncListeners":false,"disableMeta":false,"disableChangesApi":true,"disableDeltaChangesApi":true,"autoupdate":false,"serializableIndices":true,"disableFreeze":true,"ttl":null,"maxId":1,"DynamicViews":[],"events":{"insert":[],"update":[],"pre-insert":[],"pre-update":[],"close":[],"flushbuffer":[],"error":[],"delete":[null],"warning":[null]},"changes":[],"dirtyIds":[]},{"name":"$SERVICES_COLLECTION$","data":[],"idIndex":null,"binaryIndices":{},"constraints":null,"uniqueNames":["accountName"],"transforms":{},"objType":"$SERVICES_COLLECTION$","dirty":false,"cachedIndex":null,"cachedBinaryIndex":null,"cachedData":null,"adaptiveBinaryIndices":true,"transactional":false,"cloneObjects":false,"cloneMethod":"parse-stringify","asyncListeners":false,"disableMeta":false,"disableChangesApi":true,"disableDeltaChangesApi":true,"autoupdate":false,"serializableIndices":true,"disableFreeze":true,"ttl":null,"maxId":0,"DynamicViews":[],"events":{"insert":[],"update":[],"pre-insert":[],"pre-update":[],"close":[],"flushbuffer":[],"error":[],"delete":[null],"warning":[null]},"changes":[],"dirtyIds":[]},{"name":"devstoreaccount1$Deployment","data":[{"PartitionKey":"test","RowKey":"test","properties":{"PartitionKey":"test","RowKey":"test","Id":"test","Provider":"AppsAIGateway","Endpoint":"https://mytest.com","Version":"string","Timestamp":"2025-02-17T14:36:11.9931129Z","Timestamp@odata.type":"Edm.DateTime"},"lastModifiedTime":"2025-02-17T14:36:11.9931129Z","eTag":"W/\"datetime'2025-02-17T14%3A36%3A11.9931129Z'\"","meta":{"revision":0,"created":1739802971998,"version":0},"$loki":1}],"idIndex":null,"binaryIndices":{"PartitionKey":{"name":"PartitionKey","dirty":false,"values":[0]},"RowKey":{"name":"RowKey","dirty":false,"values":[0]}},"constraints":null,"uniqueNames":[],"transforms":{},"objType":"devstoreaccount1$Deployment","dirty":false,"cachedIndex":null,"cachedBinaryIndex":null,"cachedData":null,"adaptiveBinaryIndices":true,"transactional":false,"cloneObjects":false,"cloneMethod":"parse-stringify","asyncListeners":false,"disableMeta":false,"disableChangesApi":true,"disableDeltaChangesApi":true,"autoupdate":false,"serializableIndices":true,"disableFreeze":true,"ttl":null,"maxId":1,"DynamicViews":[],"events":{"insert":[],"update":[],"pre-insert":[],"pre-update":[],"close":[],"flushbuffer":[],"error":[],"delete":[null],"warning":[null]},"changes":[],"dirtyIds":[]}],"databaseVersion":1.5,"engineVersion":1.5,"autosave":true,"autosaveInterval":5000,"autosaveHandle":null,"throttledSaves":true,"options":{"persistenceMethod":"fs","autosave":true,"autosaveInterval":5000,"serializationMethod":"normal","destructureDelimiter":"$<\n"},"persistenceMethod":"fs","persistenceAdapter":null,"verbose":false,"events":{"init":[null],"loaded":[],"flushChanges":[],"close":[],"changes":[],"warning":[]},"ENV":"NODEJS"}
/opt/azurite #