c#postgresql docker-compose docker-swarm docker-stack

Exception with connecting to database which runs on separate host in docker stack

I have a trouble with networking in docker compose which runs in docker stack environment. Firt of all I have a project on .NET with dockerized Postgres database which runs in another container. I wrote this two images to docker-compose file (it's content you can see below). If I run this app on the one host everything is fine and app works correctly! But when I try to use docker-compose on multiplie hosts it raises this exception:

2022-12-22 22:44:24 Unhandled exception. System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000001, 11): Resource temporarily unavailable
2022-12-22 22:44:24    at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, ValueStopwatch stopwatch)
2022-12-22 22:44:24    at System.Net.Dns.GetHostAddresses(String hostNameOrAddress, AddressFamily family)
2022-12-22 22:44:24    at Npgsql.Internal.NpgsqlConnector.Connect(NpgsqlTimeout timeout)
2022-12-22 22:44:24    at Npgsql.Internal.NpgsqlConnector.RawOpen(SslMode sslMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken, Boolean isFirstAttempt)
2022-12-22 22:44:24    at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|203_1(NpgsqlConnector conn, SslMode sslMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken, Boolean isFirstAttempt)
2022-12-22 22:44:24    at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
2022-12-22 22:44:24    at Npgsql.UnpooledDataSource.Get(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
2022-12-22 22:44:24    at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|45_0(Boolean async, CancellationToken cancellationToken)
2022-12-22 22:44:24    at Npgsql.NpgsqlConnection.Open()
2022-12-22 22:44:24    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)
2022-12-22 22:44:24    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternal(Boolean errorsExpected)
2022-12-22 22:44:24    at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
2022-12-22 22:44:24    at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlDatabaseCreator.Exists(Boolean async, CancellationToken cancellationToken)
2022-12-22 22:44:24    at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlDatabaseCreator.Exists(Boolean async, CancellationToken cancellationToken)
2022-12-22 22:44:24    at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlDatabaseCreator.Exists()
2022-12-22 22:44:24    at Microsoft.EntityFrameworkCore.Storage.RelationalDatabaseCreator.EnsureCreated()
2022-12-22 22:44:24    at Microsoft.EntityFrameworkCore.Infrastructure.DatabaseFacade.EnsureCreated()
2022-12-22 22:44:24    at TimeManager.DAO.UserContext..ctor(IConfiguration configuration) in /src/TimeManager.DAO/UserContext.cs:line 30
2022-12-22 22:44:24    at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
2022-12-22 22:44:24    at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitDisposeCache(ServiceCallSite transientCallSite, RuntimeResolverContext context)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitIEnumerable(IEnumerableCallSite enumerableCallSite, RuntimeResolverContext context)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type serviceType)
2022-12-22 22:44:24    at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType)
2022-12-22 22:44:24    at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
2022-12-22 22:44:24    at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
2022-12-22 22:44:24    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
2022-12-22 22:44:24    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
2022-12-22 22:44:24    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
2022-12-22 22:44:24    at Microsoft.AspNetCore.Builder.WebApplication.Run(String url)
2022-12-22 22:44:24    at Program.<Main>$(String[] args) in /src/TimeManager.API/Program.cs:line 51

I guess that this exception somehow based on networking troubles in my docker compose but I don't really understand how to deal with that problem. Swarm only throw container with error from one host to another. The original idea of using stack was to have the database on a separate host but this trouble bothers me. Thank you in advance!

My docker-compose.yml file:

version: '3.4'

services:
    timemanagerapi:
        image: ****/timemanagerapi:latest
        restart: always
        ports:
            - 80:80
        depends_on:
            - postgresserver
    postgresserver:
        image: postgres
        environment:
            - POSTGRES_PASSWORD=******
        volumes:
            - pgdata:/var/lib/postgresql/data     
        ports:
            - 5432:5432


volumes:
  pgdata:

I tried to add parameter deploy to my service in docker-compose.yml but it does not help. In internet I found that I need to add --network host but according to the official docker documentation this feature does not work with Docker Desktop on Windows which I use so I can't use this method

Solution

You Compose file contains the depends_on entry. When you deploy using Docker Compose (docker-compose up -d), this controls the order in which containers are started. In your case, the database container will start first, and then your C# app container.

However, the depends_on entry is ignored when deploying through Docker Swarm (docker stack deploy ...).

When your C# app boots up, you have no way of knowing if the database is fully ready to accept requests or not, so you need to account for it.

Implement a retry mechanism for testing your connection to the database when your app is starting up. As an example, the Polly library can assist with that.
As a quick-and-dirty method, just sleep (or await Task.Delay) for a fixed amount of time to wait for the database to be ready. Try starting at 30 seconds. If this works, you can read the docker logs and adjust the time as needed.