In a Testcontainers (Java) context, I've noticed that Testcontainers fail to communicate with podman for some reason and hangs.
Here are the debug logs I can see:
[info] o.t.d.DockerClientProviderStrategy - Loaded org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy from ~/.testcontainers.properties, will try it first
[debug] o.t.d.RootlessDockerClientProviderStrategy - $XDG_RUNTIME_DIR is not set.
[debug] o.t.d.RootlessDockerClientProviderStrategy - '/opt/gitlab-runner/.docker/run' does not exist.
[debug] o.t.d.DockerClientProviderStrategy - Trying out strategy: RootlessDockerClientProviderStrategy
[debug] o.t.s.c.g.d.c.c.AbstrDockerCmd - Cmd:
[debug] c.g.d.z.s.o.a.h.c.h.i.c.InternalHttpClient - ex-00000001: preparing request execution
[debug] c.g.d.z.s.o.a.h.c.h.p.RequestAddCookies - Cookie spec selected: strict
[debug] c.g.d.z.s.o.a.h.c.h.p.RequestAuthCache - Auth cache not set in the context
[debug] c.g.d.z.s.o.a.h.c.h.i.c.ProtocolExec - ex-00000001: target auth state: UNCHALLENGED
[debug] c.g.d.z.s.o.a.h.c.h.i.c.ProtocolExec - ex-00000001: proxy auth state: UNCHALLENGED
[debug] c.g.d.z.s.o.a.h.c.h.i.c.ConnectExec - ex-00000001: acquiring connection with route {}->unix://localhost:2375
[debug] c.g.d.z.s.o.a.h.c.h.i.c.InternalHttpClient - ex-00000001: acquiring endpoint (3 MINUTES)
[debug] c.g.d.z.s.o.a.h.c.h.i.i.PoolingHttpClientConnectionManager - ex-00000001: endpoint lease request (3 MINUTES) [route: {}->unix://localhost:2375][total available: 0; route allocated: 0 of 2147483647; total allocated: 0 of 2147483647]
[debug] c.g.d.z.s.o.a.h.c.h.i.i.PoolingHttpClientConnectionManager - ex-00000001: endpoint leased [route: {}->unix://localhost:2375][total available: 0; route allocated: 1 of 2147483647; total allocated: 1 of 2147483647]
[debug] c.g.d.z.s.o.a.h.c.h.i.i.PoolingHttpClientConnectionManager - ex-00000001: acquired ep-00000000
[debug] c.g.d.z.s.o.a.h.c.h.i.c.InternalHttpClient - ex-00000001: acquired endpoint ep-00000000
[debug] c.g.d.z.s.o.a.h.c.h.i.c.ConnectExec - ex-00000001: opening connection {}->unix://localhost:2375
[debug] c.g.d.z.s.o.a.h.c.h.i.c.InternalHttpClient - ep-00000000: connecting endpoint (3 MINUTES)
[debug] c.g.d.z.s.o.a.h.c.h.i.i.PoolingHttpClientConnectionManager - ep-00000000: connecting endpoint to unix://localhost:2375 (3 MINUTES)
[debug] c.g.d.z.s.o.a.h.c.h.i.i.DefaultHttpClientConnectionOperator - http-outgoing-0: connecting to localhost/127.0.0.1:2375
[debug] c.g.d.z.s.o.a.h.c.h.i.i.DefaultHttpClientConnectionOperator - http-outgoing-0: connection established
[debug] c.g.d.z.s.o.a.h.c.h.i.i.PoolingHttpClientConnectionManager - ep-00000000: connected http-outgoing-0
[debug] c.g.d.z.s.o.a.h.c.h.i.c.InternalHttpClient - ep-00000000: endpoint connected
[debug] c.g.d.z.s.o.a.h.c.h.i.c.MainClientExec - ex-00000001: executing GET /v1.32/info HTTP/1.1
[debug] c.g.d.z.s.o.a.h.c.h.i.c.InternalHttpClient - ep-00000000: start execution ex-00000001
[debug] c.g.d.z.s.o.a.h.c.h.i.i.PoolingHttpClientConnectionManager - ep-00000000: executing exchange ex-00000001 over http-outgoing-0
[debug] c.g.d.z.s.o.a.h.c.h.headers - http-outgoing-0 >> GET /v1.32/info HTTP/1.1
[debug] c.g.d.z.s.o.a.h.c.h.headers - http-outgoing-0 >> x-tc-sid: 17aa0133-17d2-48fb-8d9c-0a6466724e15
[debug] c.g.d.z.s.o.a.h.c.h.headers - http-outgoing-0 >> User-Agent: tc-java/1.19.1
[debug] c.g.d.z.s.o.a.h.c.h.headers - http-outgoing-0 >> Accept-Encoding: gzip, x-gzip, deflate
[debug] c.g.d.z.s.o.a.h.c.h.headers - http-outgoing-0 >> Host: localhost:2375
[debug] c.g.d.z.s.o.a.h.c.h.headers - http-outgoing-0 >> Connection: keep-alive
[debug] c.g.d.z.s.o.a.h.c.h.wire - http-outgoing-0 >> "GET /v1.32/info HTTP/1.1[\r][\n]"
[debug] c.g.d.z.s.o.a.h.c.h.wire - http-outgoing-0 >> "x-tc-sid: 17aa0133-17d2-48fb-8d9c-0a6466724e15[\r][\n]"
[debug] c.g.d.z.s.o.a.h.c.h.wire - http-outgoing-0 >> "User-Agent: tc-java/1.19.1[\r][\n]"
[debug] c.g.d.z.s.o.a.h.c.h.wire - http-outgoing-0 >> "Accept-Encoding: gzip, x-gzip, deflate[\r][\n]"
[debug] c.g.d.z.s.o.a.h.c.h.wire - http-outgoing-0 >> "Host: localhost:2375[\r][\n]"
[debug] c.g.d.z.s.o.a.h.c.h.wire - http-outgoing-0 >> "Connection: keep-alive[\r][\n]"
[debug] c.g.d.z.s.o.a.h.c.h.wire - http-outgoing-0 >> "[\r][\n]"
How can I reproduce the requests done by testcontainers to troubleshoot the situation?
The requests are made against localhost:2375/v1.32/info in HTTP but over a Unix socket.
It happens that cURL support querying over a Unix socket with the following command:
curl -v --unix-socket /run/user/1000/docker.sock 'http://localhost:2375/v1.32/info'
In my case, this helped identify that one of my machines had podman incorrectly setup (due to incorrect ownership root vs. another user) as the above request would hang on this machine whereas all other machines would answer immediately.