Search code examples
linuxdockerpodman

Two rootless containers in same network could not connect

I am using podman to run the containers and have issue connecting from one container to port exposed by another container in same network.

My network info:

$ podman network ls
NETWORK ID    NAME                 DRIVER
da3b514ee40c  my-network           bridge
2f259bab93aa  podman               bridge

$ podman container inspect on container1:

"SandboxKey": "/run/user/1000/netns/netns-447e75e3-aa0e-6449-1c45-33c4af3baf84",
               "Networks": {
                    "my-network": {
                         "EndpointID": "",
                         "Gateway": "10.89.0.1",
                         "IPAddress": "10.89.0.24",
                         "IPPrefixLen": 24,
                         "IPv6Gateway": "",
                         "GlobalIPv6Address": "",
                         "GlobalIPv6PrefixLen": 0,
                         "MacAddress": "82:4c:6c:11:ac:64",
                         "NetworkID": "my-network",
                         "DriverOpts": null,
                         "IPAMConfig": null,
                         "Links": null,
                         "Aliases": [
                              "database",
                              "b81cd9ab1544"
                         ]

Container1 has exposed port:

 "NetworkMode": "bridge",
               "PortBindings": {
                    "6379/tcp": [
                         {
                              "HostIp": "127.0.0.1",
                              "HostPort": "6379"
                         }

$ podman container inspect on container2:

"SandboxKey": "/run/user/1000/netns/netns-c9ad66f3-d49d-d49a-4563-2de551197891",
               "Networks": {
                    "my-network": {
                         "EndpointID": "",
                         "Gateway": "10.89.0.1",
                         "IPAddress": "10.89.0.38",
                         "IPPrefixLen": 24,
                         "IPv6Gateway": "",
                         "GlobalIPv6Address": "",
                         "GlobalIPv6PrefixLen": 0,
                         "MacAddress": "a6:c9:e6:9d:5c:da",
                         "NetworkID": "my-network",
                         "DriverOpts": null,
                         "IPAMConfig": null,
                         "Links": null,
                         "Aliases": [
                              "49b7b13c02e7"
                         ]
                    }

Now when I try to connect to container1 from container2 using netcat I get connection refused:

[root@49b7b13c02e7 /]# nc -z -v 127.0.0.1 6379
Ncat: Connection refused.

Also I dont see any veth pairs getting created on host machine.

I remember when I used to use docker then veth pair will reflect on host machine but with podman I dont see any veth pair getting created as I start a new rootless container in podman.

Not sure what could have gone wrong, any insight where can I further look.

Solution

  • First of all thanks to @TelinovDmitri for pointing me to the right direction.

    Since I know now why things were not working hence would like to document the same.

    When ever we create 2 containers or more, with same custom network, like in my case by running both containers using command:

    $ podman run --rm --it --network my-network -p 6379:6379 container1 bash
$ podman run --rm --it --network my-network container2 bash

    This means they belong to the same network not that they have same network namespace. Network namespace is a different concept which I confused with being in same network.

    Some info on network namespace here: https://blog.scottlowe.org/2013/09/04/introducing-linux-network-namespaces/

    So once the above 2 containers get created we have environment something as below: enter image description here

    As we can see each network namespace has its own loopback address, and even the default namespace which is the namespace of the host machine has its own loopback addreess.

    Now lets suppose my container2 has network namespace ns2 as shown in diagram and container1 has network namespace ns1, thus container1 has binded port 6379 on its loopback interface inside its network namespace ns1.

    Hence when from container2 when we say connect to 127.0.0.1 6379 I get connection refused since in network namespace ns2 no one is listening on loopback interface on port 6379.

    Also now we could observe in the diagram that since both of the network namespaces are part of same network created by bridge br0 hence they can indeed communicate but for that we need to use the IP Address such that request can route properly to respective container. Since loopback address is non routable address.

    Therefore when I use IP address instead then things work fine:

    [root@87070263e84e /]# nc -z -v 10.89.0.24 6379 
Ncat: Version 7.91 ( nmap.org/ncat ) Ncat: Connected to 10.89.0.24:6379. 
Ncat: 0 bytes sent, 0 bytes received in 0.04 seconds.

    To add further even though the container1 binded the same port on host also using -p 6379:6379 still we cant access it using localhost inside container2 reason being same that even host has its own different default network namespace.

    All in all localhost inside container refers to its internal network namespace loopback interface. localhost inside container does not map to loopback interface on host or any other container unless explicitly done so.

    For rootless container in podman one can get access to loopback interface on host now with below setting, although I have not verified personally:

    Set:
slirp4netns:allow_host_loopback=true
in file:
$HOME/.config/containers/containers.conf