Search code examples
dockercontainersload-balancingdocker-swarmdocker-ingress

Docker node level load balancing not working

I have two laptops, Ubuntu-14 and Mac (Big Sur) and both of them have docker (with swarm support) installed in it.

  • I used Ubuntu as my Swarm manager (and) Mac as my worker node.
  • Ubuntu private IP is 192.168.0.14 (and) Mac private IP is 192.168.0.11 [Private IP can be shared in public without any issues because every class C network has the same IP's :P]
  • "docker swarm init --advertise-addr" was the command I used to make my Ubuntu host, the Manager (and) I entered the join command in Mac to make the Mac node to join the Swarm as worker.

So, On a highlevel, I used docker-compose.yml (which has only 1 python webservice). Using the compose file, I started a "docker stack" and then replicated the "python webservice" instance to 5. All these actions were carried out in Manager node.

  • Ubuntu Manager Node (also had 2 container instance and behaved as worker) (and) Mac Node had 3 container instances of the "python webservice". I have set up "ports" to be "80:1234" which means If I hit the port 80 of the host machine, It will redirect to the "python application webservice port" which is 1234 running inside the container.

  • When I hit the Manager IP (192.168.0.14:80) some 50 times and when I checked the logs of all the 5 containers in both Mac and Ubuntu, I found all the 2 containers in Ubuntu, got 25 hits each (in a round robin fashion) BUT, I couldn't find any logs for any of the containers present in Mac machine.

Is this an expected behavior?

  1. Only when I hit the IP address (192.168.0.11:80) of the Mac machine (worker) directly, I was able to get the logs/request hits for the containers present in the Mac machine.

So, there is two types of load balancing happening here,

  1. When I hit the IP:port (of a worker/manager), then, only the containers present in that worker/manager machine will be load balanced and served in a round-robin fashion (I can see that's the algorithm used). Let's name this load balancing type as "Container level load balancer"

  2. But, When I hit 192.168.0.14 (Manager IP), I expected the load will be balanced across all the 5 containers which is deployed across 2 nodes. Somehow this didn't work. Let's call it "Node level load balancer"

I have tried searching a lot in google for this but found nothing. Most sites are using external technologies like Nginx, HaProxy load balancers for solving "Node level load balancer".

Isn't there an out of the box support for this by docker itself?

EDIT 1 - Added docker-compose.yml as Metin asked in comment section

docker-compose.yml

version: '3'

services:
    webservice:
        image: python_ws_test
        ports:
            - '80:1234'
        command: ["python", "app.py"]
Solution

  • The main issue was, I tried to join a Linux node and a Mac node, because the docker for Mac (only SWARM, I think) is kind of broken as mentioned,

    Mac Private IP is 192.168.0.11, but somehow, 192.168.65.3 is the IP taken for the Mac worker node.

    How did I find out?

    Point 1

    => I made Mac as my Manager using "swarm init" command without any "advertise-addr" or "lister-addr" or etc.. The "docker swarm join" command I got had the IP address = 192.168.65.3. I don't know why because my Mac host IP is 192.168.0.11. This is not expected behaviour.

    => I did the same in Ubuntu to make my Ubuntu as Manager using "swarm init" raw command and the "docker swarm join" command I got had the IP address = 192.168.0.14 which is the same IP of the ubuntu host machine and that is expected behaviour.

    Point 2

    Once the stack is deployed, I tried to inspect the overlay network that's used using "docker network inspect $networkName". Linux manager node had the peers as "itself" and 192.168.65.3 which was unreachable because my Mac node's IP is 192.168.0.11. But somehow, when I auto-scaled by using "scale" command in Manager node (Ubuntu), docker manager was able to scale the containers in both Mac and Ubuntu. This is very odd.

    Default Overlay network - behaviour

    Also, "docker stack deploy" by default creates an overlay ingress network irrespective of whether you mention in docker-compose.yml or deploy command. Docker manager and nodes communicate between them on top of this network.

    Answer to the issue mentioned in question

    Docker has out of the box support for "Node level load balancing"? Yes! I was so frustrated about this odd behaviour in Mac, I installed Ubuntu 20.04 VM in my Mac and tried to use "Ubuntu 14.04" (separate Laptop / Base OS) as Manager and Ubuntu 20.04 (Virtual Machine OS) as worker node. Now, I was able to load balance between two nodes (I was getting hits in worker node), although I kept hitting the IP of manager only.

    I'll update why Mac is broken here if I get more insights. Anyone who already has the knowledge about this, please share.