NodeJS Remote Debugging with Docker

It's been two days, and I still can't get my head around this supposed-to-be-simple thing!

I want to:

Create a nodejs docker swarm service and run the nodejs code in the debugging mode
Expose the debugger port to host
Be able to connect to debugger from the host

The simplest steps to reproduce the issue:

root@host   docker service create --name dbg-nodejs --publish 4444:4444 node sleep 10d
root@host   containerID=$(docker container ls --filter name=dbg-nodejs | awk 'FNR == 2 {print $1}') ### find the containerId
root@host   docker exec -d $containerID /bin/bash -c "(echo \"console.log('\n\n\nHello from NodeJS\n\n\n');\" > /usr/test-dbg.js) && node --inspect-brk=4444 /usr/test-dbg.js"
root@host   docker exec $containerID /bin/bash -c "curl localhost:4444 -v" ### you see the output here! Great the debugging port is working within the container
root@host   curl localhost:4444 -v ### aaahhhhh <<--- CANNOT CONNECT TO the published 4444 port!!!!!!

The publishing port works fine. If I run a netcat listening to 4444 withing the container, it just works fine! Somehow node debugger is behaving differently. But it shouldn't! It's just TCP! :/

MORE INFO: So if instead of node --inspect-brk=4444 /usr/test-dbg.js I do netcat -l -p 4444, things work fine. i.e., with the default ingress network, I can connect from host to the container. Somehow the bahavior is different! They are both TCP.

Solution

Absolutely bizarre! But a port forwarding solved the issue. Everything is the same, except:

Instead of 4444, I used another port to attach nodejs debugger to, e.g., port #5555
I used socat to forward 4444 to 5555 using this command: socat -v tcp-listen:4444,reuseaddr,fork tcp:localhost:5555

Of course, you need to install socat on your docker container. But that's it!! Somehow the way Docker treat NodeJS debugger TCP is different from socat TCP! Perhaps, NodeJS debugger does not listen to port on all network interfaces and that makes it not being able to pick up the connection.

A fully working example:

root@host   docker service create --name dbg-nodejs --publish 4444:4444 node sleep 10d
root@host   containerID=$(docker container ls --filter name=dbg-nodejs | awk 'FNR == 2 {print $1}') ### find the containerId
root@host   docker exec -d $containerID /bin/bash -c "(echo \"console.log('\n\n\nHello from NodeJS\n\n\n');\" > /usr/test-dbg.js) && node --inspect-brk=5555 /usr/test-dbg.js"
root@host   docker exec $containerID /bin/bash -c "apt-get update && apt-get -y install socat"
root@host   docker exec -d $containerID /bin/bash -c "socat -v  tcp-listen:4444,reuseaddr,fork tcp:localhost:5555"
root@host   docker exec $containerID /bin/bash -c "curl localhost:4444 -v" ### you see the output here! Great the debugging port is working within the container
root@host   docker exec $containerID /bin/bash -c "curl localhost:5555 -v" ### you will see the same as we forwarded 5555 to 4444
root@host   curl localhost:4444 -v ### <<--- IT WORKS!! 4444@host --> 4444@container --> 5555@container (nodejs debugger)

NOTE: when you see "WebSockets request was expected" it means it actually got connected and got the response back from the debugger.

As for the debugging tool, I haven't been able to find a way to use webstorm yet. But chrome dev tool debugger tool and node-inspect work fine. For chrome dev tools, just copy and paste the output From the terminal when you run with node --inspect-brk flag. Make sure you adjust the port according to your port forwarding. The chrome dev tool address looks like something like this:

chrome-devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=127.0.0.1:38213/03b57c12-d8ca-4100-b852-1aacce107880

More details on debugging tools: https://nodejs.org/en/docs/inspector/