I am trying to scrape docker metrics from inside a container. I have seen the creator of Prometheus doing it this way see video here. I am not able to reproduce it.
I have exposed docker metrics page via json config.
{
"metrics-addr" : "172.17.0.1:4999",
"experimental" : true
}
I have also tried with 0.0.0.0. In any case I am able to curl from the host machine itself with no issues.
$ curl 172.17.0.1:4999/metrics | more
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0# HELP builder_builds_failed_total Number of failed image builds
# TYPE builder_builds_failed_total counter
builder_builds_failed_total{reason="build_canceled"} 0
builder_builds_failed_total{reason="build_target_not_reachable_error"} 0
...
However, I am not able to get the same metrics page from inside a container. I have tried to basi/socat image, the same one from the video. I have also tried a handful of other ways like using curl from inside the container and so on.
The image does essentially just one thing
ENV IN="172.17.0.1:4999" \
OUT="4999"
ENTRYPOINT socat -d -d TCP-L:$OUT,fork TCP:$IN
docker run --rm -p 4998:4999 basi/socat
2021/04/02 16:27:47 socat[8] N listening on AF=2 0.0.0.0:4999
curl localhost:4998/metrics
The socat images recognizes the connection, but it hangs for 1-2 minutes and times out.
2021/04/02 16:29:53 socat[8] N accepting connection from AF=2 172.17.0.1:57678 on AF=2 172.17.0.2:4999
2021/04/02 16:29:53 socat[8] N forked off child process 9
2021/04/02 16:29:53 socat[8] N listening on AF=2 0.0.0.0:4999
2021/04/02 16:29:53 socat[9] N opening connection to AF=2 172.17.0.1:4999
2021/04/02 16:32:04 socat[9] E connect(5, AF=2 172.17.0.1:4999, 16): Operation timed out
2021/04/02 16:32:04 socat[9] N exit(1)
2021/04/02 16:32:04 socat[8] N childdied(): handling signal 17
2021/04/02 16:32:04 socat[8] W waitpid(): child 9 exited with status 1
curl: (52) Empty reply from server
I am slowly wondering if it could be something in my iptables rules. I have posted below INPUT and DOCKER-USER chain. Those are the only ones I have changed myself.
sudo iptables -L -n -v
Chain INPUT (policy DROP 32060 packets, 1810K bytes)
pkts bytes target prot opt in out source destination
716K 1484M ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
599K 1069M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
16 3138 ACCEPT all -- eth0 * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
12 624 DROP all -- eth0 * 0.0.0.0/0 0.0.0.0/0
74921 139M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
So turns out issues was indeed iptables. The default policy for the INPUT chain in this case is DROP. Since the container is now trying to reach the host system itself, its traffic will come through the input chain.
So I added 2 simple rules for now, that will allow all traffic in the INPUT chain from docker0 and docker_gwbridge.
sudo iptables -A INPUT -i docker0 -j ACCEPT
sudo iptables -A INPUT -i docker_gwbridge -j ACCEPT
After that I am finally able to query docker metrics from inside the container.
docker run --rm curlimages/curl -I -m 10 172.17.0.1:4999/metrics
HTTP/1.1 200 OK
Content-Type: text/plain; version=0.0.4; charset=utf-8
Date: Sat, 03 Apr 2021 10:53:12 GMT