On my mac I am running nginx in a docker file and filebeat in a docker file.
docker run -p 80:80 nginx
The above command successfully runs nginx which I can visit in the browser and the output is printed to the console.
This is my docker file that I am building and then running:
FROM docker.elastic.co/beats/filebeat:7.8.0
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
And this is my filebeat.yml referenced in the above:
filebeat.autodiscover:
providers:
- type: docker
templates:
- condition:
contains:
docker.container.image: nginx
config:
- type: docker
containers.ids:
- "${data.docker.container.id}"
- module: nginx
access:
enabled: true
containers:
stream: "stdout"
error:
enabled: true
containers:
stream: "stderr"
output.console:
pretty: true
I run the filebeat docker image using:
sudo docker run -it -v /var/run/docker.sock:/var/run/docker.sock filebeat
The docker.sock stuff is because of an unable to connect to docker socket error I was getting (some kind of docker user error permissions I presume).
All of the above gets the filebeat running inside the docker container.
INFO [autodiscover] autodiscover/autodiscover.go:113 Starting autodiscover manager
However when I visit localhost in the browser and trigger the nginx log, the only output is occasional docker system metrics every 30 seconds:
{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":40,"time":{"ms":43}},"total":{"ticks":100,"time":{"ms":109},"value":100},"user":{"ticks":60,"time":{"ms":66}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"d3e79d62-6949-4d79-89e8-c595332c18ed","uptime":{"ms":30054}},"memstats":{"gc_next":10249440,"memory_alloc":5520104,"memory_total":17591608,"rss":55390208},"runtime":{"goroutines":23}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"console"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":2},"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}}}}
The nginx log is not being detected.
This is a potentially relevant warning I am getting:
Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled
Edit: I think this is the crux of the issue, the harvester is not finding any logs:
filebeat":{"harvester":{"open_files":0,"running":0}}
Edit2: The root cause of the problem seems to be that on the Mac the terminal can't access the log files: /var/lib/docker/containers/${data.docker.container.id}/*.log as this aren't where they are stored on the Mac... so it seems running filebeat on mac is not supported this way between dockers
Filebeat on Mac doesn't support collecting docker logs: