systemd - can out redirect logs to a file and journald

I have a ubunut machine running in AWS and want to bring the docker daemon logs to AWS Cloudwatch. Per default docker daemon is started via systemd on ubuntu and logs to the journal, and I want to keep docker logs in the journal.

But aws-cloudwatch-logs-agent does not support sending logs for a service from the journal to cloudwatch, and I therefore need a log file.

Thats why I am wondering if I can configure a systemd service to log to a file and to the journal?

I can log to a file with:

[Service]
StandardOutput=append:/var/log/dockerd.log
StandardError=append:/var/log/dockerd.log

But then now logs appear in the journal ... any ideas?

Solution

Keep the service logging to journal (i.e. default settings), but have a second service that creates the text-file logs from journal logs.

The two traditional syslog daemons, syslog-ng and rsyslog, both support pulling messages directly from journal files; many Linux distros even pre-configure them to give you standard /var/log/syslog.
```
# apt install syslog-ng
```
syslog-ng does this using the "system()" macro (which uses the "systemd-journal()" driver); rsyslog uses the "imjournal" module.

If you want to split up dockerd logs to their own file, follow syslog-ng documentation.
Journald also supports socket-based forwarding of messages to traditional syslog daemons (both of the above also support this, though pull-mode from journal files is preferred).

This is activated using "ForwardSyslog="; syslog-ng can receive the forwarded messages using "systemd-syslog()" (which is a macro for /run/systemd/journal/syslog), rsyslogd can use the regular "imuxsock".

Finally, a custom service that runs journalctl --follow might do the job:

[Service]
ExecStart=/bin/journalctl -a -u docker -f --cursor-file=/var/log/dockerd.cursor
StandardOutput=append:/var/log/dockerd.log