At midnight, a cronjob initiates logrotate to rotate 4 key log files. These 4 log files are also being sent to a log aggregation server by rsyslog. Despite running the rsyslog logrotate command after each log is rotated, sometimes, the server encounters a "Permission Denied" error that prevents rsyslog from sending the logs belonging to the new files. I cannot figure out why this is happening so unreliably.
Here are the errors reported by rsyslog. Note that they occur only on some dates:
Jul 31 00:00:01 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/celery.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 01 00:00:01 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/celery.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 02 00:00:04 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/nginx.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 09 00:00:01 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/celery.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 10 00:00:01 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/django.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 10 00:00:01 hostname rsyslogd[26343]: imfile: error with inotify API, ignoring file '/home/ubuntu/remote_logs/django.log': Permission denied [v8.32.0]
Aug 16 00:00:01 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/celery.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 16 00:00:02 hostname rsyslogd[26343]: file '/home/ubuntu/remote_logs/django.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Aug 19 00:00:03 hostname rsyslogd[856]: imfile: error with inotify API, ignoring file '/home/ubuntu/remote_logs/django.log': Permission denied [v8.32.0]
Aug 20 00:00:06 hostname rsyslogd[856]: imfile: error with inotify API, ignoring file '/home/ubuntu/remote_logs/app.log': Permission denied [v8.32.0]
Aug 30 00:00:01 hostname rsyslogd[856]: file '/home/ubuntu/remote_logs/celery.log': open error: Permission denied [v8.32.0 try http://www.rsyslog.com/e/2433 ]
Sep 03 00:00:01 hostname rsyslogd[856]: imfile: error with inotify API, ignoring file '/home/ubuntu/remote_logs/celery.log': Permission denied [v8.32.0]
Sep 03 00:00:04 hostname rsyslogd[856]: imfile: error with inotify API, ignoring file '/home/ubuntu/remote_logs/django.log': Permission denied [v8.32.0]
Logrotate is triggered by a cron job, using the root user. Here is the logrotate config:
/home/ubuntu/remote_logs/*.log {
rotate 365
daily
compress
missingok
notifempty
dateext
dateformat .%Y-%m-%d
dateyesterday
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}
/usr/lib/rsyslog/rsyslog-rotate
#!/bin/sh
if [ -d /run/systemd/system ]; then
systemctl kill -s HUP rsyslog.service
else
invoke-rc.d rsyslog rotate > /dev/null
fi
I know that the logrotate itself is succeeding, because the logrotate verbose logs indicate that, and I see the rotated log files every day. It seems like rsyslog cannot access the newly create files.
Here are the logrotate logs:
Handling 13 logs
rotating pattern: /home/ubuntu/remote_logs/*.log after 1 days (365 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 106
considering log /home/ubuntu/remote_logs/celery.log
Now: 2020-10-14 00:00
Last rotated at 2020-10-13 00:00
log needs rotating
considering log /home/ubuntu/remote_logs/django.log
Now: 2020-10-14 00:00
Last rotated at 2020-10-13 00:00
log needs rotating
considering log /home/ubuntu/remote_logs/app.log
Now: 2020-10-14 00:00
Last rotated at 2020-10-13 00:00
log needs rotating
considering log /home/ubuntu/remote_logs/nginx.log
Now: 2020-10-14 00:00
Last rotated at 2020-10-13 00:00
log needs rotating
rotating log /home/ubuntu/remote_logs/celery.log, log->rotateCount is 365
Converted ' .%Y-%m-%d' -> '.%Y-%m-%d'
dateext suffix '.2020-10-13'
glob pattern '.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'
renaming /home/ubuntu/remote_logs/celery.log to /home/ubuntu/remote_logs/celery.log.2020-10-13
creating new /home/ubuntu/remote_logs/celery.log mode = 0644 uid = 102 gid = 106
running postrotate script
switching euid to 0 and egid to 0
compressing log with: /bin/gzip
switching uid to 0 and gid to 106
rotating log /home/ubuntu/remote_logs/django.log, log->rotateCount is 365
Converted ' .%Y-%m-%d' -> '.%Y-%m-%d'
dateext suffix '.2020-10-13'
glob pattern '.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'
renaming /home/ubuntu/remote_logs/django.log to /home/ubuntu/remote_logs/django.log.2020-10-13
creating new /home/ubuntu/remote_logs/django.log mode = 0644 uid = 102 gid = 106
running postrotate script
switching euid to 0 and egid to 0
compressing log with: /bin/gzip
switching uid to 0 and gid to 106
rotating log /home/ubuntu/remote_logs/app.log, log->rotateCount is 365
Converted ' .%Y-%m-%d' -> '.%Y-%m-%d'
dateext suffix '.2020-10-13'
glob pattern '.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'
renaming /home/ubuntu/remote_logs/app.log to /home/ubuntu/remote_logs/app.log.2020-10-13
creating new /home/ubuntu/remote_logs/app.log mode = 0644 uid = 102 gid = 106
running postrotate script
switching euid to 0 and egid to 0
compressing log with: /bin/gzip
switching uid to 0 and gid to 106
rotating log /home/ubuntu/remote_logs/nginx.log, log->rotateCount is 365
Converted ' .%Y-%m-%d' -> '.%Y-%m-%d'
dateext suffix '.2020-10-13'
glob pattern '.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'
renaming /home/ubuntu/remote_logs/nginx.log to /home/ubuntu/remote_logs/nginx.log.2020-10-13
creating new /home/ubuntu/remote_logs/nginx.log mode = 0644 uid = 102 gid = 106
running postrotate script
switching euid to 0 and egid to 0
compressing log with: /bin/gzip
switching uid to 0 and gid to 106
switching euid to 0 and egid to 0
What is happening to rsyslog that prevents it from accessing the log files occasionally?
It looks like some of the files don't have permission to be created. You need to add create 0644 root root in your config file to give the necessary permission to rotate(modify) the content of the file.
You can also specify the group name create 0644 root grouname. But I will suggest using root
/home/ubuntu/remote_logs/*.log {
rotate 365
daily
compress
missingok
create 0644 root root
notifempty
dateext
dateformat .%Y-%m-%d
dateyesterday
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}