Logging into my Ubuntu machine, I get a warning that I am running out of disk space. Tracing back, I find that it is the syslogs, especially the kern.log(s) that are eating up my 1TB disk.
-rw-r----- 1 syslog adm 240G Feb 25 14:22 kern.log
-rw-r----- 1 syslog adm 516G Feb 21 07:59 kern.log.1
-rw-r----- 1 syslog adm 1.1K Feb 15 07:39 kern.log.2.gz
-rw-r----- 1 syslog adm 19K Feb 7 07:56 kern.log.3.gz
-rw-r----- 1 syslog adm 37K Feb 1 07:45 kern.log.4.gz
-rw-r----- 1 syslog adm 23G Feb 25 14:52 syslog
-rw-r----- 1 syslog adm 25G Feb 25 08:11 syslog.1
-rw-r----- 1 syslog adm 1.6G Feb 24 07:49 syslog.2.gz
-rw-r----- 1 syslog adm 1.7G Feb 23 08:18 syslog.3.gz
-rw-r----- 1 syslog adm 3.4G Feb 22 08:19 syslog.4.gz
-rw-r----- 1 syslog adm 3.6G Feb 21 07:59 syslog.5.gz
-rw-r----- 1 syslog adm 6.9G Feb 20 07:38 syslog.6.gz
-rw-r----- 1 syslog adm 7.3G Feb 19 07:36 syslog.7.gz
From the snippet above, you can easily find that kern.log and kern.log.1 is eating up 80% of my 1TB disk. I can get the space by deleting the files, but I think it won't solve the problem.
Does anyone have an idea on what the issue might be? I saw that you can get the logging level by:
cat /proc/sys/kernel/printk
and I get
4 4 1 7
Have you checked the content of those files? There's obviously something going on with your server causing events to be generated. Resolve whatever issue is causing that, and your logs should return to their normal size.
To temporary solve the issue, type
echo "" > /var/log/kern.log
echo "" > /var/log/syslog
service syslog restart
journalctl --vacuum-size=50M
You need to be root user for this: enter sudo su, your password, and then the above commands