Recently I had some disk issues on my server and unfortunately it went to load average above 20 (which was killing for this machine). It forced me to power down the machine completely and to investigate it further. (It turned out that issue was with space on /home partition – sharing file system between two disks turned out to be not-so-good-idea after all).

Nevertheless, during those perturbations I found out, that I haven’t got filesystem dedicated for /var/log, which is a good practice and I decided, that it needs to be done. First I checked how much space is used in /var/logat the moment:

du -h /var/log

52K	/var/log/exim4
5.9M	/var/log/nmon
4.0K	/var/log/samba/cores/nmbd
4.0K	/var/log/samba/cores/smbd
12K	/var/log/samba/cores
652K	/var/log/samba
88K	/var/log/apt
4.0K	/var/log/mysql
12M	/var/log/installer/cdebconf
13M	/var/log/installer
48K	/var/log/unattended-upgrades
4.0K	/var/log/iptraf
4.0K	/var/log/news
4.0K	/var/log/sysstat
12K	/var/log/fsck
4.0K	/var/log/ntpstats
2.3M	/var/log/nginx
39M	/var/log

Not so much, so I looked where I can move those files for a moment:

df -PTh | grep /dev/mapper

/dev/mapper/dziobak-root ext4  322M  178M  128M  59% /
/dev/mapper/dziobak-home ext4  169G  148G   14G  92% /home
/dev/mapper/dziobak-tmp ext4  368M   11M  339M   3% /tmp
/dev/mapper/dziobak-usr ext4  8.3G  1.0G  6.9G  13% /usr
/dev/mapper/dziobak-var ext4  2.8G  620M  2.1G  24% /var

/tmp sounds good. Last thing to check before performing any action – is there any application that is currently using /var/log folder:

lsof /var/log/*

COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 1165 root    1w   REG  254,3   483188 4039 /var/log/kern.log
rsyslogd 1165 root    2w   REG  254,3   381980 4099 /var/log/messages
rsyslogd 1165 root    5w   REG  254,3   404756   42 /var/log/syslog
rsyslogd 1165 root    6w   REG  254,3    55716 4072 /var/log/debug
rsyslogd 1165 root    7w   REG  254,3  2403297 4032 /var/log/daemon.log
rsyslogd 1165 root    8w   REG  254,3   251603 4051 /var/log/auth.log
nmon     2202 root  cwd    DIR  254,3     4096 2948 /var/log/nmon

There are two (rsyslog and nmon), so I will need to stop them before  move. But firstly I will prepare new LV dedicated to /var/log:

lvcreate -L 1G --name var-log dziobak
  Logical volume "var-log" created

Where -L 1G is size of the new LV, --name var-log is the name of LV and dziobak is the name of VG. Now it’s time to create some file system – I was  struggling between ext2 or ext4. Finally I decided to go for ext4, but I believe it’s just a matter of taste:

mkfs.ext4 /dev/dziobak/var-log

mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376

Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 24 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Simple. OK, now it’s time to stop those applications, that are using /var/log, so we can move logs to some temporary place:

service rsyslog stop
Stopping enhanced syslogd: rsyslogd.

lsof /var/log/*
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
nmon    2202 root  cwd    DIR  254,3     4096 2948 /var/log/nmon

kill -9 2202
lsof /var/log/*

Done. Let’s move those logs to /tmp then:

tar -cvvzf /tmp/var-log.tar.gz /var/log

*** Some long output ;) ***

ls -l /tmp/var-log.tar.gz
-rw-r--r-- 1 root root 10095778 Sep 27 17:10 /tmp/var-log.tar.gz

rm -rf /var/log

OK, now it’s time to attach new LV and bring back old log files:

mkdir /var/log
mount /dev/mapper/dziobak-var--log /var/log
cd /
tar xvvzf /tmp/var-log.tar.gz

Last step is to add new file system to /etc/fstab:

/dev/mapper/dziobak-var--log /var/log   ext4    defaults        0       2