Filip Chabik

DevOps Engineer, Husband & Dad.

Extend LV & FS on-line with one command

1st October 2011

Today my GNOME 3 told me, that my root partition is filling in pretty quickly and there’s not much space left. I looked into it and figured that 1 GB of additional space will suffice for some time. Since version 2.02.86 of lvm2 package, there’s a new flag for lvextend command: -r (--resizefs):

Resize underlying filesystem together with the logical volume using fsadm(8).

Neat. I decided to give it a try:

sudo lvextend -L +1G -r /dev/mapper/duckbill-root

  Extending logical volume root to 14,90 GiB
  Logical volume root successfully resized
resize2fs 1.42-WIP (25-Sep-2011)
System plików /dev/mapper/duckbill-root jest zamontowany pod /; wymagana zmiana rozmiaru w locie
old_desc_blocks = 1, new_desc_blocks = 1
Performing an on-line resize of /dev/mapper/duckbill-root to 3906560 (4k) blocks.
The filesystem on /dev/mapper/duckbill-root is now 3906560 blocks long.

Migrating to new Logical Volume

27th September 2011

Recently I had some disk issues on my server and unfortunately it went to load average above 20 (which was killing for this machine). It forced me to power down the machine completely and to investigate it further. (It turned out that issue was with space on /home partition – sharing file system between two disks turned out to be not-so-good-idea after all).

Nevertheless, during those perturbations I found out, that I haven’t got filesystem dedicated for /var/log, which is a good practice and I decided, that it needs to be done. First I checked how much space is used in /var/log at the moment:

du -h /var/log

52K	/var/log/exim4
5.9M	/var/log/nmon
4.0K	/var/log/samba/cores/nmbd
4.0K	/var/log/samba/cores/smbd
12K	/var/log/samba/cores
652K	/var/log/samba
88K	/var/log/apt
4.0K	/var/log/mysql
12M	/var/log/installer/cdebconf
13M	/var/log/installer
48K	/var/log/unattended-upgrades
4.0K	/var/log/iptraf
4.0K	/var/log/news
4.0K	/var/log/sysstat
12K	/var/log/fsck
4.0K	/var/log/ntpstats
2.3M	/var/log/nginx
39M	/var/log

Not so much, so I looked where I can move those files for a moment:

df -PTh | grep /dev/mapper

/dev/mapper/dziobak-root ext4  322M  178M  128M  59% /
/dev/mapper/dziobak-home ext4  169G  148G   14G  92% /home
/dev/mapper/dziobak-tmp ext4  368M   11M  339M   3% /tmp
/dev/mapper/dziobak-usr ext4  8.3G  1.0G  6.9G  13% /usr
/dev/mapper/dziobak-var ext4  2.8G  620M  2.1G  24% /var

/tmp sounds good. Last thing to check before performing any action – is there any application that is currently using /var/log folder:

lsof /var/log/*

COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 1165 root    1w   REG  254,3   483188 4039 /var/log/kern.log
rsyslogd 1165 root    2w   REG  254,3   381980 4099 /var/log/messages
rsyslogd 1165 root    5w   REG  254,3   404756   42 /var/log/syslog
rsyslogd 1165 root    6w   REG  254,3    55716 4072 /var/log/debug
rsyslogd 1165 root    7w   REG  254,3  2403297 4032 /var/log/daemon.log
rsyslogd 1165 root    8w   REG  254,3   251603 4051 /var/log/auth.log
nmon     2202 root  cwd    DIR  254,3     4096 2948 /var/log/nmon

There are two (rsyslog and nmon), so I will need to stop them before  move. But firstly I will prepare new LV dedicated to /var/log:

lvcreate -L 1G --name var-log dziobak
  Logical volume "var-log" created

Where -L 1G is size of the new LV, --name var-log is the name of LV and dziobak is the name of VG. Now it’s time to create some file system – I was  struggling between ext2 or ext4. Finally I decided to go for ext4, but I believe it’s just a matter of taste:

mkfs.ext4 /dev/dziobak/var-log

mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376

Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 24 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Simple. OK, now it’s time to stop those applications, that are using /var/log, so we can move logs to some temporary place:

service rsyslog stop
Stopping enhanced syslogd: rsyslogd.

lsof /var/log/*
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
nmon    2202 root  cwd    DIR  254,3     4096 2948 /var/log/nmon

kill -9 2202
lsof /var/log/*

Done. Let’s move those logs to /tmp then:

tar -cvvzf /tmp/var-log.tar.gz /var/log

*** Some long output ;) ***

ls -l /tmp/var-log.tar.gz
-rw-r--r-- 1 root root 10095778 Sep 27 17:10 /tmp/var-log.tar.gz

rm -rf /var/log

OK, now it’s time to attach new LV and bring back old log files:

mkdir /var/log
mount /dev/mapper/dziobak-var--log /var/log
cd /
tar xvvzf /tmp/var-log.tar.gz

Last step is to add new file system to /etc/fstab:

/dev/mapper/dziobak-var--log /var/log   ext4    defaults        0       2

reprepro, building packages via SSH and signing repository

26th September 2011

I’m using my stationary PC for building packages, so most of the time I’m accessing this server via SSH. Almost everything is working fine, except pinentry (curses) which isn’t asking for a password during repository signing. Fortunately, reprepro has option that you can pass in command line which will force password question. The flag is --ask-passphrase and here’s an example command:

reprepro --ask-passphrase -b .. include unstable *.changes

Updating GRUB device.map file after adding a new disk

24th August 2011

Recently I wrote about extending existing VG in LVM by adding new external hard disk on USB. Basically I covered the topic fully, although today during installation of new kernel (2.6.39 from squeeze-backports) I encounter some errors. It took my awhile while I found out, that the most important one is GRUB related:

update-grub
Generating grub.cfg ...
/usr/sbin/grub-probe: error: Couldn't find PV pv1. Check your device.map.

GRUB2 can read some information from LVM partitions – GRUB-legacy didn’t really care about disks in LVM, as he was unable to read from them. Anyway, I figured that the problem is in grub.cfg and in device.map – both files are located under /boot/grub. What’s the problem? Well, new  added to existing VG and used only to extend existing LVs disk wasn’t  recognized by GRUB (hence error message provided above). This error was  due to lack of disk information provided in device.map file. What’s the  solution? Pretty simple, actually:

mv /boot/grub/device.map /boot/grub/device.map.backup
grub-mkdevicemap
update-grub

That’s all! (:

Extending LVM & FS online with new disk

9th August 2011

Recently my brother gave me his old laptop (some Sony Vaio) and I decided, that I will make mini-shell-server from it. It actually works great this way, but I encounter  small obstacle – it has only 80G hard disk. It’s hard to extend this  disk inside, so the more straight forward way was to add new external disk. I happened to have SATA II 250G hard disk, so I just bought external enclosure Welland ME-740J with USB support (as this laptop haven’t got eSATA).

On server I have Debian Squeeze installed with 2.6.32 kernel on it, so  there was absolutely no problem with recognizing new hard disk after I plugged enclosure to USB port of the laptop (here’s an dmesg output):

[360327.763921] Initializing USB Mass Storage driver...
[360327.764262] scsi4 : SCSI emulation for USB Mass Storage devices
[360327.764376] usbcore: registered new interface driver usb-storage
[360327.764380] USB Mass Storage support registered.
[360327.764539] usb-storage: device found at 2
[360327.764541] usb-storage: waiting for device to settle before scanning
[360332.764350] usb-storage: device scan complete
[360332.767926] scsi 4:0:0:0: Direct-Access     WDC WD25 00JS-22NCB1           PQ: 0 ANSI: 2
[360332.768794] sd 4:0:0:0: Attached scsi generic sg2 type 0
[360332.769870] sd 4:0:0:0: [sdb] 488397168 512-byte logical blocks: (250 GB/232 GiB)
[360332.772147] sd 4:0:0:0: [sdb] Write Protect is off
[360332.772155] sd 4:0:0:0: [sdb] Mode Sense: 38 00 00 00
[360332.772161] sd 4:0:0:0: [sdb] Assuming drive cache: write through
[360332.775767] sd 4:0:0:0: [sdb] Assuming drive cache: write through
[360332.775788]  sdb: unknown partition table
[360332.782261] sd 4:0:0:0: [sdb] Assuming drive cache: write through
[360332.782277] sd 4:0:0:0: [sdb] Attached SCSI disk

And here’s output from fdisk -l:

Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table

As you can see, my new disk is recognized as /dev/sdb and it has no partition table. I’m going to assign it to my existing Volume Group, cause I want to extend existing Logical Volumes and not creating new ones. That’s why, using fdisk, I will create one, ~250G partition for LVM:

fdisk -cu /dev/sdb

(inside fdisk): n
Command action  
   e   extended
   p   primary partition (1-4)
(inside fdisk): p
Partition number (1-4): 1
First sector (2048-488397167, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-488397167, default 488397167):
Using default value 488397167

Hopefully this is clear. Basically, all that has been done here, was  creation of primary partition table (command n followed by p) and  assigning full space to it (default values 2048-488397167). Now command p (inside fdisk) should show something like that:

Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x7c5c7a2f

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   488397167   244197560   83  Linux

It needs to be available for LVM, so the partition type needs to be changed (still inside fdisk):

(inside fdisk): t
Selected partition 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

And here’s what command p is now showing:

Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x7c5c7a2f

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   488397167   244197560   8e  Linux LVM

Last step is to write changes that were made – command w inside fdisk will do the trick. Last check with dmesg to see, whether new partition is recognized by system:

[361136.668175]  sdb: sdb1

OK, so here’s how my Volume Group looks like before attaching new disk:

vgdisplay

  --- Volume group ---
  VG Name               dziobak
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  7
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               6
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               74.29 GiB
  PE Size               4.00 MiB
  Total PE              19018
  Alloc PE / Size       19018 / 74.29 GiB
  Free  PE / Size       0 / 0
  VG UUID               oV4Qyj-HYQe-u2uK-31KW-HmHO-4kdM-x9CAl5

Now it’s time to “attach” new disk to this Volume Group:

pvcreate /dev/sdb1
vgextend dziobak /dev/sdb1

  Volume group "dziobak" successfully extended

And here’s new vgdisplay output:

vgdisplay

  --- Volume group ---
  VG Name               dziobak
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  8
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               6
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               307.17 GiB
  PE Size               4.00 MiB
  Total PE              78636
  Alloc PE / Size       19018 / 74.29 GiB
  Free  PE / Size       59618 / 232.88 GiB
  VG UUID               oV4Qyj-HYQe-u2uK-31KW-HmHO-4kdM-x9CAl5

As you can probably see, I’ve got 232.88 GiB of new free space to  allocate inside this Volume Group. I don’t want to use all of it, as I  may change plans somewhere in the future, so I decided, that additional  100G for /home partition is all I need:

df -PTh /home

Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/dziobak-home ext4   61G   32G   27G  55% /home

Firstly, let’s check whether /home filesystem can be resized online. There’s a small applications called dumpe2fs, which, among other options, is able to print filesystems flags (-h flag). If there’s no resize_inode flag, then filesystem can’t be resized online:

dumpe2fs -h /dev/mapper/dziobak-home | grep resize_inode
dumpe2fs 1.41.12 (17-May-2010)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize

Good, we can resize:

lvextend -L+100G /dev/mapper/dziobak-home
  Extending logical volume home to 161.48 GiB
  Logical volume home successfully resized

resize2fs /dev/mapper/dziobak-home
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/dziobak-home is mounted on /home; on-line resizing required
old desc_blocks = 4, new_desc_blocks = 11
Performing an on-line resize of /dev/mapper/dziobak-home to 42332160 (4k) blocks.
The filesystem on /dev/mapper/dziobak-home is now 42332160 blocks long.

Depending on sizes and disks itself, it can take some time. After that, my /home partition is much bigger:

df -PTh /home

Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/dziobak-home ext4  159G   32G  120G  21% /home

My Volume Group still have some free space to attach in the future:

vgdisplay

  --- Volume group ---
  VG Name               dziobak
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  9
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               6
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               307.17 GiB
  PE Size               4.00 MiB
  Total PE              78636
  Alloc PE / Size       44618 / 174.29 GiB
  Free  PE / Size       34018 / 132.88 GiB
  VG UUID               oV4Qyj-HYQe-u2uK-31KW-HmHO-4kdM-x9CAl5

You’re welcome!