Filip Chabik

DevOps Engineer, Husband & Dad.

reprepro, building packages via SSH and signing repository

26th September 2011

I’m using my stationary PC for building packages, so most of the time I’m accessing this server via SSH. Almost everything is working fine, except pinentry (curses) which isn’t asking for a password during repository signing. Fortunately, reprepro has option that you can pass in command line which will force password question. The flag is --ask-passphrase and here’s an example command:

reprepro --ask-passphrase -b .. include unstable *.changes

Updating GRUB device.map file after adding a new disk

24th August 2011

Recently I wrote about extending existing VG in LVM by adding new external hard disk on USB. Basically I covered the topic fully, although today during installation of new kernel (2.6.39 from squeeze-backports) I encounter some errors. It took my awhile while I found out, that the most important one is GRUB related:

update-grub
Generating grub.cfg ...
/usr/sbin/grub-probe: error: Couldn't find PV pv1. Check your device.map.

GRUB2 can read some information from LVM partitions – GRUB-legacy didn’t really care about disks in LVM, as he was unable to read from them. Anyway, I figured that the problem is in grub.cfg and in device.map – both files are located under /boot/grub. What’s the problem? Well, new  added to existing VG and used only to extend existing LVs disk wasn’t  recognized by GRUB (hence error message provided above). This error was  due to lack of disk information provided in device.map file. What’s the  solution? Pretty simple, actually:

mv /boot/grub/device.map /boot/grub/device.map.backup
grub-mkdevicemap
update-grub

That’s all! (:

Extending LVM & FS online with new disk

9th August 2011

Recently my brother gave me his old laptop (some Sony Vaio) and I decided, that I will make mini-shell-server from it. It actually works great this way, but I encounter  small obstacle – it has only 80G hard disk. It’s hard to extend this  disk inside, so the more straight forward way was to add new external disk. I happened to have SATA II 250G hard disk, so I just bought external enclosure Welland ME-740J with USB support (as this laptop haven’t got eSATA).

On server I have Debian Squeeze installed with 2.6.32 kernel on it, so  there was absolutely no problem with recognizing new hard disk after I plugged enclosure to USB port of the laptop (here’s an dmesg output):

[360327.763921] Initializing USB Mass Storage driver...
[360327.764262] scsi4 : SCSI emulation for USB Mass Storage devices
[360327.764376] usbcore: registered new interface driver usb-storage
[360327.764380] USB Mass Storage support registered.
[360327.764539] usb-storage: device found at 2
[360327.764541] usb-storage: waiting for device to settle before scanning
[360332.764350] usb-storage: device scan complete
[360332.767926] scsi 4:0:0:0: Direct-Access     WDC WD25 00JS-22NCB1           PQ: 0 ANSI: 2
[360332.768794] sd 4:0:0:0: Attached scsi generic sg2 type 0
[360332.769870] sd 4:0:0:0: [sdb] 488397168 512-byte logical blocks: (250 GB/232 GiB)
[360332.772147] sd 4:0:0:0: [sdb] Write Protect is off
[360332.772155] sd 4:0:0:0: [sdb] Mode Sense: 38 00 00 00
[360332.772161] sd 4:0:0:0: [sdb] Assuming drive cache: write through
[360332.775767] sd 4:0:0:0: [sdb] Assuming drive cache: write through
[360332.775788]  sdb: unknown partition table
[360332.782261] sd 4:0:0:0: [sdb] Assuming drive cache: write through
[360332.782277] sd 4:0:0:0: [sdb] Attached SCSI disk

And here’s output from fdisk -l:

Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table

As you can see, my new disk is recognized as /dev/sdb and it has no partition table. I’m going to assign it to my existing Volume Group, cause I want to extend existing Logical Volumes and not creating new ones. That’s why, using fdisk, I will create one, ~250G partition for LVM:

fdisk -cu /dev/sdb

(inside fdisk): n
Command action  
   e   extended
   p   primary partition (1-4)
(inside fdisk): p
Partition number (1-4): 1
First sector (2048-488397167, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-488397167, default 488397167):
Using default value 488397167

Hopefully this is clear. Basically, all that has been done here, was  creation of primary partition table (command n followed by p) and  assigning full space to it (default values 2048-488397167). Now command p (inside fdisk) should show something like that:

Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x7c5c7a2f

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   488397167   244197560   83  Linux

It needs to be available for LVM, so the partition type needs to be changed (still inside fdisk):

(inside fdisk): t
Selected partition 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

And here’s what command p is now showing:

Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x7c5c7a2f

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   488397167   244197560   8e  Linux LVM

Last step is to write changes that were made – command w inside fdisk will do the trick. Last check with dmesg to see, whether new partition is recognized by system:

[361136.668175]  sdb: sdb1

OK, so here’s how my Volume Group looks like before attaching new disk:

vgdisplay

  --- Volume group ---
  VG Name               dziobak
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  7
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               6
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               74.29 GiB
  PE Size               4.00 MiB
  Total PE              19018
  Alloc PE / Size       19018 / 74.29 GiB
  Free  PE / Size       0 / 0
  VG UUID               oV4Qyj-HYQe-u2uK-31KW-HmHO-4kdM-x9CAl5

Now it’s time to “attach” new disk to this Volume Group:

pvcreate /dev/sdb1
vgextend dziobak /dev/sdb1

  Volume group "dziobak" successfully extended

And here’s new vgdisplay output:

vgdisplay

  --- Volume group ---
  VG Name               dziobak
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  8
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               6
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               307.17 GiB
  PE Size               4.00 MiB
  Total PE              78636
  Alloc PE / Size       19018 / 74.29 GiB
  Free  PE / Size       59618 / 232.88 GiB
  VG UUID               oV4Qyj-HYQe-u2uK-31KW-HmHO-4kdM-x9CAl5

As you can probably see, I’ve got 232.88 GiB of new free space to  allocate inside this Volume Group. I don’t want to use all of it, as I  may change plans somewhere in the future, so I decided, that additional  100G for /home partition is all I need:

df -PTh /home

Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/dziobak-home ext4   61G   32G   27G  55% /home

Firstly, let’s check whether /home filesystem can be resized online. There’s a small applications called dumpe2fs, which, among other options, is able to print filesystems flags (-h flag). If there’s no resize_inode flag, then filesystem can’t be resized online:

dumpe2fs -h /dev/mapper/dziobak-home | grep resize_inode
dumpe2fs 1.41.12 (17-May-2010)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize

Good, we can resize:

lvextend -L+100G /dev/mapper/dziobak-home
  Extending logical volume home to 161.48 GiB
  Logical volume home successfully resized

resize2fs /dev/mapper/dziobak-home
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/dziobak-home is mounted on /home; on-line resizing required
old desc_blocks = 4, new_desc_blocks = 11
Performing an on-line resize of /dev/mapper/dziobak-home to 42332160 (4k) blocks.
The filesystem on /dev/mapper/dziobak-home is now 42332160 blocks long.

Depending on sizes and disks itself, it can take some time. After that, my /home partition is much bigger:

df -PTh /home

Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/dziobak-home ext4  159G   32G  120G  21% /home

My Volume Group still have some free space to attach in the future:

vgdisplay

  --- Volume group ---
  VG Name               dziobak
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  9
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               6
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               307.17 GiB
  PE Size               4.00 MiB
  Total PE              78636
  Alloc PE / Size       44618 / 174.29 GiB
  Free  PE / Size       34018 / 132.88 GiB
  VG UUID               oV4Qyj-HYQe-u2uK-31KW-HmHO-4kdM-x9CAl5

You’re welcome!

Repartitioning LUKS-crypted LVM on Debian

26th April 2011

This is crash howto, so few important notes are needed:

  • You may and you probably will loose your data, so backup is highly recommended.
  • I’m assuming you have all needed tools to provide actions described below.
  • I’m also assuming, that you have basic knowledge of Linux command line and some of it’s tools, like fdisk for example.
  • This howto isn’t about creating encrypted LVM – it assumes you already have one.

One side note: it may be possible, to provide all of these actions using software with GUI. It’s up to you to try for example tools like palimpsest and/or system-config-lvm.

Let’s look at the starting situation:

  1. There’s one hard disk: /dev/sda (~320G).
  2. There was Windows XP installed on 80G partition (NTFS).
  3. There was later on added Debian, installed on following partitions:
    |-/boot, ext2, ~512M  
    |-LUKS-crypted LVM  
       |-/root, ext4, ~10G  
       |-swap, ~4G  
       |-/home, ext4, ~226G

This dual-boot was working swell, but I decided, that I don’t  need Windows partition anymore, as I can use KVM to use it (when needed)  as a guest. It turned out, that it would be a good idea to add some  free space to /root partition and rest of the freed space add to /home  partition (it also may be a good idea, to create dedicated partition for  KVM). Let’s play.

$ sudo fdisk -cu /dev/sda
(inside fdisk): p    (to list new partition table)
Dysk /dev/sda: 320.1 GB, bajtów: 320072933376
głowic: 255, sektorów/ścieżkę: 63, cylindrów: 38913, w sumie sektorów: 625142448
Jednostka = sektorów, czyli 1 * 512 = 512 bajtów
Rozmiar sektora (logiczny/fizyczny) w bajtach: 512 / 512
Rozmiar we/wy (minimalny/optymalny) w bajtach: 512 / 512
Identyfikator dysku: 0x64656469

Urządzenie Rozruch   Początek      Koniec   Bloków   ID  System
/dev/sda1   *          63   165427919    82713928+   7  HPFS/NTFS
/dev/sda2       165429248   166404095      487424   83  Linux
/dev/sda3       166406142   625141759   229367809    5  Rozszerzona
/dev/sda5       166406144   625141759   229367808   83  Linux

First, we need to remove Windows NTFS partition:

(inside fdisk): d    (to delete partition)
(inside fdisk): 1    (to choose which partition is to be removed)
(inside fdisk): p    (to list new partition table)

Dysk /dev/sda: 320.1 GB, bajtów: 320072933376
głowic: 255, sektorów/ścieżkę: 63, cylindrów: 38913, w sumie sektorów: 625142448
Jednostka = sektorów, czyli 1 * 512 = 512 bajtów
Rozmiar sektora (logiczny/fizyczny) w bajtach: 512 / 512
Rozmiar we/wy (minimalny/optymalny) w bajtach: 512 / 512
Identyfikator dysku: 0x64656469

Urządzenie Rozruch   Początek      Koniec   Bloków   ID  System
/dev/sda2       165429248   166404095      487424   83  Linux
/dev/sda3       166406142   625141759   229367809    5  Rozszerzona
/dev/sda5       166406144   625141759   229367808   83  Linux

OK, now save changes, exit and reboot machine:

(inside fdisk): w    (to write changes and quit fdisk)

$ sudo shutdown -r now

After reboot, launch fdisk again and create Linux LVM partition:

$ sudo fdisk -cu /dev/sda
(inside fdisk): n    (to create new partition)
(inside fdisk): 1    (to define where to place her)
(inside fdisk): 2048    (default)
(inside fdisk): 165429247    (default)
(inside fdisk): p    (to list new partition table)

Dysk /dev/sda: 320.1 GB, bajtów: 320072933376
głowic: 255, sektorów/ścieżkę: 63, cylindrów: 38913, w sumie sektorów: 625142448
Jednostka = sektorów, czyli 1 * 512 = 512 bajtów
Rozmiar sektora (logiczny/fizyczny) w bajtach: 512 / 512
Rozmiar we/wy (minimalny/optymalny) w bajtach: 512 / 512
Identyfikator dysku: 0x64656469

Urządzenie Rozruch   Początek      Koniec   Bloków   ID  System
/dev/sda1            2048   165429247    82713600   83  Linux
/dev/sda2       165429248   166404095      487424   83  Linux
/dev/sda3       166406142   625141759   229367809    5  Rozszerzona
/dev/sda5       166406144   625141759   229367808   83  Linux

(inside fdisk): t    (to change partition type)
(inside fdisk): 1    (to choose partition which is to be changed)
(inside fdisk): L    (to list available partition types)
(inside fdisk): 8e    (to choose Linux LVM)
(inside fdisk): p    (to list new partition table)

Dysk /dev/sda: 320.1 GB, bajtów: 320072933376
głowic: 255, sektorów/ścieżkę: 63, cylindrów: 38913, w sumie sektorów: 625142448
Jednostka = sektorów, czyli 1 * 512 = 512 bajtów
Rozmiar sektora (logiczny/fizyczny) w bajtach: 512 / 512
Rozmiar we/wy (minimalny/optymalny) w bajtach: 512 / 512
Identyfikator dysku: 0x64656469

Urządzenie Rozruch   Początek      Koniec   Bloków   ID  System
/dev/sda1            2048   165429247    82713600   8e  Linux LVM
/dev/sda2       165429248   166404095      487424   83  Linux
/dev/sda3       166406142   625141759   229367809    5  Rozszerzona
/dev/sda5       166406144   625141759   229367808   83  Linux

(inside fdisk): w    (to write changes and quit fdisk)

And reboot:

$ sudo shutdown -r now

Now we need to use pvcreate to initialize newly created partition to use LVM:

$ sudo pvcreate /dev/sda1
  Physical volume "/dev/sda1" successfully created

It’s time to add this physical volume to our LVM group, so we can use  free space (~80G) from it to add it to our logical volumes:

$ sudo vgextend foo /dev/sda1
  Volume group "foo" successfully extended

Where “foo” is a name of your LVM group. You may list all your logical volumes (with such informations like size etc.) using lvdisplay command. You can launch this command without specifying concrete  logical volume and it will print out informations about all of them.

Alright – it’s time to make use of this free space we gained from removing Windows:

$ sudo lvextend -L20G /dev/foo/root
  Extending logical volume root to 20,00 GiB
  Logical volume root successfully resized

Please notice, that -L command haven’t got + before 20G – it  means, that logical volume /dev/foo/root will be resized to exactly 20 GiB (20 GiB won’t be added to /dev/foo/root, cause it already had 10  GiB).

Now file system need be resized to follow new physical size:

$ sudo resize2fs /dev/foo/root

It will be done on-line, which means, you can use your system normally and you won’t have to reboot your machine after changes.

For operation described above I’ve used approx. 10 GiB from ~80 GiB  that were freed after WinXP removal. Now I want to add free space that  is left to /dev/foo/home logical volume:

$ sudo lvextend -l+100%FREE /dev/foo/home

Now file system need to follow new physical size:

$ sudo resize2fs /dev/foo/home

You may use pvdisplay, vgdisplay, lvdisplay and df (with -h flag, for example) to check whether the partitions were changed as you wanted to.

Further reading:

Newer Posts
Older Posts