Gentoo VM on SmartOS hypervisor

I wanted to have a small, “minimalistic” VMs on my hypervisor, so they would have very little footprint on resources. I decided to go with systemd and btrfs for system & service management and main/only filesystem respectively.

The only considered distributions up for that task (at least from my point of view) were Debian and Gentoo. I decided to start with the latter as I had less experience with it and wanted to learn my ways around.

Environment outline

I’m trying to keep things relatively tidy, especially on my production and semi-production boxes. Hence, I have two directories – one for storing ISOs and templates of VMs:

• /opt/iso
• /opt/templates

For installation purposes I’m using latest Gentoo ISO available:

cd /opt/iso
curl -kLO http://distfiles.gentoo.org/releases/amd64/autobuilds/current-install-amd64-minimal/install-amd64-minimal-20140904.iso

And here’s json template for Gentoo box /opt/templates/gentoo.json:

{
  "autoboot": false,
  "brand": "kvm",
  "ram": "512",
  "vcpus": "4",
  "alias": "12-gentoo",
  "resolvers": ["8.8.8.8", "8.8.4.4"],
  "nics": [
    {
     "nic_tag": "dmz0",
     "ip": "192.168.0.12",
     "gateway": "192.168.0.1",
     "netmask": "255.255.255.0",
     "allow_ip_spoofing": "1",
     "model": "virtio",
     "primary": true
    },
    {
     "nic_tag": "lan",
     "ip": "10.0.0.12",
     "netmask": "255.255.255.0",
     "model": "virtio"
    }
  ],
  "disks": [
    {
      "boot": true,
      "model": "virtio",
      "size": 10240
    }
  ]
}

I will describe some of the parts from above example, but only briefly. For more information and more details regarding each option, be sure the check official SmartOS wiki.
The first part is relatively easy and self-explanatory: autoboot is set to false, cause I will need to boot from the CD and not the disk; brand is set to kvm as this won’t be Zone; 512M ram and 4 vcpu – these settings should make compilation in this VM relatively fast; alias is just an easy to recognize name – I’m usually setting number + easily distinguishable tag for fast greping (it’s also worth noting, that number itself is also used as the ending number for the LAN network [in 10.0.0.x subnet] and in “DMZ” network [192.168.0.x subnet]); resolvers are set to standard Google DNS;
Network configuration is a bit more tricky. I assigned two NICs to the VM – one is for LAN configuration only (eth1 with no outside world access whatsoever), the second one is giving ability to VM to access internet (eth0, primary one). I’m using IPF on my Global Zone, so later on I will need to open ports and set NAT for some of them (SSH most importantly).
Please note, that if you haven’t got similar NAT configuration, you will need external IP for the VM to connect to the outside world.
Both network and disk settings are using virtio – keep that in mind as distribution you will install later on will need to have support for that. (On most distros virtio is available out of the box, but in Gentoo and relatives case you first need to compile kernel, so please do remember to compile virtio modules).

Generating & booting VM

Once template is prepared, VM generating is fairly easy:

vmadm create < /opt/templates/gentoo.json
Successfully created VM 6055824a-9a33-404a-a7b9-ef826ba79302

Note down or export the UUID as it will come in handy later on:

gentoo=6055824a-9a33-404a-a7b9-ef826ba79302
echo $gentoo
6055824a-9a33-404a-a7b9-ef826ba79302

Copy ISO to the root of the newly generated VM:

cp /opt/iso/install-amd64-minimal-20140904.iso /zones/$gentoo/root/gentoo.iso

Now it’s time to boot it up with CD as first booting device:

vmadm boot $gentoo order=cd,once=d cdrom=/gentoo.iso,ide

If, like me, you are using ipf as firewall, remember to open VNC port for connection (cause SSH is disabled by default in Gentoo LiveCD), to open port for SSH for the virtual machine (2222 for example) and to set NAT on that port to point to VM.

Check VNC port vmadm info $gentoo vnc:

{
  "vnc": {
    "host": "<hypervisor-ip>",
    "port": 52753,
    "display": 46853
  }
}

Now open SSH and VNC in /etc/ipf/ipf.conf:

[...]
# Allow SSH access to Gentoo VM
pass in quick proto tcp from any to any port = 2222 flags S/FSRPAU keep state keep frags
    [...]
# Allow VNC access to Gentoo VM
pass in quick proto tcp from any to any port = 52753 flags S/FSRPAU keep state keep frags
[...]

Set NAT in /etc/ipf/ipnat.conf:

[...]
# SSH -> Gentoo VM
rdr e1000g0 0.0.0.0/0 port 2222 -> 192.168.0.12 port 22

Reload rules:

/usr/sbin/ipf -E -Fa -v -f /etc/ipf/ipf.conf
/usr/sbin/ipnat -FC -v -f /etc/ipf/ipnat.conf

Confirm new settings via:

ipfstat -io
ipnat -l

Launch VNC client, connect to the VM and installation shall begin (:

Gentoo installation

Remember to connect via VNC client in following manner: <hypervisor-ip>:<VNC port for VM> (where VNC port is the one that was gathered via vmadm info command earlier on). Once connected make sure that both network interfaces are up and running and that at least one can reach outside world.

If both prerequisites are met, it’s time to start SSH and to set password for the root account:

service sshd start
passwd

Cool, let’s carry on with installation through SSH:

ssh -p 2222 root@<ip address of a hypervisor>

Keep in mind that 2222 is only an example and you need to set it accordingly in NAT (look up). You may also close the VNC connection and port (unless for any given reason you wish to keep it – it may come in handy when things will go south during SSH connection) right after establishing connectin via SSH.

Partitioning

Determine disk(s) name(s):

lsblk

NAME  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0    11:0    1  190M  0 rom  /mnt/cdrom
sr1    11:1    1 1024M  0 rom  
vda   252:0    0   10G  0 disk
loop0   7:0    0  161M  1 loop /mnt/livecd

In my case, I have only one disk called /dev/vda that is 10G big.

parted -a optimal /dev/vda
mklabel gpt
unit mib
mkpart gpt_bios 1 3
set 1 bios_grub on
mkpart swap 3 2051
mkpart btrfs 2051 -1
q

Above I’m using parted to create gpt partition table. There’s part of the disk for gpt handling (gpt_bios that’s 2M big) with flag for bios_grub, swap partition (2G big) and rest is for btrfs to handle.

mkswap /dev/vda2
swapon /dev/vda2
mkfs.btrfs /dev/vda3

btrfs subvolumes

OK, swap and btrfs created. Now it’s time to configure subvolumes and mount them properly:

mkdir /mnt/target
mount -t btrfs -o rw,noatime,nodiratime,space_cache /dev/vda3 /mnt/target
cd /mnt/target
btrfs subvolume create @
btrfs subvolume create @boot
btrfs subvolume create @home
btrfs subvolume create @portage
btrfs subvolume create @var
btrfs subvolume create @varlog
mount -t btrfs -o rw,noatime,nodiratime,space_cache,subvol=@ /dev/vda3 /mnt/gentoo
mkdir -p /mnt/gentoo/{boot,home,var}
mount -t btrfs -o rw,noatime,nodiratime,space_cache,nodev,nosuid,subvol=@home /dev/vda3 /mnt/gentoo/home
mount -t btrfs -o rw,noatime,nodiratime,space_cache,nodev,nosuid,noexec,subvol=@boot /dev/vda3 /mnt/gentoo/boot
mount -t btrfs -o rw,noatime,nodiratime,space_cache,subvol=@var /dev/vda3 /mnt/gentoo/var
mkdir -p /mnt/gentoo/var/log
mount -t btrfs -o rw,noatime,nodiratime,nodev,nosuid,noexec,space_cache,subvol=@varlog /dev/vda3 /mnt/gentoo/var/log
mkdir -p /mnt/gentoo/usr/portage
mount -t btrfs -o rw,noatime,nodiratime,space_cache,compress=lzo,subvol=@portage /dev/vda3 /mnt/gentoo/usr/portage

Confirm that all subvolumes have properly set mountpoints:

mount | grep btrfs
/dev/vda3 on /mnt/gentoo type btrfs (rw,noatime,nodiratime,space_cache,subvol=@)
/dev/vda3 on /mnt/gentoo/home type btrfs (rw,noatime,nodiratime,nodev,nosuid,space_cache,subvol=@home)
/dev/vda3 on /mnt/gentoo/boot type btrfs (rw,noatime,nodiratime,nodev,nosuid,noexec,space_cache,subvol=@boot)
/dev/vda3 on /mnt/gentoo/var type btrfs (rw,noatime,nodiratime,space_cache,subvol=@var)
/dev/vda3 on /mnt/gentoo/var/log type btrfs (rw,noatime,nodiratime,nodev,nosuid,noexec,space_cache,subvol=@varlog)
/dev/vda3 on /mnt/gentoo/usr/portage type btrfs (rw,noatime,nodiratime,space_cache,compress=lzo,subvol=@portage)

Let’s clean it up:

cd
umount /mnt/target
rmdir /mnt/target

I took subvolume naming convention from Ubuntu and ever since I’m using it pretty much everywhere where I’m dealing with btrfs. Feel free to try any other or stick to this one if it appeals to you too.

stage3 download & initial config

Time to download and deploy stage3:

cd /mnt/gentoo
curl -kLO http://distfiles.gentoo.org/releases/amd64/autobuilds/current-install-amd64-minimal/hardened/stage3-amd64-hardened+nomultilib-20140904.tar.bz2
tar xvjpf stage3-*.tar.bz2

I chose hardened and nomultilib, be sure to choose whatever meets your requirements.

nano -w /mnt/gentoo/etc/portage/make.conf
USE="-X systemd"
MAKEOPTS="-j5"

mirrorselect -i -o >> /mnt/gentoo/etc/portage/make.conf
mirrorselect -i -r -o >> /mnt/gentoo/etc/portage/make.conf

cp -L /etc/resolv.conf /mnt/gentoo/etc/

Usually I’m adding -X and systemd to already provided ones in USE section. I also added MAKEOPTS and set it to -j5 as I have 4 vcpus (handbook suggests to have it set to the number of CPUs + 1).
Second part is responsible for mirror selection – be sure to choose most close one to your location for faster download.

chroot

It’s time to chroot into newly prepared environment and start with the installation:

mount -t proc proc /mnt/gentoo/proc
mount --rbind /sys /mnt/gentoo/sys
mount --rbind /dev /mnt/gentoo/dev
chroot /mnt/gentoo /bin/bash
source /etc/profile
export PS1="(chroot) $PS1"

Fetch and install the latest portage snapshot on your system:

emerge-webrsync

Pick profile you wish to use (it will set USE flags accordingly):
(In my case it was already the one I wanted to use. If it differs and you would like to pick some other, you may do so via eselect profile set $number, where $number is the profile you wish to use).

eselect profile list
[13]  hardened/linux/amd64/no-multilib *

Timezone setting:

echo "Europe/Berlin" > /etc/timezone
emerge --config sys-libs/timezone-data

Locale setting:

nano -w /etc/locale.gen

egrep -v "^#|^$" /etc/locale.gen
en_US.UTF-8 UTF-8

locale-gen

eselect locale list
Available targets for the LANG variable:
[1] C
[2] en_US.utf8
[3] POSIX [ ] (free form)

eselect locale set 2

Be sure to pick locales accordingly to your needs.
Shell settings need to be reloaded:

env-update && source /etc/profile
export PS1="(chroot) $PS1"

Installation of prerequisite packages:

emerge -av dev-libs/lzo
emerge -av app-arch/lzop
emerge -av gentoo-sources
emerge -av sys-fs/btrfs-progs

This is a quick fix for btrfs fsck issue you may encounter during boot:

ln -s /sbin/btrfsck /sbin/fsck.btrfs

Kernel compilation

cd /usr/src/linux
wget -c http://deb.ianod.es/kernel/kernel-hardened-config-3.14.14.xz
unxz kernel-hardened-config-3.14.14.xz
mv kernel-hardened-config-3.14.14 .config
make menuconfig
make && make modules_install
cp arch/x86_64/boot/bzImage /boot/kernel-3.14.14-gentoo

In second step I’m downloading working config for 3.14.14 (current in Gentoo at the time of writing this post) I prepared while I was testing whether these instructions are working fine or not. Feel free to use it or to use it as a starting point – config itself was based on kernel-seeds. I added gpt, btrfs, virtio and cgroups support (thanks to the last one, it’s possible to have LXCcontainers support on this VM out of the box).

systemd installation

ln -sf /proc/self/mounts /etc/mtab
USE="-systemd" emerge -av sys-apps/dbus
emerge --update --deep --with-bdeps=y --newuse @world

Because systemd is already defined in USE in /etc/portage/make.conf last step is enough to have it installed and configured properly.

initramfs

echo "sys-kernel/dracut" >> /etc/portage/package.keywords
emerge -av sys-kernel/dracut
dracut

I’m using dracut for quick & easy initramfs generation.

Setting up /etc/fstab

It’s time to set /etc/fstab mountpoints of all subvolumes created earlier on:

cat << EOF > /etc/fstab
/dev/vda3   /       btrfs   rw,noatime,nodiratime,space_cache,subvol=@ 0 1
/dev/vda3   /home   btrfs   rw,nodev,nosuid,noatime,nodiratime,space_cache,subvol=@home 0 1
/dev/vda3   /boot   btrfs   rw,nodev,nosuid,noexec,noatime,nodiratime,space_cache,subvol=@boot 0 1
/dev/vda3   /var    btrfs   rw,noatime,nodiratime,space_cache,subvol=@var 0 1
/dev/vda3   /var/log   btrfs   rw,nodev,nosuid,noexec,noatime,nodiratime,space_cache,subvol=@varlog 0 1
/dev/vda3   /usr/portage   btrfs   rw,nodev,nosuid,noexec,noatime,nodiratime,compress=lzo,space_cache,subvol=@portage 0 1
/dev/vda2   none    swap    sw 0 0
EOF

Don’t forget about swap partition.

Setting up networking

I could use DHCP to handle networking for me, but that would require additional daemon which I don’t really need. That’s why I decided to leave network in systemd hands and make it static:

cat << EOF > /etc/conf.d/network@eth0
address=192.168.0.12
netmask=24
broadcast=192.168.0.255
gateway=192.168.0.1
EOF

cat << EOF > /etc/conf.d/network@eth1
address=10.0.0.12
netmask=24
broadcast=10.0.0.255
EOF

These are config files that will be “sourced” to systemd starting scripts. Here’s startup script for eth0 nano -w /usr/lib/systemd/system/network@eth0.service:

[Unit]
Description=Network connectivity (%i)
Wants=network.target
Before=network.target
BindsTo=sys-subsystem-net-devices-%i.device
After=sys-subsystem-net-devices-%i.device

[Service]
Type=oneshot
RemainAfterExit=yes
EnvironmentFile=/etc/conf.d/network@%i

ExecStart=/bin/ip link set dev %i up
ExecStart=/bin/ip addr add ${address}/${netmask} broadcast ${broadcast} dev %i
ExecStart=/bin/sh -c 'test -n ${gateway} && /bin/ip route add default via ${gateway}'

ExecStop=/bin/ip addr flush dev %i
ExecStop=/bin/ip link set dev %i down

[Install]
WantedBy=multi-user.target

And here’s a bit different for eth1 nano -w /usr/lib/systemd/system/network@eth1.service:

[Unit]
Description=Network connectivity (%i)
Wants=network.target
Before=network.target
BindsTo=sys-subsystem-net-devices-%i.device
After=sys-subsystem-net-devices-%i.device

[Service]
Type=oneshot
RemainAfterExit=yes
EnvironmentFile=/etc/conf.d/network@%i

ExecStart=/bin/ip link set dev %i up
ExecStart=/bin/ip addr add ${address}/${netmask} broadcast ${broadcast} dev %i

ExecStop=/bin/ip addr flush dev %i
ExecStop=/bin/ip link set dev %i down

[Install]
WantedBy=multi-user.target

Scripts are mapping config files and interfaces names via @eth (that’s what %i is there for).
iproute2 is used extensively, so be sure to have it installed:

emerge -av sys-apps/iproute2

Be sure to set DNS too:

cat << EOF > /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
EOF

GRUB installation

emerge -av sys-boot/grub

nano -w /etc/default/grub
GRUB_CMDLINE_LINUX="init=/usr/lib/systemd/systemd"

Install and configure grub. GRUB_CMDLINE_LINUX with systemd defined is already provided by default – all that is needed is to unhash this line.
After that, grub is ready ready for deployment:

mkdir -p /boot/grub
grub2-mkconfig -o /boot/grub/grub.cfg
grub2-install /dev/vda

Be sure to set password for the root account, otherwise you won’t be able to access it through SSH:

passwd

Finishing touches

Following are symbolic links for three services I want to have started after reboot:

ln -s /usr/lib/systemd/system/network@eth0.service /etc/systemd/system/multi-user.target.wants/
ln -s /usr/lib/systemd/system/network@eth1.service /etc/systemd/system/multi-user.target.wants/
ln -s /usr/lib/systemd/system/sshd.service /etc/systemd/system/

This wil pretty much bring up both network interfaces (configured earlier on) and SSH daemon.
This is it – exit, unmount & reboot:

exit
cd
umount -l /mnt/gentoo/dev{/shm,/pts,}
umount -l /mnt/gentoo{/boot,/proc,}
reboot

After reboot

Set hostname:

hostnamectl set-hostname gentoo

Summary

Once everything is done and you are able to connect to your VM, you should have fairly plain system as a starting point. There’s no dhcp, no syslog, no crontab and, most importantly, no bullshit. OS is ready to serve its purpose and is ideal for further adjustments and enhancements. Enjoy! (:

Debian is comming next.