One of my New Year’s resolution was to get back a bit closer to the lower level parts of Linux. And what’s there lower than the kernel itself? I always preferred vanilla kernel, even when I was fooling around with Gentoo, and this hasn’t changed. In December I started with preparing first builds. Nothing too fancy as it’s only for my personal usage, but I still find it worth going through. I hit four issues I needed to fix in order to be able to use these kernels on my machines:

  1. ZFS has to be supported (as I use it on my servers).
  2. Wireguard has to be built-in (cause vanilla releases occur way more often than the distro provided ones and re-running dkms each time makes no sense to me). (Worth mentioning that starting with kernel 5.6 Wireguard is going to be in the main tree).
  3. perf has to be part of the build. (Part of the linux-tools).
  4. Entire process has to be streamlined and possibly handled by some kind of CI.

All of this is done on Ubuntu 18.04 LTS (bionic) and none of the commands should require root access (except where sudo is used, duh).

Let’s start with fetching some dependencies:

sudo apt update
sudo apt install asciidoc \
                 autoconf \
                 bc \
                 binutils-dev \
                 bison \
                 build-essential \
                 byobu \
                 fakeroot \
                 flex \
                 gawk \
                 git \
                 htop \
                 kernel-package \
                 kernel-wedge \
                 libattr1-dev \
                 libaudit-dev \
                 libbabeltrace-ctf-dev \
                 libbabeltrace-dev \
                 libblkid-dev \
                 libcap-dev \
                 libdevmapper-dev \
                 libdw-dev \
                 libelf-dev \
                 libiberty-dev \
                 liblzma-dev \
                 libncurses-dev \
                 libnuma-dev \
                 libperl-dev \
                 libselinux-dev \
                 libslang2-dev \
                 libssl-dev \
                 libtool \
                 libudev-dev \
                 libunwind-dev \
                 libzstd-dev \
                 linux-generic-hwe-18.04-edge \
                 python-dev \
                 python3-distutils \
                 rsync \
                 sysstat \
                 systemtap-sdt-dev \
                 time \
                 tmux \
                 unp \
                 uuid-dev \
                 zlib1g-dev

Some of the packages above are optional (byobu, htop, sysstat, tmux etc.), but I still include them as they are not very big and they may help during the compilation or debugging problems. Feel free to omit them.

Now that’s done, let’s fetch some sources:

mkdir build
cd build
wget -c https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.4.21.tar.xz
wget -c https://kernel.hadret.com/patches/linux-tools.patch
wget -c https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.4.21/0001-base-packaging.patch \
        https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.4.21/0002-UBUNTU-SAUCE-add-vmlinux.strip-to-BOOT_TARGETS1-on-p.patch \
        https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.4.21/0003-UBUNTU-SAUCE-tools-hv-lsvmbus-add-manual-page.patch \
        https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.4.21/0004-debian-changelog.patch \
        https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.4.21/0005-configs-based-on-Ubuntu-5.4.0-8.11.patch
git clone --depth 1 --branch v0.0.20200215 https://git.zx2c4.com/wireguard-linux-compat wg
git clone --depth 1 --branch zfs-0.8.3 https://github.com/zfsonlinux/zfs.git

I first create and enter build directory and then fetch sources of the following:

  1. Stable (longterm) kernel sources, version 5.4.21. (Current as of time of writing this post).
  2. linux-tools patch, based on the one from here: [PATCH 4/4] RFC: builddeb: add linux-tools package with perf.
  3. Five patches from Ubuntu mainline kernel 5.4.21. (This step is optional but my goal was to be as close to the Ubuntu provided kernel as possible while still using vanilla with some additional patches).
  4. Git clone of tag v0.0.20200215 of Wireguard.
  5. Git clone of tag zfs-0.8.3 of ZFS.

unp linux-5.4.21.tar.xz will unpack the kernel sources. After that, the directory tree overview should look something like this:

. build
├── 0001-base-packaging.patch
├── 0002-UBUNTU-SAUCE-add-vmlinux.strip-to-BOOT_TARGETS1-on-p.patch
├── 0003-UBUNTU-SAUCE-tools-hv-lsvmbus-add-manual-page.patch
├── 0004-debian-changelog.patch
├── 0005-configs-based-on-Ubuntu-5.4.0-8.11.patch
├── linux-5.4.21
├── linux-5.4.21.tar.xz
├── linux-tools.patch
├── wg
└── zfs

Of course having these anywhere else will work too, only the paths will need to be adjusted accordingly.

Let’s do some patching:

cd linux-5.4.21
for patch in ../*.patch ; do patch -p1 < $patch ; done

OK, one of the packages installed in the dependencies was linux-generic-hwe-18.04-edge – currently it installs kernel version 5.3.0.40.97. This is going to serve us as a base for the config:

cp /boot/config-5.3.0-* .config
make olddefconfig prepare

make olddefconfig will apply old configuration from the copied .config file and set defaults to any new parts that are not covered by it. make prepare will get the kernel into the ready state for third-party modules compilation.

Let’s carry on with the ZFS part now:

cd ../zfs
./autogen.sh
./configure --enable-linux-builtin --with-linux=../linux-5.4.21 --with-linux-obj=../linux-5.4.21
./copy-builtin ../linux-5.4.21

That’s pretty much it when it comes to getting ZFS into the kernel. I’m using relative paths everywhere as a matter of convenience, but as I mentioned earlier this can be easily adjusted.

Next steps cover adding support for Wireguard:

cd ../wg
kernel-tree-scripts/jury-rig.sh ../linux-5.4.21

Now that’s done, let’s ensure that both ZFS and Wireguard are compiled in:

cd ../linux-5.4.21
make menuconfig

Hopefully images above are descriptive enough, but if not, there are two settings that need handling:

-> File systems
  -> ZFS filesystem support (NEW)

-> Networking support
  -> Networking options
    -> TCP/IP networking
      -> IP: WireGuard secure network tunnel

Once that’s done, it’s time to start compilation:

make clean
BUILD_TOOLS=true make -j $(nproc) deb-pkg LOCALVERSION=-1-hadret KDEB_PKGVERSION=$(make kernelversion)-1 > /dev/null

To dissect a bit the second, long command:

  1. BUILD_TOOLS=true: this will enable linux-tools package (including perf).
  2. make -j $(nproc): this will run make command across all of the available CPUs (cores/threads/whatever).
  3. deb-pkg: it’s the default way of building Debian packages out of kernel sources (provided in the vanilla kernel sources).
  4. LOCALVERSION and KDEB_PKGVERSION are setting appropriate versioning (including custom name in the LOCALVERSION part, which, obviously, can be adjusted).
  5. > /dev/null: this will ensure only errors are going to be printed to stdout, which may speed a bit compilation time.

After compilation is done, the following packages should be available for installation:

linux-headers-5.4.21-1-hadret_5.4.21-1_amd64.deb
linux-image-5.4.21-1-hadret-dbg_5.4.21-1_amd64.deb
linux-image-5.4.21-1-hadret_5.4.21-1_amd64.deb
linux-libc-dev_5.4.21-1_amd64.deb
linux-tools-5.4.21-1-hadret_5.4.21-1_amd64.deb

That’s it.

What about Debian?

Pretty much entire process works the same way in Debian. The main difference is that I skip the Ubuntu patches from applying and some minor APT dependencies need adjusting/skipping. There’s also one additional step necessary – Debian is using CONFIG_SYSTEM_TRUSTED_KEYS to validate the builds. If you base the default config on the one provided by kernel from Debian repository, you are going to hit an error with message along those lines:

make[4]: *** No rule to make target 'debian/certs/debian-uefi-certs.pem', needed by 'certs/x509_certificate_list'.  Stop.
make[4]: *** Waiting for unfinished jobs....

Fix is relatively easy:

sed -i 's%CONFIG_SYSTEM_TRUSTED_KEYS="debian/certs/debian-uefi-certs.pem"%CONFIG_SYSTEM_TRUSTED_KEYS=""%' .config

Continuous Integration

I will not cover the setup itself here (maybe some other time in a dedicated post), but a high-level overview instead. Here’s how I do it: I have two Ansible playbooks prepared – one for managing Cloud instances running in Hetzner (CX41) and second one running few operations on the hosts and then triggering three scripts that are preparing, building and (r)syncing ready packages. The flow is fairly simple:

  1. I commit changes in git and tag the release with 5.4.21-1 which triggers the build.
  2. First part spins up two instances in Hetzner Cloud – one with Ubuntu 18.04 (bionic) and second one with Debian 10 (buster).
  3. Second part brings in all of the dependencies, patches, clones repositories for third-party tools (ZFS and Wireguard) etc. Then it carries on with preparing the kernel, building it and syncing ready packages to my server.
  4. Third and last part ditches two instances in Hetzner Cloud to free the resources after successful build.

The entire build time takes roughly an hour and a half and is done on the two instances simultaneously.

Happy kernel compiling!