One of my New Year’s resolution was to get back a bit closer to the lower level parts of Linux. And what’s there lower than the kernel itself? I always preferred vanilla kernel, even when I was fooling around with Gentoo, and this hasn’t changed. In December I started with preparing first builds. Nothing too fancy as it’s only for my personal usage, but I still find it worth going through. I hit four issues I needed to fix in order to be able to use these kernels on my machines:
- ZFS has to be supported (as I use it on my servers).
- Wireguard has to be built-in (cause vanilla releases occur way more often than the distro provided ones and re-running
dkmseach time makes no sense to me). (Worth mentioning that starting with kernel 5.6 Wireguard is going to be in the main tree).
perfhas to be part of the build. (Part of the
- Entire process has to be streamlined and possibly handled by some kind of CI.
All of this is done on Ubuntu 18.04 LTS (bionic) and none of the commands should require root access (except where
sudo is used, duh).
Let’s start with fetching some dependencies:
sudo apt update sudo apt install asciidoc \ autoconf \ bc \ binutils-dev \ bison \ build-essential \ byobu \ fakeroot \ flex \ gawk \ git \ htop \ kernel-package \ kernel-wedge \ libattr1-dev \ libaudit-dev \ libbabeltrace-ctf-dev \ libbabeltrace-dev \ libblkid-dev \ libcap-dev \ libdevmapper-dev \ libdw-dev \ libelf-dev \ libiberty-dev \ liblzma-dev \ libncurses-dev \ libnuma-dev \ libperl-dev \ libselinux-dev \ libslang2-dev \ libssl-dev \ libtool \ libudev-dev \ libunwind-dev \ libzstd-dev \ linux-generic-hwe-18.04-edge \ python-dev \ python3-distutils \ rsync \ sysstat \ systemtap-sdt-dev \ time \ tmux \ unp \ uuid-dev \ zlib1g-dev
Some of the packages above are optional (
tmux etc.), but I still include them as they are not very big and they may help during the compilation or debugging problems. Feel free to omit them.
Now that’s done, let’s fetch some sources:
mkdir build cd build wget -c https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.4.21.tar.xz wget -c https://kernel.hadret.com/patches/linux-tools.patch wget -c https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.4.21/0001-base-packaging.patch \ https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.4.21/0002-UBUNTU-SAUCE-add-vmlinux.strip-to-BOOT_TARGETS1-on-p.patch \ https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.4.21/0003-UBUNTU-SAUCE-tools-hv-lsvmbus-add-manual-page.patch \ https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.4.21/0004-debian-changelog.patch \ https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.4.21/0005-configs-based-on-Ubuntu-5.4.0-8.11.patch git clone --depth 1 --branch v0.0.20200215 https://git.zx2c4.com/wireguard-linux-compat wg git clone --depth 1 --branch zfs-0.8.3 https://github.com/zfsonlinux/zfs.git
I first create and enter
build directory and then fetch sources of the following:
- Stable (longterm) kernel sources, version 5.4.21. (Current as of time of writing this post).
linux-toolspatch, based on the one from here: [PATCH 4/4] RFC: builddeb: add linux-tools package with perf.
- Five patches from Ubuntu mainline kernel 5.4.21. (This step is optional but my goal was to be as close to the Ubuntu provided kernel as possible while still using vanilla with some additional patches).
- Git clone of tag
- Git clone of tag
unp linux-5.4.21.tar.xz will unpack the kernel sources. After that, the directory tree overview should look something like this:
. build ├── 0001-base-packaging.patch ├── 0002-UBUNTU-SAUCE-add-vmlinux.strip-to-BOOT_TARGETS1-on-p.patch ├── 0003-UBUNTU-SAUCE-tools-hv-lsvmbus-add-manual-page.patch ├── 0004-debian-changelog.patch ├── 0005-configs-based-on-Ubuntu-5.4.0-8.11.patch ├── linux-5.4.21 ├── linux-5.4.21.tar.xz ├── linux-tools.patch ├── wg └── zfs
Of course having these anywhere else will work too, only the paths will need to be adjusted accordingly.
Let’s do some patching:
cd linux-5.4.21 for patch in ../*.patch ; do patch -p1 < $patch ; done
OK, one of the packages installed in the dependencies was
linux-generic-hwe-18.04-edge – currently it installs kernel version
220.127.116.11.97. This is going to serve us as a base for the config:
cp /boot/config-5.3.0-* .config make olddefconfig prepare
make olddefconfig will apply old configuration from the copied .config file and set defaults to any new parts that are not covered by it.
make prepare will get the kernel into the ready state for third-party modules compilation.
Let’s carry on with the ZFS part now:
cd ../zfs ./autogen.sh ./configure --enable-linux-builtin --with-linux=../linux-5.4.21 --with-linux-obj=../linux-5.4.21 ./copy-builtin ../linux-5.4.21
That’s pretty much it when it comes to getting ZFS into the kernel. I’m using relative paths everywhere as a matter of convenience, but as I mentioned earlier this can be easily adjusted.
Next steps cover adding support for Wireguard:
cd ../wg kernel-tree-scripts/jury-rig.sh ../linux-5.4.21
Now that’s done, let’s ensure that both ZFS and Wireguard are compiled in:
cd ../linux-5.4.21 make menuconfig
Hopefully images above are descriptive enough, but if not, there are two settings that need handling:
-> File systems -> ZFS filesystem support (NEW) -> Networking support -> Networking options -> TCP/IP networking -> IP: WireGuard secure network tunnel
Once that’s done, it’s time to start compilation:
make clean BUILD_TOOLS=true make -j $(nproc) deb-pkg LOCALVERSION=-1-hadret KDEB_PKGVERSION=$(make kernelversion)-1 > /dev/null
To dissect a bit the second, long command:
BUILD_TOOLS=true: this will enable
make -j $(nproc): this will run
makecommand across all of the available CPUs (cores/threads/whatever).
deb-pkg: it’s the default way of building Debian packages out of kernel sources (provided in the vanilla kernel sources).
KDEB_PKGVERSIONare setting appropriate versioning (including custom name in the
LOCALVERSIONpart, which, obviously, can be adjusted).
> /dev/null: this will ensure only errors are going to be printed to
stdout, which may speed a bit compilation time.
After compilation is done, the following packages should be available for installation:
linux-headers-5.4.21-1-hadret_5.4.21-1_amd64.deb linux-image-5.4.21-1-hadret-dbg_5.4.21-1_amd64.deb linux-image-5.4.21-1-hadret_5.4.21-1_amd64.deb linux-libc-dev_5.4.21-1_amd64.deb linux-tools-5.4.21-1-hadret_5.4.21-1_amd64.deb
What about Debian?
Pretty much entire process works the same way in Debian. The main difference is that I skip the Ubuntu patches from applying and some minor APT dependencies need adjusting/skipping. There’s also one additional step necessary – Debian is using
CONFIG_SYSTEM_TRUSTED_KEYS to validate the builds. If you base the default config on the one provided by kernel from Debian repository, you are going to hit an error with message along those lines:
make: *** No rule to make target 'debian/certs/debian-uefi-certs.pem', needed by 'certs/x509_certificate_list'. Stop. make: *** Waiting for unfinished jobs....
Fix is relatively easy:
sed -i 's%CONFIG_SYSTEM_TRUSTED_KEYS="debian/certs/debian-uefi-certs.pem"%CONFIG_SYSTEM_TRUSTED_KEYS=""%' .config
I will not cover the setup itself here (maybe some other time in a dedicated post), but a high-level overview instead. Here’s how I do it: I have two Ansible playbooks prepared – one for managing Cloud instances running in Hetzner (CX41) and second one running few operations on the hosts and then triggering three scripts that are preparing, building and (r)syncing ready packages. The flow is fairly simple:
- I commit changes in git and tag the release with
5.4.21-1which triggers the build.
- First part spins up two instances in Hetzner Cloud – one with Ubuntu 18.04 (bionic) and second one with Debian 10 (buster).
- Second part brings in all of the dependencies, patches, clones repositories for third-party tools (ZFS and Wireguard) etc. Then it carries on with preparing the kernel, building it and syncing ready packages to my server.
- Third and last part ditches two instances in Hetzner Cloud to free the resources after successful build.
The entire build time takes roughly an hour and a half and is done on the two instances simultaneously.
Happy kernel compiling!