One of my New Year’s resolution was to get back a bit closer to the lower level parts of Linux. And what’s there lower than the kernel itself? I always preferred vanilla kernel, even when I was fooling around with Gentoo, and this hasn’t changed. In December I started with preparing first builds. Nothing too fancy as it’s only for my personal usage, but I still find it worth going through. I hit four issues I needed to fix in order to be able to use these kernels on my machines:

1. ZFS has to be supported (as I use it on my servers).
2. Wireguard has to be built-in (cause vanilla releases occur way more often than the distro provided ones and re-running dkms each time makes no sense to me).¹
3. perf has to be part of the build.²
4. Entire process has to be streamlined and possibly handled by some kind of CI.

Not long ago I wrote about backporting BCC & bpftrace in Ubuntu with focus on bionic (18.04 LTS). With this new release, there’s a lot of going on: release notes. I “backported” this one already into my PPA. On that note, I did drop support for the disco (19.04) and added support for the upcoming focal (20.04).

NGINX before 1.17.7, with certain error_page configurations, allows HTTP request smuggling, as demonstrated by the ability of an attacker to read unauthorized web pages in environments where NGINX is being fronted by a load balancer.

That’s the essence of the CVE-2019-20372. Yet again, as I mentioned in my NGINX Extended post I was not going to work on 1.14.x branch any more with the exception of security updates – this is the case for such exception. Both 1.14.x for Xenial (16.04 LTS) and 1.16.x for Bionic (18.04 LTS) were patched against this vulnerability and are available from my PPA. On Docker Hub I bumped up only the 1.16.x branch as usage for 1.14.x is pretty much non-existent.

I’m following Brendan Gregg’s performance-related content for years now. I started when he was still in Joyent, later on I bought his Systems Performance book and I get back to it whenever I’m doing any profiling. Now I follow closely all of the latest work he’s doing on BPF front. There’s a small problem though. On the dependency tree of production-grade systems, I’ve got the following:

• stable OS (Debian 10 “Buster” or Ubuntu 18.04 “Bionic Beaver”)
• latest kernel (5.0+)
• latest tracing tools (BCC & bpftrace)

Two first are easy – both Debian and Ubuntu are now providing ways for having relatively new kernel up and running – former with buster-backports¹ and latter with HWE.² The third one becomes a small hassle, at least in Ubuntu world.

bpftrace is currently not available for bionic at all. There’s no technical limitation here – it’s just the package was never built for it.³ I decided to backport it then.

I already briefly wrote about the idea of having dynamically discoverable upstreams in NGINX when I covered the topic of NGINX Extended. With the boom of microservices and containers scattered all over the place there was suddenly a need for something that would serve as a single source of truth. When solutions like Mesos/Marathon or Kubernetes kicked in, notion of having services statically assigned to particular address and/or port went straight to the trash. That’s exactly where Consul comes into play. I first crossed my paths with it years ago when it was both relatively new concept and software. These days I think it’s safe to say that, along etcd, it became industry’s standard. But even with its mature state, it solves only half of the problem – it registers and allows services to discover each other for variety of connection purposes, but if there’s anything that needs to serve as an application for HTTPs reverse-proxy, it has to be relatively static. Or does it?