In the early days of the internet, massive blocks of IP addresses were allocated to entire organisations or groups. What we would now call a Class A network, a /8, or about 16 million IPv4 addresses, were handed out to organisations like AT&T (12.0.0.0/8), Apple (17.0.0.0/8), Ford (19.0.0.0/8), and Amateur Radio Digital Communications - also known as ARDC - was given the entire block of 44.0.0.0/8 for amateur radio use.

Amateur Radio operators can ask for a slice of this for amateur radio research and experimentation purposes, and I went through this process a while back and was allocated 44.31.106.0/24. This block is used to host a few Amateur Radio-related services, but to get there I had to learn how to host a public subnet, routing, BGP advertisements, and so on.

To manage this subnet I have a small Linux VPS that acts as the router. It advertises the subnet via BGP and runs a WireGuard server. Every other address on the network is another WireGuard client, or routed via a WireGuard client.

Unfortunately, I soon discovered that from time to time my entire subnet would kind of just fall off the internet. Upon further investigation I found the following in the kernel system logs:

kernel: [12314635.025901] nf_conntrack: nf_conntrack: table full, dropping packet

oops

This led me to discover that the Linux kernel can only track so many connections, and likely since this is acting as a router for a /24 network, it probably needs to deal with 255 times higher typical background noise (scanners, scrapers, NMAP, etc.) than that of a single VPS.

Fortunately this can be tweaked by writing a bigger number than the default ~7000 to /proc/sys/net/nf_conntrack_max. That doesn't seem so hard, but what if I want this value to be higher all the time, like after rebooting the server?

Well as the internet would have you believe, that is where sysctl comes in. This little tool can also be used to set kernel parameters, and has a config file in /etc/sysctl.conf - or in more recent systems, multiple config files in /etc/sysctl.d/*.conf which get loaded on startup.

So I set my parameter as a once-off, added it to the configuration file, and all is good. Right?

$ cat /etc/sysctl.d/98-conntrack-max.conf 
net.netfilter.nf_conntrack_max = 131072

This should work... right?

So after a few months and another reboot, it turns out that the answer is no.

And boy did this have me stumped.

After roping in a friend, checking systemd settings, enabling SSYTEMD_LOG_LEVEL=debug and a bunch of other diagnostics strategies, nothing worked at all.

Nothing worked, that is, until I tried a slightly different search term and stumbled across this bug on Ubuntu Launchpad from 2006.

Now surely you would think that a bug from 2006 wouldn't still be biting me in the latter half of 2024, right?

.... right??

Ha.

Haha.

Hahahahaha.

This is a bug that is seemingly unfixable, but because it is obscurely documented in man sysctl.d, it is considered to be not a problem.

The key to my problems is this little paragraph here:

The settings configured with sysctl.d files will be applied early on boot. The network interface-specific options will also be applied individually for each network interface as it shows up in the system. (More specifically, net.ipv4.conf.*, net.ipv6.conf.*, net.ipv4.neigh.*  and net.ipv6.neigh.*).

Many sysctl parameters only become available when certain kernel modules are loaded. Modules are usually loaded on demand, e.g. when certain hardware is plugged in or network brought up. This means that systemd-sysctl.service(8) which runs during early boot will not configure such parameters if they become available after it has run.

To set such parameters, it is recommended to add an udev(7) rule to set those parameters when they become available. Alternatively, a slightly simpler and less efficient option is to add the module to modules-load.d(5), causing it to be loaded statically before sysctl settings are applied (see example below).

It turns out that sysctl tries to run its configuration before nf_conntrack is loaded. And as you might now guess, setting nf_conntrack_max does absolutely nothing if there is no nf_conntrack to max.

The workaround I went with was to add nf_conntrack to /etc/modules-load.d/modules.conf. This makes nf_conntrack load earlier in the boot process, allows sysctl to configure it, gives me more maximum connections to track for my public subnet, and stops my whole subnet periodically maxing its connections and effectively going offline every so often.

I could have probably done something fancier with udev rules, as the docs suggest, but at this point I'm just happy to have it working and not have my status monitoring frequently pinging me that one or more of my Amateur Radio services just went offline.

Sigh.