Linux 2.4 in production: a year on

A year ago I migrated my home firewall to Linux 2.4 with iptables. Time for an honest year-on assessment.

What has worked

Most of it. Specifically:

iptables has been stable. Twelve months of operation without rule-engine bugs. Performance has been adequate for my workload throughout.

Connection tracking has been reliable. The conntrack module has handled the various services I run without issues. The state-tracking has caught some interesting edge cases (long-running connections that should have timed out, malformed FTP data flows) and handled them sanely.

The structured ruleset has been maintainable. A year of changes — adding services, removing services, tweaking rules — has been clean. The named sub-chain organisation continues to be more legible than my old ipchains flat structure.

The new netfilter modules have been useful. Specifically, the string match and the recent module have let me write rules I could not have expressed in ipchains. Each has caught real attack patterns this year.

What has not worked perfectly

Three specific issues.

A subtle bug in the FTP conntrack helper. Active-mode FTP from a particular client occasionally triggered a kernel oops (recovered, but logged) due to a race condition in the helper. The bug was fixed in a 2.4.x update; my hosts were affected for about a month before the fix.

Memory pressure under attack load. During Code Red at peak, the conntrack table grew large enough that memory pressure became visible. The solution was to tune the conntrack timeouts and table size; the tuning required reading source to understand.

A few iptables-save/iptables-restore quirks. The save/restore tools have some specific edge cases where they round-trip incorrectly (rules with embedded quotation marks, certain exotic match arguments). I worked around these by hand-writing the saved rules in the canonical format the tools expect.

What I would tell someone considering the migration today

Just do it. A year of operational data confirms that 2.4 with iptables is production-ready. Operators still on ipchains should plan the migration; the cost is bounded and the result is meaningfully better.

Plan the rule rewrite, do not auto-translate. The mechanical translators from ipchains to iptables produce verbose, awkward output. A hand-rewrite into the new idiom (named sub-chains, state tracking, modular matches) produces a much better ruleset.

Tune the conntrack parameters. The defaults are reasonable for most workloads but can be too small for high-traffic deployments or under attack. Knowing the relevant sysctl values is worthwhile.

What is coming next

Linux 2.5 development is active. The early signals suggest 2.6 (the next stable series) will include:

  • Native kernel support for crypto operations (allowing things like in-kernel SSL acceleration).
  • Better filesystem support for various enterprise needs.
  • Continued netfilter improvements.
  • POSIX threading support that actually works.

None of these is imminent. 2.6 is probably 2003-2004. For my own infrastructure, 2.4 is going to be the production kernel for several more years.

A small reflection

The operational discipline of running a stable kernel version is meaningfully different from chasing the latest. 2.4.x has been my production version for a year; I expect it to be for at least two more. The slow progression is the right pace for serious operations.

For anyone running mission-critical Linux infrastructure: do not run development kernels in production. The latest stable release with current security patches is the right choice; the bleeding edge is for development environments.

More as 2002 develops.


Back to all writing