Wu-FTPD, the FTP daemon developed at Washington University, is the FTP server you find on most Unix systems. It is widely deployed, well-understood, and — over the course of the last year — has become the source of a remarkable number of security advisories.
I have been keeping a notebook of FTP-related Bugtraq entries for the past twelve months. Reviewing it this week, the pattern of the bugs is more instructive than any individual one. I want to write about that pattern, because it generalises.
The advisories themselves
A partial list, off the top of the notebook:
- "Palmetto" buffer overflow (CVE-1999-0202) in the wu-ftpd authentication handler. Pre-authentication remote root.
- MAIL_ADMIN format-string vulnerability in the SITE EXEC handler. Authenticated arbitrary code execution.
- wu_fnmatch denial of service: a glob pattern with too many wildcards causes recursive expansion to consume all CPU.
- Realpath buffer overflow in handling of long paths from authenticated users.
- skey_challenge buffer overflow in S/Key authentication.
- MESSAGE_FILE format-string in the welcome message expansion.
In twelve months. From a single product. Most of them in code that is essentially the same shape: take a string from the network, do some processing, write the result somewhere. The processing is where the bug lives.
The structural cause
Wu-FTPD is written in C. Most of the bugs are buffer overflows or format-string vulnerabilities. Both classes are direct consequences of two specific facts about C:
First, C strings have no length information. They are null-terminated arrays of characters. To know the length of a C string you must either keep track of it separately or scan the string for the null. To copy a C string into a buffer of size N, you must explicitly check whether the string fits.
Second, C does not check buffer bounds. If you allocate char buf[100] and write 200 bytes to it, you have just corrupted whatever happens to be in memory adjacent to buf. The compiler does not warn you. The runtime does not stop you. The corruption is silent.
When the corrupted memory contains, say, the return address of the current function on the stack, the consequence is that the attacker can choose where the function returns to. This is the basis of the stack buffer overflow exploit that Aleph One described in Phrack a few years ago and which the discipline has been thinking about ever since.
Format-string bugs are similar in spirit. The C printf family of functions takes a format string and arguments. If the format string itself is attacker-controlled — printf(user_input) — the attacker can include format specifiers that read or write arbitrary memory.
Why these bugs persist in mature code
This is the part worth thinking about. Wu-FTPD is not new. It is heavily reviewed code. Each new bug is found in code that has been read by hundreds of people, tested by thousands, deployed on millions of machines.
The answer, I think, has three parts.
The bugs are syntactically subtle. A buffer overflow in idiomatic C looks like ordinary code. The reader has to mentally simulate the program's memory state to spot the overflow. Most reviewers do not do this for every line. Most reviewers, on most reviews, are looking for high-level logic problems, not memory-layout problems.
The reviewing population has, until recently, not been looking for them. The tradition of hostile code review — reading code from the perspective of an attacker who wants to find a way to corrupt memory — is much newer than the tradition of code review for correctness. The two skills are different. Many people who can write good code cannot find buffer overflows in good-looking code.
The codebase has accreted unsafe primitives. strcpy, sprintf, gets, strcat — every one of these can produce a buffer overflow. They are present in idiomatic C and, for legacy reasons, in the codebase of every long-running project. Replacing them all is a non-trivial refactor.
What the community is doing about it
A few responses are emerging.
Auditing as a discipline. OpenBSD's auditing project is the most public example. They are going through the source of every package in the system, line by line, looking specifically for the patterns that lead to memory-corruption bugs. The track record is good — many bugs have been found and fixed before they were exploited in the wild.
Safer language alternatives. Some of the security-relevant code is being rewritten in languages that do not have this whole class of bug — Java, Perl, more recently a few Python-based projects. The performance and integration costs are real, but for code that is not on a hot path, the trade-off is starting to look favourable.
Compiler-level mitigations. StackGuard is the most prominent — a modification to the GCC compiler that places a sentinel value (a "canary") between local variables and the return address on the stack. If the canary is overwritten, the function refuses to return. This catches a particular class of stack overflow but not all memory-corruption bugs.
Library-level replacements. OpenBSD's strlcpy and strlcat are replacements for strcpy and strcat that take a buffer size and refuse to overflow. They are now the recommended choice in OpenBSD code; they are slowly being adopted elsewhere.
What an operator should do
If you run wu-ftpd, the practical advice is:
- Run the latest version. This is obvious but often skipped. The patches are published; the version numbers are tracked; the current advisory list is readable.
- Run it in a chroot. Wu-FTPD has a chroot mode. Use it. Even if a bug is exploited, the attacker is in a small jail.
- Run it as an unprivileged user where possible. Recent versions support starting up, dropping privileges, and continuing. This limits the consequences of a compromise.
- Watch for the next advisory. It is coming. The pattern says so.
Longer-term, consider whether you actually need FTP at all. SSH and SFTP cover most of FTP's use cases without the same family of bugs. Wu-FTPD is mature C code with a long bug tail. SSH is also mature C code with its own bug tail, but the protocol is newer, the attack surface is different, and the security posture of the actual implementations is, on the whole, better.
I have removed wu-ftpd from my own machines. I do not miss it.
The general lesson
The specific story is about wu-ftpd. The general story is about a class of bug — memory corruption from unchecked input — that is going to keep recurring as long as we write internet-facing daemons in C without systematic discipline.
The discipline is improving. The tools are improving. The compiler mitigations are improving. The language alternatives are slowly maturing. The audit work in projects like OpenBSD is producing measurable improvements.
None of this is fast. Wu-FTPD will continue to ship advisories for years. The next protocol's daemon will be just as bad until the same auditing happens to it. We are at the beginning of a long, slow, mostly invisible improvement, and most of the work that produces it is under-resourced and under-acknowledged.
This is, I think, going to be the shape of memory-safety for the next decade. The bugs continue. The defences improve. The asymmetry slowly favours the defender. The improvement is real but invisible from outside the work.