Hi, I'm Carl Kosher, and this talk is a look inside the 737 Communications Management Unit
and some of the interesting things we've learned while reverse engineering systems.
But first, the obligatory who am I slide. So I'm a research scientist at the University of
Washington, but previously I was a postdoc at UC San Diego, where I did a lot of this work.
In general, I'm a low-level system security researcher, so I've worked on things such as
RFID systems, looked at enhanced driver's licenses, transit cards, and built some
anti-relaying defenses. I also led one of the original car hacking teams, so we were the first
to demonstrate a full remote compromise of a vehicle over the internet. I've reverse-engineered
wireless SCADA systems involved in power distribution. I've done some proof-of-concept
malware against DNA sequencers and analysis pipelines, and I'm also working on wet lab
automation systems these days. So of course I'm not doing this work alone. There are a number of
people involved in the AEROSEP project, and this project really came out of the previous car hacking
research that we did. So of the people listed here, the ones in the top row and listed in orange
were part of that original car hacking team. And that project had some pretty big impact.
She thinks she's going to be able to stop right at those cones. Let's make sure that she can't,
and she's going to drive right through them. We have complete control of that brake. All right,
there we go. Oh no, no, no, no, no, no, no, no, no, no. Brakes didn't work, right? Oh my god,
I can't operate the brakes at all. Oh my word, that is frightening.
So obviously these results were pretty impactful, and inspired DARPA's approximately 70 million
dollar high assurance cyber military systems, or HACMS program, to make systems provably secure
against attack. But avionic systems are already engineered to a high assurance level, albeit for
safety, not security. So we wondered if these safety assurance requirements translated into
improved security. So the 777, 787, and A380 all have new and interesting architectures,
but you can't really buy spare parts for those off of eBay. As it turns out, as 737 airframes are
retired, parts do become available from auction sites, and so that's why we decided to focus on
737s. And there's actually a great website with a ton of information about the 737 that we used in
our research, and they've even released a book. After investigating how the systems were integrated,
we decided to focus on the communications management unit, as it connects the rest of
the avionic systems to the outside world. So for example, the CMU can take messages through the VHF,
HF, or satellite radios, and relay them to other avionic systems. It's also used for sending
messages between pilots and ground. So since the CMU is sort of this central hub for all this
messaging, it has a pretty large attack surface, and it's a pretty juicy target for looking for
vulnerabilities. So aside from the display unit, all these systems live in the electronics and
equipment bay. This is a view from the E&E bay from the outside, and this is what it looks like from
the inside. So all of these boxes are called LRUs, or line replaceable units, because they slide in
and out and connect with an interconnect backplane, making them very easy to swap out for maintenance
or repair. So in the cockpit, pilots interact with these systems through these control display units.
Originally, there were separate display units for messaging and for the flight management system,
but they've since been combined into these multi-control display units, or NCDUs, where you can
access different systems through one common interface. So for example, on the left, the MCDU
is interacting with the flight management system, while on the right, the MCDU is interacting with
the communications management unit, which allows the pilots to send and receive messages, request
weather reports, do a whole bunch of things. So to put the communications management unit in context,
we first need to talk about ACARS, which started out as a simple messaging system
designed to solve one very specific problem for airlines, and that was ensuring that they paid
their employees only as much as they were entitled to. And the way that it did this is through
reporting these OOOI statuses, which basically indicates whether the plane is out of the gate,
off the ground, on the ground, or back in the gate. And this is how airlines determine
how pilots and flight attendants are paid. But since then, it's been generalized to transfer
avionics data. So for example, you can push a flight plan over ACARS, or you can request a
clearance from air traffic control. Airlines typically have their own applications that they
add on top of this. So for example, if you've ever received connecting gate information while you're
in flight, that information was likely relayed through ACARS, and it's increasingly being used
to replace traditional voice-based air traffic control communications, especially in Europe.
So ACARS works over multiple RF link types, including the legacy VHF ACARS radios,
the new VHF VDL2 data radios, over the horizon HF data radios, and even satellite links. And
since this standard came out in 1978, and has largely been unchanged since then, these messages
typically aren't encrypted. And when they are, they're often encrypted using questionable methods.
So the communications management unit has some interesting requirements.
So obviously, air traffic control is flight critical, and so that software needs to be
certified to a certain assurance level. But airlines want to add their own custom apps to
these communication management units. And it's cost prohibitive and extremely time intensive
to certify these. So how do CMUs allow uncertified apps to not taint certification?
As it turns out, there are two major vendors of CMUs for the SEM37, and they each take
a different approach to solving this problem. So the two major versions of the CMU for the SEM37
are the Collins CMU-900 and the Honeywell Mk3. So the Collins CMU-900 has 46 for running applications,
386 to handle IO, and does inter-process communication through dual-ported RAM between
the two processors. It runs the Vertex real-time operating system, which I'll talk a little bit
about later. Makes extensive use of x86 segments, which makes reverse engineering a giant pain.
And they invented their own virtual machine instruction set that apps are compiled for,
and this allows apps to be isolated without tainting certification. The other major player
is the Honeywell Mk3, and it has a slightly beefier Pentium for running its applications,
and uses PowerPC to handle IO. And similar to the CMU-900, it uses dual-ported RAM for
inter-processor communication. However, the Pentium runs Honeywell's custom DOS real-time
operating system, while the PowerPCs use Vertex, like the CMU-900. On the Honeywell Mk3, apps are
database-driven, and the database processing code is written in a verifiable language
that I'll describe later. So let's take a look inside some of these units.
So this is the CMU-900 main processor board, and on there you have the 486, RAM, a CPLD for glue
logic, a timer and interrupt controller, a real-time clock, and a serial interface chip.
The IO board is responsible for interfacing with all the 429 buses. On the board we have the 386,
the dual-ported RAM, and some custom ASICs that we're pretty sure are 429 transceivers,
along with some analogy bits to interface with the 429 bus.
So this is the Honeywell Mk3 processor board, and underneath that heatsink is a Pentium.
There's also RAM, some glue logic, and a very annoying non-volatile storage system,
which relies on a battery, which of course has gone dead by now, basically breaking the system.
So the IO board on the Honeywell Mk3 is a rather interesting beast. The first thing that pops out
to me is that there are two PowerPCs, and through our reverse engineering, it seems like only one
of them is used for IO, while the second supports reflashing the system. So in the middle there is
a CPLD, which we suspect interfaces the 429 transceivers with the PowerPC chips. There are
a number of interesting connectivity options on here. So for example, there's an Ethernet
controller on there with magnetics, probably for supporting a gate link upgrade, whereby you can
push messages to the CMU over Wi-Fi. There's also a DSP chip next to an ADC, and we suspect that's
used for sending ACARS messages over legacy VHF radios. But also interestingly enough, there's a
V.90 modem chipset, and I'm not really sure why this exists, especially with that other DSP chip
on there. Perhaps they were leaving the option open to plug a CMU into a phone line. Not really
sure that would make sense on an aircraft, but it might make sense for maintenance.
And of course, there's lots of JTAG connectors everywhere. It warms my heart. So let's take a
deep dive into how the Honeywell Mk3 is actually constructed. So to begin our reverse engineering
journey, we sacrificed one board, removed the conformal coding, pulled off the flash chips,
dumped the flash chips, and ran strings on the binaries. And what we saw next was surprising.
This program must be run under Win32. Wow. By looking for PE headers, we were able to carve
the flash ROM into a number of interesting files. It also turns out the flash had multiple
partitions. One seemed to be for flight mode, another seemed to be for update mode, and another
for recovery. So as it turns out, Honeywell wrote their own realtime OS called DOS. And one of the
interesting things about DOS is that its scheduler was ported to Java and formally verified by NASA
to ensure that it always met its realtime requirements. As you might have guessed,
it seems heavily inspired by Windows. It loads statically linked portable executables.
The APIs look a lot like Win32 or the Windows NT kernel API. And in fact, while reverse engineering
the kernel, if I didn't know what a parameter to a function was, I would just go to MSDN,
and it would probably have the right answer. And it even has a concept of a registry,
which is basically just a small binary blob of configuration parameters.
So why is DOS so similar to Windows? Is it because they wanted to develop the CMU code
against a Windows target? Well, this is pure speculation, but here's a map of Microsoft
campus in Redmond, Washington. And right across the street is Honeywell Aerospace. So again,
just speculation, but I'm guessing it was likely developed by ex-Microsoft employees.
So outside of the kernel, the bulk of the CMU code is in a single executable called CMU core.
And this is written in a language called SDL, the Specification and Description Language.
And this was created by the ITU. And it is a language that only telecoms can love.
Data structures are defined by ASN 1, and basically it's intended for building telephone
switches. If you're familiar with Erlang, it's spiritually similar. SDL lets you define loosely
coupled state machines that pass messages to each other. So it's ideal for protocol handling.
SDL has a couple of forms that it can take. There is both a graphical and a textual form.
The graphical form is shown here. So basically you write state machines that send and receive
messages or are triggered by timers. And you can also execute arbitrary C code.
Now there's a companion language to SDL called MSC, or Message Sequence Charts. And this lets
you define protocol flows and test cases to validate your SDL against. And by writing test
cases for handling 429 and ACARS messages, they can verify that the SDL code is correct.
So like I said, SDL programs are composed of loosely coupled state machines called tasks.
And in CMU Core, there are a lot of tasks. And many of them send messages to each other.
Now, I'm not going to read through this entire list here, but I encourage you to pause the video
and take a look at the list. So at this point, we have SDL code that is verified. And that SDL
code is then compiled down to C code by a product called ObjectGeode, which was made by a company
named, confusingly enough, Verilog. No relation to the hardware description language.
But how do airlines make their own apps? Well, it turns out that the SDL code actually reads
app logic from databases stored in Flash. So I'm sure you've all heard of SQL, but there's also
another database language called QWEL. And the database on the Honeywell Mark 3 is an embedded
version of Ingress QWEL. Now, the crazy thing is that queries are directly embedded in C files,
which are then preprocessed into other C files, which are then compiled and linked against.
So if you want to write a query that iterates through employee names, you can write it sort of
as a pseudo for loop, as shown here. And that will be turned into a database query, and that printf
will be executed for every row returned. So unfortunately, these queries are directly
converted into C code, so there are no query strings in the executable. And furthermore,
we don't even have any schema information, since it's known at compile time, so there is no reason
to store it in Flash. So how are these apps created? Well, Honeywell has this tool called
the Ground-Based Software Tool, which is used both to create HGI, the Honeywell-generated
information, as well as the AMI, or the Airline Modifiable Information. And basically, this tool
lets you populate the database tables that define your app. And this is used both for
certified apps made by Honeywell, as well as custom operational apps made by airlines.
So how do these state machines actually interact with real hardware? Well, these SDL messages are
serialized and sent through DOS's inter-process communication mechanism, mailboxes, to a process
called DP-RAM-IO. And DP-RAM-IO takes these messages and shoves it into the dual-ported RAM,
which is shared by the PowerPCs. And the PowerPCs handle the low-level interface to the hardware.
So as I said before, one seems to handle the PCMCI update cards and does little else. The
other one seems to handle almost everything. Unlike the Pentium chip, the PowerPCs run the
Vertex real-time OS. And the code on the PowerPCs is also written in SDL. So this makes it fairly
easy to pass messages back and forth between the x86 and PowerPCs. So Vertex is a fairly old,
but well-proven real-time operating system. And to emphasize this point, here's an ad from 1983,
where they will buy you a VW bug if you find a bug in their OS.
So there's one tricky bit about passing messages between the x86 and PowerPC.
And that is x86 is little-endian, while PowerPC is big-endian. And so the way that they actually
solve this is with one giant switch statement based on the message type to do the endianness
swap. And that code is written in C, which of course is far more promising for bugs.
And in fact, we found some. So we found some signed versus unsigned confusion and some integer
overflows as well. Fortunately, they do not seem to be exploitable. So now that we have a general
idea of how the CMU works, let's actually try going and see if we can find some real bugs.
So static analysis of this auto-generated code is hard. So we'd like to do some dynamic analysis.
Unfortunately, getting the CMU to boot was challenging. It does a lot of self-tests on
startup. If something is wrong, it just won't boot. And we spent several months with it just
blinking the fail light at us. Fortunately, we found some serial ports on the front maintenance
port, which gave us some clues as to what was going wrong. So when we powered up the CMU,
it would state that it was starting its built-in functional test. Unfortunately, after a minute,
it would start spitting out error codes. And as it turns out, this was because the APM was missing.
Now the APM is the airline personality module, which has the tail number, the aircraft configuration,
the type of aircraft it is, and a host of other settings. So if you need to replace the CMU,
you don't have to reprogram it. You just pull it out, slide in a new one, connects to the existing
airplane personality module, and the new CMU knows exactly what it needs to do. Unfortunately,
the only APM we could find was for the Collins CMU-900, which despite being somewhat specified
by an airing standard, was totally incompatible, both logically and electrically. Fortunately,
the CMU saves a copy of the last known good APM image in Flash. And so we were actually able to
create an APM emulator with an Arduino. It basically just emulated an i-squart-z EEPROM,
and we used the APM image that we pulled out of Flash. Now the second issue that we ran into
was NVRAM battery was bad, and we didn't actually know that this was preventing the CMU from booting.
We basically had to take one of the cryptic error codes out of the serial console,
find where that was in the binary, figure out what triggered that string to be printed,
and then discover that it was doing a test on the battery voltage. As it turns out, we just
removed the battery and replaced it with a power supply, and that was good enough to make it boot.
So there was actually a second serial port exposed on the front maintenance port.
And once the system booted, if you pressed enter, you got an interesting maintenance menu.
Another weird behavior is that if you provided input to the first serial port,
you would get another menu on the second serial port. So now that we can get the CMU to boot,
let's try doing something with it. And to do that, we need an MCDU and a VHF data radio.
Unfortunately, we didn't have those, so we ended up writing emulators for them and connected them
to the real CMU over the 429 buses. And this really kicked off the development of a test bed
we called Triton to support this kind of testing and security research. So the Triton test bed is
designed to support a mixture of both real and emulated devices. So for example, we have a real
communications management unit, but we don't have a real printer or airborne data loader unit.
So we've created emulators for those. And these devices are connected through a 429 message
routing daemon, which we call R429D. And R429D both supports real 429 interfaces, as well as
accepting connections from emulated avionics units. Similarly for ACARS, we have a separate daemon
that emulates different network segments. For example, we have one daemon that emulates the
air segment, so all the VHF data radios are connected to it. We also have a separate instance
for the ground segment, and we have a virtual ground station that bridges the two segments.
So a little bit about the 429 devices that the Triton test bed supports.
This box here is an Ethernet to 429 converter, and this was one of the first things that we
targeted. Unfortunately, it can only be configured with a static IP address,
and it kept getting virtually bricked by having its IP address randomly changed to something
that we didn't know. And we actually suspect that UCSD was doing network scans against this device,
and it just so happened that it sent a packet to reconfigure the IP address to some garbage
address. So after a couple of times of opening this up, pulling off the EEPROM, putting in the
correct IP address, soldering it back on, getting it back on the network, only to have it bricked
again, we decided to find a better solution. One of the primary interfaces that we use now
is this Ballard USB to 429 interface. Unfortunately, it is rather expensive.
We did also develop our own custom USB to 429 interface, which we called Scorpion.
And if you ever saw the pilot episode of that TV show, you might understand why we called it that.
It's basically an ATmegaU2 with a Holt 429 transceiver. It supports four transmit channels
and eight receive channels. And this basically interfaces with the R429D daemon to expose a
TCP IP interface to the 429 buses. So now that we have all this wired up,
we can start doing some really interesting things. So on the left here, we have our virtual MCDU,
which lets us control our real CMU. Now connected to the real CMU is our virtual VHF data radio,
which we can send real ACARS messages that we pick off over the air, alter the tail number so
that our CMU processes them, which then forwards it to our virtual printer program, which sends
it to a real credit card receipt printer. So interacting with the CMU is all well and good,
but we'd really like to have some low-level debugging capabilities. And as it turns out,
the main processor board has a Pentium debug port. Now, unfortunately, the Pentium Probe mode
is undocumented. Details are in the infamous Appendix H of the P5 external interface specification,
which was available only under NDA from Intel. And back in the mid-90s, there were a lot of
rants in Dr. Dobbs' journal about this. There was one beta version of the P5 external interface
specification that had these details, but all these copies were supposedly recalled by Intel
and destroyed. So, of course, there's no copies of this on the web. But while we were searching
for it, we did find that there was one physical copy of an unknown version at an undisclosed
university lab halfway across the world. So we sent them an email and asked if they had it,
and if so, what version it was, and that we might be interested in a scan of it.
And so they wrote back with a scan of the title page, which was version 1.3. And unfortunately,
we needed version 3. So before we could write them back and say thanks, but no thanks,
they wrote us back and said they found something very interesting inside. And sandwiched between
the pages of version 1.3 was version 3. And they happened to send us a scan of the entire thing,
including details of the Pentium micro-op instruction format, which is exactly what
you need to use the Pentium's probe mode. So armed with this information, I added P5
debugging support to OpenOCD. And so now we can use GDB against the CMU. Unfortunately,
if you halt the hardware, the watchdog timers on the other processors will notice something's
amiss and reset the entire system. And the solution for this is to basically run the CMU's
x86 firmware in QMU, but relaying messages to the power PCs over JTAG. And this is based on
some of my earlier dissertation work on a system called Surrogates, which lets you run firmware
in QMU, but proxies peripheral access over JTAG in near real time. So now we can instrument the
CMU firmware, collect IO traces, do snapshots, and do all sorts of interesting dynamic analysis.
So in summary, the communications management unit supports flight critical operations,
which requires the firmware to be certified. But airlines desire customization. And through
our reverse engineering, we've identified some somewhat unusual ways that vendors have isolated
these uncertified apps from the rest of the certified system. And finally, we built a flexible
testbed for further security research on avionic systems. So if you would like more information
about our Triton testbed or surrogate system, check out our site at aerosec.org. Thanks!