start	end	text
4540	9780	Hi, I'm Carl Kosher, and this talk is a look inside the 737 Communications Management Unit
9780	14180	and some of the interesting things we've learned while reverse engineering systems.
14260	19460	But first, the obligatory who am I slide. So I'm a research scientist at the University of
19460	24700	Washington, but previously I was a postdoc at UC San Diego, where I did a lot of this work.
24700	29160	In general, I'm a low-level system security researcher, so I've worked on things such as
29160	35840	RFID systems, looked at enhanced driver's licenses, transit cards, and built some
35840	42300	anti-relaying defenses. I also led one of the original car hacking teams, so we were the first
42300	49160	to demonstrate a full remote compromise of a vehicle over the internet. I've reverse-engineered
49160	55920	wireless SCADA systems involved in power distribution. I've done some proof-of-concept
55920	62800	malware against DNA sequencers and analysis pipelines, and I'm also working on wet lab
62800	69120	automation systems these days. So of course I'm not doing this work alone. There are a number of
69120	76180	people involved in the AEROSEP project, and this project really came out of the previous car hacking
76180	83860	research that we did. So of the people listed here, the ones in the top row and listed in orange
83860	91580	were part of that original car hacking team. And that project had some pretty big impact.
91580	94820	She thinks she's going to be able to stop right at those cones. Let's make sure that she can't,
94820	98420	and she's going to drive right through them. We have complete control of that brake. All right,
98420	106300	there we go. Oh no, no, no, no, no, no, no, no, no, no. Brakes didn't work, right? Oh my god,
106300	111960	I can't operate the brakes at all. Oh my word, that is frightening.
113480	119200	So obviously these results were pretty impactful, and inspired DARPA's approximately 70 million
119200	125860	dollar high assurance cyber military systems, or HACMS program, to make systems provably secure
125860	131920	against attack. But avionic systems are already engineered to a high assurance level, albeit for
131920	139400	safety, not security. So we wondered if these safety assurance requirements translated into
139400	148080	improved security. So the 777, 787, and A380 all have new and interesting architectures,
148080	155720	but you can't really buy spare parts for those off of eBay. As it turns out, as 737 airframes are
155720	162000	retired, parts do become available from auction sites, and so that's why we decided to focus on
162000	168760	737s. And there's actually a great website with a ton of information about the 737 that we used in
168760	174400	our research, and they've even released a book. After investigating how the systems were integrated,
174400	179180	we decided to focus on the communications management unit, as it connects the rest of
179180	188240	the avionic systems to the outside world. So for example, the CMU can take messages through the VHF,
188240	194500	HF, or satellite radios, and relay them to other avionic systems. It's also used for sending
194500	200940	messages between pilots and ground. So since the CMU is sort of this central hub for all this
200940	206240	messaging, it has a pretty large attack surface, and it's a pretty juicy target for looking for
206240	213300	vulnerabilities. So aside from the display unit, all these systems live in the electronics and
213300	220720	equipment bay. This is a view from the E&E bay from the outside, and this is what it looks like from
220720	229100	the inside. So all of these boxes are called LRUs, or line replaceable units, because they slide in
229100	235140	and out and connect with an interconnect backplane, making them very easy to swap out for maintenance
235140	241740	or repair. So in the cockpit, pilots interact with these systems through these control display units.
241740	248000	Originally, there were separate display units for messaging and for the flight management system,
248000	253700	but they've since been combined into these multi-control display units, or NCDUs, where you can
253700	259880	access different systems through one common interface. So for example, on the left, the MCDU
259880	266200	is interacting with the flight management system, while on the right, the MCDU is interacting with
266200	272400	the communications management unit, which allows the pilots to send and receive messages, request
272400	279420	weather reports, do a whole bunch of things. So to put the communications management unit in context,
279420	284880	we first need to talk about ACARS, which started out as a simple messaging system
284880	291180	designed to solve one very specific problem for airlines, and that was ensuring that they paid
291180	298360	their employees only as much as they were entitled to. And the way that it did this is through
298360	307820	reporting these OOOI statuses, which basically indicates whether the plane is out of the gate,
307820	313380	off the ground, on the ground, or back in the gate. And this is how airlines determine
313380	318560	how pilots and flight attendants are paid. But since then, it's been generalized to transfer
318560	325520	avionics data. So for example, you can push a flight plan over ACARS, or you can request a
325520	331120	clearance from air traffic control. Airlines typically have their own applications that they
331120	336780	add on top of this. So for example, if you've ever received connecting gate information while you're
336780	342460	in flight, that information was likely relayed through ACARS, and it's increasingly being used
342460	349180	to replace traditional voice-based air traffic control communications, especially in Europe.
349180	356040	So ACARS works over multiple RF link types, including the legacy VHF ACARS radios,
356040	364680	the new VHF VDL2 data radios, over the horizon HF data radios, and even satellite links. And
364680	370300	since this standard came out in 1978, and has largely been unchanged since then, these messages
370300	377480	typically aren't encrypted. And when they are, they're often encrypted using questionable methods.
378000	381910	So the communications management unit has some interesting requirements.
382300	387480	So obviously, air traffic control is flight critical, and so that software needs to be
387480	393340	certified to a certain assurance level. But airlines want to add their own custom apps to
393340	399000	these communication management units. And it's cost prohibitive and extremely time intensive
399000	406880	to certify these. So how do CMUs allow uncertified apps to not taint certification?
407620	414320	As it turns out, there are two major vendors of CMUs for the SEM37, and they each take
414680	422280	a different approach to solving this problem. So the two major versions of the CMU for the SEM37
422280	431580	are the Collins CMU-900 and the Honeywell Mk3. So the Collins CMU-900 has 46 for running applications,
431580	437360	386 to handle IO, and does inter-process communication through dual-ported RAM between
437360	443000	the two processors. It runs the Vertex real-time operating system, which I'll talk a little bit
443000	451720	about later. Makes extensive use of x86 segments, which makes reverse engineering a giant pain.
451720	456660	And they invented their own virtual machine instruction set that apps are compiled for,
456660	462420	and this allows apps to be isolated without tainting certification. The other major player
462420	468620	is the Honeywell Mk3, and it has a slightly beefier Pentium for running its applications,
468620	475380	and uses PowerPC to handle IO. And similar to the CMU-900, it uses dual-ported RAM for
475380	482280	inter-processor communication. However, the Pentium runs Honeywell's custom DOS real-time
482280	490720	operating system, while the PowerPCs use Vertex, like the CMU-900. On the Honeywell Mk3, apps are
490720	495440	database-driven, and the database processing code is written in a verifiable language
495440	499860	that I'll describe later. So let's take a look inside some of these units.
499860	507480	So this is the CMU-900 main processor board, and on there you have the 486, RAM, a CPLD for glue
507480	513980	logic, a timer and interrupt controller, a real-time clock, and a serial interface chip.
514240	522160	The IO board is responsible for interfacing with all the 429 buses. On the board we have the 386,
522160	528600	the dual-ported RAM, and some custom ASICs that we're pretty sure are 429 transceivers,
528600	533080	along with some analogy bits to interface with the 429 bus.
533900	539860	So this is the Honeywell Mk3 processor board, and underneath that heatsink is a Pentium.
539860	546460	There's also RAM, some glue logic, and a very annoying non-volatile storage system,
546460	552320	which relies on a battery, which of course has gone dead by now, basically breaking the system.
552880	559360	So the IO board on the Honeywell Mk3 is a rather interesting beast. The first thing that pops out
559360	564540	to me is that there are two PowerPCs, and through our reverse engineering, it seems like only one
564540	570540	of them is used for IO, while the second supports reflashing the system. So in the middle there is
570700	578060	a CPLD, which we suspect interfaces the 429 transceivers with the PowerPC chips. There are
578140	583060	a number of interesting connectivity options on here. So for example, there's an Ethernet
583060	590240	controller on there with magnetics, probably for supporting a gate link upgrade, whereby you can
590240	599580	push messages to the CMU over Wi-Fi. There's also a DSP chip next to an ADC, and we suspect that's
599580	607840	used for sending ACARS messages over legacy VHF radios. But also interestingly enough, there's a
607840	615600	V.90 modem chipset, and I'm not really sure why this exists, especially with that other DSP chip
615600	622660	on there. Perhaps they were leaving the option open to plug a CMU into a phone line. Not really
622660	628800	sure that would make sense on an aircraft, but it might make sense for maintenance.
629780	636340	And of course, there's lots of JTAG connectors everywhere. It warms my heart. So let's take a
636340	643620	deep dive into how the Honeywell Mk3 is actually constructed. So to begin our reverse engineering
643620	649500	journey, we sacrificed one board, removed the conformal coding, pulled off the flash chips,
649500	655720	dumped the flash chips, and ran strings on the binaries. And what we saw next was surprising.
656680	665440	This program must be run under Win32. Wow. By looking for PE headers, we were able to carve
665440	671180	the flash ROM into a number of interesting files. It also turns out the flash had multiple
671180	677920	partitions. One seemed to be for flight mode, another seemed to be for update mode, and another
677920	685940	for recovery. So as it turns out, Honeywell wrote their own realtime OS called DOS. And one of the
685940	691800	interesting things about DOS is that its scheduler was ported to Java and formally verified by NASA
691800	696880	to ensure that it always met its realtime requirements. As you might have guessed,
696880	703120	it seems heavily inspired by Windows. It loads statically linked portable executables.
703200	711340	The APIs look a lot like Win32 or the Windows NT kernel API. And in fact, while reverse engineering
711340	717380	the kernel, if I didn't know what a parameter to a function was, I would just go to MSDN,
717380	724540	and it would probably have the right answer. And it even has a concept of a registry,
724540	729540	which is basically just a small binary blob of configuration parameters.
730080	736180	So why is DOS so similar to Windows? Is it because they wanted to develop the CMU code
736180	742780	against a Windows target? Well, this is pure speculation, but here's a map of Microsoft
742780	752500	campus in Redmond, Washington. And right across the street is Honeywell Aerospace. So again,
753020	759680	just speculation, but I'm guessing it was likely developed by ex-Microsoft employees.
760500	767400	So outside of the kernel, the bulk of the CMU code is in a single executable called CMU core.
767400	773760	And this is written in a language called SDL, the Specification and Description Language.
773760	780460	And this was created by the ITU. And it is a language that only telecoms can love.
780460	786540	Data structures are defined by ASN 1, and basically it's intended for building telephone
786540	793460	switches. If you're familiar with Erlang, it's spiritually similar. SDL lets you define loosely
793460	799160	coupled state machines that pass messages to each other. So it's ideal for protocol handling.
799920	805680	SDL has a couple of forms that it can take. There is both a graphical and a textual form.
805720	811920	The graphical form is shown here. So basically you write state machines that send and receive
811920	818260	messages or are triggered by timers. And you can also execute arbitrary C code.
818860	825880	Now there's a companion language to SDL called MSC, or Message Sequence Charts. And this lets
825880	832200	you define protocol flows and test cases to validate your SDL against. And by writing test
832200	838720	cases for handling 429 and ACARS messages, they can verify that the SDL code is correct.
838980	844260	So like I said, SDL programs are composed of loosely coupled state machines called tasks.
844260	850260	And in CMU Core, there are a lot of tasks. And many of them send messages to each other.
850400	855200	Now, I'm not going to read through this entire list here, but I encourage you to pause the video
855200	862340	and take a look at the list. So at this point, we have SDL code that is verified. And that SDL
862340	868720	code is then compiled down to C code by a product called ObjectGeode, which was made by a company
868720	874860	named, confusingly enough, Verilog. No relation to the hardware description language.
874860	880880	But how do airlines make their own apps? Well, it turns out that the SDL code actually reads
880880	887820	app logic from databases stored in Flash. So I'm sure you've all heard of SQL, but there's also
887820	894040	another database language called QWEL. And the database on the Honeywell Mark 3 is an embedded
894040	901020	version of Ingress QWEL. Now, the crazy thing is that queries are directly embedded in C files,
901020	906020	which are then preprocessed into other C files, which are then compiled and linked against.
906680	911700	So if you want to write a query that iterates through employee names, you can write it sort of
911700	918700	as a pseudo for loop, as shown here. And that will be turned into a database query, and that printf
918700	925280	will be executed for every row returned. So unfortunately, these queries are directly
925280	931000	converted into C code, so there are no query strings in the executable. And furthermore,
931000	936200	we don't even have any schema information, since it's known at compile time, so there is no reason
936200	943760	to store it in Flash. So how are these apps created? Well, Honeywell has this tool called
943760	951300	the Ground-Based Software Tool, which is used both to create HGI, the Honeywell-generated
951300	959160	information, as well as the AMI, or the Airline Modifiable Information. And basically, this tool
959160	966160	lets you populate the database tables that define your app. And this is used both for
966160	972040	certified apps made by Honeywell, as well as custom operational apps made by airlines.
973000	979080	So how do these state machines actually interact with real hardware? Well, these SDL messages are
979080	986500	serialized and sent through DOS's inter-process communication mechanism, mailboxes, to a process
986500	993820	called DP-RAM-IO. And DP-RAM-IO takes these messages and shoves it into the dual-ported RAM,
993820	1001280	which is shared by the PowerPCs. And the PowerPCs handle the low-level interface to the hardware.
1001280	1009560	So as I said before, one seems to handle the PCMCI update cards and does little else. The
1009560	1015400	other one seems to handle almost everything. Unlike the Pentium chip, the PowerPCs run the
1015400	1022900	Vertex real-time OS. And the code on the PowerPCs is also written in SDL. So this makes it fairly
1022900	1031000	easy to pass messages back and forth between the x86 and PowerPCs. So Vertex is a fairly old,
1031000	1037320	but well-proven real-time operating system. And to emphasize this point, here's an ad from 1983,
1037320	1042120	where they will buy you a VW bug if you find a bug in their OS.
1043960	1048980	So there's one tricky bit about passing messages between the x86 and PowerPC.
1049440	1056860	And that is x86 is little-endian, while PowerPC is big-endian. And so the way that they actually
1056860	1063020	solve this is with one giant switch statement based on the message type to do the endianness
1063020	1070200	swap. And that code is written in C, which of course is far more promising for bugs.
1071000	1077880	And in fact, we found some. So we found some signed versus unsigned confusion and some integer
1077880	1084700	overflows as well. Fortunately, they do not seem to be exploitable. So now that we have a general
1084700	1090520	idea of how the CMU works, let's actually try going and see if we can find some real bugs.
1091100	1097740	So static analysis of this auto-generated code is hard. So we'd like to do some dynamic analysis.
1098220	1105540	Unfortunately, getting the CMU to boot was challenging. It does a lot of self-tests on
1105540	1112240	startup. If something is wrong, it just won't boot. And we spent several months with it just
1112240	1118840	blinking the fail light at us. Fortunately, we found some serial ports on the front maintenance
1118840	1124780	port, which gave us some clues as to what was going wrong. So when we powered up the CMU,
1124780	1130340	it would state that it was starting its built-in functional test. Unfortunately, after a minute,
1130340	1136720	it would start spitting out error codes. And as it turns out, this was because the APM was missing.
1136720	1144500	Now the APM is the airline personality module, which has the tail number, the aircraft configuration,
1144500	1149820	the type of aircraft it is, and a host of other settings. So if you need to replace the CMU,
1149820	1156160	you don't have to reprogram it. You just pull it out, slide in a new one, connects to the existing
1156160	1162880	airplane personality module, and the new CMU knows exactly what it needs to do. Unfortunately,
1162880	1170580	the only APM we could find was for the Collins CMU-900, which despite being somewhat specified
1170580	1178740	by an airing standard, was totally incompatible, both logically and electrically. Fortunately,
1178740	1186240	the CMU saves a copy of the last known good APM image in Flash. And so we were actually able to
1186240	1193120	create an APM emulator with an Arduino. It basically just emulated an i-squart-z EEPROM,
1193120	1198560	and we used the APM image that we pulled out of Flash. Now the second issue that we ran into
1198560	1205940	was NVRAM battery was bad, and we didn't actually know that this was preventing the CMU from booting.
1206360	1211320	We basically had to take one of the cryptic error codes out of the serial console,
1211320	1218220	find where that was in the binary, figure out what triggered that string to be printed,
1218220	1224580	and then discover that it was doing a test on the battery voltage. As it turns out, we just
1224580	1229960	removed the battery and replaced it with a power supply, and that was good enough to make it boot.
1230400	1235260	So there was actually a second serial port exposed on the front maintenance port.
1235260	1242000	And once the system booted, if you pressed enter, you got an interesting maintenance menu.
1242200	1248100	Another weird behavior is that if you provided input to the first serial port,
1248100	1255740	you would get another menu on the second serial port. So now that we can get the CMU to boot,
1255740	1263200	let's try doing something with it. And to do that, we need an MCDU and a VHF data radio.
1263200	1268280	Unfortunately, we didn't have those, so we ended up writing emulators for them and connected them
1268280	1275880	to the real CMU over the 429 buses. And this really kicked off the development of a test bed
1275880	1283680	we called Triton to support this kind of testing and security research. So the Triton test bed is
1283680	1290740	designed to support a mixture of both real and emulated devices. So for example, we have a real
1290740	1297580	communications management unit, but we don't have a real printer or airborne data loader unit.
1297580	1304840	So we've created emulators for those. And these devices are connected through a 429 message
1304840	1314340	routing daemon, which we call R429D. And R429D both supports real 429 interfaces, as well as
1314340	1320360	accepting connections from emulated avionics units. Similarly for ACARS, we have a separate daemon
1320360	1326880	that emulates different network segments. For example, we have one daemon that emulates the
1326880	1333560	air segment, so all the VHF data radios are connected to it. We also have a separate instance
1333560	1338500	for the ground segment, and we have a virtual ground station that bridges the two segments.
1339920	1345120	So a little bit about the 429 devices that the Triton test bed supports.
1345120	1350980	This box here is an Ethernet to 429 converter, and this was one of the first things that we
1350980	1356660	targeted. Unfortunately, it can only be configured with a static IP address,
1356660	1361920	and it kept getting virtually bricked by having its IP address randomly changed to something
1361920	1370240	that we didn't know. And we actually suspect that UCSD was doing network scans against this device,
1370240	1376460	and it just so happened that it sent a packet to reconfigure the IP address to some garbage
1376460	1382620	address. So after a couple of times of opening this up, pulling off the EEPROM, putting in the
1382620	1387560	correct IP address, soldering it back on, getting it back on the network, only to have it bricked
1387560	1393540	again, we decided to find a better solution. One of the primary interfaces that we use now
1393540	1400120	is this Ballard USB to 429 interface. Unfortunately, it is rather expensive.
1400600	1407400	We did also develop our own custom USB to 429 interface, which we called Scorpion.
1407400	1414040	And if you ever saw the pilot episode of that TV show, you might understand why we called it that.
1414040	1422400	It's basically an ATmegaU2 with a Holt 429 transceiver. It supports four transmit channels
1422400	1430080	and eight receive channels. And this basically interfaces with the R429D daemon to expose a
1430080	1436900	TCP IP interface to the 429 buses. So now that we have all this wired up,
1436900	1442980	we can start doing some really interesting things. So on the left here, we have our virtual MCDU,
1442980	1451740	which lets us control our real CMU. Now connected to the real CMU is our virtual VHF data radio,
1451740	1458280	which we can send real ACARS messages that we pick off over the air, alter the tail number so
1458280	1464580	that our CMU processes them, which then forwards it to our virtual printer program, which sends
1464580	1480250	it to a real credit card receipt printer. So interacting with the CMU is all well and good,
1480250	1485210	but we'd really like to have some low-level debugging capabilities. And as it turns out,
1485210	1491830	the main processor board has a Pentium debug port. Now, unfortunately, the Pentium Probe mode
1491830	1500090	is undocumented. Details are in the infamous Appendix H of the P5 external interface specification,
1500090	1506030	which was available only under NDA from Intel. And back in the mid-90s, there were a lot of
1506030	1511830	rants in Dr. Dobbs' journal about this. There was one beta version of the P5 external interface
1511830	1517430	specification that had these details, but all these copies were supposedly recalled by Intel
1517430	1523490	and destroyed. So, of course, there's no copies of this on the web. But while we were searching
1523490	1530430	for it, we did find that there was one physical copy of an unknown version at an undisclosed
1530430	1537810	university lab halfway across the world. So we sent them an email and asked if they had it,
1537810	1544010	and if so, what version it was, and that we might be interested in a scan of it.
1544210	1551730	And so they wrote back with a scan of the title page, which was version 1.3. And unfortunately,
1551730	1557250	we needed version 3. So before we could write them back and say thanks, but no thanks,
1557250	1562670	they wrote us back and said they found something very interesting inside. And sandwiched between
1562670	1571050	the pages of version 1.3 was version 3. And they happened to send us a scan of the entire thing,
1571050	1576270	including details of the Pentium micro-op instruction format, which is exactly what
1576270	1582190	you need to use the Pentium's probe mode. So armed with this information, I added P5
1582190	1588810	debugging support to OpenOCD. And so now we can use GDB against the CMU. Unfortunately,
1588810	1593490	if you halt the hardware, the watchdog timers on the other processors will notice something's
1593490	1599790	amiss and reset the entire system. And the solution for this is to basically run the CMU's
1599790	1608350	x86 firmware in QMU, but relaying messages to the power PCs over JTAG. And this is based on
1608350	1613870	some of my earlier dissertation work on a system called Surrogates, which lets you run firmware
1613870	1620530	in QMU, but proxies peripheral access over JTAG in near real time. So now we can instrument the
1620530	1626870	CMU firmware, collect IO traces, do snapshots, and do all sorts of interesting dynamic analysis.
1627610	1632370	So in summary, the communications management unit supports flight critical operations,
1632370	1639010	which requires the firmware to be certified. But airlines desire customization. And through
1639010	1645430	our reverse engineering, we've identified some somewhat unusual ways that vendors have isolated
1645430	1651930	these uncertified apps from the rest of the certified system. And finally, we built a flexible
1651930	1657910	testbed for further security research on avionic systems. So if you would like more information
1657910	1664590	about our Triton testbed or surrogate system, check out our site at aerosec.org. Thanks!