00:00:00.000-->00:00:04.000 >>Um right, so my talk is Are all BSDs Created Equally? A survey of BSD kernel 00:00:04.000-->00:00:09.000 vulnerabilities. Um so who am I? My name is Ilja van Sprundel um I uh work at IOActive I’m the 00:00:11.083-->00:00:16.292 director of penetration testing, uh I do pen test, code review and I break stuff for fun and 00:00:16.292-->00:00:22.958 profit, basically, and then it’s somebody else’s job to fix it. [laughter] Um so my outline is 00:00:22.958-->00:00:27.500 sort of really easy, I have a one slide introduction that sort of puts something out there and 00:00:27.500-->00:00:32.583 then I tried to collect some data uh which is essentially vulnerabilities over the years 00:00:32.583-->00:00:37.583 in uh BSD or BSD- BSD kernels and then sort of lacking more than what I want um I sort of do 00:00:41.333-->00:00:47.542 this sort of test by audit and then basically um I sort of rung through some common attack 00:00:47.542-->00:00:54.250 surface and some issues I found uh and then I sort of of uh lay out some results and draw a 00:00:54.250-->00:01:00.667 couple conclusions. Um so what’s this talk about? About BSD kernel vulnerabilities, a 00:01:00.667-->00:01:06.458 comparison between different BSD flavors uh the audience I think should be- pretty self 00:01:06.458-->00:01:11.375 explanatory, right? If you’re a low level security guy, or you’re UNIX or BSD geek, or if 00:01:11.375-->00:01:17.000 you’re a Linux guy you might enjoy it too. Um or people generally curious about you know 00:01:17.000-->00:01:22.000 low level OS internals, I think they might enjoy this um as far as knowledge goes um I mean I do 00:01:24.125-->00:01:30.083 expect that you have some basic understanding of the BSD kernel um it doesn’t have to be too 00:01:30.083-->00:01:36.750 deep but just a general concept would be nice. Um yeah before I wanna continue um this is not 00:01:36.750-->00:01:42.875 just- I didn’t just create this out of thin air, um there was work by others that I’ve uh sort 00:01:42.875-->00:01:48.583 of relied upon and sort of- that came before me um and I’m sure I’ve forgot some because you 00:01:48.583-->00:01:53.458 always do uh if you feel like you should be on the list and you’re not I’m- I’m the only guy 00:01:53.458-->00:02:00.125 at fault here, um but most all of these guys had basically had contributions at some level at- 00:02:00.125-->00:02:05.125 in sort of the uh- offensive BSD kernel stuff. So this is my entire intro it’s basically a 00:02:09.333-->00:02:14.333 small rant by Theo De Raadt um from about 12 years ago um where he gave some Forbes magazine 00:02:17.167-->00:02:22.167 review about the Linux guys and how they suck um and he follows up with some uh- some uh- um 00:02:24.292-->00:02:29.500 some posts online and he goes uh ‘well you know if the Linux people actually cared about code 00:02:29.500-->00:02:35.167 Quality as we do, they wouldn’t have had as many you know local kernel holes in the last year’ 00:02:35.167-->00:02:40.917 and he goes like ‘how many has it been, like 20 so far?’ right? That’s the entire slide um and 00:02:40.917-->00:02:46.625 now that thing is 12 years old but somehow that’s been stuck in my head I’ve never gotten it out 00:02:46.625-->00:02:50.583 and I’ve been like ‘well you know is that- is that really true? Like, can you back that 00:02:50.583-->00:02:57.042 up?’ um and so I went to look- looking for some data and sure enough most of these things have 00:02:57.042-->00:03:02.083 CVEs um and there’s a website that collects them all and you basically go there and say, 00:03:02.083-->00:03:05.167 ‘please show me all Linux vulnerabilities’ and it gives you this beautiful grid and 00:03:05.167-->00:03:11.167 says, ‘these are all the bugs per year, types of bugs’ all that type of stuff. Um and so if 00:03:11.167-->00:03:17.750 you look at this thing you- you sort of see from 99 til 2007 um and this- this- this- this 00:03:17.750-->00:03:23.833 grid’s actually slightly older this is from uh about three weeks ago the numbers from two 00:03:23.833-->00:03:28.833 thousand and s- 2017 aren’t quite right, it says 346 there, but it’s really today it’s 353, 00:03:32.417-->00:03:37.208 that’s the number of Linux kernel vulnerabilities this year, right? Um but if you start 00:03:37.208-->00:03:42.958 looking at the- at the- at sort of the bo- the- the top, 99 til about 2003 you know the numbers 00:03:42.958-->00:03:48.250 were pretty mild right it was like oh you know 5 anywhere between 5 and 20ish right? So 00:03:48.250-->00:03:53.542 that- and then- and then sort of as of uh 2004 it kind of sort of- you know it sort of hockey 00:03:53.542-->00:03:57.667 sticked that goes on right? Um so the numbers basically get worse and worse and worse, more 00:03:57.667-->00:04:01.625 and more security vulnerability is discovered and disclosed every year in Linux kernel, 00:04:01.625-->00:04:07.458 essentially, right? So now what do those numbers look like for the BSDs? Well, I went to the 00:04:07.458-->00:04:12.292 CVedetails dot come and said uh you know what were BSD kernel vulnerabilities and sort of 00:04:12.292-->00:04:15.250 [mouth noise] zilch and it says net BSD kernel vulnerabilities nothing, CVe kernel 00:04:15.250-->00:04:20.208 vulnerabilities, zero, well I know that’s not true um [laughs] I had some before, they should 00:04:20.208-->00:04:25.208 have at least shown up um but I do know they publish their own advisories um and I guess to get 00:04:29.542-->00:04:32.000 it so BSD it’s neurota um and it’s it’s essentially the same thing except it’s uh much 00:04:32.000-->00:04:36.958 s**ttier to pa- to get data out of um so I spent like an evening writing some uh web crawl and 00:04:36.958-->00:04:41.333 some s**ty protocol that parses HTML and then like f***k around in Excel to get the numbers 00:04:41.333-->00:04:46.333 lined up. Um but I did that and so you sort of you know build a similar ish table um and you’ll 00:04:48.792-->00:04:55.500 see that the- the numbers from 1999 to about 2003 more or less are in the same ballpark as 00:04:55.500-->00:05:01.833 Linux uh and then Linux after 2003 kind of you know goes like this but then all the BSDs kind 00:05:01.833-->00:05:07.292 of sort of stay the same, I mean it’s- it more or less line up, goes up, goes down a bit, it’s 00:05:07.292-->00:05:12.792 usually anywhere between zero and like 20 or something um and that- that hasn’t changed so if 00:05:12.792-->00:05:18.833 you look at numbers even for this year, they’re more or less in line. Um so looking at these 00:05:18.833-->00:05:23.708 numbers um Theo was right, you know, the Linux guys are doing a lot worse than the BSD guys 00:05:23.708-->00:05:28.708 right? A very astute observation um and when he said 20 in 2005 um that was you know a very 00:05:31.625-->00:05:36.625 conservative view if you look at the table here, it’s 133 not 20 [laughs] so you know he was uh- 00:05:38.708-->00:05:44.708 he was off but uh the- his- his sort of uh the impression he wanted to give seems to sound 00:05:44.708-->00:05:50.083 right um. But then I wonder well, you know if we see this kind of hockey stick um in the 00:05:50.083-->00:05:54.417 Linux numbers why don’t we see it in the BSD numbers, right? Is it really true? Is it- is it- is 00:05:54.417-->00:06:00.542 it code quality, is that what it is? And so I wanna sort of- in my presentation I want to I 00:06:00.542-->00:06:03.000 wanted to basically figure out okay are these numbers equal footing, are the Linux guys 00:06:03.000-->00:06:09.458 doing something different, or is everything else the same, and it’s all about code quality. Um 00:06:09.458-->00:06:13.125 and so when I was tal- thinking about this I said well you know there- there’s you know this uh- 00:06:13.125-->00:06:17.833 this many eyeballs thing right? And and and I know I know cause the many eyeballs thing is 00:06:17.833-->00:06:23.250 mostly like you know this sort of it’s not really true and of course not all bugs are shallow 00:06:23.250-->00:06:27.042 no matter how many people you have looking at code, some bugs are just ridiculously hard to 00:06:27.042-->00:06:33.833 find. Um but the general idea behind the many eyeballs thing I do like and so I’m- I’m thinking 00:06:33.833-->00:06:38.208 is there some truth to it? Um cause surely there are more eyes looking at the Linux kernel code 00:06:38.208-->00:06:43.333 than there are at OpenBSD just by sheer number right? Um and so I wanted to figure out if 00:06:43.333-->00:06:48.333 that’s- if that has- if that has remotely anything to do with this. Okay so um first of all I- 00:06:50.583-->00:06:55.875 I sort of looked around and sort of has anybody else done this before right? Cause the way you 00:06:55.875-->00:06:59.500 would do this is you go and look for vulnerabilities in these things and see what the numbers 00:06:59.500-->00:07:04.500 are. Um and it turns out somebody did that um 15 years ago, Um so- uh uh uh a very very 00:07:06.875-->00:07:11.833 nice Australian guy named um Silvio and he did uh um he did a Black Hat presentation about 00:07:11.833-->00:07:15.792 this um and I and I vaguely remembered this and I looked over the slides and I’m like 00:07:15.792-->00:07:19.375 this- you know you get so nostalgic looking at it again and you’re like all these slides 00:07:19.375-->00:07:22.667 are so f***king awesome um and then you start looking at them and then- and then it sort of 00:07:22.667-->00:07:27.875 get to the last three or four conclusion slides and and um Silvio basically goes well you 00:07:27.875-->00:07:31.500 know there’s really not much of a quality difference between the BSDs and the Linux in terms of 00:07:31.500-->00:07:36.500 security. Um now that was over a decade ago, it was 15 years ago, so have things changed, right? 00:07:38.667-->00:07:43.375 The- the history conclusions are 3 years before oh Theo’s rant so things may have changed in those 00:07:43.375-->00:07:49.917 15 years, right? Um the time spent um in the slides compared between BSD and Linux is like he 00:07:49.917-->00:07:54.833 spent like a week on all the BSDs combined and then like 3 months on Linux right? So the- 00:07:54.833-->00:08:00.667 even- even- like there that’s apples and oranges right? And then the last one sort of the um 00:08:00.667-->00:08:05.958 uh Silvio basically put out this okay I’m going to go look for trivial integer overflows and 00:08:05.958-->00:08:09.958 some info leaks and and so that’s kind of it, right? And so what I wanted to do is I was 00:08:09.958-->00:08:14.625 like I don’t want to limit myself to just those two I wanna be a little bit wider and then 00:08:14.625-->00:08:19.625 see if I can find some other stuff as well and how easy those things are to find. Um so I sort 00:08:22.500-->00:08:28.250 of set- set out and said, okay I’ll do the same thing Silvio did, just 15 years later um so I 00:08:28.250-->00:08:33.250 spend a April, May, June, and a little bit of July um auditing BSD source code. Um and when I 00:08:36.167-->00:08:41.333 set out to do this I was like okay well um where where do I think the bugs are? Where would 00:08:41.333-->00:08:45.792 they be, right? Whats- what’s the interesting attack surface right? And so you come up with 00:08:45.792-->00:08:50.792 this sort of short list of you know where I think they are, right. Number one eh- I mean 00:08:50.792-->00:08:55.458 it’s stuff we’re- a user can talk to it essentially, right? So number one is system calls, 00:08:55.458-->00:08:59.042 right? Users are gonna have to talk to the kernel and they’re gonna have to do it through 00:08:59.042-->00:09:04.125 system calls, right? Uh number two is obviously TCP/IP Stack, right? If you’re network 00:09:04.125-->00:09:07.750 connected [cough] someone's going to be sending you packets at some point so that’s attack 00:09:07.750-->00:09:11.625 surface and so that’s sort of the super common that’s the one that you know my- my mom could 00:09:11.625-->00:09:15.958 come up with those and then sort of a little list of like slightly less common to like you 00:09:15.958-->00:09:21.583 know sort of rare-ish or exotic or something, right? Um and so at the top of the list is sort 00:09:21.583-->00:09:26.750 of the drivers and specifically you know ioctls um cause you know most devices need just 00:09:26.750-->00:09:32.458 someone to talk to it and configure it, and get data out of it Um [coughs] and that’s 00:09:32.458-->00:09:38.542 sort of uh- I started thinking a bit more and I was like oh well these bs- BSDs um most of em you 00:09:38.542-->00:09:43.667 know it comes with these compat layers um that’s probably interesting to go and find- look 00:09:43.667-->00:09:48.875 for bugs um and then um I was doing something else where I had this idea that I’ll- I’ll talk 00:09:48.875-->00:09:53.750 about in a second and based upon that I was like oh you know the uh the trap handlers are 00:09:53.750-->00:09:57.458 probably interesting too and this is where it gets sort of get a little bit more and more 00:09:57.458-->00:10:02.583 esoteric and then I sort of did okay well what about filesystems and it turns out there’s some 00:10:02.583-->00:10:07.042 attack surfaces in filesystems as well and then I’m like okay well, there’s also networking 00:10:07.042-->00:10:12.042 that isn’t um TCP/IP stack like you know wifi, I mean it kind of ties into it but it’s really I 00:10:14.167-->00:10:18.625 mean it’s not, you know you open up TCP to illustrate it it doesn’t talk about you know- uh 00:10:18.625-->00:10:24.250 a wifi frames right? Um and there’s other things like bluetooth and IrDA and I wanted 00:10:24.250-->00:10:28.542 to cover those too, but I ran out of time so I’ll just in that section I’ll only talk about the 00:10:28.542-->00:10:34.500 wifi stack. Um and then I was like, okay let’s just dive right in, and what I’m gonna do is 00:10:34.500-->00:10:38.917 I’ll sort of um have a little slide that sort of introduces the thing and then I’ll show you 00:10:38.917-->00:10:43.667 a little bit of code of a- of a bug and I’ll describe the bug and for 4 or 5 of them I’ll 00:10:43.667-->00:10:48.667 basically run through a little bit of a demo [coughs] So let’s uh [clears throat] start off 00:10:51.792-->00:10:56.792 with syscalls. Obvious attack surface, right? Um it has exactly how the user talks uh to 00:10:59.125-->00:11:05.833 the kernel um without system calls your OS would be essentially useless. Um it says 00:11:05.833-->00:11:10.958 the BSDs have you know evolved since the 70s um they no longer have 4 different system calls uh 00:11:10.958-->00:11:15.292 they- their- they number in the hundreds uh and they do all sorts of funky stuff and 00:11:15.292-->00:11:20.000 performance improvements. Um and these, I mean, th- uh- essentially these are things 00:11:20.000-->00:11:25.250 like you know, open a file, read a file, you know, create a network connection, send- read 00:11:25.250-->00:11:30.292 something from a network connection, close a file, you know, send a message, get my 00:11:30.292-->00:11:35.583 UID, that- that’s the kind of stuff it does, right? Um and so FreeBSD at the top of the list, 00:11:35.583-->00:11:42.375 they have like 550 ish and then um NetBSD is slightly below it with 480 and then at the bottom 00:11:42.375-->00:11:48.333 is like OpenBSD they only have 330 system calls um and functionality wise you might say 00:11:48.333-->00:11:52.958 okay well you wanna have more, wanna know less because that you know have more functionality and 00:11:52.958-->00:11:56.750 while that may be true, from a security perspective, the less system calls the better, right? 00:11:56.750-->00:12:03.458 Um cause no- less system calls means less attack surface, means more security, right? Um so just 00:12:03.458-->00:12:08.583 looking at these numbers you go like oh go OpenBSD, right? Um and then I- one of my 00:12:08.583-->00:12:13.042 assumptions basically that well given that their such obvious attack surface and they’re so 00:12:13.042-->00:12:17.458 well tested and they’re constantly used by apps this is probably you know less likely to 00:12:17.458-->00:12:24.125 contain security bugs, right? [laughs] and so let me show some code, so that’s a system call, 00:12:24.125-->00:12:28.667 this one basically you know drops a sys- a sys log message right? And it takes a buffer and 00:12:28.667-->00:12:32.625 it takes a length and it takes your buffer and your length and it passes on through this 00:12:32.625-->00:12:38.250 function called dosens uh- dosendsyslog and dosendsyslog basically takes the length field 00:12:38.250-->00:12:44.417 and then basically puts it in uh um in on uh uh IO structure and then from there sort of takes 00:12:44.417-->00:12:48.292 out a link- the length field again and then passes that along to uh the kernel malloc 00:12:48.292-->00:12:54.750 function. Um now the kernel malloc function is sort of um historically in BSDs it was very 00:12:54.750-->00:13:00.500 special um the free [inaudible] BSDs have sort of evolved away from that um but OpenBSD still 00:13:00.500-->00:13:07.000 has the original model where um malloc basically goes and says if your length is too big um 00:13:07.000-->00:13:10.750 you’re doing something really horrible and instead of just hanging your system, I’m going 00:13:10.750-->00:13:16.833 to c**p out um and basically panic and then it looks like this, your system essentially 00:13:16.833-->00:13:22.958 dead after you do this. Um and so essentially this is the uh it’s a- it’s an unbound malloc 00:13:22.958-->00:13:28.958 in OpenBSD um that triggers a kernel panic um and it’s- it’s easily triggerable through 00:13:28.958-->00:13:33.958 dosendsyslog system call. And let me see if I can- man I ca- I really can’t see that [mumbles 00:13:38.833-->00:13:43.833 under breath] is that it? What? One more? Uh this guy, there we go okay, so that’s a [laughs] I- 00:13:49.625-->00:13:54.625 I- I seriously can’t see it cause my screen’s different than what’s there uh anyway so that 00:13:57.750-->00:14:04.625 should tell you the U name that is NetBSD uh- OpenBSD sorry, right? Yes, yes! And I- I 00:14:04.625-->00:14:11.000 precompiled the syslog binary um and I will be throwing the physical thread out after my 00:14:11.000-->00:14:16.167 talk uh I just don’t know where yet um but if you- if you wanna figure out where it is um shoot 00:14:16.167-->00:14:21.375 me an email and I’ll tell you. So if we run this thing, boom there you go. RIght there that’s 00:14:21.375-->00:14:26.375 a kernel panic, the box is dead. >>Woo! >>Awesome, thank you! [applause] [laughs] there we go, 00:14:35.917-->00:14:40.917 oh yeah hold on. Yeah that looks about right. Okay so that definitely triggers a kernel 00:14:44.375-->00:14:51.292 panic, right? Okay let’s uh do an- let’s look at a non system call cause we don’t have just 00:14:51.292-->00:14:56.875 wanna look at OpenBSD right? So there’s this system code called kldstat which is um uh in uh uh 00:14:56.875-->00:15:03.750 FreeBSD basically it basically you call system call and you say hey give me a um a list of 00:15:03.750-->00:15:08.083 loaded drivers or for that particular driver, tell me everything you know about it and 00:15:08.083-->00:15:13.792 sort of this- syscall goes back and says, here’s what I got. And the way that works is um it 00:15:13.792-->00:15:17.375 calls this thing called copyin which takes the data from the user and then it calls kern 00:15:17.375-->00:15:21.375 kldstat and it basically collects a bunch of data and puts it in this structure called 00:15:21.375-->00:15:26.417 the stat structure um and I don’t have the function for um kldstat in here cause I didn’t 00:15:26.417-->00:15:30.625 want to spoil- I didn’t want to like pollute um the slide but essentially what this function 00:15:30.625-->00:15:35.625 does is it fills out two strings uh that can be about 1000 bytes long um and it sort of um copies 00:15:38.292-->00:15:43.833 an alternate string into it uh but then everything else after the no byte is never initialized 00:15:43.833-->00:15:49.333 so you basically end up with about a half a page for the info leak stuff just data that’s on 00:15:49.333-->00:15:55.375 the stack never initialized, right? Um [clears throat] this is in uh FreeBSD 11 is when I- 00:15:55.375-->00:16:00.542 when I looked at it went back and saw that this book has been around for ehhh more or less 00:16:00.542-->00:16:05.542 about a decade um so [clears throat] let me see if I can demo that too. [mumbles under breath] 00:16:11.917-->00:16:16.917 okay this works better. Uh I think that one’s right, yes So this basically um triggers the 00:16:24.750-->00:16:29.167 bug over and over and over and what you’re going to see is like a long stream of like random 00:16:29.167-->00:16:34.167 kernel bytes um [indiscernible] boom there you go! All that stuff is that’s all you’re 00:16:39.000-->00:16:43.958 leaking, you’re leaking, you’re leaking. Yeah, there’s some weird bug where sometimes it 00:16:43.958-->00:16:47.792 hangs but it basically just keeps leaking and leaking and leaking, and leaking, that’s 00:16:47.792-->00:16:52.667 endless amounts of [indiscernible] um I was thinking of doing a demo where I 00:16:52.667-->00:16:57.667 grabbed some stuff out of it um but I didn’t um ran out of time. Um. Oh yeah um d**n it. There we 00:17:10.292-->00:17:15.292 go, okay. Oh yeah. Um [clears throat] so now that I’ve seen two of these bugs um and I have 00:17:22.333-->00:17:28.167 several several more uh syscall bugs uh I didn't want to spend all my time just demoing those 00:17:28.167-->00:17:32.667 those things um but essentially you know the previous assumption where we go ‘oh well you know 00:17:32.667-->00:17:35.625 these things are well tested, they’ve been around forever, it’s highly unlikely to get 00:17:35.625-->00:17:40.500 bugs’ well that’s not really true, turns out bugs do occur in system calls with some 00:17:40.500-->00:17:47.292 regularity um and particularly newly added system calls, right? Um that do senses locking has 00:17:47.292-->00:17:51.500 only been around for one version um the KLD one happened to be happens to have been around for 00:17:51.500-->00:17:58.000 a decade um there’s a bunch of others like um I have- I have a bunch of other like FreeBSD ones 00:17:58.000-->00:18:04.500 um like you know that show up um so you know that the assumption is isn’t really true like if you 00:18:04.500-->00:18:11.250 just assume syscalls won’t have bugs well that just isn’t true. Okay so let’s move on to the 00:18:11.250-->00:18:16.583 TCP/IP Stack right, hopefully presumably everybody knows um uh what that is and what that means 00:18:16.583-->00:18:21.708 um it you know it handles networks you know [clears throat] the low level stuff IPV4 00:18:21.708-->00:18:26.292 and 6, Udp and tcp and the sequence numbers and you know, Ipsec and how it all ties 00:18:26.292-->00:18:30.542 together and then how you route it back and forth to userland app and those kind of things. Um 00:18:30.542-->00:18:35.750 this is obviously very well known attack surface [clears throat] it’s been around 00:18:35.750-->00:18:39.792 forever, again the same assumption here is that this is very well tested code, it’s been 00:18:39.792-->00:18:46.292 around since the early 80s uh very unlikely to find security bugs in there. Right So here’s 00:18:46.292-->00:18:51.792 another piece of uh OpenBSD code this basically takes uh P- pppoe vert net packets and sort of 00:18:51.792-->00:18:56.417 passes them all to to this handler and it sort of loops over the the packet data and it 00:18:56.417-->00:19:00.292 looks for a tag in the length when it finds a tag in the length it goes into the switch 00:19:00.292-->00:19:05.542 and the switch goes ‘oh hey, this tag, this tag, this tag’ in this particular case, there’s a 00:19:05.542-->00:19:11.458 um an error [clears throat] some error tag in this case I guess SName error and it basically 00:19:11.458-->00:19:15.333 goes there and says ‘okay let’s me- set the error message to this’ and then send a flag which 00:19:15.333-->00:19:20.042 is error tag and then end of the uh switch case you kind of there’s this code that kind of 00:19:20.042-->00:19:25.292 goes ‘oh hey, if we got this particular error message go and and and go do this’ and then um 00:19:25.292-->00:19:30.292 they obviously pull out the thing out of the packet um before they can actually touch 00:19:30.292-->00:19:36.625 the bytes and this is- this is like a bizarre um uh uh they have these things called Mbufs 00:19:36.625-->00:19:41.667 which are basically very pa- specific buffer structures that are designed to hold network 00:19:41.667-->00:19:45.250 data, right? And there’s all sorts of APIs around them to work with them and so they’re, 00:19:45.250-->00:19:50.417 they’re made in a way so you can chain it together so that if you wanna like append or prepend or 00:19:50.417-->00:19:55.583 like ch- swatch- uh switch stuff out as you’re building a packet that- that they work, right. And 00:19:55.583-->00:20:00.250 as part of that um before you ever touch any of these things um you- you you’re not going to 00:20:00.250-->00:20:04.292 need to have a continuous buffer, right? Um and so there’s a function called Pulldown and 00:20:04.292-->00:20:07.833 basically what what Pulldown does is it says ‘okay what I’m gonna do is I’m gonna look at 00:20:07.833-->00:20:11.708 all of these things chained together, um, and when I successfully when I- when I- 00:20:11.708-->00:20:16.125 when I complete and it’s successful, I’m going to hand you back a you know a buffer 00:20:16.125-->00:20:20.875 that is completely continuous and you can start using that’, right, and so it does that um 00:20:20.875-->00:20:26.083 and if that works it goes on and does what it’s supposed to do. Now if that fails, it bails out 00:20:26.083-->00:20:31.042 um and it basically goes to this thing called Done and then Done basically goes, says, ‘Oh yeah 00:20:31.042-->00:20:36.292 this is your original uh Mbuf let’s go and free that thing’ right? Um and so that code looks 00:20:36.292-->00:20:42.250 fine except if you look at the main page for Pulldown um it says in uh the fine print it 00:20:42.250-->00:20:47.333 goes ‘well if Pulldown fails it frees the Mbuf and then it tells you it failed’ um so what’s 00:20:47.333-->00:20:52.125 happening here basically is if the Pulldown fails, the Mbuf is freed, and then it goes and 00:20:52.125-->00:20:57.125 frees the Mbuf again, right? So s- uh uh sorry. Um so uh uh a pretty trivial case of um you 00:21:01.167-->00:21:07.958 know double free um this isn’t o- I found it in OpenBSD 6 point 1 and it turns out it’s been 00:21:07.958-->00:21:13.042 there for a decade and change um and then um one of the things I did is I tried to cross 00:21:13.042-->00:21:17.292 reference, because they do have a fair amount of shared code, so I tried to cross reference um 00:21:17.292-->00:21:22.250 and I looked at the NetBSD code base and then I saw that in February of this year they 00:21:22.250-->00:21:26.750 discovered this bug independently, and they fixed it, but uh there was never an 00:21:26.750-->00:21:31.750 announcement made, it was silently fixed Um [laughs] [applause] and so again the- the 00:21:36.417-->00:21:40.458 previous assumption that this is well tested code, and it’s highly unlikely there’s a bug in 00:21:40.458-->00:21:45.583 there, well that isn’t really true, it’s- I mean it was well tested code, it’ll still have 00:21:45.583-->00:21:52.542 bugs uh and the other thing I discovered is that um so these mbufs are incredibly complicated 00:21:52.542-->00:21:57.958 and it’s very error prone, right? To- uh to do it right and never make a mistake, it’s kinda 00:21:57.958-->00:22:03.250 hard and one of the things is that um a lot of APIs have sort of handled these things um it 00:22:03.250-->00:22:08.000 depends on their failure cases, so about half of them basically go and say ‘okay if I fail I 00:22:08.000-->00:22:11.750 don’t free this buffer’ and the other half go, ‘if I fail I’m going to free these buffers’ 00:22:11.750-->00:22:16.208 right? And so that creates an inconsistency where if you’re a developer, right and you’re 00:22:16.208-->00:22:21.208 you’re- assume the API uh uh um frees on failure um and so uh you basically say ‘okay I’m not 00:22:23.500-->00:22:27.458 gonna free this thing because it frees on failure’ but if the API really doesn’t free on failure, 00:22:27.458-->00:22:31.375 then you’re leaking memory and the opposite oc- occurs too, right? Where if you go and say 00:22:31.375-->00:22:35.542 ‘okay I’m pretty sure it’s API on the return- on the failure doesn’t free- all the- will free 00:22:35.542-->00:22:41.042 it, you know it’s double free so this- it sort of you know inconsistency among API behavior 00:22:41.042-->00:22:45.292 um uh combined with the complexity of of the mbuf structure sort of creates this 00:22:45.292-->00:22:50.292 um situation where like it’s pretty d***n hard to get things right every time. Right. [clears 00:22:55.083-->00:22:58.875 throat] so let’s move on to drivers right. Ioctls you know everybody's favorite hack 00:22:58.875-->00:23:04.458 surface um you know lots and lots of drivers are out there um B- BSDs they’re no different, 00:23:04.458-->00:23:10.292 they have piles and piles of drivers for all sorts of stuff um and you know eh- obviously 00:23:10.292-->00:23:15.792 BSDs UNIX and in UNIX you know everything’s a file so the way you expose a device to userland 00:23:15.792-->00:23:21.083 is you put it in slash dev right? And you give it some name and obviously these things 00:23:21.083-->00:23:26.833 basically expose file operations to userland right? Like file and open and read and write and 00:23:26.833-->00:23:31.833 ioctl and things like that. And I guess you can mmap some of them too, things like that. Um 00:23:34.042-->00:23:38.000 [clears throat] so you know ioctl obviously is where you have an attack surface okay so 00:23:38.000-->00:23:43.583 uh let’s take a look at an ioctl, right? So um that BSD has this device called crypto um and 00:23:43.583-->00:23:48.583 basically what crypto does is um it takes uh uh uh an ioctl input and I actually- um anyway- it 00:23:51.042-->00:23:55.917 takes uh an ioctl input and it basically goes and says ‘okay well here’s uh here’s some 00:23:55.917-->00:24:00.083 structure that comes from userland and it contains this uh this account field’, and what 00:24:00.083-->00:24:04.625 we're gonna do is we’re going to create an array of number of account fields um and then we’re 00:24:04.625-->00:24:08.708 going to start running data into it, right? And then we’re going to start using it, and then 00:24:08.708-->00:24:12.917 we’re going to start like, manipulating stuff. Um and as as you can see there’s a count time 00:24:12.917-->00:24:18.167 size of the structure um and anybody who knows anything about writing code and security knows 00:24:18.167-->00:24:24.792 that that is a very class integer overflow. Um [clears throat] right that’s integer 00:24:24.792-->00:24:29.292 overflow right there, and that’s your memory corruption block, right there. The- I didn’t- 00:24:29.292-->00:24:32.542 again I didn’t want to put more code on the slide cause it would just pollute things but that 00:24:32.542-->00:24:38.167 crypto dev mkey stuff is basically you know starts uh uh uh writing stuff to to that 00:24:38.167-->00:24:43.167 buffer that was previously allocated. Um so yeah that’s uh uh uh uh classic integer 00:24:45.500-->00:24:50.500 overflow leaks memory corruption um it’s in NetBSD it has been there uh s- about 9 ish years um 00:24:54.792-->00:24:59.792 and then hold on let me see Yeah let’s uh There. So this is basically hopefully fingers 00:25:17.708-->00:25:21.792 crossed gonna trigger the um integer overflow and then cause memory corruption and it’ll 00:25:21.792-->00:25:27.458 basically trap into the debugger. There you go. Boom, it’s on- it’s gone, memory 00:25:27.458-->00:25:32.458 corrupted, rebooted [applause] [clears throat] so here’s another one this is uh uh this 00:25:49.542-->00:25:54.875 is uh um this is a device on FreeBSD called ksyms which basically gives you the ability 00:25:54.875-->00:25:59.292 to expose kernel symbols to userland, right? Um and it basically as I mentioned 00:25:59.292-->00:26:03.333 earlier, most of these drivers have an open callback, and this one’s no different. And when you 00:26:03.333-->00:26:06.833 uh when you call the open system call it goes to this callback and this callback basically sets 00:26:06.833-->00:26:11.292 up a few things so you can later do operations on it, and then it returns successfully. Uh one of 00:26:11.292-->00:26:17.583 the things it does in this case is basically um so it creates um a file descriptor storage 00:26:17.583-->00:26:21.833 specific private data which means there’s kernel private data associated with the final 00:26:21.833-->00:26:25.500 descriptor you’re going to use in userland. Um in this particular case what they do is 00:26:25.500-->00:26:30.958 um they uh uh look at the pointer to your uh um that’s basically contains all the data 00:26:30.958-->00:26:36.250 needed to describe your virtual address space for your- for your process and stores it in that 00:26:36.250-->00:26:41.250 pointer or or yeah inside that uh uh driver specific um uh uh uh storage um and then basically 00:26:43.583-->00:26:47.792 returns successfully. And then um later on there’s another system call that basically gets 00:26:47.792-->00:26:53.292 called if you do an mmap in that particular file descriptor um and as you can see it references 00:26:53.292-->00:26:59.833 um that particular pointer that was previously saved, right that uh sc pmap is saved here and 00:26:59.833-->00:27:04.833 it’s referenced here, right? Okay so the bug here is that uh um uh that pointer using mmap 00:27:08.083-->00:27:12.958 doesn’t necessarily still have to be valid then the one using open right and so the way that 00:27:12.958-->00:27:17.167 works is um- ‘cause usually you say okay well the file descriptor associated with the 00:27:17.167-->00:27:20.875 process and so as long as the process is around the file descriptor will be around and so 00:27:20.875-->00:27:26.208 that virtual address space map will still be around except you know OpenBSD er- er- at least 00:27:26.208-->00:27:31.458 the BSDs in UNIX in general is smarter than that and it gives you the option to basically not 00:27:31.458-->00:27:35.625 just hold on to a file descriptor but instead go to a different process and say, ‘hey 00:27:35.625-->00:27:38.833 you can now have this file descriptor too’ and then once that guy has the file 00:27:38.833-->00:27:43.167 descriptor, you can say to the other process, ‘okay well you go and die’ and so the other 00:27:43.167-->00:27:48.292 process is gone and so all data associated with it is entirely gone and expired and now you 00:27:48.292-->00:27:51.750 have that second process, and the second process is still holding onto the file descriptor 00:27:51.750-->00:27:56.708 and all of a sudden it says ‘oh well now sounds like a good time to call mmap on this thing’ and 00:27:56.708-->00:28:00.333 mmap goes around and says ‘oh look I have this- I have all these pointers to uh this 00:28:00.333-->00:28:05.750 particular process, this virtual memory uh space’ um and it goes and references that, does all 00:28:05.750-->00:28:09.583 sort of things with that, except the original process is gone, it’s dead, it’s entirely 00:28:09.583-->00:28:15.542 expired, um and so this guy um ends up- if you do that, you know it ends up doing really 00:28:15.542-->00:28:20.542 really horrible things. Um and I would love to demo this but um well I have a pock ‘cause it’s a 00:28:23.750-->00:28:28.792 s**tty race to hit uh this one took me a couple days to hit um I don’t have that- I don’t have 00:28:28.792-->00:28:34.292 that long um to show you um I will be releasing code um so if you wanna play a- play around 00:28:34.292-->00:28:38.958 with this uh you can just for now you you’ll have to do with a screenshot of a kernel panic 00:28:41.917-->00:28:46.917 [clears throat] okay so that’s it for this sort of super common attack surface. Uh so the compat 00:28:49.542-->00:28:56.000 code is another one where it’s like okay this is an interesting uh attack surface um and so uh 00:28:56.000-->00:29:01.125 the one there is basically so most BSDs have this thing where they tr- um offer additional 00:29:01.125-->00:29:07.458 binary support uh so for example uh uh FreeBSD NetBSD and up until a couple of years ago 00:29:07.458-->00:29:11.292 OpenBSD had support for Linux and what that means is you can have a Linux compiled binary, 00:29:11.292-->00:29:15.917 you would run it on FreeBSD, and it would just run. Not just ‘cause you have to set up like 00:29:15.917-->00:29:20.792 an environment but by and large the binary would work. Um and it could do other things like uh 00:29:20.792-->00:29:25.250 you know emulate [indiscernible] environment of the 64 bit version of of of of that BSD you 00:29:25.250-->00:29:32.250 have um a- a- and it could do um like old versions of the OS, things like that. Um and the way 00:29:32.250-->00:29:36.083 that works is basically what they do is they emulate a whole bunch of system calls, they fake 00:29:36.083-->00:29:39.458 a- the system call that you expect it to be and then they sort of funnel it to the real 00:29:39.458-->00:29:45.500 system call and then sort of you know parse the stuff back and forth um they uh now s- uh um 00:29:45.500-->00:29:51.333 compat layers are a great idea from uh a user functionality perspective, they are an 00:29:51.333-->00:29:56.958 absolutely terrible idea from a security perspective um and uh Theo De Raadt who’s the you know 00:29:56.958-->00:30:01.750 the leader of of OpenBSD and and you know a pretty go- d***n good developer um you know had some 00:30:01.750-->00:30:05.708 really interesting insight of the compat layers and if anybody would know, it would be him, he 00:30:05.708-->00:30:09.833 goes, ‘well you know, people who rely on this stuff don’t care enough to maintain it and the 00:30:09.833-->00:30:14.375 people who work on the actual kernel uh don’t give a d***n about compat layers ‘cause they 00:30:14.375-->00:30:18.167 don’t use ‘em’ um and he says ‘cultures aren’t aligned in the same direction’ and then he 00:30:18.167-->00:30:22.208 says, ‘compat layers, rot very quickly’ which means you write one today and tomorrow it’s 00:30:22.208-->00:30:27.208 gonna be just useless and just have nothing but security bugs in it. Um so here’s the uh uh um 00:30:29.708-->00:30:36.458 here’s the NetBSD um this is it’s this like svr4 which is like fairly ancient 00:30:36.458-->00:30:41.000 [indescribable] thing um and they basically emulate that on NetBSD so if you have original 00:30:41.000-->00:30:46.000 binary compiled for that and you move to binary NetBSD it’ll still work and it has this thing 00:30:46.000-->00:30:51.958 called streams which is sort of a predecessor to sockets ish um and so what they do is, NetBSD 00:30:51.958-->00:30:57.792 doesn’t support stream so what they do is they fake it and they sort of put a socket underneath. 00:30:57.792-->00:31:01.708 Um and so the- the original svr4 binaries will think they have a stream even though it’s a socket 00:31:01.708-->00:31:07.250 um and so this thing basically it’s it’s essentially uh um it’s essentially an an ioctl that’s 00:31:07.250-->00:31:11.667 supposed to do a socket operation um and it copies in the data structure and they put 00:31:11.667-->00:31:17.083 it in the BSD structure, and it passed on to this netaddr the sockaddr in and netaddr sockaddr 00:31:17.083-->00:31:21.375 in basically looks at this uh structure and it says ‘oh this thing contains uh uh you know a 00:31:21.375-->00:31:25.500 32 bit offset inside it’s structure I’m gonna use this offset and I’m gonna add it to 00:31:25.500-->00:31:30.042 my pointer and I’m gonna give you back a pointer to where you can start reading from’ uh 00:31:30.042-->00:31:35.042 problem there being is there’s absolutely- uh no validation being performed um and so 00:31:37.708-->00:31:43.250 basically you can end up with a wild pointer that points anywhere um which you can either 00:31:43.250-->00:31:48.042 use to trigger uh kernel panic you know your box goes down or you can if- if you know what 00:31:48.042-->00:31:51.458 you’re doing you can very selectively pick pieces of memory and start leeking you 00:31:51.458-->00:31:56.625 know very selective pieces of of kernel memory. Um and then what’s really beautiful is that 00:31:56.625-->00:31:59.792 at the top of this file someone said ‘well we’re going to pretend to have streams, we 00:31:59.792-->00:32:05.833 don’t really, oh yeah by the way this code is really gross’ [laughs] Um so yeah this was 00:32:05.833-->00:32:12.042 found in uh uh NetBSD 7 point 1 but this things been around for uh uh 21 years uh it’s been 00:32:12.042-->00:32:17.417 there since NetBSD 1 point 2, it’s been around for a very very long time um and it- it gives 00:32:17.417-->00:32:23.292 you the ability to read arbitrary ish kernel memory or cause kernel panic. And so I 00:32:23.292-->00:32:27.375 submitted this bug to the uh NetBSD guys uh and they go ‘oh s**t yeah this thing- this thing 00:32:27.375-->00:32:32.625 is tot- total piece of s***t’ and they go uh ‘we fix- we fixed a gazillion bugs in this svr4 00:32:32.625-->00:32:39.042 thing uh we should have never enabled this by default it is an absolute minefield’ um and so 00:32:39.042-->00:32:42.958 when I submitted the bugs and they fixed them they also changed uh their OS config file 00:32:42.958-->00:32:48.292 to be like- and and basically turn off svr4 support um if you want it now you’re gonna have to 00:32:48.292-->00:32:53.292 recompile your kernel. [laughs] [applause] Okay so another thing I was thinking, I was like okay, 00:33:01.250-->00:33:05.333 well, okay we have these things called trap handlers and um this is where things get a little bit 00:33:05.333-->00:33:11.417 exotic um so trap handlers were these sort of callback that you reg- that like very low level OS 00:33:11.417-->00:33:16.500 code registers sort of when it boots up and it’s goes eh- basically goes to the os- or the 00:33:16.500-->00:33:21.167 OS basically sets things up and so when some kind of a hardware thing happens these things get 00:33:21.167-->00:33:24.958 called, right? Some kind of exceptional fault happens and these are things like division 00:33:24.958-->00:33:29.750 by zero, a system call, a breakpoint, invalid memory access, um when it- when the 00:33:29.750-->00:33:33.042 hardware sees any of these things it sort of goes ‘oh I don’t know what this is but I’m 00:33:33.042-->00:33:36.833 gonna call this function pointer and we’ll have that deal with it’, right? Um some of them can 00:33:36.833-->00:33:41.250 be triggered by userland and some of them can be triggered by kernel um and the kernel has to 00:33:41.250-->00:33:45.042 trigger them correctly um the thing though with trap handlers is that they are unbelievably 00:33:45.042-->00:33:50.250 nasty pieces of code um and exceptionally architecture specific. Every architecture and 00:33:50.250-->00:33:54.875 like every little miniscule like little dif will have between architectures things are very 00:33:54.875-->00:33:58.500 different and then every little dif within the architecture might still have like tiny 00:33:58.500-->00:34:04.875 little changes between them um it’s an absolute pain to try and understand these things um and 00:34:04.875-->00:34:09.917 it is- it’s a ridiculous pain to look at this code so I didn’t look at it. Um so I said okay 00:34:09.917-->00:34:14.167 but I’d still like to play with this a little bit, where do I go from here? And I was like ‘Okay 00:34:14.167-->00:34:20.917 well what can I do to trigger um uh uh eh- exceptions an- and faults’ well what if I just 00:34:20.917-->00:34:26.250 start executing random instructions, what happens? Um surely they’ll trigger a bunch 00:34:26.250-->00:34:31.750 of uh uh uh uh uh uh uh e- exceptions and traps and then the OS has to handle it and then 00:34:31.750-->00:34:38.125 I have a slightly more um bigger fuzzer, this is essentially it this is uh sort of a one slide 00:34:38.125-->00:34:44.125 um sort of uh uh trap handler fuzzer where essentially what you do is you map a piece of 00:34:44.125-->00:34:48.167 code read and write in executable you sort of start a loop and every time you get into 00:34:48.167-->00:34:53.333 loop you basically you know read out op dev urandom- uh a page of random garbage, you fork off a 00:34:53.333-->00:34:57.333 process, you hand that phd process and the process basically goes, function pointer 00:34:57.333-->00:35:02.125 call’ and then surely crashes and it does this over and over and over and over and over again 00:35:02.125-->00:35:08.833 um [laughs] and essentially I did this on the previous Z and you know there’s a uh a Xen trap 00:35:08.833-->00:35:13.833 that kinda blows up my face and then [applause] [laughs] and there’s some other signal stuff 00:35:17.292-->00:35:21.500 that uh um sorta blows up and I have a demo of this, I’m not sure if it’s gonna work ‘cause 00:35:21.500-->00:35:27.667 this thing sometimes blows up fast and sometimes it takes awhile but I could at least um I 00:35:27.667-->00:35:32.667 could at least try and and show this thing. Uh Oh man this is so horrible ehh. Tu- uh god 00:36:13.833-->00:36:20.542 d***n it! [laughter] I got em! Okay there we go, awesome! [applause] Wow look at that, 00:36:20.542-->00:36:25.542 this is I- this is a horrible set up I got here [chuckles] And plus my eyesight sucks um so 00:36:29.333-->00:36:34.333 this page functions for the [indiscernible] processes I don’t know if it’ll uh I may 00:36:38.750-->00:36:45.083 have to come back to it later ‘cause um give it a few minutes it might die um >>So these are 00:36:45.083-->00:36:50.583 all crashes of that second thread but like how many can- >>Yeah this has- doesn’t crash 00:36:50.583-->00:36:56.167 the kernel yet but eventually it’ll- it’ll crash the kernel um if we’re lucky, fingers crossed 00:36:56.167-->00:37:01.750 um I got a hit earlier this morning and- and usually it only takes a couple minutes but it’s 00:37:01.750-->00:37:07.208 like random garbage so it’s like >>And how are you recording which [inaudible] [speaker 00:37:07.208-->00:37:10.875 inhales] >>Uh so not in this version but I have a version that basically uh has utp socket 00:37:10.875-->00:37:14.917 and sends it to a server and so the server basically goes and says ‘okay this is- this is the 00:37:14.917-->00:37:18.083 garbage it generated- this garbage it generated’ and so when it crashes I look at the 00:37:18.083-->00:37:22.708 last packet and I go like ‘okay let’s decode what this- set of it- set of bites are like you 00:37:22.708-->00:37:29.333 can feed it to like Ida or something and it’ll tell you, this is what it means. Um anyway 00:37:29.333-->00:37:34.333 I’ll- I’ll get back to this later, I hope it dies [laughs] Uh um but essentially so yeah 00:37:41.458-->00:37:45.542 there’s uh uh uh uh uh uh a Xen trap I hit and then there was another sendsignal trap I hit 00:37:45.542-->00:37:50.542 which is this one um and so hopefully I ca- I can show this uh uh in a little while um 00:37:53.958-->00:37:57.542 [clears throat] so let’s mo- move on to file systems um so obviously when you think file 00:37:57.542-->00:38:01.458 systems you say okay well you have a file system image and you can mount it and obviously if 00:38:01.458-->00:38:06.333 you or or you know have USB stick and plug it in and have that mounted and if somebody 00:38:06.333-->00:38:11.292 doesn’t parse the file system correctly then you know bugs show up right? That is number 00:38:11.292-->00:38:15.708 one that is an easy attack surface. [sniffles] Um there’s another one that I sort of- I 00:38:15.708-->00:38:19.875 was playing with and I was like, okay that could be interesting, so in the last couple years all 00:38:19.875-->00:38:26.000 three of the BSDs have built in support for fuse, um and I don’t know if you guys know what fuse 00:38:26.000-->00:38:30.167 is but fuse essentially just a mechanism that allows you to have userland file systems, and 00:38:30.167-->00:38:34.583 the idea being there is that you know, writing file system sin kernel is really, really hard, 00:38:34.583-->00:38:38.792 um and so if you ofed userland it’s okay if your file system does something bad or crashes, 00:38:38.792-->00:38:43.792 the OS won’t go down. Um and so what that requires is, um, your normal BSD uh VFS layer now has 00:38:47.750-->00:38:54.000 to support a file system that’s in user as opposed to one in kernel um if you’re talking, if 00:38:54.000-->00:38:58.250 kernel- the kernel is talking everyone’s trusted, if kernel’s user’s talking, they’re not 00:38:58.250-->00:39:01.250 trusted. So you’re now taking this layer that is always talking to someone that it 00:39:01.250-->00:39:07.250 trusted and you’re basically now feeding it a bunch of data from userland that’s no longer 00:39:07.250-->00:39:13.500 trusted um so the VFS layer- layer- layers in theory should have been hardened um for fuse. 00:39:13.500-->00:39:19.792 Um so I looked at fuse for all three BSDs and basically sort of my impression was the NetBSD- 00:39:19.792-->00:39:24.125 fuse-they’re- they’re- this is funny right, so all the B- they’re family right? They all 00:39:24.125-->00:39:27.583 fork from each other except when fuse came around they all said ‘nope we’re all gonna sort of 00:39:27.583-->00:39:32.292 make our own little thing’ and so you have three distinct uh fuse implementations for the 00:39:32.292-->00:39:38.292 three different uh BSDs um and so NetBSD sort of um in my view made the most complete 00:39:38.292-->00:39:43.167 implementation which allows for- for the most options and flexibility. Um FreeBSD I think 00:39:43.167-->00:39:47.542 has the most controlled one in terms of uh the amount of stuff they’ll take from userland in 00:39:47.542-->00:39:53.875 terms of file system is highly controlled um and so what that means is there’s relatively 00:39:53.875-->00:39:58.958 little opportunity uh for consumers to make mistakes but there’s more opportunity for the 00:39:58.958-->00:40:03.958 parsing inside the fuse code itself to make mistakes. Um and then um OpenBSD sort of um uh 00:40:07.000-->00:40:12.542 has sort of just minimal implementation I mean they all uh um none of em on ioctl but 00:40:12.542-->00:40:16.917 they implement all of the other uh file operations. Um D***n I’m gonna have to skip through this 00:40:16.917-->00:40:21.125 ‘cause I’ve got only a few more minutes left I think and there’s conclusions I want to get to um 00:40:21.125-->00:40:26.125 but anyway this is the getcwd system call um for um uh for the BSDs [clears throat] uh for 00:40:29.500-->00:40:36.250 OpenBSD and that- you can hit that from- from fuse um anyway let’s- let- let’s skip over the 00:40:36.250-->00:40:42.667 file system stuff. Yeah the network is interesting you know there’s some bugs in the 00:40:42.667-->00:40:46.625 networks- in the network stack where if you send a packet the box goes down. Um [mouse 00:40:46.625-->00:40:52.750 clicking] blah blah blah oh and this is one of these things where there’s all- all these uh 00:40:52.750-->00:40:56.833 wireless drivers that have like these straight straight out heap smashes where they’re not 00:40:56.833-->00:41:01.833 expecting a malicious radio even though they’re taking data over over USB and so if you have 00:41:01.833-->00:41:05.875 malicio- k- k- a malicious radio all of a sudden the OS has all sorts of heap overflows. But 00:41:05.875-->00:41:11.417 let’s skip through that through right there’s some miscellaneous stuff I want to talk about but I 00:41:11.417-->00:41:16.417 don’t have time [laughs] um what oh yeah um so yeah results. Um I looked at this stuff for about 00:41:21.667-->00:41:26.667 uh 3 months um uh I found about 30 or so bugs in FreeBSD, uh 25 or so in OpenBSD, 60 ish in 00:41:34.667-->00:41:39.000 NetBSD, and it’s- thing is it’s hard to sa- ‘cause I sa-, I give approximate numbers because some 00:41:39.000-->00:41:43.042 of these things are like, it’s a single function and it contains 5 buffer overflows, okay is that 00:41:43.042-->00:41:48.042 1 bug or 5, right? It’s- so I try to sort of be like ‘err it’s this many’ um depends how you 00:41:50.500-->00:41:54.875 make- or how you sort of look at these things but I think these numbers more or less make sense. 00:41:54.875-->00:42:01.125 Um in terms of bugs I’ve seen it’s like a very long list of, I mean there’s the straight heap 00:42:01.125-->00:42:05.042 and and stack smash like in those wireless drivers uh a couple of race conditions, 00:42:05.042-->00:42:09.750 expired pointers, double frees, recursion bugs [chuckles] integer issues, um very very 00:42:09.750-->00:42:14.792 interesting refcount issues um obviously overflows but also there was one uh NetBSD bug 00:42:14.792-->00:42:19.500 where they’re like ‘okay we’ll take a refcount on this v-note’ and then ‘okay we’re gonna take 00:42:19.500-->00:42:21.958 the mount point out and then we’re gonna drop refcount and now we’re gonna play with the 00:42:21.958-->00:42:25.625 mount point’ well there’s nothing tying them together so the mount point goes away, 00:42:25.625-->00:42:29.500 you're- your’re meaning well, pointers. Anyway, other things that I found were info leaks, 00:42:29.500-->00:42:34.583 out of bound reads, NULL derefs, logic bugs, um a typo where they literally called their own 00:42:34.583-->00:42:41.417 instructor, it was something something TX and it called something something RX um 00:42:41.417-->00:42:48.167 division by zero bugs, uh kernel panics and memory leaks. So um [clears throat] found bugs in 00:42:48.167-->00:42:54.625 all 3 um and basically you know, in the attack surfaces I discovered uh I mentioned 00:42:54.625-->00:43:01.000 earlier uh and sort of among the range of the bugs that I- that I just listed. Um winner loser is 00:43:01.000-->00:43:05.125 is in my impression fairly easy, when I started looking at this stuff OpenBSD came out as a 00:43:05.125-->00:43:11.583 clear winner um for a number of reasons, um they have massively, massively uh reduced their 00:43:11.583-->00:43:17.625 attack surface, um they got rid of loadable module support, they uh um have relatively few 00:43:17.625-->00:43:23.500 devices, um [chuckles] got rid of all of their compat code, including Linux, [chuckles] they 00:43:23.500-->00:43:27.958 deleted their bluetooth stack ‘cause it sucks [chuckles] um they have- more- they have more 00:43:27.958-->00:43:32.458 than 200 system- 200 less system calls than FreeBSD um and they cut support for a bunch of 00:43:32.458-->00:43:37.375 really s***tty architectures. Um on top of that um when Theo said they care about code quality, 00:43:37.375-->00:43:42.833 they do. Um they have virtually no integer overflows or signedness bugs they’re as good 00:43:42.833-->00:43:46.667 as gone in most places, except the ones where they didn’t know about the attack surface, like 00:43:46.667-->00:43:52.458 the wireless drivers. Um oh- yeah- um and then also very few info leaks, so the bugs they 00:43:52.458-->00:43:58.375 know about and the patterns that they know um they look for them you know like a madman and- and- 00:43:58.375-->00:44:04.208 they eradicate them. Um NetBSDs one of these where it’s like okay, comparatively they’re the 00:44:04.208-->00:44:08.042 clear loser, um tons of lazy code and compat code so that’s- you got an attack surface right 00:44:08.042-->00:44:14.542 there um [clears throat] they seem to be less consistent um when it comes to uh um code 00:44:14.542-->00:44:19.500 quality and too many signedness bugs. And I know how this sounds but- and I’m not trying to dis 00:44:19.500-->00:44:24.792 them because I know it’s incredibly hard to write an OS, right? If you think it’s easy to 00:44:24.792-->00:44:29.500 build, maintain, improve an operating system, you should try it, it’s really hard, like you 00:44:29.500-->00:44:34.500 will cry yourself to sleep. And then somewhere in the middle is FreeBSD um now when I started 00:44:36.875-->00:44:40.542 reporting these bugs um sort of- here’s the response that I got, right? When I emailed the 00:44:40.542-->00:44:46.042 OpenBSD guys uh it took about a week or so to get an email back from Theo and fir- he fir- he’s 00:44:46.042-->00:44:50.500 like ‘oh you know I’m so sorry I took a week, I was out on vacation’ and then within- you 00:44:50.500-->00:44:54.792 know after that week like bugs started rolling out, bug fixes started rolling out, you know uh 00:44:54.792-->00:44:59.583 the next couple of days, everything got fixed within a 2 week period. Uh FreeBSD guys 00:44:59.583-->00:45:06.208 responded as well um in uh uh about a week or so and then filed all the bugs, I don’t know 00:45:06.208-->00:45:11.500 the exact status of the bugs. NetBSD and this is where they really shine though, so I was 00:45:11.500-->00:45:13.500 kind of a lazy b*****d um and I was sort of uh uh [chuckles] uh procrastinating and I was like 00:45:13.500-->00:45:15.500 ‘oh I don't want to write up all these 60 bugs’ so I did only got them wrote up right before I got 00:45:15.500-->00:45:20.500 to Vegas and so then I sent them these bugs a couple days ago [chuckles] which is really 00:45:26.583-->00:45:30.250 unfair of me and I was thinking like ‘oh how am I going to deal with this, I don’t want to like 00:45:30.250-->00:45:35.250 make them look bad’ um and then something happened that was like ridiculous um they fixed all 60 00:45:37.250-->00:45:42.250 bugs overnight. [laughter] [applause] Which is unbelievably impressive, and I told them 00:45:49.750-->00:45:53.708 that, and I told them like if you wanna call me out on being an a*****e you can, but I am in 00:45:53.708-->00:45:59.583 awe of you guys ‘cause that is insane. Um additionally they also turned off SVR4 which I 00:45:59.583-->00:46:02.750 think is a great deal and when they emailed me back they said ‘we have disabled this thing, 00:46:02.750-->00:46:09.250 something we should have done a very very long time ago’ Um so yeah pretty much and and my 00:46:09.250-->00:46:14.500 insights, more conclusions. Bugs are still easy to find. Uh very- various levels of quality. Um 00:46:14.500-->00:46:21.125 most consistent uh was OpenBSD um yeah one more point I wanna make is uh maintainers of these 00:46:21.125-->00:46:25.208 BSDs should talk to each other ‘cause I had several bugs where one guy had the bug, and then 00:46:25.208-->00:46:29.667 the other guy fixed it like 6 months ago, right? And it goes both ways, right, it’s OpenBSD 00:46:29.667-->00:46:34.625 and NetBSD and it’s FreeBSD um and uh I got another funny story where- funny story about that 00:46:34.625-->00:46:38.250 about a guy I met at a bar yesterday, [inaudible] he’s a developer, but I don’t have time 00:46:38.250-->00:46:43.583 to get into it. Um yeah code base is is OpenBSD is small, FreeBSD’s the biggest and so 00:46:43.583-->00:46:47.167 it’s interesting to look at these things uh it’s clear OpenBSD has the least amount of 00:46:47.167-->00:46:51.292 attack surfaces, FreeBSD likely has the biggest one. Um this obviously plays a part you know 00:46:51.292-->00:46:55.625 if you- if you don’t have code for something then you can’t have bugs in it, right? [laughs] 00:46:55.625-->00:46:59.125 and there’s a thing between accidental versus planned, right? Uh some things didn’t get 00:46:59.125-->00:47:02.833 ac- get- get implemented, just didn’t get to it and other things you made a conscious 00:47:02.833-->00:47:07.833 choice to sort of get rid of it. Um so yeah the many eyeballs thing, I think it’s a factor. Um 00:47:11.042-->00:47:14.583 based on my results I I think it matters, I think code quality alone doesn’t account for 00:47:14.583-->00:47:19.708 discrepancy, I think code quality has something to do- uh I think the the many eyeballs 00:47:19.708-->00:47:24.417 thing has something to do with it uh my my view is that you know say what you will about the 00:47:24.417-->00:47:28.792 guys looking at the Linux kernel, and you could say a lot about them, um there are just 00:47:28.792-->00:47:34.875 simply so many more of them and it shows in the numbers. Um yeah and then this thing leak- this 00:47:34.875-->00:47:40.500 thing sort of presentation got out before I did this thing, and so the internet saw this and 00:47:40.500-->00:47:47.292 they you know had some questions and and and I don’t I’m pretty much out of time but I wanna 00:47:47.292-->00:47:50.792 ‘cause somebody asked about uh um DragonFly BSD and I did not look at DragonFly BSD, I 00:47:50.792-->00:47:57.417 considered it uh maybe I should've, I just didn’t, I ran out of time. Um and then some 00:47:57.417-->00:48:02.417 links, and then that’s pretty much it, I think i’m out of time right? [applause] Awesome.