00:00:00.042-->00:00:05.958 >>So as he said, I'm Sarah Zatko and, uh, I'm going to be talking about just the interesting 00:00:05.958-->00:00:10.750 tidbits from our last year's worth of research. And then introducing everyone to the work 00:00:10.750-->00:00:15.625 we've been doing with Consumer Reports and talking about how you can help on that or how you 00:00:15.625-->00:00:20.625 can get involved. So for those who don't know what is CITL, it's a nonprofit research 00:00:23.833-->00:00:29.500 organization focused on being sort of like Consumer Reports but focused specifically on 00:00:29.500-->00:00:34.750 software risks and safety. So some people get confused. They think we're just evaluating 00:00:34.750-->00:00:40.917 security software. We're evaluating all software. Um. And that includes desktop, binaries, 00:00:40.917-->00:00:45.167 or things off of IoT devices that's sort of what we're getting into now. There will be 00:00:45.167-->00:00:50.167 a little bit about that later. But, uh, so well anything that's a binary. And, uh, the goal is 00:00:53.375-->00:00:58.458 that we test things and then publish results just like Consumer Reports does. Right now 00:00:58.458-->00:01:05.125 where we are in that process is we've got the testing part pretty solid, but, uh, we're 00:01:05.125-->00:01:10.583 working now on coming up with nice looking, readable reports. Uh. Scaling everything up so 00:01:10.583-->00:01:16.708 that we test everything rather than, you know, just a, you know, a product here or there 00:01:16.708-->00:01:23.125 when you want to do, uh, entire industry like this, uh, requires more scale than what you 00:01:23.125-->00:01:29.333 initially set up for an approved concept. And then, uh, we're building up our partnerships for 00:01:29.333-->00:01:36.000 publishing 'cause, again, we're not publishers. We'd rather give data to people like Consumer 00:01:36.000-->00:01:41.667 Reports and other organizations and let them disseminate the information to their audiences. 00:01:41.667-->00:01:46.667 So that's the sort of stuff we're working on right now. And the end goal is, of course, to 00:01:49.000-->00:01:55.708 make the software industry safer and easier for people to navigate. You know. The people 00:01:55.708-->00:02:00.542 who care to make the right decisions about minimizing their risk have no clue how to do it 00:02:00.542-->00:02:06.833 'cause, not any fault of theirs, but just an entire lack of usable information. So that's 00:02:06.833-->00:02:11.833 the, uh, problem we're looking to solve in the long term. The, uh- our software testing right 00:02:14.333-->00:02:20.958 now... we do static analysis which is different than most people's. Our static analysis is 00:02:20.958-->00:02:26.125 that we look at what application armoring features are there and how well are they implemented. 00:02:26.125-->00:02:31.208 What functions were called and which of those are historically really risky or just straight 00:02:31.208-->00:02:36.917 up, bad. And, uh, um, what libraries are people linking to and what are the features of 00:02:36.917-->00:02:42.917 those libraries. And then, you know, complexity of the code. And we can measure all that to 00:02:42.917-->00:02:49.458 our satisfaction, but there's a problem when you want to take the hundred or two hundred 00:02:49.458-->00:02:54.417 features you extracted about a binary and you want to turn it into a single score. Like this 00:02:54.417-->00:02:59.417 got a 75 out of 100. Because it's hard to tell what thing is worth how many points. You know, 00:03:01.625-->00:03:06.625 are, like, stack guards worth 15 points or 10 or 20? I don't know. Because there haven't been 00:03:09.625-->00:03:15.542 a lot of studies that look at how effective are the safety measures that we use and trust. 00:03:15.542-->00:03:20.875 The, um... In most other industries, that sort of data would be pretty fundamental. 00:03:20.875-->00:03:26.167 Something you could take for granted that it existed because the first step is that somebody 00:03:26.167-->00:03:31.583 comes up with their bright, new idea and publishes how and why it works and we hope, one day, 00:03:31.583-->00:03:36.750 everybody will be using it. But before you get to that universal adoption, there's clinical 00:03:36.750-->00:03:41.000 trials or there's small scale deployment and then they're studies to see did it do what we 00:03:41.000-->00:03:46.708 thought it would do and how well did it do that thing. The security industry doesn't do 00:03:46.708-->00:03:51.833 that. I hadn't realized what a weird blind spot that was until I needed that data for this 00:03:51.833-->00:03:58.167 effort and then I was, like, "Wait! We don't do this at all?" That's, I mean, it's just sort 00:03:58.167-->00:04:02.792 of mind-blowing when you realize that this is an entire gap. But it makes sense 'cause that 00:04:02.792-->00:04:08.417 stuff's not as sexy. It's not as exciting. So people- and there's enough other problems that 00:04:08.417-->00:04:15.167 people don't do it. So that's what we're focusing on in the immediate future, is that we 00:04:15.167-->00:04:20.250 have a lot of static analysis data. We're putting together a lot of dynamic analysis and 00:04:20.250-->00:04:25.583 fuzzing data to go with it so that we can do those studies about how impactful are 00:04:25.583-->00:04:31.500 different elements we're looking for and how much do they affect a final score. Uh, that way when 00:04:31.500-->00:04:36.167 we do finally publish results, they'll be numbers we're ready to get in a fight over. You 00:04:36.167-->00:04:42.083 know. Things that are really solid and that we really stand by. And this'll... when we 00:04:42.083-->00:04:48.292 publish that corpus of data, the fuzzing data and the static analysis data, that'll be a big 00:04:48.292-->00:04:53.500 deal for a lot of other people too, because we're not the only ones who are frustrated by that 00:04:53.500-->00:04:59.250 lack of quantification for impact for the things that we hope to be industry standards. 00:04:59.250-->00:05:06.125 You know, if somebody is looking to push for universal adoption of a particular safety feature, 00:05:06.125-->00:05:10.583 say, you know, the FTC or someone like that, it's very hard to make the argument 00:05:10.583-->00:05:16.500 without the studies saying, "This is why you need this." And, uh, you know, it's also... 00:05:16.500-->00:05:22.167 I believe that this gap in the body of existing research is at least partially responsible for 00:05:22.167-->00:05:29.042 the success of snake oil salesmen in our industry because for the stuff that really has 00:05:29.042-->00:05:34.417 suts-substance and does something, and the snake oil, the same argument is being made. 00:05:34.417-->00:05:40.375 I'm an expert and this works. Trust me. You know, the non-expert doesn't have the sort 00:05:40.375-->00:05:44.458 of data that they'd have for any other sort of decision making process. And it looks the same 00:05:44.458-->00:05:48.917 to them. So whichever one better marketed and prettier-looking is the one they're going to go for. 00:05:48.917-->00:05:55.167 Also snake oil people usually make more broad promises. So, you know, they'll go for the one 00:05:55.167-->00:05:58.917 that says, "This will make you totally secure." Not the one where they're like, "Yeah! I 00:05:58.917-->00:06:05.458 think this'll help." You know. So I'm very excited about the research we're doing right now 00:06:05.458-->00:06:10.500 to build up that corpus of data and that's what we're hoping to have as our next big thing, is 00:06:10.500-->00:06:15.500 releasing that. But for now, on to our talk and what do we have, uh, to show you today. So, uh, 00:06:18.833-->00:06:23.833 as I said, we're ramping up our fuzzing capabilities so that we can build up the second half of 00:06:23.833-->00:06:29.417 the data we need for our big longitudinal studies. And, uh, so we're going to look at the 00:06:29.417-->00:06:33.667 really early stuff coming out of there and talk a little bit about the fuzzing framework that 00:06:33.667-->00:06:38.500 we're building up. And then we're going to get into the interesting tidbits from our 00:06:38.500-->00:06:44.292 static analysis so far. So we're going to look at comparisons of major OS's, a couple Smart TV's, 00:06:44.292-->00:06:49.958 a couple Amazon AMI's. And then for applications, we're going to look at browsers on all those 00:06:49.958-->00:06:56.083 major OS's. And then, uh, revisit Microsoft Office for OSX, 'cause, uh, there's an 00:06:56.083-->00:07:01.375 interesting case study there. And, um, finally we're going to talk about our work with 00:07:01.375-->00:07:07.500 Consumer Reports on the digital standard which, again, I'm very excited about. It's nice to have 00:07:07.500-->00:07:12.500 so much work to do that you're really, uh, feel like it's going to make a difference sometime. 00:07:15.375-->00:07:22.375 You know. Uh. Ok. So fuzzing. We've fuzzed about 300 binaries so far which might sound like a 00:07:22.375-->00:07:28.292 lot in some context, but, uh, given that we've done static analysis on over 100-->000, it's 00:07:28.292-->00:07:34.375 not where it needs to be yet. Um. And what we're building is a system that can do fuzzing in a 00:07:34.375-->00:07:39.375 fully automated fashion. So set up a VM, install the software to be tested, take a bunch of 00:07:42.042-->00:07:47.333 existing test harnesses and try them out on the binaries. If something fits, then you fuzz it 00:07:47.333-->00:07:53.417 and then do the crash triage to identify primitives all in an automated fashion. When most 00:07:53.417-->00:07:59.792 people are doing fuzzing, their goal is to hit a particular target. But what we want is just 00:07:59.792-->00:08:04.333 breath- we want as many different kinds of binaries as possible. So if something 00:08:04.333-->00:08:08.667 doesn't work, we ditch it and just move on to the next thing. And, you know, so that means we 00:08:08.667-->00:08:15.125 can build up a larger set of data, although probably a little idiosyn-syncratic looking to 00:08:15.125-->00:08:21.750 somebody who had a different goal in mind. And, uh, to do this, we're customizing AFL and 00:08:21.750-->00:08:26.750 Triforce QMU and also borrowing some stuff out of the cyber grand challenge. And, uh, that 00:08:28.917-->00:08:33.500 put together is called CITL-Fuzz, very creatively. Maybe we'll have a better name 00:08:33.500-->00:08:39.333 for it somewhere down the road. Uh. Names aren't totally my specialty if you saw the talk 00:08:39.333-->00:08:44.333 title. But, uh. So we've fuzzed about 200 binaries with AFL and about 100 with CITL-Fuzz so far. 00:08:50.958-->00:08:57.667 So for the CITL-Fuzz results, those are- it's not a terribly rich data set at the moment, but 00:08:57.667-->00:09:03.667 one interesting thing you d- can do is look at the packages where we fuzz multiple binaries and 00:09:03.667-->00:09:08.667 see how well did those packages do as a whole. So, um, what we have here is package name, 00:09:11.333-->00:09:17.042 number of binaries, how many of those crashed that as a percent. And then total number of unique 00:09:17.042-->00:09:22.042 crashes for that package. And, uh, you can see the winners here are Libc-bin and Cook with five 00:09:25.875-->00:09:31.875 and four binaries respectively and zero crashes. Those were the biggest sets of binaries to not 00:09:31.875-->00:09:38.417 have any crashes found. Uh, the runner up would be Canna which had a handful of crashes for its 00:09:38.417-->00:09:44.583 six binaries. That's still better than anything else in its range of, uh, binary counts. The 00:09:44.583-->00:09:49.583 losers were, uh, Biosquid, Elfutils, and then Cdftools. The first two of those had 100 00:09:52.083-->00:09:58.167 percent crash rate for their eight binaries tested. And Biosquid had several hundred 00:09:58.167-->00:10:05.042 crashes per binary so that was definitely the, uh, biggest loser out of our tests so far. 00:10:05.042-->00:10:10.042 For, uh, the AFL data, we had static analysis to go with it so then we can start doing some 00:10:14.625-->00:10:19.958 really preliminary data science of the type we're going to be doing more and more of as our 00:10:19.958-->00:10:25.125 data set gets to be bigger. Right now, uh, we're only getting numbers of unique 00:10:25.125-->00:10:30.250 crashes and the signals. But later on we'll have primitives identified and then we'll be 00:10:30.250-->00:10:35.250 able to do more complex, uh, math magic. But, uh, for now, we have the numbers of unique 00:10:38.042-->00:10:44.917 crashes. And then we tried correlating those with risky and bad functions. You know, so 00:10:44.917-->00:10:48.958 risky and bad functions mean what you think. They're the functions that somebody looks 00:10:48.958-->00:10:53.875 for when they're bug hunting because they know that people screw them up a lot. Uh a- bad 00:10:53.875-->00:10:58.875 is worse than risky. You know. Um. So, uh, the highest correlations that we found were 00:11:01.708-->00:11:08.500 for execlp and execv which, you know, had a pretty high correlation rate with numbers of 00:11:08.500-->00:11:14.333 crashes. Uh. And, you know, it- this is, again, really early days and a pretty small data 00:11:14.333-->00:11:20.042 set. So it's an exciting, early tidbit and tells us there's going to be some really cool 00:11:20.042-->00:11:25.042 stuff when we, uh, get this fully op- up and running. So, uh, onto our static analysis 00:11:29.875-->00:11:36.375 data where we've got a lot more to work with. Uh. So first up, operating systems. Um. You know, 00:11:36.375-->00:11:42.375 an operating system has thousands, maybe tens of thousands of binaries. Each 00:11:42.375-->00:11:47.375 binary has a score. So the way we show an entire environment and its stances that we do a 00:11:50.792-->00:11:55.792 histogram of all the scores in that operating system. And, uh, you can't see the numbers here 00:11:57.833-->00:12:01.625 because I didn't want to get too much on one slide, but you don't really have to. You just need to 00:12:01.625-->00:12:07.208 see the shape of the distribution. Uh. So the X-axis here are all the same. They go 00:12:07.208-->00:12:12.208 from negative ten to 110. And, uh, low numbers are bad, they mean softer targets. Higher 00:12:14.292-->00:12:19.292 numbers are good. So the further to the right your distribution gets, the better off you are. 00:12:21.292-->00:12:27.417 And, uh, the Y-axis is not the same on all of these because it made things sort of unreadable. 00:12:27.417-->00:12:32.417 Uh. But it's not really important for this story here, uh, 'cause, you know, uh, what 00:12:35.250-->00:12:40.333 we're looking for again is the shape. So what we've called out is the fifth, fiftieth, and 00:12:40.333-->00:12:45.208 ninety-fifth percentile marks 'cause this allows you to see how long the tails are in either 00:12:45.208-->00:12:50.750 direction. We particularly care about the left tail 'cause that's the s- the low hanging 00:12:50.750-->00:12:55.792 fruit in your environment. The longer that tail gets, the more soft targets you have lying 00:12:55.792-->00:13:01.667 around and the less you've done to clean up that low hanging fruit. So, um, Windows 10 does 00:13:01.667-->00:13:07.250 actually very well. Uh, which, you know, you might know from seeing the new safety features 00:13:07.250-->00:13:12.250 and improvements that they had in that release. The fifth percentile mark is pretty close 00:13:12.250-->00:13:18.667 to the fiftieth. And, uh, they're really very consistent in the application armoring 00:13:18.667-->00:13:23.250 features that they're including which is why the distribution looks so different than all the 00:13:23.250-->00:13:29.333 other ones. It's a lot more uniform in the safety practices. So then you end up with a lot 00:13:29.333-->00:13:34.750 more things in the same bin. So for example, that biggest bin there is the sixtyfive to 00:13:34.750-->00:13:39.750 seventy and there's 5,500 binaries in there. So that's the most consistent off-the-shelf, 00:13:42.500-->00:13:47.500 uh, use of safety features that we've found so far. Uh. OSX El Capitan has scores in generally 00:13:51.208-->00:13:56.208 the same region on, like, the chart as OSX- I mean, as Windows. But it's, uh- the fifth 00:13:59.458-->00:14:06.042 percentile mark has moved a lot further down. So it's, uh, got a lot of the same safety features 00:14:06.042-->00:14:12.167 but they're not being as consistently applied. And, uh, there's just a lot more low 00:14:12.167-->00:14:18.375 hanging fruit hanging around. So the percentiles all move down a bit. The fiftieth is almost the 00:14:18.375-->00:14:23.167 same, but the thing we care about most is that that, uh, fifth percentile mark went down 00:14:23.167-->00:14:28.167 by almost fourteen points. And then on Linux, again, the fifth percentile mark moves noticeably 00:14:31.833-->00:14:36.833 further, uh, down the scale. And, uh, the distribution gets a little lower as a whole. So, you 00:14:40.042-->00:14:46.750 know, this isn't any sort of surprise to most bug hunters because this correlates exactly 00:14:46.750-->00:14:52.500 with the prices for exploits in these environments. And that makes sense 'cause we're trying 00:14:52.500-->00:14:59.167 to assess the cost to newly exploit software. And, uh, so, you know, the, uh- you get paid 00:14:59.167-->00:15:03.958 for your work. The thing that's harder to exploit, you get more money. And- But it's nice to 00:15:03.958-->00:15:09.458 know that the Pwn2own values for these exploits correlated so closely with our results. It's 00:15:09.458-->00:15:16.042 good confirmation that we are measuring what we want to measure correctly. And the last 00:15:16.042-->00:15:21.875 bit of data that we have for standard OS's is that more recently we got, uh, some data 00:15:21.875-->00:15:27.833 on Sierra so then we can do a comparison between Sierra and El Cap to see what improvements 00:15:27.833-->00:15:32.833 have been made. And, uh, so, uh Sierra got a bit bigger. No surprise there. Everything 00:15:35.042-->00:15:40.500 always gets bigger. And then, uh, ASLR's application rate on El Capitan was already pretty 00:15:40.500-->00:15:45.500 high, but it inched a little close to 100 percent. Um. The Heap DEP was the only safety 00:15:48.125-->00:15:53.125 feature to have a decrease in use. Uh. Heap DEP data execution prevention, you know, one of the 00:15:56.000-->00:16:01.000 exploit mitigation features we look for. And this is explained by the fact that most modern OSX 00:16:03.375-->00:16:08.375 compilers, if you compile something as 64 bit, then the flag for Heap data execution 00:16:10.500-->00:16:16.417 prevention is off. So the increase in 64 bit matches pretty closely with the decrease 00:16:16.417-->00:16:21.417 in Heap DEP. Uh. This is a worrisome trend in that any time you have a flag that's off, uh, 00:16:24.875-->00:16:29.542 there's the chance that that safety feature will not be used. Some people argue it doesn't 00:16:29.542-->00:16:35.750 matter. That most modern OSX environments enable Heap data execution prevention by default, 00:16:35.750-->00:16:40.083 but that's not always true. In some cases, it'll see you don't have this flag and it'll say, 00:16:40.083-->00:16:45.083 "Oh. You're not compatible. I'll turn it off." And, uh, you know, there's nothing that will go bad 00:16:47.125-->00:16:53.958 if you have the flag on so have the flag on if you can do it. You know. Uh. So I'm not clear 00:16:53.958-->00:16:59.167 on why that's a trend, but it is. And, uh, it'd be nice if that got turned around sometime 00:16:59.167-->00:17:04.167 soon. Uh. We also look at source fortification. And, uh, the numb- percent of binaries that 00:17:09.333-->00:17:14.333 were fortified was about the same. Although the number that were unfortified went down. So 00:17:14.333-->00:17:19.333 good. Uh. You might notice that these numbers don't add up to 100 percent. That's because the 00:17:22.167-->00:17:26.375 way that we tell if there was source fortification is that we look for the unfortified 00:17:26.375-->00:17:32.458 versions of these functions and we look for the fortified versions, and, you know, if you 00:17:32.458-->00:17:36.750 don't have either in your binary, we don't know what flags you had. And it doesn't really 00:17:36.750-->00:17:41.750 matter, you know, 'cause it's irrelevant. It didn't make any change. And then, uh, the last 00:17:44.875-->00:17:50.958 bit is looking at what percent of the binaries had functions from our different categories. 00:17:50.958-->00:17:57.792 So good, bad, risky, and ick all mean exactly what you think. Uh. Ick is just the couple functions 00:17:57.792-->00:18:03.375 that no one should ever use in commercial code. Uh, there's not... really it is just two or 00:18:03.375-->00:18:07.292 three functions. So, you know, it's, uh... you have to be special to get in there. And, 00:18:07.292-->00:18:12.292 uh, here again the story was pretty positive. Bad and risky both went down by about ten 00:18:15.125-->00:18:21.292 points. I mean, ten percent. And then, uh, good was actually the biggest increase we saw across 00:18:21.292-->00:18:25.625 the board. Uh. Going from eight percent to twentyfive percent of binaries with good functions in 00:18:25.625-->00:18:30.625 them was, you know, a really pleasant surprise. And, uh, so next, I'm going to look at a 00:18:33.792-->00:18:38.792 chart that shows the impact of corporate policies for software installation. Uh, so, I'll do 00:18:41.667-->00:18:47.458 some explaining on this one. Just bear with me. The, uh, blue bars- the ones that are in the 00:18:47.458-->00:18:53.958 middle row- are the base installation for a large enterprise organizations- well, 00:18:53.958-->00:19:00.625 large-ish... like, 1-->000 plus employees. Uh. Their base OSX install for new employees. So 00:19:00.625-->00:19:06.250 this is the clean installation- everything that's on it when they give it to the employee. 00:19:06.250-->00:19:11.667 The little gray bars that you can hardly see in front of the blue bars are what executables 00:19:11.667-->00:19:18.458 from that clean installation their employees actually use. So which ones got executed. And 00:19:18.458-->00:19:23.458 then the giant orange bars are what ex-executables get run by employees when the company gives 00:19:26.292-->00:19:31.292 them the right to install whatever software they want. So, the- if you're looking at the 00:19:35.500-->00:19:42.333 live attack surface being presented by your organization, those gray bars versus those 00:19:42.333-->00:19:47.333 orange bars are the comparison. They, uh, um, and obviously, you can't have just those tiny bars. 00:19:49.875-->00:19:54.792 You have to allow them to install some things, but being judicious in what you allow is 00:19:54.792-->00:19:59.958 important, because if you aren't, you're increasing your attack surface by a sizable 00:19:59.958-->00:20:05.000 amount. And the... in particular, the low hanging fruit increases a great deal. 00:20:05.000-->00:20:10.000 The negative scores for the clean installs stop at negative ten, but on the dirty install, 00:20:12.250-->00:20:17.250 they go down to negative thirtyfive. So you're increasing your lower tail by a lot. And 00:20:21.042-->00:20:25.583 again that's not something that's a surprise to anyone, but nobody's quantified it and 00:20:25.583-->00:20:30.583 showed somebody a chart of, you know, "Here's what you're do-doing with this policy." Um. 00:20:33.208-->00:20:37.292 Ok. And next we're going to look at a few different linux distributions. The first one is 00:20:37.292-->00:20:43.500 the same linux that you saw earlier. This is what you get as the off-the-shelf linux sixteen- 00:20:43.500-->00:20:49.167 Ubuntu 16. And then, uh, the one below it is an example of what you get when you take a 00:20:49.167-->00:20:54.167 off-the-shelf linux install and do a modest hardening effort. So no co- no code modification, but 00:20:56.792-->00:21:02.083 you take out the stuff you obviously don't need and you recompile it with all the modern 00:21:02.083-->00:21:07.625 safety features enabled that won't get in the way of normal operations. You know, this is a 00:21:07.625-->00:21:12.500 very reasonable thing to expect from any major vendor when they're putting something out 00:21:12.500-->00:21:19.458 with a linux environment. Obviously that's not what they do right now, 'cause nobody 00:21:19.458-->00:21:24.458 makes that modest, uh, effort if no one's going to notice. Uh. So what you see if you take apart a 00:21:27.125-->00:21:33.042 Smart TV and get the binaries is, uh, the two things we have on the bottom here. A Samsung 00:21:33.042-->00:21:38.042 Smart TV and then a LG Smart TV. And, uh, these are worse than the off-the-shelf linux. They're 00:21:40.958-->00:21:46.792 lower distribution, uh, they have lower points overall 'cause they're missing a lot of 00:21:46.792-->00:21:53.500 fundamental safety features. Uh, you know, they have- they're smaller. They're 4-->000 and 2-->000 00:21:53.500-->00:21:59.375 binaries respectively. But they're clearly not, uh, doing all the things a modern linux 00:21:59.375-->00:22:04.375 environment does from a safety perspective. And, uh, for a comparison, if you look at 00:22:06.625-->00:22:13.417 Amazon's AMIs for linux environments, they're the same size. These are both 2-->000 00:22:13.417-->00:22:20.083 binaries so, you know, very comparable footprint. But this is what happens when you care 00:22:20.083-->00:22:25.125 about the security of your linux environment. The distributions are much higher and they've 00:22:25.125-->00:22:29.250 taken out all the low-hanging fruit. The scores on the other ones went well into the negative 00:22:29.250-->00:22:36.083 numbers, but here the minimum scores are forty and thirty respectively. So that's a nice 00:22:36.083-->00:22:42.125 accomplishment and it shows that it's entirely possible to have a full linux distribution in the 00:22:42.125-->00:22:47.125 footprint that they- these IOT devices have while doing everything more correctly. So to 00:22:51.958-->00:22:58.042 see why those Smart TVs' scores were as low as they are here's, uh, some data similar to what we 00:22:58.042-->00:23:04.208 saw for the OSX stuff, uh, just showing all the safety features that the Smart TVs don't have. 00:23:04.208-->00:23:09.208 Uh. I've highlighted in light orange and light red the, uh, bad and really bad numbers 00:23:12.333-->00:23:17.333 respectively. Uh. You know... math terms. And, uh, I'm comparing it to the harden linux 00:23:19.583-->00:23:25.042 because they should have made some effort to harden. That's the bar they should be compared 00:23:25.042-->00:23:30.042 to. So, uh, ASLR's lower than the standard. Because the standard is that we're getting 00:23:32.083-->00:23:37.083 really close to 100 percent and then, uh, RELRO is not existent. That's a- a linux-specific 00:23:40.417-->00:23:46.875 application armoring feature that's really pretty common these days. And, uh, stack 00:23:46.875-->00:23:51.875 guards and fortification were basically nonexistent on LG. They were there on Samsung but 00:23:53.958-->00:23:58.958 much lower than industry standard. And, you know, that's all reflected in their scores. 00:24:02.250-->00:24:07.625 So that's it for, uh, full environment views of operating systems. Now we're going to use 00:24:07.625-->00:24:12.625 those histograms as the context for scores for individual applications. And so you see the 00:24:14.792-->00:24:21.375 same histogram that we had for OSX earlier. Only now it's showing you where the different 00:24:21.375-->00:24:26.208 applications' scores fall. And below it we've got our hardening line, softer targets to the 00:24:26.208-->00:24:31.708 left, harder targets to the right. And, uh, we're calling out where specific applications 00:24:31.708-->00:24:36.708 of note fell on them. So for OSX browsers, uh, the Firefox score was not great. Uh. Safari was 00:24:41.708-->00:24:48.375 middle of the pack. And then Chrome did decently. Not amazing, but decently. And, uh, 00:24:48.375-->00:24:53.375 you know, this is a... I don't know. Not maybe a earth shattering result to some 00:24:56.208-->00:25:01.208 people, but again it's nice to quantify things. For linux, just to have a third one to throw in 00:25:03.583-->00:25:08.708 we did Opera. They're clearly a little bit further behind the times in terms of some of the 00:25:08.708-->00:25:15.375 safety features. Um. And, uh, for Chrome and Firefox- they presented an interesting case. 00:25:15.375-->00:25:21.333 So here, first I'm showing the scores that they got just for the executable- the main 00:25:21.333-->00:25:26.375 executable alone. Not taking into account the libraries they're calling. I'm doing this 00:25:26.375-->00:25:33.375 because viewed that way, they got the same score. Granted, uh, the one bit of static analysis 00:25:33.375-->00:25:39.333 we're still working on is our evaluation of sandboxing. If one that gets finished, Chrome will 00:25:39.333-->00:25:46.167 probably gain a few points so they'll probably, uh, get a little further ahead of Firefox 00:25:46.167-->00:25:52.500 there. But right now, they're neck and neck. Except when you bring libraries into account 00:25:52.500-->00:25:57.500 where Firefox falls fifteen points behind Chrome. And, you know, this is important to point 00:25:59.917-->00:26:05.292 out because this was Firefox's home turf. This was their chance to shine, but, you know, if you 00:26:05.292-->00:26:09.250 bring in libraries that are scoring negative tens, then that's going to, you know, be 00:26:09.250-->00:26:14.250 reflected in your permanent record. And, uh, you know, they, uh... A lot of times when people 00:26:17.083-->00:26:22.917 are figuring out what libraries to use, they think first about functionality because that is 00:26:22.917-->00:26:28.958 admittedly very important. As is what license its using. But third place should be its 00:26:28.958-->00:26:33.958 security stance. Uh. People fall into the trap of not my code, not my problem, but it's code 00:26:36.417-->00:26:42.667 you're linking to from your product and choosing to execute when your customer runs your 00:26:42.667-->00:26:47.667 product. So you should care about it. Uh, that mini rant over. Onto Windows browsers. So, 00:26:53.500-->00:27:00.083 uh, here we looked at Firefox, Chrome, and Edge. And, uh, again, Chrome will probably gain 00:27:00.083-->00:27:05.625 a few points when we get the fir- uh, sandboxing thing in place. But for now, Edge being 00:27:05.625-->00:27:11.625 on its home turf... It got a near-perfect score. Uh, slightly better than perfect 'cause, uh, 00:27:11.625-->00:27:16.875 they did- they got bonus points for a couple things that very few people do. And, uh, um... So 00:27:16.875-->00:27:21.875 the- not a huge surprise there. Firefox, uh, is again away from their home turf so they fell 00:27:25.125-->00:27:31.542 down a bit. And, uh, if we were going to pick one of these Firefox binaries to count as the 00:27:31.542-->00:27:37.792 Firefox binary, uh, we would pick the 32 bit because that was the one that we got when we went 00:27:37.792-->00:27:42.708 to their main website to download. You know, I mean, it didn't have explicitly labeled 00:27:42.708-->00:27:48.167 as 32 bit but when we checked, that's what it was and, you know, it took a little bit of 00:27:48.167-->00:27:54.000 hunting to find the 64 bit one so your average consumer is not going to do that. So, uh, you 00:27:54.000-->00:27:59.000 know, you get credit for what you make available. And, uh, again, uh, the pwn2own values 00:28:01.333-->00:28:08.083 for exploits match up pretty closely with the, uh, rankings that we'd give these, uh, 00:28:08.083-->00:28:13.625 applications. You know, Microsoft Edge and Google Chrome are at the top. Safari's in the 00:28:13.625-->00:28:18.625 middle. And then Firefox is relatively cheap. Um. And then our last bit of static analysis 00:28:22.250-->00:28:28.000 data that we're looking at is Microsoft Office for OSX. Last year we beat up a bit on them 00:28:28.000-->00:28:33.000 for Office 2011. People rightly asked, "Ok. What about 2016?" So we bought that and tested it. 00:28:35.750-->00:28:40.750 And, uh, presents an interesting story. Uh. So here's those bad, uh, 2011 scores that we had last 00:28:46.750-->00:28:53.417 year. Uh. In particular, their auto update scoring of seven was pretty embarrassing. And their 00:28:53.417-->00:28:59.792 average for all binaries in that package was around a sixteen so, uh, you know, just, uh, not a 00:28:59.792-->00:29:04.792 good going all around. But when you look at Office 2016, uh, they really did much better. The 00:29:08.833-->00:29:14.125 average binary score increased by sixty points. And the auto update went from a seven to a 00:29:14.125-->00:29:20.917 sixty-four. And, uh, you know when a new Office Suite comes out, I'm not in a big rush to 00:29:20.917-->00:29:27.125 get it because all the features that I need, I already have and am using in the current one and 00:29:27.125-->00:29:31.583 I know that all i- all is going to happen is they'll have moved a bunch of buttons on me. It's 00:29:31.583-->00:29:37.042 going to take me a week to be productive in it again. And so, you know, it's not one of my 00:29:37.042-->00:29:42.250 priorities. But sometimes there's more hidden features and this is an example of that. So 00:29:42.250-->00:29:47.417 if somebody who's responsible for purchasing decisions for a large organization, had this 00:29:47.417-->00:29:51.958 sort of data available, then they'd be like, "Ok guys. Sorry. You have to find out new 00:29:51.958-->00:29:56.958 locations for buttons." Like, you know, the- we're upgrading. So, uh, done with our data that 00:30:00.125-->00:30:05.708 we've collected. Now onto the work we've been doing with Consumer Reports. Uh. In 00:30:05.708-->00:30:12.500 particular, uh, the digital standard which launched earlier this year. And, uh, you can go 00:30:12.500-->00:30:18.500 to this website to see the first draft we put together. And what this is is that Consumer Reports 00:30:18.500-->00:30:24.708 recognizing the need for software evaluation for their own product reviews, uh, asked 00:30:24.708-->00:30:30.833 us to be part of a group they're forming. Uh, the other groups are all focused on digital 00:30:30.833-->00:30:36.333 rights. Uh, privacy, one of them is focused more on governance side, there's data sharing... 00:30:36.333-->00:30:41.333 You know, they're, uh, trying to be a pretty broad view of a consumer digital rights. And, 00:30:44.125-->00:30:50.667 uh, together we put together this, uh, uh, first draft of the digital standard which is our 00:30:50.667-->00:30:57.208 first step at a testing standard for how you would evaluate software or IOT products from a 00:30:57.208-->00:31:02.917 digital rights perspective. And, uh, we've done a couple of initial rounds of testing based 00:31:02.917-->00:31:08.292 on the standard we put together, uh, including the browsers and Smart TVs that you saw earlier 00:31:08.292-->00:31:14.542 in the talk. And all those other organizations tested the same devices but since the full 00:31:14.542-->00:31:20.000 report hasn't been published yet, I'm sharing our, uh, side of that 'cause that's the part 00:31:20.000-->00:31:25.875 we're allowed to share. Um. Right. So if you go to the website and look at the digital 00:31:25.875-->00:31:31.167 standard. This is what you'll see. Uh. Everything gets more technical the further to the 00:31:31.167-->00:31:36.708 right you go. So the leftmost column is the name for a particular criteria like 00:31:36.708-->00:31:42.958 vulnerability disclosure program. After that is, uh, lay-person readable material on 00:31:42.958-->00:31:48.708 what that criteria means. After that is a slightly more technical description of what 00:31:48.708-->00:31:53.583 indicators we're checking for when we're evaluating that criteria. And then after that is 00:31:53.583-->00:31:59.333 a brief description of the test procedure. So if something has a little green check mark by it 00:31:59.333-->00:32:05.792 like this one does, that means we're pretty happy with this one. It's pretty fully baked and 00:32:05.792-->00:32:10.250 we feel like we have a test procedure in place that scales well enough for Consumer 00:32:10.250-->00:32:16.750 Reports' needs or some other similar kind of effort. If something has a little flask 00:32:16.750-->00:32:23.000 like the next one does, uh, that yellow flask means usually that we can test this for a single 00:32:23.000-->00:32:28.625 product, but we don't think it scales well enough for testing an entire product vertical. Uh. 00:32:28.625-->00:32:34.167 So, you know, there will be a test procedure there but it's something that needs work in 00:32:34.167-->00:32:40.792 order to figure out how to make it, uh, be more automated. And then if something has a little 00:32:40.792-->00:32:45.750 red exclamation point, that means that none of the groups that are part of this coalition 00:32:45.750-->00:32:51.167 right now have an in-house capability to test this thing. And, uh, that's why that 00:32:51.167-->00:32:56.167 column's blank. And, uh... So if you go to this site, all of this is linked to the GitHub page so, 00:32:59.375-->00:33:04.542 you know, if you read a criteria or description of something and you think we didn't word it 00:33:04.542-->00:33:11.083 right or we're barking up the wrong tree, you can click and go to GitHub and comment and 00:33:11.083-->00:33:16.083 propose a change. If you have some way of turning one of our orange or red icons into a 00:33:19.333-->00:33:25.083 little green check mark, then you can let us know that and we'd love to get input on how to 00:33:25.083-->00:33:30.083 make our overall procedure more complete. The- there was some question as to if we don't have 00:33:32.250-->00:33:36.250 the ability to test something why are we including it? It's because we think those things 00:33:36.250-->00:33:41.167 are important and that an evaluation of a product would be incomplete if it wasn't there. 00:33:41.167-->00:33:46.542 But, uh, you know, and us not having the in-house capability to test it right now doesn't 00:33:46.542-->00:33:52.208 change that. And in this case what I mean by "us" is the total, uh- all the groups that 00:33:52.208-->00:33:57.500 are part of the digital standard. Not CITL. We're just a little... We're only responsible 00:33:57.500-->00:34:02.500 for a couple of these criteria, thankfully. So, um, if you are interested in the digital 00:34:05.500-->00:34:10.500 standard and want to be a part of that, I, uh, encourage people to go to the website and, uh, 00:34:12.583-->00:34:18.833 get on GitHub and or look at our bits where we are missing something and think about 00:34:18.833-->00:34:24.542 whether that could be a new research project. Um. And then, uh, if you have anything follow 00:34:24.542-->00:34:29.542 up for Cons- for CITL, our contact info is at the top here. So thank you very much. 00:34:34.792-->00:34:39.792 [Applause] Um and, uh, are we doing questions? I don't know. [off mic response] Uh. Yeah, I 00:34:53.667-->00:34:58.667 can do ten minutes of questions. [off mic question] Oh. We're a nonprofit so, uh, we're figuring 00:35:04.458-->00:35:11.208 out a way to support the organization while sharing as much data as we can. Uh. But 00:35:11.208-->00:35:17.667 public service is the main goal. Uh. Sorry, I forgot to repeat that question. Um, but he just 00:35:17.667-->00:35:22.667 wanted to know whether, uh, what the business model was. Yes? [off mic question] Uh, so he 00:35:29.500-->00:35:34.875 wanted to know if we're looking at historically impactful vuln-vulnerabilities and those 00:35:34.875-->00:35:40.125 binaries. And we're not looking at historic data right now 'cause we've got more than 00:35:40.125-->00:35:45.208 enough present day data and we're, uh, using the fuzzing data to compare against. When we 00:35:45.208-->00:35:51.208 publish our corpus of all the static analysis data and fuzzing data, then somebody else would 00:35:51.208-->00:35:55.250 be welcome to do that study. It's just the, you know, so-many-hours-in-a-day kind of 00:35:55.250-->00:36:00.250 deal. Anyone else? It's a little hard to see the audience. You. [off mic question] Uh. Could 00:36:15.250-->00:36:20.250 you- could you say that louder? [repeated off mic question] Uh. So are we looking at 00:36:26.250-->00:36:31.583 dependencies for software? Uh, if it's a library that it links to then we look at that. Then we 00:36:31.583-->00:36:36.458 look at the libraries it links to. We go down that whole tree. If it's some sort of more 00:36:36.458-->00:36:41.375 indirect, uh, dependency that we can't see through library linkage, I don't know what that 00:36:41.375-->00:36:45.750 would be but, then we wouldn't find it. But yeah, we look at the whole tree of library 00:36:45.750-->00:36:50.750 dependencies. Anyone else? We're all good? No- there? Oh, okay. We're all good. Ok. Great! An- 00:36:59.500-->00:37:04.500 oh, wait. You have a question? [off mic question] I'm not currently on Twitter. [off mic 00:37:12.500-->00:37:17.500 statement] What? [off mic statement] Um. Yeah, we're in between DARPA officers. Ok. So 00:37:22.458-->00:37:27.458 everyone's good? Uh, then have a great Def Con. [applause]