1 00:00:00,000 --> 00:00:01,751 VAAGN TOUKHARIAN: Good day. 2 00:00:01,918 --> 00:00:02,999 My name is Vaagn. 3 00:00:02,999 --> 00:00:06,999 TIGRAN GEVORGYAN: And my name is Tigran. 4 00:00:06,999 --> 00:00:11,209 VAAGN TOUKHARIAN: We will talk about HTTP time bandit 5 00:00:11,209 --> 00:00:17,334 but before we start our talk, I have one thing to ask. 6 00:00:17,334 --> 00:00:19,375 This is our first talk at DEF CON. 7 00:00:19,375 --> 00:00:21,792 So don't be nice to us. 8 00:00:22,042 --> 00:00:25,834 Be very, very, very gentle to us. 9 00:00:25,918 --> 00:00:26,918 Please. 10 00:00:26,999 --> 00:00:28,209 Thanks. 11 00:00:28,667 --> 00:00:29,999 (Applause). 12 00:00:30,334 --> 00:00:32,375 So about us. 13 00:00:32,501 --> 00:00:35,375 Tigran and I work for Qualys. 14 00:00:35,375 --> 00:00:37,292 We break stuff for fun. 15 00:00:37,417 --> 00:00:40,167 Sometimes we fix them too. 16 00:00:40,417 --> 00:00:46,334 We are interested in time travel and we'll talk about that a bit later. 17 00:00:46,667 --> 00:00:48,999 And we both love to tri, we do triathlons. 18 00:00:50,918 --> 00:00:53,417 So what we going to talk about? 19 00:00:53,999 --> 00:00:58,250 The presentation short, but effective maybe. 20 00:00:58,667 --> 00:01:01,709 You will judge by leaving the room or not. 21 00:01:02,459 --> 00:01:05,999 It's going to be about another application, 22 00:01:05,999 --> 00:01:10,667 DOS method and we'll then show you the tool. 23 00:01:11,209 --> 00:01:17,167 We'll go through some stats and we'll talk about what are good 24 00:01:17,167 --> 00:01:22,083 or bad uses of tool or method and then we'll glance 25 00:01:22,083 --> 00:01:26,792 over the defense against this method. 26 00:01:29,083 --> 00:01:30,417 Why? 27 00:01:30,417 --> 00:01:31,959 Why do we do this? 28 00:01:32,083 --> 00:01:35,999 It's obvious because we wanted to have time machine. 29 00:01:35,999 --> 00:01:37,626 We couldn't buy any on eBay. 30 00:01:38,626 --> 00:01:43,999 We could, but it was similar to what you see on the screen. 31 00:01:43,999 --> 00:01:45,584 And that really doesn't work. 32 00:01:46,250 --> 00:01:49,501 So we thought we will make our own time machine 33 00:01:49,501 --> 00:01:52,125 and I think we succeeded. 34 00:01:53,083 --> 00:01:58,501 The time machine that we have, does one thing and does it well. 35 00:01:58,999 --> 00:02:04,584 It can give you the future state of your web server given 36 00:02:04,584 --> 00:02:09,999 the resources you are going to push on it. 37 00:02:10,125 --> 00:02:11,834 About it a bit later. 38 00:02:14,626 --> 00:02:19,292 So we going to talk about application layer, 39 00:02:19,292 --> 00:02:22,375 DOS tool or method. 40 00:02:22,375 --> 00:02:28,334 For us, DOS tool are two different kinds. 41 00:02:28,501 --> 00:02:35,209 There are some that are not that smart and another group that is a bit smarter. 42 00:02:35,209 --> 00:02:37,542 Not smart apart is pushing your resource over and 43 00:02:37,542 --> 00:02:41,125 over and over and hoping the resource will die. 44 00:02:41,626 --> 00:02:47,751 And then you do that from multiple instances, it works. 45 00:02:48,083 --> 00:02:50,375 But that doesn't have feedback. 46 00:02:50,375 --> 00:02:55,999 And sometimes a load is near symmetrical. 47 00:02:56,334 --> 00:03:02,417 Smarter bots like slow loris and slow HTTP test, and PKI abuse, 48 00:03:02,417 --> 00:03:08,999 they are smarter and they are asymmetrical in nature. 49 00:03:09,584 --> 00:03:15,999 Our proposed method is going to be striving to be asymmetrical. 50 00:03:16,083 --> 00:03:20,667 TIGRAN GEVORGYAN: So how do we do this? 51 00:03:20,876 --> 00:03:23,709 So the point of this talk, the information from future 52 00:03:23,709 --> 00:03:27,667 is picking the resources out of all the available ones, picking 53 00:03:27,667 --> 00:03:31,501 the ones that are most painful for the server. 54 00:03:32,209 --> 00:03:34,792 While people say, oh, servers done feel pain, we 55 00:03:34,792 --> 00:03:36,626 all know they do. 56 00:03:37,542 --> 00:03:39,626 So how do we do this? 57 00:03:39,626 --> 00:03:42,709 We spider the website, we collect, compile a list of the resources 58 00:03:42,709 --> 00:03:46,083 and then we start measuring certain things, for example, 59 00:03:46,083 --> 00:03:50,083 the download speed or the start to download time. 60 00:03:51,125 --> 00:03:55,999 What this gives us is a lot of data and what deals very well with a lot 61 00:03:55,999 --> 00:03:58,999 of data and that's statistics. 62 00:03:58,999 --> 00:04:02,999 So we use statistics to filter through the data. 63 00:04:03,083 --> 00:04:06,626 We calculate the average of all the download times. 64 00:04:06,626 --> 00:04:10,375 That gives us a baseline, what is a normal transfer time. 65 00:04:10,667 --> 00:04:16,125 And then we can tell oh, these resources are slower or faster. 66 00:04:16,375 --> 00:04:23,167 So we use to formalize this, we use the we used mean as the measure 67 00:04:23,167 --> 00:04:27,959 of central tendency, and we calculate this 68 00:04:27,959 --> 00:04:30,999 standard deviation. 69 00:04:30,999 --> 00:04:34,876 The resources that have high standard deviation, that means that 70 00:04:34,876 --> 00:04:39,959 they are inaccurate in terms of they can be fast or they can be slow, 71 00:04:39,959 --> 00:04:42,751 but we can't really tell. 72 00:04:42,751 --> 00:04:44,459 So we throw them away. 73 00:04:44,999 --> 00:04:49,999 We select resources that have slower download times or start 74 00:04:49,999 --> 00:04:55,501 to download times and we think that this is related to CPU load, and 75 00:04:55,501 --> 00:05:00,125 the reasoning is that all the transfer delays are the same 76 00:05:00,125 --> 00:05:05,083 for all the resources, the network related ones. 77 00:05:05,083 --> 00:05:08,999 So if some resource has higher transfer time, that must be resource specific 78 00:05:08,999 --> 00:05:11,792 and that makes it interesting. 79 00:05:11,999 --> 00:05:17,375 So this, for example, is a chart of resource download for a website 80 00:05:17,375 --> 00:05:22,751 and you can see that the majority of resources fall close together 81 00:05:22,751 --> 00:05:25,334 in some middle range. 82 00:05:25,459 --> 00:05:29,999 There are some outliers, so fast outliers and some slow ones. 83 00:05:29,999 --> 00:05:32,459 The ones that are interesting are the slow ones. 84 00:05:32,459 --> 00:05:33,459 The question is. 85 00:05:33,459 --> 00:05:35,501 What causes these resources to be slow? 86 00:05:35,501 --> 00:05:40,626 Is it the CPU load or is it just the sleep standard on the page. 87 00:05:41,999 --> 00:05:45,999 So small demonstration and not to anger the demo Gods, 88 00:05:45,999 --> 00:05:47,999 we did a video. 89 00:05:49,167 --> 00:05:50,501 (Laughter). 90 00:05:50,501 --> 00:05:53,501 So here the tool is running against a locally hosted web server 91 00:05:53,501 --> 00:05:58,083 and when it's done it, generates a list of URLs with the measurement mean, 92 00:05:58,083 --> 00:06:00,626 the standard deviation. 93 00:06:01,083 --> 00:06:04,459 While this is nice, it's not very, very useful. 94 00:06:04,459 --> 00:06:10,083 Instead, the tool can be constructed to output this information in the HTML. 95 00:06:10,292 --> 00:06:14,876 We see it in increasing order. 96 00:06:14,918 --> 00:06:19,209 So the HTML resources down at the bottom are the fast and 97 00:06:19,209 --> 00:06:23,209 the PHP resources are near the top. 98 00:06:23,834 --> 00:06:27,584 And this is because PHP requires the PHP interpreter invocation, 99 00:06:27,584 --> 00:06:30,959 which takes some additional time and the resource 100 00:06:30,959 --> 00:06:35,999 at the top are some artificially made pie calculation, which understandably 101 00:06:35,999 --> 00:06:38,083 takes a long time. 102 00:06:38,292 --> 00:06:41,834 And so XML is very nice because it can be head 103 00:06:41,834 --> 00:06:45,542 into some other tools and I have one example which 104 00:06:45,542 --> 00:06:48,876 is an XXLT which transforms this information 105 00:06:48,876 --> 00:06:52,709 into something that can be visualized. 106 00:06:52,792 --> 00:06:56,876 So you will see a scatter chart of download speed 107 00:06:56,876 --> 00:07:02,501 and standard deviation and there are four quadrants here. 108 00:07:02,999 --> 00:07:04,999 The top quadrant, the top two quadrants, they are 109 00:07:04,999 --> 00:07:08,250 the resources that have high standard deviation, which means that 110 00:07:08,250 --> 00:07:10,999 the measurements were inaccurate. 111 00:07:10,999 --> 00:07:12,542 So since we cannot trust these resources 112 00:07:12,542 --> 00:07:14,501 we throw them away. 113 00:07:14,501 --> 00:07:17,626 Instead we concentrate on the resources on the bottom half. 114 00:07:17,999 --> 00:07:22,125 On the right side, these resources are fast resources, 115 00:07:22,125 --> 00:07:24,999 and consistently so. 116 00:07:24,999 --> 00:07:26,876 So they are not bottlenecks. 117 00:07:26,999 --> 00:07:28,999 Instead, we concentrate on the resources 118 00:07:28,999 --> 00:07:33,999 on the lower left quadrant, and closer to the zero zero point, the better I mean, 119 00:07:33,999 --> 00:07:37,999 the painful the resource would be for the server. 120 00:07:38,999 --> 00:07:50,459 So the tool also can be can generate data for visualization in other means. 121 00:07:50,959 --> 00:07:55,999 For example, the next one, whenever it comes, it will be the tool 122 00:07:55,999 --> 00:08:01,417 is instructed to generate graph dot language graph. 123 00:08:01,417 --> 00:08:03,999 So here you can see a small website with transitions 124 00:08:03,999 --> 00:08:07,999 and each transition is marked with a transfer time. 125 00:08:08,125 --> 00:08:11,334 So slowly the resources could be highlighted. 126 00:08:11,334 --> 00:08:14,334 I couldn't do a good job here, but the capability is there. 127 00:08:14,584 --> 00:08:19,125 So next, the tool is going to scan real website, built 128 00:08:19,125 --> 00:08:24,167 with concrete CMS, and so we give it the URL to start with, 129 00:08:24,167 --> 00:08:29,542 how deep into the website to go into and how many measurements 130 00:08:29,542 --> 00:08:32,876 for each resource to take and write it 131 00:08:32,876 --> 00:08:37,999 into concrete.xml which will be processed later. 132 00:08:38,125 --> 00:08:41,999 So once it's done, it's crawling right now. 133 00:08:41,999 --> 00:08:46,167 Once it's done, it will generate the XML and then what can be done 134 00:08:46,167 --> 00:08:51,751 with this XML is, again, feed back into the tool, which will correlate 135 00:08:51,751 --> 00:08:55,918 and find out which resources are slow in its opinion, 136 00:08:55,918 --> 00:09:00,250 and it picks some PHP resources out of that. 137 00:09:00,334 --> 00:09:04,999 And so it will start sending paralyzed requests to these resources 138 00:09:04,999 --> 00:09:07,334 and measure whether some service 139 00:09:07,334 --> 00:09:09,999 degradation happened. 140 00:09:09,999 --> 00:09:13,292 It will output, what was the original measurements 141 00:09:13,292 --> 00:09:16,542 and what were the new ones. 142 00:09:16,542 --> 00:09:19,918 So by looking at these deltas, you can figure out whether this 143 00:09:19,918 --> 00:09:22,999 is truly CPU heavy resource or not. 144 00:09:24,417 --> 00:09:29,417 So now a bit about asynchronous nature. 145 00:09:29,417 --> 00:09:34,999 So this slide shows three screens, the top one is the top utility output 146 00:09:34,999 --> 00:09:37,751 of the web server. 147 00:09:37,876 --> 00:09:40,334 The bottom right one is the client. 148 00:09:40,334 --> 00:09:44,501 You notice the client has 9, 9.5% usage, and that's mostly 149 00:09:44,501 --> 00:09:47,375 because of the recorder. 150 00:09:49,751 --> 00:09:55,999 I'm launching the tool five times to why the most heavy resources 151 00:09:55,999 --> 00:10:02,999 and the load on the client side, it was not it goes up to 9.5%. 152 00:10:02,999 --> 00:10:09,501 So negligible, but the load on the server side goes up to 80 to 82%, 153 00:10:09,501 --> 00:10:14,751 which is, you know, asynchronous in my book. 154 00:10:14,999 --> 00:10:16,167 Okay. 155 00:10:16,459 --> 00:10:18,751 So this concludes the demo. 156 00:10:18,999 --> 00:10:23,834 VAAGN TOUKHARIAN: And now that we saw the tool, let's talk 157 00:10:23,834 --> 00:10:27,375 about what can be done with it. 158 00:10:27,834 --> 00:10:31,167 You can use it for the good of the humanity. 159 00:10:31,250 --> 00:10:38,125 As a QA tool, for example, or to figure out if you are vulnerable 160 00:10:38,125 --> 00:10:44,709 to attacks described a minute ago by just going in and taking 161 00:10:44,709 --> 00:10:51,459 all of those CPU hogs and debugging them or getting rid of them, 162 00:10:51,459 --> 00:10:53,999 rewriting them. 163 00:10:55,334 --> 00:11:01,709 Or you can use it for bad which is almost what Tigran showed two 164 00:11:01,709 --> 00:11:07,834 minutes ago, where he paralyzes the resources and tries to see 165 00:11:07,834 --> 00:11:13,959 if the degradation happens, imagining that being all automated 166 00:11:13,959 --> 00:11:20,709 and one click, boom, you go find the worst resources and then do more 167 00:11:20,709 --> 00:11:26,959 analysis by figuring out which ones are the worst in real life, 168 00:11:26,959 --> 00:11:33,083 which ones degrade, and then you push them even more. 169 00:11:33,292 --> 00:11:37,999 And then, of course, there is the ugly, which is the bad, 170 00:11:37,999 --> 00:11:41,167 but distributed basically. 171 00:11:42,876 --> 00:11:49,667 So that is what we have right now and what we were thinking 172 00:11:49,667 --> 00:11:57,501 of on adding on the tool or a method is a couple of tweaks. 173 00:11:57,834 --> 00:12:01,375 One would be the measurement of degradation, 174 00:12:01,375 --> 00:12:07,083 more precise measurement of degradation, in parallel testing, 175 00:12:07,083 --> 00:12:13,834 which will go to nuances like understanding load balancers. 176 00:12:13,834 --> 00:12:18,083 We don't know how load balancers would behave right now, 177 00:12:18,083 --> 00:12:24,083 how our calculations will be affected by load balancers. 178 00:12:24,209 --> 00:12:27,999 Those things need to be worked out. 179 00:12:28,125 --> 00:12:32,209 Then we have a couple of other things that would be nice 180 00:12:32,209 --> 00:12:37,999 to add, like SQL wildcard injection into forms that we think are search 181 00:12:37,999 --> 00:12:42,459 forms, or in any way are fit to SQL query. 182 00:12:45,417 --> 00:12:52,083 And then another thing would be figuring out if state reset is pricey, 183 00:12:52,083 --> 00:12:56,417 to see if when you are forcing an application 184 00:12:56,417 --> 00:13:01,125 to generate new tokens for session ID or whatever, 185 00:13:01,125 --> 00:13:04,250 it takes any CPU load. 186 00:13:04,250 --> 00:13:09,959 That has to be calculated, estimated, or guessed, or doctored. 187 00:13:12,209 --> 00:13:14,999 And now let's talk about defense. 188 00:13:14,999 --> 00:13:20,167 Well, what can be done on the defense side? 189 00:13:20,209 --> 00:13:23,999 Load balancers are probably good measure here. 190 00:13:24,876 --> 00:13:33,751 Our guess would be mod security here, and you can use the tool to identify 191 00:13:33,751 --> 00:13:39,459 and fix memory, DBCPU hogs make your application 192 00:13:39,459 --> 00:13:46,751 work smoother and be less vulnerable to this attack. 193 00:13:47,292 --> 00:13:50,999 And slide deck has a reference number one, 194 00:13:50,999 --> 00:13:55,999 which has a link to very simple mod security setup that 195 00:13:55,999 --> 00:14:00,709 most probably going to cover 90% of cases. 196 00:14:00,751 --> 00:14:05,584 It's done by some other guys but we find it's effective. 197 00:14:05,999 --> 00:14:09,751 And then there is one more thing that Tigran thinks can 198 00:14:09,751 --> 00:14:11,292 be done. 199 00:14:11,292 --> 00:14:13,999 TIGRAN GEVORGYAN: So I'm lazy, right? 200 00:14:13,999 --> 00:14:17,250 If I can off load something to the computer, I prefer that. 201 00:14:17,250 --> 00:14:21,417 So the idea here is to design a mode security extension that 202 00:14:21,417 --> 00:14:26,667 will keep track of the website usage, meaning it will keep track 203 00:14:26,667 --> 00:14:31,542 of what resources the chain of resources that was requested, 204 00:14:31,542 --> 00:14:37,999 the response times that were witnessed and all that, and it will track and tag 205 00:14:37,999 --> 00:14:40,751 out of place requests. 206 00:14:40,751 --> 00:14:43,083 For example, if a resource is requested all the time 207 00:14:43,083 --> 00:14:46,209 through some chain of events, and this resource 208 00:14:46,209 --> 00:14:49,334 is requested immediately, that would, you know, 209 00:14:49,334 --> 00:14:52,751 warrant some special consideration. 210 00:14:52,792 --> 00:14:54,918 Same goes for state coherent checks where 211 00:14:54,918 --> 00:14:58,209 a resource is always requested with some state and now 212 00:14:58,209 --> 00:15:01,626 the request comes in without a state. 213 00:15:02,125 --> 00:15:05,292 That's an indication to pay some close attention 214 00:15:05,292 --> 00:15:07,334 to that as well. 215 00:15:11,667 --> 00:15:12,999 Yeah. 216 00:15:12,999 --> 00:15:14,417 With that, we thank you. 217 00:15:14,417 --> 00:15:16,417 VAAGN TOUKHARIAN: We are on time. 218 00:15:21,999 --> 00:15:23,584 Thank you very much. 219 00:15:23,584 --> 00:15:26,959 (Applause) Thanks to Sean for setting up the lab for us.