Good day. My name is Van. And here is Tigran. Hi. My name is Tigran Gevorkin.
We are going to talk about HTTP time bandit. But before we start our talk, I have one thing
to ask. This is our first talk at DEF CON. So don't be nice to us. Be very, very, very
gentle to us. Please. Thanks. So about us. Tigran and I work for Qualys. We break stuff
for farm. Sometimes we fix them, too. We are interested in time travel. And we'll talk
about that a bit later. And we both love to try. We do triathlons. So what we're going
to talk about.
The presentation, short but effective maybe. You will judge by leaving the room or not.
It's going to be about another application there, DOS method. And we'll then show you
the tool. We'll go through some stats. And we'll talk about what are good or bad uses
of the tool or method. And then we'll glance over the defense.
Defends against this method. Why? Why do we do this? It's obvious because we wanted to
have time machine. We couldn't buy any on eBay. We could, but it was similar to what
you see on the screen. And that really doesn't work. So we thought we would make our own
time machine. And I think we succeeded.
machine that we have does one thing and does it well. It can give you the future state
of your web server given the resources you're going to push on it. About it a bit later.
So we're going to talk about application layer, DOS tool or method. For us, those tools in
application layer are of two different kinds. There are some that are not that smart and
another group that is a bit smarter. Non-smart approach is just pushing the resource over
and over and over and over, hoping that the resource will die.
And then you do that from multiple instances and it works. But that doesn't have feedback
sometimes and load is near symmetrical. Smarter bots like things like slow lorry, slow HTTP
test, PKI abuse and whatnot, they are smarter and they are asymmetrical in nature. Our proposal
method is going to be striving to be asymmetrical. So how do we do this? So the point of this
tool, the information from future, is picking the resources out of all the available ones,
picking the ones that are most painful for the server. While people say servers don't
feel pain, we all know they do. So how do we do this? We spider the web
site, we collect a compiler list of the resources and then we start measuring certain things,
for example, the download speed or the start to download time. And what this gives us is
a lot of data. And what deals very well with a lot of data and that's statistics. So we
use statistics to filter through the data. We calculate the average of all the download
times. That gives us a baseline, what is a normal transfer time.
And then we can tell, oh, these resources are slower or faster.
So we used to formalize this, we used the ‑‑ we used mean as the measure of central tendency
and we calculate the standard deviation. The resources that have high standard deviation,
that means that they are inaccurate in terms of they can be fast or they can be slow, but
‑‑
we can't really tell. So we throw them away. We select resources that have slower download
times or start to download times. And we think that this is related to CPU load. And
the reasoning is that all the transfer delays are the same for all the resources, the network
related ones. And so if some resource has higher transfer time, that must be resource-specific.
And that makes it interesting. So this, for example, is a chart of resource
download for a website. And you can see that the majority of resources fall close together
in some middle range. There are some outliers, some fast outliers, and some slow ones. And
the ones that are interesting are the slow ones. The question is, what causes these
resources to be slow? Is it the CPU load, or is it just a slip somewhere? On the ‑‑
on the page? So a small demonstration, and not to anger the demo gods, we did a video.
. Okay. So here the tool is running against
the locally hosted web server, and when it's done, it generates a list of URLs with the
measure mean, the standard deviation. While this is nice, it's not very, very useful.
Instead, the tool can be instructed to output all this information in an XML. And it will
So here we see the resources sorted by their download speed in increasing order.
So the resources, the HTML resources down at the bottom are the fast resources.
And the PHP resources are near the top.
And this is because PHP requires the PHP interpreter invocation which takes some additional time.
And the resource at the top is some artificially made Pi calculation which understandably takes
a long time.
And so XML is very nice because it can be fed into some other tools.
And I have one example which is an XSLT that transforms this information into something
that can be visualized.
So you'll see a scatter chart of download speed and standard deviation.
And there are four quadrants here.
The top quadrants, two quadrants, they are the resources that have ‑‑ that are the
highest standard deviation which means that the measurements were inaccurate.
So since we cannot trust these resources, we throw them away.
Instead we concentrate on the resources on the bottom half.
On the right side, these resources are fast resources and consistently so.
So they are not bottlenecks.
Instead we concentrate on the resources on the lower left quadrant.
And closer to the zero, zero point, the better ‑‑ I mean, the painful the resource would be for
the server.
So the tool also can be ‑‑ can generate data for visualization in other means.
For example, the next one, whenever it comes, it will be ‑‑ the tool is instructed
to generate graph this is dot language graph.
So here you can see a small website with transitions and each transition is marked with the transition
time.
So slowly resources could be highlighted.
I didn't do a good job here but the capability is there.
So next, the tool is going to scan a real website built with concrete CMS.
And so we give it the URL to start with, how deep into the website to go into and how
many measurements for each resource to take.
And write all that into concrete.xml.
Okay.
Which will be processed later.
So once it's done, it's crawling right now.
So once it's done, it will generate the XML.
And then what can be done with this XML is, again, feed it back into the tool, which
will correlate and find out which resources are slow in its opinion.
And it picked some PHP resources out of that.
And so it will start sending parallelized requests to these resources.
And measure whether some service degradation happened.
And it will output what was the original measurements and what were the new ones.
So by looking at these deltas, you can figure out whether this is truly a CPU heavy resource
or not.
So now a bit about this asynchronous nature.
So this slide shows three screens.
The top one is top.
The top utility output of the web server.
The bottom right one is the client.
So you can notice that the client has about nine, nine and a half percent CPU usage.
And this is mostly because of the recording software.
And the server has zero.
So on the bottom left, I'm launching the tool five times to request the most heavy resources.
And the load on the client side, it was nine.
It goes up to nine and a half percent.
So negligible.
But the load on the server side goes up to 80, 82%, which is, you know, asynchronous
in my book.
Okay.
So this concludes the demo.
Now that we saw the tool, let's talk about what can be done with it.
You can use it for the good of the humanity.
As a QA tool, for example.
Or.
To figure out if you are vulnerable to attacks described a minute ago.
By just going in and taking all those CPU hogs and debugging them or getting rid of
them, rewriting them.
Or you can use it for bad, which is almost what Tigran showed two minutes ago, where,
here.
He paralyzes the worst resources and tries to see if degradation happens.
Imagine that being all automated.
One click.
Boom.
You go find the worst resources and then do more analysis by figuring out which ones are
the worst in real life, which ones degrade, and then you push them even more.
And.
And then, of course, there is the ugly, which is the bad, but distributed, basically.
So that is what we have right now, and what we were thinking of on adding on the tool
or method is a couple of tweaks.
One would be the measurement of degradation, more precise measurement of degradation in
parallel testing, which will go into nuances like understanding load balancers.
We don't know how load balancers would behave right now, how our calculations will be affected
by load balancers.
Those things need to be worked.
Okay.
Worked out.
Then we have a couple of other things that would be nice to add, like SQL white card
injection into forms that we think are search forms, are in any way are fed to SQL query.
And then another thing would be figuring out if state reset is pricey to see if when you
are forcing an application.
If you are forcing an application to generate new tokens for session ID or whatever, it
takes any CPU load.
That has to be calculated, estimated, or guessed, or doctored.
Now let's talk about defense.
What can be done on the defense side?
Load balancers are probably a good measure here.
Yeah.
Yeah.
So our guess would be nonsticky mode would work here.
And of course use the tool or similar tools to identify unfixed memory, DB, CPU hogs,
make your application work smoother, and be less vulnerable to this attack.
And slide deck has a reference number one, which has a link to a very simple code that's
simple mod security set up that most probably going to cover 90% of cases. It's done by
some other guys, but we find it's effective. And then there is one more thing that Tigran
thinks can be done. So I'm lazy, right? If I can offload something
to the computer, I prefer that. So the idea here is to design a mod security extension
that will keep in track of the website usage, meaning it will keep track of what resources,
the chain of resources that was requested, the response times that were witnessed and
all that. And it will track and tag out of place requests.
For example, if a resource is requested all the time through some chain of events and
this resource is requested immediately, that would, you know, warrant some special consideration
consideration. Same goes for state coherence checks where a resource is always requested
with some state and now a request comes in without a state. That's an indication to play
and pay some close attention to that as well. With that, we thank you.
We are in time. Thank you very much.
And thanks to Sean for setting up the lab for us.