Next talk that we have, that we have, and I guess how the discard presentation, you know, it was ironic because when we were reading the CFPs, we were looking at the CFP and I was looking at the description and it was really eerily familiar.
It sounded really, really familiar because, does anyone remember the, not too long ago, there was a DDoS, a Distributed Analysis Service Attack against Dyn, and it took down some of the big websites back in October of 2016.
It also affected all of each and every one of my courses as well.
I write, coincidentally, two weeks before.
All that mess happened in October of 2016, which took out some of the biggest services on the internet.
I was putting together content and material to talk about Distributed Analysis Service Attacks, and I was looking.
One of my resources that was just absolutely fantastic was an IP spoofing by Marek Bajewski, and sure enough, when we opened,
when we read the abstract with, of course, the name and the title were, the name and the title were covered, once I opened it up, it was so spooky, like, sure enough, here it is.
So now, it's very coincidental, and I guess things happen for a reason, Marek, I think things happen for a reason, and that's why we're here.
Ladies and gentlemen, from CloudFirm, Marek Bajewski.
No one said my surname is easy.
All right, thank you for having me here.
First thing, the room is fairly packed, so if you have a free seat near you, can you raise your hand so people at the back, if they want to sit, they would know that there are a couple of seats in front, especially left.
So, yeah, please do grab a seat.
This is going to be a long talk.
Yep, there are still seats in front.
And also, the back is super cold, so it's much warmer here.
You're not found to be.
All right, so let me get started.
So, this is talk about IP spoofing, and I dare you, I will say the phrase IP spoofing at least 100 times.
So, if you are not in technology, you can at least count if I'm...
I will do that.
And I promise that I will not say cyber any time.
So, it's not about cyber, it's about IP spoofing.
If you look at this talk kind of from the higher level point of view, you can sum it up as how the Internet works, what's broken in the Internet fundamentally, and how we as the Internet community can fix it eventually.
That's the optimistic view.
But more specifically, this talk is going to be about IP spoofing, what it is, how it allows...
large attacks, and finally, how to fix it, so with a positive note at the end.
All right, so what is IP spoofing?
I'm sure everyone here in this room knows this is Pocket Hacking Village, but still, let's give a couple of slides of introduction, what I'm going to talk about.
So, by IP spoofing in this talk, I mean the capability of our network cards, of our devices, Internet devices, of transmitting packets over the Internet, and on our phone, on our laptop.
Everywhere, on every connected device, the network card itself doesn't know anything about the higher level protocols.
So, from the hardware point of view, you can transmit whatever the hell you want over the network, and we often forget about it.
You know, people program in JavaScript these days, and whatever, Python, and they forget that network card inherently is a device that we control.
So, you can basically transmit whatever you want over the network, and that includes the headers, all the headers.
That includes the IP header.
So, once again, fundamentally, you can put whatever data you need, you want, in the IP header.
And by IP spoofing in this talk, I mean specifically putting the fake or malicious IP address in the source IP port on the IP packet.
So, basically overriding the source IP in the packets.
And you can totally do that.
This is absolutely legitimate feature of our network stack.
So, what's the problem?
Why this is bad?
Why this is something we haven't talked about?
It's bad because it's...
It fundamentally enables impersonation.
So, from the receiving host point of view, if you ever receive any packet from the internet, basically there is no way you can find out if the packet was actually originated in the source IP it says, or was it actually fake and created by someone else, some potentially malicious actor.
Basically, there is no way from the receiving side to know that.
And that's pretty bad.
And we learned that it's pretty bad years ago.
This is a photo of Kevin Mitnick, who used to be, or still is, one of the most famous hackers in the universe, in the galaxy.
So, one of his best attacks was against Mr. Shimomura.
And in 1995, he basically took over Shimomura's TCP session and injected some commands in his TCP session, therefore granting himself root.
And he was able to do that only because he was able to spoof packets.
It was because he was able to fake the packets.
Put magical commands there and the gram can self-access.
That was in 1995.
So, quite a while ago.
Since then, IP spoofing was a problem.
And number and number and number of different attacks, different exploits, different kind of fundamentally internet issues.
In 1996, we had a wave of SYN floods, which I would say kind of the problem of SYN floods was solved only about two or three years ago.
But...
SYN flooding is a different story.
In 1998, we have my favorite attack, sorry, scanning technique called idle scanning.
This was invented by Antires, who now is working on Redis.
But if you haven't read, if you don't know what idle scanning is, I highly recommend it.
It's a super interesting scanning technique, again, based on requiring IP spoofing.
Then we had a whole bunch of problems with TCP implementations.
I think the most well-known are the ones when a malicious attacker, by forging,
reset packet, was able to drop BGP connections, so that could, that led to pretty serious problems in the internet.
And then there was another, you know, 10 years, and in 2008, we all know the famous Dan Kaminski's DNS bug, DNS issues,
which again were only triggered, could only be triggered if you could do IP spoofing.
And I would say since about 2013, we have a new wave of attacks, of the big denial of service attacks, which again are all caused by IP spoofing.
But, you know, that was on the attacking side.
What's on the defense side?
Are we making any progress as the internet community?
Is there anything happening?
Of course.
The first formal definition that, you know, IP spoofing is something, but it was around 1998.
Around 2000, there was a famous BCP-38 published.
It basically says, yep, the internet works as it works, but we should not allow end hosts to forge packets,
to send packets from the internet.
Because the IPs, they don't belong to them.
Unfortunately, BCP-38 is fairly naive.
So then BCP-84, four years later, came in and said, you know, it's harder than we thought, but still, we should do that.
And there are many working groups and many kind of initiatives that right now try to fight IP spoofing.
So there is the EETF working group called SAVI.
There is a working group in the industry.
And then there is the spoofer.ky.org project.
Which tries to track down the IP spoofing around the world.
So this is a chart from the spoofer.ky.org project.
And it basically shows that around 56% of ASs of networks on the internet do not allow spoofing.
So it's good.
Not everyone allows spoofing.
But still, the whole problem is not solved yet.
There is around 27% of ASs of networks that do allow IP spoofing.
So, you know, the problem still exists in the internet.
Even though it was first problematic in 1998.
And not only that, there is the whole industry around amplification and around attacks using IP spoofing.
So if you use your Google skills properly, you'll be able to find services like this one.
So spoofing enabled offshore servers when you can, for Bitcoin, buy bandwidth, not even bandwidth, buy servers and do whatever spoofing you want.
So this is a problem.
This is a proper industry.
This is a problem.
By the way, in the terms of service, they do tell you that if their ISP disables spoofing.
If their ISP tries to enforce it.
If their ISP tries to enforce filtering, they will not give you the Bitcoin's back.
All right.
So what's the big deal about spoofing?
Well, in my opinion, it enables the large attacks.
So if you open news outlets and read about the big, I don't know, ransom emails requiring, I don't know, banks to pay up or kind of attacks that actually were influencing some institutions.
In most cases, not all, but most cases, those big attacks, the biggest attacks will be caused by IP spoofing.
In one way or another.
Now, there is, there needs to be a disclaimer here.
I'm not, this is not about the Mirai attacks.
So we can speak about the Mirai and Dyn at the end in the questions section because they were a bit different.
But except for this case, the major attacks you've heard were almost all about IP spoofing.
All right.
So who am I?
Why do I care?
Why do I know anything about it?
So I work at Cloudflare and we operate a global reverse proxy network.
We have a very large network.
We have our servers all around the globe.
We operate on almost all continents.
We have customers from all backgrounds, from government agencies to social portals.
So, and we also have plenty of customers.
So we see quite a good cross-section of what's happening on the internet.
This is a chart of daily attacks that we categorize as denial of service attacks.
This is per day.
And you can see that some days are quite quiet with, I don't know, 50.
Yeah.
Or 60 events that we categorize as denial of service attacks.
And some of them are more busy with up to 1,400 events.
So we see a good chunk of the weird things on the internet.
And I won't lie, most of the attacks that we see are fairly small.
They are not really anything exciting.
But some of the attacks that we do see are super large, are gigantic.
And over the years, we've published a number of blog posts describing those attacks in big details.
And in this talk, I'm going to show you two major types of attacks we've seen.
And I'm going to also emphasize why they are problematic because of the IP spoofing.
All right.
So one of the attacks I'm going to talk about is called, I will call direct attack.
And the other one, it will be amplification.
So let's start with the direct attack.
So what is it?
Why do I put it in a separate category?
So this is an attack.
This is an attack when the attacker is just transmitting packets directly to us, directly to the target, without any amplification, any reflection in the middle.
It's just packets from the source, the target.
That's it.
Fairly simple.
But we cannot retrace the attacker because the source IPs of the attack are spoofed.
So we don't know who actually originated them.
About a year ago, we published a story which we dubbed the winter of attacks, in which case the attacker had about 400 gigs of capacity, of spoofing capacity.
These are the charts that we published.
They look fairly shady.
But basically we were hit with about 200, on the peak, 200 million packets per second of the attack traffic.
And in the peak as well, about 400 gigabytes per second of the attack traffic.
So it was fairly substantial.
And once again, it was direct attack.
So there exists an attacker in the world, at least a year ago, that had 400 gigabytes of spoofing capacity.
So the interesting things about this type of attack is that they are counterproductive.
usually originate in only a couple of places
and a couple of, I would say, big data
centers or on big
ISPs. So when we
see them, we see them only in a couple
of places around the world.
So that's how we notice.
Okay, so once
we see such an attack, what do we do? Well, first
we have to make sure that we are kept online,
that our servers are operating, that our
customers do not feel the problem,
that the internet still
works for all of
our customers. So what do we do?
Well, we obviously try to figure out what is hitting
us. So we run TCP dump
or some other tools and try to figure out
what is in the packets. And in this
case, those 400 gigs, this
was a SYN flood. So it was a very, very, very large
SYN flood. And
it's interesting because it had to be
SYN floods have to be direct. There's
no way to reflect SYN floods. So basically
if you see a SYN flood, this is direct attacks.
In many of the SYN floods, the
IP packets, the IP
sources are not spoofed.
So it's just flood of SYN
packets.
With a source IP address belonging to the
attacker. But in that case, it's easy to recognize
who is attacking you. In our situation,
the source IPs were obfuscated. They were
just random numbers.
Okay. So how do we keep ourselves online
when we see 400 gigs of
SYN floods? Well, we mitigate it.
And our favorite mitigation
technique is called BPF.
This is an example of IP tables command
with BPFs inside. So what is a BPF?
BPF is a bytecode. It's a
program that you can stick
inside your firewall, inside your IP firewall.
And run it within IP tables.
In this bytecode, you can read data from the packet
and you can basically have an almost
fully fledged program running inside
your firewall, which is super cool.
So on this
command line prompt, you can
see a bunch of
numbers. What are those numbers?
As I mentioned, they're a program.
Here is the same program. It's just
decompiled to more assembly-like format.
So once again,
it's almost fully fledged program.
And by the way,
if you ever run a TCP dump
or Wireshark, you should be familiar with BPFs
because that's exactly the interface
between TCP dump and the kernel.
So if you use TCP dump and type
something like port 80, this will be compiled
down to this bytecode.
All right.
So more specifically, we can't really
use the TCP dump expressions because they are a bit
too limited. So we wrote a series of tools
that we actually use to mitigate the big attacks.
They're all
open-sourced, so on our GitHub you can find
BPF tools. And that's exactly
the bytecodes that we run to
mitigate attacks, including
SYNFLATs.
Now, there needs to be a disclaimer here.
Although iptables is fast and BPFs are
super fast and it all
works reasonably well,
there are still some limitations.
iptables run fairly late
in the kernel network stack.
So you can do much, much faster by
doing some magical tricks like kernel bypass.
But I won't go there
because there's a new development in Linux
kernel called XDP.
It's an express data path. It's available from
kernel 4.9. And right now you need
to have a magical, specific network card.
Only a couple of vendors support it.
But it's super cool. It's really great.
But that's it.
I won't say much because tomorrow
2 p.m. my colleague will give a talk
exactly about XDP. So
stay tuned and be here tomorrow
at 2 p.m.
All right. So we mentioned how
we keep our services online, how
we drop the packets on the floor.
And sometimes we
want to identify who's actually behind those attacks.
We want to dig deeper into
who is actually sending the traffic,
where the traffic originates.
So
basically we need to identify,
since the source IPs are spoofed, we need to identify
from which physical cable
the packets were delivered, like where
they are coming from.
We have no number of
cables connected to our routers.
So what we do is
we log into our routers and we chase the graphs.
This is
a chart. This is a couple of charts from
one of our routers. And here you should be
able to clearly see that there are some
inbound spikes hitting
whatever 10 gigs on the
inbound, on the kind of bottom left chart.
So it's easy to identify
from which interface, from which physical cable
the attack is coming from, even though
the source IPs are obfuscated.
All right. So then we follow the wires.
Then we look on our router
carefully and we try to figure out who's on the
other side of the cable. Who's actually
sending us the traffic?
And there are generally three types
of cables that our routers have. Three types
of connectivity that we have on our routers.
This is basically how the internet works, by the way.
So the
first type of cabling is called
PNIs, direct peering, or cross-connects.
It's basically a direct connection, a direct cable
laid from our router
to someone else's router, when this
someone else is an entity similar to
our company. So it could be
Google, it could be Akamai, it could
be AWS, it could be
any other number of providers.
So basically
another entity on the internet.
The second type of cabling that we
have is a cable that goes from our router
to local internet exchanges.
I will speak more about
internet exchanges, but basically internet exchanges
are a collaboration of
local ISPs, and it's a single place
where they all connect. And finally
there is the connectivity to the
transit providers or internet carriers
where basically it is paid for traffic.
We pay for
to deliver packets to the internet. So it's kind of a
normal connectivity to the internet.
Alright, so let's imagine that the attack
came from direct peering. So what do we do in this case?
What do we do if we know that, you know, the attack
packets came from this specific cable
which has this another entity on the
other side? So it's easy. It's easy because
you can just call up them and say, you
know, guys, you're delivering us junk, please
don't attack us. Please don't do that.
And the good news is that this usually
works. This usually works well. So the
relationship between us and other
part that we cross-connect to is usually
very friendly. It's in both our interests
to, you know, to keep the relationship
okay. We do have sometimes situations
when the ISP on the other side does
respond to our calls and say, okay, I
will kick off the customer that's actually
attacking you. And then the problem
reappears couple of days or weeks later.
But this is fine. Like, we can call them
again and in the worst case we can just
de-peer them and force the attack traffic
to go over the internet and increase
their costs. So basically there is
some discussion here. So that's great.
So what about the other two options? What if
the attack comes over the internet
exchange or what if the spoofed attack comes
over internet carrier and transit
providers? Unfortunately in this case we
cannot do much. So this is a pretty sad
story here. So let me explain to you why
this is the case. Okay, so let's focus on
internet exchanges for a while. So this is a
picture of, I think, Seattle internet
exchange. It's basically a gigantic layer
to switch with plenty of cables in, with
local internet providers connected. And
so what's the issue? The issue is that
internet exchanges are layer 2 switches, so
they are ethernet switches, while
routers are layer 3 beasts, layer 3
entities. And from router point of view,
router is there to route packets. So it
takes packets, it throws away the
ethernet frame as quickly as it can.
That's its sole job. It's to route packets
on the IP layer. And then it routes it back
to our data centers. And that's an issue.
Because from the router point of view,
all that router knows is that, yep, there
are packets, they are DAC packets, they are
coming from internet exchange. That's all
I know. I don't know anything else. So
basically we can't really figure out
which of the local ISPs, which of the local
connected parties is actually originating
traffic. Now if you think about it, this
data is there because it is in the
ethernet frame.
The source MAC address usually should
point out to the router that is
belonging to the entity that is sending
the junk traffic. But unfortunately, as I
mentioned, router doesn't, our router
doesn't really look at that because it's
on a different layer. So this is pretty
sad.
Okay, so what's in the third case?
What happens if the attack comes over
internet carriers or transit providers?
What can we do in such a case? Well, we
often call them up and say, you know guys,
you're delivering us packets, which are
just junk, which are just invalid packets.
And they say, well, it's because you paid
us to do that. And that's true, like the
internet carriers are there to deliver
packets. We paid them exactly for this
job. And by the way, I don't think that
it is their job to filter the
packets. It's censorship, it's not really
their job. So it's kind of fine, but
still, they should be able to do
something about it. They should be able
to inspect their network and figure out
which of their customers is actually
originating the traffic. Unfortunately,
that's not the case. They can't really do
that. So again, pretty sad story. We see
packets flying in from our internet
carrier. We have no idea who is actually
originating them. So that all leads to the
conclusion that basically, in most cases,
tracing back these spoofed packets is
impossible. And that's pretty bad.
And that's pretty bad because we cannot really
push back against it. We cannot really
figure out which data
centers are evil, which IPs are evil, because we
just don't know.
Okay, so let's deep and deep
dig into a bit more details into the
attacks I showed, which is the winter of
attacks, which I mentioned already. I can
actually share a couple of more details
about this specific attack. So this
attack was on the third category. This
attack, this 400 gigs, was coming to us
over paid-for internet carrier. And
in this specific attack, the source IP
addresses were belonging to an AS
number belonging to Hurricane
Electric. Hurricane Electric is a big, a big
internet carrier. They are one of the more
competent ones. But anyway, from what we
knew, all we knew is that we see attack
traffic, we see SYN flood, and the SYN flood
comes from IP ranges belonging to
Hurricane Electric. Okay. That's pretty bad.
But then we noticed that actually the
attack was coming to our Los Angeles
data center. And in Los Angeles we
actually have a direct connectivity to
Hurricane Electric. So that leads to two
conclusions, to two options. One option is
that, okay, the attack was actually coming from Hurricane Electric.
you know, it is possible, but in that case, it should be delivered over direct link to us.
It shouldn't be delivered over paid for internet.
So maybe they misconfigured the routing.
If so, they should fire their router admins because they just paid for a big attack traffic.
So let's say that's not the case.
So the other interpretation is, of course, that our internet carrier has a client, has a customer,
who is spoofing packets and pretending to be Hurricane Electric.
But this is a nice situation, a nice explanation that our internet carrier actually doesn't know,
does not know if the packets are valid or not.
From the internet carrier's point of view, these packets are originating in Hurricane Electric,
and that can be okay, that can be fine.
It's just we know that that wasn't the case in this situation.
Alright, let me show you a couple of more examples of similar attacks.
But before I show you more details, we need to draw, we need to have vocabulary to speak about those attacks,
those IP spoofing attacks.
So this is how we are going to show it.
So this is a picture from XKCD by Randall Monroe from about 10 years old.
And he attempted to draw a map of the internet.
It's not easy.
So what he did is he used a mathematical thing called Hilbert curve.
And the Hilbert curve is a way to draw a line over two-dimensional space that will kind of group things together.
So in each of the squares you can see here, there is a slash 8 subnet.
So you can see top right corner, there are some reserved blocks.
Again, kind of top right corner, there is a multicast group.
So basically all these slash 8s are kind of close together in a square.
If you look at slash 16s, they will also be in a smaller square and so on and so forth.
So it's a pretty cool way of visualizing IP space, basically.
Okay, let me show you one of the attacks.
So this is a Hilbert curve drawn by me.
You can see some hashed out blocks.
These blocks are basically unroutable IP addresses.
And in the middle there is 127 slash 8.
So kind of reserved look back IP range.
Okay, so in this attack for each of these source IPs that we saw, there was, I drew the black dot.
Okay, so plenty of IP addresses scattered all around.
Is that interesting?
What's exciting about this?
Well, not much until you realize that this is,
these are IP ranges.
These are IP addresses belonging to China Telecom.
So again, this is the attack.
This is China Telecom.
Attack, China Telecom.
So the interpretation of this is fairly hard.
Like, I'm not saying China Telecom attacked us.
I'm saying the IP addresses belonging to China Telecom were used in the attack.
And from my point of view, it's super hard to actually figure out what's happening behind the scenes.
So there are two interpretations possible.
One interpretation is that someone has a large botnet in China.
I know it is totally possible that someone has a gigantic botnet in China and is just using it to attack us.
That's fairly sane. It's possible.
In such case, the IP addresses will not be spoofed.
They will be properly legitimate IP addresses belonging to malicious bots in China.
On the other hand, if you look at that again, they seem to be quite well distributed.
Again, I'm not an expert.
Maybe China Telecom IP addresses are distributed evenly around the China population.
I don't know if that's the case.
It's possible.
It could also be the case that someone has just went through the IP allocations and is just spoofing the China Telecom IP addresses.
There is no way to figure it out.
Okay.
So another example of an exciting attack.
This one.
This is my favorite.
So in this case, you can see that the source IPs are just uniformly distributed across the whole IP space.
Including the reserved blocks.
Including 255.
So the top one, including 0.
So the bottom one.
Including reserved blocks like 127.8.
So if you ask me, do I see attacks from loopback?
Yes, absolutely.
Every second of the day, I see malicious packets coming from 127.01 attacking my network.
So, you know.
There is that.
There are then variations on top of that.
So, for example, this is a very similar chart with uniformly distributed source IP addresses across the whole IP space.
But with the reserved blocks skipped.
Skipped.
Now, the question is, why did it happen?
Was it the attacker that is smart?
And, you know, when they have the routine that is forging the source IP addresses.
Is the attacker just, you know, filtering the IP addresses and say, okay, there is 0.
So maybe let's not put the obviously wrong ones.
But I don't think that's the case.
So I think what's actually happening is that their ISP is filtering the traffic.
But again, there is no way for me to figure it out.
There is next one.
Next very uniform distribution.
Why this particular one?
This particular pattern I have no idea.
Maybe someone will know.
And another one.
This may again look like a fairly nice either gigantic botnet from all around the world.
Or just someone spoofing some weird stuff with a bit more density and kind of the lower IP ranges.
But no.
If you look at this particular chart carefully.
You would notice.
If you are a network expert.
You would notice that those dots correspond exactly to the nodes.
They are non-routable IP ranges.
So in current internet.
Almost every IP address belongs to someone these days.
But there are still plenty of IP blocks that are not routable.
That don't exist in the global BGP routing table.
So in this case.
The attacker was fairly sophisticated.
The attacker went for the global BGP routing table.
Inverted it.
So figured out, okay, so what blocks are not present there.
And only spoofed these.
Why such?
I don't know.
But here you are.
All right.
So that's all I have on the direct attacks.
When I actually can see the source IP addresses.
I can actually argue about it.
Again, I can't really track it back.
To anyone.
But at least I see them.
But then there is a second category of attacks.
Called the amplification attacks.
About two weeks ago.
Three weeks ago.
We published a story about SSDP amplification.
These charts are not as exciting.
But basically they show that the attack we received.
To our network.
Was of about 10.000.
About 40 million packets per second.
And the volume was about 110 gigabytes per second.
So you know.
A sizable attack.
Okay.
So what is an amplification attack?
Just a couple of slides on it.
The general idea is that.
There exists UDP servers in the world.
UDP protocols in the world.
With a basic request response protocols.
And the idea of those protocols.
Is that the client creates a request.
And the server responds.
That's it.
It's DNS and it works exactly like that.
You know that because NDP works exactly like that.
And a number of other protocols work in the same fashion.
So what's the problem?
Well the problem is that.
If you can spoof packets.
You can forge the request.
So you can convince a legitimate server.
To answer to a request.
That wasn't really asked in the first place.
And therefore you can cause the server.
Legitimate server.
To trigger legitimate answer to a target.
And that way you can basically.
Amplify the attack.
So the main idea is that.
In many of those protocols.
The responses are much much much larger than the requests.
For example in DNS.
In DNS the request is usually small.
And the response is usually much larger.
And if that's the case.
If this is the case in the UDP protocol you're using.
You can create an amplification.
Of a big size.
So in this case it's like 10 times.
But it's not uncommon to see 20 times.
Or 80 times amplifications.
And this is great from the attacker point of view.
Because you know.
Possessing one gigabit per second of attack.
Of spoofing capacity.
You are able to generate much much much larger.
Attack volume.
Going to the target.
Also you never use a single UDP server.
You have to scale it out.
There is no single UDP server.
That can generate too much traffic.
So that's what we saw those couple of weeks ago.
So what we saw was.
An attack which again.
On our side had about.
12 gigabits per second.
There were about 490,000.
Reflectors used.
So a sizeable chunk.
A big number of reflectors.
And we believe that the attacker.
Only had about 5 gigs of spoofing capacity.
That's what we can know from.
Looking at the protocol deeper.
Alright.
So again what do we do in case of session attack.
Well the first thing as always.
Is we try to keep ourselves online.
So how do we do that.
There are a number of techniques we are using.
But once.
The most interesting one is.
That we are.
This attack is nicely distributed geographically.
This is because all of those reflectors.
All those 940,000 reflectors around the world.
They don't belong in a single place in the world.
They are scattered all around the globe.
And this is good from our point of view.
Because the attack traffic.
Is nicely delivered.
To all our data centers.
We can do that because.
We run an anycast network.
So we advertise the same IP ranges.
All over the world.
Therefore the attack packets will be delivered.
To the closest data center.
That is to the reflectors servers.
So again.
Our architecture allows us.
To basically split the traffic very evenly.
Across our data centers.
Okay.
So what do we do next.
Well blocking these attacks is usually very very simple.
These attacks usually happen.
Usually come from a specific protocol.
So if it's SSDP.
The servers will always send packets.
From port 1900.
So just dropping port 1900.
With the protocol UDP.
On their firewall will do the job.
So it's really fairly simple.
It's a bit harder for things like DNS.
Because you can't really drop all DNS traffic.
But still fundamentally.
That's a fairly simple thing.
Okay and how do we identify the sources.
How do we know who actually is attacking us.
And the answer is that we don't.
And we cannot do that.
And no one can do that.
So the packets that are delivered to us.
Are legitimate.
The Reflector servers deliver us responses.
The requests we never asked in the first place.
But still the packets were legitimate.
The source IPs are absolutely correct.
So if you wanted to trace it back.
We would need to ask the internet.
Dear internet.
Who in the internet forged our source IP addresses.
And there is just no.
You just can't ask this question.
Alright.
So what about the amplifications.
So in the other blog post.
That we published.
We mentioned a couple of other protocols used.
But these are usually the usual suspects.
So it's usually NTP.
SSDP is the biggest one we saw.
There is also DNS.
And a couple of other protocols.
And most interestingly.
There are a couple of gaming protocols.
So I think we will see a new interesting attacks.
That are using gaming protocols.
Like Call of Duty.
Or Source.
Because.
These protocols are not audited well enough.
On the attack sizes.
The amplification attacks.
Are usually fairly small.
So the average size of the attacks.
We noticed in last six months.
Was about 7 gigabits per second.
So you know.
Not gigantic.
Sizeable for small internet players.
But you know.
Not gigantic.
The maximum was 100 gigs.
As I mentioned.
This is a bit outdated.
Up to 44 gigabits per second.
Or NTP attacks.
I think the maximum NTP one we saw.
Was 64 gigabits.
Just over last six months.
Alright.
That's almost it.
So basically.
There are kind of three themes in this talk so far.
One is that IP spoofing is generally bad.
And it allows all kinds of problems.
And most importantly.
The big attacks.
The second thing is that.
In order to survive the big attacks.
It's pretty unfortunate.
Because that means that only big internet players can survive.
The largest attacks.
And also the third one is that fighting against those attacks.
Is basically impossible.
There is no way to trace the attacker.
There is no way to put pressure on the attacker.
We just don't know who is on the other side of the wire.
Okay.
So how the internet can fix it.
How we can influence our colleagues.
How we can buy bandwidth.
What can we do as internet community to fix it eventually.
Okay.
So let's tackle it step by step.
So first.
We have to understand.
That these attacks are possible.
Only because of IP spoofing.
We would not speak here about amplifications.
If it wasn't about IP spoofing.
I wouldn't speak about SYN floods.
If it wasn't about IP spoofing.
Yes.
SYN floods would have existed.
But I would be able to just sue or put pressure on the people.
That are actually sending me the malicious traffic.
So what can we do about IP spoofing.
Well fundamentally we can't really do much.
Because that's how the internet works.
But we should continue the usual efforts.
We should continue promoting BCP38.
And basically promote good network practices.
These things are not dead.
We just need to spend more time on it.
But we can do even more things.
There is this spooferkaida.org project.
Which I mentioned already.
They give you.
If you want you can download an application.
Which will run in background.
And basically test network that you connect to.
For IP spoofing.
This is pretty cool.
So you can basically as you go around the world.
You can basically see what networks are.
Allows spoofing to what degree.
And finally.
We vendors can do a lot.
For example right now the RPF.
So reverse path filtering.
Sorry forwarding.
Is not enabled by default.
If you buy a new router.
It should be enabled by default.
There are reasons why you may want to disable it.
But first it should be enabled.
But again we have to appreciate.
That actually doing filtering is hard.
Filtering must be done close to the source.
So it's basically the anti-ISPs responsibility.
And you know many ISPs.
As I showed 56% of the ISPs.
Already fixed their networks.
But there are still plenty of ISPs around the world.
Which just either don't know that this is a problem.
Or don't care.
So we will be left for next years.
With the incompetent ISPs.
So we will be left with the problem.
And we will be left with new strategies as well.
As we go.
Okay.
The second thing is network capacity.
What can we do to avoid.
Everyone to have gigantic pipes.
In order to intake.
The amplification attacks for example.
And here I believe.
We have a technological technical solution.
Which is called BGP FlowSpec.
I'm a big fan of BGP FlowSpec.
BGP FlowSpec is basically.
A protocol created on top of BGP.
Which allows.
You to serve firewall rules.
Up the routing chain.
So basically.
Your router.
Over a BGP session.
Can tell to our routers.
By the way.
Here are my firewall rules.
I don't want some of the traffic.
I don't want the SSDP traffic.
I want the amplification traffic.
Drop it as close as possible.
And this is great.
Because it allows you to relieve the congestion.
On the last link.
We had a couple of years ago.
We had a number of outages caused by that.
I mean sorry.
FlowSpec worked perfectly.
It dropped all the attack traffic.
100% of the attack traffic.
Unfortunately it also dropped.
Normal traffic.
But these are technical problems.
And I'm sure they are fixable.
And I'm sure they are all fixed already.
This particular outage was caused.
By us putting IPv6 IP addresses.
In FlowSpec rules.
Which was not supported at the time.
And then the FlowSpec shines.
Because it can cross AS boundaries.
It's a firewall language.
Which can basically cross ASs.
So that's again great.
You can basically move your rules up the chain.
Now.
Again.
I won't lie.
Most people that use FlowSpec.
Use it within one domain.
Within their AS number.
It doesn't cross the boundary.
So it's not as useful.
But I think it is.
It should be deployed.
Kind of as the cross AS thing.
And then there is.
At least one success story.
In Russia.
So Ruscom I think a year ago.
Created a service.
Which basically enables their customers.
To send FlowSpec firewall rules.
Up the chain.
So there is at least one internet carrier.
That does this thing right.
So it is possible.
So next time.
You choose your bandwidth.
Vendor.
You choose who to buy the bandwidth from.
Ask about FlowSpec.
Okay.
And finally.
What about tracing bug.
Can we ever find out who is behind the attacks.
Can we know that.
The answer is yes.
And again.
There is a technical solution.
Which is called NetFlow.
And again.
We use routers.
To send packet samples.
Actually not packet.
But flow samples.
To a central location.
But this time within one AS number.
Within your domain.
So it doesn't cross.
So there is no privacy issues.
To cross data across ASs.
And this data can be collected.
And can be analyzed.
And with this data.
If configured properly.
With good tool chain.
Then your customers actually.
Originated the traffic.
Who is actually behind it.
Who is actually being paid money.
To deliver the spoof traffic.
So yeah.
So we could just call them.
In such a case.
And ask.
And they will just run a simple command line tool.
And basically be able to answer those questions.
But once again.
The point is right now.
It's impossible.
The ISPs do not support FlowSpec.
They don't say that okay.
Using NetFlow and basically logging traffic.
On disk ISPs is bad.
For the privacy.
This is true.
I'm basically asking for more logging.
But there are ways around it.
There are ways to find a compromise.
So for example.
I'm only interested in the really, really, really.
Big, big, big attacks.
I'm only interested in the gigantic spikes.
Which threaten the stability of the internet.
I really don't care about your customer connections.
So you can set the sampling rate.
Or largest depending how you look at that.
Maximum value.
So which is one I think in 64,000 of flows.
So unless you are sending 64,000 of flows.
A second.
Your privacy should be reasonably preserved.
And second thing is.
I care only about recent attacks.
I care only about last whatever.
48 hours of traffic.
So the logs can be aggressively rotated.
This is fine.
Okay.
So there is a tangent.
That it is a bit harder for the internet exchanges.
But again there are technical solutions to that.
One of the solutions is called.
Mug accounting.
Which is a feature in our routers.
Which allows us to do something.
And get more statistics about the mug addresses.
On that network.
And the second thing is.
Is that internet exchanges.
So this kind of no man's land between ISPs.
Should play more active role in guarding the internet.
Internet exchanges should have.
Should provide tools to figure out.
Where the attacks are originated.
Okay.
All right.
That's it.
So a quick recap is.
In order to prevent the IP spoofing.
We have to promote BCP38.
This is the root of the actual problem.
But unfortunately it will not fix it in short time.
In the meantime.
We should definitely promote flow spec.
Because this allows smaller players.
To avoid buying gigantic network capacity.
So basically this will.
Enable smaller players to survive big attacks.
And finally.
Net flow is necessary.
To figure out who is behind attacks.
To trace them back.
And basically solve the problem.
By just putting pressure on the attacking parties.
Thank you very much.
That's it.
That's all I have.
Actually.
I'm lying.
I have 30 more slides.
With more technical details.
But I think we should stop here.
Thank you.
Anyone have any questions?
Please.
You're welcome.
Take the microphone.
Anyone?
I have a t-shirt.
To give away.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
have probes, and they run probes on the internet. I think nowadays they run their software on
your machines, so if you install their piece of software, it will basically probe the network
and upload the data. There are some privacy concerns because they don't want to reveal
your IP address, so there's kind of a bigger subject. But basically they do some active
probing. But how, what specifically spoofing they are talking about, is it spoofing within
the IP ranges of this ISP or globally, I don't know, but it's there. So spoofer.kaida.org
is the data source here.
Just curious, so the bandwidth attacks, the DDoSes get the big numbers, do you have any
idea, like relation, usually the app falls first, right?
Okay, the question is about...
What is the app falling first? Yes, in this talk we spoke specifically about layer three
attacks with spoofed IP addresses, so a very specific niche of the attacks. There are tons,
tons, tons of more attacks, L7 attacks. Interesting tangent, for example, is DNS attacks. So
we at Cloudflare operate DNS, and DNS is connectionless, it's UDP, so we see packets which are kind
of both L7 and both L3 because they are spoofed, but on the other hand they are legit.
So we're like, do we use DNS packets? And yes, the answer is usually yes. Our DNS application
will never be able to do 100 million packets per second, which we often see on DNS layers.
Sorry, we used to see. So yeah, so for that we are using raise limiting on IP tables
layer, but again, BPFs is the main methodology. So for the connectionless stuff, for the UDP
stuff, it's usually fairly easy to block and you see our DNS, sorry, BPF tools. For DL7
things on TCP, it's completely different subject. And you can see that the connectionless stuff,
and you always see the proper IP address there. So if the worst come to the end, you can
always block on the IP addresses. So it's way easier to mitigate. You can just blacklist
the whole internet. I mean, that was actually kind of touched on my question, which was
some of the stuff that was hitting TCP services on layer 7. Could you look for connections
that were not being set up properly and just blacklist them using some sort of automated
The question is, does the problem exist on TCP? No. If there is a TCP connection handshake,
then I know the other entity IP address, so.
Well, you had showed a TCP dump where you were showing connections on port 80.
These are SYN packets. So SYN flats are, it's the first packet in the connection handshake.
So SYN flats are fairly normal thing. Like everyone knows these deal with SYN flats. So it's
basically a layer 3 attack. So it's not a layer 7 attack then on SYN flats.
Sorry, but I mean, could you do...
Can you do stuff to mitigate based on, you know, sanity checks, whether or not the incoming
data was representing a legitimate user versus... Do you understand what I'm trying to ask?
Not really, but if you are asking about the details on the SYN packets, then again, BPF tools.
BPF tools is what we use for...
I'm wondering what I can do at an app layer.
Okay, so the app layer. So what about the app layer?
Higher level.
Okay. Can I ask myself a question?
So the Dyn attacks. I mentioned this talk is not about that. So from my point of view,
it's not super sure what were the Dyn attacks. They were the biggest published attack on
the internet with 620 gigabits per second. So why haven't I spent more time on that?
So first is that it's not obvious what exactly was hitting... It was Krebs before Dyn. So
it was hitting Krebs, this particular attack blog.
But from what we do know is that it was GRE packets, which... Without spoofed IP addresses.
And also what we do know is that the attack had large post uploads. So it is possible
this attack was not spoofed. So that's why I haven't mentioned it here. So yes, there
are... Sometimes it happens that there are big attacks which are actually from legitimate
IP addresses. But then I would argue it's fairly easy to block them.
Thank you very much.