1
00:00:00,000 --> 00:00:18,260
Next talk that we have, that we have, and I guess how the discard presentation, you know, it was ironic because when we were reading the CFPs, we were looking at the CFP and I was looking at the description and it was really eerily familiar.

2
00:00:18,260 --> 00:00:39,560
It sounded really, really familiar because, does anyone remember the, not too long ago, there was a DDoS, a Distributed Analysis Service Attack against Dyn, and it took down some of the big websites back in October of 2016.

3
00:00:40,180 --> 00:00:44,680
It also affected all of each and every one of my courses as well.

4
00:00:45,040 --> 00:00:48,240
I write, coincidentally, two weeks before.

5
00:00:48,260 --> 00:00:55,780
All that mess happened in October of 2016, which took out some of the biggest services on the internet.

6
00:00:56,260 --> 00:01:06,160
I was putting together content and material to talk about Distributed Analysis Service Attacks, and I was looking.

7
00:01:06,320 --> 00:01:18,040
One of my resources that was just absolutely fantastic was an IP spoofing by Marek Bajewski, and sure enough, when we opened,

8
00:01:18,260 --> 00:01:31,840
when we read the abstract with, of course, the name and the title were, the name and the title were covered, once I opened it up, it was so spooky, like, sure enough, here it is.

9
00:01:32,160 --> 00:01:41,820
So now, it's very coincidental, and I guess things happen for a reason, Marek, I think things happen for a reason, and that's why we're here.

10
00:01:42,080 --> 00:01:46,120
Ladies and gentlemen, from CloudFirm, Marek Bajewski.

11
00:01:53,480 --> 00:01:55,740
No one said my surname is easy.

12
00:01:56,540 --> 00:01:58,400
All right, thank you for having me here.

13
00:01:59,020 --> 00:02:11,140
First thing, the room is fairly packed, so if you have a free seat near you, can you raise your hand so people at the back, if they want to sit, they would know that there are a couple of seats in front, especially left.

14
00:02:12,180 --> 00:02:13,800
So, yeah, please do grab a seat.

15
00:02:13,940 --> 00:02:15,240
This is going to be a long talk.

16
00:02:21,780 --> 00:02:23,320
Yep, there are still seats in front.

17
00:02:26,060 --> 00:02:29,640
And also, the back is super cold, so it's much warmer here.

18
00:02:31,060 --> 00:02:32,140
You're not found to be.

19
00:02:35,880 --> 00:02:37,780
All right, so let me get started.

20
00:02:37,960 --> 00:02:44,380
So, this is talk about IP spoofing, and I dare you, I will say the phrase IP spoofing at least 100 times.

21
00:02:44,800 --> 00:02:48,220
So, if you are not in technology, you can at least count if I'm...

22
00:02:48,260 --> 00:02:49,180
I will do that.

23
00:02:49,680 --> 00:02:53,620
And I promise that I will not say cyber any time.

24
00:02:53,880 --> 00:02:55,720
So, it's not about cyber, it's about IP spoofing.

25
00:02:57,280 --> 00:03:09,320
If you look at this talk kind of from the higher level point of view, you can sum it up as how the Internet works, what's broken in the Internet fundamentally, and how we as the Internet community can fix it eventually.

26
00:03:10,120 --> 00:03:11,300
That's the optimistic view.

27
00:03:11,860 --> 00:03:18,060
But more specifically, this talk is going to be about IP spoofing, what it is, how it allows...

28
00:03:18,260 --> 00:03:22,260
large attacks, and finally, how to fix it, so with a positive note at the end.

29
00:03:23,540 --> 00:03:25,180
All right, so what is IP spoofing?

30
00:03:25,280 --> 00:03:33,740
I'm sure everyone here in this room knows this is Pocket Hacking Village, but still, let's give a couple of slides of introduction, what I'm going to talk about.

31
00:03:34,160 --> 00:03:47,880
So, by IP spoofing in this talk, I mean the capability of our network cards, of our devices, Internet devices, of transmitting packets over the Internet, and on our phone, on our laptop.

32
00:03:48,260 --> 00:03:54,960
Everywhere, on every connected device, the network card itself doesn't know anything about the higher level protocols.

33
00:03:55,500 --> 00:04:01,380
So, from the hardware point of view, you can transmit whatever the hell you want over the network, and we often forget about it.

34
00:04:01,500 --> 00:04:08,900
You know, people program in JavaScript these days, and whatever, Python, and they forget that network card inherently is a device that we control.

35
00:04:10,180 --> 00:04:16,560
So, you can basically transmit whatever you want over the network, and that includes the headers, all the headers.

36
00:04:16,740 --> 00:04:18,020
That includes the IP header.

37
00:04:18,260 --> 00:04:22,400
So, once again, fundamentally, you can put whatever data you need, you want, in the IP header.

38
00:04:22,760 --> 00:04:31,920
And by IP spoofing in this talk, I mean specifically putting the fake or malicious IP address in the source IP port on the IP packet.

39
00:04:32,120 --> 00:04:36,120
So, basically overriding the source IP in the packets.

40
00:04:36,840 --> 00:04:38,060
And you can totally do that.

41
00:04:38,540 --> 00:04:41,260
This is absolutely legitimate feature of our network stack.

42
00:04:42,480 --> 00:04:43,520
So, what's the problem?

43
00:04:43,620 --> 00:04:44,320
Why this is bad?

44
00:04:44,440 --> 00:04:46,460
Why this is something we haven't talked about?

45
00:04:47,080 --> 00:04:48,240
It's bad because it's...

46
00:04:48,260 --> 00:04:50,160
It fundamentally enables impersonation.

47
00:04:50,440 --> 00:05:07,860
So, from the receiving host point of view, if you ever receive any packet from the internet, basically there is no way you can find out if the packet was actually originated in the source IP it says, or was it actually fake and created by someone else, some potentially malicious actor.

48
00:05:08,200 --> 00:05:11,680
Basically, there is no way from the receiving side to know that.

49
00:05:11,900 --> 00:05:12,800
And that's pretty bad.

50
00:05:13,340 --> 00:05:16,900
And we learned that it's pretty bad years ago.

51
00:05:16,900 --> 00:05:26,120
This is a photo of Kevin Mitnick, who used to be, or still is, one of the most famous hackers in the universe, in the galaxy.

52
00:05:26,780 --> 00:05:31,440
So, one of his best attacks was against Mr. Shimomura.

53
00:05:31,660 --> 00:05:40,800
And in 1995, he basically took over Shimomura's TCP session and injected some commands in his TCP session, therefore granting himself root.

54
00:05:40,960 --> 00:05:44,280
And he was able to do that only because he was able to spoof packets.

55
00:05:44,500 --> 00:05:46,880
It was because he was able to fake the packets.

56
00:05:46,900 --> 00:05:49,860
Put magical commands there and the gram can self-access.

57
00:05:50,340 --> 00:05:51,580
That was in 1995.

58
00:05:51,940 --> 00:05:53,140
So, quite a while ago.

59
00:05:53,780 --> 00:05:57,580
Since then, IP spoofing was a problem.

60
00:05:57,780 --> 00:06:06,440
And number and number and number of different attacks, different exploits, different kind of fundamentally internet issues.

61
00:06:07,160 --> 00:06:15,540
In 1996, we had a wave of SYN floods, which I would say kind of the problem of SYN floods was solved only about two or three years ago.

62
00:06:16,000 --> 00:06:16,520
But...

63
00:06:16,900 --> 00:06:18,040
SYN flooding is a different story.

64
00:06:18,640 --> 00:06:24,060
In 1998, we have my favorite attack, sorry, scanning technique called idle scanning.

65
00:06:24,640 --> 00:06:27,880
This was invented by Antires, who now is working on Redis.

66
00:06:28,320 --> 00:06:32,400
But if you haven't read, if you don't know what idle scanning is, I highly recommend it.

67
00:06:32,520 --> 00:06:36,940
It's a super interesting scanning technique, again, based on requiring IP spoofing.

68
00:06:37,900 --> 00:06:41,440
Then we had a whole bunch of problems with TCP implementations.

69
00:06:41,640 --> 00:06:46,680
I think the most well-known are the ones when a malicious attacker, by forging,

70
00:06:46,680 --> 00:06:54,060
reset packet, was able to drop BGP connections, so that could, that led to pretty serious problems in the internet.

71
00:06:55,020 --> 00:07:03,660
And then there was another, you know, 10 years, and in 2008, we all know the famous Dan Kaminski's DNS bug, DNS issues,

72
00:07:03,880 --> 00:07:07,340
which again were only triggered, could only be triggered if you could do IP spoofing.

73
00:07:07,620 --> 00:07:15,720
And I would say since about 2013, we have a new wave of attacks, of the big denial of service attacks, which again are all caused by IP spoofing.

74
00:07:16,680 --> 00:07:18,740
But, you know, that was on the attacking side.

75
00:07:18,860 --> 00:07:19,720
What's on the defense side?

76
00:07:19,820 --> 00:07:21,660
Are we making any progress as the internet community?

77
00:07:22,000 --> 00:07:23,500
Is there anything happening?

78
00:07:23,720 --> 00:07:24,180
Of course.

79
00:07:24,520 --> 00:07:31,880
The first formal definition that, you know, IP spoofing is something, but it was around 1998.

80
00:07:32,540 --> 00:07:36,560
Around 2000, there was a famous BCP-38 published.

81
00:07:37,060 --> 00:07:45,180
It basically says, yep, the internet works as it works, but we should not allow end hosts to forge packets,

82
00:07:45,340 --> 00:07:46,660
to send packets from the internet.

83
00:07:46,680 --> 00:07:49,260
Because the IPs, they don't belong to them.

84
00:07:50,660 --> 00:07:52,580
Unfortunately, BCP-38 is fairly naive.

85
00:07:52,840 --> 00:07:59,620
So then BCP-84, four years later, came in and said, you know, it's harder than we thought, but still, we should do that.

86
00:08:01,120 --> 00:08:07,600
And there are many working groups and many kind of initiatives that right now try to fight IP spoofing.

87
00:08:07,760 --> 00:08:10,320
So there is the EETF working group called SAVI.

88
00:08:10,540 --> 00:08:13,920
There is a working group in the industry.

89
00:08:14,080 --> 00:08:16,660
And then there is the spoofer.ky.org project.

90
00:08:16,680 --> 00:08:20,020
Which tries to track down the IP spoofing around the world.

91
00:08:20,980 --> 00:08:24,360
So this is a chart from the spoofer.ky.org project.

92
00:08:24,500 --> 00:08:32,560
And it basically shows that around 56% of ASs of networks on the internet do not allow spoofing.

93
00:08:32,640 --> 00:08:33,200
So it's good.

94
00:08:33,300 --> 00:08:34,640
Not everyone allows spoofing.

95
00:08:34,700 --> 00:08:37,040
But still, the whole problem is not solved yet.

96
00:08:37,340 --> 00:08:41,640
There is around 27% of ASs of networks that do allow IP spoofing.

97
00:08:41,780 --> 00:08:43,760
So, you know, the problem still exists in the internet.

98
00:08:43,880 --> 00:08:46,660
Even though it was first problematic in 1998.

99
00:08:47,580 --> 00:08:54,040
And not only that, there is the whole industry around amplification and around attacks using IP spoofing.

100
00:08:54,140 --> 00:08:58,260
So if you use your Google skills properly, you'll be able to find services like this one.

101
00:08:58,400 --> 00:09:07,300
So spoofing enabled offshore servers when you can, for Bitcoin, buy bandwidth, not even bandwidth, buy servers and do whatever spoofing you want.

102
00:09:07,720 --> 00:09:08,640
So this is a problem.

103
00:09:08,740 --> 00:09:09,400
This is a proper industry.

104
00:09:09,620 --> 00:09:10,160
This is a problem.

105
00:09:10,300 --> 00:09:15,100
By the way, in the terms of service, they do tell you that if their ISP disables spoofing.

106
00:09:15,100 --> 00:09:16,660
If their ISP tries to enforce it.

107
00:09:16,660 --> 00:09:18,940
If their ISP tries to enforce filtering, they will not give you the Bitcoin's back.

108
00:09:20,640 --> 00:09:21,220
All right.

109
00:09:22,040 --> 00:09:24,440
So what's the big deal about spoofing?

110
00:09:24,560 --> 00:09:26,540
Well, in my opinion, it enables the large attacks.

111
00:09:26,760 --> 00:09:39,300
So if you open news outlets and read about the big, I don't know, ransom emails requiring, I don't know, banks to pay up or kind of attacks that actually were influencing some institutions.

112
00:09:39,740 --> 00:09:46,060
In most cases, not all, but most cases, those big attacks, the biggest attacks will be caused by IP spoofing.

113
00:09:46,100 --> 00:09:46,640
In one way or another.

114
00:09:46,660 --> 00:09:49,840
Now, there is, there needs to be a disclaimer here.

115
00:09:50,020 --> 00:09:51,940
I'm not, this is not about the Mirai attacks.

116
00:09:52,140 --> 00:09:57,540
So we can speak about the Mirai and Dyn at the end in the questions section because they were a bit different.

117
00:09:57,720 --> 00:10:03,900
But except for this case, the major attacks you've heard were almost all about IP spoofing.

118
00:10:05,540 --> 00:10:06,180
All right.

119
00:10:06,220 --> 00:10:06,960
So who am I?

120
00:10:07,040 --> 00:10:07,740
Why do I care?

121
00:10:08,260 --> 00:10:10,480
Why do I know anything about it?

122
00:10:10,840 --> 00:10:15,700
So I work at Cloudflare and we operate a global reverse proxy network.

123
00:10:16,040 --> 00:10:16,640
We have a very large network.

124
00:10:16,640 --> 00:10:18,280
We have our servers all around the globe.

125
00:10:18,400 --> 00:10:20,440
We operate on almost all continents.

126
00:10:20,780 --> 00:10:26,760
We have customers from all backgrounds, from government agencies to social portals.

127
00:10:27,160 --> 00:10:29,580
So, and we also have plenty of customers.

128
00:10:29,820 --> 00:10:35,460
So we see quite a good cross-section of what's happening on the internet.

129
00:10:36,380 --> 00:10:40,160
This is a chart of daily attacks that we categorize as denial of service attacks.

130
00:10:41,220 --> 00:10:42,780
This is per day.

131
00:10:42,940 --> 00:10:46,540
And you can see that some days are quite quiet with, I don't know, 50.

132
00:10:46,560 --> 00:10:46,620
Yeah.

133
00:10:46,640 --> 00:10:51,340
Or 60 events that we categorize as denial of service attacks.

134
00:10:51,720 --> 00:10:54,240
And some of them are more busy with up to 1,400 events.

135
00:10:54,860 --> 00:10:58,340
So we see a good chunk of the weird things on the internet.

136
00:10:58,880 --> 00:11:02,000
And I won't lie, most of the attacks that we see are fairly small.

137
00:11:02,180 --> 00:11:04,020
They are not really anything exciting.

138
00:11:04,240 --> 00:11:07,800
But some of the attacks that we do see are super large, are gigantic.

139
00:11:08,500 --> 00:11:15,940
And over the years, we've published a number of blog posts describing those attacks in big details.

140
00:11:15,940 --> 00:11:22,180
And in this talk, I'm going to show you two major types of attacks we've seen.

141
00:11:22,300 --> 00:11:26,820
And I'm going to also emphasize why they are problematic because of the IP spoofing.

142
00:11:27,820 --> 00:11:28,360
All right.

143
00:11:29,020 --> 00:11:32,080
So one of the attacks I'm going to talk about is called, I will call direct attack.

144
00:11:32,240 --> 00:11:33,440
And the other one, it will be amplification.

145
00:11:34,760 --> 00:11:36,420
So let's start with the direct attack.

146
00:11:38,200 --> 00:11:39,300
So what is it?

147
00:11:39,780 --> 00:11:42,300
Why do I put it in a separate category?

148
00:11:44,520 --> 00:11:45,700
So this is an attack.

149
00:11:45,700 --> 00:11:52,860
This is an attack when the attacker is just transmitting packets directly to us, directly to the target, without any amplification, any reflection in the middle.

150
00:11:53,000 --> 00:11:55,080
It's just packets from the source, the target.

151
00:11:55,260 --> 00:11:55,740
That's it.

152
00:11:56,560 --> 00:11:57,280
Fairly simple.

153
00:11:57,640 --> 00:12:01,700
But we cannot retrace the attacker because the source IPs of the attack are spoofed.

154
00:12:01,700 --> 00:12:03,660
So we don't know who actually originated them.

155
00:12:04,860 --> 00:12:14,660
About a year ago, we published a story which we dubbed the winter of attacks, in which case the attacker had about 400 gigs of capacity, of spoofing capacity.

156
00:12:15,860 --> 00:12:17,720
These are the charts that we published.

157
00:12:18,060 --> 00:12:19,820
They look fairly shady.

158
00:12:20,000 --> 00:12:27,140
But basically we were hit with about 200, on the peak, 200 million packets per second of the attack traffic.

159
00:12:27,440 --> 00:12:31,100
And in the peak as well, about 400 gigabytes per second of the attack traffic.

160
00:12:31,240 --> 00:12:32,280
So it was fairly substantial.

161
00:12:32,600 --> 00:12:34,160
And once again, it was direct attack.

162
00:12:34,320 --> 00:12:40,240
So there exists an attacker in the world, at least a year ago, that had 400 gigabytes of spoofing capacity.

163
00:12:42,500 --> 00:12:45,680
So the interesting things about this type of attack is that they are counterproductive.

164
00:12:45,700 --> 00:12:47,520
usually originate in only a couple of places

165
00:12:47,520 --> 00:12:49,560
and a couple of, I would say, big data

166
00:12:49,560 --> 00:12:51,440
centers or on big

167
00:12:51,440 --> 00:12:53,280
ISPs. So when we

168
00:12:53,280 --> 00:12:55,620
see them, we see them only in a couple

169
00:12:55,620 --> 00:12:56,800
of places around the world.

170
00:12:57,960 --> 00:12:59,620
So that's how we notice.

171
00:13:00,420 --> 00:13:01,440
Okay, so once

172
00:13:01,440 --> 00:13:03,340
we see such an attack, what do we do? Well, first

173
00:13:03,340 --> 00:13:05,240
we have to make sure that we are kept online,

174
00:13:05,480 --> 00:13:07,320
that our servers are operating, that our

175
00:13:07,320 --> 00:13:09,100
customers do not feel the problem,

176
00:13:09,260 --> 00:13:11,580
that the internet still

177
00:13:11,580 --> 00:13:12,680
works for all of

178
00:13:12,680 --> 00:13:15,540
our customers. So what do we do?

179
00:13:15,700 --> 00:13:17,560
Well, we obviously try to figure out what is hitting

180
00:13:17,560 --> 00:13:19,660
us. So we run TCP dump

181
00:13:19,660 --> 00:13:21,720
or some other tools and try to figure out

182
00:13:21,720 --> 00:13:23,580
what is in the packets. And in this

183
00:13:23,580 --> 00:13:25,460
case, those 400 gigs, this

184
00:13:25,460 --> 00:13:27,680
was a SYN flood. So it was a very, very, very large

185
00:13:27,680 --> 00:13:29,460
SYN flood. And

186
00:13:29,460 --> 00:13:31,640
it's interesting because it had to be

187
00:13:31,640 --> 00:13:33,620
SYN floods have to be direct. There's

188
00:13:33,620 --> 00:13:35,480
no way to reflect SYN floods. So basically

189
00:13:35,480 --> 00:13:37,300
if you see a SYN flood, this is direct attacks.

190
00:13:37,560 --> 00:13:39,420
In many of the SYN floods, the

191
00:13:39,860 --> 00:13:41,040
IP packets, the IP

192
00:13:41,040 --> 00:13:43,400
sources are not spoofed.

193
00:13:43,400 --> 00:13:45,520
So it's just flood of SYN

194
00:13:45,520 --> 00:13:45,680
packets.

195
00:13:45,700 --> 00:13:47,620
With a source IP address belonging to the

196
00:13:47,620 --> 00:13:49,760
attacker. But in that case, it's easy to recognize

197
00:13:49,760 --> 00:13:51,400
who is attacking you. In our situation,

198
00:13:51,680 --> 00:13:53,600
the source IPs were obfuscated. They were

199
00:13:53,600 --> 00:13:54,340
just random numbers.

200
00:13:55,480 --> 00:13:57,500
Okay. So how do we keep ourselves online

201
00:13:57,500 --> 00:13:59,420
when we see 400 gigs of

202
00:13:59,420 --> 00:14:01,660
SYN floods? Well, we mitigate it.

203
00:14:02,240 --> 00:14:03,480
And our favorite mitigation

204
00:14:03,480 --> 00:14:04,880
technique is called BPF.

205
00:14:05,520 --> 00:14:07,120
This is an example of IP tables command

206
00:14:07,120 --> 00:14:09,460
with BPFs inside. So what is a BPF?

207
00:14:09,780 --> 00:14:11,340
BPF is a bytecode. It's a

208
00:14:12,160 --> 00:14:13,440
program that you can stick

209
00:14:13,440 --> 00:14:15,380
inside your firewall, inside your IP firewall.

210
00:14:15,700 --> 00:14:16,920
And run it within IP tables.

211
00:14:17,100 --> 00:14:19,140
In this bytecode, you can read data from the packet

212
00:14:19,140 --> 00:14:21,580
and you can basically have an almost

213
00:14:21,580 --> 00:14:23,500
fully fledged program running inside

214
00:14:23,500 --> 00:14:25,020
your firewall, which is super cool.

215
00:14:25,640 --> 00:14:26,420
So on this

216
00:14:28,420 --> 00:14:29,640
command line prompt, you can

217
00:14:29,640 --> 00:14:31,100
see a bunch of

218
00:14:31,780 --> 00:14:33,760
numbers. What are those numbers?

219
00:14:34,000 --> 00:14:35,160
As I mentioned, they're a program.

220
00:14:35,560 --> 00:14:37,440
Here is the same program. It's just

221
00:14:37,440 --> 00:14:39,700
decompiled to more assembly-like format.

222
00:14:40,260 --> 00:14:41,200
So once again,

223
00:14:41,360 --> 00:14:43,080
it's almost fully fledged program.

224
00:14:44,240 --> 00:14:45,520
And by the way,

225
00:14:45,520 --> 00:14:47,300
if you ever run a TCP dump

226
00:14:47,300 --> 00:14:49,660
or Wireshark, you should be familiar with BPFs

227
00:14:49,660 --> 00:14:51,220
because that's exactly the interface

228
00:14:51,220 --> 00:14:53,220
between TCP dump and the kernel.

229
00:14:53,480 --> 00:14:55,300
So if you use TCP dump and type

230
00:14:55,300 --> 00:14:57,260
something like port 80, this will be compiled

231
00:14:57,260 --> 00:14:58,480
down to this bytecode.

232
00:15:00,900 --> 00:15:01,860
All right.

233
00:15:01,920 --> 00:15:03,500
So more specifically, we can't really

234
00:15:03,500 --> 00:15:05,460
use the TCP dump expressions because they are a bit

235
00:15:05,460 --> 00:15:07,600
too limited. So we wrote a series of tools

236
00:15:07,600 --> 00:15:09,540
that we actually use to mitigate the big attacks.

237
00:15:10,700 --> 00:15:11,400
They're all

238
00:15:11,400 --> 00:15:13,560
open-sourced, so on our GitHub you can find

239
00:15:13,560 --> 00:15:15,420
BPF tools. And that's exactly

240
00:15:15,520 --> 00:15:17,220
the bytecodes that we run to

241
00:15:17,220 --> 00:15:19,060
mitigate attacks, including

242
00:15:19,060 --> 00:15:19,880
SYNFLATs.

243
00:15:21,040 --> 00:15:23,180
Now, there needs to be a disclaimer here.

244
00:15:23,320 --> 00:15:25,320
Although iptables is fast and BPFs are

245
00:15:25,320 --> 00:15:27,040
super fast and it all

246
00:15:27,040 --> 00:15:28,640
works reasonably well,

247
00:15:28,900 --> 00:15:30,100
there are still some limitations.

248
00:15:31,540 --> 00:15:33,360
iptables run fairly late

249
00:15:33,360 --> 00:15:35,100
in the kernel network stack.

250
00:15:35,500 --> 00:15:37,340
So you can do much, much faster by

251
00:15:37,340 --> 00:15:39,220
doing some magical tricks like kernel bypass.

252
00:15:39,560 --> 00:15:40,820
But I won't go there

253
00:15:40,820 --> 00:15:43,260
because there's a new development in Linux

254
00:15:43,260 --> 00:15:44,300
kernel called XDP.

255
00:15:45,520 --> 00:15:47,120
It's an express data path. It's available from

256
00:15:47,120 --> 00:15:49,380
kernel 4.9. And right now you need

257
00:15:49,380 --> 00:15:51,320
to have a magical, specific network card.

258
00:15:51,440 --> 00:15:52,720
Only a couple of vendors support it.

259
00:15:52,820 --> 00:15:55,480
But it's super cool. It's really great.

260
00:15:56,160 --> 00:15:57,120
But that's it.

261
00:15:57,220 --> 00:15:59,120
I won't say much because tomorrow

262
00:15:59,120 --> 00:16:01,040
2 p.m. my colleague will give a talk

263
00:16:01,040 --> 00:16:03,100
exactly about XDP. So

264
00:16:03,100 --> 00:16:05,260
stay tuned and be here tomorrow

265
00:16:05,260 --> 00:16:05,920
at 2 p.m.

266
00:16:07,160 --> 00:16:09,300
All right. So we mentioned how

267
00:16:09,300 --> 00:16:11,320
we keep our services online, how

268
00:16:11,320 --> 00:16:13,260
we drop the packets on the floor.

269
00:16:14,320 --> 00:16:15,380
And sometimes we

270
00:16:15,380 --> 00:16:17,100
want to identify who's actually behind those attacks.

271
00:16:17,220 --> 00:16:18,900
We want to dig deeper into

272
00:16:18,900 --> 00:16:21,280
who is actually sending the traffic,

273
00:16:21,540 --> 00:16:22,880
where the traffic originates.

274
00:16:24,560 --> 00:16:25,040
So

275
00:16:25,040 --> 00:16:26,500
basically we need to identify,

276
00:16:26,820 --> 00:16:29,180
since the source IPs are spoofed, we need to identify

277
00:16:29,180 --> 00:16:30,780
from which physical cable

278
00:16:30,780 --> 00:16:32,680
the packets were delivered, like where

279
00:16:32,680 --> 00:16:33,940
they are coming from.

280
00:16:34,640 --> 00:16:36,060
We have no number of

281
00:16:36,640 --> 00:16:38,240
cables connected to our routers.

282
00:16:39,580 --> 00:16:40,880
So what we do is

283
00:16:40,880 --> 00:16:42,940
we log into our routers and we chase the graphs.

284
00:16:44,640 --> 00:16:45,240
This is

285
00:16:45,380 --> 00:16:46,960
a chart. This is a couple of charts from

286
00:16:46,960 --> 00:16:49,100
one of our routers. And here you should be

287
00:16:49,100 --> 00:16:50,620
able to clearly see that there are some

288
00:16:50,620 --> 00:16:52,600
inbound spikes hitting

289
00:16:52,600 --> 00:16:54,980
whatever 10 gigs on the

290
00:16:54,980 --> 00:16:57,060
inbound, on the kind of bottom left chart.

291
00:16:57,320 --> 00:16:58,920
So it's easy to identify

292
00:16:58,920 --> 00:17:01,140
from which interface, from which physical cable

293
00:17:01,140 --> 00:17:03,160
the attack is coming from, even though

294
00:17:03,160 --> 00:17:04,300
the source IPs are obfuscated.

295
00:17:05,080 --> 00:17:07,160
All right. So then we follow the wires.

296
00:17:07,360 --> 00:17:09,000
Then we look on our router

297
00:17:09,000 --> 00:17:11,000
carefully and we try to figure out who's on the

298
00:17:11,000 --> 00:17:13,060
other side of the cable. Who's actually

299
00:17:13,060 --> 00:17:13,900
sending us the traffic?

300
00:17:15,380 --> 00:17:17,280
And there are generally three types

301
00:17:17,280 --> 00:17:19,320
of cables that our routers have. Three types

302
00:17:19,320 --> 00:17:21,020
of connectivity that we have on our routers.

303
00:17:21,200 --> 00:17:23,120
This is basically how the internet works, by the way.

304
00:17:24,780 --> 00:17:25,360
So the

305
00:17:25,360 --> 00:17:27,220
first type of cabling is called

306
00:17:27,220 --> 00:17:29,300
PNIs, direct peering, or cross-connects.

307
00:17:29,340 --> 00:17:31,360
It's basically a direct connection, a direct cable

308
00:17:31,980 --> 00:17:33,180
laid from our router

309
00:17:33,180 --> 00:17:35,220
to someone else's router, when this

310
00:17:35,220 --> 00:17:37,160
someone else is an entity similar to

311
00:17:37,160 --> 00:17:38,940
our company. So it could be

312
00:17:38,940 --> 00:17:41,180
Google, it could be Akamai, it could

313
00:17:41,180 --> 00:17:43,120
be AWS, it could be

314
00:17:43,120 --> 00:17:45,000
any other number of providers.

315
00:17:46,620 --> 00:17:47,420
So basically

316
00:17:47,420 --> 00:17:48,920
another entity on the internet.

317
00:17:49,520 --> 00:17:51,220
The second type of cabling that we

318
00:17:51,220 --> 00:17:53,300
have is a cable that goes from our router

319
00:17:53,300 --> 00:17:54,380
to local internet exchanges.

320
00:17:55,500 --> 00:17:56,980
I will speak more about

321
00:17:56,980 --> 00:17:59,360
internet exchanges, but basically internet exchanges

322
00:17:59,360 --> 00:18:01,400
are a collaboration of

323
00:18:01,400 --> 00:18:03,460
local ISPs, and it's a single place

324
00:18:03,460 --> 00:18:05,540
where they all connect. And finally

325
00:18:05,540 --> 00:18:06,900
there is the connectivity to the

326
00:18:07,560 --> 00:18:09,320
transit providers or internet carriers

327
00:18:09,320 --> 00:18:11,440
where basically it is paid for traffic.

328
00:18:11,560 --> 00:18:12,020
We pay for

329
00:18:13,360 --> 00:18:15,360
to deliver packets to the internet. So it's kind of a

330
00:18:15,380 --> 00:18:16,780
normal connectivity to the internet.

331
00:18:17,060 --> 00:18:19,300
Alright, so let's imagine that the attack

332
00:18:19,300 --> 00:18:22,220
came from direct peering. So what do we do in this case?

333
00:18:22,380 --> 00:18:24,560
What do we do if we know that, you know, the attack

334
00:18:24,560 --> 00:18:26,640
packets came from this specific cable

335
00:18:26,640 --> 00:18:29,080
which has this another entity on the

336
00:18:29,080 --> 00:18:31,440
other side? So it's easy. It's easy because

337
00:18:31,440 --> 00:18:34,740
you can just call up them and say, you

338
00:18:34,740 --> 00:18:36,880
know, guys, you're delivering us junk, please

339
00:18:36,880 --> 00:18:39,060
don't attack us. Please don't do that.

340
00:18:39,320 --> 00:18:41,360
And the good news is that this usually

341
00:18:41,360 --> 00:18:44,480
works. This usually works well. So the

342
00:18:45,840 --> 00:18:47,480
relationship between us and other

343
00:18:47,480 --> 00:18:49,120
part that we cross-connect to is usually

344
00:18:49,120 --> 00:18:51,360
very friendly. It's in both our interests

345
00:18:51,360 --> 00:18:52,720
to, you know, to keep the relationship

346
00:18:52,720 --> 00:18:55,300
okay. We do have sometimes situations

347
00:18:55,300 --> 00:18:58,680
when the ISP on the other side does

348
00:18:58,680 --> 00:19:01,340
respond to our calls and say, okay, I

349
00:19:01,340 --> 00:19:03,340
will kick off the customer that's actually

350
00:19:03,340 --> 00:19:05,200
attacking you. And then the problem

351
00:19:05,200 --> 00:19:06,960
reappears couple of days or weeks later.

352
00:19:07,180 --> 00:19:08,820
But this is fine. Like, we can call them

353
00:19:08,820 --> 00:19:11,380
again and in the worst case we can just

354
00:19:11,380 --> 00:19:13,600
de-peer them and force the attack traffic

355
00:19:13,600 --> 00:19:15,080
to go over the internet and increase

356
00:19:15,080 --> 00:19:17,080
their costs. So basically there is

357
00:19:17,080 --> 00:19:18,380
some discussion here. So that's great.

358
00:19:19,400 --> 00:19:22,460
So what about the other two options? What if

359
00:19:22,460 --> 00:19:23,780
the attack comes over the internet

360
00:19:23,780 --> 00:19:26,240
exchange or what if the spoofed attack comes

361
00:19:26,240 --> 00:19:28,500
over internet carrier and transit

362
00:19:28,500 --> 00:19:30,940
providers? Unfortunately in this case we

363
00:19:30,940 --> 00:19:33,240
cannot do much. So this is a pretty sad

364
00:19:33,240 --> 00:19:35,900
story here. So let me explain to you why

365
00:19:35,900 --> 00:19:40,980
this is the case. Okay, so let's focus on

366
00:19:40,980 --> 00:19:43,300
internet exchanges for a while. So this is a

367
00:19:43,600 --> 00:19:45,180
picture of, I think, Seattle internet

368
00:19:45,180 --> 00:19:47,300
exchange. It's basically a gigantic layer

369
00:19:47,300 --> 00:19:50,740
to switch with plenty of cables in, with

370
00:19:50,740 --> 00:19:53,900
local internet providers connected. And

371
00:19:53,900 --> 00:19:56,000
so what's the issue? The issue is that

372
00:19:56,000 --> 00:19:58,600
internet exchanges are layer 2 switches, so

373
00:19:58,600 --> 00:20:00,560
they are ethernet switches, while

374
00:20:01,180 --> 00:20:03,820
routers are layer 3 beasts, layer 3

375
00:20:03,820 --> 00:20:06,780
entities. And from router point of view,

376
00:20:06,940 --> 00:20:08,980
router is there to route packets. So it

377
00:20:08,980 --> 00:20:10,980
takes packets, it throws away the

378
00:20:10,980 --> 00:20:12,980
ethernet frame as quickly as it can.

379
00:20:13,600 --> 00:20:16,000
That's its sole job. It's to route packets

380
00:20:16,000 --> 00:20:18,060
on the IP layer. And then it routes it back

381
00:20:18,060 --> 00:20:20,460
to our data centers. And that's an issue.

382
00:20:20,940 --> 00:20:22,320
Because from the router point of view,

383
00:20:22,440 --> 00:20:24,880
all that router knows is that, yep, there

384
00:20:24,880 --> 00:20:27,260
are packets, they are DAC packets, they are

385
00:20:27,260 --> 00:20:29,120
coming from internet exchange. That's all

386
00:20:29,120 --> 00:20:31,160
I know. I don't know anything else. So

387
00:20:31,160 --> 00:20:33,480
basically we can't really figure out

388
00:20:33,480 --> 00:20:35,820
which of the local ISPs, which of the local

389
00:20:35,820 --> 00:20:38,080
connected parties is actually originating

390
00:20:38,080 --> 00:20:40,280
traffic. Now if you think about it, this

391
00:20:40,280 --> 00:20:42,400
data is there because it is in the

392
00:20:42,400 --> 00:20:42,920
ethernet frame.

393
00:20:43,600 --> 00:20:46,860
The source MAC address usually should

394
00:20:46,860 --> 00:20:49,080
point out to the router that is

395
00:20:49,080 --> 00:20:50,500
belonging to the entity that is sending

396
00:20:50,500 --> 00:20:52,640
the junk traffic. But unfortunately, as I

397
00:20:52,640 --> 00:20:54,240
mentioned, router doesn't, our router

398
00:20:54,240 --> 00:20:56,280
doesn't really look at that because it's

399
00:20:56,280 --> 00:20:57,820
on a different layer. So this is pretty

400
00:20:57,820 --> 00:20:58,060
sad.

401
00:20:59,060 --> 00:21:02,140
Okay, so what's in the third case?

402
00:21:03,000 --> 00:21:05,420
What happens if the attack comes over

403
00:21:05,420 --> 00:21:08,140
internet carriers or transit providers?

404
00:21:08,700 --> 00:21:11,020
What can we do in such a case? Well, we

405
00:21:11,020 --> 00:21:12,740
often call them up and say, you know guys,

406
00:21:12,740 --> 00:21:15,180
you're delivering us packets, which are

407
00:21:15,180 --> 00:21:17,220
just junk, which are just invalid packets.

408
00:21:18,480 --> 00:21:21,420
And they say, well, it's because you paid

409
00:21:21,420 --> 00:21:23,880
us to do that. And that's true, like the

410
00:21:23,880 --> 00:21:25,320
internet carriers are there to deliver

411
00:21:25,320 --> 00:21:27,320
packets. We paid them exactly for this

412
00:21:27,320 --> 00:21:29,100
job. And by the way, I don't think that

413
00:21:29,100 --> 00:21:31,820
it is their job to filter the

414
00:21:31,820 --> 00:21:33,320
packets. It's censorship, it's not really

415
00:21:33,320 --> 00:21:35,320
their job. So it's kind of fine, but

416
00:21:35,320 --> 00:21:36,920
still, they should be able to do

417
00:21:36,920 --> 00:21:38,280
something about it. They should be able

418
00:21:38,280 --> 00:21:40,260
to inspect their network and figure out

419
00:21:40,260 --> 00:21:42,420
which of their customers is actually

420
00:21:42,420 --> 00:21:44,060
originating the traffic. Unfortunately,

421
00:21:44,200 --> 00:21:45,620
that's not the case. They can't really do

422
00:21:45,620 --> 00:21:48,220
that. So again, pretty sad story. We see

423
00:21:48,220 --> 00:21:49,680
packets flying in from our internet

424
00:21:49,680 --> 00:21:52,440
carrier. We have no idea who is actually

425
00:21:52,440 --> 00:21:55,600
originating them. So that all leads to the

426
00:21:55,600 --> 00:21:57,440
conclusion that basically, in most cases,

427
00:21:57,480 --> 00:21:59,900
tracing back these spoofed packets is

428
00:21:59,900 --> 00:22:01,420
impossible. And that's pretty bad.

429
00:22:02,300 --> 00:22:03,780
And that's pretty bad because we cannot really

430
00:22:03,780 --> 00:22:05,200
push back against it. We cannot really

431
00:22:05,200 --> 00:22:07,120
figure out which data

432
00:22:07,120 --> 00:22:09,320
centers are evil, which IPs are evil, because we

433
00:22:09,320 --> 00:22:09,760
just don't know.

434
00:22:10,620 --> 00:22:12,400
Okay, so let's deep and deep

435
00:22:12,420 --> 00:22:14,660
dig into a bit more details into the

436
00:22:14,660 --> 00:22:17,800
attacks I showed, which is the winter of

437
00:22:17,800 --> 00:22:20,320
attacks, which I mentioned already. I can

438
00:22:20,320 --> 00:22:21,720
actually share a couple of more details

439
00:22:21,720 --> 00:22:24,840
about this specific attack. So this

440
00:22:24,840 --> 00:22:26,680
attack was on the third category. This

441
00:22:26,680 --> 00:22:29,360
attack, this 400 gigs, was coming to us

442
00:22:29,360 --> 00:22:33,160
over paid-for internet carrier. And

443
00:22:33,160 --> 00:22:35,080
in this specific attack, the source IP

444
00:22:35,080 --> 00:22:37,380
addresses were belonging to an AS

445
00:22:37,380 --> 00:22:39,160
number belonging to Hurricane

446
00:22:39,160 --> 00:22:42,260
Electric. Hurricane Electric is a big, a big

447
00:22:42,420 --> 00:22:44,620
internet carrier. They are one of the more

448
00:22:44,620 --> 00:22:47,220
competent ones. But anyway, from what we

449
00:22:47,220 --> 00:22:49,580
knew, all we knew is that we see attack

450
00:22:49,580 --> 00:22:51,620
traffic, we see SYN flood, and the SYN flood

451
00:22:51,620 --> 00:22:53,340
comes from IP ranges belonging to

452
00:22:53,340 --> 00:22:55,820
Hurricane Electric. Okay. That's pretty bad.

453
00:22:56,320 --> 00:22:58,580
But then we noticed that actually the

454
00:22:58,580 --> 00:22:59,900
attack was coming to our Los Angeles

455
00:22:59,900 --> 00:23:02,000
data center. And in Los Angeles we

456
00:23:02,000 --> 00:23:03,380
actually have a direct connectivity to

457
00:23:03,380 --> 00:23:06,420
Hurricane Electric. So that leads to two

458
00:23:06,420 --> 00:23:09,260
conclusions, to two options. One option is

459
00:23:09,260 --> 00:23:12,400
that, okay, the attack was actually coming from Hurricane Electric.

460
00:23:12,420 --> 00:23:18,460
you know, it is possible, but in that case, it should be delivered over direct link to us.

461
00:23:18,620 --> 00:23:20,180
It shouldn't be delivered over paid for internet.

462
00:23:20,380 --> 00:23:22,800
So maybe they misconfigured the routing.

463
00:23:23,020 --> 00:23:27,980
If so, they should fire their router admins because they just paid for a big attack traffic.

464
00:23:28,580 --> 00:23:30,560
So let's say that's not the case.

465
00:23:30,640 --> 00:23:34,840
So the other interpretation is, of course, that our internet carrier has a client, has a customer,

466
00:23:34,960 --> 00:23:37,420
who is spoofing packets and pretending to be Hurricane Electric.

467
00:23:37,420 --> 00:23:42,960
But this is a nice situation, a nice explanation that our internet carrier actually doesn't know,

468
00:23:43,040 --> 00:23:44,720
does not know if the packets are valid or not.

469
00:23:44,960 --> 00:23:49,340
From the internet carrier's point of view, these packets are originating in Hurricane Electric,

470
00:23:49,540 --> 00:23:51,280
and that can be okay, that can be fine.

471
00:23:51,520 --> 00:23:54,360
It's just we know that that wasn't the case in this situation.

472
00:23:56,240 --> 00:23:59,480
Alright, let me show you a couple of more examples of similar attacks.

473
00:24:00,040 --> 00:24:05,540
But before I show you more details, we need to draw, we need to have vocabulary to speak about those attacks,

474
00:24:05,700 --> 00:24:06,940
those IP spoofing attacks.

475
00:24:08,220 --> 00:24:10,820
So this is how we are going to show it.

476
00:24:11,180 --> 00:24:17,260
So this is a picture from XKCD by Randall Monroe from about 10 years old.

477
00:24:17,340 --> 00:24:19,400
And he attempted to draw a map of the internet.

478
00:24:20,020 --> 00:24:20,880
It's not easy.

479
00:24:21,240 --> 00:24:26,020
So what he did is he used a mathematical thing called Hilbert curve.

480
00:24:26,340 --> 00:24:32,380
And the Hilbert curve is a way to draw a line over two-dimensional space that will kind of group things together.

481
00:24:33,000 --> 00:24:37,160
So in each of the squares you can see here, there is a slash 8 subnet.

482
00:24:37,800 --> 00:24:40,400
So you can see top right corner, there are some reserved blocks.

483
00:24:41,080 --> 00:24:43,640
Again, kind of top right corner, there is a multicast group.

484
00:24:43,880 --> 00:24:48,900
So basically all these slash 8s are kind of close together in a square.

485
00:24:49,140 --> 00:24:52,900
If you look at slash 16s, they will also be in a smaller square and so on and so forth.

486
00:24:53,040 --> 00:24:56,720
So it's a pretty cool way of visualizing IP space, basically.

487
00:24:57,820 --> 00:24:59,440
Okay, let me show you one of the attacks.

488
00:25:00,120 --> 00:25:03,020
So this is a Hilbert curve drawn by me.

489
00:25:03,140 --> 00:25:04,760
You can see some hashed out blocks.

490
00:25:05,000 --> 00:25:07,300
These blocks are basically unroutable IP addresses.

491
00:25:08,000 --> 00:25:11,600
And in the middle there is 127 slash 8.

492
00:25:11,740 --> 00:25:14,940
So kind of reserved look back IP range.

493
00:25:15,220 --> 00:25:23,520
Okay, so in this attack for each of these source IPs that we saw, there was, I drew the black dot.

494
00:25:23,700 --> 00:25:27,380
Okay, so plenty of IP addresses scattered all around.

495
00:25:27,660 --> 00:25:28,660
Is that interesting?

496
00:25:28,900 --> 00:25:31,600
What's exciting about this?

497
00:25:31,840 --> 00:25:35,660
Well, not much until you realize that this is,

498
00:25:35,860 --> 00:25:37,400
these are IP ranges.

499
00:25:37,420 --> 00:25:39,340
These are IP addresses belonging to China Telecom.

500
00:25:39,520 --> 00:25:41,340
So again, this is the attack.

501
00:25:41,660 --> 00:25:42,740
This is China Telecom.

502
00:25:42,880 --> 00:25:44,540
Attack, China Telecom.

503
00:25:44,960 --> 00:25:47,240
So the interpretation of this is fairly hard.

504
00:25:47,400 --> 00:25:48,980
Like, I'm not saying China Telecom attacked us.

505
00:25:49,060 --> 00:25:51,920
I'm saying the IP addresses belonging to China Telecom were used in the attack.

506
00:25:52,520 --> 00:25:56,840
And from my point of view, it's super hard to actually figure out what's happening behind the scenes.

507
00:25:57,400 --> 00:25:58,860
So there are two interpretations possible.

508
00:25:59,340 --> 00:26:04,320
One interpretation is that someone has a large botnet in China.

509
00:26:04,320 --> 00:26:09,460
I know it is totally possible that someone has a gigantic botnet in China and is just using it to attack us.

510
00:26:09,660 --> 00:26:11,360
That's fairly sane. It's possible.

511
00:26:11,620 --> 00:26:14,100
In such case, the IP addresses will not be spoofed.

512
00:26:14,100 --> 00:26:19,180
They will be properly legitimate IP addresses belonging to malicious bots in China.

513
00:26:19,800 --> 00:26:23,860
On the other hand, if you look at that again, they seem to be quite well distributed.

514
00:26:24,420 --> 00:26:25,640
Again, I'm not an expert.

515
00:26:26,140 --> 00:26:30,580
Maybe China Telecom IP addresses are distributed evenly around the China population.

516
00:26:30,900 --> 00:26:32,460
I don't know if that's the case.

517
00:26:32,540 --> 00:26:32,920
It's possible.

518
00:26:34,320 --> 00:26:41,440
It could also be the case that someone has just went through the IP allocations and is just spoofing the China Telecom IP addresses.

519
00:26:42,040 --> 00:26:43,660
There is no way to figure it out.

520
00:26:44,780 --> 00:26:45,180
Okay.

521
00:26:45,280 --> 00:26:47,800
So another example of an exciting attack.

522
00:26:48,940 --> 00:26:49,640
This one.

523
00:26:49,800 --> 00:26:50,820
This is my favorite.

524
00:26:51,740 --> 00:26:56,740
So in this case, you can see that the source IPs are just uniformly distributed across the whole IP space.

525
00:26:57,020 --> 00:26:58,320
Including the reserved blocks.

526
00:26:58,500 --> 00:26:59,600
Including 255.

527
00:27:00,240 --> 00:27:02,300
So the top one, including 0.

528
00:27:02,480 --> 00:27:03,320
So the bottom one.

529
00:27:04,320 --> 00:27:06,320
Including reserved blocks like 127.8.

530
00:27:06,960 --> 00:27:10,500
So if you ask me, do I see attacks from loopback?

531
00:27:10,660 --> 00:27:11,480
Yes, absolutely.

532
00:27:11,660 --> 00:27:17,640
Every second of the day, I see malicious packets coming from 127.01 attacking my network.

533
00:27:18,780 --> 00:27:19,440
So, you know.

534
00:27:19,520 --> 00:27:20,160
There is that.

535
00:27:21,320 --> 00:27:23,400
There are then variations on top of that.

536
00:27:23,600 --> 00:27:28,940
So, for example, this is a very similar chart with uniformly distributed source IP addresses across the whole IP space.

537
00:27:29,560 --> 00:27:32,320
But with the reserved blocks skipped.

538
00:27:33,300 --> 00:27:33,560
Skipped.

539
00:27:35,000 --> 00:27:37,180
Now, the question is, why did it happen?

540
00:27:37,400 --> 00:27:39,620
Was it the attacker that is smart?

541
00:27:39,860 --> 00:27:43,440
And, you know, when they have the routine that is forging the source IP addresses.

542
00:27:43,780 --> 00:27:48,180
Is the attacker just, you know, filtering the IP addresses and say, okay, there is 0.

543
00:27:48,300 --> 00:27:50,460
So maybe let's not put the obviously wrong ones.

544
00:27:50,900 --> 00:27:52,160
But I don't think that's the case.

545
00:27:52,440 --> 00:27:55,640
So I think what's actually happening is that their ISP is filtering the traffic.

546
00:27:56,320 --> 00:27:58,400
But again, there is no way for me to figure it out.

547
00:27:59,820 --> 00:28:00,820
There is next one.

548
00:28:01,140 --> 00:28:02,900
Next very uniform distribution.

549
00:28:03,520 --> 00:28:04,300
Why this particular one?

550
00:28:04,320 --> 00:28:05,460
This particular pattern I have no idea.

551
00:28:05,680 --> 00:28:06,540
Maybe someone will know.

552
00:28:07,780 --> 00:28:08,780
And another one.

553
00:28:09,040 --> 00:28:15,420
This may again look like a fairly nice either gigantic botnet from all around the world.

554
00:28:15,640 --> 00:28:22,560
Or just someone spoofing some weird stuff with a bit more density and kind of the lower IP ranges.

555
00:28:22,820 --> 00:28:23,440
But no.

556
00:28:24,040 --> 00:28:26,660
If you look at this particular chart carefully.

557
00:28:26,880 --> 00:28:27,800
You would notice.

558
00:28:28,040 --> 00:28:29,440
If you are a network expert.

559
00:28:29,580 --> 00:28:34,300
You would notice that those dots correspond exactly to the nodes.

560
00:28:34,320 --> 00:28:35,460
They are non-routable IP ranges.

561
00:28:35,900 --> 00:28:37,500
So in current internet.

562
00:28:37,600 --> 00:28:40,120
Almost every IP address belongs to someone these days.

563
00:28:40,680 --> 00:28:43,200
But there are still plenty of IP blocks that are not routable.

564
00:28:43,320 --> 00:28:45,600
That don't exist in the global BGP routing table.

565
00:28:45,860 --> 00:28:47,200
So in this case.

566
00:28:47,240 --> 00:28:48,340
The attacker was fairly sophisticated.

567
00:28:48,700 --> 00:28:50,960
The attacker went for the global BGP routing table.

568
00:28:51,560 --> 00:28:52,220
Inverted it.

569
00:28:52,340 --> 00:28:55,140
So figured out, okay, so what blocks are not present there.

570
00:28:55,280 --> 00:28:56,280
And only spoofed these.

571
00:28:56,660 --> 00:28:57,360
Why such?

572
00:28:57,540 --> 00:28:58,040
I don't know.

573
00:28:58,120 --> 00:28:59,000
But here you are.

574
00:29:00,220 --> 00:29:00,820
All right.

575
00:29:00,820 --> 00:29:03,120
So that's all I have on the direct attacks.

576
00:29:03,320 --> 00:29:05,400
When I actually can see the source IP addresses.

577
00:29:05,720 --> 00:29:07,400
I can actually argue about it.

578
00:29:07,560 --> 00:29:08,900
Again, I can't really track it back.

579
00:29:09,040 --> 00:29:09,520
To anyone.

580
00:29:09,700 --> 00:29:11,060
But at least I see them.

581
00:29:11,720 --> 00:29:13,340
But then there is a second category of attacks.

582
00:29:13,780 --> 00:29:14,960
Called the amplification attacks.

583
00:29:17,260 --> 00:29:18,540
About two weeks ago.

584
00:29:18,660 --> 00:29:19,200
Three weeks ago.

585
00:29:19,280 --> 00:29:21,780
We published a story about SSDP amplification.

586
00:29:23,200 --> 00:29:25,200
These charts are not as exciting.

587
00:29:25,440 --> 00:29:27,660
But basically they show that the attack we received.

588
00:29:27,800 --> 00:29:28,760
To our network.

589
00:29:29,740 --> 00:29:30,800
Was of about 10.000.

590
00:29:30,820 --> 00:29:32,060
About 40 million packets per second.

591
00:29:32,300 --> 00:29:35,700
And the volume was about 110 gigabytes per second.

592
00:29:35,840 --> 00:29:36,220
So you know.

593
00:29:36,340 --> 00:29:37,960
A sizable attack.

594
00:29:39,300 --> 00:29:39,820
Okay.

595
00:29:39,900 --> 00:29:41,500
So what is an amplification attack?

596
00:29:41,800 --> 00:29:42,820
Just a couple of slides on it.

597
00:29:43,120 --> 00:29:44,860
The general idea is that.

598
00:29:45,680 --> 00:29:48,580
There exists UDP servers in the world.

599
00:29:48,640 --> 00:29:49,700
UDP protocols in the world.

600
00:29:50,140 --> 00:29:52,640
With a basic request response protocols.

601
00:29:53,460 --> 00:29:55,020
And the idea of those protocols.

602
00:29:55,180 --> 00:29:56,960
Is that the client creates a request.

603
00:29:57,160 --> 00:29:58,120
And the server responds.

604
00:29:58,460 --> 00:29:58,840
That's it.

605
00:30:00,820 --> 00:30:02,160
It's DNS and it works exactly like that.

606
00:30:02,300 --> 00:30:04,480
You know that because NDP works exactly like that.

607
00:30:04,600 --> 00:30:07,240
And a number of other protocols work in the same fashion.

608
00:30:07,620 --> 00:30:08,400
So what's the problem?

609
00:30:09,160 --> 00:30:10,160
Well the problem is that.

610
00:30:10,240 --> 00:30:11,540
If you can spoof packets.

611
00:30:11,820 --> 00:30:13,280
You can forge the request.

612
00:30:13,940 --> 00:30:15,900
So you can convince a legitimate server.

613
00:30:16,160 --> 00:30:17,420
To answer to a request.

614
00:30:17,560 --> 00:30:19,600
That wasn't really asked in the first place.

615
00:30:19,920 --> 00:30:22,480
And therefore you can cause the server.

616
00:30:22,600 --> 00:30:23,200
Legitimate server.

617
00:30:23,480 --> 00:30:26,320
To trigger legitimate answer to a target.

618
00:30:26,700 --> 00:30:28,240
And that way you can basically.

619
00:30:29,840 --> 00:30:30,680
Amplify the attack.

620
00:30:30,820 --> 00:30:32,300
So the main idea is that.

621
00:30:32,380 --> 00:30:33,600
In many of those protocols.

622
00:30:33,820 --> 00:30:36,820
The responses are much much much larger than the requests.

623
00:30:37,040 --> 00:30:37,960
For example in DNS.

624
00:30:38,160 --> 00:30:39,780
In DNS the request is usually small.

625
00:30:39,980 --> 00:30:41,460
And the response is usually much larger.

626
00:30:41,800 --> 00:30:43,040
And if that's the case.

627
00:30:43,200 --> 00:30:46,200
If this is the case in the UDP protocol you're using.

628
00:30:46,400 --> 00:30:48,020
You can create an amplification.

629
00:30:48,180 --> 00:30:49,580
Of a big size.

630
00:30:49,960 --> 00:30:51,800
So in this case it's like 10 times.

631
00:30:52,180 --> 00:30:54,100
But it's not uncommon to see 20 times.

632
00:30:54,140 --> 00:30:55,080
Or 80 times amplifications.

633
00:30:55,560 --> 00:30:57,600
And this is great from the attacker point of view.

634
00:30:57,600 --> 00:30:58,640
Because you know.

635
00:30:58,840 --> 00:31:01,080
Possessing one gigabit per second of attack.

636
00:31:01,280 --> 00:31:02,340
Of spoofing capacity.

637
00:31:02,640 --> 00:31:04,960
You are able to generate much much much larger.

638
00:31:05,920 --> 00:31:06,700
Attack volume.

639
00:31:07,960 --> 00:31:08,880
Going to the target.

640
00:31:11,880 --> 00:31:14,040
Also you never use a single UDP server.

641
00:31:14,320 --> 00:31:15,280
You have to scale it out.

642
00:31:15,360 --> 00:31:16,480
There is no single UDP server.

643
00:31:16,620 --> 00:31:17,900
That can generate too much traffic.

644
00:31:18,840 --> 00:31:20,900
So that's what we saw those couple of weeks ago.

645
00:31:22,140 --> 00:31:23,320
So what we saw was.

646
00:31:23,420 --> 00:31:25,540
An attack which again.

647
00:31:25,660 --> 00:31:26,840
On our side had about.

648
00:31:27,600 --> 00:31:28,660
12 gigabits per second.

649
00:31:29,300 --> 00:31:31,860
There were about 490,000.

650
00:31:32,560 --> 00:31:33,240
Reflectors used.

651
00:31:33,500 --> 00:31:34,500
So a sizeable chunk.

652
00:31:34,800 --> 00:31:36,380
A big number of reflectors.

653
00:31:36,720 --> 00:31:38,380
And we believe that the attacker.

654
00:31:38,500 --> 00:31:40,260
Only had about 5 gigs of spoofing capacity.

655
00:31:41,180 --> 00:31:42,360
That's what we can know from.

656
00:31:42,760 --> 00:31:44,260
Looking at the protocol deeper.

657
00:31:45,700 --> 00:31:46,140
Alright.

658
00:31:46,280 --> 00:31:49,320
So again what do we do in case of session attack.

659
00:31:49,640 --> 00:31:51,320
Well the first thing as always.

660
00:31:51,500 --> 00:31:53,020
Is we try to keep ourselves online.

661
00:31:54,300 --> 00:31:55,340
So how do we do that.

662
00:31:55,460 --> 00:31:56,980
There are a number of techniques we are using.

663
00:31:57,600 --> 00:31:58,020
But once.

664
00:31:58,120 --> 00:32:00,100
The most interesting one is.

665
00:32:00,900 --> 00:32:01,640
That we are.

666
00:32:02,460 --> 00:32:04,840
This attack is nicely distributed geographically.

667
00:32:05,020 --> 00:32:07,040
This is because all of those reflectors.

668
00:32:07,120 --> 00:32:09,720
All those 940,000 reflectors around the world.

669
00:32:09,900 --> 00:32:12,340
They don't belong in a single place in the world.

670
00:32:12,480 --> 00:32:14,180
They are scattered all around the globe.

671
00:32:14,520 --> 00:32:16,340
And this is good from our point of view.

672
00:32:16,380 --> 00:32:17,580
Because the attack traffic.

673
00:32:17,740 --> 00:32:18,900
Is nicely delivered.

674
00:32:19,060 --> 00:32:20,200
To all our data centers.

675
00:32:21,400 --> 00:32:22,580
We can do that because.

676
00:32:23,820 --> 00:32:25,420
We run an anycast network.

677
00:32:25,420 --> 00:32:27,780
So we advertise the same IP ranges.

678
00:32:27,980 --> 00:32:28,900
All over the world.

679
00:32:29,180 --> 00:32:31,020
Therefore the attack packets will be delivered.

680
00:32:31,120 --> 00:32:32,440
To the closest data center.

681
00:32:33,400 --> 00:32:35,320
That is to the reflectors servers.

682
00:32:35,660 --> 00:32:36,200
So again.

683
00:32:36,640 --> 00:32:38,660
Our architecture allows us.

684
00:32:38,740 --> 00:32:41,220
To basically split the traffic very evenly.

685
00:32:41,340 --> 00:32:42,200
Across our data centers.

686
00:32:43,600 --> 00:32:44,120
Okay.

687
00:32:44,200 --> 00:32:45,160
So what do we do next.

688
00:32:45,440 --> 00:32:48,200
Well blocking these attacks is usually very very simple.

689
00:32:48,640 --> 00:32:50,000
These attacks usually happen.

690
00:32:50,180 --> 00:32:52,560
Usually come from a specific protocol.

691
00:32:53,220 --> 00:32:54,360
So if it's SSDP.

692
00:32:55,420 --> 00:32:57,280
The servers will always send packets.

693
00:32:57,440 --> 00:32:58,140
From port 1900.

694
00:32:58,800 --> 00:33:00,920
So just dropping port 1900.

695
00:33:01,300 --> 00:33:02,380
With the protocol UDP.

696
00:33:02,700 --> 00:33:04,320
On their firewall will do the job.

697
00:33:04,540 --> 00:33:06,100
So it's really fairly simple.

698
00:33:06,440 --> 00:33:08,340
It's a bit harder for things like DNS.

699
00:33:08,580 --> 00:33:10,380
Because you can't really drop all DNS traffic.

700
00:33:10,660 --> 00:33:11,580
But still fundamentally.

701
00:33:11,820 --> 00:33:13,420
That's a fairly simple thing.

702
00:33:14,720 --> 00:33:16,580
Okay and how do we identify the sources.

703
00:33:16,840 --> 00:33:18,560
How do we know who actually is attacking us.

704
00:33:18,880 --> 00:33:20,360
And the answer is that we don't.

705
00:33:20,380 --> 00:33:21,240
And we cannot do that.

706
00:33:21,320 --> 00:33:22,040
And no one can do that.

707
00:33:25,420 --> 00:33:26,560
So the packets that are delivered to us.

708
00:33:26,600 --> 00:33:27,300
Are legitimate.

709
00:33:27,620 --> 00:33:30,380
The Reflector servers deliver us responses.

710
00:33:30,720 --> 00:33:32,420
The requests we never asked in the first place.

711
00:33:32,500 --> 00:33:34,040
But still the packets were legitimate.

712
00:33:34,200 --> 00:33:35,680
The source IPs are absolutely correct.

713
00:33:36,600 --> 00:33:38,240
So if you wanted to trace it back.

714
00:33:38,320 --> 00:33:40,260
We would need to ask the internet.

715
00:33:40,580 --> 00:33:41,440
Dear internet.

716
00:33:41,540 --> 00:33:43,960
Who in the internet forged our source IP addresses.

717
00:33:44,360 --> 00:33:45,520
And there is just no.

718
00:33:45,680 --> 00:33:46,980
You just can't ask this question.

719
00:33:48,780 --> 00:33:49,260
Alright.

720
00:33:49,360 --> 00:33:50,920
So what about the amplifications.

721
00:33:50,920 --> 00:33:53,280
So in the other blog post.

722
00:33:53,400 --> 00:33:53,840
That we published.

723
00:33:53,980 --> 00:33:55,500
We mentioned a couple of other protocols used.

724
00:33:56,540 --> 00:33:58,400
But these are usually the usual suspects.

725
00:33:59,860 --> 00:34:01,000
So it's usually NTP.

726
00:34:01,760 --> 00:34:03,240
SSDP is the biggest one we saw.

727
00:34:03,560 --> 00:34:04,700
There is also DNS.

728
00:34:05,100 --> 00:34:06,580
And a couple of other protocols.

729
00:34:07,060 --> 00:34:08,040
And most interestingly.

730
00:34:08,180 --> 00:34:09,160
There are a couple of gaming protocols.

731
00:34:09,540 --> 00:34:13,040
So I think we will see a new interesting attacks.

732
00:34:14,000 --> 00:34:15,540
That are using gaming protocols.

733
00:34:16,420 --> 00:34:17,360
Like Call of Duty.

734
00:34:17,440 --> 00:34:18,000
Or Source.

735
00:34:20,160 --> 00:34:20,640
Because.

736
00:34:20,920 --> 00:34:22,580
These protocols are not audited well enough.

737
00:34:23,820 --> 00:34:24,900
On the attack sizes.

738
00:34:25,100 --> 00:34:26,240
The amplification attacks.

739
00:34:26,400 --> 00:34:27,760
Are usually fairly small.

740
00:34:28,440 --> 00:34:29,900
So the average size of the attacks.

741
00:34:30,100 --> 00:34:31,680
We noticed in last six months.

742
00:34:31,920 --> 00:34:34,720
Was about 7 gigabits per second.

743
00:34:35,400 --> 00:34:36,040
So you know.

744
00:34:36,080 --> 00:34:36,540
Not gigantic.

745
00:34:36,740 --> 00:34:38,380
Sizeable for small internet players.

746
00:34:38,620 --> 00:34:38,940
But you know.

747
00:34:38,980 --> 00:34:39,320
Not gigantic.

748
00:34:40,480 --> 00:34:42,060
The maximum was 100 gigs.

749
00:34:42,180 --> 00:34:42,820
As I mentioned.

750
00:34:42,980 --> 00:34:43,780
This is a bit outdated.

751
00:34:50,920 --> 00:34:52,680
Up to 44 gigabits per second.

752
00:34:53,080 --> 00:34:54,420
Or NTP attacks.

753
00:34:54,640 --> 00:34:56,260
I think the maximum NTP one we saw.

754
00:34:56,300 --> 00:34:57,680
Was 64 gigabits.

755
00:34:57,820 --> 00:34:59,300
Just over last six months.

756
00:35:01,300 --> 00:35:01,740
Alright.

757
00:35:04,080 --> 00:35:05,260
That's almost it.

758
00:35:05,840 --> 00:35:06,580
So basically.

759
00:35:06,600 --> 00:35:08,540
There are kind of three themes in this talk so far.

760
00:35:08,760 --> 00:35:10,760
One is that IP spoofing is generally bad.

761
00:35:11,040 --> 00:35:12,900
And it allows all kinds of problems.

762
00:35:13,100 --> 00:35:14,240
And most importantly.

763
00:35:14,260 --> 00:35:14,800
The big attacks.

764
00:35:15,420 --> 00:35:16,760
The second thing is that.

765
00:35:16,860 --> 00:35:18,120
In order to survive the big attacks.

766
00:35:20,920 --> 00:35:21,460
It's pretty unfortunate.

767
00:35:21,660 --> 00:35:24,040
Because that means that only big internet players can survive.

768
00:35:24,200 --> 00:35:25,100
The largest attacks.

769
00:35:25,880 --> 00:35:28,640
And also the third one is that fighting against those attacks.

770
00:35:28,760 --> 00:35:29,560
Is basically impossible.

771
00:35:29,880 --> 00:35:31,100
There is no way to trace the attacker.

772
00:35:31,460 --> 00:35:33,180
There is no way to put pressure on the attacker.

773
00:35:33,420 --> 00:35:35,260
We just don't know who is on the other side of the wire.

774
00:35:36,920 --> 00:35:37,320
Okay.

775
00:35:37,380 --> 00:35:38,920
So how the internet can fix it.

776
00:35:39,000 --> 00:35:41,560
How we can influence our colleagues.

777
00:35:42,400 --> 00:35:43,700
How we can buy bandwidth.

778
00:35:43,940 --> 00:35:46,400
What can we do as internet community to fix it eventually.

779
00:35:47,620 --> 00:35:48,020
Okay.

780
00:35:48,020 --> 00:35:49,760
So let's tackle it step by step.

781
00:35:51,460 --> 00:35:52,300
So first.

782
00:35:52,780 --> 00:35:54,220
We have to understand.

783
00:35:54,540 --> 00:35:55,780
That these attacks are possible.

784
00:35:55,900 --> 00:35:57,180
Only because of IP spoofing.

785
00:35:57,700 --> 00:35:59,460
We would not speak here about amplifications.

786
00:35:59,800 --> 00:36:01,420
If it wasn't about IP spoofing.

787
00:36:01,540 --> 00:36:03,020
I wouldn't speak about SYN floods.

788
00:36:03,280 --> 00:36:04,420
If it wasn't about IP spoofing.

789
00:36:04,500 --> 00:36:04,660
Yes.

790
00:36:04,880 --> 00:36:06,020
SYN floods would have existed.

791
00:36:06,300 --> 00:36:09,880
But I would be able to just sue or put pressure on the people.

792
00:36:09,940 --> 00:36:11,460
That are actually sending me the malicious traffic.

793
00:36:13,600 --> 00:36:15,060
So what can we do about IP spoofing.

794
00:36:15,060 --> 00:36:16,780
Well fundamentally we can't really do much.

795
00:36:16,940 --> 00:36:18,400
Because that's how the internet works.

796
00:36:18,680 --> 00:36:20,560
But we should continue the usual efforts.

797
00:36:20,720 --> 00:36:22,380
We should continue promoting BCP38.

798
00:36:22,700 --> 00:36:25,380
And basically promote good network practices.

799
00:36:26,660 --> 00:36:29,380
These things are not dead.

800
00:36:29,800 --> 00:36:32,040
We just need to spend more time on it.

801
00:36:32,820 --> 00:36:34,300
But we can do even more things.

802
00:36:34,960 --> 00:36:36,940
There is this spooferkaida.org project.

803
00:36:37,040 --> 00:36:37,760
Which I mentioned already.

804
00:36:38,340 --> 00:36:40,820
They give you.

805
00:36:40,900 --> 00:36:42,480
If you want you can download an application.

806
00:36:43,020 --> 00:36:44,100
Which will run in background.

807
00:36:45,060 --> 00:36:46,900
And basically test network that you connect to.

808
00:36:46,980 --> 00:36:47,760
For IP spoofing.

809
00:36:47,800 --> 00:36:48,560
This is pretty cool.

810
00:36:48,700 --> 00:36:50,860
So you can basically as you go around the world.

811
00:36:50,960 --> 00:36:53,100
You can basically see what networks are.

812
00:36:53,240 --> 00:36:54,480
Allows spoofing to what degree.

813
00:36:55,720 --> 00:36:56,560
And finally.

814
00:36:57,360 --> 00:36:59,180
We vendors can do a lot.

815
00:36:59,700 --> 00:37:02,400
For example right now the RPF.

816
00:37:02,580 --> 00:37:04,440
So reverse path filtering.

817
00:37:04,620 --> 00:37:05,260
Sorry forwarding.

818
00:37:05,400 --> 00:37:06,800
Is not enabled by default.

819
00:37:06,940 --> 00:37:08,140
If you buy a new router.

820
00:37:08,380 --> 00:37:09,400
It should be enabled by default.

821
00:37:09,940 --> 00:37:11,920
There are reasons why you may want to disable it.

822
00:37:15,060 --> 00:37:15,940
But first it should be enabled.

823
00:37:18,520 --> 00:37:20,220
But again we have to appreciate.

824
00:37:20,360 --> 00:37:22,040
That actually doing filtering is hard.

825
00:37:22,280 --> 00:37:24,160
Filtering must be done close to the source.

826
00:37:24,580 --> 00:37:26,460
So it's basically the anti-ISPs responsibility.

827
00:37:27,220 --> 00:37:28,440
And you know many ISPs.

828
00:37:28,600 --> 00:37:30,860
As I showed 56% of the ISPs.

829
00:37:30,900 --> 00:37:31,920
Already fixed their networks.

830
00:37:32,320 --> 00:37:34,980
But there are still plenty of ISPs around the world.

831
00:37:35,060 --> 00:37:36,820
Which just either don't know that this is a problem.

832
00:37:36,940 --> 00:37:37,460
Or don't care.

833
00:37:38,060 --> 00:37:40,260
So we will be left for next years.

834
00:37:40,360 --> 00:37:42,380
With the incompetent ISPs.

835
00:37:42,580 --> 00:37:43,680
So we will be left with the problem.

836
00:37:45,060 --> 00:37:45,820
And we will be left with new strategies as well.

837
00:37:46,380 --> 00:37:47,160
As we go.

838
00:37:48,100 --> 00:37:48,620
Okay.

839
00:37:48,780 --> 00:37:51,660
The second thing is network capacity.

840
00:37:51,860 --> 00:37:53,220
What can we do to avoid.

841
00:37:53,580 --> 00:37:55,600
Everyone to have gigantic pipes.

842
00:37:55,820 --> 00:37:57,360
In order to intake.

843
00:37:58,680 --> 00:38:00,500
The amplification attacks for example.

844
00:38:01,000 --> 00:38:02,140
And here I believe.

845
00:38:02,220 --> 00:38:04,200
We have a technological technical solution.

846
00:38:04,440 --> 00:38:05,680
Which is called BGP FlowSpec.

847
00:38:06,300 --> 00:38:08,280
I'm a big fan of BGP FlowSpec.

848
00:38:08,860 --> 00:38:10,380
BGP FlowSpec is basically.

849
00:38:10,500 --> 00:38:12,980
A protocol created on top of BGP.

850
00:38:13,900 --> 00:38:14,720
Which allows.

851
00:38:15,060 --> 00:38:17,440
You to serve firewall rules.

852
00:38:17,700 --> 00:38:20,840
Up the routing chain.

853
00:38:21,100 --> 00:38:21,500
So basically.

854
00:38:21,600 --> 00:38:22,420
Your router.

855
00:38:22,660 --> 00:38:23,700
Over a BGP session.

856
00:38:23,880 --> 00:38:25,140
Can tell to our routers.

857
00:38:25,200 --> 00:38:25,720
By the way.

858
00:38:25,880 --> 00:38:27,120
Here are my firewall rules.

859
00:38:27,240 --> 00:38:28,460
I don't want some of the traffic.

860
00:38:28,680 --> 00:38:29,780
I don't want the SSDP traffic.

861
00:38:30,220 --> 00:38:31,300
I want the amplification traffic.

862
00:38:31,520 --> 00:38:33,500
Drop it as close as possible.

863
00:38:33,880 --> 00:38:34,500
And this is great.

864
00:38:34,640 --> 00:38:36,340
Because it allows you to relieve the congestion.

865
00:38:36,520 --> 00:38:37,280
On the last link.

866
00:38:45,660 --> 00:38:47,140
We had a couple of years ago.

867
00:38:47,240 --> 00:38:49,080
We had a number of outages caused by that.

868
00:38:49,220 --> 00:38:49,600
I mean sorry.

869
00:38:49,820 --> 00:38:50,920
FlowSpec worked perfectly.

870
00:38:51,200 --> 00:38:52,800
It dropped all the attack traffic.

871
00:38:54,100 --> 00:38:55,540
100% of the attack traffic.

872
00:38:56,620 --> 00:38:57,880
Unfortunately it also dropped.

873
00:38:59,260 --> 00:39:00,140
Normal traffic.

874
00:39:00,480 --> 00:39:01,880
But these are technical problems.

875
00:39:02,380 --> 00:39:04,020
And I'm sure they are fixable.

876
00:39:04,200 --> 00:39:05,320
And I'm sure they are all fixed already.

877
00:39:05,680 --> 00:39:07,160
This particular outage was caused.

878
00:39:07,220 --> 00:39:10,080
By us putting IPv6 IP addresses.

879
00:39:10,220 --> 00:39:11,000
In FlowSpec rules.

880
00:39:11,060 --> 00:39:12,400
Which was not supported at the time.

881
00:39:15,060 --> 00:39:17,120
And then the FlowSpec shines.

882
00:39:17,660 --> 00:39:20,200
Because it can cross AS boundaries.

883
00:39:20,740 --> 00:39:22,280
It's a firewall language.

884
00:39:22,920 --> 00:39:24,840
Which can basically cross ASs.

885
00:39:25,020 --> 00:39:26,140
So that's again great.

886
00:39:26,340 --> 00:39:28,760
You can basically move your rules up the chain.

887
00:39:29,260 --> 00:39:29,620
Now.

888
00:39:29,980 --> 00:39:30,540
Again.

889
00:39:30,640 --> 00:39:31,220
I won't lie.

890
00:39:31,420 --> 00:39:32,960
Most people that use FlowSpec.

891
00:39:33,020 --> 00:39:34,220
Use it within one domain.

892
00:39:34,420 --> 00:39:35,380
Within their AS number.

893
00:39:35,540 --> 00:39:36,880
It doesn't cross the boundary.

894
00:39:37,440 --> 00:39:39,400
So it's not as useful.

895
00:39:39,400 --> 00:39:40,440
But I think it is.

896
00:39:40,560 --> 00:39:41,420
It should be deployed.

897
00:39:41,980 --> 00:39:45,300
Kind of as the cross AS thing.

898
00:39:45,460 --> 00:39:47,080
And then there is.

899
00:39:47,220 --> 00:39:48,760
At least one success story.

900
00:39:49,200 --> 00:39:50,000
In Russia.

901
00:39:50,520 --> 00:39:52,060
So Ruscom I think a year ago.

902
00:39:52,300 --> 00:39:53,540
Created a service.

903
00:39:53,700 --> 00:39:55,380
Which basically enables their customers.

904
00:39:55,540 --> 00:39:57,820
To send FlowSpec firewall rules.

905
00:39:57,920 --> 00:39:58,560
Up the chain.

906
00:39:58,780 --> 00:40:01,160
So there is at least one internet carrier.

907
00:40:01,280 --> 00:40:02,400
That does this thing right.

908
00:40:02,640 --> 00:40:03,260
So it is possible.

909
00:40:03,860 --> 00:40:04,740
So next time.

910
00:40:04,800 --> 00:40:06,960
You choose your bandwidth.

911
00:40:07,560 --> 00:40:08,000
Vendor.

912
00:40:08,160 --> 00:40:09,900
You choose who to buy the bandwidth from.

913
00:40:10,620 --> 00:40:11,640
Ask about FlowSpec.

914
00:40:13,620 --> 00:40:14,060
Okay.

915
00:40:14,120 --> 00:40:14,700
And finally.

916
00:40:14,860 --> 00:40:15,860
What about tracing bug.

917
00:40:16,080 --> 00:40:18,400
Can we ever find out who is behind the attacks.

918
00:40:19,180 --> 00:40:20,240
Can we know that.

919
00:40:20,480 --> 00:40:21,420
The answer is yes.

920
00:40:21,660 --> 00:40:22,080
And again.

921
00:40:22,140 --> 00:40:23,000
There is a technical solution.

922
00:40:23,140 --> 00:40:23,840
Which is called NetFlow.

923
00:40:26,160 --> 00:40:27,040
And again.

924
00:40:33,260 --> 00:40:33,600
We use routers.

925
00:40:33,800 --> 00:40:35,900
To send packet samples.

926
00:40:36,360 --> 00:40:37,200
Actually not packet.

927
00:40:37,280 --> 00:40:37,940
But flow samples.

928
00:40:38,180 --> 00:40:39,100
To a central location.

929
00:40:39,700 --> 00:40:41,660
But this time within one AS number.

930
00:40:41,840 --> 00:40:42,940
Within your domain.

931
00:40:43,100 --> 00:40:43,720
So it doesn't cross.

932
00:40:44,040 --> 00:40:45,580
So there is no privacy issues.

933
00:40:45,660 --> 00:40:47,980
To cross data across ASs.

934
00:40:48,300 --> 00:40:50,140
And this data can be collected.

935
00:40:50,320 --> 00:40:50,980
And can be analyzed.

936
00:40:51,860 --> 00:40:53,340
And with this data.

937
00:40:53,480 --> 00:40:54,640
If configured properly.

938
00:40:55,020 --> 00:40:55,980
With good tool chain.

939
00:41:03,260 --> 00:41:04,340
Then your customers actually.

940
00:41:04,700 --> 00:41:06,300
Originated the traffic.

941
00:41:06,640 --> 00:41:07,620
Who is actually behind it.

942
00:41:07,700 --> 00:41:09,980
Who is actually being paid money.

943
00:41:10,180 --> 00:41:13,240
To deliver the spoof traffic.

944
00:41:13,840 --> 00:41:14,240
So yeah.

945
00:41:14,280 --> 00:41:15,120
So we could just call them.

946
00:41:15,220 --> 00:41:15,840
In such a case.

947
00:41:15,920 --> 00:41:16,600
And ask.

948
00:41:16,740 --> 00:41:18,600
And they will just run a simple command line tool.

949
00:41:18,960 --> 00:41:20,880
And basically be able to answer those questions.

950
00:41:21,360 --> 00:41:22,240
But once again.

951
00:41:22,300 --> 00:41:23,180
The point is right now.

952
00:41:23,240 --> 00:41:23,560
It's impossible.

953
00:41:24,620 --> 00:41:27,480
The ISPs do not support FlowSpec.

954
00:41:33,260 --> 00:41:33,780
They don't say that okay.

955
00:41:34,200 --> 00:41:36,040
Using NetFlow and basically logging traffic.

956
00:41:36,320 --> 00:41:38,140
On disk ISPs is bad.

957
00:41:38,860 --> 00:41:39,980
For the privacy.

958
00:41:40,800 --> 00:41:41,540
This is true.

959
00:41:42,000 --> 00:41:43,840
I'm basically asking for more logging.

960
00:41:44,140 --> 00:41:45,280
But there are ways around it.

961
00:41:45,460 --> 00:41:47,120
There are ways to find a compromise.

962
00:41:47,880 --> 00:41:48,720
So for example.

963
00:41:48,900 --> 00:41:51,080
I'm only interested in the really, really, really.

964
00:41:51,200 --> 00:41:52,300
Big, big, big attacks.

965
00:41:52,580 --> 00:41:54,480
I'm only interested in the gigantic spikes.

966
00:41:54,660 --> 00:41:56,660
Which threaten the stability of the internet.

967
00:41:57,000 --> 00:41:59,340
I really don't care about your customer connections.

968
00:41:59,800 --> 00:42:01,500
So you can set the sampling rate.

969
00:42:03,260 --> 00:42:04,740
Or largest depending how you look at that.

970
00:42:05,480 --> 00:42:06,140
Maximum value.

971
00:42:06,340 --> 00:42:08,540
So which is one I think in 64,000 of flows.

972
00:42:08,820 --> 00:42:11,960
So unless you are sending 64,000 of flows.

973
00:42:12,040 --> 00:42:12,320
A second.

974
00:42:12,420 --> 00:42:15,740
Your privacy should be reasonably preserved.

975
00:42:16,200 --> 00:42:17,340
And second thing is.

976
00:42:17,440 --> 00:42:19,200
I care only about recent attacks.

977
00:42:19,380 --> 00:42:20,680
I care only about last whatever.

978
00:42:20,820 --> 00:42:22,900
48 hours of traffic.

979
00:42:23,100 --> 00:42:25,300
So the logs can be aggressively rotated.

980
00:42:25,600 --> 00:42:26,340
This is fine.

981
00:42:28,460 --> 00:42:28,920
Okay.

982
00:42:28,960 --> 00:42:29,760
So there is a tangent.

983
00:42:29,880 --> 00:42:31,840
That it is a bit harder for the internet exchanges.

984
00:42:33,260 --> 00:42:34,920
But again there are technical solutions to that.

985
00:42:35,260 --> 00:42:36,860
One of the solutions is called.

986
00:42:37,000 --> 00:42:37,840
Mug accounting.

987
00:42:38,060 --> 00:42:40,700
Which is a feature in our routers.

988
00:42:40,800 --> 00:42:42,240
Which allows us to do something.

989
00:42:42,300 --> 00:42:44,740
And get more statistics about the mug addresses.

990
00:42:44,740 --> 00:42:45,420
On that network.

991
00:42:45,820 --> 00:42:47,220
And the second thing is.

992
00:42:48,180 --> 00:42:49,320
Is that internet exchanges.

993
00:42:49,600 --> 00:42:52,300
So this kind of no man's land between ISPs.

994
00:42:52,440 --> 00:42:55,280
Should play more active role in guarding the internet.

995
00:42:55,460 --> 00:42:57,520
Internet exchanges should have.

996
00:42:57,700 --> 00:42:59,740
Should provide tools to figure out.

997
00:42:59,840 --> 00:43:01,080
Where the attacks are originated.

998
00:43:01,440 --> 00:43:01,720
Okay.

999
00:43:03,260 --> 00:43:03,640
All right.

1000
00:43:04,140 --> 00:43:04,700
That's it.

1001
00:43:04,880 --> 00:43:06,160
So a quick recap is.

1002
00:43:06,800 --> 00:43:08,700
In order to prevent the IP spoofing.

1003
00:43:08,760 --> 00:43:10,300
We have to promote BCP38.

1004
00:43:10,700 --> 00:43:12,340
This is the root of the actual problem.

1005
00:43:12,660 --> 00:43:15,200
But unfortunately it will not fix it in short time.

1006
00:43:15,560 --> 00:43:16,420
In the meantime.

1007
00:43:16,520 --> 00:43:18,080
We should definitely promote flow spec.

1008
00:43:18,740 --> 00:43:20,840
Because this allows smaller players.

1009
00:43:20,960 --> 00:43:23,660
To avoid buying gigantic network capacity.

1010
00:43:23,960 --> 00:43:25,040
So basically this will.

1011
00:43:26,040 --> 00:43:28,300
Enable smaller players to survive big attacks.

1012
00:43:28,620 --> 00:43:29,540
And finally.

1013
00:43:29,700 --> 00:43:30,680
Net flow is necessary.

1014
00:43:33,260 --> 00:43:34,340
To figure out who is behind attacks.

1015
00:43:34,480 --> 00:43:35,060
To trace them back.

1016
00:43:35,220 --> 00:43:36,740
And basically solve the problem.

1017
00:43:37,920 --> 00:43:40,440
By just putting pressure on the attacking parties.

1018
00:43:41,940 --> 00:43:42,760
Thank you very much.

1019
00:43:44,120 --> 00:43:44,780
That's it.

1020
00:43:45,060 --> 00:43:45,800
That's all I have.

1021
00:43:52,440 --> 00:43:52,860
Actually.

1022
00:43:53,000 --> 00:43:53,740
I'm lying.

1023
00:43:53,900 --> 00:43:55,020
I have 30 more slides.

1024
00:43:55,220 --> 00:43:56,440
With more technical details.

1025
00:43:56,800 --> 00:43:57,960
But I think we should stop here.

1026
00:43:57,960 --> 00:43:58,060
Thank you.

1027
00:43:59,080 --> 00:44:00,620
Anyone have any questions?

1028
00:44:01,100 --> 00:44:01,360
Please.

1029
00:44:01,820 --> 00:44:02,420
You're welcome.

1030
00:44:02,520 --> 00:44:03,160
Take the microphone.

1031
00:44:08,540 --> 00:44:09,100
Anyone?

1032
00:44:09,480 --> 00:44:11,060
I have a t-shirt.

1033
00:44:11,180 --> 00:44:11,740
To give away.

1034
00:44:15,560 --> 00:44:16,120
.

1035
00:44:17,020 --> 00:44:17,580
.

1036
00:44:17,580 --> 00:44:17,720
.

1037
00:44:17,720 --> 00:44:18,020
.

1038
00:44:18,020 --> 00:44:18,840
.

1039
00:44:18,840 --> 00:44:19,060
.

1040
00:44:21,220 --> 00:44:22,620
.

1041
00:44:25,020 --> 00:44:25,720
.

1042
00:44:30,780 --> 00:44:32,180
.

1043
00:44:32,180 --> 00:44:32,200
.

1044
00:44:32,200 --> 00:44:33,020
.

1045
00:44:33,020 --> 00:44:33,120
.

1046
00:44:33,120 --> 00:44:33,140
.

1047
00:44:33,140 --> 00:44:33,180
.

1048
00:44:33,180 --> 00:44:34,140
.

1049
00:44:36,920 --> 00:44:38,320
.

1050
00:44:39,740 --> 00:44:41,140
.

1051
00:44:44,500 --> 00:44:45,200
.

1052
00:44:45,200 --> 00:44:45,640
.

1053
00:44:56,900 --> 00:44:57,620
.

1054
00:44:57,620 --> 00:44:59,040
.

1055
00:44:59,780 --> 00:45:00,500
.

1056
00:45:00,500 --> 00:45:01,200
.

1057
00:45:01,200 --> 00:45:01,220
.

1058
00:45:01,220 --> 00:45:01,360
.

1059
00:45:01,360 --> 00:45:01,640
.

1060
00:45:01,640 --> 00:45:01,760
.

1061
00:45:01,760 --> 00:45:01,800
.

1062
00:45:10,380 --> 00:45:11,100
.

1063
00:45:11,100 --> 00:45:11,140
.

1064
00:45:11,900 --> 00:45:11,980
.

1065
00:45:19,960 --> 00:45:20,040
.

1066
00:45:23,620 --> 00:45:23,700
.

1067
00:45:23,700 --> 00:45:23,740
.

1068
00:45:24,260 --> 00:45:24,340
.

1069
00:45:24,340 --> 00:45:24,360
.

1070
00:45:24,360 --> 00:45:24,380
.

1071
00:45:24,380 --> 00:45:24,400
.

1072
00:45:24,400 --> 00:45:24,420
.

1073
00:45:24,420 --> 00:45:24,460
.

1074
00:45:35,880 --> 00:45:35,960
.

1075
00:45:35,960 --> 00:45:36,000
.

1076
00:45:41,160 --> 00:45:41,580
.

1077
00:45:49,600 --> 00:45:50,020
.

1078
00:45:57,400 --> 00:45:57,820
.

1079
00:45:57,820 --> 00:45:57,840
.

1080
00:45:57,840 --> 00:45:58,060
.

1081
00:45:58,060 --> 00:45:58,220
.

1082
00:45:58,220 --> 00:45:58,420
.

1083
00:45:58,420 --> 00:45:58,440
.

1084
00:45:58,440 --> 00:45:58,460
.

1085
00:45:58,460 --> 00:45:58,500
.

1086
00:45:58,500 --> 00:45:58,540
.

1087
00:45:59,420 --> 00:45:59,840
.

1088
00:46:02,980 --> 00:46:03,400
.

1089
00:46:07,180 --> 00:46:07,600
.

1090
00:46:08,400 --> 00:46:08,560
.

1091
00:46:23,000 --> 00:46:23,160
.

1092
00:46:25,000 --> 00:46:25,160
.

1093
00:46:25,160 --> 00:46:25,240
.

1094
00:46:25,240 --> 00:46:25,260
.

1095
00:46:25,260 --> 00:46:25,280
.

1096
00:46:25,280 --> 00:46:25,300
.

1097
00:46:25,300 --> 00:46:25,320
.

1098
00:46:28,540 --> 00:46:28,700
.

1099
00:46:32,900 --> 00:46:33,060
.

1100
00:46:33,060 --> 00:46:33,140
.

1101
00:46:38,740 --> 00:46:39,400
.

1102
00:46:49,900 --> 00:46:50,560
.

1103
00:46:53,580 --> 00:46:54,240
.

1104
00:46:54,240 --> 00:46:54,400
.

1105
00:46:55,740 --> 00:46:56,400
.

1106
00:46:56,400 --> 00:46:56,580
.

1107
00:46:56,580 --> 00:46:56,600
.

1108
00:46:56,600 --> 00:46:56,620
.

1109
00:46:56,620 --> 00:46:56,960
.

1110
00:46:56,960 --> 00:46:57,280
.

1111
00:47:00,780 --> 00:47:01,440
.

1112
00:47:01,440 --> 00:47:01,580
.

1113
00:47:01,580 --> 00:47:01,600
.

1114
00:47:04,640 --> 00:47:05,300
.

1115
00:47:06,760 --> 00:47:07,240
.

1116
00:47:15,700 --> 00:47:16,180
.

1117
00:47:17,340 --> 00:47:17,820
.

1118
00:47:17,820 --> 00:47:18,060
.

1119
00:47:18,060 --> 00:47:18,440
.

1120
00:47:18,440 --> 00:47:18,460
.

1121
00:47:18,460 --> 00:47:18,960
.

1122
00:47:18,960 --> 00:47:18,980
.

1123
00:47:18,980 --> 00:47:19,000
.

1124
00:47:19,000 --> 00:47:19,200
.

1125
00:47:19,200 --> 00:47:19,240
.

1126
00:47:35,300 --> 00:47:42,240
have probes, and they run probes on the internet. I think nowadays they run their software on

1127
00:47:42,240 --> 00:47:46,200
your machines, so if you install their piece of software, it will basically probe the network

1128
00:47:46,200 --> 00:47:50,780
and upload the data. There are some privacy concerns because they don't want to reveal

1129
00:47:50,780 --> 00:47:54,680
your IP address, so there's kind of a bigger subject. But basically they do some active

1130
00:47:54,680 --> 00:48:00,580
probing. But how, what specifically spoofing they are talking about, is it spoofing within

1131
00:48:00,580 --> 00:48:08,160
the IP ranges of this ISP or globally, I don't know, but it's there. So spoofer.kaida.org

1132
00:48:08,160 --> 00:48:09,160
is the data source here.

1133
00:48:18,480 --> 00:48:25,580
Just curious, so the bandwidth attacks, the DDoSes get the big numbers, do you have any

1134
00:48:25,580 --> 00:48:28,140
idea, like relation, usually the app falls first, right?

1135
00:48:28,960 --> 00:48:30,460
Okay, the question is about...

1136
00:48:30,460 --> 00:48:37,640
What is the app falling first? Yes, in this talk we spoke specifically about layer three

1137
00:48:38,140 --> 00:48:43,760
attacks with spoofed IP addresses, so a very specific niche of the attacks. There are tons,

1138
00:48:43,860 --> 00:48:49,500
tons, tons of more attacks, L7 attacks. Interesting tangent, for example, is DNS attacks. So

1139
00:48:49,500 --> 00:48:56,460
we at Cloudflare operate DNS, and DNS is connectionless, it's UDP, so we see packets which are kind

1140
00:48:56,460 --> 00:49:00,320
of both L7 and both L3 because they are spoofed, but on the other hand they are legit.

1141
00:49:00,460 --> 00:49:04,540
So we're like, do we use DNS packets? And yes, the answer is usually yes. Our DNS application

1142
00:49:04,540 --> 00:49:09,460
will never be able to do 100 million packets per second, which we often see on DNS layers.

1143
00:49:10,400 --> 00:49:15,340
Sorry, we used to see. So yeah, so for that we are using raise limiting on IP tables

1144
00:49:15,340 --> 00:49:19,820
layer, but again, BPFs is the main methodology. So for the connectionless stuff, for the UDP

1145
00:49:19,820 --> 00:49:26,380
stuff, it's usually fairly easy to block and you see our DNS, sorry, BPF tools. For DL7

1146
00:49:26,380 --> 00:49:30,440
things on TCP, it's completely different subject. And you can see that the connectionless stuff,

1147
00:49:30,440 --> 00:49:36,420
and you always see the proper IP address there. So if the worst come to the end, you can

1148
00:49:36,420 --> 00:49:40,800
always block on the IP addresses. So it's way easier to mitigate. You can just blacklist

1149
00:49:41,160 --> 00:49:47,500
the whole internet. I mean, that was actually kind of touched on my question, which was

1150
00:49:47,500 --> 00:49:55,080
some of the stuff that was hitting TCP services on layer 7. Could you look for connections

1151
00:49:55,080 --> 00:50:00,140
that were not being set up properly and just blacklist them using some sort of automated

1152
00:50:00,440 --> 00:50:04,220
The question is, does the problem exist on TCP? No. If there is a TCP connection handshake,

1153
00:50:04,340 --> 00:50:07,280
then I know the other entity IP address, so.

1154
00:50:07,500 --> 00:50:13,100
Well, you had showed a TCP dump where you were showing connections on port 80.

1155
00:50:13,700 --> 00:50:18,640
These are SYN packets. So SYN flats are, it's the first packet in the connection handshake.

1156
00:50:18,760 --> 00:50:24,900
So SYN flats are fairly normal thing. Like everyone knows these deal with SYN flats. So it's

1157
00:50:24,900 --> 00:50:28,480
basically a layer 3 attack. So it's not a layer 7 attack then on SYN flats.

1158
00:50:28,580 --> 00:50:30,100
Sorry, but I mean, could you do...

1159
00:50:30,440 --> 00:50:37,520
Can you do stuff to mitigate based on, you know, sanity checks, whether or not the incoming

1160
00:50:37,520 --> 00:50:43,040
data was representing a legitimate user versus... Do you understand what I'm trying to ask?

1161
00:50:43,240 --> 00:50:48,400
Not really, but if you are asking about the details on the SYN packets, then again, BPF tools.

1162
00:50:48,580 --> 00:50:50,000
BPF tools is what we use for...

1163
00:50:50,000 --> 00:50:51,200
I'm wondering what I can do at an app layer.

1164
00:50:53,000 --> 00:50:55,060
Okay, so the app layer. So what about the app layer?

1165
00:50:55,580 --> 00:50:56,520
Higher level.

1166
00:50:56,800 --> 00:51:00,420
Okay. Can I ask myself a question?

1167
00:51:02,440 --> 00:51:08,620
So the Dyn attacks. I mentioned this talk is not about that. So from my point of view,

1168
00:51:08,680 --> 00:51:13,100
it's not super sure what were the Dyn attacks. They were the biggest published attack on

1169
00:51:13,100 --> 00:51:18,640
the internet with 620 gigabits per second. So why haven't I spent more time on that?

1170
00:51:18,940 --> 00:51:25,120
So first is that it's not obvious what exactly was hitting... It was Krebs before Dyn. So

1171
00:51:25,120 --> 00:51:28,320
it was hitting Krebs, this particular attack blog.

1172
00:51:28,960 --> 00:51:36,400
But from what we do know is that it was GRE packets, which... Without spoofed IP addresses.

1173
00:51:36,740 --> 00:51:44,240
And also what we do know is that the attack had large post uploads. So it is possible

1174
00:51:44,240 --> 00:51:48,900
this attack was not spoofed. So that's why I haven't mentioned it here. So yes, there

1175
00:51:48,900 --> 00:51:53,380
are... Sometimes it happens that there are big attacks which are actually from legitimate

1176
00:51:53,380 --> 00:51:56,520
IP addresses. But then I would argue it's fairly easy to block them.

1177
00:51:59,660 --> 00:52:00,360
Thank you very much.