1
00:00:08,460 --> 00:00:10,760
Alright, this is our agenda for today.

2
00:00:10,760 --> 00:00:13,240
First, we'll be going over what WAFs are.

3
00:00:13,240 --> 00:00:15,140
I think many of you already know what they are.

4
00:00:15,140 --> 00:00:17,100
Otherwise, you probably wouldn't be here.

5
00:00:17,400 --> 00:00:20,180
Just so we start with an even play for everyone.

6
00:00:20,460 --> 00:00:26,920
Then, we'll go over what we've learned at Datadog operating a very large-scale WAF and the struggle that comes with it.

7
00:00:27,340 --> 00:00:32,260
We'll try to address a bit how we deal with false positives and false negatives.

8
00:00:32,580 --> 00:00:38,040
And then, we'll finish by sharing how we try to make them a bit better.

9
00:00:38,600 --> 00:00:40,880
Alright, who are we first?

10
00:00:41,460 --> 00:00:42,260
I'm Emmanuelle.

11
00:00:42,260 --> 00:00:43,760
I'm a software engineer at Datadog.

12
00:00:43,760 --> 00:00:44,760
I'm based in Paris.

13
00:00:44,760 --> 00:00:47,420
And I've worked on machine learning models for the past four years.

14
00:00:47,420 --> 00:00:49,300
So, not a security background at all.

15
00:00:49,460 --> 00:00:53,940
And actually, last year, I decided to join Datadog to work on our application security product.

16
00:00:54,440 --> 00:00:56,220
And this is my co-presenter, Emile.

17
00:00:56,860 --> 00:00:58,500
So, I'm Emile Gospier.

18
00:00:58,500 --> 00:01:02,780
I'm a security researcher at Datadog, also based in Paris.

19
00:01:02,780 --> 00:01:06,460
I've been working on a WAF and a RASP solution for the last five years.

20
00:01:06,560 --> 00:01:09,940
First, at Screen, then Datadog, following an acquisition in 2021.

21
00:01:10,500 --> 00:01:12,300
And, yeah, let's talk about WAFs.

22
00:01:12,300 --> 00:01:13,660
You may know about them.

23
00:01:14,020 --> 00:01:26,600
If you don't, that's a security tool that was created in the mid-90s, when web applications started to become more complex, more useful, and thus, a more enticing target.

24
00:01:27,120 --> 00:01:32,160
We didn't really have anything to do to help protect them.

25
00:01:32,160 --> 00:01:36,160
So, we looked at other tools, and a promising one was firewalls.

26
00:01:36,160 --> 00:01:38,060
They're working really great at the network level.

27
00:01:38,060 --> 00:01:42,720
So, the idea was, let's take them and bring them to the application layer.

28
00:01:43,080 --> 00:01:54,280
And instead of running on specific rules, like this IP, this port, this whatever, it was more, yeah, let's run regexes or dictionaries to find suspicious patterns in that.

29
00:01:54,280 --> 00:01:57,540
And if you find them in the incoming request, that's probably because it's bad.

30
00:01:58,460 --> 00:02:00,920
And, yeah, that's pretty much it.

31
00:02:03,080 --> 00:02:07,180
WAFs today are pretty much the backbone of runtime application security.

32
00:02:07,180 --> 00:02:08,760
And that's for a few reasons.

33
00:02:08,760 --> 00:02:11,460
First, they are super easy to explain.

34
00:02:11,500 --> 00:02:15,320
In just a few words, I was able to give you a good view of what they are.

35
00:02:16,180 --> 00:02:18,440
They're also incredibly easy to customize.

36
00:02:18,440 --> 00:02:26,380
Because they're based on regexes and dictionaries, you have an entire industry of rulesets that you can use, or you can even customize them yourself.

37
00:02:26,860 --> 00:02:28,640
They're also really easy to demo.

38
00:02:28,640 --> 00:02:34,220
All you need to do to show that your WAF works is plug it on the Internet.

39
00:02:34,500 --> 00:02:41,160
And just the noise from the Internet is going to create a bunch of alerts, and, hey, you have a great security tool, you're protecting everyone.

40
00:02:42,580 --> 00:02:45,660
And finally, it never hurts, it's actually recorded by compliance.

41
00:02:45,800 --> 00:02:51,800
So if you're dealing with credit cards, yeah, you need a WAF, basically.

42
00:02:51,940 --> 00:02:56,860
So, yeah, all of that came together to make WAF an incredibly popular product.

43
00:02:58,840 --> 00:03:03,400
However, WAF tends to give a bit of a false sense of security.

44
00:03:03,440 --> 00:03:08,400
Because what a WAF does is that it looks for suspicious requests.

45
00:03:08,400 --> 00:03:11,240
It's running regexes, dictionaries, that's what it does.

46
00:03:11,540 --> 00:03:19,040
But then it means that if you can make your malicious attack look unsuspicious, then it's going to go through.

47
00:03:19,560 --> 00:03:27,900
And if you happen to have a user that sends another request that happens to look suspicious, they need to be matched.

48
00:03:28,340 --> 00:03:31,760
And there is not really anything you can do about it.

49
00:03:31,760 --> 00:03:33,340
That's WAF working as design.

50
00:03:34,200 --> 00:03:40,640
So, yeah, a few alternatives were tried over time, like RASP, IASP, but they never really went anywhere.

51
00:03:40,960 --> 00:03:43,460
No solution really got anywhere as popular as WAF.

52
00:03:45,160 --> 00:03:46,680
We talked about what they are.

53
00:03:46,980 --> 00:03:49,300
Emmanuel is going to tell you about how to make one.

54
00:03:49,300 --> 00:03:50,800
Yeah, how do we build a WAF?

55
00:03:50,800 --> 00:03:54,240
Well, first, of course, you need a web application because that's what you want to protect.

56
00:03:54,240 --> 00:03:59,600
In front of that, most of the time, you're going to put a reverse proxy to route all the incoming traffic to your web application.

57
00:04:00,020 --> 00:04:00,680
All right.

58
00:04:00,900 --> 00:04:02,500
Now, let's build a WAF.

59
00:04:02,600 --> 00:04:05,160
First element that we need is a rule engine, of course.

60
00:04:05,160 --> 00:04:07,760
We're going to put it on the reverse proxy because it's very convenient.

61
00:04:07,760 --> 00:04:12,960
It's where we're going to be able to see all the incoming traffic and filter out the malicious request if we need to.

62
00:04:13,800 --> 00:04:15,960
Second element we need, a rule set.

63
00:04:16,360 --> 00:04:21,020
Here, we're mentioning the OWASP core rule set because it's a very popular one across the industry.

64
00:04:21,020 --> 00:04:25,240
It's open source, so you can also use it to build your own rule set on top of it.

65
00:04:25,240 --> 00:04:29,080
Here, you can see we mentioned PL1 for Paranoia Level 1.

66
00:04:29,580 --> 00:04:36,140
What it means is it's supposed to be not too noisy, so it's supposed to be a set of rules that shouldn't trigger too many alerts for us.

67
00:04:36,160 --> 00:04:36,580
Okay.

68
00:04:36,580 --> 00:04:37,160
All right.

69
00:04:37,160 --> 00:04:37,740
Well, good.

70
00:04:37,740 --> 00:04:39,620
Last element we need, the monitoring.

71
00:04:40,200 --> 00:04:46,660
It can be just a text file where you push all of your alerts, or it can be a super fancy dashboard that you've built.

72
00:04:48,220 --> 00:04:48,940
Okay.

73
00:04:49,200 --> 00:04:49,880
We're all set.

74
00:04:49,880 --> 00:04:51,240
I think we can activate the WAF.

75
00:04:51,240 --> 00:04:53,900
And this is more or less what happened the first time we did it.

76
00:04:53,900 --> 00:04:55,480
We started crying.

77
00:04:55,740 --> 00:04:57,600
We got a bunch of alerts.

78
00:04:57,640 --> 00:04:59,880
And we were like, okay, what do we do with this?

79
00:05:00,360 --> 00:05:04,640
And we took a step back and were like, okay, this is actually a great learning experiment.

80
00:05:05,080 --> 00:05:06,560
So what did we learn, Emil?

81
00:05:08,640 --> 00:05:14,240
So the learnings we are going to share are based on us operating a WAF at a pretty large scale.

82
00:05:14,240 --> 00:05:18,840
We are talking around a billion requests per second across hundreds of thousands of organizations.

83
00:05:20,060 --> 00:05:28,420
And the first and most important learning we took away from that is that the accuracy of a WAF rule is totally unpredictable at scale.

84
00:05:28,620 --> 00:05:30,800
You can think that your rule is perfect.

85
00:05:30,960 --> 00:05:33,380
Until it's tested, you have no idea.

86
00:05:35,600 --> 00:05:38,560
So they will tend to converge quite fast.

87
00:05:38,780 --> 00:05:45,060
So if you can test them on a broad enough set of data, within a few hours you will have a good idea of how it's going to behave.

88
00:05:45,060 --> 00:05:59,640
But if you only tested them on WordPress or CMS, and you deployed them on an e-commerce website, I mean, you can have a French customer going to share their address, which has a quote and OR, and then the WAF is going to freak out.

89
00:05:59,640 --> 00:06:02,320
I mean, of course, this is a SQL injection, no?

90
00:06:03,020 --> 00:06:03,880
Yeah.

91
00:06:05,580 --> 00:06:19,780
The other important consequence from that is that even if you do very simple modifications, like you just expand it a little bit further, like you add a couple of duplication steps, you need to test them again because you don't know what you are missing before.

92
00:06:20,720 --> 00:06:30,120
The other side of this coin, however, is that even if you have a crazy rule that shouldn't work, it's usually worth testing them just in case.

93
00:06:30,460 --> 00:06:40,820
To illustrate that, we're going to go back like to late 2021, when AppSec just caught fire, and we had log4j drop on us a few weeks before Christmas.

94
00:06:41,440 --> 00:06:48,760
And for those of you who don't know, it was an RCE in the log4j library, which is used by Java application to log.

95
00:06:48,760 --> 00:06:52,220
And if you don't know, Java applications love to log.

96
00:06:53,340 --> 00:07:03,740
So when it came out, the initial PLC was something like that, where if you could get log4j to log this, you had an RCE.

97
00:07:04,780 --> 00:07:07,160
So yeah, everyone started scanning all over the place.

98
00:07:07,860 --> 00:07:13,680
Pretty much every WAF vendor or user set up their own pattern, and we did too.

99
00:07:13,720 --> 00:07:15,560
The first version was something like that.

100
00:07:15,560 --> 00:07:24,200
So, okay, we just match gndi, which is the entry point of the bug in log4j, and then pretty much everything that follows.

101
00:07:24,440 --> 00:07:25,720
We test it.

102
00:07:25,840 --> 00:07:26,600
Okay, it's fine.

103
00:07:26,600 --> 00:07:28,000
It triggers a little bit.

104
00:07:28,400 --> 00:07:30,400
No obvious pattern of false positive.

105
00:07:30,400 --> 00:07:31,440
Okay, good to go.

106
00:07:31,440 --> 00:07:32,300
We can ship it.

107
00:07:33,200 --> 00:07:34,960
And yeah, it worked well.

108
00:07:35,200 --> 00:07:39,760
Very quickly, however, the attackers tried to find bypasses for that.

109
00:07:39,760 --> 00:07:42,280
So the first one they found was they could use a different schema.

110
00:07:42,320 --> 00:07:48,720
The initial PLC used LDAP to perform the RCE, but actually you could use another schema, so they just did.

111
00:07:49,300 --> 00:07:53,960
In our case, we were not really looking at LDAP, so we didn't really care.

112
00:07:54,020 --> 00:07:58,340
But then they decided to step things up.

113
00:07:58,340 --> 00:08:05,440
So they started to use the string interpolation features of log4j to obfuscate their pattern further.

114
00:08:06,300 --> 00:08:09,580
So, okay, we need to complexify quite a bit the pattern.

115
00:08:09,580 --> 00:08:16,960
So every single character we had before became this whole mess to support some amount of obfuscation.

116
00:08:17,080 --> 00:08:19,460
Then attackers decided to step things up further.

117
00:08:19,460 --> 00:08:23,080
So, okay, now you have completely spurious blocks that are just going to disappear.

118
00:08:23,720 --> 00:08:26,300
Okay, we can expand the pattern further.

119
00:08:26,300 --> 00:08:28,460
This is becoming a bit unwieldy.

120
00:08:28,460 --> 00:08:33,060
For each character you have in the initial pattern, you have this whole mess there.

121
00:08:33,600 --> 00:08:36,120
And attackers just got crazy.

122
00:08:38,640 --> 00:08:43,660
This could either be the most complex waffle ever, or we need a different type strategy.

123
00:08:43,660 --> 00:08:48,160
So when we saw that, we basically tested two rules.

124
00:08:48,200 --> 00:08:53,700
One monstrosity that could deal with that, or something much simpler.

125
00:08:55,080 --> 00:08:59,760
And we are shocked, but this never triggered the false positive.

126
00:09:00,840 --> 00:09:06,720
Like, that's why we're telling you that you need to test rules, because this shouldn't work.

127
00:09:08,160 --> 00:09:14,720
And, yeah, the last bypasses we found at the end were attempts to use non-ASCII characters.

128
00:09:14,820 --> 00:09:18,280
So here, like, log4j would do some kind of unicode normalization.

129
00:09:18,860 --> 00:09:20,560
Yeah, at this point you need to do the same.

130
00:09:20,560 --> 00:09:21,900
And that's expensive.

131
00:09:24,660 --> 00:09:30,340
Yeah, and when we put this in production, it triggered roughly half a million triggers in a week.

132
00:09:30,340 --> 00:09:33,640
And that was before the acquisition of Datadog, so at a much smaller scale.

133
00:09:35,700 --> 00:09:42,700
We told you about how you design a rule, but there is a greater framework behind it and some constraints.

134
00:09:42,700 --> 00:09:45,860
So let Emmanuel tell you about the dilemma of building a WAF rule.

135
00:09:45,860 --> 00:09:52,200
Yes, so when you build a WAF rule, we try to kind of display this into a graph.

136
00:09:53,300 --> 00:09:59,620
On the horizontal axis, you can have a very broad rule that will catch everything, or a very specific rule.

137
00:09:59,880 --> 00:10:10,160
And on the vertical axis, you can have either a rule that's very easy to bypass, but doesn't give you too many false positives, or here on the top, it triggers a lot of false positives, but it's hard to bypass.

138
00:10:10,880 --> 00:10:25,280
All right, based on that, we try to define a max amount of noise that we're willing to accept, meaning if a rule starts triggering more false positives than this threshold, we're not even going to put it into prediction because it's not worth the hassle.

139
00:10:26,180 --> 00:10:33,840
Okay, now an ideal rule we could try to build would be something very broad that's going to catch every XSS attack that we're getting.

140
00:10:34,360 --> 00:10:34,940
Dope.

141
00:10:34,940 --> 00:10:38,380
And it doesn't trigger too many false positives and it's not too easy to bypass.

142
00:10:38,380 --> 00:10:40,140
All right, it's right in there in the middle.

143
00:10:40,140 --> 00:10:50,180
On the other side, you could also try to build a rule that's very, very specific, meaning it could only target a very specific XSS vulnerability for Jupyter notebooks.

144
00:10:51,160 --> 00:10:54,560
And same, not too easy to bypass and not too many false positives.

145
00:10:54,820 --> 00:10:55,380
Dope.

146
00:10:55,420 --> 00:11:02,660
Well, that would be an ideal rule because actually what happens most of the time when you try to build this rule is you end up right there.

147
00:11:02,860 --> 00:11:09,440
If you do a very broad rule, you're going to have a lot of false positives and if you do a very specific rule, you're going to be bypassed very easily.

148
00:11:11,020 --> 00:11:12,220
So what happens?

149
00:11:12,540 --> 00:11:16,900
Most of the time, you're going to be in this yellow area right there, the space of operation.

150
00:11:16,920 --> 00:11:20,120
And what you want to do, ideally, is to be right there in the middle.

151
00:11:20,120 --> 00:11:24,140
So you are doing, kind of, having the best of both worlds, so to speak.

152
00:11:25,300 --> 00:11:30,160
All right, so we've talked about how we don't want too many false positives because they are nasty.

153
00:11:30,520 --> 00:11:34,000
Well, let me tell you or kind of remind you why we don't want them.

154
00:11:34,080 --> 00:11:36,300
First, they are going to create an alert fatigue.

155
00:11:36,300 --> 00:11:40,260
So, yeah, there is nothing worse than you come into work and you see 10,000 plus alerts.

156
00:11:40,260 --> 00:11:41,380
What do I do with this?

157
00:11:41,680 --> 00:11:45,020
It's not worth looking at it and it's not actionable.

158
00:11:45,840 --> 00:11:47,260
Second reason why.

159
00:11:47,260 --> 00:11:49,280
Well, it can have a business impact.

160
00:11:49,560 --> 00:11:56,180
If you have rules in blocking mode, you might be blocking legitimate customers with a wrong setup.

161
00:11:56,460 --> 00:11:58,760
And if you're blocking customers, you're losing business.

162
00:11:58,760 --> 00:12:00,260
If you're losing business, you're losing money.

163
00:12:00,260 --> 00:12:01,640
So you don't want that either.

164
00:12:01,860 --> 00:12:06,740
And the last reason why you don't want false positives is because you might end up collecting PII by accident.

165
00:12:06,740 --> 00:12:11,940
Meaning you might collect some private information like addresses, emails, phone numbers.

166
00:12:11,940 --> 00:12:14,080
And this might end up in your log triggers.

167
00:12:14,920 --> 00:12:17,920
And because of that, you're going to have a security issue on your hands.

168
00:12:17,920 --> 00:12:19,580
So , yeah, you don't want that either.

169
00:12:20,720 --> 00:12:24,520
Overall, what you need to keep in mind is that it's all about finding this compromise.

170
00:12:24,520 --> 00:12:28,900
You want a rule that doesn't have too many false positives but still brings you value.

171
00:12:30,420 --> 00:12:31,380
All right.

172
00:12:31,820 --> 00:12:38,980
In all this time working on false positives, at Datadog, we've been able to see a bunch of them and we've tried to classify them into different types of families.

173
00:12:39,400 --> 00:12:44,500
The first family we've encountered most of the time is based on what they are.

174
00:12:44,600 --> 00:12:48,440
Meaning you get a bunch of triggers coming from free text.

175
00:12:48,760 --> 00:12:51,680
For example, you're running an application like Stack Overflow.

176
00:12:51,680 --> 00:12:56,080
You have a bunch of people pasting SQL injection all the time because they're asking questions about it.

177
00:12:56,080 --> 00:13:00,420
And this will trigger your WAF if it's set up in a way where it looks at free text.

178
00:13:01,380 --> 00:13:12,680
Second family you can have is if you're running any type of infrastructure that's having an admin panel, running SQL queries against your DB.

179
00:13:12,680 --> 00:13:15,400
This is going to look like SQL injection to your WAF.

180
00:13:15,400 --> 00:13:19,100
So let's say you're running phpMyAdmin that could happen most of the time.

181
00:13:19,960 --> 00:13:27,660
And also, if your application is relying on some concept like you're doing a lot of queries all over the place.

182
00:13:27,660 --> 00:13:29,920
For example, at Datadog, that's an issue that we have.

183
00:13:29,920 --> 00:13:34,400
We're running a ton of queries everywhere and it looks like SQL injection a lot of the time.

184
00:13:34,400 --> 00:13:35,820
So what do we do with that?

185
00:13:36,300 --> 00:13:42,000
Most of the time, when we run these queries, they have the keyword query in them.

186
00:13:42,000 --> 00:13:46,100
So we decided to pass list the word query to get rid of those false positives.

187
00:13:47,780 --> 00:13:52,620
And different type of rules are going to be sensitive to different type of false positive families.

188
00:13:52,660 --> 00:13:54,460
So you have to kind of look for them.

189
00:13:54,700 --> 00:14:06,980
For example, if you manage to detect some false positive based on the free text, what you can do is just pass list free text for some of the rules that trigger on them.

190
00:14:09,720 --> 00:14:11,320
We've talked about families.

191
00:14:11,320 --> 00:14:13,120
So now, how do we catch them?

192
00:14:13,300 --> 00:14:19,620
Because it's all good and fun, but how do we actually find them in our application when we have 10 plus thousand triggers?

193
00:14:20,000 --> 00:14:21,700
Well, you can do a bunch of statistics.

194
00:14:21,700 --> 00:14:27,100
So yeah, that's where the ML engineer comes into the room and tell you you should do statistics on your triggers.

195
00:14:28,540 --> 00:14:31,800
You can look at application or trigger trends.

196
00:14:31,800 --> 00:14:34,480
So what I mean is, let's say you're...

197
00:14:34,480 --> 00:14:39,100
you can look at your global traffic trends and see how many queries you're getting.

198
00:14:39,220 --> 00:14:42,080
And then what you can do is look at your trigger trends.

199
00:14:42,080 --> 00:14:50,360
If you have a rule that has a traffic that follows a lot your global application traffic, then it's probably a false positive here.

200
00:14:50,580 --> 00:14:53,580
We took an example of something with your data dog here.

201
00:14:54,340 --> 00:14:57,520
Yeah, you have a rule here, the yellow rule, that's triggering a lot of the time.

202
00:14:57,520 --> 00:15:06,420
It's like, I don't know, you cannot really see it here, but 250k triggers most of the time and across all days, every day.

203
00:15:06,440 --> 00:15:09,420
So that's where we're like, okay, we have a false positive on our hands.

204
00:15:09,900 --> 00:15:13,280
And here, on the other side, you can see here a purple bump.

205
00:15:13,280 --> 00:15:16,320
It's actually a security scanner and this is a rule that works perfectly fine.

206
00:15:16,320 --> 00:15:18,340
You can see it's not spread all over the place.

207
00:15:19,000 --> 00:15:29,240
If you're in doubt and you don't know if the trends that you're seeing are actually kind of matching the trends of your traffic, you can also look at the IPs that are triggering your alerts.

208
00:15:29,240 --> 00:15:37,700
If you have a very broad range, meaning everyone that's coming into your application is actually triggering your rule, you also have a rule that's having a false positive issue.

209
00:15:38,380 --> 00:15:43,920
And on this side here, I decided to kind of show you what a Nelfy-ish traffic would look like.

210
00:15:43,920 --> 00:15:45,880
So you have a bunch of bumps, ups and downs.

211
00:15:45,880 --> 00:15:54,380
It's not super regular, so this is typically what Nelfy rules would be outputting in terms of number of triggers.

212
00:15:54,720 --> 00:15:55,760
All right.

213
00:15:56,240 --> 00:16:02,620
Now let's take it a step further and let's see if we can actually do any kind of machine learning path processing.

214
00:16:03,720 --> 00:16:05,880
Yeah, you can start with embeddings.

215
00:16:05,880 --> 00:16:10,800
So I don't know if any of you are familiar with embeddings, but they are used, for example, in LLMs.

216
00:16:10,800 --> 00:16:14,800
So it's very trendy at the moment and you can use them to...

217
00:16:14,800 --> 00:16:16,880
basically use them as a hash function.

218
00:16:16,880 --> 00:16:22,120
You give it a payload, so it could be any type of string, and then it will give you back a vector of numbers.

219
00:16:22,520 --> 00:16:28,580
Here, it's going to be much larger than a 2D vector, but for the sake of this example, I kept a 2D vector.

220
00:16:29,000 --> 00:16:31,940
And what you can do is, you put it in some kind of graph.

221
00:16:32,700 --> 00:16:34,600
Here, okay, we put it there.

222
00:16:34,600 --> 00:16:43,820
Then, as we have a new payload coming in, that we know is also a false positive, we also put it through the embedding function and we get a new representation.

223
00:16:44,140 --> 00:16:50,020
And they are kind of clustered together, so we can kind of define a false positive area, a false positive cluster.

224
00:16:50,680 --> 00:17:00,100
And what this means is that when you have a new payload coming in and it's actually a real trigger, you're able to see it, because it's further away from all the false positive clusters you've been able to build.

225
00:17:00,240 --> 00:17:06,340
So you can kind of deduce false positives or new false positives based on what you already know.

226
00:17:07,120 --> 00:17:16,040
All right, old fun games, but if you're lazy and you don't want to do all this manual clustering, it's very cumbersome, you can do some more fancy things.

227
00:17:16,040 --> 00:17:18,100
So you can do classification, for example.

228
00:17:19,020 --> 00:17:26,600
You can use machine learning models like the classic old-school ones, or you can even use OpenAI LLMs, if you feel like it.

229
00:17:28,460 --> 00:17:37,000
If you do that, some of them will work out of the box, so it's super nice, and some others, if you want it really tailored to your use case, it's not going to work, so you will need to fine-tune them.

230
00:17:38,320 --> 00:17:42,760
And let me tell you, this is a hassle, like we've tried, and this is very tricky to do.

231
00:17:43,340 --> 00:17:50,380
You need to build a legit data set, so a lot of real triggers and a lot of false positives to actually make it make sense.

232
00:17:50,620 --> 00:17:54,020
And here, I took a plain example that we had when we did a trial.

233
00:17:54,080 --> 00:18:04,400
So we gave it an embedding, a bunch of payloads, and we picked an OpenAI LLMs, we fine-tuned it, and we gave it an actual SQL injection payload, and we were like, OK, what is this?

234
00:18:04,400 --> 00:18:06,360
And it replied, it's a true attack.

235
00:18:06,400 --> 00:18:07,820
Dope, OK, it's working.

236
00:18:08,000 --> 00:18:18,360
And then we did the same thing, but we just put a comment after, and it was like, oh no, no, this is not an attack, this is actually just someone randomly coming to your application and pasting that.

237
00:18:18,860 --> 00:18:22,660
So yeah, this is a blatant case of false negatives, and this can happen a lot.

238
00:18:22,660 --> 00:18:30,740
So if you're going the route of doing some fancy machine learning to deal with your false positives, just try to build a good data set.

239
00:18:30,940 --> 00:18:33,760
And yeah, it's just fancy statistics, basically.

240
00:18:34,280 --> 00:18:35,040
All right.

241
00:18:36,260 --> 00:18:39,340
What if we talk about the blind spots of the WAF now?

242
00:18:40,160 --> 00:18:44,460
So that's the part of the talk where the security vendor tells you it's totally fine, the product doesn't work.

243
00:18:44,460 --> 00:18:50,080
No , more seriously, WAFs have a problem with obfuscation.

244
00:18:50,240 --> 00:18:53,100
If you hide your payload, it's going to go through.

245
00:18:53,100 --> 00:19:03,620
And this is mostly because of what the internet is, which is a bunch of RFCs from the late 80s, early 90s, and all of the weird implementation bugs we had since then.

246
00:19:03,740 --> 00:19:08,260
And as a vendor, you can only really easily find the first ones.

247
00:19:08,580 --> 00:19:11,740
What are we talking about with, like, those weird bugs?

248
00:19:11,840 --> 00:19:21,540
You have a web server that I'm not going to name that happens to perform recursive decoding of URLs and support non-serial syntax.

249
00:19:21,740 --> 00:19:27,060
What this means is that you have this monstrosity here and it's going to be converted into an open bracket.

250
00:19:27,060 --> 00:19:27,520
Why?

251
00:19:27,520 --> 00:19:30,260
Because, I guess, some engineers hate us.

252
00:19:31,620 --> 00:19:32,300
SQL.

253
00:19:32,300 --> 00:19:33,240
That's great.

254
00:19:33,240 --> 00:19:34,040
It's a language.

255
00:19:34,040 --> 00:19:34,900
It's standardized.

256
00:19:34,980 --> 00:19:37,540
Every database engine is using the same.

257
00:19:37,580 --> 00:19:37,960
Right?

258
00:19:37,960 --> 00:19:38,280
No.

259
00:19:38,280 --> 00:19:39,080
No, of course not.

260
00:19:39,080 --> 00:19:41,460
MySQL decided to make some commands conditional.

261
00:19:41,460 --> 00:19:46,800
You have this command and in specific version of MySQL, it's not going to be a command anymore.

262
00:19:48,680 --> 00:19:49,800
Thank you.

263
00:19:50,920 --> 00:19:57,000
Last but not least, you can't define a list of value in a URL parameter.

264
00:19:57,180 --> 00:19:59,060
Like, it's just not supported by the other specs.

265
00:19:59,060 --> 00:20:03,160
So, if a developer wants to do it, of course, they only send it in the body.

266
00:20:03,160 --> 00:20:04,860
That's what it's there for.

267
00:20:05,240 --> 00:20:06,180
No, of course not.

268
00:20:06,180 --> 00:20:10,980
Every single web server does something different and passes, like, turns an array of parameters into a list.

269
00:20:10,980 --> 00:20:19,750
The best way to deal with this kind of obfuscation is to work around it completely.

270
00:20:19,750 --> 00:20:27,050
We have one advantage as defenders, which is that we know what the application on the other side of the WAF is.

271
00:20:27,050 --> 00:20:28,170
We have some insight.

272
00:20:28,170 --> 00:20:32,050
We know that we could know the language, the framework, what it's expecting.

273
00:20:32,370 --> 00:20:40,350
And using that, you could tell the WAF to decode the incoming parameter just like the application would do it.

274
00:20:40,350 --> 00:20:42,910
It does not mean that we would be able to decode everything.

275
00:20:43,110 --> 00:20:44,450
But in practice, you don't care.

276
00:20:44,450 --> 00:20:49,450
Because if the application can't decode a parameter, then the parameter can't harm it.

277
00:20:50,090 --> 00:20:51,350
It's not perfect.

278
00:20:51,350 --> 00:20:58,510
Specifically, it won't work if the parameter is integrated by something else, so specifically XSS, where the final target is a browser.

279
00:20:58,510 --> 00:21:04,230
But it's very effective against all kinds of URL encoding, HTTP shenanigans.

280
00:21:04,230 --> 00:21:09,510
Like, if you can just extract what the application is going to see, then you're golden.

281
00:21:09,510 --> 00:21:14,610
Moreover, another problem WAFs need to deal with is performance.

282
00:21:14,650 --> 00:21:19,550
The way most WAFs are set up is they add their latency to the response time of your application.

283
00:21:19,610 --> 00:21:32,510
Meaning, if your application answers in, like, two milliseconds, and you spent a lot of resources to optimize it, and the WAF just barges in and adds, like, 100 milliseconds of latency, that's unacceptable.

284
00:21:32,790 --> 00:21:36,150
So WAFs need to be able to fit within those constraints.

285
00:21:36,330 --> 00:21:45,790
Moreover, even if you're not under attack, we notice that even a small minority of long payload is enough to tank the performance.

286
00:21:45,950 --> 00:21:48,130
Like, surprisingly little.

287
00:21:49,670 --> 00:21:56,390
What this means is that you have a few constraints that WAFs are never really going to be able to work around of.

288
00:21:56,390 --> 00:22:07,230
So, for instance, if your WAF is not passing the body, and the application is expecting a large JSON payload, then all the attacker has to do is to put the attack at the end of the payload, and they're in.

289
00:22:09,010 --> 00:22:12,930
Some WAFs also have some constraints on the time they are allowed to take.

290
00:22:12,930 --> 00:22:18,470
And if they start taking, like, let's say more than five milliseconds, all right, they disconnect, they are like, there's too much impact.

291
00:22:19,310 --> 00:22:27,650
And all an attacker has to do is to increase the load on the WAF just by sending a bunch of requests, and, yeah, they're in.

292
00:22:28,670 --> 00:22:31,850
And, yeah, finally, the obfuscation, the WAF can do it.

293
00:22:31,910 --> 00:22:36,250
It can try to parse the parameter as a browser would do it, but it's expensive.

294
00:22:36,250 --> 00:22:39,250
So you need to figure out how do you want to spend your budget.

295
00:22:39,250 --> 00:22:49,150
And you might decide, okay, this decoding would be very effective, but actually if I use it on all of my rules, I'm, like, spending half my time just doing decoding and not actually running rules.

296
00:22:50,250 --> 00:22:54,590
There are a few solutions to that, such as running the WAF async.

297
00:22:55,130 --> 00:23:07,110
They all have, like, terrible problems, from not being able to block to you could direct all of your traffic to some kind of cloud infrastructure, but then you're adding latency, you're saying all of your traffic don't know where.

298
00:23:07,390 --> 00:23:09,310
So, yeah, that's kind of terrible.

299
00:23:11,570 --> 00:23:16,170
Finally, WAF, and I'm going to blow your mind, don't know what they don't see.

300
00:23:16,190 --> 00:23:21,670
So if you have an attack that the WAF don't catch, the WAF won't be able to tell you that you are under attack.

301
00:23:22,070 --> 00:23:26,450
What this means is that to be effective with WAF, you kind of need to do threat intel.

302
00:23:26,450 --> 00:23:33,510
You need to be able to do quick and quiet deploy of rules to be able to know what's happening in the wild.

303
00:23:33,510 --> 00:23:35,330
Doing some kind of shadow deployments.

304
00:23:36,050 --> 00:23:39,350
The scale at which you do that depends on what you can do.

305
00:23:39,350 --> 00:23:43,570
Like, it can be an only pot, but then you're mostly going to catch against the noise of the internet.

306
00:23:43,870 --> 00:23:52,690
You may be able to do it on your internal services, but, yeah, you'll get a better idea of what an actual attacker is going to do for your kind of application.

307
00:23:52,950 --> 00:23:59,290
And if you're a vendor and can work with it with your customers, you can even deploy it in customer environments.

308
00:23:59,290 --> 00:24:04,830
And that's amazing, because then you see a very broad diversity of attacks.

309
00:24:04,830 --> 00:24:07,910
Like, you see what the actual threat landscape is like.

310
00:24:08,290 --> 00:24:15,430
And when you do and you catch someone, WAFs are really good at leveraging mistakes.

311
00:24:15,430 --> 00:24:20,690
If you find any identifying pattern in an attacker, you can leverage it.

312
00:24:20,690 --> 00:24:23,910
Be it user agent, they're using this old version of Firefox.

313
00:24:23,930 --> 00:24:25,910
All right, I'm just going to match everything from that.

314
00:24:25,910 --> 00:24:27,990
It could be the IP or the range of IPs.

315
00:24:27,990 --> 00:24:30,670
It could be the domain they're fetching their attacks from.

316
00:24:30,670 --> 00:24:36,490
It's very common for accesses to load the payload from some given domain.

317
00:24:36,670 --> 00:24:38,330
Or, and that's the best one, typos.

318
00:24:38,550 --> 00:24:39,510
The attacker does a typo.

319
00:24:39,510 --> 00:24:44,670
All right, I just want to match all of the typo headers.

320
00:24:44,730 --> 00:24:50,910
And then once you have all of that, you dump all of the attacker traffic and you burn it all at once.

321
00:24:51,170 --> 00:24:53,270
And yeah, destroy that toolkit.

322
00:24:55,750 --> 00:25:03,110
So, we talked a bit about what WAFs are, what their issues are, what the false positive, false negative, what they're good at.

323
00:25:03,110 --> 00:25:06,050
Let's talk about making them better for end users.

324
00:25:06,950 --> 00:25:19,010
The first and most important point is you need to embrace the strength of the WAF and stop seeing a WAF as a protection tool.

325
00:25:19,610 --> 00:25:21,330
It won't be very good at that.

326
00:25:21,330 --> 00:25:28,950
If you want something to catch every single exploit going to your application, there are things, there are products that do that.

327
00:25:28,950 --> 00:25:30,250
There are RAS, there are IAS.

328
00:25:30,370 --> 00:25:32,750
There are some solutions, but WAFs are not it.

329
00:25:32,750 --> 00:25:34,810
What a WAF is, is a monitoring tool.

330
00:25:34,810 --> 00:25:40,310
A WAF will tell you that you're under attack and give you a feeling for the kind of attacks that the attacker is sending.

331
00:25:40,310 --> 00:25:42,110
It won't give you everything.

332
00:25:42,250 --> 00:25:44,270
If you wanted to do it, you can.

333
00:25:44,290 --> 00:25:47,230
It's called, like, here, this rule, trade secret.

334
00:25:47,270 --> 00:25:48,710
You just match that star.

335
00:25:49,210 --> 00:25:50,490
You can get everything.

336
00:25:50,730 --> 00:25:52,330
It will be totally unusable.

337
00:25:54,910 --> 00:25:57,130
Next one, rule testing.

338
00:25:57,890 --> 00:26:01,850
You can have a good turnkey experience in a WAF.

339
00:26:01,850 --> 00:26:07,110
But for that, rules need to have been tested on every single kind of workloads.

340
00:26:07,150 --> 00:26:10,130
And this requires massive scale.

341
00:26:10,130 --> 00:26:17,750
On top of it, whoever is doing the rule qualification will need to decide how much noise is good enough.

342
00:26:17,750 --> 00:26:24,470
And that's going to depend from a deployment perspective, from a company perspective, from a... yeah.

343
00:26:24,470 --> 00:26:27,230
It even changes on an engineer basis.

344
00:26:29,090 --> 00:26:36,690
The best way for someone to guarantee that the rule is going to work is to do some kind of shadow deployment, meaning running the rules on your environment without blocking.

345
00:26:37,630 --> 00:26:38,890
And, yeah.

346
00:26:39,170 --> 00:26:43,050
Doing all of that, you need to make sure not to catch any PII because otherwise you're in trouble.

347
00:26:45,050 --> 00:26:45,890
All right.

348
00:26:45,890 --> 00:26:49,090
Next up, you can try to leverage your application context.

349
00:26:49,090 --> 00:27:00,410
Meaning, if you're able to give data about your application to your WAF, like, telling it which language you're running, telling it which framework you're using, then, in that case, you'll be able to increase the precision.

350
00:27:00,410 --> 00:27:01,590
Let me give you an example.

351
00:27:01,610 --> 00:27:03,650
You're running a JavaScript application.

352
00:27:03,650 --> 00:27:05,410
You're receiving a bunch of PHP attacks.

353
00:27:05,410 --> 00:27:07,490
Well, you don't really care about those.

354
00:27:07,490 --> 00:27:09,090
They're not going to hurt your application.

355
00:27:09,090 --> 00:27:12,090
So, what you need to do is just discard those.

356
00:27:12,330 --> 00:27:20,030
If you're running some kind of PHP, for example, the SilverStripe framework, it was vulnerable to XSS in older versions.

357
00:27:20,710 --> 00:27:26,310
Well, if you're not running, if you're running up to the latest version, you also don't care about the alerts that are targeting the older version.

358
00:27:26,310 --> 00:27:27,570
So, yeah.

359
00:27:27,570 --> 00:27:29,990
If you're able to do that, you'll increase your accuracy.

360
00:27:30,110 --> 00:27:46,390
And one other side effect that can be good for you is you're also able to detect threat actors that will be sending, for example, Java attacks to your app when you're not running any Java, and you can kind of increase the attention that you're giving to this malicious IP and gather a bunch of threat intel on top of that.

361
00:27:46,810 --> 00:27:56,150
The only downside is it's going to be a bit challenging, meaning you have to have your security team work with your engineering team, and God knows it can be tricky to do sometimes.

362
00:27:56,310 --> 00:28:04,210
So, if you're able to do that, you can build this feedback loop that will be improving greatly your experience with the WAF.

363
00:28:05,790 --> 00:28:20,770
And the last element that you can try is to tune the response to the attack, meaning if you're able to catch a specific IP that's been targeting your application for a while, you could dynamically roll out a new rule set for this specific IP and catch more attacks coming from it,

364
00:28:20,770 --> 00:28:23,750
and you could catch everything and see what's coming up.

365
00:28:23,770 --> 00:28:36,270
And , yeah, if you're able to do that, and on top of that, what you could try, if you're unsure if the actor is actually malicious, we should be able to enable different types of responses.

366
00:28:36,270 --> 00:28:40,930
First, we should be able to have a capture and then only block after.

367
00:28:41,910 --> 00:28:46,410
Remember, we talked a bit about how we can do some analysis of false positives.

368
00:28:46,410 --> 00:28:55,570
If we're also able to leverage that to feed that back to the WAF and build dynamically a bunch of passive rules so we don't have as many false positives, it's even better.

369
00:28:55,570 --> 00:29:00,250
We could be even more confident when we actually enable the WAF in blocking mode on some rules.

370
00:29:01,190 --> 00:29:04,210
All right, this is our final takeaways.

371
00:29:04,830 --> 00:29:11,030
If you're a defender, look at your WAF more as a monitoring system than as a protection system.

372
00:29:11,250 --> 00:29:17,350
You should try to launch your false positives because this way you will be able to leverage the actual alerts that you get.

373
00:29:17,350 --> 00:29:22,330
And if you're building some kind of rule set, make sure that you keep in some flexibility for your engineering team.

374
00:29:22,330 --> 00:29:30,050
Meaning if you do very restrictive type of rule set and as soon as your engineers are changing the app, they're breaking everything, it's not worth it.

375
00:29:30,470 --> 00:29:39,870
If you are a vendor, well, I think it should be our collective mission to propose to our users a very low noise rule set.

376
00:29:40,110 --> 00:29:46,770
And then also support them in their efforts to actually analyze their triggers and the trends that they have.

377
00:29:47,530 --> 00:29:53,230
And if you're collecting any type of threat intelligence, please share it across all of your customers so everyone can benefit from it.

378
00:29:54,070 --> 00:29:59,070
And last, if you're an attacker, well, you're in luck because there will always be bypasses to the WAF.

379
00:29:59,070 --> 00:30:02,270
We've talked about them because of the WAF's nature.

380
00:30:02,570 --> 00:30:11,270
So keep in mind that you should probably rotate your infrastructure because we will leverage all the mistakes that you make and if you don't rotate, we'll catch you and we'll thank you for that.

381
00:30:11,270 --> 00:30:18,270
And then if you have to invest in one type of application, please only invest in ambiguous interpreted obfuscation.

382
00:30:18,270 --> 00:30:20,790
HTTP is worthless.

383
00:30:21,130 --> 00:30:24,050
Alright, we're at the end.

384
00:30:24,170 --> 00:30:25,890
Thank you for listening to us.

385
00:30:25,890 --> 00:30:31,690
applause music music music music