[00:08.460 --> 00:10.760]  Alright, this is our agenda for today.
[00:10.760 --> 00:13.240]  First, we'll be going over what WAFs are.
[00:13.240 --> 00:15.140]  I think many of you already know what they are.
[00:15.140 --> 00:17.100]  Otherwise, you probably wouldn't be here.
[00:17.400 --> 00:20.180]  Just so we start with an even play for everyone.
[00:20.460 --> 00:22.720]  Then, we'll go over what we've learned at Datadog
[00:22.720 --> 00:24.660]  operating a very large-scale WAF
[00:24.660 --> 00:26.920]  and the struggle that comes with it.
[00:27.340 --> 00:29.840]  We'll try to address a bit how we deal with
[00:29.840 --> 00:32.260]  false positives and false negatives.
[00:32.580 --> 00:35.800]  And then, we'll finish by sharing
[00:35.800 --> 00:38.040]  how we try to make them a bit better.
[00:38.600 --> 00:40.880]  Alright, who are we first?
[00:41.460 --> 00:43.760]  I'm Emmanuelle. I'm a software engineer at Datadog.
[00:43.760 --> 00:44.760]  I'm based in Paris.
[00:44.760 --> 00:46.200]  And I've worked on machine learning models
[00:46.200 --> 00:47.420]  for the past four years.
[00:47.420 --> 00:49.300]  So, not a security background at all.
[00:49.460 --> 00:51.480]  And actually, last year, I decided to join Datadog
[00:51.480 --> 00:53.940]  to work on our application security product.
[00:54.440 --> 00:56.220]  And this is my co-presenter, Emile.
[00:56.860 --> 00:58.500]  So, I'm Emile Gospier.
[00:58.500 --> 01:01.780]  I'm a security researcher at Datadog,
[01:01.780 --> 01:02.780]  also based in Paris.
[01:02.780 --> 01:04.860]  I've been working on a WAF and a RASP solution
[01:04.860 --> 01:06.460]  for the last five years.
[01:06.560 --> 01:08.120]  First, at Screen, then Datadog,
[01:08.120 --> 01:09.940]  following an acquisition in 2021.
[01:10.500 --> 01:12.300]  And, yeah, let's talk about WAFs.
[01:12.300 --> 01:13.660]  You may know about them.
[01:14.020 --> 01:16.020]  If you don't, that's a security tool
[01:16.020 --> 01:18.940]  that was created in the mid-90s,
[01:18.940 --> 01:22.900]  when web applications started to become more complex,
[01:22.900 --> 01:26.600]  more useful, and thus, a more enticing target.
[01:27.120 --> 01:29.100]  We didn't really have anything to do
[01:29.100 --> 01:32.160]  to help protect them.
[01:32.160 --> 01:33.980]  So, we looked at other tools,
[01:33.980 --> 01:36.160]  and a promising one was firewalls.
[01:36.160 --> 01:38.060]  They're working really great at the network level.
[01:38.060 --> 01:40.220]  So, the idea was, let's take them
[01:40.220 --> 01:42.720]  and bring them to the application layer.
[01:43.080 --> 01:44.900]  And instead of running on specific rules,
[01:44.900 --> 01:47.000]  like this IP, this port, this whatever,
[01:47.000 --> 01:50.220]  it was more, yeah, let's run regexes or dictionaries
[01:50.220 --> 01:54.280]  to find suspicious patterns in that.
[01:54.280 --> 01:56.300]  And if you find them in the incoming request,
[01:56.300 --> 01:57.540]  that's probably because it's bad.
[01:58.460 --> 02:00.920]  And, yeah, that's pretty much it.
[02:03.080 --> 02:05.080]  WAFs today are pretty much the backbone
[02:05.080 --> 02:07.180]  of runtime application security.
[02:07.180 --> 02:08.760]  And that's for a few reasons.
[02:08.760 --> 02:11.460]  First, they are super easy to explain.
[02:11.500 --> 02:13.480]  In just a few words, I was able to give you
[02:13.660 --> 02:15.320]  a good view of what they are.
[02:16.180 --> 02:18.440]  They're also incredibly easy to customize.
[02:18.440 --> 02:20.880]  Because they're based on regexes and dictionaries,
[02:20.880 --> 02:22.880]  you have an entire industry of rulesets
[02:22.880 --> 02:26.380]  that you can use, or you can even customize them yourself.
[02:26.860 --> 02:28.640]  They're also really easy to demo.
[02:28.640 --> 02:32.080]  All you need to do to show that your WAF works
[02:32.080 --> 02:34.220]  is plug it on the Internet.
[02:34.500 --> 02:37.280]  And just the noise from the Internet
[02:37.280 --> 02:38.720]  is going to create a bunch of alerts,
[02:38.720 --> 02:40.080]  and, hey, you have a great security tool,
[02:40.080 --> 02:41.160]  you're protecting everyone.
[02:42.580 --> 02:44.140]  And finally, it never hurts,
[02:44.140 --> 02:45.660]  it's actually recorded by compliance.
[02:45.800 --> 02:49.640]  So if you're dealing with credit cards,
[02:50.400 --> 02:51.800]  yeah, you need a WAF, basically.
[02:51.940 --> 02:54.340]  So, yeah, all of that came together
[02:54.340 --> 02:56.860]  to make WAF an incredibly popular product.
[02:58.840 --> 03:01.300]  However, WAF tends to give
[03:01.420 --> 03:03.400]  a bit of a false sense of security.
[03:03.440 --> 03:04.720]  Because what a WAF does
[03:04.720 --> 03:08.400]  is that it looks for suspicious requests.
[03:08.400 --> 03:10.040]  It's running regexes, dictionaries,
[03:10.040 --> 03:11.240]  that's what it does.
[03:11.540 --> 03:13.340]  But then it means that if you can make
[03:13.340 --> 03:17.660]  your malicious attack look unsuspicious,
[03:17.660 --> 03:19.040]  then it's going to go through.
[03:19.560 --> 03:21.360]  And if you happen to have a user
[03:21.360 --> 03:22.580]  that sends another request
[03:22.580 --> 03:26.760]  that happens to look suspicious,
[03:26.760 --> 03:27.900]  they need to be matched.
[03:28.340 --> 03:31.760]  And there is not really anything you can do about it.
[03:31.760 --> 03:33.340]  That's WAF working as design.
[03:34.200 --> 03:37.060]  So, yeah, a few alternatives were tried over time,
[03:37.060 --> 03:38.700]  like RASP, IASP,
[03:38.700 --> 03:40.640]  but they never really went anywhere.
[03:40.960 --> 03:43.460]  No solution really got anywhere as popular as WAF.
[03:45.160 --> 03:46.680]  We talked about what they are.
[03:46.980 --> 03:49.300]  Emmanuel is going to tell you about how to make one.
[03:49.300 --> 03:50.800]  Yeah, how do we build a WAF?
[03:50.800 --> 03:52.640]  Well, first, of course, you need a web application
[03:52.640 --> 03:54.240]  because that's what you want to protect.
[03:54.240 --> 03:55.800]  In front of that, most of the time,
[03:55.800 --> 03:56.640]  you're going to put a reverse proxy
[03:56.640 --> 03:59.600]  to route all the incoming traffic to your web application.
[04:00.020 --> 04:00.680]  All right.
[04:00.900 --> 04:02.500]  Now, let's build a WAF.
[04:02.600 --> 04:05.160]  First element that we need is a rule engine, of course.
[04:05.160 --> 04:06.440]  We're going to put it on the reverse proxy
[04:06.440 --> 04:07.760]  because it's very convenient.
[04:07.760 --> 04:09.340]  It's where we're going to be able to see
[04:09.340 --> 04:10.880]  all the incoming traffic and filter out
[04:10.880 --> 04:12.960]  the malicious request if we need to.
[04:13.800 --> 04:15.960]  Second element we need, a rule set.
[04:16.360 --> 04:18.380]  Here, we're mentioning the OWASP core rule set
[04:18.380 --> 04:21.020]  because it's a very popular one across the industry.
[04:21.020 --> 04:22.920]  It's open source, so you can also use it
[04:22.920 --> 04:25.240]  to build your own rule set on top of it.
[04:25.240 --> 04:29.080]  Here, you can see we mentioned PL1 for Paranoia Level 1.
[04:29.580 --> 04:32.240]  What it means is it's supposed to be not too noisy,
[04:32.240 --> 04:33.920]  so it's supposed to be a set of rules
[04:33.920 --> 04:36.140]  that shouldn't trigger too many alerts for us.
[04:36.160 --> 04:37.160]  Okay. All right.
[04:37.160 --> 04:37.740]  Well, good.
[04:37.740 --> 04:39.620]  Last element we need, the monitoring.
[04:40.200 --> 04:43.220]  It can be just a text file where you push all of your alerts,
[04:43.220 --> 04:46.660]  or it can be a super fancy dashboard that you've built.
[04:48.220 --> 04:48.940]  Okay.
[04:49.200 --> 04:49.880]  We're all set.
[04:49.880 --> 04:51.240]  I think we can activate the WAF.
[04:51.240 --> 04:52.660]  And this is more or less what happened
[04:52.660 --> 04:53.900]  the first time we did it.
[04:53.900 --> 04:55.480]  We started crying.
[04:55.740 --> 04:57.600]  We got a bunch of alerts.
[04:57.640 --> 04:59.880]  And we were like, okay, what do we do with this?
[05:00.360 --> 05:02.180]  And we took a step back and were like, okay,
[05:02.180 --> 05:04.640]  this is actually a great learning experiment.
[05:05.080 --> 05:06.560]  So what did we learn, Emil?
[05:08.640 --> 05:10.700]  So the learnings we are going to share
[05:10.700 --> 05:14.240]  are based on us operating a WAF at a pretty large scale.
[05:14.240 --> 05:16.580]  We are talking around a billion requests per second
[05:16.580 --> 05:18.840]  across hundreds of thousands of organizations.
[05:20.060 --> 05:22.260]  And the first and most important learning
[05:22.260 --> 05:26.280]  we took away from that is that the accuracy of a WAF rule
[05:26.280 --> 05:28.420]  is totally unpredictable at scale.
[05:28.620 --> 05:30.800]  You can think that your rule is perfect.
[05:30.960 --> 05:33.380]  Until it's tested, you have no idea.
[05:35.600 --> 05:38.560]  So they will tend to converge quite fast.
[05:38.780 --> 05:42.600]  So if you can test them on a broad enough set of data,
[05:42.600 --> 05:44.120]  within a few hours you will have a good idea
[05:44.120 --> 05:45.060]  of how it's going to behave.
[05:45.060 --> 05:49.300]  But if you only tested them on WordPress or CMS,
[05:49.300 --> 05:52.100]  and you deployed them on an e-commerce website,
[05:52.100 --> 05:53.580]  I mean, you can have a French customer
[05:53.580 --> 05:55.660]  going to share their address,
[05:55.660 --> 05:58.280]  which has a quote and OR,
[05:58.280 --> 05:59.640]  and then the WAF is going to freak out.
[05:59.640 --> 06:02.320]  I mean, of course, this is a SQL injection, no?
[06:03.020 --> 06:03.880]  Yeah.
[06:05.580 --> 06:07.660]  The other important consequence from that
[06:07.660 --> 06:10.640]  is that even if you do very simple modifications,
[06:10.640 --> 06:13.020]  like you just expand it a little bit further,
[06:13.020 --> 06:16.240]  like you add a couple of duplication steps,
[06:16.240 --> 06:17.340]  you need to test them again
[06:17.340 --> 06:19.780]  because you don't know what you are missing before.
[06:20.720 --> 06:23.280]  The other side of this coin, however,
[06:23.280 --> 06:25.260]  is that even if you have a crazy rule
[06:25.260 --> 06:26.940]  that shouldn't work,
[06:27.720 --> 06:30.120]  it's usually worth testing them just in case.
[06:30.460 --> 06:32.260]  To illustrate that, we're going to go back
[06:32.260 --> 06:34.700]  like to late 2021,
[06:34.700 --> 06:37.240]  when AppSec just caught fire,
[06:37.240 --> 06:40.820]  and we had log4j drop on us a few weeks before Christmas.
[06:41.440 --> 06:43.100]  And for those of you who don't know,
[06:43.100 --> 06:45.960]  it was an RCE in the log4j library,
[06:45.960 --> 06:48.760]  which is used by Java application to log.
[06:48.760 --> 06:49.820]  And if you don't know,
[06:49.820 --> 06:52.220]  Java applications love to log.
[06:53.340 --> 06:55.400]  So when it came out,
[06:55.400 --> 06:57.460]  the initial PLC was something like that,
[06:57.460 --> 07:02.000]  where if you could get log4j to log this,
[07:02.000 --> 07:03.740]  you had an RCE.
[07:04.780 --> 07:07.160]  So yeah, everyone started scanning all over the place.
[07:07.860 --> 07:11.080]  Pretty much every WAF vendor or user
[07:11.080 --> 07:13.680]  set up their own pattern, and we did too.
[07:13.720 --> 07:15.560]  The first version was something like that.
[07:15.560 --> 07:18.520]  So, okay, we just match gndi,
[07:18.520 --> 07:21.840]  which is the entry point of the bug in log4j,
[07:21.840 --> 07:24.200]  and then pretty much everything that follows.
[07:24.440 --> 07:25.720]  We test it.
[07:25.840 --> 07:26.600]  Okay, it's fine.
[07:26.600 --> 07:28.000]  It triggers a little bit.
[07:28.400 --> 07:30.400]  No obvious pattern of false positive.
[07:30.400 --> 07:31.440]  Okay, good to go.
[07:31.440 --> 07:32.300]  We can ship it.
[07:33.200 --> 07:34.960]  And yeah, it worked well.
[07:35.200 --> 07:36.760]  Very quickly, however,
[07:36.760 --> 07:39.760]  the attackers tried to find bypasses for that.
[07:39.760 --> 07:40.800]  So the first one they found was
[07:40.800 --> 07:42.280]  they could use a different schema.
[07:42.320 --> 07:45.540]  The initial PLC used LDAP to perform the RCE,
[07:45.540 --> 07:47.120]  but actually you could use another schema,
[07:47.120 --> 07:48.720]  so they just did.
[07:49.300 --> 07:52.720]  In our case, we were not really looking at LDAP,
[07:52.720 --> 07:53.960]  so we didn't really care.
[07:54.020 --> 07:58.340]  But then they decided to step things up.
[07:58.340 --> 07:59.400]  So they started to use
[07:59.400 --> 08:02.300]  the string interpolation features of log4j
[08:02.300 --> 08:05.440]  to obfuscate their pattern further.
[08:06.300 --> 08:09.580]  So, okay, we need to complexify quite a bit the pattern.
[08:09.580 --> 08:12.180]  So every single character we had before
[08:12.180 --> 08:14.340]  became this whole mess
[08:14.340 --> 08:16.960]  to support some amount of obfuscation.
[08:17.080 --> 08:19.460]  Then attackers decided to step things up further.
[08:19.460 --> 08:21.180]  So, okay, now you have completely spurious blocks
[08:21.180 --> 08:23.080]  that are just going to disappear.
[08:23.720 --> 08:26.300]  Okay, we can expand the pattern further.
[08:26.300 --> 08:28.460]  This is becoming a bit unwieldy.
[08:28.460 --> 08:31.380]  For each character you have in the initial pattern,
[08:31.380 --> 08:33.060]  you have this whole mess there.
[08:33.600 --> 08:36.120]  And attackers just got crazy.
[08:38.640 --> 08:42.120]  This could either be the most complex waffle ever,
[08:42.120 --> 08:43.660]  or we need a different type strategy.
[08:43.660 --> 08:45.440]  So when we saw that,
[08:45.440 --> 08:48.160]  we basically tested two rules.
[08:48.200 --> 08:52.120]  One monstrosity that could deal with that,
[08:52.120 --> 08:53.700]  or something much simpler.
[08:55.080 --> 08:56.640]  And we are shocked,
[08:56.640 --> 08:59.760]  but this never triggered the false positive.
[09:00.840 --> 09:03.120]  Like, that's why we're telling you
[09:03.660 --> 09:04.540]  that you need to test rules,
[09:04.540 --> 09:06.720]  because this shouldn't work.
[09:08.160 --> 09:11.760]  And, yeah, the last bypasses we found at the end
[09:11.760 --> 09:14.720]  were attempts to use non-ASCII characters.
[09:14.820 --> 09:17.000]  So here, like, log4j would do some kind of
[09:17.000 --> 09:18.280]  unicode normalization.
[09:18.860 --> 09:20.560]  Yeah, at this point you need to do the same.
[09:20.560 --> 09:21.900]  And that's expensive.
[09:24.660 --> 09:26.960]  Yeah, and when we put this in production,
[09:26.960 --> 09:30.340]  it triggered roughly half a million triggers in a week.
[09:30.340 --> 09:32.040]  And that was before the acquisition of Datadog,
[09:32.040 --> 09:33.640]  so at a much smaller scale.
[09:35.700 --> 09:38.900]  We told you about how you design a rule,
[09:38.900 --> 09:41.640]  but there is a greater framework behind it
[09:41.640 --> 09:42.700]  and some constraints.
[09:42.700 --> 09:44.460]  So let Emmanuel tell you about the dilemma
[09:44.460 --> 09:45.860]  of building a WAF rule.
[09:45.860 --> 09:48.200]  Yes, so when you build a WAF rule,
[09:48.700 --> 09:52.200]  we try to kind of display this into a graph.
[09:53.300 --> 09:55.060]  On the horizontal axis,
[09:55.060 --> 09:57.420]  you can have a very broad rule that will catch everything,
[09:57.420 --> 09:59.620]  or a very specific rule.
[09:59.880 --> 10:01.180]  And on the vertical axis,
[10:01.180 --> 10:04.220]  you can have either a rule that's very easy to bypass,
[10:04.220 --> 10:05.820]  but doesn't give you too many false positives,
[10:05.820 --> 10:08.980]  or here on the top, it triggers a lot of false positives,
[10:08.980 --> 10:10.160]  but it's hard to bypass.
[10:10.880 --> 10:12.880]  All right, based on that,
[10:12.880 --> 10:15.560]  we try to define a max amount of noise
[10:15.560 --> 10:16.960]  that we're willing to accept,
[10:16.960 --> 10:19.920]  meaning if a rule starts triggering more false positives
[10:19.920 --> 10:20.960]  than this threshold,
[10:20.960 --> 10:22.500]  we're not even going to put it into prediction
[10:22.500 --> 10:25.280]  because it's not worth the hassle.
[10:26.180 --> 10:29.640]  Okay, now an ideal rule we could try to build
[10:29.640 --> 10:30.800]  would be something very broad
[10:30.800 --> 10:33.840]  that's going to catch every XSS attack that we're getting.
[10:34.360 --> 10:34.940]  Dope.
[10:34.940 --> 10:36.640]  And it doesn't trigger too many false positives
[10:36.640 --> 10:38.380]  and it's not too easy to bypass.
[10:38.380 --> 10:40.140]  All right, it's right in there in the middle.
[10:40.140 --> 10:41.360]  On the other side,
[10:41.360 --> 10:43.100]  you could also try to build a rule
[10:43.100 --> 10:44.500]  that's very, very specific,
[10:44.500 --> 10:48.660]  meaning it could only target a very specific XSS vulnerability
[10:48.660 --> 10:50.180]  for Jupyter notebooks.
[10:51.160 --> 10:52.840]  And same, not too easy to bypass
[10:52.840 --> 10:54.560]  and not too many false positives.
[10:54.820 --> 10:55.380]  Dope.
[10:55.420 --> 10:57.260]  Well, that would be an ideal rule
[10:57.260 --> 10:59.420]  because actually what happens most of the time
[10:59.420 --> 11:00.760]  when you try to build this rule
[11:00.760 --> 11:02.660]  is you end up right there.
[11:02.860 --> 11:04.400]  If you do a very broad rule,
[11:04.400 --> 11:05.520]  you're going to have a lot of false positives
[11:05.520 --> 11:07.420]  and if you do a very specific rule,
[11:07.420 --> 11:09.440]  you're going to be bypassed very easily.
[11:11.020 --> 11:12.220]  So what happens?
[11:12.540 --> 11:14.940]  Most of the time, you're going to be in this yellow area
[11:14.940 --> 11:16.900]  right there, the space of operation.
[11:16.920 --> 11:18.540]  And what you want to do, ideally,
[11:18.540 --> 11:20.120]  is to be right there in the middle.
[11:20.120 --> 11:21.920]  So you are doing, kind of,
[11:21.920 --> 11:24.140]  having the best of both worlds, so to speak.
[11:25.300 --> 11:26.920]  All right, so we've talked about
[11:26.920 --> 11:28.600]  how we don't want too many false positives
[11:28.600 --> 11:30.160]  because they are nasty.
[11:30.520 --> 11:32.800]  Well, let me tell you or kind of remind you
[11:32.800 --> 11:34.000]  why we don't want them.
[11:34.080 --> 11:36.300]  First, they are going to create an alert fatigue.
[11:36.300 --> 11:37.640]  So, yeah, there is nothing worse
[11:37.640 --> 11:40.260]  than you come into work and you see 10,000 plus alerts.
[11:40.260 --> 11:41.380]  What do I do with this?
[11:41.680 --> 11:43.920]  It's not worth looking at it
[11:43.920 --> 11:45.020]  and it's not actionable.
[11:45.840 --> 11:47.260]  Second reason why.
[11:47.260 --> 11:49.280]  Well, it can have a business impact.
[11:49.560 --> 11:51.800]  If you have rules in blocking mode,
[11:51.800 --> 11:53.680]  you might be blocking legitimate customers
[11:53.680 --> 11:56.180]  with a wrong setup.
[11:56.460 --> 11:57.860]  And if you're blocking customers,
[11:57.860 --> 11:58.760]  you're losing business.
[11:58.760 --> 12:00.260]  If you're losing business, you're losing money.
[12:00.260 --> 12:01.640]  So you don't want that either.
[12:01.860 --> 12:03.700]  And the last reason why you don't want false positives
[12:03.700 --> 12:06.740]  is because you might end up collecting PII by accident.
[12:06.740 --> 12:09.380]  Meaning you might collect some private information
[12:09.380 --> 12:11.940]  like addresses, emails, phone numbers.
[12:11.940 --> 12:14.080]  And this might end up in your log triggers.
[12:14.920 --> 12:16.120]  And because of that,
[12:16.120 --> 12:17.920]  you're going to have a security issue on your hands.
[12:17.920 --> 12:19.580]  So, yeah, you don't want that either.
[12:20.720 --> 12:22.520]  Overall, what you need to keep in mind is that
[12:22.520 --> 12:24.520]  it's all about finding this compromise.
[12:24.520 --> 12:26.820]  You want a rule that doesn't have too many false positives
[12:26.820 --> 12:28.900]  but still brings you value.
[12:30.420 --> 12:31.380]  All right.
[12:31.820 --> 12:34.020]  In all this time working on false positives,
[12:34.020 --> 12:36.220]  at Datadog, we've been able to see a bunch of them
[12:36.220 --> 12:37.400]  and we've tried to classify them
[12:37.400 --> 12:38.980]  into different types of families.
[12:39.400 --> 12:41.100]  The first family we've encountered
[12:41.100 --> 12:44.500]  most of the time is based on what they are.
[12:44.600 --> 12:48.440]  Meaning you get a bunch of triggers coming from free text.
[12:48.760 --> 12:50.500]  For example, you're running an application
[12:50.500 --> 12:51.680]  like Stack Overflow.
[12:51.680 --> 12:54.640]  You have a bunch of people pasting SQL injection all the time
[12:54.640 --> 12:56.080]  because they're asking questions about it.
[12:56.080 --> 12:57.140]  And this will trigger your WAF
[12:57.140 --> 13:00.420]  if it's set up in a way where it looks at free text.
[13:01.380 --> 13:03.340]  Second family you can have is
[13:03.340 --> 13:04.900]  if you're running any type of infrastructure
[13:04.900 --> 13:07.140]  that's having an admin panel,
[13:07.140 --> 13:12.680]  running SQL queries against your DB.
[13:12.680 --> 13:15.400]  This is going to look like SQL injection to your WAF.
[13:15.400 --> 13:16.960]  So let's say you're running phpMyAdmin
[13:16.960 --> 13:19.100]  that could happen most of the time.
[13:19.960 --> 13:24.600]  And also, if your application is relying on some concept
[13:24.600 --> 13:27.660]  like you're doing a lot of queries all over the place.
[13:27.660 --> 13:29.920]  For example, at Datadog, that's an issue that we have.
[13:29.920 --> 13:31.780]  We're running a ton of queries everywhere
[13:31.780 --> 13:34.400]  and it looks like SQL injection a lot of the time.
[13:34.400 --> 13:35.820]  So what do we do with that?
[13:36.300 --> 13:39.700]  Most of the time, when we run these queries,
[13:39.700 --> 13:42.000]  they have the keyword query in them.
[13:42.000 --> 13:44.280]  So we decided to pass list the word query
[13:44.280 --> 13:46.100]  to get rid of those false positives.
[13:47.780 --> 13:49.800]  And different type of rules are going to be sensitive
[13:49.800 --> 13:52.620]  to different type of false positive families.
[13:52.660 --> 13:54.460]  So you have to kind of look for them.
[13:54.700 --> 13:58.800]  For example, if you manage to detect
[13:58.800 --> 14:01.280]  some false positive based on the free text,
[14:01.280 --> 14:04.920]  what you can do is just pass list free text
[14:04.920 --> 14:06.980]  for some of the rules that trigger on them.
[14:09.720 --> 14:11.320]  We've talked about families.
[14:11.320 --> 14:13.120]  So now, how do we catch them?
[14:13.300 --> 14:15.300]  Because it's all good and fun,
[14:15.300 --> 14:17.540]  but how do we actually find them in our application
[14:17.540 --> 14:19.620]  when we have 10 plus thousand triggers?
[14:20.000 --> 14:21.700]  Well, you can do a bunch of statistics.
[14:21.700 --> 14:24.360]  So yeah, that's where the ML engineer comes into the room
[14:24.360 --> 14:27.100]  and tell you you should do statistics on your triggers.
[14:28.540 --> 14:31.800]  You can look at application or trigger trends.
[14:31.800 --> 14:34.480]  So what I mean is, let's say you're...
[14:34.480 --> 14:36.600]  you can look at your global traffic trends
[14:36.600 --> 14:39.100]  and see how many queries you're getting.
[14:39.220 --> 14:42.080]  And then what you can do is look at your trigger trends.
[14:42.080 --> 14:43.580]  If you have a rule that has a traffic
[14:43.580 --> 14:48.220]  that follows a lot your global application traffic,
[14:48.220 --> 14:50.360]  then it's probably a false positive here.
[14:50.580 --> 14:53.580]  We took an example of something with your data dog here.
[14:54.340 --> 14:56.160]  Yeah, you have a rule here, the yellow rule,
[14:56.160 --> 14:57.520]  that's triggering a lot of the time.
[14:57.520 --> 14:59.600]  It's like, I don't know, you cannot really see it here,
[14:59.600 --> 15:04.280]  but 250k triggers most of the time
[15:04.280 --> 15:06.420]  and across all days, every day.
[15:06.440 --> 15:07.440]  So that's where we're like,
[15:07.440 --> 15:09.420]  okay, we have a false positive on our hands.
[15:09.900 --> 15:11.380]  And here, on the other side,
[15:11.380 --> 15:13.280]  you can see here a purple bump.
[15:13.280 --> 15:14.380]  It's actually a security scanner
[15:14.380 --> 15:16.320]  and this is a rule that works perfectly fine.
[15:16.320 --> 15:18.340]  You can see it's not spread all over the place.
[15:19.000 --> 15:20.660]  If you're in doubt and you don't know
[15:20.660 --> 15:22.620]  if the trends that you're seeing
[15:22.620 --> 15:26.100]  are actually kind of matching the trends of your traffic,
[15:26.100 --> 15:29.240]  you can also look at the IPs that are triggering your alerts.
[15:29.240 --> 15:31.080]  If you have a very broad range,
[15:31.080 --> 15:32.760]  meaning everyone that's coming into your application
[15:32.760 --> 15:34.640]  is actually triggering your rule,
[15:34.640 --> 15:37.700]  you also have a rule that's having a false positive issue.
[15:38.380 --> 15:39.500]  And on this side here,
[15:39.500 --> 15:41.180]  I decided to kind of show you
[15:41.180 --> 15:43.920]  what a Nelfy-ish traffic would look like.
[15:43.920 --> 15:45.880]  So you have a bunch of bumps, ups and downs.
[15:45.880 --> 15:47.200]  It's not super regular,
[15:47.200 --> 15:50.620]  so this is typically what Nelfy rules
[15:50.620 --> 15:54.380]  would be outputting in terms of number of triggers.
[15:54.720 --> 15:55.760]  All right.
[15:56.240 --> 15:58.180]  Now let's take it a step further
[15:58.180 --> 15:59.640]  and let's see if we can actually do
[15:59.640 --> 16:02.620]  any kind of machine learning path processing.
[16:03.720 --> 16:05.880]  Yeah, you can start with embeddings.
[16:05.880 --> 16:08.320]  So I don't know if any of you are familiar with embeddings,
[16:08.320 --> 16:10.800]  but they are used, for example, in LLMs.
[16:10.800 --> 16:11.960]  So it's very trendy at the moment
[16:12.780 --> 16:14.800]  and you can use them to...
[16:14.800 --> 16:16.880]  basically use them as a hash function.
[16:16.880 --> 16:18.320]  You give it a payload,
[16:18.320 --> 16:19.540]  so it could be any type of string,
[16:19.540 --> 16:22.120]  and then it will give you back a vector of numbers.
[16:22.520 --> 16:25.240]  Here, it's going to be much larger than a 2D vector,
[16:25.240 --> 16:27.140]  but for the sake of this example,
[16:27.140 --> 16:28.580]  I kept a 2D vector.
[16:29.000 --> 16:29.860]  And what you can do is,
[16:29.860 --> 16:31.940]  you put it in some kind of graph.
[16:32.700 --> 16:34.600]  Here, okay, we put it there.
[16:34.600 --> 16:37.360]  Then, as we have a new payload coming in,
[16:37.360 --> 16:39.320]  that we know is also a false positive,
[16:39.780 --> 16:42.220]  we also put it through the embedding function
[16:42.220 --> 16:43.820]  and we get a new representation.
[16:44.140 --> 16:46.160]  And they are kind of clustered together,
[16:46.160 --> 16:48.820]  so we can kind of define a false positive area,
[16:48.900 --> 16:50.020]  a false positive cluster.
[16:50.680 --> 16:51.740]  And what this means is that
[16:51.740 --> 16:53.040]  when you have a new payload coming in
[16:53.040 --> 16:54.960]  and it's actually a real trigger,
[16:55.420 --> 16:56.540]  you're able to see it,
[16:56.540 --> 16:58.700]  because it's further away from all the false positive clusters
[16:58.700 --> 17:00.100]  you've been able to build.
[17:00.240 --> 17:03.820]  So you can kind of deduce false positives
[17:03.820 --> 17:04.780]  or new false positives
[17:04.780 --> 17:06.340]  based on what you already know.
[17:07.120 --> 17:08.740]  All right, old fun games,
[17:08.740 --> 17:10.180]  but if you're lazy
[17:10.180 --> 17:12.680]  and you don't want to do all this manual clustering,
[17:12.680 --> 17:13.600]  it's very cumbersome,
[17:13.600 --> 17:16.040]  you can do some more fancy things.
[17:16.040 --> 17:18.100]  So you can do classification, for example.
[17:19.020 --> 17:20.880]  You can use machine learning models
[17:20.880 --> 17:23.160]  like the classic old-school ones,
[17:23.160 --> 17:25.420]  or you can even use OpenAI LLMs,
[17:25.420 --> 17:26.600]  if you feel like it.
[17:28.460 --> 17:29.280]  If you do that,
[17:29.280 --> 17:30.660]  some of them will work out of the box,
[17:30.660 --> 17:31.560]  so it's super nice,
[17:31.560 --> 17:32.920]  and some others,
[17:32.920 --> 17:34.620]  if you want it really tailored to your use case,
[17:34.620 --> 17:35.400]  it's not going to work,
[17:35.400 --> 17:37.000]  so you will need to fine-tune them.
[17:38.320 --> 17:39.520]  And let me tell you,
[17:39.520 --> 17:40.300]  this is a hassle,
[17:40.300 --> 17:41.100]  like we've tried,
[17:41.100 --> 17:42.760]  and this is very tricky to do.
[17:43.340 --> 17:45.600]  You need to build a legit data set,
[17:45.600 --> 17:47.140]  so a lot of real triggers
[17:47.140 --> 17:48.000]  and a lot of false positives
[17:48.000 --> 17:50.380]  to actually make it make sense.
[17:50.620 --> 17:51.100]  And here,
[17:51.100 --> 17:52.120]  I took a plain example
[17:52.120 --> 17:54.020]  that we had when we did a trial.
[17:54.080 --> 17:55.260]  So we gave it an embedding,
[17:55.420 --> 17:56.620]  a bunch of payloads,
[17:56.620 --> 17:59.580]  and we picked an OpenAI LLMs,
[17:59.580 --> 18:00.340]  we fine-tuned it,
[18:00.340 --> 18:02.960]  and we gave it an actual SQL injection payload,
[18:02.960 --> 18:03.320]  and we were like,
[18:03.320 --> 18:04.400]  OK, what is this?
[18:04.400 --> 18:05.200]  And it replied,
[18:05.200 --> 18:06.360]  it's a true attack.
[18:06.400 --> 18:07.820]  Dope, OK, it's working.
[18:08.000 --> 18:09.580]  And then we did the same thing,
[18:09.580 --> 18:11.540]  but we just put a comment after,
[18:11.540 --> 18:12.420]  and it was like,
[18:12.420 --> 18:13.020]  oh no, no,
[18:13.020 --> 18:14.280]  this is not an attack,
[18:14.280 --> 18:15.380]  this is actually just someone
[18:15.380 --> 18:16.760]  randomly coming to your application
[18:16.760 --> 18:18.360]  and pasting that.
[18:18.860 --> 18:19.680]  So yeah,
[18:19.680 --> 18:21.540]  this is a blatant case of false negatives,
[18:21.540 --> 18:22.660]  and this can happen a lot.
[18:22.660 --> 18:24.000]  So if you're going the route
[18:24.000 --> 18:25.580]  of doing some fancy machine learning
[18:25.580 --> 18:28.080]  to deal with your false positives,
[18:28.080 --> 18:30.740]  just try to build a good data set.
[18:30.940 --> 18:31.340]  And yeah,
[18:31.340 --> 18:33.760]  it's just fancy statistics, basically.
[18:34.280 --> 18:35.040]  All right.
[18:36.260 --> 18:37.700]  What if we talk about
[18:37.700 --> 18:39.340]  the blind spots of the WAF now?
[18:40.160 --> 18:41.340]  So that's the part of the talk
[18:41.340 --> 18:42.700]  where the security vendor tells you
[18:42.700 --> 18:43.240]  it's totally fine,
[18:43.240 --> 18:44.460]  the product doesn't work.
[18:44.460 --> 18:46.720]  No, more seriously,
[18:46.720 --> 18:50.080]  WAFs have a problem with obfuscation.
[18:50.240 --> 18:51.860]  If you hide your payload,
[18:51.860 --> 18:53.100]  it's going to go through.
[18:53.100 --> 18:54.340]  And this is mostly because
[18:54.340 --> 18:55.880]  of what the internet is,
[18:55.880 --> 18:57.580]  which is a bunch of RFCs
[18:57.580 --> 19:00.220]  from the late 80s, early 90s,
[19:00.220 --> 19:02.200]  and all of the weird implementation bugs
[19:02.200 --> 19:03.620]  we had since then.
[19:03.740 --> 19:04.720]  And as a vendor,
[19:04.720 --> 19:06.340]  you can only really easily
[19:06.340 --> 19:08.260]  find the first ones.
[19:08.580 --> 19:09.900]  What are we talking about
[19:09.900 --> 19:11.740]  with, like, those weird bugs?
[19:11.840 --> 19:13.520]  You have a web server
[19:13.520 --> 19:14.720]  that I'm not going to name
[19:15.540 --> 19:17.100]  that happens to perform
[19:17.720 --> 19:19.660]  recursive decoding of URLs
[19:19.660 --> 19:21.540]  and support non-serial syntax.
[19:21.740 --> 19:22.720]  What this means is that
[19:22.720 --> 19:24.300]  you have this monstrosity here
[19:24.300 --> 19:25.820]  and it's going to be converted
[19:25.820 --> 19:27.060]  into an open bracket.
[19:27.060 --> 19:27.520]  Why?
[19:27.520 --> 19:28.640]  Because, I guess,
[19:28.640 --> 19:30.260]  some engineers hate us.
[19:31.620 --> 19:32.300]  SQL.
[19:32.300 --> 19:33.240]  That's great.
[19:33.240 --> 19:34.040]  It's a language.
[19:34.040 --> 19:34.900]  It's standardized.
[19:34.980 --> 19:36.420]  Every database engine
[19:36.420 --> 19:37.540]  is using the same.
[19:37.580 --> 19:37.960]  Right?
[19:37.960 --> 19:38.280]  No.
[19:38.280 --> 19:39.080]  No, of course not.
[19:39.080 --> 19:40.060]  MySQL decided to make
[19:40.060 --> 19:41.460]  some commands conditional.
[19:41.460 --> 19:42.600]  You have this command
[19:42.600 --> 19:45.140]  and in specific version of MySQL,
[19:45.140 --> 19:45.860]  it's not going to be
[19:45.940 --> 19:46.800]  a command anymore.
[19:48.680 --> 19:49.800]  Thank you.
[19:50.920 --> 19:52.260]  Last but not least,
[19:52.260 --> 19:55.140]  you can't define a list of value
[19:55.140 --> 19:57.000]  in a URL parameter.
[19:57.180 --> 19:58.340]  Like, it's just not supported
[19:58.340 --> 19:59.060]  by the other specs.
[19:59.060 --> 20:00.020]  So, if a developer
[20:00.020 --> 20:00.660]  wants to do it,
[20:00.660 --> 20:01.260]  of course,
[20:01.260 --> 20:03.160]  they only send it in the body.
[20:03.160 --> 20:04.860]  That's what it's there for.
[20:05.240 --> 20:06.180]  No, of course not.
[20:06.180 --> 20:07.020]  Every single web server
[20:07.020 --> 20:07.780]  does something different
[20:07.780 --> 20:08.460]  and passes,
[20:09.100 --> 20:10.280]  like, turns an array of parameters
[20:10.280 --> 20:10.980]  into a list.
[20:10.980 --> 20:15.530]  The best way to deal
[20:15.530 --> 20:16.650]  with this kind of obfuscation
[20:16.650 --> 20:19.750]  is to work around it completely.
[20:19.750 --> 20:21.010]  We have one advantage
[20:21.010 --> 20:22.290]  as defenders,
[20:22.290 --> 20:23.970]  which is that we know
[20:23.970 --> 20:24.970]  what the application
[20:24.970 --> 20:27.050]  on the other side of the WAF is.
[20:27.050 --> 20:28.170]  We have some insight.
[20:28.170 --> 20:28.730]  We know that we could
[20:28.730 --> 20:29.890]  know the language,
[20:29.890 --> 20:30.790]  the framework,
[20:30.790 --> 20:32.050]  what it's expecting.
[20:32.370 --> 20:34.070]  And using that,
[20:34.070 --> 20:35.110]  you could tell the WAF
[20:35.110 --> 20:37.750]  to decode the incoming parameter
[20:37.750 --> 20:39.330]  just like the application
[20:39.330 --> 20:40.350]  would do it.
[20:40.350 --> 20:40.850]  It does not mean
[20:40.850 --> 20:41.450]  that we would be able
[20:41.450 --> 20:42.910]  to decode everything.
[20:43.110 --> 20:43.730]  But in practice,
[20:43.730 --> 20:44.450]  you don't care.
[20:44.450 --> 20:45.210]  Because if the application
[20:45.210 --> 20:46.550]  can't decode a parameter,
[20:47.250 --> 20:49.450]  then the parameter can't harm it.
[20:50.090 --> 20:51.350]  It's not perfect.
[20:51.350 --> 20:52.370]  Specifically, it won't work
[20:52.370 --> 20:53.570]  if the parameter is integrated
[20:53.570 --> 20:54.570]  by something else,
[20:54.570 --> 20:56.610]  so specifically XSS,
[20:56.610 --> 20:57.390]  where the final target
[20:57.390 --> 20:58.510]  is a browser.
[20:58.510 --> 20:59.590]  But it's very effective
[20:59.590 --> 21:00.570]  against all kinds
[21:00.570 --> 21:02.930]  of URL encoding,
[21:02.930 --> 21:04.230]  HTTP shenanigans.
[21:04.230 --> 21:05.630]  Like, if you can just extract
[21:05.630 --> 21:06.210]  what the application
[21:06.210 --> 21:08.450]  is going to see,
[21:08.450 --> 21:09.510]  then you're golden.
[21:09.510 --> 21:12.050]  Moreover, another problem
[21:12.050 --> 21:13.130]  WAFs need to deal with
[21:13.130 --> 21:14.610]  is performance.
[21:14.650 --> 21:15.990]  The way most WAFs are set up
[21:15.990 --> 21:17.310]  is they add their latency
[21:17.310 --> 21:18.450]  to the response time
[21:18.450 --> 21:19.550]  of your application.
[21:19.610 --> 21:21.550]  Meaning, if your application
[21:22.470 --> 21:24.270]  answers in, like, two milliseconds,
[21:24.270 --> 21:25.770]  and you spent a lot of resources
[21:25.770 --> 21:26.990]  to optimize it,
[21:26.990 --> 21:28.610]  and the WAF just barges in
[21:28.610 --> 21:29.650]  and adds, like, 100 milliseconds
[21:29.650 --> 21:30.830]  of latency,
[21:30.830 --> 21:32.510]  that's unacceptable.
[21:32.790 --> 21:34.450]  So WAFs need to be able
[21:34.450 --> 21:36.150]  to fit within those constraints.
[21:36.330 --> 21:39.010]  Moreover, even if you're not
[21:39.010 --> 21:40.270]  under attack,
[21:40.270 --> 21:42.230]  we notice that even a small
[21:42.230 --> 21:43.430]  minority of long payload
[21:43.430 --> 21:45.790]  is enough to tank the performance.
[21:45.950 --> 21:48.130]  Like, surprisingly little.
[21:49.670 --> 21:50.850]  What this means is that
[21:50.850 --> 21:52.390]  you have a few constraints
[21:52.390 --> 21:53.590]  that WAFs are never really
[21:53.590 --> 21:54.650]  going to be able to
[21:55.170 --> 21:56.390]  work around of.
[21:56.390 --> 21:57.530]  So, for instance,
[21:57.530 --> 21:58.550]  if your WAF is not passing
[21:58.550 --> 21:59.390]  the body,
[21:59.390 --> 22:00.630]  and the application is expecting
[22:00.810 --> 22:02.390]  a large JSON payload,
[22:02.390 --> 22:03.710]  then all the attacker has to do
[22:03.710 --> 22:04.830]  is to put the attack
[22:04.830 --> 22:05.770]  at the end of the payload,
[22:05.770 --> 22:07.230]  and they're in.
[22:09.010 --> 22:11.530]  Some WAFs also have some constraints
[22:11.530 --> 22:12.930]  on the time they are allowed to take.
[22:12.930 --> 22:14.370]  And if they start taking, like,
[22:14.370 --> 22:15.730]  let's say more than five milliseconds,
[22:15.730 --> 22:16.610]  all right, they disconnect,
[22:16.610 --> 22:17.270]  they are like,
[22:17.270 --> 22:18.470]  there's too much impact.
[22:19.310 --> 22:20.690]  And all an attacker has to do
[22:20.690 --> 22:21.530]  is to increase the load
[22:21.530 --> 22:22.870]  on the WAF just by sending
[22:23.070 --> 22:25.690]  a bunch of requests,
[22:25.690 --> 22:27.650]  and, yeah, they're in.
[22:28.670 --> 22:29.830]  And, yeah, finally,
[22:29.830 --> 22:30.610]  the obfuscation,
[22:30.610 --> 22:31.850]  the WAF can do it.
[22:31.910 --> 22:33.810]  It can try to parse the parameter
[22:33.810 --> 22:35.230]  as a browser would do it,
[22:35.230 --> 22:36.250]  but it's expensive.
[22:36.250 --> 22:37.850]  So you need to figure out
[22:37.850 --> 22:39.250]  how do you want to spend your budget.
[22:39.250 --> 22:40.150]  And you might decide,
[22:40.150 --> 22:42.990]  okay, this decoding would be very effective,
[22:42.990 --> 22:43.630]  but actually if I use it
[22:43.630 --> 22:44.470]  on all of my rules,
[22:44.470 --> 22:46.210]  I'm, like, spending half my time
[22:46.210 --> 22:46.950]  just doing decoding
[22:46.950 --> 22:49.150]  and not actually running rules.
[22:50.250 --> 22:52.130]  There are a few solutions to that,
[22:52.130 --> 22:54.590]  such as running the WAF async.
[22:55.130 --> 22:56.970]  They all have, like, terrible problems,
[22:56.970 --> 22:58.410]  from not being able to block
[22:58.410 --> 23:00.730]  to you could direct all of your traffic
[23:00.730 --> 23:02.590]  to some kind of cloud infrastructure,
[23:02.590 --> 23:04.450]  but then you're adding latency,
[23:04.450 --> 23:05.350]  you're saying all of your traffic
[23:06.010 --> 23:07.110]  don't know where.
[23:07.390 --> 23:09.310]  So, yeah, that's kind of terrible.
[23:11.570 --> 23:12.930]  Finally, WAF,
[23:12.930 --> 23:14.110]  and I'm going to blow your mind,
[23:14.110 --> 23:16.170]  don't know what they don't see.
[23:16.190 --> 23:17.530]  So if you have an attack
[23:17.530 --> 23:18.610]  that the WAF don't catch,
[23:18.970 --> 23:20.070]  the WAF won't be able to tell you
[23:20.070 --> 23:21.670]  that you are under attack.
[23:22.070 --> 23:23.050]  What this means is that
[23:23.050 --> 23:24.950]  to be effective with WAF,
[23:24.950 --> 23:26.450]  you kind of need to do threat intel.
[23:26.450 --> 23:27.450]  You need to be able to do
[23:27.450 --> 23:30.650]  quick and quiet deploy of rules
[23:30.650 --> 23:31.550]  to be able to know
[23:32.250 --> 23:33.510]  what's happening in the wild.
[23:33.510 --> 23:35.330]  Doing some kind of shadow deployments.
[23:36.050 --> 23:37.710]  The scale at which you do that
[23:37.710 --> 23:39.350]  depends on what you can do.
[23:39.350 --> 23:40.290]  Like, it can be an only pot,
[23:40.290 --> 23:41.550]  but then you're mostly going to catch
[23:42.010 --> 23:43.570]  against the noise of the internet.
[23:43.870 --> 23:45.050]  You may be able to do it
[23:45.050 --> 23:46.770]  on your internal services,
[23:46.770 --> 23:49.510]  but, yeah, you'll get a better idea
[23:49.510 --> 23:50.970]  of what an actual attacker
[23:50.970 --> 23:52.690]  is going to do for your kind of application.
[23:52.950 --> 23:54.110]  And if you're a vendor
[23:54.110 --> 23:55.770]  and can work with it
[23:55.770 --> 23:57.230]  with your customers,
[23:57.230 --> 23:58.270]  you can even deploy it
[23:58.270 --> 23:59.290]  in customer environments.
[23:59.290 --> 24:00.510]  And that's amazing,
[24:00.510 --> 24:01.210]  because then you see
[24:01.750 --> 24:02.970]  a very broad diversity
[24:02.970 --> 24:04.830]  of attacks.
[24:04.830 --> 24:06.010]  Like, you see what the actual
[24:06.010 --> 24:07.910]  threat landscape is like.
[24:08.290 --> 24:09.890]  And when you do
[24:10.430 --> 24:11.970]  and you catch someone,
[24:12.670 --> 24:14.010]  WAFs are really good
[24:14.010 --> 24:15.430]  at leveraging mistakes.
[24:15.430 --> 24:17.990]  If you find any identifying pattern
[24:17.990 --> 24:19.470]  in an attacker,
[24:19.470 --> 24:20.690]  you can leverage it.
[24:20.690 --> 24:22.070]  Be it user agent,
[24:22.070 --> 24:23.910]  they're using this old version of Firefox.
[24:23.930 --> 24:24.770]  All right, I'm just going to match
[24:24.770 --> 24:25.910]  everything from that.
[24:25.910 --> 24:26.550]  It could be the IP
[24:26.550 --> 24:27.990]  or the range of IPs.
[24:27.990 --> 24:28.970]  It could be the domain
[24:28.970 --> 24:30.670]  they're fetching their attacks from.
[24:30.670 --> 24:31.910]  It's very common for accesses
[24:31.910 --> 24:34.370]  to load the payload
[24:34.370 --> 24:36.490]  from some given domain.
[24:36.670 --> 24:38.330]  Or, and that's the best one, typos.
[24:38.550 --> 24:39.510]  The attacker does a typo.
[24:39.510 --> 24:40.450]  All right, I just want to match
[24:40.450 --> 24:44.670]  all of the typo headers.
[24:44.730 --> 24:46.470]  And then once you have all of that,
[24:46.470 --> 24:48.230]  you dump all of the attacker traffic
[24:48.710 --> 24:50.910]  and you burn it all at once.
[24:51.170 --> 24:53.270]  And yeah, destroy that toolkit.
[24:55.750 --> 24:57.370]  So, we talked a bit about
[24:57.370 --> 24:58.510]  what WAFs are,
[24:58.510 --> 25:00.170]  what their issues are,
[25:00.170 --> 25:01.290]  what the false positive,
[25:01.290 --> 25:02.090]  false negative,
[25:02.090 --> 25:03.110]  what they're good at.
[25:03.110 --> 25:04.630]  Let's talk about making them better
[25:04.630 --> 25:06.050]  for end users.
[25:06.950 --> 25:09.410]  The first and most important point
[25:09.410 --> 25:11.170]  is you need to
[25:12.430 --> 25:13.590]  embrace the strength
[25:13.590 --> 25:14.810]  of the WAF
[25:15.470 --> 25:17.430]  and stop seeing a WAF
[25:17.430 --> 25:19.010]  as a protection tool.
[25:19.610 --> 25:21.330]  It won't be very good at that.
[25:21.330 --> 25:22.370]  If you want something to catch
[25:22.370 --> 25:23.610]  every single exploit
[25:24.110 --> 25:26.230]  going to your application,
[25:26.230 --> 25:26.970]  there are things,
[25:26.970 --> 25:28.950]  there are products that do that.
[25:28.950 --> 25:30.250]  There are RAS, there are IAS.
[25:30.370 --> 25:31.810]  There are some solutions,
[25:31.810 --> 25:32.750]  but WAFs are not it.
[25:32.750 --> 25:33.490]  What a WAF is,
[25:33.490 --> 25:34.810]  is a monitoring tool.
[25:34.810 --> 25:35.890]  A WAF will tell you
[25:35.890 --> 25:36.770]  that you're under attack
[25:36.770 --> 25:38.110]  and give you a feeling
[25:38.110 --> 25:39.010]  for the kind of attacks
[25:39.010 --> 25:40.310]  that the attacker is sending.
[25:40.310 --> 25:42.110]  It won't give you everything.
[25:42.250 --> 25:43.550]  If you wanted to do it,
[25:43.550 --> 25:44.270]  you can.
[25:44.290 --> 25:45.310]  It's called, like,
[25:45.310 --> 25:46.370]  here, this rule,
[25:46.370 --> 25:47.230]  trade secret.
[25:47.270 --> 25:48.710]  You just match that star.
[25:49.210 --> 25:50.490]  You can get everything.
[25:50.730 --> 25:52.330]  It will be totally unusable.
[25:54.910 --> 25:56.290]  Next one,
[25:56.290 --> 25:57.130]  rule testing.
[25:57.890 --> 26:00.750]  You can have a good turnkey experience
[26:00.750 --> 26:01.850]  in a WAF.
[26:01.850 --> 26:02.850]  But for that,
[26:02.850 --> 26:04.370]  rules need to have been tested
[26:04.370 --> 26:07.110]  on every single kind of workloads.
[26:07.150 --> 26:10.130]  And this requires massive scale.
[26:10.130 --> 26:11.470]  On top of it,
[26:11.470 --> 26:13.330]  whoever is doing the rule qualification
[26:13.330 --> 26:14.930]  will need to decide
[26:14.930 --> 26:17.750]  how much noise is good enough.
[26:17.750 --> 26:18.770]  And that's going to depend
[26:18.770 --> 26:21.850]  from a deployment perspective,
[26:21.850 --> 26:23.350]  from a company perspective,
[26:23.350 --> 26:24.470]  from a... yeah.
[26:24.470 --> 26:27.230]  It even changes on an engineer basis.
[26:29.090 --> 26:30.070]  The best way
[26:30.070 --> 26:31.190]  for someone to guarantee
[26:31.190 --> 26:32.070]  that the rule is going to work
[26:32.070 --> 26:33.650]  is to do some kind of shadow deployment,
[26:33.650 --> 26:34.610]  meaning running the rules
[26:34.610 --> 26:35.470]  on your environment
[26:35.470 --> 26:36.690]  without blocking.
[26:37.630 --> 26:38.890]  And, yeah.
[26:39.170 --> 26:40.190]  Doing all of that,
[26:40.190 --> 26:40.650]  you need to make sure
[26:40.650 --> 26:41.630]  not to catch any PII
[26:41.630 --> 26:43.050]  because otherwise you're in trouble.
[26:45.050 --> 26:45.890]  All right.
[26:45.890 --> 26:46.730]  Next up,
[26:46.730 --> 26:47.610]  you can try to leverage
[26:47.610 --> 26:49.090]  your application context.
[26:49.090 --> 26:49.630]  Meaning,
[26:49.630 --> 26:51.270]  if you're able to give data
[26:51.270 --> 26:52.710]  about your application to your WAF,
[26:52.710 --> 26:53.450]  like, telling it
[26:53.450 --> 26:54.970]  which language you're running,
[26:54.970 --> 26:55.350]  telling it
[26:55.350 --> 26:57.310]  which framework you're using,
[26:57.310 --> 26:58.470]  then, in that case,
[26:58.470 --> 26:59.290]  you'll be able
[26:59.290 --> 27:00.410]  to increase the precision.
[27:00.410 --> 27:01.590]  Let me give you an example.
[27:01.610 --> 27:01.990]  You're running
[27:02.170 --> 27:03.650]  a JavaScript application.
[27:03.650 --> 27:04.370]  You're receiving a bunch
[27:04.370 --> 27:05.410]  of PHP attacks.
[27:05.410 --> 27:06.610]  Well, you don't really care
[27:06.610 --> 27:07.490]  about those.
[27:07.490 --> 27:08.210]  They're not going to hurt
[27:08.210 --> 27:09.090]  your application.
[27:09.090 --> 27:10.510]  So, what you need to do
[27:10.510 --> 27:12.090]  is just discard those.
[27:12.330 --> 27:12.990]  If you're running
[27:12.990 --> 27:14.930]  some kind of PHP,
[27:14.930 --> 27:15.350]  for example,
[27:15.350 --> 27:17.370]  the SilverStripe framework,
[27:17.370 --> 27:18.750]  it was vulnerable to XSS
[27:18.750 --> 27:20.030]  in older versions.
[27:20.710 --> 27:21.690]  Well, if you're not running,
[27:21.690 --> 27:22.090]  if you're running
[27:22.090 --> 27:23.250]  up to the latest version,
[27:23.250 --> 27:23.910]  you also don't care
[27:23.910 --> 27:24.450]  about the alerts
[27:24.450 --> 27:25.190]  that are targeting
[27:25.190 --> 27:26.310]  the older version.
[27:26.310 --> 27:27.570]  So, yeah.
[27:27.570 --> 27:28.490]  If you're able to do that,
[27:28.490 --> 27:29.990]  you'll increase your accuracy.
[27:30.110 --> 27:31.210]  And one other side effect
[27:31.210 --> 27:32.450]  that can be good for you
[27:32.450 --> 27:33.610]  is you're also able
[27:33.610 --> 27:35.330]  to detect threat actors
[27:35.330 --> 27:36.270]  that will be sending,
[27:36.270 --> 27:36.670]  for example,
[27:36.670 --> 27:37.870]  Java attacks to your app
[27:37.870 --> 27:38.930]  when you're not running
[27:38.930 --> 27:39.450]  any Java,
[27:39.450 --> 27:40.710]  and you can kind of increase
[27:41.530 --> 27:42.330]  the attention
[27:42.330 --> 27:42.910]  that you're giving
[27:42.910 --> 27:44.050]  to this malicious IP
[27:44.050 --> 27:44.870]  and gather a bunch
[27:44.870 --> 27:45.390]  of threat intel
[27:45.390 --> 27:46.390]  on top of that.
[27:46.810 --> 27:47.950]  The only downside
[27:47.950 --> 27:48.570]  is it's going to be
[27:48.650 --> 27:49.450]  a bit challenging,
[27:49.450 --> 27:50.790]  meaning you have to have
[27:50.790 --> 27:51.690]  your security team
[27:51.690 --> 27:53.010]  work with your engineering team,
[27:53.010 --> 27:53.570]  and God knows
[27:53.570 --> 27:54.530]  it can be tricky
[27:54.530 --> 27:56.150]  to do sometimes.
[27:56.310 --> 27:57.870]  So, if you're able
[27:57.870 --> 27:58.530]  to do that,
[27:58.530 --> 27:58.910]  you can build
[27:58.910 --> 28:00.530]  this feedback loop
[28:00.530 --> 28:02.090]  that will be improving
[28:02.090 --> 28:02.410]  greatly
[28:02.410 --> 28:03.250]  your experience
[28:03.250 --> 28:04.210]  with the WAF.
[28:05.790 --> 28:07.370]  And the last element
[28:07.370 --> 28:08.110]  that you can try
[28:08.110 --> 28:09.150]  is to tune the response
[28:09.150 --> 28:09.650]  to the attack,
[28:09.650 --> 28:10.870]  meaning if you're able
[28:10.870 --> 28:11.950]  to catch a specific IP
[28:11.950 --> 28:12.690]  that's been targeting
[28:12.690 --> 28:13.270]  your application
[28:13.270 --> 28:14.490]  for a while,
[28:14.490 --> 28:15.510]  you could dynamically
[28:15.510 --> 28:16.590]  roll out a new rule set
[28:16.590 --> 28:17.730]  for this specific IP
[28:17.730 --> 28:19.470]  and catch more attacks
[28:19.470 --> 28:20.770]  coming from it,
[28:20.770 --> 28:21.210]  and you could catch
[28:21.210 --> 28:22.170]  everything and see
[28:22.170 --> 28:23.750]  what's coming up.
[28:23.770 --> 28:25.130]  And, yeah,
[28:25.130 --> 28:25.570]  if you're able
[28:25.570 --> 28:26.050]  to do that,
[28:26.050 --> 28:27.210]  and on top of that,
[28:27.670 --> 28:28.610]  what you could try,
[28:28.610 --> 28:29.270]  if you're unsure
[28:29.270 --> 28:30.090]  if the actor
[28:30.090 --> 28:31.610]  is actually malicious,
[28:31.610 --> 28:32.510]  we should be able
[28:32.510 --> 28:33.730]  to enable
[28:33.730 --> 28:34.530]  different types
[28:35.170 --> 28:36.270]  of responses.
[28:36.270 --> 28:36.590]  First,
[28:36.590 --> 28:37.310]  we should be able
[28:37.310 --> 28:38.690]  to have a capture
[28:38.690 --> 28:39.810]  and then only
[28:39.810 --> 28:40.930]  block after.
[28:41.910 --> 28:42.530]  Remember,
[28:42.530 --> 28:42.990]  we talked a bit
[28:42.990 --> 28:44.230]  about how we can do
[28:44.230 --> 28:45.450]  some analysis
[28:45.450 --> 28:46.410]  of false positives.
[28:46.410 --> 28:47.270]  If we're also able
[28:47.270 --> 28:48.190]  to leverage that
[28:48.190 --> 28:49.030]  to feed that back
[28:49.030 --> 28:49.650]  to the WAF
[28:49.650 --> 28:51.610]  and build dynamically
[28:51.810 --> 28:52.730]  a bunch of passive rules
[28:52.730 --> 28:53.510]  so we don't have
[28:53.510 --> 28:54.510]  as many false positives,
[28:54.510 --> 28:55.570]  it's even better.
[28:55.570 --> 28:56.190]  We could be even
[28:56.190 --> 28:56.890]  more confident
[28:56.890 --> 28:57.950]  when we actually enable
[28:57.950 --> 28:59.030]  the WAF in blocking
[28:59.030 --> 29:00.250]  mode on some rules.
[29:01.190 --> 29:02.050]  All right,
[29:02.050 --> 29:02.970]  this is our
[29:02.970 --> 29:04.210]  final takeaways.
[29:04.830 --> 29:06.890]  If you're a defender,
[29:06.890 --> 29:07.590]  look at your WAF
[29:07.590 --> 29:08.790]  more as a monitoring system
[29:08.790 --> 29:11.030]  than as a protection system.
[29:11.250 --> 29:12.050]  You should try to
[29:12.050 --> 29:12.830]  launch your false positives
[29:12.830 --> 29:13.530]  because this way
[29:13.530 --> 29:14.090]  you will be able
[29:14.090 --> 29:14.930]  to leverage
[29:14.930 --> 29:16.390]  the actual alerts
[29:16.390 --> 29:17.350]  that you get.
[29:17.350 --> 29:18.610]  And if you're building
[29:18.610 --> 29:19.470]  some kind of rule set,
[29:19.470 --> 29:20.390]  make sure that you keep in
[29:20.390 --> 29:21.090]  some flexibility
[29:21.090 --> 29:22.330]  for your engineering team.
[29:22.330 --> 29:23.390]  Meaning if you do
[29:23.390 --> 29:24.650]  very restrictive
[29:24.650 --> 29:25.430]  type of rule set
[29:25.430 --> 29:26.410]  and as soon as
[29:26.410 --> 29:27.010]  your engineers
[29:27.010 --> 29:27.970]  are changing the app,
[29:27.970 --> 29:28.850]  they're breaking everything,
[29:28.850 --> 29:30.050]  it's not worth it.
[29:30.470 --> 29:31.670]  If you are a vendor,
[29:31.670 --> 29:32.550]  well, I think
[29:32.550 --> 29:33.110]  it should be
[29:33.110 --> 29:33.990]  our collective mission
[29:33.990 --> 29:35.730]  to propose
[29:35.730 --> 29:36.850]  to our users
[29:37.210 --> 29:38.910]  a very low noise
[29:38.910 --> 29:39.870]  rule set.
[29:40.110 --> 29:40.530]  And then
[29:41.330 --> 29:42.390]  also support them
[29:42.390 --> 29:43.230]  in their efforts
[29:43.230 --> 29:44.390]  to actually analyze
[29:44.390 --> 29:44.970]  their triggers
[29:44.970 --> 29:45.710]  and the trends
[29:45.710 --> 29:46.770]  that they have.
[29:47.530 --> 29:48.130]  And if you're collecting
[29:48.130 --> 29:48.610]  any type of
[29:48.610 --> 29:49.450]  threat intelligence,
[29:49.450 --> 29:50.110]  please share it
[29:50.110 --> 29:51.470]  across all of your customers
[29:51.470 --> 29:52.390]  so everyone can benefit
[29:52.390 --> 29:53.230]  from it.
[29:54.070 --> 29:54.790]  And last,
[29:54.790 --> 29:55.650]  if you're an attacker,
[29:55.650 --> 29:56.530]  well, you're in luck
[29:56.530 --> 29:57.370]  because there will
[29:57.370 --> 29:58.470]  always be bypasses
[29:58.470 --> 29:59.070]  to the WAF.
[29:59.070 --> 30:00.010]  We've talked about them
[30:00.690 --> 30:02.270]  because of the WAF's nature.
[30:02.570 --> 30:03.610]  So keep in mind
[30:03.610 --> 30:04.230]  that you should probably
[30:04.230 --> 30:05.050]  rotate your infrastructure
[30:05.050 --> 30:06.530]  because we will leverage
[30:06.530 --> 30:07.070]  all the mistakes
[30:07.070 --> 30:07.690]  that you make
[30:07.690 --> 30:09.370]  and if you don't rotate,
[30:09.370 --> 30:10.070]  we'll catch you
[30:10.070 --> 30:10.770]  and we'll thank you
[30:10.770 --> 30:11.270]  for that.
[30:11.270 --> 30:11.670]  And then
[30:12.430 --> 30:13.410]  if you have to invest
[30:13.410 --> 30:14.810]  in one type of application,
[30:14.810 --> 30:16.170]  please only invest
[30:16.430 --> 30:17.070]  in ambiguous
[30:17.070 --> 30:18.270]  interpreted obfuscation.
[30:18.270 --> 30:20.790]  HTTP is worthless.
[30:21.130 --> 30:22.150]  Alright,
[30:22.150 --> 30:24.050]  we're at the end.
[30:24.170 --> 30:24.730]  Thank you for
[30:24.730 --> 30:25.890]  listening to us.
[30:25.890 --> 30:26.390]  applause
[30:29.390 --> 30:29.850]  music
[30:29.850 --> 30:30.470]  music
[30:30.470 --> 30:31.610]  music
[30:31.610 --> 30:31.690]  music