WEBVTT 00:08.280 --> 00:11.160 Thanks for having me here at the AppSec Village. 00:11.160 --> 00:14.720 So, I'm very glad to be here in person. 00:14.720 --> 00:19.140 The last talk that I did, it was a virtual event. 00:19.140 --> 00:26.700 We were in the middle of COVID, so we, like, I did a talk in AI Village, also Red Team Village. 00:26.700 --> 00:33.100 So, it's good to be back in person and doing this with the wired mic. 00:33.100 --> 00:41.280 I feel like an 80s comedian, so I have to, like, make sure that my movement is within this range. 00:41.500 --> 00:53.520 So, I will be talking about my research on the use of generative adversarial networks for automating the penetration testing of web application firewalls. 00:53.520 --> 01:01.260 So, it's an appliance that we have all been using, and here is a little bit about myself. 01:01.260 --> 01:04.000 I am a cybersecurity researcher. 01:04.000 --> 01:10.600 I work for a company, Sixense, as a senior AppSec engineer. 01:10.780 --> 01:15.720 The company builds AI-based sales software. 01:16.280 --> 01:25.020 So, that is one of the reasons that I joined the organization, because they were using AI for building cool products. 01:25.020 --> 01:29.580 And my background was mainly in use of AI in cybersecurity. 01:29.580 --> 01:38.880 So, my PhD, I did with specialization of using artificial intelligence in the field of cybersecurity. 01:38.880 --> 01:45.580 I also work with a lot of students in cybersecurity defense competition. 01:45.580 --> 01:57.340 So, if you are looking for talented individuals who have gone through our CCDC program, shoot me a message on LinkedIn and I can connect you to the folks. 01:58.220 --> 02:02.540 And most of my research information is on my website. 02:03.840 --> 02:10.760 So, here is a brief overview of what we will be discussing during the next half an hour or so. 02:10.760 --> 02:16.700 We will be talking about what kind of web application threats that we are looking at. 02:16.700 --> 02:21.660 We will look at what a web application firewall looks like. 02:21.660 --> 02:27.300 And then, where does this generative adversarial network come into picture? 02:27.300 --> 02:32.440 Like, why are we using this technology to bombard these WAFs? 02:32.600 --> 02:38.560 We will look at a particular case of a firewall known as mod security. 02:38.560 --> 02:44.720 I am doing some experimental work on Cloudflare and AWS WAF as well. 02:44.720 --> 02:51.560 So, I'll talk about that and then we will conclude the talk. 02:51.560 --> 02:59.640 So, we have all been working with different web application technologies, like application servers. 02:59.640 --> 03:08.100 We have NGINX, Apache, stuff like that protecting our backend infrastructure. 03:08.100 --> 03:17.280 We have multiple technologies, like we were using SQL, then we started using NoSQL, GraphQL. 03:17.280 --> 03:41.500 A lot of technology is coming up and a lot of the issues that have been there since a long time, like weak TLS, like server-side encryption not being up to par, like we want to use 256-bit, but the application using 128-bit of encryption. 03:41.500 --> 03:45.840 So, a lot of these issues in the web application threats. 03:45.840 --> 04:12.120 We see a lot of issues in the client and server-side technologies that render us vulnerable to cross-site scripting attacks, XXE, and a lot of weak protocols in terms of how we do authentication and authorization, like now we know that we had authentication bypass as we are moving towards API technologies. 04:12.120 --> 04:25.500 We see a lot of insecure direct object reference IDORs and things like that, and we have multiple types of data stores at the backend. 04:25.500 --> 04:32.520 So, we have to understand the logic flow there and deal with those security issues. 04:33.840 --> 04:48.500 So, if we look at the OWASP top 10 over the years, some of the issues like broken access control, weak cryptographic issues, they have been there for a long time. 04:48.640 --> 04:52.360 Some of you might be doing a lot of design reviews as well. 04:52.360 --> 04:53.640 We call it threat modeling. 04:53.640 --> 04:57.180 So, we want to analyze trust boundaries. 04:57.420 --> 05:02.980 Those are some additional issues that have came up in recent times. 05:03.300 --> 05:11.280 Vulnerable and outdated software components, we all do patch Tuesdays like if you are a Microsoft shop. 05:11.280 --> 05:22.220 So, a lot of that has been present and some of it is gaining higher volume as we deploy more and more web technologies. 05:23.420 --> 05:33.620 And monitoring and logging, like a lot of you going through SOC audits and compliance requirements, you need to provide proof of your logs. 05:33.620 --> 05:35.080 Are you doing monitoring? 05:35.080 --> 05:36.700 Are you doing it enough? 05:37.000 --> 05:46.560 If you do it in an insecure manner, does it disclose some sort of key information in your data? 05:48.480 --> 05:53.560 So, let's have a look at the typical example of a web attack. 05:53.560 --> 06:05.580 Like, if I am talking about cross-site scripting, the user logs into an application and suppose it has a DOM-based XSS vulnerability. 06:05.580 --> 06:15.520 So, user, being naive, hovers over a comment that has the script injected by the malicious attacker. 06:15.520 --> 06:31.360 And that triggers the user request to the attacker's URL, like an ATOI function or a JavaScript call to a remote server that is owned by the attacker. 06:31.360 --> 06:43.620 So, this malicious response executes the attacker's JavaScript and you end up providing your session token to the attacker. 06:45.060 --> 07:00.760 Nowadays, we have protection like MFA to make sure that we are having an additional layer of protection, but we also have attacks like evil jinx and stuff like that that can do even MFA bypass. 07:04.100 --> 07:18.200 So, we have defense mechanisms in place as well, like we have intrusion detection system, intrusion prevention system, firewalls at multiple layers, and these are based on signature database. 07:18.380 --> 07:26.620 And they are as good as the attack signatures we have configured for these tools. 07:26.760 --> 07:41.820 We also have input sanitization, like we want to make sure that only trusted set of characters are allowed, or we also do encoding to deal with this input data. 07:42.300 --> 07:58.040 And there are truncation techniques in place as well, like we want to make sure that we are not ingesting a huge amount of payload as input, so we have defense in places. 07:58.520 --> 08:06.340 And web application firewall, it combines a series of defense mechanisms. 08:06.440 --> 08:23.000 So, you might be using something like AWS WAF or CloudFlare, and you have user agent based blocking, you have your rate limiting rules in place, things of that nature. 08:23.600 --> 08:31.340 So, web application firewall combines some of these techniques like these attack signatures in its database. 08:32.780 --> 08:54.680 So, essentially it's a layer 7 appliance that looks at the incoming metadata at the layer 7, it checks what kind of user agent do you have, like if you have wget or curl, suppose, in your user agent, or if it's a crawling bot that is hitting your application, 08:54.680 --> 09:00.090 you can basically block that based on the attack signature. 09:01.420 --> 09:23.940 And as you can see, it can be deployed in front of NGNX, your application load balancer in AWS, and it goes through its signature list, and if there is a positive match on that attack signature, it will drop that traffic, it will rate limit, whatnot. 09:23.940 --> 09:25.900 You can do a lot of customization. 09:26.760 --> 09:38.860 So, if it is based on a fixed set of rules, the question we can ask is like, can there be a variation of that rule that somebody can exploit? 09:42.120 --> 09:49.440 So, let's talk about the tale of web application firewall, mod security based WAF. 09:49.520 --> 10:16.660 It is deployed across a lot of known web technologies like Apache, NGNX, ISS, and what it has like a typical WAF is a core set of rules that checks the signature for OWASP top 10 or bad user agent, like if you see a curl request coming in your user agent, 10:16.660 --> 10:19.060 you have rules to block that out. 10:19.060 --> 10:30.720 It looks for SQL injections, session hijacking type of exploits, so it has different configuration levels. 10:30.720 --> 10:40.580 Basically, there is some testing that is done across multiple levels that you get in the web application firewall. 10:40.960 --> 10:59.200 Starting with the paranoia level zero, where you get essentially no protection, to level one, where it is minimal set of rules, some false positives, to paranoia level two, where you have a lot more false positives, a lot broader set of rules. 10:59.580 --> 11:17.120 If you are using something like AWS WAF, you can have rules from Fortinet, some other vendors, so it is essentially like an expanded set of rules as you go through the different paranoia levels. 11:17.820 --> 11:28.740 So at level three, you have rules that target PII, bank level information, like DLP protection, things of that nature. 11:28.780 --> 11:34.740 And level four is like a full nuclear level, so you have a lot more rules. 11:34.740 --> 11:53.320 But as the number of rules grow in size, you get a lot of false positives as well, and the graph on the right side of the picture shows the requests that were done with the different attack types. 11:53.320 --> 12:06.330 So there were redirect attempts, LFI, RFI, and you can see that as the PL increases, it becomes incredibly hard to bypass the firewall filters. 12:06.330 --> 12:13.420 So even at level two and three, most of these payloads are blocked based on signature. 12:16.300 --> 12:20.400 So how do we circumvent this? 12:20.400 --> 12:49.800 If we look at an application that is running a PHP system command and it takes the user input and basically prints that out and executes this, and if this application supposes a NGINX server, it is protected by the mod security WAF, and I try to send this command like a bincat etc password, 12:49.800 --> 12:54.640 this will be blocked because there is a direct signature. 12:55.420 --> 13:02.860 What if I try to do a variation of this and try to send something that has wildcard characters? 13:03.020 --> 13:14.660 That will be blocked as well because how the WAF operates is it de-obfuscates the payload and executes it in the interpreted context of the browser. 13:15.580 --> 13:23.840 But if you do a question mark base like individual character substitution, you can see that payload passes through. 13:23.840 --> 13:26.160 WAF does not block that. 13:26.660 --> 13:30.820 But for how long can I do it manually? 13:30.820 --> 13:40.220 As a web application tester, do you want to test individual payloads one by one, finding those minor variations and testing those out? 13:40.220 --> 13:42.340 That's very, very difficult. 13:44.800 --> 13:51.560 So it's a game of cat and mouse, and that is the motivation of generative adversarial networks. 13:52.740 --> 14:00.740 So how do you play that kind of cat and mouse game with the web application firewall and try to fool it out? 14:00.880 --> 14:07.380 So on the left you can see some blocked expressions that will be blocked by the firewall. 14:07.840 --> 14:22.400 But as we know that we have different browsers, like some are running maybe Silverlight, old legacy systems, some are on .NET, so you can have variations of these payloads that will basically pass through. 14:22.640 --> 14:35.820 So for each blocked expression, you can have analogous variation that will bypass the web application firewall signature and basically pass through. 14:38.320 --> 15:01.220 So talking about generative modeling, it's essentially an unsupervised technique, and unsupervised learning technique, basically it tries to identify some emerging patterns or regularities in the data, comparing it to a supervised model where we have everything labeled, 15:01.520 --> 15:09.260 like you will have your training data labeled, test data labeled in case of spam filtering and all. 15:09.860 --> 15:27.240 Unsupervised modeling is based on some patterns where you don't know the exact labels, but you can bucket common behavior together like buying patterns of users on Amazon or something, so clustering is an example of unsupervised modeling. 15:28.020 --> 15:32.660 So generative adversarial network does something intelligent. 15:32.660 --> 15:54.420 It projects this unsupervised learning technique as a supervised model where it provides the labels of correct and fake, like basically correct attack patterns or fake attack patterns to a neural network classifier known as a discriminator. 15:54.620 --> 16:16.060 So generator is a kind of learning model that starts with random noise, tries to generate some patterns, sends it over to discriminator, hey, is this a correct image or not, and then discriminator says that no, it's not correct, and based on that it filters out the data. 16:16.060 --> 16:31.180 So you might have seen like deep fakes and voice recording that are very similar to like President Obama, they superimpose his face on some video, so those are generated by generative models. 16:31.180 --> 16:43.260 So with each iteration, the generator brings in new data, it learns what parameter variation was rejected, and it improves itself. 16:46.280 --> 16:57.020 So generator and discriminator, like these two machine learning models, are trained together, so they are locked in a zero-sum game. 16:57.020 --> 17:16.300 Basically one player loses and another player wins, like some of the two-player card games, they are zero-sum games, like one person generates a positive payoff based on his moves, another person loses and he gets a negative payoff. 17:16.460 --> 17:32.680 So with each iteration, the discriminator basically improves its capability of detecting these malicious attacks, and generators are becoming more and more smart with each iteration. 17:32.680 --> 18:00.520 So how we penalize these models is, suppose I send an attack payload to discriminator and it fails to detect it as a true attack, and it executes as a true attack, the discriminator's model is penalized for making that error in judgment, and then it is forced to update its model parameter and improve in the next round. 18:00.520 --> 18:25.060 Whereas if the generator sends a payload to the discriminator and it is detected and blocked by firewall, the generator's model is penalized, so they keep on competing with each other in this adversarial game until there is no more improvement and both are at a stable position that no more learning can be done. 18:28.510 --> 18:33.670 So starting with random noise, does it make sense in case of text data? 18:33.670 --> 18:40.390 We are not trying to generate a fake image of Mona Lisa, so text is different. 18:41.230 --> 18:51.660 So we use this variation called conditional GAN, where we provide labels not only to the discriminator, but also generator. 18:52.090 --> 19:07.850 So generator also starts with some basic information that helps it train faster, helps in the convergence, and basically improves the process of generating these new payloads. 19:11.480 --> 19:19.530 So in summary, we provide the attack samples to the generator. 19:19.890 --> 19:25.430 We bombard the web application firewall with these payloads. 19:25.430 --> 19:29.110 Then we learn, did it lead to an error? 19:29.110 --> 19:30.640 Did it lead to a warning? 19:30.930 --> 19:33.610 Did the payload execute successfully? 19:33.770 --> 19:41.790 And the objective is to generate payloads that lead to a positive execution. 19:41.790 --> 19:52.950 So if it's a reflected XSS, it should lead to a pop-up message and it should bypass the web application firewall. 19:52.950 --> 20:00.490 So as long as those two criteria are satisfied, basically the generator receives the reward. 20:00.970 --> 20:05.070 And at the end of it, we update the model of generator. 20:05.070 --> 20:12.010 Like if it is generating a set of positive payloads, well and good. 20:12.930 --> 20:17.170 Basically, we can update the attack signature database. 20:17.170 --> 20:22.450 So essentially we are improving our web application firewall over a period of time. 20:24.050 --> 20:26.350 But is it that easy? 20:26.350 --> 20:31.170 Like can I pass the same input to a GAN model? 20:31.170 --> 20:35.810 Like image data set and text data sets are fundamentally different. 20:35.810 --> 20:39.350 So that is where the main challenge of this problem was. 20:39.690 --> 20:49.070 Because in image, you are dealing with these RGB pixels and you are projecting them to your embedding layer. 20:49.350 --> 20:51.390 But text has semantics. 20:51.390 --> 20:52.530 It has structure. 20:52.530 --> 20:54.910 It has meaning. 20:54.910 --> 20:56.730 There is a vocabulary associated. 20:56.730 --> 21:00.950 So it is, in fact, not that simple. 21:01.110 --> 21:16.650 So when we are talking about text or character data, we need to have embedded layers where we project our text bits into one hot encoding. 21:16.650 --> 21:24.590 So we have a sparse matrix of zeros and one where we project these text characters. 21:24.830 --> 21:26.430 But that leads to problems. 21:26.430 --> 21:29.470 Like two problems. 21:29.470 --> 21:33.890 One is model parameters don't update beyond a point. 21:33.890 --> 21:36.970 And second is known as vanishing gradient. 21:36.970 --> 21:44.730 So essentially in text data, there is a model collapse. 21:44.730 --> 21:50.930 Like beyond a point, it is very hard to distinguish real payloads from fake payloads. 21:50.930 --> 21:58.490 And discriminator is basically not telling any new information to a generator to help it improve. 21:58.490 --> 22:10.990 So we use a variation known as Wassertime distance where we use like sum of probability distribution to deal with this. 22:10.990 --> 22:19.510 Second issue is that the XSS payload itself, it's not essentially a text with vocabulary. 22:19.510 --> 22:32.090 So it's a combination of these characters like your opening bracket, then you have some pattern like alert, null characters, then you have closing brackets. 22:32.090 --> 22:36.990 So it's a state action representation. 22:36.990 --> 22:49.810 So we need to have some kind of sequential model that understands that this particular sequence of characters, it worked well for my model. 22:49.970 --> 22:59.530 So what we do to deal with this variation is we use the generator as a reinforcement learning model. 22:59.530 --> 23:06.450 What it means is that we will have the individual characters as states. 23:06.450 --> 23:10.950 And next character will be based on like the action pair. 23:10.950 --> 23:18.010 So a policy in this reinforcement learning model will be like the correct sequence. 23:18.010 --> 23:28.690 So if you have a pair of correct sequence that work well, that are like bypassing your web application firewall, that's a positive policy. 23:28.690 --> 23:35.150 And using this reinforcement learning model, it basically improves over time. 23:38.360 --> 23:44.620 So as we can see that if there is a positive signature match, we update our generator model. 23:44.620 --> 23:52.420 And if there is a payload that is not able to bypass our WAF, we basically reject that. 23:52.640 --> 23:56.260 So we did something known as semantic tokenization. 23:56.260 --> 23:59.160 Like I used these payloads as input. 23:59.160 --> 24:12.060 I used burp intruder to basically check what is the HTML response of this data that I'm throwing at the web application firewall. 24:12.060 --> 24:13.980 Is it triggering any alerts? 24:13.980 --> 24:15.720 Is it leading to true warnings? 24:15.720 --> 24:17.520 Is it leading to errors? 24:17.660 --> 24:22.040 And we use that to generate our initial labels. 24:22.040 --> 24:29.360 Like this is the subset from the XSS data set that worked. 24:29.360 --> 24:36.500 Now let me use it to train my generator and generate variations of these payloads that will work as well. 24:38.920 --> 24:51.580 So in semantic tokenization, we basically understand the labels with each payload, like what leads to a node, what leads to a warning, what leads to an error. 24:51.620 --> 24:59.400 And we pass that to the discriminator and we understand the response it created. 24:59.400 --> 25:06.920 So with each iteration, we learn like our policy and model parameter updates. 25:06.920 --> 25:15.560 And when there is no update happening in future, basically we are at a stable position. 25:15.560 --> 25:20.300 And these positive payloads then we can add to our WAF signature database. 25:20.300 --> 25:23.460 So that's how we improve. 25:24.820 --> 25:31.240 So we keep track of the logs that are generated at the modSecurityWAF level. 25:31.240 --> 25:40.620 Like for example, this one was triggered by one of the PHP payloads that was denied. 25:40.620 --> 25:44.700 So basically generator will be penalized in these cases. 25:44.700 --> 25:49.660 Like if it generates the variation that are blocked, it will be penalized. 25:51.620 --> 26:01.580 So some more things that we need to do is we have to semantically understand how the modSecurityWAF behaves. 26:01.640 --> 26:06.880 So attacks are verified over the vulnerable applications protected by WAF. 26:06.880 --> 26:16.620 But modSecurity also uses this PhantomJS library with the WebKit and JavaScript API. 26:16.880 --> 26:35.160 So basically we are generating attack payloads, throwing them on the modSecurityWAF that is protecting our NGINX application, learning the behavior, and then updating our generator with each iteration. 26:35.420 --> 26:43.130 And at the end of it, we can use these new signatures to basically update our web application firewall. 26:47.580 --> 27:01.520 So the module of the WebKit that modSecurity uses, it uses like a variation of partial non-alphanumeric characters and it de-obfuscates the payload. 27:01.520 --> 27:14.500 So those were the cases where we needed to understand like what sort of semantic variations do we need to put in place to bypass like these particular set of rule sets on this WAF. 27:14.500 --> 27:23.160 So as you can see that if we provide some sort of obfuscated characters, the WAF is not able to detect that. 27:23.160 --> 27:27.540 So we need to understand that semantically and encode this in our model. 27:27.540 --> 27:31.820 So that is another challenge that we faced. 27:31.820 --> 27:37.040 Let me do the demo first and then we can do the results. 27:37.040 --> 27:38.380 We can look at the results. 27:41.920 --> 27:51.200 So the pre-training part, it takes hours and hours, so that will not be showcased in the demo, but I will update those models. 27:51.420 --> 27:58.300 So we know that this is a reflected XSS, that this will be present in this vulnerable web application. 28:01.520 --> 28:08.320 So now we know that what is the particular input parameter we have to bombard. 28:08.320 --> 28:15.680 So we compile this list in the intruder, and we hammer the application with this. 28:15.680 --> 28:29.260 We have the capability of reading the response that we got, so we can have the annotation like, did it lead to error, warning, 403, 200, what sort of response did it cause. 28:32.020 --> 28:39.800 So we collect all this data and we use this data to update the labels of our training data. 28:40.620 --> 28:53.440 And we also check, like I use Docker to host some of these services, so we also check the logs that are generated in the mod security web application firewall while we are doing that. 28:53.440 --> 28:58.960 So we keep track of the response, it emitted or did it trigger WAF or not. 28:58.960 --> 29:05.780 So both this information is used to update our labels of a successful attack. 29:05.780 --> 29:13.960 So as you can see here, it matched the signature database and it triggered an alert. 29:18.220 --> 29:37.880 And in the next step, what we did do is compile a list of all of these and basically I use Selenium to automatically bombard these and understand which of these are generating positive alerts. 29:38.180 --> 29:40.320 That will come up in a second. 29:45.950 --> 29:56.650 Yeah, so for automated testing, basically you can also use Selenium web driver to basically throw these payloads at a web application. 29:56.650 --> 30:04.530 You just need to understand the semantics of the input and how it is composed. 30:09.330 --> 30:15.990 So this is the automated stuff that's happening, like nothing. 30:16.630 --> 30:24.830 So it will tell you which of these payloads are triggering alerts, like it's currently bombarding it with payloads. 30:28.540 --> 30:38.620 So no alerts triggered and then it starts popping up some positive payloads that trigger alert and that's how we update our signature database. 30:38.620 --> 30:45.660 These are the set of payloads that bypass WAF and triggered alert on the website. 30:49.860 --> 31:02.380 So it's pretty easy to check which of these triggered alert, like there is a webhook in Selenium that you can use to sort of understand the output. 31:09.160 --> 31:25.490 So looking at some of the results that we understood from the model is basically there are two losses of discriminator, like DLoss1 and DLoss2. 31:25.550 --> 31:32.230 The DLoss1 tells us, is discriminator able to detect real attacks as real? 31:32.230 --> 31:36.970 And DLoss2 tells us, is it detecting fake samples as fake? 31:37.130 --> 31:46.670 So the goal as we update the signature database is to make sure that the performance of discriminator improves with this new data. 31:46.930 --> 31:58.870 So you can see that the loss of discriminator is lowest around 250 epochs, so that's how we improve the performance of our discriminator. 32:00.010 --> 32:05.310 And yeah, it converges to a stable value around 250 epochs. 32:07.090 --> 32:21.730 So we also noticed that conditional GAN, where we provide the labels to the generator, it is performing better than like a plain vanilla GAN where we don't provide any information to the generator. 32:21.730 --> 32:47.610 So first of all, like CGAN or conditional GAN converges faster, but when we used different variations of the payload, in the run 4, the vanilla GAN was not able to understand most of these attacks and the performance was very poor on a particular batch of this input data. 32:47.610 --> 32:53.110 Whereas the conditional GAN, it had sort of a stable performance. 32:53.110 --> 33:01.670 So these are the percentage of payloads that were from the data set generated that were able to bypass the WAF. 33:01.670 --> 33:13.070 So conditional GAN performs consistently better at bypassing the web application firewall if we see from the generalizability perspective. 33:14.230 --> 33:24.350 So in conclusion, we utilized the conditional GAN to generate attack payloads that were able to bypass the web application firewall. 33:24.510 --> 33:38.930 The generated samples found the variation of payloads that were able to circumvent the rules of Privilege Level 1 and 2 of ModSecurityWAF. 33:39.690 --> 33:47.430 And the sample generated can be used to improve the signatures of our web application firewall database. 33:48.110 --> 34:01.410 And this can help us deal with some unknown variation of attacks, like APT attackers occasionally come with variations of like the normal attacks that are hard to detect. 34:01.410 --> 34:05.030 So this can help us in that coverage as well. 34:06.670 --> 34:21.070 So next steps for this research, we have a paper that is coming up, which has all sorts of results or architecture and a lot more information, as well as some results on AWS WAF. 34:22.530 --> 34:30.230 We are trying to understand how it behaves on other commercial firewalls like CloudFlare. 34:30.330 --> 34:36.290 And yeah, so as Chris mentioned in the previous talk, like your vendors need to research you. 34:36.290 --> 34:42.530 So CloudFlare, in fact, reach out to me and ask like, can you also test it on our WAF? 34:42.530 --> 34:49.690 So that is like a good thing from that vendor that they were tracking that I'm presenting on this topic and they reached out to me. 34:49.690 --> 34:53.850 So I'll be looking at CloudFlare as well, like how it behaves. 34:54.350 --> 35:10.630 And again, we have kind of moved to the era of large language models like BART and GPT models, where we want to use these enhanced models to basically generate payloads. 35:10.630 --> 35:12.730 So it's not very simple. 35:12.730 --> 35:23.910 It will take some intuitive engineering, like I tried asking chat GPT, like, can you send me some payloads that can help me bypass the web application firewall? 35:23.910 --> 35:32.450 So it will reject it outright saying that you are trying to do something malicious, but then people try to send a variation of query. 35:32.450 --> 35:37.570 Can you send me payloads which can help me protect my web application firewall? 35:37.570 --> 35:47.050 So you can project the query that way, but the payloads that I got from output were completely garbage, so they don't have any semantic as of now. 35:47.050 --> 35:55.570 So it will take some intuition and engineering to understand, like, how we can use large language models to generate these variations. 35:56.150 --> 36:02.270 I would like to acknowledge my colleague, Krith Shekhar Jha, who is a PhD student. 36:02.450 --> 36:08.670 He helped me with a lot of these experiments and is also a co-author of mine on the paper. 36:08.670 --> 36:15.890 So with that, I will conclude the talk, and I'll be open to any questions.