[00:08.690 --> 00:15.290]  So my name is Jason Haddix, and this is Gunnar, and we are hackers.
[00:16.010 --> 00:22.930]  We created a data, I guess, data repository and tooling called...
[00:23.510 --> 00:25.310]  You didn't want that picture in the slides?
[00:25.310 --> 00:26.450]  No, no, no, it's a good picture.
[00:26.450 --> 00:27.790]  That's a great picture, dude.
[00:27.790 --> 00:28.930]  I appreciate it.
[00:29.530 --> 00:33.890]  And so we created some tooling and some data repository stuff
[00:33.890 --> 00:37.150]  for people primarily who are doing application security testing.
[00:38.550 --> 00:41.050]  Okay. So a little bit of history.
[00:41.290 --> 00:44.950]  So about... oh, I don't know.
[00:45.750 --> 00:50.930]  2016, I spoke at DEF CON on the main stage on this topic.
[00:50.950 --> 00:52.890]  And the talk was called HUNT.
[00:52.890 --> 00:55.930]  It was called HUNT Data-Driven Web Hacking and Manual Testing.
[00:56.510 --> 01:00.070]  Now, HUNT, these fine people at the bottom
[01:00.070 --> 01:02.410]  were the ones who helped me put that talk together.
[01:02.430 --> 01:05.350]  What we're doing today is enhancing it a whole bunch.
[01:05.350 --> 01:09.670]  So let me talk about the problem space we're working in right now.
[01:09.750 --> 01:13.990]  So right now, a seasoned hacker, application security hacker,
[01:13.990 --> 01:18.130]  can kind of look at a website, and for some reason,
[01:18.130 --> 01:20.250]  they've developed, like, a spidey sense.
[01:20.250 --> 01:22.710]  If you've been doing this for 10 years or 15 years,
[01:22.710 --> 01:24.050]  you look at a website and you're like,
[01:24.050 --> 01:26.390]  oh, yeah, that's probably vulnerable to X, right?
[01:26.470 --> 01:30.050]  And that's just, like, something that people talk about all the time.
[01:30.050 --> 01:31.450]  It's like, oh, I don't know really how I hack.
[01:31.450 --> 01:34.510]  I just kind of, you know, I look at the site and I do this thing.
[01:34.510 --> 01:37.890]  That is not helpful for new people getting into application security.
[01:37.890 --> 01:41.750]  Just saying, oh, you'll develop this skill set in 15 years, right?
[01:42.330 --> 01:45.050]  And so a lot of them call it, like, their spidey sense.
[01:45.150 --> 01:49.590]  But really, it's experience and context that are driving these decisions
[01:49.590 --> 01:52.770]  when you look at an application and know where to hack it
[01:52.770 --> 01:55.630]  and how to hack that section of the application.
[01:56.550 --> 01:59.730]  So how do we shortcut experience and context?
[01:59.730 --> 02:01.410]  And in general, you should never do this, right?
[02:01.410 --> 02:04.750]  But in our cases, we're going to try to help new people get some ideas
[02:04.750 --> 02:09.290]  about where to hack certain things in the web application assessment arena.
[02:10.470 --> 02:12.730]  Okay. So this is a URL.
[02:12.730 --> 02:14.510]  Have any of you seen a URL before?
[02:15.710 --> 02:17.750]  Woo! There we go. There we go.
[02:17.750 --> 02:19.750]  All right. So here is a URL.
[02:19.830 --> 02:22.950]  And at the end here, we have ID.
[02:23.550 --> 02:26.150]  And so the whole structure is, like, you have your protocol,
[02:26.150 --> 02:28.070]  and then you have your subdomain and your domain,
[02:28.070 --> 02:29.970]  and then you have your file or resource name,
[02:29.970 --> 02:34.450]  and then you pass that, usually a parameter and a parameter value.
[02:34.890 --> 02:39.570]  Now, our talk and research is all about these things.
[02:39.570 --> 02:41.050]  Parameters, right here.
[02:41.050 --> 02:45.070]  Because they can be used in a sense and fashion
[02:45.070 --> 02:46.970]  to give you the context we're looking for
[02:46.970 --> 02:49.530]  and guide you into what to test.
[02:50.830 --> 02:55.350]  Okay. So in 2016, when we did this research,
[02:56.830 --> 03:00.230]  I mean, I won't call it stole, because I worked for them at the time,
[03:00.230 --> 03:02.670]  but I didn't really get permission, but, you know, whatever.
[03:03.710 --> 03:07.730]  Basically, I went over the dataset of every bug bounty on BugCrowd.
[03:07.730 --> 03:09.210]  Anybody know who BugCrowd is?
[03:09.790 --> 03:10.950]  Yeah. Okay. Cool.
[03:10.950 --> 03:13.590]  So BugCrowd is a bug bounty platform.
[03:13.670 --> 03:16.670]  And at the time, it had over 600 bounties, internal and external.
[03:16.870 --> 03:18.970]  And what I did is I had the data engineers chop off the domain,
[03:18.970 --> 03:21.210]  so I could not identify who the companies were.
[03:21.210 --> 03:22.970]  But they gave me the URL data.
[03:22.970 --> 03:26.970]  And then for every bug that existed on a URL,
[03:28.190 --> 03:36.790]  I parsed out the parameters for that bug class and the bug URL.
[03:36.870 --> 03:38.850]  So that was about two programs...
[03:38.850 --> 03:43.370]  or every program had, on average, about two targets per bug bounty.
[03:43.630 --> 03:47.750]  So targets are like www.defcon.org, forums.defcon.org,
[03:47.750 --> 03:50.090]  media.defcon.org, etc., etc., etc.
[03:50.870 --> 03:57.590]  So each target had about 15 average parameters per application.
[03:57.990 --> 03:59.850]  And many were many, many more.
[03:59.850 --> 04:03.950]  So approximately, we saw about 18,000 parameters
[04:03.950 --> 04:06.210]  with vulnerabilities on them in this dataset.
[04:06.210 --> 04:09.050]  Which is probably the biggest dataset, I think,
[04:09.050 --> 04:12.490]  that's ever been parsed for vulnerability research in web before.
[04:14.370 --> 04:16.290]  Or 18,000. Sorry.
[04:16.290 --> 04:19.390]  So in the 18,000 parameters that we found,
[04:19.390 --> 04:21.330]  that were anonymized,
[04:21.330 --> 04:24.390]  we reduced them to make sure that we got the vulns
[04:24.390 --> 04:26.190]  that we wanted to investigate, the vulnerabilities.
[04:26.190 --> 04:29.330]  So things like cross-site scripting, SQL injection, command injection.
[04:29.330 --> 04:31.970]  Things that you find in AppSec testing, right?
[04:31.970 --> 04:34.630]  So we reduced it to ones we wanted to do research on.
[04:34.630 --> 04:37.710]  And then we also reduced it to only P1s and P2s
[04:37.710 --> 04:40.790]  or high-severity critical vulns.
[04:41.130 --> 04:45.250]  Then we sorted by recurring instances of when that took place.
[04:45.250 --> 04:51.090]  And then we included the 5 to 10 or 15, in some cases, we expanded to,
[04:51.090 --> 04:54.470]  the reoccurring instances per vulnerability category.
[04:54.910 --> 04:58.990]  And then we looked at those and then manually reviewed them
[04:58.990 --> 05:02.830]  and were like, OK, well, I know with my 15 years of context
[05:02.830 --> 05:04.450]  and that team of five people,
[05:04.450 --> 05:07.050]  there's some permutations of this thing that I always want to look at,
[05:07.050 --> 05:09.390]  so I'll add those just off the top of my head.
[05:09.390 --> 05:11.230]  And then we also looked at a whole bunch of repos
[05:11.230 --> 05:13.010]  that include fuzzing data.
[05:13.010 --> 05:16.230]  Which was like PentestDB, FuzzDB, and SecLists,
[05:16.230 --> 05:19.390]  which I run SecLists with Daniel Meisler.
[05:19.430 --> 05:21.610]  So, we made these lists.
[05:21.610 --> 05:26.190]  And what these lists were, were just lists of parameters in websites
[05:26.190 --> 05:29.130]  that were vulnerable ones to a certain class of vulnerability.
[05:29.190 --> 05:32.330]  But because we had seen certain ones so many times,
[05:32.330 --> 05:34.070]  we started to tell our friends, like,
[05:34.070 --> 05:36.790]  every time you see a next parameter,
[05:36.790 --> 05:42.130]  probably you should check it for redirects and LFIs and RFIs
[05:42.130 --> 05:43.490]  and stuff like that.
[05:46.220 --> 05:49.660]  So, this talk, SUS Params,
[05:49.660 --> 05:53.180]  is about releasing the tooling to do that yourself,
[05:53.180 --> 05:56.120]  and jacking up the lists to the nth degree.
[05:56.420 --> 05:58.300]  And analyzing that data.
[05:58.540 --> 06:03.140]  So, SUS Params, obviously SUS and parameters.
[06:03.960 --> 06:06.200]  We did a whole bunch of work with AI,
[06:06.200 --> 06:07.460]  which Gunnar's going to talk about,
[06:07.460 --> 06:09.040]  to actually do the data parsing.
[06:09.040 --> 06:11.580]  And people say that data parsing with AI is impossible.
[06:11.580 --> 06:12.700]  I'm here to tell you it's not.
[06:12.700 --> 06:15.000]  We used it very effectively in this talk,
[06:15.000 --> 06:16.420]  or in this set of tools.
[06:16.440 --> 06:17.800]  We're going to release the lists,
[06:17.800 --> 06:20.000]  and then we're also going to work with a friend of ours,
[06:20.000 --> 06:21.840]  xmlhacker, who will give you a way to
[06:21.840 --> 06:25.180]  implement this in your day-to-day life using his tool.
[06:25.180 --> 06:28.120]  So, I'm going to pass it over to Gunnar as we go.
[06:30.060 --> 06:30.860]  Alright.
[06:31.760 --> 06:35.380]  So, right now, what we started with on this release
[06:35.380 --> 06:37.860]  of the repo and the data,
[06:37.860 --> 06:39.980]  the next target we went for was HackerOne.
[06:39.980 --> 06:43.580]  We don't have any internal digs in HackerOne, right?
[06:43.580 --> 06:45.200]  So, it's all the public data.
[06:45.280 --> 06:47.660]  Anyone who knows HackerOne knows that there's a very long list
[06:47.660 --> 06:50.920]  called the hacktivity that is publicly disclosed data.
[06:52.580 --> 06:53.460]  Perfect.
[06:53.460 --> 06:55.080]  So, we used AI to parse this data,
[06:55.080 --> 06:56.040]  which I'll talk about later.
[06:56.040 --> 06:58.120]  But this is what hacktivity looks like.
[06:58.640 --> 07:02.200]  So, this would be kind of annoying to scrape,
[07:02.200 --> 07:04.140]  and web scraping nowadays with dynamic websites
[07:04.140 --> 07:05.740]  and all that stuff gets very annoying.
[07:06.200 --> 07:09.080]  So, huge shout-out to HackerOne
[07:09.080 --> 07:12.420]  for making this whole list populated by a GraphQL query
[07:12.420 --> 07:13.700]  in the backend.
[07:13.700 --> 07:15.960]  So, you can actually just script the GraphQL query,
[07:15.960 --> 07:17.920]  it's a paginated GraphQL query,
[07:17.920 --> 07:22.280]  and you can get all, I think, 15,000 publicly disclosed reports
[07:22.280 --> 07:26.520]  on every public program on HackerOne.
[07:26.520 --> 07:28.440]  Then, the next thing that's nice about it
[07:28.440 --> 07:30.580]  is normally when you click on one of these,
[07:30.580 --> 07:32.980]  it brings you to a HackerOne report,
[07:32.980 --> 07:36.240]  which is now also stuff that would be very annoying to scrape
[07:36.240 --> 07:37.700]  for text.
[07:37.700 --> 07:41.920]  Well, again, all of the HackerOne reports come in JSON format.
[07:41.960 --> 07:44.160]  If you take the URL of the HackerOne report
[07:44.160 --> 07:46.320]  and tack on .json at the end,
[07:46.320 --> 07:48.280]  it shows up in JSON format,
[07:48.280 --> 07:50.200]  because that's how it gets pulled out of their backend.
[07:50.320 --> 07:54.860]  So, what we did is I went and grabbed all of the report IDs
[07:54.860 --> 07:56.620]  out of the GraphQL endpoint,
[07:56.620 --> 07:59.940]  and then saved all of the JSON format of all those reports,
[07:59.940 --> 08:01.820]  and we ended up with right around 15,000.
[08:01.820 --> 08:03.700]  Again, it's in the repo if you want to see it.
[08:03.700 --> 08:05.960]  I saved all the data from every step,
[08:05.960 --> 08:09.200]  so if you want to reproduce it for some other data set,
[08:09.200 --> 08:12.560]  you can follow the steps and make sure it looks very similar.
[08:13.640 --> 08:16.540]  So, the AI parsing discussion.
[08:16.640 --> 08:19.000]  So, we tried to do a whole bunch of different things
[08:19.520 --> 08:21.880]  with AI to parse this data.
[08:21.980 --> 08:24.280]  As a lot of people know who have messed with AI,
[08:24.280 --> 08:27.780]  I'm not an AI expert myself,
[08:27.780 --> 08:31.360]  but AI output is normally never very uniform.
[08:31.360 --> 08:33.540]  You normally can't get AI to just give you JSON output
[08:33.540 --> 08:34.460]  and only JSON output.
[08:34.460 --> 08:36.980]  You normally can't get it to give you really clean output
[08:36.980 --> 08:39.540]  that you can count on to be the exact same every single time.
[08:39.740 --> 08:42.610]  So, just telling some of the AI,
[08:42.980 --> 08:45.660]  hey, I'm going to let you read this text that I pulled,
[08:45.660 --> 08:47.480]  and then you give me the parameter.
[08:47.860 --> 08:49.980]  Sadly, it doesn't exactly work that way yet.
[08:50.340 --> 08:52.140]  Maybe soon, but not quite yet.
[08:52.780 --> 08:54.600]  So, oh, sorry, misclicked.
[08:54.600 --> 08:59.060]  So, what we did instead was GPT 3.5 Turbo,
[08:59.060 --> 09:01.340]  you can use with the OpenAI API.
[09:01.340 --> 09:04.060]  I messed with that and messed with temperature
[09:04.060 --> 09:05.220]  and all that kind of other stuff,
[09:05.220 --> 09:07.680]  and it works sometimes, doesn't work all the time.
[09:08.220 --> 09:10.880]  You pay the extra 20 bucks and use GPT 4,
[09:10.880 --> 09:14.560]  and it's much, much better at understanding HTTP requests,
[09:14.560 --> 09:19.080]  HTTP URLs, code, JSON, whatever it may be.
[09:19.220 --> 09:24.180]  And we were able to use AI to basically take the mountain of text
[09:24.180 --> 09:27.420]  that you would get from 15,000 bug bounty reports
[09:27.420 --> 09:30.160]  and reduce it down to basically one sentence
[09:30.160 --> 09:33.640]  that either had parameters in quotes,
[09:33.640 --> 09:35.240]  parameters in a bulleted list,
[09:35.780 --> 09:40.120]  or basically something that makes it basically rejectable,
[09:40.120 --> 09:41.540]  which is going to be the next step.
[09:41.840 --> 09:46.260]  So it takes this massive thing of text, reads it,
[09:46.260 --> 09:47.380]  and then will tell you, like,
[09:47.380 --> 09:49.760]  this report has parameters that are vulnerable or not,
[09:49.760 --> 09:51.100]  this is what they are.
[09:51.220 --> 09:53.060]  And since the reports come from HackerOne,
[09:53.060 --> 09:57.560]  already broken into cross-site scripting, command injection,
[09:57.560 --> 09:59.020]  SQL injection, whatever it may be,
[09:59.020 --> 10:00.980]  you don't have to use AI to do that.
[10:01.140 --> 10:02.800]  You just can already split it up as it goes
[10:02.800 --> 10:04.840]  and then ingest it into AI that way.
[10:05.420 --> 10:07.300]  And at the end of all the slides,
[10:07.300 --> 10:08.460]  I'm going to go through the repo,
[10:08.460 --> 10:10.920]  and I'll show you exactly what I did.
[10:11.640 --> 10:14.560]  We used HackerOne, like Jason said.
[10:14.560 --> 10:16.540]  You can use what we did,
[10:16.540 --> 10:18.340]  and we're releasing the scripts,
[10:18.340 --> 10:19.900]  of very few changes.
[10:19.900 --> 10:21.640]  If you have, within your company,
[10:21.640 --> 10:24.220]  your own internal data set of either incidents you've had,
[10:24.220 --> 10:25.660]  or you run your own bug bounty program,
[10:25.660 --> 10:26.780]  and you have your own internal reports,
[10:26.780 --> 10:27.840]  and you want to know,
[10:27.840 --> 10:29.900]  hey, can I make this data set just for my stuff?
[10:29.920 --> 10:31.580]  Absolutely. That's the point of the project.
[10:31.580 --> 10:34.560]  We're releasing our set, our new set of lists,
[10:34.560 --> 10:36.960]  based on Jason's list, and this new data,
[10:36.960 --> 10:40.500]  for bug bounty hunters, solopreneurs, whatever it may be.
[10:40.500 --> 10:42.360]  But we're also releasing the scripts,
[10:42.360 --> 10:44.140]  and we're going to explain how we did it.
[10:44.160 --> 10:46.380]  That way, if you have something very similar
[10:46.380 --> 10:47.300]  that you would like to generate
[10:47.300 --> 10:49.300]  your own honed list of SUS params for,
[10:49.300 --> 10:50.540]  you can also do it.
[10:51.480 --> 10:53.540]  So, as Jason mentioned,
[10:53.540 --> 10:56.620]  we have a buddy who goes by xnailhacker on Twitter,
[10:56.620 --> 10:58.700]  who makes a burp extension called GAP.
[10:59.080 --> 11:01.400]  GAP originally would collect,
[11:01.400 --> 11:03.260]  as stuff goes through your proxy,
[11:03.260 --> 11:05.040]  it will collect links,
[11:05.040 --> 11:06.980]  and it will collect parameters, yes?
[11:06.980 --> 11:07.380]  Yep.
[11:07.380 --> 11:09.680]  From anything that flows through your proxy.
[11:09.680 --> 11:12.200]  It does it automatically, and then there's a GAP tab
[11:12.200 --> 11:14.380]  in your burp suite that you can click on
[11:14.380 --> 11:15.740]  and read through them all and filter them all
[11:15.740 --> 11:17.120]  and highlight and all that stuff.
[11:17.120 --> 11:18.900]  He was nice enough to work with us
[11:18.900 --> 11:23.200]  and implement the list that we created into GAP.
[11:23.200 --> 11:25.480]  So the new version that's releasing today
[11:25.480 --> 11:27.640]  with GAP will now do,
[11:28.120 --> 11:30.160]  is not only will it collect parameters,
[11:30.160 --> 11:33.240]  but if you check a little box in the config,
[11:33.240 --> 11:34.800]  it will create burp issues,
[11:34.800 --> 11:36.640]  just little burp issues for you,
[11:36.640 --> 11:40.720]  for all your parameters that show up in the SUS params list.
[11:41.100 --> 11:43.920]  So if you have a parameter that shows up in a URL,
[11:43.920 --> 11:46.820]  it will automatically put it in the list for you.
[11:46.820 --> 11:49.040]  You can go to GAP, pull down the list of burp issues,
[11:49.040 --> 11:50.260]  and you can see all your parameters
[11:50.260 --> 11:52.320]  that show up in SUS params.
[11:52.400 --> 11:54.740]  So we'll show that to you in a screenshot.
[11:54.740 --> 11:57.440]  Yep. So this is part of what it looks like.
[11:57.440 --> 11:59.440]  Is there two? Oh, no. Okay. So there's just the one.
[11:59.440 --> 12:01.160]  So this is kind of what it looks like.
[12:01.160 --> 12:03.220]  So when it, you can basically pull the list
[12:03.220 --> 12:05.740]  and it'll say like, okay, so it found an ID parameter.
[12:05.900 --> 12:08.900]  Like people in the past have tested that for XSS,
[12:08.900 --> 12:11.920]  IDOR, SQLI, SSTI, whatever it may be.
[12:13.180 --> 12:14.080]  That kind of thing.
[12:14.180 --> 12:15.980]  You can turn it off and on,
[12:15.980 --> 12:18.580]  but this is basically what Jason was talking about,
[12:18.580 --> 12:20.280]  where if you're maybe newer to the scene,
[12:20.280 --> 12:22.040]  or you maybe are doing a fast, you know,
[12:22.040 --> 12:24.660]  you're doing a fast engagement or something like that,
[12:24.660 --> 12:26.440]  this is something to really help you
[12:26.440 --> 12:28.220]  kind of narrow it down and maybe get to stuff
[12:28.220 --> 12:29.960]  that might be vulnerable a little quicker.
[12:31.960 --> 12:32.260]  Okay.
[12:32.620 --> 12:34.380]  Did I miss anything before the demo?
[12:34.380 --> 12:35.580]  Yeah. Cool.
[12:36.800 --> 12:38.500]  Where's the escape key? There we go.
[12:39.020 --> 12:41.740]  Okay. So I'll leave it up at the end,
[12:41.740 --> 12:45.220]  but this is the repo for the project.
[12:46.600 --> 12:48.820]  I was going to say, can we zoom in?
[12:48.820 --> 12:51.160]  I wasn't allowed to use my Mac, so...
[12:52.020 --> 12:53.780]  So this is a repo for the project.
[12:54.340 --> 12:56.140]  Again, at the top, anyone who knows
[12:56.140 --> 12:57.900]  it is TomNoNone, correct, is GF?
[12:58.660 --> 12:59.460]  Yes.
[12:59.820 --> 13:01.560]  So there's a tool that some people use in automation
[13:01.560 --> 13:02.820]  just called GF.
[13:02.820 --> 13:05.180]  So there's GF patterns for the list at the top, too,
[13:05.180 --> 13:06.360]  if you use that tool.
[13:07.260 --> 13:10.900]  And then as we go down...
[13:10.900 --> 13:13.020]  So this query I saved.
[13:14.300 --> 13:16.500]  This was the original query that I pulled.
[13:16.500 --> 13:18.160]  This is your GraphQL query.
[13:18.180 --> 13:19.720]  So if you want to try and pull hacktivity
[13:19.980 --> 13:21.220]  a little bit different than I do,
[13:21.220 --> 13:23.840]  this is the GraphQL query that you start with.
[13:23.880 --> 13:25.100]  So that's just thrown into a text file
[13:25.100 --> 13:26.920]  just in case, you know, so you don't have to
[13:26.920 --> 13:28.080]  hunt it down.
[13:28.720 --> 13:29.980]  That kind of thing.
[13:31.460 --> 13:33.300]  After that, HackerOne and BugCrowd,
[13:33.300 --> 13:35.380]  we did try and pull BugCrowd data publicly.
[13:35.440 --> 13:38.420]  The CrowdStream is much smaller than hacktivity.
[13:38.420 --> 13:41.060]  So size of dataset-wise,
[13:41.060 --> 13:44.300]  the hunt dataset is vastly larger,
[13:44.300 --> 13:46.820]  obviously, than what is on the CrowdStream.
[13:47.020 --> 13:48.680]  Whereas HackerOne has much bigger...
[13:48.680 --> 13:50.200]  Hacktivity is just much bigger.
[13:50.200 --> 13:52.340]  So HackerOne.py, that file,
[13:52.340 --> 13:54.480]  will basically run...
[13:54.480 --> 13:55.740]  It does take a little bit,
[13:55.740 --> 13:57.280]  because it's paginated GraphQL queries,
[13:57.280 --> 13:59.200]  and there's quite a bit of JSON files.
[13:59.360 --> 14:01.660]  But what it will do is it will save them
[14:01.660 --> 14:02.880]  in this volns.json.
[14:02.880 --> 14:05.420]  Again, I'm not going to try and open it,
[14:05.420 --> 14:06.580]  because it's very big.
[14:06.740 --> 14:08.680]  But this is the last pull that we did,
[14:08.680 --> 14:09.580]  if you're interested.
[14:09.580 --> 14:12.300]  Otherwise, just running that HackerOne.py
[14:12.300 --> 14:16.160]  will make a new JSON for you,
[14:16.160 --> 14:17.020]  and you can start over.
[14:17.020 --> 14:18.900]  Because obviously, there's new stuff getting disclosed
[14:18.900 --> 14:20.860]  on Hacktivity all the time.
[14:21.500 --> 14:23.520]  If you want to take this list and not run it,
[14:23.520 --> 14:25.340]  and regenerate the whole list,
[14:25.760 --> 14:27.440]  it's very easy in the query
[14:27.440 --> 14:29.100]  that's in the HackerOne.py file
[14:29.100 --> 14:31.140]  to just add a filter
[14:31.140 --> 14:33.100]  that says from X date to X date.
[14:33.100 --> 14:34.900]  So you can be like, ah, just grab the last month.
[14:34.900 --> 14:37.300]  So you don't have to run through all 18,000
[14:37.300 --> 14:38.900]  that there will be at some point.
[14:41.860 --> 14:43.700]  Next, this POC.py,
[14:43.700 --> 14:44.740]  this is our proof of concept
[14:44.740 --> 14:46.180]  for how we did AI.
[14:46.180 --> 14:48.180]  Again, this will change
[14:48.720 --> 14:51.300]  depending on what type of text you're putting in,
[14:51.300 --> 14:53.140]  what type of data you're putting in, etc.
[14:53.200 --> 14:54.460]  What we did is
[14:54.460 --> 14:56.240]  I took the JSON file
[14:56.860 --> 14:58.800]  and there was just one of the strings
[14:58.800 --> 15:00.800]  was just all the text that was in the
[15:00.800 --> 15:02.560]  description of the report. So it was pretty easy.
[15:02.560 --> 15:04.480]  You just take the text and you put it into
[15:04.480 --> 15:06.660]  OpenAI. So how we did that
[15:06.660 --> 15:08.860]  is you just put in your OpenAI token at the top.
[15:08.860 --> 15:11.160]  Again, GPT-4 for this use case
[15:11.160 --> 15:12.940]  worked way better
[15:12.940 --> 15:14.480]  than GPT-3.
[15:14.760 --> 15:17.020]  Someone out there who is more of an AI
[15:17.020 --> 15:19.080]  wizard than I am,
[15:19.080 --> 15:21.540]  play with it. If you get it to work, let me know.
[15:22.420 --> 15:23.220]  But GPT-4
[15:23.220 --> 15:24.840]  does it basically out of the box.
[15:25.200 --> 15:26.940]  This GPT conversation
[15:26.940 --> 15:28.860]  function that you see at the top is
[15:28.860 --> 15:30.520]  basically the out-of-the-box
[15:31.320 --> 15:33.160]  conversation API call that they would
[15:33.160 --> 15:34.960]  recommend you make using
[15:34.960 --> 15:36.520]  GPT-4.
[15:36.960 --> 15:38.860]  So the one thing I did
[15:38.860 --> 15:40.980]  play with that you can either turn off and on
[15:40.980 --> 15:42.720]  if you want to change the script,
[15:42.720 --> 15:45.020]  is this is using the chat completion
[15:45.020 --> 15:46.900]  part of the API, which
[15:46.900 --> 15:49.120]  technically speaking will save a chat
[15:49.120 --> 15:50.760]  if you want it to.
[15:50.760 --> 15:52.840]  I clear the chat every time
[15:52.840 --> 15:55.160]  because I'm doing 18,000
[15:55.160 --> 15:56.980]  pieces of text. And if you
[15:56.980 --> 15:58.660]  keep trying to hand OpenAI
[15:58.660 --> 16:00.900]  an 18,000 thing
[16:00.900 --> 16:02.760]  long conversation, it will very
[16:02.760 --> 16:04.440]  quickly hit the token limit
[16:04.440 --> 16:06.160]  and you can't do it anymore.
[16:06.160 --> 16:08.620]  If you want to do something with
[16:08.620 --> 16:10.360]  conversations and keep some back-end
[16:10.360 --> 16:12.240]  data, you can probably play with that
[16:12.240 --> 16:14.360]  and actually use a tokenizer.
[16:14.580 --> 16:16.520]  I use TickToken. You can use
[16:16.620 --> 16:18.580]  a tokenizer and probably keep the back
[16:18.580 --> 16:20.560]  half of the conversation if you wanted to and maybe
[16:20.560 --> 16:22.480]  enrich the data even further. But
[16:22.480 --> 16:25.540]  what I did is I did one piece of text in a conversation,
[16:25.540 --> 16:26.600]  had OpenAI handle
[16:26.600 --> 16:28.380]  it, and started over with a fresh
[16:28.380 --> 16:30.120]  conversation. Just
[16:30.120 --> 16:31.840]  throwing that out there.
[16:32.460 --> 16:34.060]  Then all it's doing is, again, there's going to
[16:34.060 --> 16:35.980]  be a function for each one.
[16:35.980 --> 16:38.280]  I split it into functions
[16:38.280 --> 16:39.980]  that way if you want to just rip the function
[16:39.980 --> 16:42.040]  and do it for whatever your own thing is.
[16:42.120 --> 16:44.320]  This goes through the JSON and does all the XSS,
[16:44.320 --> 16:45.740]  all the SSRF, and we'll show
[16:45.740 --> 16:48.120]  the categories at the end.
[16:48.360 --> 16:50.080]  But this is basically just for every category
[16:50.080 --> 16:51.920]  there's a function for it.
[16:52.200 --> 16:52.500]  So...
[16:53.700 --> 16:54.500]  What?
[16:55.780 --> 16:56.840]  Okay, yeah.
[16:57.880 --> 16:59.940]  These are just... and all the functions
[16:59.940 --> 17:02.040]  are very similar. And then
[17:02.040 --> 17:04.020]  again, this is the conversation that will grab the
[17:04.020 --> 17:05.960]  conversation and print it out to
[17:05.960 --> 17:07.480]  the, as you can see here, to the
[17:08.020 --> 17:10.060]  volumetype-file in the test.
[17:10.120 --> 17:12.020]  So this test directory,
[17:12.020 --> 17:14.060]  this is what I was talking about,
[17:14.060 --> 17:15.540]  and this one I can open.
[17:15.620 --> 17:17.640]  This is not too big, hopefully.
[17:17.920 --> 17:20.140]  So this is what I was talking about. Data parsing
[17:20.140 --> 17:21.240]  in AI is
[17:22.000 --> 17:24.160]  interesting, and the output is never uniform.
[17:24.160 --> 17:25.860]  But, if you even look at these
[17:25.860 --> 17:27.740]  top couple things, you'll notice
[17:27.740 --> 17:30.220]  there's pretty obvious
[17:30.220 --> 17:32.060]  patterns in the output.
[17:32.060 --> 17:34.200]  First of all, 90%
[17:34.200 --> 17:35.960]  of those parameters that it gave me, it says
[17:35.960 --> 17:38.020]  the vulnerable parameter is, and then
[17:38.020 --> 17:39.960]  the vulnerable parameter is normally in some
[17:39.960 --> 17:41.140]  kind of quotation.
[17:42.200 --> 17:44.160]  So I can use regex
[17:44.160 --> 17:46.220]  on this dataset, which is now much
[17:46.220 --> 17:47.960]  smaller than the mountains
[17:47.960 --> 17:50.280]  of text that I just was putting into OpenAI.
[17:50.760 --> 17:52.060]  And it now also
[17:52.060 --> 17:53.940]  eliminated any report that it read
[17:53.940 --> 17:56.280]  and didn't find the parameter in the report.
[17:56.280 --> 17:58.060]  If it didn't find any parameter names in the report,
[17:58.060 --> 18:00.060]  those are now eliminated. So not only did I eliminate
[18:00.220 --> 18:02.100]  a huge amount of reports, but I
[18:02.100 --> 18:04.160]  lowered the amount of text to much lower
[18:04.160 --> 18:06.100]  where you can easily regex through this and it happens
[18:06.100 --> 18:08.220]  instantly. And we regex
[18:08.220 --> 18:10.140]  for any type of quotation,
[18:10.140 --> 18:12.240]  any type of list, anything that comes after
[18:12.400 --> 18:14.100]  a colon, a whole bunch of
[18:14.100 --> 18:15.820]  things. That
[18:16.440 --> 18:16.880]  is
[18:23.670 --> 18:24.110]  in...
[18:24.110 --> 18:25.190]  Oop, wrong one.
[18:26.030 --> 18:27.190]  This one, sorry.
[18:29.530 --> 18:30.990]  So these, again, this
[18:30.990 --> 18:32.910]  file here, when you look at it,
[18:32.910 --> 18:34.970]  and again, this is something you can play with too. If your data
[18:34.970 --> 18:36.950]  looks a little bit different, this one you can play with
[18:36.950 --> 18:38.770]  too. But this, I put little
[18:40.990 --> 18:42.950]  comments in the Python so you can know what
[18:42.950 --> 18:44.790]  each of the regex does. Thanks to
[18:44.790 --> 18:46.890]  ChadGPT for making my regexes, by the way, to
[18:46.890 --> 18:47.950]  look through its own data.
[18:50.230 --> 18:50.870]  So,
[18:50.870 --> 18:52.890]  like I said, there's stuff for quotes, there's stuff
[18:52.890 --> 18:54.870]  for single quotes, the dashes, there's
[18:54.870 --> 18:56.990]  stuff for REST API endpoints that lands
[18:56.990 --> 18:59.170]  inside two backslashes,
[18:59.170 --> 19:01.130]  colons, everything else. Like the list goes on and
[19:01.130 --> 19:03.730]  on. Then what it does is it takes,
[19:03.730 --> 19:05.270]  as it goes through the regexes,
[19:05.270 --> 19:07.070]  those regexes create their own list of data
[19:07.070 --> 19:09.670]  that they parsed out of the much smaller text,
[19:09.670 --> 19:11.150]  you combine all those files
[19:11.150 --> 19:13.090]  for each category, and you
[19:13.090 --> 19:15.170]  end up in here
[19:17.990 --> 19:18.950]  where now
[19:18.950 --> 19:20.990]  you actually have sorted, counted
[19:21.570 --> 19:22.050]  parameters
[19:23.650 --> 19:25.130]  from the output.
[19:25.530 --> 19:27.230]  And from here is how
[19:27.230 --> 19:29.210]  we then basically turned it
[19:29.210 --> 19:30.830]  into categories and lists.
[19:30.830 --> 19:33.370]  So, I think...
[19:34.210 --> 19:35.230]  Alright.
[19:35.230 --> 19:36.770]  So, let's look at
[19:36.770 --> 19:39.110]  the categories.
[19:43.450 --> 19:48.850]  Okay. So,
[19:48.850 --> 19:50.870]  we looked at file
[19:50.870 --> 19:52.870]  inclusion. What parameters
[19:52.870 --> 19:54.690]  when you do web application
[19:54.690 --> 19:56.850]  testing are most likely vulnerable to
[19:56.850 --> 19:59.590]  file inclusion of some sort.
[19:59.610 --> 20:01.070]  So, that is something
[20:01.070 --> 20:03.130]  like the parameter name lang, type,
[20:03.130 --> 20:05.210]  text domain, name, token, URL,
[20:05.210 --> 20:06.910]  path, etc., app, platform,
[20:06.910 --> 20:09.230]  resource type, file, default language,
[20:09.230 --> 20:10.890]  directory, API path name.
[20:10.890 --> 20:12.990]  So, if you've ever written an application, you probably
[20:12.990 --> 20:14.930]  use one of these keywords to reference
[20:14.930 --> 20:16.470]  how you handle a file
[20:16.470 --> 20:18.950]  or anything, right? And so
[20:18.950 --> 20:20.910]  in the English language,
[20:20.910 --> 20:22.910]  in my 15
[20:22.910 --> 20:24.790]  years of testing web applications,
[20:24.790 --> 20:26.830]  I don't see any on here that I haven't
[20:26.830 --> 20:28.750]  found a file inclusion button on. I've found
[20:28.750 --> 20:31.010]  at least one on all of these parameters.
[20:31.550 --> 20:32.690]  We also did
[20:33.230 --> 20:35.330]  insecure direct object reference.
[20:35.590 --> 20:36.690]  So, of all of those
[20:36.690 --> 20:38.510]  two datasets, which at this point
[20:38.510 --> 20:40.710]  encompasses over 32,000
[20:40.710 --> 20:42.730]  vulnerable parameters,
[20:42.730 --> 20:45.050]  out of all of those, out of all of the IDORs
[20:45.050 --> 20:46.610]  that existed there, these are the most
[20:46.610 --> 20:49.250]  statistically likely ones that have IDOR.
[20:49.330 --> 20:50.870]  The ID parameter,
[20:50.870 --> 20:53.790]  the user ID, username, email,
[20:53.790 --> 20:55.990]  extended data, username,
[20:55.990 --> 20:57.230]  data ID, etc.
[20:57.410 --> 21:00.110]  We also did redirect vulnerabilities.
[21:00.710 --> 21:01.570]  So, redirect,
[21:01.570 --> 21:03.690]  very common, has a high score in
[21:03.690 --> 21:05.670]  URL. The
[21:05.670 --> 21:08.030]  parameter next, redirect,
[21:08.030 --> 21:09.810]  return, host,
[21:09.810 --> 21:11.550]  you can see that redirect and return appear
[21:11.550 --> 21:13.670]  multiple times under permutations.
[21:14.050 --> 21:15.510]  So, it makes sense, right?
[21:15.510 --> 21:17.610]  A redirect vulnerability lies
[21:17.610 --> 21:19.690]  in these parameters. Now, one thing is
[21:19.690 --> 21:21.830]  that we looked at parameter data, but if your application
[21:21.830 --> 21:23.630]  uses RESTful URLs, you can
[21:23.630 --> 21:25.410]  just substitute
[21:25.410 --> 21:27.990]  these keyword terms for
[21:27.990 --> 21:29.950]  parameters instead. You could just look
[21:29.950 --> 21:31.430]  for them in the path instead of
[21:31.430 --> 21:32.990]  parameters.
[21:34.690 --> 21:35.810]  So, we also did SQL
[21:35.810 --> 21:37.910]  injection. Obviously,
[21:37.910 --> 21:39.730]  the biggest offender in SQL injection, because it
[21:39.730 --> 21:42.470]  fits the database nomenclature as ID,
[21:42.470 --> 21:44.050]  then email, refer, limit,
[21:44.050 --> 21:46.370]  search, token, username, password.
[21:46.830 --> 21:47.890]  Pro bounty tip, always
[21:47.890 --> 21:49.730]  try SQL injection in the refer,
[21:49.730 --> 21:51.590]  keep a blind SQL injection payload
[21:51.590 --> 21:53.290]  in your refer at all times.
[21:53.630 --> 21:55.930]  Hit that one many, many, many times.
[21:57.110 --> 21:57.770]  Server-side request
[21:57.770 --> 21:59.770]  forgery, we did the same
[21:59.770 --> 22:01.810]  thing here. Same thing,
[22:01.810 --> 22:03.770]  URL, source,
[22:03.770 --> 22:05.750]  start, host, those make a lot of sense
[22:05.750 --> 22:07.610]  for server-side request forgery.
[22:08.290 --> 22:10.090]  Server-side template injection,
[22:10.090 --> 22:11.770]  we didn't get great data for this, so we
[22:11.770 --> 22:13.650]  just... So, the GF patterns have
[22:13.650 --> 22:15.630]  more together. Okay, so hold on,
[22:15.630 --> 22:17.530]  we do have some stuff for server-side template
[22:17.530 --> 22:19.710]  injection. And then cross-site scripting.
[22:20.590 --> 22:21.730]  So, yeah.
[22:21.730 --> 22:23.390]  So these are the places
[22:23.390 --> 22:25.650]  where you're most likely to find vulnerable parameters.
[22:26.470 --> 22:27.670]  And the GF,
[22:27.670 --> 22:29.750]  so when you go up to the GF patterns,
[22:29.750 --> 22:31.390]  the GF patterns, so that was the
[22:31.390 --> 22:33.370]  output for only HackerOne, right?
[22:33.370 --> 22:35.730]  So HackerOne actually didn't have,
[22:35.730 --> 22:37.470]  at least that we found, using the parsing
[22:37.470 --> 22:39.270]  methods we found, didn't have anything for
[22:39.270 --> 22:41.430]  SSTI. Maybe it's not as common,
[22:41.430 --> 22:43.510]  maybe, you know, there's a possibility we missed
[22:43.510 --> 22:45.330]  it, either way. But,
[22:45.330 --> 22:47.270]  there's still old data from
[22:47.270 --> 22:49.310]  Hunt, that's still, like, it's old,
[22:49.310 --> 22:51.290]  but it's still very useful, and it's still the
[22:51.290 --> 22:53.390]  same parameters that are being used. So the GF patterns
[22:53.390 --> 22:55.330]  that we made are the full list
[22:55.330 --> 22:57.210]  of both, right? That's where they both
[22:57.210 --> 22:59.510]  come together, and same thing, the SUS parameters,
[22:59.510 --> 23:00.930]  if you use the SUS parameters list
[23:01.770 --> 23:03.330]  on XNL Hacker's GAP
[23:03.330 --> 23:05.350]  extension, that will be both data sets
[23:05.350 --> 23:07.350]  put together, right? So the
[23:07.350 --> 23:09.310]  GF patterns are basically the end
[23:09.310 --> 23:11.290]  product of Jason's old research, with
[23:11.290 --> 23:13.330]  the five people that were at the bottom,
[23:13.330 --> 23:16.330]  and our research this time, put together,
[23:16.330 --> 23:17.370]  and basically
[23:17.370 --> 23:19.190]  the goal of this is, A,
[23:19.190 --> 23:21.290]  to close the gap in experience,
[23:21.290 --> 23:23.670]  but again, I really want to reiterate,
[23:23.670 --> 23:25.630]  you can do this with anything.
[23:25.770 --> 23:27.470]  There's CVE data out
[23:27.470 --> 23:29.430]  there, there's XSS data out there,
[23:29.430 --> 23:31.590]  we will probably do that at some point,
[23:31.590 --> 23:33.270]  as we continue to play with the repo
[23:33.270 --> 23:35.510]  and bring in the database.
[23:35.630 --> 23:37.130]  Again, if you have internal data
[23:37.130 --> 23:39.210]  that's, you know, pertaining
[23:39.210 --> 23:41.250]  only to your company,
[23:41.250 --> 23:43.410]  make your own lists. The scripts
[23:43.410 --> 23:44.830]  are supposed to be out there, and
[23:44.830 --> 23:46.470]  the repo is public.
[23:47.490 --> 23:49.310]  Specifically so that this can
[23:49.310 --> 23:51.310]  maybe generate some other folks out
[23:51.310 --> 23:53.290]  there, to make maybe some different kind
[23:53.290 --> 23:55.270]  of lists for stuff that you would also want to see.
[23:55.570 --> 23:57.290]  This is kind of an opportunity,
[23:57.290 --> 23:59.370]  hopefully, to again close the experience gap
[23:59.370 --> 24:01.210]  in more fields than
[24:01.210 --> 24:03.590]  just vulnerable parameters.
[24:03.850 --> 24:05.270]  So we had planned to do
[24:05.270 --> 24:07.290]  CVE stuff, and we put it
[24:07.290 --> 24:09.210]  in the abstract for the talk. The problem
[24:09.210 --> 24:10.930]  we ran into is that
[24:10.930 --> 24:13.030]  many CVEs are like
[24:13.030 --> 24:15.190]  what I like to call ghost CVEs.
[24:15.190 --> 24:16.450]  They're CVEs that people
[24:17.090 --> 24:19.270]  say the bug class, and they put it out on CVE
[24:19.270 --> 24:21.070]  details or some other place, but they don't
[24:21.070 --> 24:22.870]  actually give you any exploitation data, right?
[24:22.870 --> 24:25.170]  They just said X product, how to cross-site scripting,
[24:25.170 --> 24:27.170]  whatever. And then the ones that do
[24:27.170 --> 24:29.130]  have proof of concepts, which include URL
[24:29.130 --> 24:31.030]  data, which we need to do this,
[24:31.030 --> 24:33.030]  are often on blogs. And so we have to do
[24:33.030 --> 24:35.210]  two steps of crawling, and
[24:35.210 --> 24:37.170]  we're close to figuring out all
[24:37.170 --> 24:39.090]  that. And so when
[24:39.090 --> 24:41.170]  we get that done, we'll probably try to tackle
[24:41.170 --> 24:43.230]  this data set. And we figured the last
[24:43.230 --> 24:45.030]  five years would be relevant. That's what we had planned
[24:45.030 --> 24:47.210]  for the talk. So we got stymied
[24:47.370 --> 24:49.110]  a little bit on this data set, but we really want to
[24:49.110 --> 24:50.710]  include it in the list as well.
[24:51.430 --> 24:52.950]  The other one we wanted to do
[24:52.950 --> 24:55.170]  was XSS.com,
[24:56.130 --> 24:57.110]  which if you've ever seen
[24:57.110 --> 24:58.770]  that site, it's purely
[24:58.770 --> 25:00.990]  XSS proof of concepts.
[25:01.450 --> 25:02.310]  Oh my god.
[25:07.570 --> 25:08.290]  So this is
[25:08.290 --> 25:10.610]  XSS.com. It's a
[25:10.610 --> 25:12.550]  hacker website where
[25:13.310 --> 25:14.490]  gray hat hackers
[25:15.470 --> 25:17.110]  post a bunch of
[25:17.110 --> 25:18.790]  XSS vulnerabilities,
[25:18.790 --> 25:20.890]  usually when they don't get paid through a bug bounty.
[25:20.890 --> 25:23.190]  So they have a total of
[25:23.190 --> 25:25.010]  45,000 total XSS
[25:25.010 --> 25:26.830]  proof of concepts on XSS.com.
[25:26.830 --> 25:29.370]  So that one would specifically
[25:29.370 --> 25:32.810]  enrich the XSS data.
[25:32.810 --> 25:33.850]  Some other ones
[25:33.850 --> 25:36.330]  that we might look at in the future
[25:34.290 --> 25:36.290]  is actually looking at
[25:36.330 --> 25:38.210]  instead of looking at
[25:38.210 --> 25:40.410]  the bugs that come through HackerOne
[25:40.410 --> 25:42.070]  or BugCrowd, we could look at
[25:42.510 --> 25:44.330]  researcher lists. We could figure
[25:44.330 --> 25:46.310]  out a way to scrape their blogs and look at their
[25:46.310 --> 25:48.470]  personal research and see where they find cross-site
[25:48.470 --> 25:50.310]  scripting vulnerabilities, which was actually
[25:50.310 --> 25:52.490]  the first iteration of anything like this
[25:52.490 --> 25:54.090]  when stuff like this happened in
[25:54.090 --> 25:56.470]  2014 or 2012 or something like that.
[25:56.470 --> 25:58.310]  So the possibilities are endless for us
[25:58.310 --> 25:59.870]  to keep enriching the lists.
[26:00.510 --> 26:02.410]  We think that this project will help a lot
[26:02.410 --> 26:03.270]  of people.
[26:03.270 --> 26:05.530]  And I think
[26:06.790 --> 26:07.270]  that's
[26:07.270 --> 26:08.550]  all we have.
[26:08.610 --> 26:11.050]  So last thing, I will say for people curious
[26:11.050 --> 26:12.990]  about cost,
[26:12.990 --> 26:14.950]  because the few people I've talked to about the talk
[26:14.950 --> 26:17.670]  before we gave it, that was the first question that came up.
[26:18.330 --> 26:19.290]  I parsed
[26:19.290 --> 26:21.390]  all of the HackerOne reports
[26:21.390 --> 26:22.830]  on a personal
[26:22.830 --> 26:24.950]  OpenAI token, and it was right
[26:24.950 --> 26:26.410]  around $30.
[26:27.330 --> 26:29.610]  So it's not, you know,
[26:29.610 --> 26:31.070]  not too bad.
[26:32.910 --> 26:33.990]  Any questions?
[26:35.010 --> 26:36.330]  Kind of talk fast.
[26:37.710 --> 26:38.510]  No? Cool.
[26:38.510 --> 26:39.170]  Thank you very much.
[26:39.170 --> 26:40.410]  Oh, wait, we got one. Sorry.
[26:40.790 --> 26:42.310]  Sorry, I killed you guys.
[26:46.270 --> 26:46.990]  Yeah.
[26:54.570 --> 26:55.190]  Oh, yeah.
[26:55.910 --> 26:56.870]  Oh, yeah.
[26:57.010 --> 26:58.370]  With every model.
[26:58.370 --> 27:00.870]  It will do it sometimes.
[27:02.290 --> 27:04.030]  Let me repeat the question.
[27:04.030 --> 27:06.050]  So the question was, did we try
[27:06.050 --> 27:08.290]  prompt engineering to change
[27:08.290 --> 27:10.350]  the prompt to pull out the data
[27:10.350 --> 27:11.890]  in the format that we wanted?
[27:11.950 --> 27:14.270]  If we said, give us the parameters
[27:14.270 --> 27:15.750]  this one way, did we try more
[27:16.270 --> 27:17.730]  specification in what we wanted?
[27:17.730 --> 27:20.270]  Did we change it to say different words?
[27:20.370 --> 27:21.830]  And the answer is yes.
[27:22.070 --> 27:24.130]  So we tried a lot of that with a lot of the different
[27:24.130 --> 27:25.010]  models.
[27:25.010 --> 27:26.770]  Even, because obviously
[27:26.770 --> 27:28.770]  it will definitely struggle to
[27:28.770 --> 27:30.930]  give you a JSON output or something like that.
[27:30.930 --> 27:32.810]  But even just trying to beg it to be like
[27:32.810 --> 27:34.810]  please just always put it in quotes.
[27:34.810 --> 27:36.730]  Like no matter what the extra stuff is,
[27:36.730 --> 27:38.390]  just please use double quotes.
[27:38.530 --> 27:39.930]  It'll do it.
[27:39.930 --> 27:42.250]  It looks almost identical.
[27:42.490 --> 27:44.590]  It's immensely frustrating.
[27:45.750 --> 27:47.250]  Was there any other questions?
[27:47.450 --> 27:48.290]  Yes, sir.
[28:02.080 --> 28:03.540]  Boom.
[28:04.080 --> 28:05.600]  Yeah.
[28:13.480 --> 28:15.920]  You could try.
[28:15.920 --> 28:17.520]  I feel so bad for your
[28:17.520 --> 28:19.080]  open AI token.
[28:19.820 --> 28:21.440]  Like I guess
[28:21.440 --> 28:23.300]  I never just like let it fully
[28:23.300 --> 28:25.280]  exhaust itself into infinity.
[28:25.280 --> 28:27.400]  But I tried doing retries.
[28:27.400 --> 28:29.500]  I even tried going to the quoted
[28:29.500 --> 28:31.500]  version, and then taking the quoted
[28:31.500 --> 28:33.300]  version, like slowly going
[28:33.300 --> 28:35.480]  downhill, and it still is...
[28:36.280 --> 28:37.500]  Yeah, it has a really hard
[28:37.500 --> 28:39.420]  time creating structured output like
[28:39.420 --> 28:41.080]  JSON. It just doesn't work.
[28:41.340 --> 28:43.500]  I don't know why, but it hallucinates
[28:43.500 --> 28:46.180]  formats, it adds extra characters.
[28:46.560 --> 28:47.380]  It's just because
[28:47.380 --> 28:48.940]  it's trained on next characters, right?
[28:48.940 --> 28:51.360]  It just doesn't create structured JSON
[28:51.360 --> 28:52.120]  very well.
[28:52.920 --> 28:55.360]  Not saying it can't be improved. Again, I'm not
[28:55.360 --> 28:57.320]  saying I'm the expert. This is just like what we've
[28:57.320 --> 28:59.360]  tried. Again, that is probably
[28:59.360 --> 29:01.380]  if you play with this project, that is
[29:01.380 --> 29:02.640]  probably one of the areas to
[29:03.400 --> 29:05.640]  play with, right? Yeah, I mean we wanted that to work.
[29:05.640 --> 29:07.180]  We really did want that to work.
[29:11.540 --> 29:12.760]  Yep, yep. So I
[29:12.760 --> 29:14.960]  actually counted... so with the regex
[29:14.960 --> 29:16.120]  I actually counted
[29:16.820 --> 29:18.580]  how many lines I was actually
[29:18.580 --> 29:20.640]  catching with the regex versus how many I was just like
[29:20.640 --> 29:22.860]  dropping. And the regex
[29:22.860 --> 29:24.600]  was able to catch like 98%.
[29:24.600 --> 29:26.920]  Right? So I mean there's like 2%
[29:26.920 --> 29:28.560]  there that probably still had
[29:28.560 --> 29:30.020]  malformed data or whatever, but like
[29:30.500 --> 29:32.600]  that was as close as I could get. Without
[29:32.600 --> 29:34.860]  literally making like a
[29:34.860 --> 29:36.720]  whole list of like 80 regexes for
[29:36.720 --> 29:38.640]  every single... at that point you're just manually going through
[29:38.640 --> 29:40.740]  the file anyway. And I... why
[29:40.740 --> 29:43.540]  even script it, right? So, yeah.
[29:43.540 --> 29:44.700]  So this is the repo and you guys
[29:44.700 --> 29:46.720]  can use it. Any other questions?
[29:46.720 --> 29:47.120]  Yes.
[29:47.120 --> 29:47.920]  ...
[29:56.640 --> 29:59.220]  That's actually a really great question.
[29:59.220 --> 30:00.940]  So the question was, could you
[30:02.000 --> 30:03.320]  subcategorize these into
[30:04.040 --> 30:05.200]  parameters that were most
[30:05.200 --> 30:07.120]  likely vulnerable on an industry vertical?
[30:07.380 --> 30:08.980]  Like finance or banking
[30:08.980 --> 30:11.060]  or automotive or something like that.
[30:11.400 --> 30:13.120]  You could with the HackerOne data
[30:13.120 --> 30:15.200]  I believe because they... I think they reference
[30:15.200 --> 30:17.280]  the vertical, don't they? So you can
[30:17.280 --> 30:19.180]  with the HackerOne data if you're going to go
[30:19.180 --> 30:21.700]  deep into it because they always reference the target.
[30:21.920 --> 30:23.080]  The publicly disclosed reports
[30:23.080 --> 30:25.200]  always say a target. So if you're willing to
[30:25.200 --> 30:27.340]  make a master list of like for instance
[30:27.340 --> 30:29.280]  the HackerOne targets and say okay this target
[30:29.280 --> 30:31.040]  is financial, this target is like
[30:31.040 --> 30:33.080]  XYZ or gaming or whatever it may be.
[30:33.220 --> 30:35.260]  You could probably do that with AI pretty easily.
[30:35.500 --> 30:37.100]  Yeah. So it is possible. Yep.
[30:38.440 --> 30:39.840]  Any other questions?
[30:40.940 --> 30:41.360]  Okay.
[30:41.360 --> 30:42.480]  Now you guys can clap for Gunnar.
[30:42.480 --> 30:43.560]  Thanks guys.