WEBVTT 00:00.000 --> 00:00.960 Hey, everyone. 00:00.960 --> 00:04.180 My name is Jason, and thanks for joining us this morning. 00:04.380 --> 00:15.520 Super big thanks to the Red Team Village for having me and all the other speakers on and the staff for facilitating everything that's going on with the conference for DEF CON for hosting the Village. 00:15.520 --> 00:17.340 A big place in my heart for DEF CON. 00:17.340 --> 00:22.320 So today I'm going to talk a little bit about my methodology for recon. 00:22.420 --> 00:34.880 I've been doing this talk called the Bug Hunters Methodology for five years now, and I basically update it every year, and I talk about new tools and techniques in different spaces of the Bug Bounty scene and Red Teaming. 00:34.880 --> 00:36.780 So today we're going to go over the recon stuff. 00:37.820 --> 00:46.100 Now, the Bug Hunters Methodology, TBHM, is a presentation, like I said, I've been running for a long time, and basically it's too big for any one conference slot, so I split it into two sections. 00:46.100 --> 00:53.340 One is recon, which we're going over today, and the other is application analysis, which is on-site hacking and stuff like that. 00:53.340 --> 00:59.360 So today we're going to do recon, and I'm working on this year's version of application analysis, haven't finished it yet. 00:59.360 --> 01:00.800 I'm a giant slacker, sorry. 01:01.340 --> 01:03.240 So yeah, today we're going to go over recon. 01:04.480 --> 01:05.640 A little bit about me. 01:06.180 --> 01:10.140 I'm a husband, father, hacker, gamer, and I stream sometimes. 01:10.140 --> 01:14.840 These are my socials, so if you want to reach out and ask a question after the talk or something like that, just hit me up. 01:15.540 --> 01:26.200 I was a bug hunter, and then I worked at Bug Crowd for many years, and now I'm the head of security at Ubisoft, and so I lead the security team there. 01:26.220 --> 01:28.520 Awesome team, great people. 01:28.740 --> 01:32.120 And I'm also a gamer at heart, so I play some video games. 01:32.120 --> 01:36.380 These are my kids, Arcadia and Avalon in the top picture. 01:36.380 --> 01:38.380 I took them to DEF CON for the first year. 01:38.380 --> 01:40.440 Last year I did DEF CON Kids or Roots. 01:40.440 --> 01:42.260 It was amazing, they had a great time. 01:42.260 --> 01:46.640 And then that's my son Arlen, him and I being pirates in the bottom right-hand corner. 01:46.640 --> 01:48.220 It's a little bit about me. 01:48.220 --> 01:49.640 I've been doing pen testing for about... 01:51.720 --> 01:53.440 a long time, I'm old. 01:55.460 --> 01:59.640 I have a lot of related experience to this talk. 01:59.780 --> 02:10.940 The first thing I want to talk about is today we're going to go over recon methodology, and we do recon in red teaming engagements, and we do recon in bug bounties primarily. 02:10.940 --> 02:14.100 And you can do it in some wide-scope pen tests as well. 02:14.200 --> 02:24.140 And one of the first things I talk about is when you're doing an assessment, some sort of assessment, whether it's one of those three, you have to have a way that jives with you to record your work. 02:24.140 --> 02:27.180 Otherwise, you're just throwing darts and forgetting what you're doing. 02:27.180 --> 02:33.460 And this is counterproductive to basically keeping organized. 02:33.460 --> 02:37.560 So the first thing that I talk about here is how I keep notes. 02:37.620 --> 02:41.460 And I use a tool called X-Mind, which is a mind-mapping tool. 02:41.460 --> 02:50.420 And basically, I create nodes, and I create methodologies as checklists inside these nodes, and I track all my domains and which ones I've worked on by color codes. 02:50.560 --> 02:52.520 So this is something I do. 02:52.520 --> 02:59.740 I've seen other people do a lot of project tracking for assessments in just Notepad or Vim, or there's some specialized pen test tools. 02:59.760 --> 03:04.620 If you're really fancy and you have a subscription, you can probably do it in something fancy like Dratus or something like that. 03:04.980 --> 03:10.100 So there's a lot of tools, but it's just really important to track your work when you're doing recon. 03:10.100 --> 03:20.140 Recon is the art of finding as many assets as possible related to a target, and it can get pretty data-heavy and dense as you do it. 03:20.140 --> 03:25.720 So having a way that works for you to track your data is the first thing that's going to guide you to success. 03:25.900 --> 03:30.000 So this is an example of how I track my data when I'm on a target. 03:30.000 --> 03:36.080 So Tesla, Tesla Motors, has a pretty open-scope bounty on Bug Crowd. 03:36.220 --> 03:39.500 And this is how I would start off my project with Tesla. 03:39.500 --> 03:43.780 So in the middle, I have a node in this mind-mapping program I use called xMind. 03:44.000 --> 03:49.920 And then on the left-hand side, I have their autonomous system numbers, their ASNs. 03:49.920 --> 03:52.060 I have any acquisitions they've made. 03:52.060 --> 03:56.980 I have some other notes, LinkedIn discovery, and a link to their reverse-who-is information. 03:57.380 --> 04:01.920 And on the right-hand side, I'm starting to build out their root domains, or their seeds. 04:01.920 --> 04:10.260 Roots or seeds are terms for things like something.com, or teslamotors.com, or tesla.com, or SolarCity. 04:11.020 --> 04:14.660 And so I've started to build out their seeds on the right-hand side. 04:14.660 --> 04:17.380 So you can see Tesla Motors, Tesla, SolarCity, etc. 04:17.880 --> 04:21.180 And as I do this, I'm going to start collecting a lot of data. 04:21.180 --> 04:24.340 As I do recon on Tesla, I'm going to collect a lot of data. 04:24.340 --> 04:35.360 So that turns into this, which on the left-hand side, you can see for tesla.com, that seed, I've enumerated all their subdomains and all of the live links. 04:35.360 --> 04:39.380 So these are all the live links for tesla.com, for their subdomains. 04:39.380 --> 04:43.980 So things like pages.tesla.com, shop.tesla.com, etc. 04:44.520 --> 04:47.240 And then you can drill down into any one of these. 04:47.240 --> 05:00.600 And so if I drill into www.tesla.com, then I have my methodology notes for that single site, which inside of that node, I keep all of the questions I ask myself when I'm basically looking to hack a website. 05:00.600 --> 05:02.920 Like, does the site have multiple user roles? 05:02.920 --> 05:04.140 How does it reference users? 05:04.140 --> 05:05.840 How does it handle special characters? 05:06.060 --> 05:07.700 What are dynamic parameters? 05:07.700 --> 05:09.260 Does it have an API component? 05:09.260 --> 05:11.000 What kind of errors am I seeing? 05:11.000 --> 05:12.860 Does it have file uploads? 05:12.860 --> 05:15.460 Have I done JavaScript analysis for paths? 05:15.460 --> 05:17.580 Have I done content discovery? 05:17.940 --> 05:26.520 All these questions that you're normally going to keep in your methodology, I apply these all to each one of these subnodes. 05:26.520 --> 05:30.280 And then I color code my work based on where I am. 05:30.280 --> 05:38.420 And this is one of the most important things, I think, is that everything that is not filled in hasn't been worked on yet, so most of this hasn't been worked on yet. 05:38.420 --> 05:44.700 But everything in orange means I'm currently working on something, and everything in green, which is there's nothing green on here, I've already done. 05:45.400 --> 05:55.680 And so just the way I can do this, and then I can add checkmarks and other stuff in XMind to these mind maps helps me track where I am, so I can easily put down a project and pick it back up if I want. 05:56.980 --> 06:01.120 Okay, so today's mission, we're going to talk about wide recon. 06:01.120 --> 06:05.480 This is the art of finding as many assets related to a target as possible. 06:05.480 --> 06:11.260 So you're a red teamer, or you're a bug bounty hunter, and you have a scope of X company. 06:11.460 --> 06:17.420 And one of the things you need to do is identify all the websites that they own. 06:17.420 --> 06:25.440 Because the more websites you identify, or the more infrastructure you identify, the better your chances of getting in or finding a bounty, right? 06:25.440 --> 06:29.600 And this is a core component in both of those skill sets. 06:29.600 --> 06:33.520 And I break down what we call recon into a couple of domains. 06:33.520 --> 06:51.140 So finding first the in-scope domains via the program or your project brief, you're finding acquisitions for the company or the target, doing ASN enumeration, doing reverse whois, doing a whole bunch of subdomain enumeration, and then doing port analysis. 06:51.140 --> 07:05.040 And then we're going to go into some related topics like some vulnerability scanning and some automation information related to recon, because you'll probably do it in this phase of your workflow. 07:06.220 --> 07:13.300 Okay, so you've decided you're going to do an assessment or you've been handed a red team assessment or you've been handed a bug bounty that's wide scope, right? 07:13.300 --> 07:14.380 That's where we are right now. 07:14.380 --> 07:18.080 So the first thing we need to do is parse the program page. 07:18.080 --> 07:22.600 So I'm going to use a bounty here, a couple bounties to illustrate where you would start. 07:22.600 --> 07:28.240 So here is the brief page for the aforementioned Tesla for BugCraft. 07:28.240 --> 07:34.000 So in their brief, they have star.tesla.com, star.tesla.cn, Tesla Motors and Tesla Services. 07:34.000 --> 07:43.040 And then they also have this catch-all sentence here that says any host verified or owned by Tesla Motors that you can find is also included in scope. 07:43.040 --> 07:45.620 So this is what we consider a wide scope bug bounty. 07:45.620 --> 07:48.060 There's lots of stuff that you can do with this. 07:48.260 --> 07:51.180 So your first four seed domains are already listed here. 07:51.180 --> 07:54.140 Tesla.services, TeslaMotors.com, TeslaCN, and Tesla.com. 07:54.140 --> 07:56.800 So we're going to focus on these four and add them to our list. 07:57.000 --> 07:59.640 And then anything else we can find that is Tesla's is also fair game. 07:59.640 --> 08:00.520 So that's great. 08:00.800 --> 08:10.320 If you look at a platform like HackerOne, they also have a pretty wide scope bounty and probably the most prolific bounty available on the internet which is Verizon Media. 08:10.320 --> 08:16.980 And this is like the granddaddy, the biggest scope program I think I know of in the bug bounty scene. 08:16.980 --> 08:25.020 So Verizon Media has gobbled up so many brands and they have so much infrastructure related to these brands and sites and subdomains. 08:25.020 --> 08:25.900 It's giant. 08:25.900 --> 08:31.920 So a lot of hackers make their whole career hacking on Verizon Media, which is awesome. 08:31.920 --> 08:34.840 And they're a great team and they support the bug bounty community. 08:34.840 --> 08:43.440 They also have this catch-all kind of phrase in their scope that says if you've found a vulnerability that affects an asset belonging to them but it's not in scope, report it to the program. 08:43.440 --> 08:46.200 And so it's a really advanced team to work with. 08:46.200 --> 08:51.180 You couldn't even fit all of the domains that you have for this program on one page, which I didn't even try. 08:51.180 --> 08:56.780 So you will parse your seed domains and your in-scope domains from the program page to start off. 08:56.780 --> 09:00.100 And this will give you a starting place to work from . 09:00.760 --> 09:06.400 The next thing that I do when I'm scoping out a company is I want to find all their acquisitions. 09:06.400 --> 09:09.080 And there's a couple reasons I want to do this. 09:09.140 --> 09:22.960 I want to basically see if any of the infrastructure for those acquired companies has not been taken off the line or any of the subdomains or websites or integrations that were too hard to port over to new infrastructure are still online. 09:23.260 --> 09:26.800 This gives me more and more infrastructure and sites to hack. 09:26.800 --> 09:30.580 So to find the acquisitions, I use a tool called Crunchbase. 09:30.580 --> 09:34.240 And Crunchbase is a business intelligence portal. 09:34.240 --> 09:43.020 Basically, you add any business name to this search box on Crunchbase and you'll be able to look at that company and find out what are their employees? 09:43.080 --> 09:44.680 Are there investor rounds? 09:44.680 --> 09:46.220 What acquisitions have they had? 09:46.220 --> 09:47.800 What are their socials, etc.? 09:47.800 --> 09:50.560 So here I've used the example of Twitch.com. 09:50.560 --> 09:52.620 We're streaming on Twitch today, which is awesome. 09:53.000 --> 09:56.300 And Twitch as an organization here has an entry. 09:56.300 --> 09:59.620 And if I click on that, I get a lot of information about Twitch. 09:59.620 --> 10:10.400 And I can see that under their acquisitions tab, they have acquired four companies in the last eight years, Revlo, Clipmind, Curse, and Goodgame. 10:10.400 --> 10:14.700 And then also, they were acquired themselves on the left-hand side by Amazon. 10:14.840 --> 10:25.400 So this gives me a good idea of what they used to be, where I can also look for some infrastructure if it's a recent acquisition, like Revlo, for instance. 10:25.400 --> 10:30.500 If I click on Revlo in this page, it'll take me to Revlo's page, which will give me their main domain, which is Revlo.com. 10:30.500 --> 10:35.300 So I might put that in scope of a wide scope assessment or a Red Team assessment. 10:35.680 --> 10:37.500 Acquisitions are important to keep track of. 10:37.500 --> 10:46.100 Now, the one thing that you want to make sure here is that on some of those brief pages that we looked at in the last couple pages, some of these are explicitly out of scope. 10:46.120 --> 10:53.320 So they'll say in the brief, especially for Tesla, SolarCity is not in scope of their wide scope bounty. 10:53.320 --> 11:00.240 So even though we found that as an acquisition of Twitch when I looked on this page, when I've done it many times, it's explicitly out of scope. 11:00.240 --> 11:06.680 So remember to refer to your project page or understand what is okay and what is not okay to hack or do recon on. 11:07.960 --> 11:24.000 So after I find the acquisitions, which has given me seed domains like Revlo.com and new seed domains other than Tesla Motors and Tesla and stuff like that, or any of these top-level seed domains, what I want to do is I want to find this company's autonomous system numbers. 11:24.000 --> 11:32.420 So every company that gets large enough ends up having a collection of networks basically applied an AS number to them. 11:32.420 --> 11:39.020 And an AS number, autonomous system number, is just a collection of your known IP ranges for your networks, for your infrastructure. 11:39.080 --> 11:43.760 And so you can find someone's autonomous system number in many ways. 11:43.760 --> 11:51.560 There's many search engines to do this kind of lookup, but I use one called Hurricane Electric or bgp.he.net. 11:51.560 --> 11:59.100 I use this when I'm doing manual searching because it has a freeform text box at the top and I can just search the keyword Twitch. 11:59.100 --> 12:08.300 And here you can see I found Twitch's two autonomous system numbers, AS46489 and AS397153. 12:08.300 --> 12:13.840 And then their associated IP ranges and IPv4 ranges and IPv6 ranges. 12:13.840 --> 12:19.240 So this is a good representation of the infrastructure or the IP space that they own. 12:19.240 --> 12:24.180 And I can assume that it belongs to... I can guarantee that it belongs to them. 12:24.260 --> 12:29.480 I don't have to worry about scope really here because I know that all this is owned by them. 12:29.480 --> 12:37.700 So if they have that catch-all phrase in their bounty project or you've got the go-ahead in your red team engagement to open scope, then you know this is your target. 12:37.880 --> 12:40.500 One thing that this doesn't represent is their cloud assets. 12:40.500 --> 12:43.020 So this is all owned IP space. 12:43.200 --> 12:47.380 It doesn't represent things like AWS and Azure and GCP ranges. 12:47.380 --> 12:51.620 So you won't find stuff that they host in those environments inside of these ranges. 12:52.180 --> 12:54.580 You just want to be cognizant of that. 12:54.580 --> 12:55.500 So here I've searched Twitch. 12:55.500 --> 12:57.040 I got their ASNs. 12:57.060 --> 13:07.900 With every tool or method here in my methodology, I like to give you a manual way to do it because context is important and getting used to the idea is important. 13:07.900 --> 13:14.800 And I like to give you an automated way to do it in case you're going to script up your own recon framework, which is all the fad these days and I have one of my own and everything. 13:14.800 --> 13:25.500 So there are two tools you can use to parse ASNs and IP ranges from companies that exist on the command line if you're going to script this stuff up yourself. 13:25.720 --> 13:28.280 So one is called Metabigor. 13:28.640 --> 13:30.980 I'm still not sure if I'm saying that name of the tool right. 13:30.980 --> 13:32.160 By JesseJJJ. 13:32.600 --> 13:39.580 And this one utilizes the site that we just saw, bgp.he.net and another site called ASN Lookup. 13:39.580 --> 13:46.000 And it will grab the information from those websites, scrape it off and give you your IP ranges for an organization. 13:46.000 --> 13:48.660 The other one is called ASN Lookup by Yasin. 13:48.740 --> 13:49.740 Same idea. 13:49.740 --> 13:52.560 It uses a different data source called the MaxMind database. 13:52.560 --> 14:01.640 It'll pull off their IPv4 ranges from an ASN and you can give these both tools a term instead of an AS number or anything like that. 14:01.640 --> 14:05.240 So they will pull it off based on that keyword that we're looking for. 14:05.240 --> 14:07.220 In this case, Tesla or something like that. 14:07.240 --> 14:14.280 Now, one thing to know about these tools is they are looking off your keyword here, which we put on the command line, Tesla. 14:14.440 --> 14:17.160 And there are multiple companies with Tesla in the name. 14:17.160 --> 14:21.480 There's like Tesla Research Lab or something like that in Sweden or something like that. 14:22.140 --> 14:31.020 Just be sure that when you get these ranges and you start seeing these sites, you can identify, oh no, I might have gotten one that's actually not Tesla Motors. 14:32.140 --> 14:35.880 Both these tools, I verified, pull up the right data for Tesla. 14:35.880 --> 14:43.660 But just be cognizant of, since you're using a keyword here to search, that you don't get other IP ranges that are not your targets. 14:43.660 --> 14:45.900 You'll have to verify somewhat manually there. 14:47.140 --> 14:49.640 Okay, so we've gotten some IP space from the ASNs. 14:49.640 --> 14:50.980 We've gotten some seed domains. 14:50.980 --> 14:54.260 We want to continue gathering as many seed domains as possible. 14:55.020 --> 14:59.120 We have the AS number, and for here we're going to use Twitch as an example again. 14:59.480 --> 15:03.300 Their autonomous system number is 46489. 15:03.660 --> 15:09.100 And here what we want to do is we're going to get our first introduction to Amass. 15:09.100 --> 15:14.920 And Amass is a framework for domain intelligence, I would call it, I guess. 15:14.920 --> 15:16.380 And it is written by Jeff Foley. 15:16.380 --> 15:18.880 There is a workshop you should absolutely sign up for. 15:18.880 --> 15:22.360 He's going to do a full workshop on Amass and how to use it. 15:22.520 --> 15:26.380 Jeff is as associated to the Red Team Village, so super cool. 15:26.580 --> 15:32.320 And here we're going to use Amass and feed it our autonomous system number, 46489. 15:32.320 --> 15:38.720 And what it's going to do is it's going to go out to all of the IP ranges that are represented by that autonomous system number. 15:38.720 --> 15:42.620 And it's going to scan the certificates for all HTTPS sites. 15:42.620 --> 15:49.580 And then it's going to parse those certificates and tell us, hey, here are the seed domains I found that are owned by them. 15:49.580 --> 15:52.940 And here you can see that we found some seed domains that we didn't know about before. 15:52.940 --> 15:55.820 Justin.tv, which is the company Twitch used to be. 15:57.360 --> 16:02.360 TTVNW.net, which is, I think, associated to Twitch's streaming protocol. 16:02.780 --> 16:05.040 Twitch.tv, obviously, which is their main site. 16:05.100 --> 16:06.400 TwitchCon, which is their conference. 16:06.400 --> 16:08.460 And SocialCam, which I have no idea what is. 16:08.960 --> 16:19.080 And all the links for these tools and the methods are posted in the slides, so you can click through when the slides get released and go grab the tools and play with them on your own. 16:19.080 --> 16:25.440 So here we've already expanded our assessment a lot, right? 16:25.440 --> 16:34.040 We have a whole bunch of IP space, we have a whole bunch of top-level domain seed domains, and we have the seed domains already given to us by our brief page. 16:34.680 --> 16:37.140 Okay, so the next method is reverse whois. 16:37.320 --> 16:43.760 And what reverse whois does is it will take the whois entry of... you can search your domain in Twitch.tv here. 16:43.820 --> 16:45.880 And you can do this with a lot of sites. 16:45.880 --> 16:53.040 I use huoxi.com because it's pretty cheap and you can get a free API key for so many uses. 16:53.260 --> 17:03.880 And you basically supply it your domain and it will give you the whois data and then you can use that whois data to correlate who else has registered stuff. 17:03.880 --> 17:07.520 So an example here is we gave this site huoxi.com, twitch.tv. 17:07.520 --> 17:16.940 And then it said that over the course of the years, there have been several companies' names related to the whois data and also registered emails. 17:17.200 --> 17:28.020 And so here you can see that the company, as of March 26, 2015, was justin.tv and the email in the domain was domainmaster at justin.tv. 17:28.080 --> 17:31.240 And then later in 2015, it changed to twitchinteractive. 17:31.240 --> 17:38.600 And if you click the button right next to those entries, you can see there's 20 other domains still registered to justin.tv, which could be interesting to us. 17:38.600 --> 17:42.960 And there's 575 domains registered under twitchinteractive. 17:42.960 --> 17:50.600 And so we can start pulling back more data, more subdomains, more root domains from these links when we click on them. 17:50.600 --> 17:54.640 Now, one thing to know here is that whois data is registered data. 17:54.640 --> 18:04.420 And so a lot of these companies will park domains to combat phishing and to hold stuff for marketing campaigns and stuff like that. 18:04.420 --> 18:05.600 So they might not be live sites. 18:05.600 --> 18:06.580 They might just be parsed. 18:06.580 --> 18:16.140 So this is a medium fidelity type of technique here, but I have found some really good stuff using reverse whois and the registrant data of these sites. 18:16.920 --> 18:19.400 There is a tool to do this type of analysis. 18:19.400 --> 18:22.980 It's called domlink, written by Vincent Yu via security on Twitter. 18:23.000 --> 18:26.840 And it uses that exact site, hoaxy.com. 18:27.100 --> 18:36.400 And you can basically give it a domain and it will find every other associated domain by both registrant email and organization name or company name. 18:36.420 --> 18:39.260 And it will do it on the command line and return it to you in a script. 18:39.260 --> 18:40.260 And it's recursive. 18:40.260 --> 18:41.840 So a really cool tool here. 18:41.840 --> 18:45.600 You can use this to automate some of that reverse whois lookup. 18:45.600 --> 18:46.300 It's called domlink. 18:46.300 --> 18:47.400 The tool is called domlink. 18:48.880 --> 19:00.980 So the next method that we have now that we've maybe gotten some more seeds and subdomains from reverse whois and we're building out our list of things that we can hack is we want to look at the ad and analytics relationships of our target site. 19:00.980 --> 19:06.580 We want to see what other sites are using the same ad and analytics codes as our main site. 19:06.580 --> 19:17.220 And this will give us a good understanding of maybe what are their most popular domains and maybe what are some of their less popular domains that are still using the codes and we didn't know about before. 19:17.340 --> 19:27.080 So every page usually embeds a Google Analytics code or a New Relic code or some kind of ad or analytics code in the main page. 19:27.080 --> 19:40.320 So here if we use a site called builtwith.com, we can search twitch.tv and twitch.tv will give us a list under the relationship profile tab of all of the keys that are on twitch.tv. 19:40.340 --> 19:47.960 And then you can click on those and see what other sites are using the same ad and analytics codes and maybe we didn't know about those before. 19:47.960 --> 19:53.280 So here we can... it's hard to see on the bottom but we have a couple entries that we didn't know about before. 19:53.420 --> 19:57.280 manvsgame.tv, twitc.tv. 19:57.360 --> 20:07.420 We knew about twitchcon.com from the previous method but we're starting to build up more domains that we can see that are related to our target. 20:07.420 --> 20:11.620 So ad and analytics relationship profiles is a pretty powerful technique. 20:11.660 --> 20:14.620 I have used it for success a couple times. 20:14.620 --> 20:17.220 You can also do this directly from Firefox or Chrome. 20:17.220 --> 20:21.920 They have an extension for this site called built with and you can just install it. 20:21.920 --> 20:31.500 You make a free account and then you visit your site in the browser, click the Chrome extension button and it will give you the relationship information right in your browser and you can start clicking around there. 20:31.500 --> 20:36.680 So pretty easy to institute while you're doing your recon in a browser. 20:38.760 --> 20:42.080 So you can also use a tool to do this in the command line. 20:42.080 --> 20:48.380 So Malik, after my first iteration of this talk, quickly scripted up an awesome tool to do this. 20:48.680 --> 20:53.400 It's called getrelationship.py and you just pass it your target here. 20:53.400 --> 21:01.200 Here he's targeted Uber.com which also has a bounty program and passed it to this Python script and it will pull out on the command line all of the related domains. 21:01.200 --> 21:14.860 And you'll have to manually kind of verify these but the ones I would look at first are the ones that actually have Uber in the domain somewhere like daft.uber or driveuber.co.nz or things like that. 21:14.860 --> 21:19.300 So you'll have to sift through this but you can do it on the command line now too. 21:21.870 --> 21:28.230 Then you can just use some general Google hacking or Google-fu to try to find associated sites to your target. 21:28.410 --> 21:39.230 You can use the copyright text, the terms of service, and the privacy policy and you can just copy those from the bottom of any of the pages and here you can see Twitch interactive ink would be something I'd Google for. 21:39.330 --> 21:48.890 And then you can see any other page that is hosting that text at the bottom of their page which could be something that you didn't know about before in your recon so far. 21:48.890 --> 21:57.690 So you can just do this manually and you can use other search operators like in URL Twitch or you can just look for these policies. 21:57.690 --> 22:02.630 One tip here is that you don't just want to target a certain year. 22:02.630 --> 22:05.990 Like here I have 2019 but you also want to check 2018 and 2017. 22:06.050 --> 22:15.370 In fact, the more stuff you find that goes back farther in age, the more likely it is to be vulnerable and less likely to have a lot of security assessment associated to it. 22:15.370 --> 22:20.190 So probably more vulnerable for your Red Team engagement or your bug bounty. 22:22.170 --> 22:27.330 Alright, the next method that we're going to use to find some seed domains or related infrastructure is Shodan. 22:27.330 --> 22:33.670 I have to admit I'm not a Shodan ninja but I know a lot of Red Teamers are and I actually love Shodan and it's just a failing of mine. 22:33.670 --> 22:34.550 I need to get better at it. 22:34.550 --> 22:44.970 But there's a ton of search operators we could use here on our domain twitch.tv and what Shodan is is it's the site that is a site that hosts information from an infrastructure based spider. 22:45.010 --> 23:08.470 And a spider goes to every site on the internet and basically captures its HTTP responses, its technology stack, its certificate data and cross links it so you can click on any of these pieces of information and find out other things it's seen related and what organizations it's related to and what common tech that they're using, 23:08.470 --> 23:12.070 what web servers, what JavaScript framers, you could do anything. 23:12.070 --> 23:20.770 So Shodan is a really powerful tool and you can query our domain here and try to find related infrastructure. 23:20.770 --> 23:37.630 For instance, in one of the entries when I just didn't even use a great operator here, I just searched my domain twitch.tv I ended up seeing in the SSL certificate data of the search there was twitch.amazon.eu and now I need to ask myself a question like is that in scope, 23:37.630 --> 23:37.790 right? 23:37.790 --> 23:47.230 Like that's not verbatim a Twitch domain but in a red team assessment if that's related to Twitch infrastructure, well maybe it's a target that I can go after. 23:47.230 --> 23:49.690 So Shodan can give you a lot of good information. 23:50.330 --> 23:53.730 Okay , so we found seed domains, stuff to start with. 23:53.810 --> 23:57.810 Now what we're going to do is we're going to try to find subdomains for those seed domains. 23:58.170 --> 24:10.470 So these are things like www is a subdomain but also admin.tesla.com or forums.tesla.com or something like that or forums.twitch.tv or whatever. 24:10.470 --> 24:12.290 We're going to try to find subdomains. 24:12.450 --> 24:19.030 And so this idea of starting with seeds and then moving to subs really is going to multiply. 24:19.030 --> 24:22.250 The more seeds you find the more subdomain enumeration you can do. 24:22.250 --> 24:26.870 The more subdomains you find is related to how many sites you find. 24:26.870 --> 24:33.170 The more sites you find, the more successful you'll be inside of your red team engagement or bug bounty program. 24:33.470 --> 24:40.310 So for subdomain enumeration I use three-ish different methods to get subdomain information. 24:40.310 --> 24:42.610 One is LinkedIn JavaScript discovery. 24:42.790 --> 24:47.590 Another one is subdomain scraping techniques and then subdomain brute force. 24:47.590 --> 24:49.930 And then there's some auxiliary stuff I use as well. 24:49.930 --> 24:51.310 So we're going to run into those right now. 24:51.310 --> 24:54.250 And this is the bulk of what a lot of tools are doing right now. 24:55.070 --> 24:58.150 And so you'll see a lot of favorites in these slides. 24:58.150 --> 25:05.670 So the first area we're going to use to find subdomain data is or the first technique is linked discovery and JavaScript discovery. 25:06.410 --> 25:08.190 So linked discovery, what is it? 25:08.190 --> 25:16.810 It's basically using a web spider to visit a site like twitch.tv land on it and spider all the HTML links. 25:16.810 --> 25:17.730 Pretty simple. 25:17.730 --> 25:25.350 And any of those that come back with a subdomain we just add to our list of in-scope targets. 25:25.350 --> 25:27.790 And so this is pretty self-explanatory. 25:27.790 --> 25:31.770 It happens naturally when you're using tools like burp suite and you're spidering a site. 25:31.770 --> 25:33.830 But I'm going to walk you through how I do it. 25:33.830 --> 25:41.750 I'm going to use burp 1.7 because I like the UI better but it works as well with the crawler or the scanner in burp 2.0. 25:41.750 --> 25:43.710 So use whatever you want. 25:43.710 --> 25:45.510 And then I'm going to give you some other methods to do it. 25:45.510 --> 25:49.770 So the first thing we do is we load up burp suite which is on the right here. 25:49.770 --> 25:51.490 It's a side-by-side to our site. 25:51.630 --> 25:54.990 And we just visit our site through burp suite the proxy. 25:55.350 --> 25:58.650 And burp suite the proxy will capture all this data. 25:58.650 --> 26:00.830 We've only visited one page. 26:00.830 --> 26:12.390 If you visit twitch.tv ever, you see that it has cross-linked and it does request a lot of stuff because it's a streaming media site and so there's a lot of stuff going in the background when you go to twitch.tv. 26:13.110 --> 26:15.330 So you can see all that data on the right-hand side. 26:15.330 --> 26:20.950 This is everything that's either seen been linked on this page or it's actually been requested. 26:20.950 --> 26:24.510 Actually been requested is the stuff in black, seen is the stuff that's in gray. 26:24.950 --> 26:30.770 And so what you do is you visit your page and then you have to set up a rule, some type of rule. 26:30.770 --> 26:38.270 And you can do this by going into the target tab and the scope tab and then you click this box that says use advanced scope control. 26:38.570 --> 26:44.250 And what I do is I enter in just a keyword here in advanced scope control which I just say twitch, right? 26:44.250 --> 26:51.070 So I want to see in burp anything that has twitch in the URL at all, no matter where it is, what has that word twitch. 26:51.070 --> 27:00.630 And you could also add some more here because we've seen in our previous analysis TWT is also part of their domain naming nomenclature, right? 27:00.630 --> 27:13.470 And so you can add a couple of these scope rules and then you go back to your sitemap and you click on the ribbon bar up at top or the filter bar and then you say show only in scope items. 27:13.650 --> 27:17.870 So now this will just show in our sitemap all the twitch.tv links, right? 27:17.870 --> 27:22.030 So we already have a good list of subdomains here, right? 27:22.050 --> 27:26.570 Sentinel.twitchservice.net API.twitch.tv. 27:26.590 --> 27:29.030 So we have a lot of stuff with twitch in the name here. 27:29.030 --> 27:31.270 We know that most of this is probably related to twitch. 27:31.270 --> 27:35.130 This gives us exponential areas to hack, right? 27:35.130 --> 27:36.130 And this is pretty good. 27:36.130 --> 27:46.930 But if we select all these and then we spider them using burp spider, it will then go to all of them and find their links and find subdomains references in their pages. 27:46.930 --> 27:50.990 And we can do this recursively until there's nothing left to find. 27:50.990 --> 27:54.730 So here I'm going to select all of these and then I'm going to spider them. 27:54.730 --> 28:00.970 And now you can see, since I had them selected, everything in orange on the right-hand side was stuff I knew about before I spidered. 28:00.970 --> 28:04.810 And then after I spidered with burp, everything in white is new stuff. 28:04.810 --> 28:08.730 And you can see I found a combination of a lot of stuff here. 28:08.730 --> 28:20.010 I found subdomains for twitch.tv and I've also found some new root and seed domains like twitchapp.net and twitchservice.net and x-twitch.tv. 28:20.190 --> 28:21.970 So this is a hybrid technique. 28:21.970 --> 28:24.950 It can find you new seed domains or roots and subdomains. 28:24.950 --> 28:27.570 And so now I have all of this domain data. 28:27.570 --> 28:32.210 I can now select all of these and spider them again and spider those pages for links. 28:32.210 --> 28:35.350 And eventually you can do this until you get kind of spider fatigue. 28:36.230 --> 28:40.870 And eventually you'll have a large sitemap buildup of targets, which is awesome. 28:41.410 --> 28:44.490 Now, how do you get this data out of Burp Suite? 28:44.530 --> 28:47.030 There's not a great way to do this. 28:47.030 --> 28:50.730 Pro has a thing called Engagement Tools. 28:50.730 --> 28:54.790 And in Engagement Tools, you can do this thing called Analyze Target. 28:54.890 --> 29:02.090 And in Analyze Target, you can create an HTML report of all the targets in your site tree that are selected. 29:02.090 --> 29:02.890 So that's what I do. 29:02.890 --> 29:04.790 I select everything in the site tree. 29:04.790 --> 29:08.450 I go Engagement Tools, Analyze Target, Generate HTML Report. 29:08.450 --> 29:14.290 And then in the HTML report, I have a parsable list of the domains that it's seen at the beginning of that HTML report. 29:14.290 --> 29:17.510 And then I take that and I put it in a text file for later analysis. 29:17.510 --> 29:19.590 And then I dump it into my mind map as well. 29:20.930 --> 29:21.630 Okay. 29:21.630 --> 29:26.950 So the whole linked discovery thing counts on Burp Spider, right? 29:26.950 --> 29:42.070 A lot of times I hadn't seen before the last couple of years, great options to do this in the command line because it would require a lot of custom coding, bash scripting or Python to create my own spider and do that same process. 29:42.210 --> 29:47.750 Well, now there does exist some tools that are command line spiders with bug hunters and red teamers in mind. 29:47.750 --> 29:48.290 There's two. 29:48.290 --> 29:51.090 There's one called Ghost Spider written by Jesse JJJ. 29:51.570 --> 29:56.810 And it also designates the types of things it's parsing when it's visiting a URL. 29:56.810 --> 29:58.670 So it'll give you... 30:25.120 --> 30:25.480 Okay. 30:25.480 --> 30:26.440 Hopefully you guys can hear me. 30:27.100 --> 30:28.140 Headphone problems. 30:28.140 --> 30:29.300 Going to plug it in. 30:29.780 --> 30:30.580 Okay. 30:31.340 --> 30:32.420 Great, great, great. 30:32.420 --> 30:33.940 I was getting beeping in my ears. 30:33.940 --> 30:34.940 So maybe it's just running batteries. 30:34.940 --> 30:35.640 My bad. 30:35.700 --> 30:36.200 Okay. 30:36.200 --> 30:39.420 So you have two spiders here that you can use in the command line. 30:39.420 --> 30:42.440 Ghost Spider and Hackcrawler by HackLuke. 30:42.440 --> 30:44.220 And both these are awesome. 30:44.460 --> 30:54.200 They both have some functionality that will parse out the types of things you're getting, like JavaScript files, subdomains, URL endpoints. 30:54.560 --> 30:57.940 Some of them will give you, I think, parameter names and stuff like that. 30:57.940 --> 31:06.040 So you could institute the whole process of link discovery by scripting up Ghost Spider or Hackcrawler if you wanted. 31:06.040 --> 31:13.760 I still use Burp, but these are invaluable tools to have at your disposal, just like a crawler that can do some analysis. 31:14.240 --> 31:16.960 So keep them bookmarked for when you might need them. 31:18.420 --> 31:18.860 Okay. 31:18.860 --> 31:25.720 So the next place we're going to get subdomain information is by analyzing some JavaScript. 31:26.020 --> 31:38.080 And one of the tools I like here, just because it has this kind of little added benefit that I think that I haven't seen many places before, but maybe some other tools are starting to implement it now, is a tool called Subdomainizer by Neeraj Edwards. 31:38.440 --> 31:57.260 And what it'll do is it'll take a JavaScript file, you have to point it to a JavaScript file, and it will parse out all the cloud services, the subdomains, and it'll do this little extra thing where it uses the Shannon entropy formula or algorithm to identify things that look like API keys hard-coded in JavaScript, 31:57.260 --> 32:05.500 which is already kind of a vulnerability if you find a private API key or credential hard-coded in JavaScript, which I know sounds crazy to a lot of people. 32:05.500 --> 32:10.440 Like, you would just find a hard-coded private API key, but this happens all the time. 32:10.440 --> 32:12.440 Like, all the time it happens. 32:12.860 --> 32:20.820 And so this is what I like to call a forward-thinking type of thing, is using an algorithm like this to identify keys. 32:20.820 --> 32:24.840 Sometimes it's a little noisy, but a lot of times it finds you good stuff. 32:24.840 --> 32:31.180 So I like this tool to point at all the JavaScript files I found already on some of these sites. 32:31.460 --> 32:42.240 If you're just looking for subdomain information, there's another tool called Subscraper by Celian Collins, which has recursion built into it, which can do this method as well, but it doesn't do the API key part. 32:42.740 --> 32:55.280 So point this at JavaScript files you find on your site, and you can get back a whole bunch of subdomains and cloud services that the site might use or the target might use. 32:56.540 --> 32:56.860 All right. 32:56.860 --> 33:01.080 So that's link discovery and JavaScript analysis for subdomains. 33:01.080 --> 33:06.320 Now we're going to get into the big meats of what a lot of people do is subdomain scraping. 33:06.420 --> 33:16.120 Now the idea of subdomain scraping is going out to these websites on the internet that have search boxes, basically. 33:17.080 --> 33:24.740 There's all these projects on the internet, search engines, security websites, certificate projects and stuff like that. 33:24.740 --> 33:26.120 They all do different stuff. 33:26.120 --> 33:29.700 So Robtex gives you infrastructure information. 33:29.700 --> 33:36.280 The Wayback Machine houses information about domains and their responses years past. 33:36.280 --> 33:38.720 Everyone's probably used the Wayback Machine. 33:39.120 --> 33:44.540 The certificate sources down here, some of them are certificate projects to provide certificate transparency. 33:45.000 --> 33:47.440 Search engines, obviously you know what search engines are. 33:47.600 --> 33:56.040 And then there's a whole bunch of security sites that do different things, like give you a rating on how malicious a URL or a site might be. 33:56.040 --> 34:00.680 The common thing that these all have is either have an API or a search box where you can put in a domain. 34:00.700 --> 34:05.760 And they will search that domain and they will tell you anything they've seen related to that domain. 34:05.800 --> 34:16.620 And if you put in a domain like Tesla .com, the information that comes back is parsable and you could possibly find out that they know about some subdomains of Tesla.com that we don't know about. 34:16.620 --> 34:19.360 So this is the process of subdomain scraping. 34:19.360 --> 34:26.200 We're going to go to all of these sources, all of these sources and ask them, hey, what do you know about Tesla.com or Twitch.tv? 34:26.200 --> 34:28.780 Do you know of any subdomains that maybe I don't know of? 34:28.800 --> 34:29.360 Okay, cool. 34:29.360 --> 34:30.360 Let's do that. 34:30.420 --> 34:33.920 Now, there are many more sources that are on this page, right? 34:33.920 --> 34:39.980 These are only a subset and new sources to parse are coming out as fast as new websites are coming out. 34:39.980 --> 34:50.200 So the tools have to keep up adding new novel sources to find subdomain data or URL data as they mature. 34:50.720 --> 34:54.180 Now, this is the example of using a search engine like Google to do it, right? 34:54.180 --> 35:05.680 So here, what we're doing is we're searching with the search operators site.twitch.tv and we're saying, I already know about www.twitch.tv, so we're saying minus www.twitch.tv. 35:05.680 --> 35:12.140 And we already know about watch.twitch.tv, so minus watch.twitch.tv and minus dev.twitch.tv because we already know about that. 35:12.140 --> 35:24.260 So now, Google's only showing us things that are not those and so we can do this process, keep on minusing out domains until Google doesn't know anything more about subdomains related to twitch.tv. 35:24.520 --> 35:31.720 And that way, we can basically get a full inventory of what Google knows and what Google knows about subdomains for twitch.tv. 35:31.720 --> 35:34.480 So this is an example of doing it manually with Google. 35:34.520 --> 35:36.380 But luckily, you don't have to do this yourself. 35:36.380 --> 35:40.800 There's tools out there that will do this type of analysis, this scraping for you. 35:41.880 --> 35:45.900 The first one is Amass by Jeff Foley and the Amass team. 35:45.900 --> 35:47.860 There's a whole team behind this. 35:47.860 --> 35:51.380 And there's two tools I use here, Amass and Subfinder, but we're going to look at Amass first. 35:51.580 --> 36:00.980 Amass has probably the most sources for any subdomain scraping tool in existence in its enum section. 36:00.980 --> 36:09.280 Amass is a framework for domain information, but the one that actually pulls this method's results, subdomain scraping, is called Amass Enum. 36:09.940 --> 36:20.420 And here on the right-hand side, you can see we gave it twitch.tv and it went out to a whole bunch of sources it's had in its databases and it reached out to those web pages using curl or whatever. 36:20.480 --> 36:22.180 I don't know if they use headless chromium. 36:22.180 --> 36:23.620 I don't know exactly what they're using. 36:23.620 --> 36:30.400 And then they parse those pages for subdomains for twitch.tv and they give us a list of all the subdomains for twitch.tv. 36:30.960 --> 36:39.500 And if we do this on all of our seed domains that we've gathered, we have now started to exponentially build out the amount of sites that we have to attack. 36:39.500 --> 36:42.480 So this tool is invaluable. 36:42.600 --> 36:51.980 Amass is becoming the go-to tool for all subdomain enumeration, especially this thing, subdomain scraping. 36:52.020 --> 36:56.080 The other tool I used here is oh, also, this is still Amass. 36:56.080 --> 37:04.040 What Amass does at the end of a run, not only does it give you all the subdomains, but it also gives you this great table, which I feel like is underutilized a little bit. 37:04.040 --> 37:13.460 And this table tells you, okay, I discovered 439 subdomains and here is where they were inside of these ASNs. 37:13.460 --> 37:25.780 And you can see that most of twitches were in Amazon's ranges, which makes sense because they were acquired by Amazon, but you can see some of them were in other ASNs, right? 37:25.780 --> 37:37.480 And I've had instances where, not on Twitch, but on other projects where there was a whole ASN related to the company I didn't know about that I found out because Amass built this table. 37:37.480 --> 37:43.880 They're like, hey, 70 of these subdomains we discovered appeared in this ASN or these IP ranges. 37:43.880 --> 37:48.720 And then I look at that and I'm like, oh, I wasn't initially looking at those ranges. 37:48.720 --> 37:49.440 Weird. 37:49.440 --> 37:53.300 Let's go back to the beginning of my workflow and start enumerating those ranges. 37:53.300 --> 37:55.320 So this table is super cool. 37:55.320 --> 38:00.020 It also gives you information on their third party kind of tools they're using. 38:00.700 --> 38:07.080 Here you can see that Twitch is using Bitly and SendGrid and some other stuff. 38:07.080 --> 38:09.520 So it also gives you information there in Fastly. 38:11.040 --> 38:15.280 Okay, the other tool I use for subdomain scraping is Subfinder. 38:15.280 --> 38:24.400 Originally written by Iceman and Michael Skelton, I think now navigated to the ProjectDiscovery.io team, which is a group of bug hunters releasing some stellar tools. 38:24.760 --> 38:30.760 And they also have multiple sources, extensible output, also a really great tool. 38:30.760 --> 38:33.840 Both of these tools are really good. 38:34.960 --> 38:36.900 Each one of them have different sources. 38:36.900 --> 38:45.640 So what I end up doing is in my automation, I run both of them and then I cat the output together and sort it and unique it from both the tools. 38:45.640 --> 38:50.020 So they both have a couple of different sources and a lot of the same sources. 38:50.020 --> 38:53.360 So that's what I do for my stuff is I just use both of them. 38:55.460 --> 39:03.540 Okay, so this one is somewhat new and it has been integrated into Amass a little bit. 39:04.400 --> 39:04.700 But 39:08.060 --> 39:16.520 I've had inconsistency with run between this standalone tool, GitHub subdomains.py and the output of Amass. 39:16.520 --> 39:21.640 So I still use this independently inside of my automation when I'm looking for subdomains. 39:21.640 --> 39:29.400 So this tool is called GitHub subdomains.py and it's still scraping and what this is doing is it's going out to GitHub as a source. 39:29.400 --> 39:44.700 And basically you're providing GitHub and if you've ever been on GitHub, they have the search box up top and you're saying search for twitch.tv and then anything that comes back with twitch.tv as a piece of source code, it will parse up the subdomains from that piece of source code. 39:44.840 --> 39:53.420 Now this was written by Gwendolyn Kuik and I still don't know if I'm saying Gwendolyn's name right and that's okay. 39:53.540 --> 39:58.460 And Gwendolyn wrote a whole suite of tools for GitHub enumeration. 39:58.460 --> 40:03.020 How to find secret keys in GitHub related to an organization. 40:03.020 --> 40:04.680 How to pull email addresses out. 40:04.800 --> 40:10.140 He has a wonderful blog and it's linked here in this slide with a whole bunch of GitHub tools. 40:10.140 --> 40:13.340 Now GitHub subdomains is just looking for subdomains. 40:13.920 --> 40:25.580 The thing about using this tool is that the GitHub API, not the tool is just kind of instable and doesn't give you like somewhat returns rate limited results sometimes. 40:25.580 --> 40:55.060 So what I do inside of my automation and this has given me subdomains I haven't found anywhere else using this method is I run GitHub search or GitHub subdomains.py like five times and I sleep in between each run so that I give it a little while for the rate limiting to die down and then I give it a big sleep at the end before I run another one and then that seems and then I cat all those results together and unique them and that seems to give me more consistency when parsing the API. 40:55.060 --> 40:56.780 And that has nothing to do with the tool that Gwendo wrote. 40:56.780 --> 41:00.140 It's all about what GitHub does with their search API. 41:00.140 --> 41:07.940 So this is an awesome tool to find lesser known subdomains and I found some great stuff using this tool. 41:09.160 --> 41:21.040 Okay, the next one is show sub go by incogbite and this one will parse using a number of search operators with your API key. 41:21.040 --> 41:23.340 It will parse showdown. 41:23.460 --> 41:27.020 So I include this in my script as well. 41:27.320 --> 41:35.380 Again, this is one of those tools that some of the subdomain frameworks like Amass and Subfinder might do for you. 41:35.400 --> 41:39.860 I find using the standalone tool for some reason just works better for me. 41:39.860 --> 41:46.020 So I don't know exactly why that is but I've run them side by side and gotten different outputs so I still use this verbatim. 41:46.020 --> 41:48.200 And it's also a fast running script. 41:48.200 --> 41:51.320 It's not like I'm waiting for it to complete for 5-10 minutes or something like that. 41:51.320 --> 41:53.000 It usually runs in a minute or two. 41:53.060 --> 41:58.980 So it doesn't skin off my nose if I have to wait an extra couple of minutes to ensure that I get coverage out of showdown. 41:58.980 --> 42:09.640 So show sub go will take a domain and your showdown API key and parse showdown using some search operators and give you back all the subdomains related to here twitch.tv. 42:11.880 --> 42:19.600 Okay, so the last... well, not the last, but one of the other methods to look for subdomain enumeration or subdomain scraping is the cloud ranges, right? 42:19.600 --> 42:31.280 So we talked about this a little bit earlier is that we have all this IP space and now we've started to identify subdomains and you'll notice that some of those subdomains are resolving to infrastructure or sites in the cloud. 42:31.880 --> 42:46.600 Now there's this idea that not a lot of people have been doing is just going out recently or not recently but people have been doing it for a while but it hasn't been much public is just scanning the entire ranges for AWS, GCP and Azure for SSL sites, 42:46.600 --> 42:46.720 right? 42:46.720 --> 42:55.280 Anything that responds to... anything that responds to, you know, or responds to a connection on 443. 42:55.280 --> 43:09.560 And you basically scan the cloud ranges and you scan it by IP address and then when it responds it'll give you its SSL certificate and then you parse the organization name or the domain name out of the certificate data. 43:09.580 --> 43:15.580 And then you look at that data and you say does it match my target twitch.tv or does it have the keyword twitch in it? 43:15.580 --> 43:22.700 And you're like, cool, I found some stuff that these people have put in the cloud that wasn't part of their ASN that I probably didn't know about before. 43:23.100 --> 43:26.360 Now that's a tremendous amount of scanning, right? 43:26.360 --> 43:28.040 Those ranges are huge. 43:28.260 --> 43:35.480 And you can do it yourself with some tools like mass scan and you can script it up yourself to scan but 43:39.380 --> 43:44.180 it's going to cost you a little bit of money on your VPS that you're using, etc. 43:44.260 --> 43:51.760 There's a service by Sam Erb and a DEF CON talk he did two years ago where he created a service called bufferover .run. 43:51.760 --> 43:57.420 It's an API and you can give it a domain and it will go out and he does this scanning every once in a while. 43:57.420 --> 44:06.700 And so you basically take that data, you parse it, and then you get a list of subdomains that were in the cloud ranges. 44:06.940 --> 44:13.540 Now, I think he runs his scans every two weeks and updates the service so it's not exactly live data. 44:14.220 --> 44:17.620 And this is one of the sources included in a mass. 44:17.700 --> 44:21.500 Again, I just wanted to outline the single-shot tool here. 44:21.500 --> 44:25.260 You probably could feel safe using a mass to get most of this data back. 44:25.260 --> 44:29.120 But yeah, you can use the service as well. 44:29.120 --> 44:32.900 So this method also finds some really good stuff. 44:32.900 --> 44:47.880 A lot of people are putting shadow IT infrastructure and registering websites on a credit card or just not paying attention or thinking that anybody will find their cloud infrastructure because they've never published those domains anywhere other than internally. 44:48.260 --> 44:54.300 And so you can find a lot of things for your target organization that are just sitting in the cloud, pretty much unsecured. 44:54.300 --> 45:04.960 In fact, it's been a big part of my research lately is scanning the cloud ranges and just finding wickedly undersecured stuff because people just don't think you'll ever find it. 45:04.960 --> 45:06.200 So this is a good method. 45:07.320 --> 45:07.900 All right. 45:07.900 --> 45:10.300 So we've scraped a lot of stuff to find subdomains. 45:10.300 --> 45:12.740 Now we want to brute force for subdomains. 45:13.040 --> 45:23.120 This is just a tried and true method that I'm sure every red team or pen tester has done before, usually with a tool like Fierce or one of the other tools like that in the past 10 years. 45:23.420 --> 45:31.860 You just try to give a word out of a dictionary and add it before your company.com and see if it resolves. 45:31.860 --> 45:33.680 It's a pretty simple method. 45:34.220 --> 45:47.280 Now, iteration in this field has come along in the last four years where a lot of the tools that we were using to do this were great, but they were using one DNS resolver, one DNS server to resolve. 45:47.440 --> 45:49.240 And it took a long time. 45:49.240 --> 45:58.160 I remember running Fierce as a pen tester 10 years ago, and it just took so long to finish a large dictionary to do subdomain brute forcing. 45:58.340 --> 46:09.480 And in the last four years, the idea of using multiple resolvers to speed up the process was pioneered by MassDNS. 46:09.480 --> 46:12.340 And MassDNS was the king for a little while. 46:12.340 --> 46:17.180 Now Amass also includes the idea of using multiple resolvers. 46:17.180 --> 46:26.180 So Amass uses eight DNS resolvers to parse DNS data when it brute forces by default. 46:26.180 --> 46:30.580 And you can add even more by using some flags. 46:30.640 --> 46:34.960 So here we're just running Amass over, you know, Twitch again. 46:34.960 --> 46:40.760 And we're doing brute forcing this time with the dash brute option. 46:40.760 --> 46:46.940 And then we're adding the source so we can see where some of these came from with the scraping part. 46:46.940 --> 46:51.140 And then you can specify the number of resolvers with dash RF. 46:51.140 --> 46:53.540 So RF means resolver file, I think. 46:53.660 --> 47:03.560 And if you basically give it a list of DNS resolvers or DNS servers that you trust, it will use more than one to do your brute forcing. 47:03.560 --> 47:05.900 So this speeds up your brute forcing significantly. 47:06.480 --> 47:11.720 I've also heard that another alternative to using Amass for this is ASDNS brute. 47:11.720 --> 47:15.420 I haven't used it yet, but I've heard it's also wicked fast and has multiple resolvers. 47:15.420 --> 47:16.940 So if you wanted an alternative. 47:18.380 --> 47:21.740 Shuffle DNS by the project discovery team also does this. 47:21.740 --> 47:24.740 I think it's a wrapper around mass DNS. 47:24.880 --> 47:26.920 It is a wrapper around mass DNS. 47:26.920 --> 47:37.340 And if you prefer to break out that type of brute force from Amass for whatever reason, whether it's stability or mass , you know, is taking too long for you. 47:37.340 --> 47:44.420 I haven't benchmarked the tools side by side, but a lot of people like Shuffle DNS as well to do subdomain brute forcing. 47:45.540 --> 47:51.300 So a subdomain brute force tool is only as good as the dictionary or word list you give it, right? 47:51.300 --> 47:55.240 Because it's just trying to resolve a whole bunch of words in front of your target, right ? 47:55.240 --> 47:56.400 Twitch.tv. 47:56.680 --> 48:03.340 And so in this mind , there's two kind of ways that you can approach what word list you feed to these tools. 48:03.340 --> 48:27.320 One is a tailored word list where you can build one based off of words that appear, brand names that appear, you know, you can build like a contextual based word list to your target, which Tom Nom Nom, who's a prolific bug hunter, awesome human, have a lot of respect for this guy and the tools he makes and the contributions he makes to the community. 48:27.380 --> 48:36.960 He did a talk at NahomCon, which is a conference a little while ago now, a couple months ago, where he did a whole talk on word list and how to generate contextual word list for your target. 48:36.960 --> 48:39.800 So I suggest going to watch that if you want to make some tailored word list. 48:39.800 --> 48:41.840 It's called Who, What, When, Where word list. 48:42.180 --> 48:45.280 And then you can also use a massive word list. 48:45.340 --> 48:49.960 And so over the years, we've had many, many tools that do DNS brute forcing. 48:49.960 --> 48:57.860 I went out and I took the word list for all of them and put them into one, and it's called all.txt. 48:57.860 --> 48:59.340 It's linked in this presentation. 48:59.340 --> 48:59.960 People use it. 48:59.960 --> 49:00.720 It's on my GitHub. 49:01.100 --> 49:06.020 And it basically sort and uniques all of the DNS names that you've seen forever. 49:06.020 --> 49:07.520 Now it has a lot of crap in it. 49:07.520 --> 49:08.220 It's true. 49:08.220 --> 49:09.240 It's a lot of lines. 49:09.240 --> 49:10.560 It's like a million or two million. 49:10.560 --> 49:11.080 I can't remember. 49:11.080 --> 49:11.820 It's a ton. 49:12.040 --> 49:17.620 With using the multiple resolvers, it actually doesn't take too long to do the brute forcing. 49:18.300 --> 49:27.640 I don't necessarily care if I'm sending crap to a DNS resolver like 0001.twitch.tv. 49:27.640 --> 49:38.140 Obviously that's usually not going to resolve, but the list has some gems in it that I just can't get past, you know, like its efficacy. 49:38.140 --> 49:43.020 So I use all.txt when I'm doing subdomain brute forcing, and I think it's pretty good. 49:44.460 --> 49:49.980 There are some newer school pieces of research to pull out subdomain enumeration, right? 49:50.220 --> 50:00.320 That file that I made parsed out all of the kind of subdomain brute forcing tools that have existed for the last, you know, 10 years or something like that and put them into one file. 50:00.320 --> 50:06.460 But there's some new research by the team at AssetNote, which is an attack surface mapping company. 50:06.500 --> 50:19.360 And they did some cool research using Google BigQuery to generate and discover subdomains that were used on, like, the Alexa top 10 or Alexa top 1,000 or 10,000 or something like that. 50:19.360 --> 50:20.400 Or Reddit. 50:20.400 --> 50:24.480 Or anytime someone referenced a URL on Reddit, what was the subdomain in that link? 50:24.480 --> 50:26.740 Or Stack Overflow. 50:26.740 --> 50:35.320 Anytime someone referenced a URL on Stack Overflow, they parsed these sites with BigQuery, and then they made these subdomain lists that you can use. 50:35.320 --> 50:37.720 So they called this project the Common Speak project. 50:37.720 --> 50:44.620 I've integrated Common Speak 1 into the list, all.txt, but Common Speak 2 came out a couple years ago. 50:44.620 --> 50:46.340 It's not in all.txt. 50:46.500 --> 50:49.180 It's got some other sites that they decided to target. 50:49.180 --> 50:54.760 I recommend checking it out for kind of newer school research on what subdomain names look like. 50:55.340 --> 50:57.780 Then there's this idea of alteration scanning. 50:57.780 --> 50:59.580 This is a type of brute forcing. 50:59.600 --> 51:06.160 So you have dev.company.com, but you could also have dev1, dev2, or dev-1, or dev.1. 51:06.600 --> 51:07.640 company.com. 51:07.640 --> 51:12.560 And so this technique was pioneered by Shubbs and Nathy when they wrote a tool called AltDNS. 51:12.560 --> 51:19.320 It's now been this permutation or alteration scanning, whatever you want to call it, has been built into a mass. 51:19.320 --> 51:25.720 So you can use a mass and it will try to find these naming conventions for you. 51:25.820 --> 51:32.200 And it will sometimes give you gold because people name stuff predictably in their subdomains. 51:33.640 --> 51:39.680 Some of the things I've done with permutation scanning is bypassing web application firewalls. 51:39.680 --> 51:54.820 So where I've had SQL injection on a main target and getting blocked by like a web application firewall, I've managed to find a permutation like dubdub2 and then managed to bypass the firewall because it wasn't applied to dubdub2. 51:55.020 --> 52:06.520 I've also managed to bypass things like Akamai by finding their origin via predictably named alterations like origin-sub or origin.sub domain to bypass filtering to go to the source. 52:06.520 --> 52:13.140 So these are some things you can use that origin scanning to do, as well as just find new surface to attack. 52:14.300 --> 52:18.300 All right, so now we're going to go into some other stuff that's related to Widescope Reon. 52:18.300 --> 52:27.300 We have seed domains, we've got a lot of seed domains, a lot of subdomains right now, and we should have a pretty good map of the infrastructure that belongs to this company. 52:28.040 --> 52:37.660 All right, so one that I didn't talk about, and this is new to this specific talk, that's why I called it 4.002, is favicon analysis. 52:37.800 --> 52:42.720 So there is this idea that every page in the tab at the top of your browser has a favicon, right? 52:42.720 --> 52:46.260 You see that little Tesla in the bottom left-hand corner of my Chrome tab? 52:46.260 --> 52:47.540 That's a favicon. 52:47.960 --> 52:58.860 Now, favicons are little images, and what you can do is you can take a hash of that favicon, and then you can search for the hash of the favicon on Shodan. 52:58.920 --> 53:09.560 And Shodan will then show you every other site that has that favicon, which will find you some gems, because people tend to reuse favicons on a lot of their sites of domains you might not have seen before. 53:09.800 --> 53:12.200 There are some newer tools to do this. 53:12.200 --> 53:14.620 I used to do this a lot in my recon testing. 53:14.640 --> 53:18.800 I then took it out for a little while, and I recently put it back in. 53:18.800 --> 53:24.100 So favicon analysis is kind of a fringe technique, but it's pretty cool. 53:24.100 --> 53:28.220 So there's a new tool called favfreakout by Devansh Batham. 53:28.500 --> 53:33.460 They're at Twitter at Asmodeus, and basically he does a couple things here. 53:33.460 --> 53:35.060 He parses from Shodan. 53:35.240 --> 53:42.820 He'll also look for hashes in different places, but he also uses the hash. 53:43.060 --> 53:50.660 He also has hashes for common infrastructure, like a scanner would, to look for different types of infrastructure. 53:50.660 --> 53:59.280 So on the bottom right-hand side, you can see that he has some fingerprints on different types of things, like Spring Boot pages or Big IP pages or Slack instances. 53:59.280 --> 54:00.720 They all have common favicons. 54:00.720 --> 54:14.360 And so when you scan a list of URLs or a set of domains on port 80 or 443 and you retrieve their favicon and their hash matches one of those hashes, you know, oh, shoot, I've stumbled upon a Slack instance for this organization. 54:14.500 --> 54:17.980 And you may not have found that doing anything else. 54:17.980 --> 54:24.080 You can also do the same thing if you scan the cloud and you correlate one of the previous techniques with this one. 54:24.080 --> 54:29.660 Let me find things that have my domain and the certificate data and have these favicon hashes associated to them. 54:29.660 --> 54:39.020 So this is a fringe kind of analysis technique to find even more kind of esoteric related sites. 54:40.780 --> 54:41.440 Okay. 54:41.440 --> 54:45.860 So then we're going to go back to a tried and true method is port scanning. 54:45.860 --> 54:47.180 We have a lot of domains now. 54:47.180 --> 54:50.820 We have a lot of C domains and subdomains and sites that we can work with. 54:50.960 --> 54:57.680 We want to port scan them because they may have services that are not 80 or 443 on these pieces of infrastructure. 54:57.680 --> 55:04.860 And so the, you know, the tried and true hacker education will tell you to use nmap here. 55:05.040 --> 55:07.620 But I use mass scan because I believe it's faster. 55:07.620 --> 55:12.740 It has a rewritten TCP IP stack, true multi threading . 55:12.740 --> 55:16.680 It's written in C, you know, directly calls, you know, a lot of stuff. 55:16.980 --> 55:28.560 So mass scan, in my experience, has been faster than using nmap, even with flags like min parallelism, which there was a huge debate on Twitter the other day of which is faster, min parallelism or using mass scan and people were like, whatever. 55:28.560 --> 55:43.220 When I run a scanning tool, a port scanning tool across, you know, 400,000 hosts, I have always found mass scan and it's syntax, it's advantages to beat out nmap. 55:43.280 --> 55:45.340 So that's just my personal experience. 55:45.340 --> 55:47.600 If you really like nmap to do this, you totally can. 55:47.600 --> 55:49.900 And this is strictly for finding ports. 55:49.900 --> 55:51.360 It's not to do service scanning, right? 55:51.360 --> 55:52.940 Nmap obviously wins in those areas. 55:52.940 --> 55:59.200 If you're going to do banner analysis, service scanning, script scanning, like that all pans down. 55:59.300 --> 56:01.000 Mass scan only does one thing, finds open ports. 56:01.000 --> 56:01.800 That's it. 56:02.000 --> 56:11.260 So what I do, also, if you want to learn how to use mass scan, Daniel Miesler, one of my best friends in the whole world, he wrote a study guide on mass scan and all of its syntax. 56:11.260 --> 56:13.600 It's one of the best ones I've seen, even better than man page. 56:13.600 --> 56:19.160 So go check that out on how to use mass scan and how to set the flags correctly and how to scan. 56:19.720 --> 56:26.840 So what I do is I'll take mass scan and then the problem about mass scan is it only scans IP addresses. 56:26.840 --> 56:28.580 It won't scan a domain name. 56:28.840 --> 56:33.840 So you can use a tool called dnmasscan, which will convert your domains that we have, right? 56:33.840 --> 56:41.460 We have subdomains and seed domains and it'll convert them into IP addresses and then scan them with mass scan and tell you all the open ports on each IP address. 56:41.460 --> 56:53.060 So what I do is I take mass scan and I scan that over all of my subdomains and then I feed that output, because I know it's open, to nmap to do service scanning. 56:53.340 --> 56:59.140 And service scanning will start to give me more information on these services that are open on those boxes. 56:59.140 --> 57:14.000 And then what I do is I do a quick default credential spray across everything that has certain services open, which is FTP, SMTP, SSH, Telnet, any type of SQL database, basic authorization and some other stuff. 57:14.140 --> 57:15.680 And so this is a tool that I use to do that. 57:15.680 --> 57:16.960 It's called Brute Spray. 57:17.120 --> 57:35.220 Brute Spray will take the output of your nmap scan, so you feed mass scan to nmap, nmap does the full service scan and outputs an XML file, and then you feed that XML file to Brute Spray to do a quick credential, default credential brute force against all of the services that allow you to do that. 57:35.220 --> 57:46.700 And so I've found some good wins doing this, logging straight into SQL databases and SSH and just some horrible stuff that people leave unsecured on the Internet, so on services. 57:47.800 --> 57:48.820 All right. 57:49.080 --> 57:53.340 So while I'm doing all of this stuff, right, we're in kind of the other category right now. 57:53.340 --> 58:01.480 So while I'm doing all of this analysis, and I have most of this automated in a giant ugly shell script that I use, and we'll talk about recon frameworks in a second. 58:01.480 --> 58:04.780 But while I'm doing all this, it takes a little while for these tools to run. 58:04.780 --> 58:16.520 The conglomeration of tools and techniques we've already talked about takes anywhere between 5 and 15 minutes, maybe a little bit more if I have a lot of subdomains or it's a big project. 58:16.520 --> 58:19.440 It'll take to run all of these tools and give me output. 58:19.440 --> 58:27.360 So while I'm doing this, I do a technique called GitHub dorking and this has found me many, many great things. 58:28.240 --> 58:32.800 And what you do is you basically just go to GitHub and type in your domain as a search operator. 58:32.800 --> 58:33.880 So twitch.tv. 58:33.880 --> 58:44.140 And you go to GitHub and you type in twitch.tv and then you just start browsing source code that has referenced twitch.tv or tesla.com or whatever, teslamotors.com. 58:44.460 --> 58:54.140 And you can find all kinds of sensitive data that former employees or current employees have accidentally put on GitHub. 58:54.140 --> 58:58.700 And so I have built a script just to build these search queries for me. 58:58.700 --> 58:59.800 These dorks, I call them. 58:59.800 --> 59:01.140 And it's right here in a gist. 59:01.140 --> 59:02.300 You can grab it. 59:02.440 --> 59:21.720 And it looks for things like my domain twitch.tv that I'm looking at right now and password or npmrcauth or dockerconfig or pemprivate for certificates or s3config or htpassword or credentials or bashrcprofiles or sshconfigs. 59:21.760 --> 59:25.320 And so it searches these key terms along with the domain. 59:25.500 --> 59:33.600 And if it comes up that someone has accidentally put this on GitHub, it's usually automatically a finding and it'll help me get into other systems. 59:33.600 --> 59:36.400 Let me give you an example here of something that happened in the real world. 59:36.900 --> 59:39.400 I found an admin page for a site . 59:39.580 --> 59:41.520 I couldn't do anything with it. 59:41.520 --> 59:44.300 I fuzzed the form. 59:44.300 --> 59:46.160 Great, didn't work, etc., etc. 59:46.300 --> 59:56.460 Then I found some dude who had basically posted a password not for that specific site, but it was related to my domain on GitHub for some other system. 59:56.640 --> 59:59.680 That system was internal, so I couldn't access it. 59:59.680 --> 01:00:04.660 But then I tried to use that password and username on this admin portal I had found that I had no luck before. 01:00:04.660 --> 01:00:08.380 Bam, got in, stole credit cards, millions of records of data. 01:00:08.380 --> 01:00:09.680 It was game over at that point. 01:00:09.680 --> 01:00:11.940 So this can help. 01:00:11.940 --> 01:00:19.520 There's a whole awesome talk here called GitHub Recon and Sensitive Data Exposure on Bugcrowd University by the gentleman. 01:00:19.580 --> 01:00:25.400 It's probably the best primer on doing this type of dorking to find sensitive stuff on GitHub that I've ever seen. 01:00:25.400 --> 01:00:27.220 I highly recommend you check that out. 01:00:27.220 --> 01:00:34.420 And then Gwendolyn also wrote a GitHub search tool that allows you to do this in the command line and not in the browser, which is... 01:00:40.580 --> 01:00:48.780 Okay, so now you have a bunch of subdomains, and I think I'm running close to time, so I'm going to try to hurry it up here. 01:00:49.540 --> 01:00:54.720 You have a bunch of subdomains here, and now we want to prioritize which ones we test, right? 01:00:54.720 --> 01:00:59.720 So what you can do is feed all of your subdomains to a tool that does screenshotting. 01:00:59.940 --> 01:01:02.620 And there are several tools that do this. 01:01:02.620 --> 01:01:04.440 Currently, I use Eyewitness. 01:01:04.440 --> 01:01:05.460 I was using Aquatone. 01:01:05.460 --> 01:01:07.820 I go back and forth, but there's four tools here. 01:01:07.860 --> 01:01:10.760 Aquatone, HTTP Screenshot, Eyewitness, and WitnessMe. 01:01:11.060 --> 01:01:22.340 All are tools that help you take screenshots, and then some do, like, additional analysis on your domains that you feed it, but normally I just use them for screenshotting. 01:01:22.340 --> 01:01:23.520 It doesn't matter which one you use. 01:01:23.520 --> 01:01:24.080 Try them out. 01:01:24.080 --> 01:01:24.880 See which one you like. 01:01:24.880 --> 01:01:26.000 I like Eyewitness. 01:01:26.100 --> 01:01:27.940 I feed it a list of domains. 01:01:28.140 --> 01:01:41.040 It will take those domains, visit them with the headless browser, take a screenshot of the page, and then I will just look at that folder and see, okay, which one of these looks like I want to prioritize first? 01:01:41.240 --> 01:01:43.280 Does it look like it redirects to the main site? 01:01:43.280 --> 01:01:46.000 Well, obviously, I'm not looking for the main site right now. 01:01:46.720 --> 01:01:48.840 It's some kind of back-end admin portal. 01:01:48.840 --> 01:01:50.320 Okay, I want to prioritize that. 01:01:50.320 --> 01:01:52.740 So screenshots can help you prioritize your work. 01:01:54.900 --> 01:02:00.400 When you're given a large list of subdomains like you have now, you can start to look for some vulnerabilities. 01:02:00.400 --> 01:02:01.960 One is subdomain takeover. 01:02:02.180 --> 01:02:13.340 There's a repo called CanITakeOverXYZ by edoverflow, and it gives you a list of all of these services and their fingerprints that might indicate that you can take over that subdomain. 01:02:13.340 --> 01:02:17.800 I'm not going to go crazy into subdomain takeover because it's short on time, obviously. 01:02:18.060 --> 01:02:21.180 But you can use a tool to check for subdomain takeovers. 01:02:21.180 --> 01:02:26.960 The best one right now is Nuclei and SubOver, I think, are my two favorites right now. 01:02:26.960 --> 01:02:33.600 SubOver was an independent tool, has been since ported over into the Nuclei framework that Project Discovery is making. 01:02:33.700 --> 01:02:35.220 But go check out Nuclei. 01:02:35.220 --> 01:02:42.720 It has the most subdomain takeover checks that I've seen for any tool. 01:02:42.920 --> 01:02:44.860 And I would check that out. 01:02:44.860 --> 01:02:50.860 And you can run it just across a list of large domains, and it'll tell you, hey, possible subdomain takeover at this address. 01:02:50.860 --> 01:02:55.860 So this is another wide scope tool that I use in the end part of my recon when I have all these subdomains. 01:02:55.860 --> 01:02:59.960 And you can find some volumes just straight off of using this. 01:03:01.240 --> 01:03:03.740 Okay, so we're going to blow through automation real quick. 01:03:03.920 --> 01:03:09.680 Okay, so when you have some of the tools and you have a long methodology like I have, you end up automating it, right? 01:03:09.680 --> 01:03:12.580 I write a horrible bash script to do my stuff. 01:03:13.500 --> 01:03:16.660 It works for me, but it could be better. 01:03:16.800 --> 01:03:19.460 Some of the tools that I use don't do certain things. 01:03:19.460 --> 01:03:20.720 They're not threaded. 01:03:21.260 --> 01:03:29.520 They don't take certain types of inputs, like list inputs or glob notation like Nmap does or range notation like Nmap does. 01:03:29.520 --> 01:03:32.580 And sometimes I need to feed a tool with another tool. 01:03:32.580 --> 01:03:34.540 And so I can't do that. 01:03:35.040 --> 01:03:41.800 So Michael Skelton, also known as Kadingo, wrote a tool called Interlace, which basically wraps around other tools and lets them do those things. 01:03:41.800 --> 01:03:42.580 It will thread them. 01:03:42.580 --> 01:03:44.940 It'll allow you to take different sources of input. 01:03:45.280 --> 01:03:46.880 It'll allow you to distribute tools. 01:03:46.880 --> 01:03:56.120 Here is an example of a blog written by Hackloop, which talks about basically threading Nikto, which Nikto doesn't support inherent threading. 01:03:56.120 --> 01:04:03.240 So you can use Interlace to basically glue together a lot of stuff that your other tools don't do. 01:04:03.700 --> 01:04:06.840 Any tool written by TomNomNom is awesome in automation, right? 01:04:07.840 --> 01:04:09.940 HttpProbe, WaybackURLs, Meg. 01:04:10.560 --> 01:04:14.140 TomNomNom has several talks out there talking about his tools and how they work. 01:04:14.260 --> 01:04:22.640 I use HttpProbe a lot for the glue between me finding subdomains and you feed it a lot of subdomains that you have found after your initial analysis. 01:04:22.640 --> 01:04:27.720 And then it tells you which ones have actual live listening web servers associated to them. 01:04:27.720 --> 01:04:29.180 So that's HttpProbe. 01:04:29.580 --> 01:04:32.360 WaybackURLs will find you URLs associated to any... 01:04:32.980 --> 01:04:37.000 old URLs associated to any site that you're currently looking at. 01:04:37.020 --> 01:04:39.580 Meg is like a directory brute forcer, but for many hosts. 01:04:39.580 --> 01:04:41.640 And I use this to find some stuff too. 01:04:41.640 --> 01:04:45.180 So all of TomNomNom's tools are just amazing. 01:04:45.800 --> 01:04:46.520 Alright, last part. 01:04:46.520 --> 01:04:47.300 I promise. 01:04:48.060 --> 01:04:50.960 Omar, are we okay to finish this last little bit? 01:04:52.280 --> 01:04:53.160 Absolutely. 01:04:53.880 --> 01:04:54.680 Awesome. 01:04:55.460 --> 01:04:58.200 Okay, so maybe recon is not your thing. 01:04:58.200 --> 01:05:03.000 I definitely run into hackers who are like, this is the most boring part of assessment to me. 01:05:03.000 --> 01:05:06.720 And then I run into other people who are like, yeah, recon is awesome. 01:05:06.720 --> 01:05:08.280 It's like my favorite part of the assessment. 01:05:08.280 --> 01:05:10.900 So it could be that recon is not really your thing. 01:05:10.900 --> 01:05:12.940 Finding all these sites is not your thing. 01:05:12.940 --> 01:05:14.760 Hacking the sites is more your thing. 01:05:14.820 --> 01:05:15.960 And that's cool. 01:05:15.960 --> 01:05:21.640 There are a lot of new tools out here these days that basically automate all this for you. 01:05:21.640 --> 01:05:23.340 And they're called recon frameworks. 01:05:23.820 --> 01:05:33.740 So if you've ever looked at a video game before, like Diablo, every once in a while, content creators will make these tiers of builds for Diablo or something like that. 01:05:33.740 --> 01:05:36.820 So I've put recon frameworks into a couple of tiers. 01:05:36.820 --> 01:05:39.160 C tier, B tier, A tier, and S tier. 01:05:39.160 --> 01:05:44.460 C tier are recon frameworks that are built around scripting up other tools in Bash or Python. 01:05:44.820 --> 01:05:45.700 They're step-based. 01:05:45.700 --> 01:05:47.080 They don't really have a workflow. 01:05:47.080 --> 01:05:50.220 They only have a few techniques and they're not really super extensible. 01:05:50.220 --> 01:05:51.500 But they work really well. 01:05:51.500 --> 01:05:53.560 Let me put out a big disclaimer here. 01:05:53.560 --> 01:05:58.520 My tools that I use personally on all bug hunts are C tier tools. 01:05:58.680 --> 01:06:01.440 And they work for what I want and they do. 01:06:01.440 --> 01:06:02.540 And they're great. 01:06:02.540 --> 01:06:05.440 So there's nothing bad to say about a C tier tool. 01:06:05.440 --> 01:06:09.560 A B tier tool, in my mind, and this is all very rough, right? 01:06:09.760 --> 01:06:14.960 These classifications are not in any way super serious. 01:06:14.960 --> 01:06:30.000 But a B tier tool, in my mind, has some of its own modules it's doing, some of its own sources, maybe has a GUI, maybe has some workflow in it where they're bringing data in from the end of the recon process back to the beginning if they find new stuff. 01:06:30.000 --> 01:06:32.600 It has more techniques than a C tier. 01:06:32.760 --> 01:06:39.220 But it still runs that point in time and it's still working on flat files to track the data for recon. 01:06:39.280 --> 01:06:43.660 A tier is maybe writing all of their own modules. 01:06:43.760 --> 01:06:49.380 These tools start to have some GUIs, some of them they run iteratively. 01:06:49.380 --> 01:06:53.180 So like Cron or something like that, they'll run at a schedule. 01:06:53.180 --> 01:06:58.940 And they start to manage things via database so you can compare scans and recon scans over time. 01:06:59.100 --> 01:07:02.980 And then S tier are kind of the highest level of what's out right now. 01:07:02.980 --> 01:07:04.520 They write a lot of their own modules. 01:07:04.520 --> 01:07:05.300 They have a GUI. 01:07:05.300 --> 01:07:06.260 They run iteratively. 01:07:06.260 --> 01:07:08.180 They manage all their data via database. 01:07:08.180 --> 01:07:11.480 They scale across multiple boxes to make the scanning faster. 01:07:11.480 --> 01:07:17.980 They send alerts back to the user via email and text and Slack and whatever when they find new things. 01:07:17.980 --> 01:07:20.880 They have some novel techniques that not a lot of other people are doing. 01:07:20.880 --> 01:07:22.820 So these are how I classify some of the frameworks. 01:07:22.820 --> 01:07:27.580 I'm going to show some of them to you now and you can pick one that works for you if you're not into this recon stuff. 01:07:28.140 --> 01:07:28.900 Warning. 01:07:29.000 --> 01:07:30.380 I had to put this in here. 01:07:30.380 --> 01:07:31.920 I don't want to hurt anyone's feelings. 01:07:31.920 --> 01:07:34.760 I am scaling some tools and giving them grades. 01:07:34.760 --> 01:07:35.640 It's not serious. 01:07:35.640 --> 01:07:36.880 It's my rough experience. 01:07:36.880 --> 01:07:37.940 My gut feel. 01:07:38.380 --> 01:07:43.560 All these tools are wonderful and I respect the authors of them so darn much of just putting code out there. 01:07:43.560 --> 01:07:48.620 I can't say enough about the people that open source their code and even C tier. 01:07:48.620 --> 01:07:51.560 Like I said, my tool is a C tier tool and it works for me. 01:07:51.560 --> 01:07:54.640 So this is just subjective and based off my own experience. 01:07:55.300 --> 01:07:55.760 Okay. 01:07:55.760 --> 01:08:02.140 So some C tier tools that you could look into that are wrapping around a lot of other existing tools. 01:08:02.140 --> 01:08:07.500 One of the ones called ultimate recon here I like is listed here. 01:08:07.500 --> 01:08:17.980 And the reason I like ultimate recon is because it's using the new nuclei scanner that's out by project discovery and it implements finding subdomains using a couple different tools. 01:08:17.980 --> 01:08:19.620 It will then take all of those. 01:08:19.620 --> 01:08:23.720 It will port scan them and then it will run nuclei templates on them. 01:08:23.720 --> 01:08:25.840 So it's pretty good. 01:08:25.840 --> 01:08:26.660 I like it. 01:08:26.660 --> 01:08:32.560 All of these different C tier tools they're all choosing their own tools to wrap around that they like. 01:08:32.560 --> 01:08:34.440 So there's no best one really. 01:08:34.440 --> 01:08:38.500 But I tend to like ultimate recon in this one. 01:08:38.500 --> 01:08:41.280 There's also other good ones in here too. 01:08:41.280 --> 01:08:42.780 So check these out. 01:08:43.240 --> 01:08:44.560 B tier frameworks. 01:08:44.560 --> 01:08:47.440 This is lazy recon by Captain Milo. 01:08:47.440 --> 01:08:49.500 And here you can see his workflow. 01:08:49.500 --> 01:08:51.780 Very similar to what we talked about in this presentation. 01:08:51.780 --> 01:08:54.380 Amass and sub finder combine those two. 01:08:54.380 --> 01:08:56.040 You get a final subdomain count. 01:08:56.040 --> 01:09:05.020 Then you start to do some vulnerability analysis with subjack and core scanner to find some vulnerabilities related around subdomain takeover and cores vulnerabilities. 01:09:05.020 --> 01:09:07.540 Then you start brute forcing. 01:09:07.560 --> 01:09:09.640 Then you start port scanning. 01:09:09.640 --> 01:09:11.460 Then you do some screenshots. 01:09:11.620 --> 01:09:14.940 And then you have a final output. 01:09:15.000 --> 01:09:18.240 So a lot of stuff we talked about today is implemented inside of this workflow. 01:09:18.240 --> 01:09:22.700 So lazy recon is what I like to point out inside of the B tier here. 01:09:24.180 --> 01:09:26.080 A tier, there's a couple. 01:09:26.200 --> 01:09:28.300 They all teeter on being S tier, right? 01:09:28.700 --> 01:09:34.060 I wonder if I should even really have like an S tier and an A tier. 01:09:34.440 --> 01:09:37.860 Usually the thing that's separating some of these is GUI from the other ones. 01:09:37.860 --> 01:09:40.540 So the one I have referenced here on this slide is called find domain. 01:09:40.540 --> 01:09:42.600 And find domain does pretty much everything. 01:09:42.600 --> 01:09:50.700 It does subdomain discovery and scraping and brute force and all of that via amass and sub finder and asset finder and everything. 01:09:50.960 --> 01:09:52.920 And then it stores it in a database. 01:09:52.920 --> 01:09:54.180 It does iterative scanning. 01:09:54.180 --> 01:09:55.140 It'll text you. 01:09:55.140 --> 01:10:01.140 If it finds new domains from its last scan, it'll email you, which you see in the bottom left hand corner, which is great. 01:10:01.140 --> 01:10:02.920 The only thing it lacks is a GUI. 01:10:02.980 --> 01:10:04.520 It's a command line based tool. 01:10:04.520 --> 01:10:07.900 So it kind of teeters on doing a lot of the stuff we want to do. 01:10:08.040 --> 01:10:12.180 And in the enterprise version, it does port scanning and screenshots. 01:10:12.640 --> 01:10:17.540 And that version is a for pay version, but it's affordable to bounty hunters. 01:10:17.540 --> 01:10:20.160 Some of those are really good subscriptions to pay for. 01:10:20.160 --> 01:10:21.880 So find domain, pretty cool. 01:10:22.960 --> 01:10:24.420 Here's our S tier framework. 01:10:24.420 --> 01:10:32.040 Some of them you'll never get to use because they're commercial products, but I include them here to give people ideas as to what they should be working towards in their frameworks. 01:10:32.220 --> 01:10:37.540 Intrigue.io is an asset discovery tool written by Jonathan Cran, who's a friend of mine. 01:10:38.800 --> 01:10:45.580 He builds Intrigue and he has a GUI for it and it's a SaaS platform and businesses buy it to map their attack surface. 01:10:45.580 --> 01:10:48.100 But it's doing the same techniques that we've talked about today. 01:10:48.740 --> 01:11:04.100 The cool thing about Intrigue is even if you don't pay for the subscription of the SaaS version that he's selling to businesses, he open sources almost all of the code for Intrigue.io so you can run your own hosted version or at least see what he's doing and his kind of analysis. 01:11:04.100 --> 01:11:05.760 So Intrigue is pretty cool. 01:11:05.840 --> 01:11:11.400 Asset note is another one that's a B2B type play for recon. 01:11:11.400 --> 01:11:14.260 It's a team of former bug hunters who made this awesome platform. 01:11:14.260 --> 01:11:21.340 This is probably the gold standard of asset management or attack surface management tools. 01:11:21.340 --> 01:11:23.160 You can see that it's pretty. 01:11:23.160 --> 01:11:26.920 It breaks things up by assets that need attention. 01:11:26.960 --> 01:11:28.420 It's got a lot of graphing. 01:11:28.680 --> 01:11:32.180 It does a lot of custom vulnerability checking. 01:11:32.420 --> 01:11:39.060 So asset note is probably, it's S tier, but it's a high price tag so you're probably not going to use it as a bug hunter or a red teamer. 01:11:39.060 --> 01:11:43.780 But it is kind of the gold standard that a lot of the tool makers should probably work towards. 01:11:45.000 --> 01:11:50.900 Spiderfoot is another one that does a lot of OSINT type domain correlation information. 01:11:50.900 --> 01:11:54.500 It does some of the techniques we've talked about in this presentation. 01:11:54.500 --> 01:11:55.760 They're adding more all the time. 01:11:55.760 --> 01:12:03.020 I know the author is really strictly interested in adding more bug bounty focused features in it. 01:12:03.020 --> 01:12:06.720 And it's got a great graphing library and monitoring and everything. 01:12:06.720 --> 01:12:08.160 So Spiderfoot's pretty cool. 01:12:08.320 --> 01:12:12.360 The unreleased project discovery, we've talked about project discovery a couple times in the presentation. 01:12:12.360 --> 01:12:15.500 That team, their unreleased framework looks to be a killer. 01:12:15.500 --> 01:12:16.560 Looks to be awesome. 01:12:16.560 --> 01:12:18.680 I'm really excited about it when it comes out. 01:12:19.120 --> 01:12:27.580 So yeah, you can see here a dashboard of a project, all your discovered domains and seeds, what the scan process is. 01:12:27.580 --> 01:12:36.000 And then if you dig into any of those, you get screenshots, port scans, technology identification of the site, its activity, its changes over time. 01:12:36.000 --> 01:12:39.840 So the discovery framework when it comes out will be pretty cool. 01:12:40.840 --> 01:12:44.680 Jails is a vulnerability scanner written by Jesse, JJJ and team. 01:12:44.760 --> 01:12:46.640 It is also really good. 01:12:46.640 --> 01:12:50.800 It is a GUI flow that wraps around some command line stuff. 01:12:50.800 --> 01:12:57.880 It does vulnerability scanning, which, you know, if you're working with Nessus or, you know, not a lot of red teams are scanning with Nessus . 01:12:57.880 --> 01:13:06.060 But if you want to look for some very pointed stuff, you can use Jails and look at their kind of CVE checks that they use. 01:13:06.060 --> 01:13:11.760 And you can see signatures there for the Zoho management page and fuel CMS RCE. 01:13:11.760 --> 01:13:14.600 So they're looking for some cool volume Jails. 01:13:14.600 --> 01:13:15.940 I really like it. 01:13:16.180 --> 01:13:18.640 Osmedias is awesome as well. 01:13:18.640 --> 01:13:21.660 It's very similar to some of the previous pages we've seen. 01:13:21.840 --> 01:13:24.380 It'll take all of your domains. 01:13:24.380 --> 01:13:27.200 It will do... it'll take all your subdomains. 01:13:27.200 --> 01:13:28.560 It will resolve them to IP. 01:13:28.560 --> 01:13:29.940 It will do port scanning on them. 01:13:29.940 --> 01:13:31.760 It'll tell you the technologies. 01:13:32.720 --> 01:13:34.420 It will do screenshots. 01:13:34.420 --> 01:13:35.580 It'll do all kinds of stuff. 01:13:35.580 --> 01:13:39.320 So Osmedias is also pretty easy to stand up and really good. 01:13:39.360 --> 01:13:41.960 Huntersuite.io, also same kind of idea. 01:13:41.960 --> 01:13:47.880 You can see technology parsing, services, domain discovery, etc. 01:13:49.100 --> 01:13:52.600 Bounty.offensive.ai, same type of deal, right? 01:13:52.600 --> 01:13:59.800 Includes some vulnerability scanning, technology fingerprinting, subdomain finding, and then attack graphing. 01:14:00.060 --> 01:14:03.640 Reengine is the one I used last week or a week before. 01:14:03.640 --> 01:14:06.200 Reengine is pretty sick. 01:14:06.260 --> 01:14:08.800 You can have projects in it that are separate. 01:14:08.800 --> 01:14:10.720 It'll do the subdomain scanning. 01:14:10.720 --> 01:14:18.040 It basically takes the methodology that I outlined in this presentation and turns it into a tool, a hosted tool. 01:14:18.400 --> 01:14:21.920 And they're making some improvements every week. 01:14:21.920 --> 01:14:29.560 So lots of potential on Reengine, which I highly suggest checking out. 01:14:30.240 --> 01:14:32.820 And then Scout PDP on the SecApps team. 01:14:32.820 --> 01:14:35.720 They've been working on tools for well over a decade. 01:14:35.820 --> 01:14:41.360 And they have built out Scout, which is a paid offering, but it's affordable enough for bounty hunters. 01:14:41.360 --> 01:14:42.300 Same idea, right? 01:14:42.400 --> 01:14:46.800 Projects, subdomain enumeration, related domains, reverse DNS. 01:14:46.800 --> 01:14:48.300 It supports screenshotting. 01:14:48.300 --> 01:14:49.120 It's awesome. 01:14:50.540 --> 01:14:51.960 Awesome tool. 01:14:52.280 --> 01:14:56.000 And then lastly, Nuclei. 01:14:56.000 --> 01:15:02.420 I talked about it very briefly, but anyone who's not using Nuclei in their bounty scanning right now is at a disadvantage. 01:15:02.440 --> 01:15:06.800 They're coming out with some epic templates for CVE identification. 01:15:08.080 --> 01:15:11.660 And you can basically build your own via a YAML file. 01:15:11.660 --> 01:15:17.920 And it's a scanner that'll go out and scan all your domains for vulnerabilities, subdomain takeovers, all kinds of stuff. 01:15:17.920 --> 01:15:21.400 So I really have been using Nuclei to great success lately. 01:15:22.320 --> 01:15:23.280 And that's it. 01:15:23.280 --> 01:15:24.100 That's all I got. 01:15:24.100 --> 01:15:25.700 So that's the Bug Hunters methodology. 01:15:25.960 --> 01:15:27.680 Thanks for giving me the time today. 01:15:27.680 --> 01:15:28.280 I really appreciate it. 01:15:28.280 --> 01:15:29.680 Sorry I went over a little bit. 01:15:29.680 --> 01:15:30.600 No worries. 01:15:30.600 --> 01:15:34.120 Thank you so much for the presentation and amazing support. 01:15:34.780 --> 01:15:40.120 Once again, from the bottom of my heart, thank you for supporting DEF CON and the DEF CON Red Team Village. 01:15:40.120 --> 01:15:47.400 And a reminder for everyone here, please take a look at all the talks and activities that we have in our website. 01:15:47.400 --> 01:15:52.400 We have the CTF, of course, the Cyber Raid contest as well. 01:15:52.680 --> 01:15:55.440 Tons and tons of activities throughout this weekend. 01:15:55.700 --> 01:15:59.460 The link should be in the description at the bottom of your stream. 01:15:59.460 --> 01:15:59.980 And all that stuff.