Hey, everyone. My name is Jason, and thanks for joining us this morning.
Super big thanks to the Red Team Village for having me and all the other speakers on
and the staff for facilitating everything that's going on with the conference for DEF CON
for hosting the Village. A big place in my heart for DEF CON.
So today I'm going to talk a little bit about my methodology for recon.
I've been doing this talk called the Bug Hunters Methodology for five years now,
and I basically update it every year, and I talk about new tools and techniques
in different spaces of the Bug Bounty scene and Red Teaming.
So today we're going to go over the recon stuff.
Now, the Bug Hunters Methodology, TBHM, is a presentation, like I said,
I've been running for a long time, and basically it's too big for any one conference slot,
so I split it into two sections. One is recon, which we're going over today,
and the other is application analysis, which is on-site hacking and stuff like that.
So today we're going to do recon, and I'm working on this year's version
of application analysis, haven't finished it yet. I'm a giant slacker, sorry.
So yeah, today we're going to go over recon.
A little bit about me. I'm a husband, father, hacker, gamer, and I stream sometimes.
These are my socials, so if you want to reach out and ask a question
after the talk or something like that, just hit me up.
I was a bug hunter, and then I worked at Bug Crowd for many years,
and now I'm the head of security at Ubisoft,
and so I lead the security team there. Awesome team, great people.
And I'm also a gamer at heart, so I play some video games.
These are my kids, Arcadia and Avalon in the top picture.
I took them to DEF CON for the first year. Last year I did DEF CON Kids or Roots.
It was amazing, they had a great time. And then that's my son Arlen,
him and I being pirates in the bottom right-hand corner.
It's a little bit about me. I've been doing pen testing for about...
a long time, I'm old.
I have a lot of related experience to this talk.
The first thing I want to talk about is today we're going to go over
recon methodology, and we do recon in red teaming engagements,
and we do recon in bug bounties primarily. And you can do it
in some wide-scope pen tests as well. And one of the first things
I talk about is when you're doing an assessment, some sort of assessment,
whether it's one of those three, you have to have a way
that jives with you to record your work.
Otherwise, you're just throwing darts and forgetting what you're doing.
And this is counterproductive to basically keeping organized.
So the first thing that I talk about here is how I keep notes.
And I use a tool called X-Mind, which is a mind-mapping tool.
And basically, I create nodes, and I create methodologies
as checklists inside these nodes, and I track all my domains
and which ones I've worked on by color codes.
So this is something I do. I've seen other people do
a lot of project tracking for assessments in just Notepad or Vim,
or there's some specialized pen test tools. If you're really fancy
and you have a subscription, you can probably do it in something fancy
like Dratus or something like that. So there's a lot of tools,
but it's just really important to track your work when you're doing recon.
Recon is the art of finding as many assets as possible
related to a target, and it can get pretty data-heavy and dense
as you do it. So having a way that works for you to track your data
is the first thing that's going to guide you to success.
So this is an example of how I track my data
when I'm on a target. So Tesla, Tesla Motors,
has a pretty open-scope bounty on Bug Crowd.
And this is how I would start off my project with Tesla.
So in the middle, I have a node in this mind-mapping program
I use called xMind. And then on the left-hand side,
I have their autonomous system numbers, their ASNs.
I have any acquisitions they've made. I have some other notes,
LinkedIn discovery, and a link to their reverse-who-is information.
And on the right-hand side, I'm starting to build out
their root domains, or their seeds. Roots or seeds
are terms for things like something.com,
or teslamotors.com, or tesla.com, or SolarCity.
And so I've started to build out their seeds
on the right-hand side. So you can see Tesla Motors,
Tesla, SolarCity, etc. And as I do this,
I'm going to start collecting a lot of data. As I do recon
on Tesla, I'm going to collect a lot of data. So that turns into
this, which on the left-hand side, you can see
for tesla.com, that seed,
I've enumerated all their subdomains and all of the live links.
So these are all the live links for tesla.com,
for their subdomains. So things like pages.tesla.com,
shop.tesla.com, etc.
And then you can drill down into any one of these.
And so if I drill into www.tesla.com, then I have
my methodology notes for that single site, which
inside of that node, I keep
all of the questions I ask myself when I'm basically
looking to hack a website. Like, does the site have multiple user roles?
How does it reference users? How does it handle special characters?
What are dynamic parameters? Does it have an API component?
What kind of errors am I seeing? Does it have
file uploads? Have I done JavaScript analysis
for paths? Have I done content discovery?
All these questions that you're normally going to
keep in your methodology, I apply
these all to each one of these subnodes. And then I
color code my work based on where I am.
And this is one of the most important things, I think,
is that everything that is not filled in hasn't been
worked on yet, so most of this hasn't been worked on yet.
But everything in orange means I'm currently working on something,
and everything in green, which is there's nothing green on here, I've already done.
And so just the way I can do this,
and then I can add checkmarks and other stuff in XMind
to these mind maps helps me track where I am, so I can easily
put down a project and pick it back up if I want.
Okay, so today's mission, we're going to talk about
wide recon. This is the art of finding as
many assets related to a target as possible.
So you're a red teamer, or you're a bug bounty hunter, and you have
a scope of X company.
And one of the things you need to do
is identify all the websites that they own.
Because the more websites you identify, or the more
infrastructure you identify, the better your chances
of getting in or finding a bounty, right? And this is a
core component in both of those skill sets.
And I break down what we call recon into
a couple of domains. So finding first the in-scope
domains via the program or your project
brief, you're finding acquisitions
for the company or the target,
doing ASN enumeration, doing reverse whois,
doing a whole bunch of subdomain enumeration, and then doing port analysis.
And then we're going to go into some related topics
like some vulnerability scanning
and some automation
information related to recon, because you'll probably do it
in this phase of your workflow.
Okay, so you've decided you're going to do an assessment
or you've been handed a red team assessment or you've been handed a bug bounty
that's wide scope, right? That's where we are right now. So the first thing
we need to do is parse the program page.
So I'm going to use a bounty here, a couple bounties to illustrate
where you would start. So here is
the brief page for the aforementioned Tesla
for BugCraft. So in their brief, they have
star.tesla.com, star.tesla.cn, Tesla Motors
and Tesla Services. And then they also have this
catch-all sentence here that says any host verified
or owned by Tesla Motors that you can find
is also included in scope. So this is what we consider a wide scope
bug bounty. There's lots of stuff that you can do with this.
So your first four seed domains are already listed here.
Tesla.services, TeslaMotors.com, TeslaCN, and Tesla.com.
So we're going to focus on these four and add them to our list.
And then anything else we can find that is Tesla's is also fair game.
So that's great. If you look at a platform like HackerOne,
they also have a pretty wide scope bounty
and probably the most prolific bounty available on the internet
which is Verizon Media. And this is like the granddaddy,
the biggest scope program I think
I know of in the bug bounty scene. So Verizon Media
has gobbled up so many brands and they have
so much infrastructure related to these brands and sites
and subdomains. It's giant. So a lot of hackers
make their whole career hacking on
Verizon Media, which is awesome. And they're a great team
and they support the bug bounty community. They also have this
catch-all kind of phrase in their scope that says if you've found
a vulnerability that affects an asset belonging to them but it's not
in scope, report it to the program. And so it's a
really advanced team to work with. You couldn't even fit
all of the domains that you have for this program on one page, which I
didn't even try. So you will parse your seed domains
and your in-scope domains from the program page
to start off. And this will give you a starting place to
work from. The next thing that I do
when I'm scoping out a company is I want to find
all their acquisitions. And there's a couple reasons I want to do this.
I want to basically see
if any of the infrastructure for those acquired companies has
not been taken off the line or any of the subdomains
or websites or integrations that were too hard to port
over to new infrastructure are still online.
This gives me more and more infrastructure and sites to
hack. So to find the acquisitions, I use
a tool called Crunchbase. And Crunchbase is a business
intelligence portal. Basically, you add any business name
to this search box on Crunchbase and you'll be able to
look at that company and find out what
are their employees? Are there investor rounds?
What acquisitions have they had? What are their socials, etc.?
So here I've used the example of Twitch.com. We're streaming
on Twitch today, which is awesome. And Twitch
as an organization here has an entry. And if I click
on that, I get a lot of information about Twitch.
And I can see that under their acquisitions tab,
they have acquired four companies
in the last eight years, Revlo,
Clipmind, Curse, and Goodgame. And then also,
they were acquired themselves on the left-hand side by Amazon.
So this gives me a good idea of
what they used to be, where I can also
look for some infrastructure if it's a recent acquisition,
like Revlo, for instance. If I click on Revlo
in this page, it'll take me to Revlo's page, which will give me their main domain,
which is Revlo.com. So I might put that in scope of
a wide scope assessment or a Red Team assessment.
Acquisitions are important to keep track of. Now, the one thing that you
want to make sure here is that on some of those brief pages that we
looked at in the last couple pages, some of these are
explicitly out of scope. So they'll say in the brief,
especially for Tesla, SolarCity is not in
scope of their wide scope bounty.
So even though we found that as an acquisition of Twitch
when I looked on this page, when I've done it many times,
it's explicitly out of scope. So remember to refer to your project page
or understand what is okay and what is not okay to hack
or do recon on.
So after I find the acquisitions, which has given me
seed domains like Revlo.com and new
seed domains other than Tesla Motors and Tesla and stuff
like that, or any of these top-level seed domains,
what I want to do is I want to find this company's autonomous
system numbers. So every company that gets
large enough ends up having a collection of networks
basically applied an AS number to them.
And an AS number, autonomous system number, is just a collection
of your known IP ranges for your networks, for your
infrastructure. And so you can find someone's
autonomous system number in many ways. There's
many search engines to do this kind of
lookup, but I use one called Hurricane Electric
or bgp.he.net. I use this
when I'm doing manual searching because it has a freeform
text box at the top and I can just search the keyword
Twitch. And here you can see I found Twitch's
two autonomous system numbers, AS46489
and AS397153.
And then their associated IP ranges
and IPv4 ranges and IPv6 ranges.
So this is a good representation of the infrastructure
or the IP space that they own. And I can
assume that it belongs to... I can guarantee
that it belongs to them. I don't have to worry about
scope really here because I know that all this is
owned by them. So if they have that catch-all phrase in their
bounty project or you've got the go-ahead in your red team engagement
to open scope, then you know this is your target.
One thing that this doesn't represent is their cloud assets.
So this is all owned IP space.
It doesn't represent things like AWS and Azure and GCP
ranges. So you won't find stuff that they host
in those environments inside of these ranges.
You just want to be cognizant of that. So here I've searched
Twitch. I got their ASNs. With every
tool or method here in my methodology,
I like to give you a manual way to do it
because context is important and getting used to the idea
is important. And I like to give you an automated way to do it in case you're going to script
up your own recon framework, which is all the fad these days
and I have one of my own and everything. So there are
two tools you can use to parse ASNs
and IP ranges from companies that
exist on the command line if you're going to script this stuff up yourself.
So one is called Metabigor.
I'm still not sure if I'm saying that name of the tool right. By
JesseJJJ. And this one utilizes
the site that we just saw, bgp.he.net
and another site called ASN Lookup. And it will
grab the information from those websites, scrape it off and give you
your IP ranges for an organization. The other one
is called ASN Lookup by Yasin. Same idea.
It uses a different data source called the MaxMind database.
It'll pull off their IPv4 ranges from an ASN
and you can give these both tools
a term instead of an AS number or anything like that.
So they will pull it off based on that keyword
that we're looking for. In this case, Tesla or something like that.
Now, one thing to know about these tools is they are
looking off your keyword here, which we put on the command line,
Tesla. And there are multiple companies with
Tesla in the name. There's like Tesla Research Lab or something
like that in Sweden or something like that.
Just be sure that when you get these ranges and you start seeing
these sites, you can identify, oh no, I might have gotten
one that's actually not Tesla Motors.
Both these tools, I verified, pull up the right
data for Tesla. But just be cognizant
of, since you're using a keyword here to search, that you don't
get other IP ranges that are not your targets.
You'll have to verify somewhat manually there.
Okay, so we've gotten some IP space from the ASNs.
We've gotten some seed domains. We want to continue gathering
as many seed domains as possible.
We have the AS number, and for here
we're going to use Twitch as an example again.
Their autonomous system number is 46489.
And here what we want to do is we're going to
get our first introduction to Amass.
And Amass is a framework for
domain intelligence, I would call it, I guess.
And it is written by Jeff Foley. There is a workshop you should absolutely
sign up for. He's going to do a full workshop on Amass
and how to use it. Jeff is as associated
to the Red Team Village, so super cool. And here we're going
to use Amass and feed it our autonomous
system number, 46489. And what it's going to do
is it's going to go out to all of the IP ranges that
are represented by that autonomous system number.
And it's going to scan the certificates for all HTTPS
sites. And then it's going to parse those certificates and tell us,
hey, here are the seed domains I found
that are owned by them. And here you can see that we found some
seed domains that we didn't know about before. Justin.tv, which is the
company Twitch used to be. TTVNW.net,
which is, I think, associated to
Twitch's streaming protocol. Twitch.tv,
obviously, which is their main site. TwitchCon, which is their conference.
And SocialCam, which I have no idea what is.
And all the links for these tools and the methods
are posted in the slides, so you can click
through when the slides get released and go grab the tools
and play with them on your own. So here we've already expanded
our assessment
a lot, right? We have a whole bunch of IP space,
we have a whole bunch of top-level domain seed domains,
and we have the seed domains already given to us by our
brief page. Okay, so the next method is
reverse whois. And what reverse whois does is it will take the
whois entry of... you can search your domain
in Twitch.tv here. And you can do this
with a lot of sites. I use huoxi.com because
it's pretty cheap and you can get a free API
key for so many uses. And
you basically supply it your domain and it will give
you the whois data and then you can
use that whois data to correlate who else has
registered stuff. So an example here is we gave this site
huoxi.com, twitch.tv. And then it said
that over the course of the years, there have been several
companies' names related to the whois data and also
registered emails. And so here you
can see that the company, as of March 26,
2015, was justin.tv
and the email in the domain was domainmaster
at justin.tv. And then later in 2015, it changed
to twitchinteractive. And if you click the button right next
to those entries, you can see there's 20 other domains still
registered to justin.tv, which could be interesting to us. And there's
575 domains registered
under twitchinteractive. And so we can start pulling back more
data, more subdomains, more root domains from
these links when we click on them. Now,
one thing to know here is that whois data is
registered data. And so a lot of these companies
will park domains to combat
phishing and to hold stuff for
marketing campaigns and stuff like that. So they might not be live sites.
They might just be parsed. So this is a medium fidelity
type of technique here, but I have found some
really good stuff using reverse whois and the registrant
data of these sites. There is
a tool to do this type of analysis. It's called domlink,
written by Vincent Yu via security on Twitter. And
it uses that exact site, hoaxy.com.
And you can basically give it a domain
and it will find every other associated domain by both
registrant email and organization name or company
name. And it will do it on the command line and return it to you
in a script. And it's recursive. So a really cool tool
here. You can use this to automate some of that reverse whois
lookup. It's called domlink. The tool is called domlink.
So the next method that we have now
that we've maybe gotten some more seeds and subdomains from
reverse whois and we're building out our list of things that we can
hack is we want to look at the ad and analytics
relationships of our target site. We want to see what
other sites are using the same ad and analytics codes
as our main site. And this will give us
a good understanding of maybe what are their most
popular domains and maybe what are some of their less popular domains
that are still using the codes and we didn't know about before.
So every page usually embeds a
Google Analytics code or a New Relic code
or some kind of ad or analytics code
in the main page. So here if we use a site called
builtwith.com, we can search twitch.tv
and twitch.tv will give us a list under the
relationship profile tab of all of the keys that are on
twitch.tv. And then you can
click on those and see what other sites are using the same
ad and analytics codes and maybe we didn't know
about those before. So here we can... it's hard to see on the bottom
but we have a couple entries that we didn't know about before.
manvsgame.tv, twitc.tv.
We knew about twitchcon.com
from the previous method but we're starting to build up more
domains that we can see that are related to
our target. So ad and
analytics relationship profiles is a pretty powerful technique.
I have used it for success a couple
times. You can also do this directly from Firefox or Chrome.
They have an extension for this site called built with and
you can just install it. You make a free account
and then you visit your site in the browser, click the
Chrome extension button and it will give you the relationship information
right in your browser and you can start clicking around there.
So pretty easy to institute while you're
doing your recon in a browser.
So you can also use a tool to do this
in the command line. So Malik, after
my first iteration of this talk, quickly scripted up an
awesome tool to do this. It's called getrelationship.py
and you just pass it your target here.
Here he's targeted Uber.com which also has a bounty
program and passed it to this Python script and it will pull out on the command
line all of the related domains. And you'll have to
manually kind of verify these but the ones I would look at first
are the ones that actually have Uber in the
domain somewhere like
daft.uber or driveuber.co.nz
or things like that. So you'll have to
sift through this but you can do it on the command line now too.
Then you can just use some general
Google hacking or Google-fu to try to find
associated sites to your target. You can use
the copyright text, the terms of service, and the
privacy policy and you can just copy those from the bottom of any of the pages
and here you can see Twitch interactive ink would be something
I'd Google for. And then you can see any other page that is
hosting that text at the bottom of their page
which could be something that you didn't know about before
in your recon so far. So you can just do
this manually and you can use
other search operators like in URL Twitch or you can
just look for these policies. One tip here is
that you don't just want to target a certain
year. Like here I have 2019 but you also want to check 2018
and 2017. In fact, the more stuff you find
that goes back farther in age, the more
likely it is to be vulnerable and less likely to have a lot of security
assessment associated to it. So probably more vulnerable for your
Red Team engagement or your bug bounty.
Alright, the next
method that we're going to use to find some seed domains or
related infrastructure is Shodan. I have to admit I'm not a Shodan
ninja but I know a lot of Red Teamers are and I actually love Shodan
and it's just a failing of mine. I need to get better at it. But there's
a ton of search operators we could use here on our domain twitch.tv
and what Shodan is is it's the site that is a
site that hosts information from an infrastructure
based spider. And a spider goes to every site on the
internet and basically captures
its HTTP responses, its
technology stack, its certificate data
and cross links it
so you can click on any of these pieces of information and find out
other things it's seen related and what organizations it's
related to and what common tech that they're
using, what web servers, what JavaScript
framers, you could do anything. So Shodan is a really powerful tool
and you can query our
domain here and try to find
related infrastructure. For instance, in one of the entries when I just
didn't even use a great operator here, I just searched my domain twitch.tv
I ended up seeing in the SSL certificate data
of the search there was
twitch.amazon.eu and now I need
to ask myself a question like is that in scope, right? Like that's not
verbatim a Twitch domain but in a
red team assessment if that's related to Twitch infrastructure, well
maybe it's a target that I can go after.
So Shodan can give you a lot of good information.
Okay, so we found seed domains, stuff to start
with. Now what we're going to do is we're going to try to find subdomains
for those seed domains. So these are things
like www is a subdomain
but also admin.tesla.com or
forums.tesla.com or something like that
or forums.twitch.tv or whatever. We're going to try to find
subdomains. And so this idea of starting
with seeds and then moving to subs really
is going to multiply. The more seeds you find
the more subdomain enumeration you can do. The more subdomains
you find is related to how many
sites you find. The more sites you find, the more successful you'll be
inside of your red team engagement or bug bounty
program. So for subdomain enumeration
I use three-ish
different methods to get subdomain information. One is
LinkedIn JavaScript discovery. Another one is
subdomain scraping techniques and then subdomain
brute force. And then there's some auxiliary stuff I use as well.
So we're going to run into those right now. And this is the bulk of what a lot of tools
are doing right now. And so
you'll see a lot of favorites in these slides. So the first area
we're going to use to find subdomain data is
or the first technique is linked discovery and JavaScript
discovery. So linked discovery, what is it?
It's basically using a web spider to
visit a site like twitch.tv
land on it and spider all the
HTML links. Pretty simple. And any of those that
come back with a subdomain
we just add to our list of in-scope targets.
And so this is pretty self-explanatory. It happens
naturally when you're using tools like burp suite and you're spidering
a site. But I'm going to walk you through how I do it.
I'm going to use burp 1.7 because I like the UI better but it
works as well with the crawler or the scanner
in burp 2.0. So use whatever
you want. And then I'm going to give you some other methods to do it. So the first thing
we do is we load up burp suite which is on the right
here. It's a side-by-side to our site. And
we just visit our site through burp suite the proxy.
And burp suite the proxy will capture all this
data. We've only visited one page. If you visit
twitch.tv ever, you see that it has
cross-linked and it does request a lot
of stuff because it's a streaming media site and so there's a lot
of stuff going in the background when you go to twitch.tv.
So you can see all that data on the right-hand side. This is everything that's
either seen been linked on this page
or it's actually been requested. Actually been requested is the
stuff in black, seen is the stuff that's in gray.
And so what you do is you visit your page
and then you have to set up a rule, some type of rule.
And you can do this by going into the target tab and
the scope tab and then you click this box that says
use advanced scope control. And what I do is I enter
in just a keyword here in advanced scope control
which I just say twitch, right? So I want to see
in burp anything that has twitch in the URL
at all, no matter where it is, what has that word twitch. And you could also
add some more here because we've seen in our previous analysis
TWT is also part of their
domain naming nomenclature, right?
And so you can add a couple of these scope rules
and then you go back to your sitemap and you
click on the ribbon bar up at top or the filter
bar and then you say show only
in scope items. So now this will just show
in our sitemap all the twitch.tv links, right? So we already
have a good list of subdomains here, right?
Sentinel.twitchservice.net
API.twitch.tv. So we have a lot
of stuff with twitch in the name here. We know that most of this is probably
related to twitch. This gives us exponential
areas to hack, right? And this is pretty good.
But if we select all these and then we spider them
using burp spider, it will then go to all of
them and find their links and find subdomains
references in their pages. And we can do this recursively
until there's nothing left to find. So here I'm
going to select all of these and then I'm going to spider
them. And now you can see, since I had them selected,
everything in orange on the right-hand side was stuff I knew about before I
spidered. And then after I spidered with burp,
everything in white is new stuff. And you can see I found
a combination of a lot of stuff here. I found
subdomains for twitch.tv and I've also found some new
root and seed domains like twitchapp.net
and twitchservice.net and
x-twitch.tv. So this is a
hybrid technique. It can find you new seed domains or roots
and subdomains. And so now I have all of this domain
data. I can now select all of these and spider them again
and spider those pages for links. And eventually you can do
this until you get kind of spider fatigue.
And eventually you'll have a large sitemap buildup
of targets, which is awesome. Now, how
do you get this data out of Burp Suite? There's
not a great way to do this. Pro has
a thing called Engagement Tools.
And in Engagement Tools, you can do this thing called
Analyze Target. And in Analyze Target, you can create
an HTML report of all the targets
in your site tree that are selected. So that's what I do.
I select everything in the site tree. I go Engagement
Tools, Analyze Target, Generate HTML Report. And then
in the HTML report, I have a parsable list of
the domains that it's seen at the beginning of that HTML report. And then I
take that and I put it in a text file for later analysis.
And then I dump it into my mind map as well.
Okay. So the whole
linked discovery thing counts on Burp
Spider, right? A lot of times I hadn't seen
before the last couple of years, great options to do this
in the command line because it would require
a lot of custom coding, bash scripting
or Python to create my own spider and do that same
process. Well, now there does exist some tools that are
command line spiders with bug hunters and red teamers
in mind. There's two. There's one called Ghost Spider written by Jesse
JJJ. And it also designates
the types of things it's parsing when it's visiting a URL.
So it'll give you...
Okay. Hopefully you guys can hear me. Headphone problems.
Going to plug it in.
Okay. Great, great, great. I was getting beeping in my ears.
So maybe it's just running batteries. My bad. Okay.
So you have two spiders here that you can use in the command line.
Ghost Spider and Hackcrawler by HackLuke.
And both these are awesome. They both have some functionality
that will parse out the types of things
you're getting, like JavaScript files,
subdomains, URL endpoints.
Some of them will give you, I think, parameter names and stuff like that.
So you could institute the whole process
of link discovery by scripting up
Ghost Spider or Hackcrawler if you wanted.
I still use Burp, but these are invaluable tools
to have at your disposal, just like a crawler that can do
some analysis. So keep them bookmarked
for when you might need them.
Okay. So the next place we're going to get subdomain information
is by analyzing
some JavaScript. And one of the tools I like here,
just because it has this kind of little added benefit that I think
that I haven't seen many places before, but maybe some other tools
are starting to implement it now, is a tool called
Subdomainizer by Neeraj Edwards. And what it'll do
is it'll take a JavaScript file, you have to point it to a JavaScript
file, and it will parse out all the cloud services,
the subdomains, and it'll do this little extra
thing where it uses the Shannon entropy formula
or algorithm to identify things that
look like API keys hard-coded in JavaScript,
which is already kind of a vulnerability if you find a private API
key or credential hard-coded in JavaScript,
which I know sounds crazy to a lot of people. Like, you would just find
a hard-coded private API key, but
this happens all the time. Like, all the time it happens.
And so this is
what I like to call a forward-thinking type of thing, is using
an algorithm like this to identify keys. Sometimes
it's a little noisy, but a lot of times it finds you
good stuff. So I like this tool to point at all the
JavaScript files I found already on
some of these sites. If you're just looking for subdomain
information, there's another tool called Subscraper by Celian
Collins, which has recursion built into it, which
can do this method as well, but it doesn't do the API key part.
So point this at JavaScript files you find
on your site, and you can get back a whole bunch of
subdomains and cloud
services that the site might use or the target might use.
All right. So that's
link discovery and JavaScript analysis for
subdomains. Now we're going to get into the big meats of
what a lot of people do is subdomain scraping.
Now the idea of subdomain scraping is
going out to these websites on the
internet that have search boxes, basically.
There's all these projects on the internet, search
engines, security websites, certificate
projects and stuff like that. They all do different stuff.
So Robtex gives you infrastructure
information. The Wayback Machine houses
information about domains and their responses
years past. Everyone's probably used the Wayback
Machine. The certificate sources down
here, some of them are certificate projects to provide certificate
transparency. Search engines, obviously you know what search
engines are. And then there's a whole bunch of security sites
that do different things, like give you a
rating on how malicious a URL or a site might be.
The common thing that these all have is either have an API or a
search box where you can put in a domain. And they will search
that domain and they will tell you anything they've seen
related to that domain. And if you put in a domain like
Tesla.com, the information that comes back is
parsable and you could possibly find out that they know about
some subdomains of Tesla.com that we don't know about.
So this is the process of subdomain scraping.
We're going to go to all of these sources, all of these sources and ask
them, hey, what do you know about Tesla.com or
Twitch.tv? Do you know of any subdomains that maybe I don't know of?
Okay, cool. Let's do that. Now,
there are many more sources that are on this page, right?
These are only a subset and new sources
to parse are coming out as fast as new websites are coming out.
So the tools have to keep up adding new
novel sources to find subdomain data or URL
data as they
mature. Now, this is the example of using
a search engine like Google to do it, right? So here, what we're doing
is we're searching with the search operators
site.twitch.tv and we're saying, I already know about
www.twitch.tv, so we're saying
minus www.twitch.tv. And we already know about
watch.twitch.tv, so minus watch.twitch.tv
and minus dev.twitch.tv because we already know about that. So now,
Google's only showing us things that are not those
and so we can do this process, keep on
minusing out domains until Google doesn't know anything
more about subdomains related to twitch.tv.
And that way, we can basically get a full
inventory of what Google knows and what Google knows about subdomains
for twitch.tv. So this is an example of doing it manually
with Google. But luckily, you don't have to do this yourself. There's
tools out there that will do this type of analysis, this scraping
for you. The first
one is Amass by Jeff Foley and the Amass team.
There's a whole team behind this. And there's two tools I use
here, Amass and Subfinder, but we're going to look at Amass first.
Amass has probably the most
sources for any subdomain
scraping tool in existence in its enum section.
Amass is a framework for domain information,
but the one that actually pulls this method's
results, subdomain scraping, is called Amass Enum.
And here on the right-hand side, you can see we gave it twitch.tv
and it went out to a whole bunch of sources it's had
in its databases and it reached out to those web pages
using curl or whatever. I don't know if they
use headless chromium. I don't know exactly what they're using.
And then they parse those pages for subdomains for twitch.tv
and they give us a list of all the subdomains for twitch.tv.
And if we do this on all of our
seed domains that we've gathered, we have now started to exponentially
build out the amount of sites that we have to attack.
So this tool is invaluable.
Amass is becoming the go-to
tool for all subdomain enumeration,
especially this thing, subdomain
scraping. The other tool I used here is
oh, also, this is still Amass. What Amass does at the end
of a run, not only does it give you all the subdomains,
but it also gives you this great table, which I feel like is underutilized
a little bit. And this table tells you, okay, I discovered
439 subdomains
and here is where they were inside
of these ASNs. And you can see that most
of twitches were in
Amazon's ranges, which makes sense because they were acquired by Amazon,
but you can see some of them were in other
ASNs, right? And I've had instances where,
not on Twitch, but on other projects where
there was a whole ASN related to the company
I didn't know about that I found out because
Amass built this table. They're like, hey, 70
of these subdomains we discovered appeared in this ASN
or these IP ranges. And then I look at that and I'm like,
oh, I wasn't initially looking at those ranges.
Weird. Let's go back to the beginning of my workflow
and start enumerating those ranges. So this table is
super cool. It also gives you information on their third
party kind of tools they're using.
Here you can see that
Twitch is using Bitly and SendGrid
and some other stuff. So it also gives you information there in Fastly.
Okay, the other tool I use for
subdomain scraping is Subfinder.
Originally written by Iceman and Michael Skelton, I think
now navigated to the ProjectDiscovery.io team, which is
a group of bug hunters releasing some stellar tools.
And they also have multiple sources,
extensible output, also a really great tool.
Both of these tools are really good.
Each one of them have different
sources. So what I end up doing is in my automation, I run both of them
and then I cat the output together
and sort it and unique it from both the tools.
So they both have a couple of different sources
and a lot of the same sources. So that's what I do for my stuff
is I just use both of them.
Okay, so this one is somewhat new
and it has been integrated into
Amass a little bit.
But
I've had inconsistency with run
between this standalone tool, GitHub
subdomains.py and the output of Amass.
So I still use this independently inside of my automation
when I'm looking for subdomains.
So this tool is called GitHub subdomains.py
and it's still scraping and what this is doing is it's going out to GitHub
as a source. And basically you're providing GitHub
and if you've ever been on GitHub, they have the search box up top
and you're saying search for twitch.tv
and then anything that comes back with twitch.tv
as a piece of source code, it will parse up the subdomains
from that piece of source code.
Now this was written by Gwendolyn Kuik
and I still don't know if I'm saying Gwendolyn's name
right and that's okay.
And Gwendolyn wrote a whole suite of tools for GitHub
enumeration. How to find secret keys in
GitHub related to an organization. How to pull email
addresses out. He has a wonderful blog
and it's linked here in this slide with a whole bunch of GitHub
tools. Now GitHub subdomains is just looking for subdomains.
The thing about using this tool
is that the GitHub API, not the tool
is just kind of instable and
doesn't give you like somewhat returns rate limited
results sometimes. So what I do inside of my
automation and this has given me subdomains I haven't
found anywhere else using this method is I run GitHub
search or GitHub subdomains.py
like five times and I sleep in between each run
so that I give it a little while for the rate limiting to die down
and then I give it a big sleep at the end before I run
another one and then that seems
and then I cat all those results together and unique them
and that seems to give me more consistency
when parsing the API. And that has nothing to do with the tool that Gwendo
wrote. It's all about what GitHub does with their
search API. So this is an awesome tool to
find lesser known subdomains and I found some
great stuff using this tool.
Okay, the next one is
show sub go by incogbite
and this one will parse
using a number of search operators with your
API key. It will parse showdown.
So I include this in my script as well.
Again, this is one of those tools that some of
the subdomain frameworks like Amass
and Subfinder might do for you.
I find using the standalone tool for
some reason just works better for me. So I don't know exactly why that is
but I've run them side by side and gotten different outputs so I still use this
verbatim. And it's also a fast
running script. It's not like I'm waiting for it to complete for 5-10
minutes or something like that. It usually runs in a minute or two. So it doesn't
skin off my nose if I have to wait an extra couple of minutes to
ensure that I get coverage out of showdown. So
show sub go will take a domain and your
showdown API key and parse showdown using some search operators
and give you back all the subdomains related to here
twitch.tv.
Okay, so the last... well, not the last, but one of the
other methods to look for subdomain enumeration
or subdomain scraping is the cloud ranges, right? So we talked about this
a little bit earlier is that we have all this IP space
and now we've started to identify subdomains and you'll notice that some
of those subdomains are resolving to
infrastructure or sites in the cloud. Now
there's this idea that not a lot of people have been doing is just
going out recently or not recently but
people have been doing it for a while but it hasn't been much public is just
scanning the entire ranges for AWS, GCP
and Azure for SSL sites, right? Anything that
responds to... anything that responds
to, you know, or responds
to a connection on 443. And you basically
scan the cloud ranges and you scan
it by IP address and then when it responds
it'll give you its SSL certificate and then you parse
the organization name or the domain name out of the
certificate data. And then you look at that data and you say
does it match my target twitch.tv or does it have
the keyword twitch in it? And you're like, cool, I found some stuff
that these people have put in the cloud that wasn't part of their
ASN that I probably didn't know about before. Now
that's a tremendous amount of scanning, right? Those
ranges are huge. And you can do it
yourself with some tools like mass scan and you
can script it up yourself to scan but
it's going to cost you a little bit of money
on your VPS that you're using, etc. There's a
service by Sam Erb and a DEF CON talk he did
two years ago where he created a service called
bufferover.run. It's an API and you can give it a domain
and it will go out and he does this scanning
every once in a while. And so you basically
take that data, you parse it, and then you get
a list of subdomains that were in
the cloud ranges. Now, I think he runs his
scans every two weeks and updates the service so
it's not exactly live data. And
this is one of the sources included in a mass.
Again, I just wanted to outline the
single-shot tool here. You probably could feel safe using a mass
to get most of this data back. But
yeah, you can use the service as well. So
this method also finds some really good
stuff. A lot of people are putting shadow IT infrastructure
and registering websites on a credit card or
just not paying attention or thinking that anybody
will find their cloud infrastructure because they've never published
those domains anywhere other than internally.
And so you can find a lot of things for your
target organization that are just sitting in the cloud, pretty much unsecured.
In fact, it's been a big part of my research
lately is scanning the cloud ranges and
just finding wickedly undersecured
stuff because people just don't think you'll ever find it. So this is a good method.
All right. So we've scraped
a lot of stuff to find subdomains. Now we want to brute force
for subdomains. This is just a tried
and true method that I'm sure every red team or pen tester has
done before, usually with a tool like Fierce or
one of the other tools like that in the past 10 years.
You just try to give a word out of a dictionary
and add it before your company.com
and see if it resolves. It's a pretty simple
method. Now, iteration in this field has
come along in the last four years where
a lot of the tools that we were using to do this
were great, but they were using one DNS resolver,
one DNS server to resolve. And
it took a long time. I remember running Fierce as a pen tester
10 years ago, and it just
took so long to finish a large dictionary to do
subdomain brute forcing. And in the last
four years,
the idea of using multiple
resolvers to speed up the process was pioneered by
MassDNS. And MassDNS was the king for a little
while. Now Amass also includes
the idea of using multiple resolvers. So Amass
uses eight DNS resolvers
to parse DNS data when it brute
forces by default. And you can add even more
by using some flags. So
here we're just running Amass over, you know,
Twitch again. And we're doing brute forcing this time
with the dash brute
option. And then we're adding the source so we can
see where some of these came from with the scraping
part. And then you can specify the number of resolvers
with dash RF. So RF means resolver
file, I think. And if you basically
give it a list of DNS resolvers or DNS servers
that you trust, it will use more than one to do
your brute forcing. So this speeds up your brute
forcing significantly. I've also heard that
another alternative to using Amass for this is
ASDNS brute. I haven't used it yet, but I've heard it's
also wicked fast and has multiple resolvers. So if you wanted an
alternative. Shuffle DNS by the project
discovery team also does this. I think it's a
wrapper around mass DNS. It is a
wrapper around mass DNS. And if you prefer to
break out that type of brute force from Amass
for whatever reason, whether it's stability or
mass, you know, is taking too long for you.
I haven't benchmarked the tools side by side, but a
lot of people like Shuffle DNS as well to do
subdomain brute forcing. So a
subdomain brute force tool is only as good as the
dictionary or word list you give it, right? Because it's just
trying to resolve a whole bunch of words in front of your
target, right? Twitch.tv. And so
in this mind, there's two kind of ways that you can
approach what word list you feed to these tools. One
is a tailored word list where you can
build one based off of words that appear,
brand names that appear, you know, you can build
like a contextual based word list to your
target, which Tom Nom Nom, who's a prolific
bug hunter, awesome human, have a lot of respect
for this guy and the tools he makes and the
contributions he makes to the community. He did a talk
at NahomCon, which is a conference a little while
ago now, a couple months ago, where he did a whole talk
on word list and how to generate contextual word list for your target.
So I suggest going to watch that if you want to make some tailored word list. It's called
Who, What, When, Where word list. And then
you can also use a massive word list. And so over
the years, we've had many, many tools that do
DNS brute forcing. I went out and I took the
word list for all of them and put them into
one, and it's called all.txt. It's
linked in this presentation. People use it. It's on my GitHub.
And it basically sort and uniques all of the DNS
names that you've seen forever. Now it has a lot of
crap in it. It's true. It's a lot of lines. It's like a million
or two million. I can't remember. It's a ton. With
using the multiple resolvers, it actually doesn't
take too long to do the brute forcing. I
don't necessarily care if I'm sending crap to
a DNS resolver like
0001.twitch.tv. Obviously
that's usually not going to resolve, but
the list has some gems in it that I
just can't get past, you know, like
its efficacy. So I use all.txt
when I'm doing subdomain brute forcing, and I think it's pretty good.
There are some
newer school pieces of research to pull out
subdomain enumeration, right? That file
that I made parsed out all of the kind of
subdomain brute forcing tools that have existed for the last, you know,
10 years or something like that and put them into one file.
But there's some new research by the team at AssetNote,
which is an attack surface mapping company.
And they did some cool research using
Google BigQuery to generate and discover
subdomains that were used on, like, the Alexa
top 10 or Alexa top 1,000 or 10,000
or something like that. Or Reddit. Or anytime
someone referenced a URL on Reddit, what was the subdomain in that link?
Or Stack Overflow. Anytime someone
referenced a URL on Stack Overflow, they parsed these sites
with BigQuery, and then they made these
subdomain lists that you can use. So they called this project
the Common Speak project. I've integrated Common Speak 1
into the list, all.txt,
but Common Speak 2 came out a couple years ago. It's not in
all.txt. It's got some other sites that they decided
to target. I recommend checking it out for kind of newer school
research on what subdomain names look like.
Then there's this idea of alteration scanning.
This is a type of brute forcing. So you have
dev.company.com, but you could also have dev1,
dev2, or dev-1, or dev.1.
company.com. And so this technique was
pioneered by Shubbs and Nathy when they wrote a tool called AltDNS.
It's now been this permutation
or alteration scanning, whatever you want to call it, has been built
into a mass. So you can use a mass
and it will try to find these naming conventions
for you. And it will sometimes
give you gold because people name stuff predictably
in their subdomains.
Some of the things I've done with permutation scanning
is bypassing web application firewalls.
So where I've had SQL injection on a main
target and getting blocked by like a web application
firewall, I've managed to find a permutation
like dubdub2 and then managed to bypass the
firewall because it wasn't applied to dubdub2.
I've also managed to bypass things like
Akamai by finding their origin via predictably
named alterations like origin-sub or
origin.sub domain to bypass filtering to go to the source.
So these are some things you can use that origin scanning to
do, as well as just find new surface to
attack.
All right, so now we're going to go into some other stuff that's related to
Widescope Reon. We have seed domains, we've got a lot of seed
domains, a lot of subdomains right now, and we should have a pretty good
map of the infrastructure that belongs to this company.
All right, so one that I didn't talk about,
and this is new to this specific talk,
that's why I called it 4.002,
is favicon analysis. So there is this idea
that every page in the tab at the top of your browser
has a favicon, right? You see that little Tesla in the bottom left-hand corner
of my Chrome tab? That's a favicon.
Now, favicons are little
images, and what you can do is you can take a hash
of that favicon, and then you can search for the hash
of the favicon on Shodan. And Shodan
will then show you every other site that has that favicon,
which will find you some gems, because people tend to reuse
favicons on a lot of their sites of domains you might not have seen
before. There are some newer tools to do this.
I used to do this a lot in my recon testing.
I then took it out for a little while, and I recently
put it back in. So favicon analysis is kind of
a fringe technique, but it's pretty
cool. So there's a new tool called favfreakout by
Devansh Batham. They're at Twitter at
Asmodeus, and basically he does a couple
things here. He parses from Shodan.
He'll also look for
hashes in different places, but he also uses
the hash. He also has hashes for
common infrastructure, like a scanner would,
to look for different types of infrastructure. So on the bottom
right-hand side, you can see that he has some
fingerprints on different types of things, like Spring Boot pages
or Big IP pages or Slack instances. They all have common
favicons. And so when you scan a list of
URLs or a set of domains on
port 80 or 443 and you retrieve their favicon and their
hash matches one of those hashes, you know, oh, shoot, I've stumbled
upon a Slack instance for this organization. And you
may not have found that doing anything else.
You can also do the same thing if you scan the cloud
and you correlate one of the previous techniques with this one.
Let me find things that have my domain and the certificate data and have
these favicon hashes associated to them. So this is
a fringe kind of analysis technique to find
even more kind of esoteric
related sites.
Okay. So then we're
going to go back to a tried and true method
is port scanning. We have a lot of domains now. We have a lot of C domains
and subdomains and sites that we can work with.
We want to port scan them because they may have services that are
not 80 or 443 on these pieces of
infrastructure. And so the, you know,
the tried and true hacker education
will tell you to use nmap here. But I use
mass scan because I believe it's faster. It has a
rewritten TCP IP stack, true multi
threading. It's written in C, you know, directly
calls, you know, a lot of stuff. So mass scan, in my
experience, has been faster than using nmap, even with flags
like min parallelism, which there was a huge debate on Twitter
the other day of which is faster, min parallelism or using
mass scan and people were like, whatever. When I run
a scanning tool, a port scanning tool across,
you know, 400,000 hosts,
I have always found mass scan and it's
syntax, it's advantages
to beat out nmap. So that's just my personal
experience. If you really like nmap to do this, you totally can.
And this is strictly for finding ports. It's not to do
service scanning, right? Nmap obviously wins in those areas. If you're
going to do banner analysis, service scanning, script
scanning, like that all pans down. Mass scan
only does one thing, finds open ports. That's it. So what I
do, also, if you want to learn how to use mass scan,
Daniel Miesler, one of my best friends in the whole world,
he wrote a study guide on mass scan and all of its syntax.
It's one of the best ones I've seen, even better than man page. So go check
that out on how to use mass scan and how to set the flags
correctly and how to scan. So
what I do is I'll take mass scan and then the problem
about mass scan is it only scans IP
addresses. It won't scan a domain name. So
you can use a tool called dnmasscan, which will convert
your domains that we have, right? We have subdomains and
seed domains and it'll convert them into IP addresses and then scan them with mass
scan and tell you all the open ports on each IP address.
So what I do is I take mass scan
and I scan that over all of my subdomains
and then I feed that output, because I know it's
open, to nmap to do service scanning. And
service scanning will start to give me more information
on these services that are open on those boxes. And then what
I do is I do a quick default credential spray
across everything that has certain services open,
which is FTP, SMTP, SSH, Telnet,
any type of SQL database, basic
authorization and some other stuff. And so this is a tool
that I use to do that. It's called Brute Spray. Brute Spray will
take the output of your nmap scan, so you feed mass
scan to nmap, nmap does the full service
scan and outputs an XML file, and then you feed that
XML file to Brute Spray to do a quick
credential, default credential brute force against all
of the services that allow you to do that. And so I've found
some good wins doing this, logging
straight into SQL databases and SSH
and just some horrible stuff that people leave
unsecured on the Internet, so on services.
All right. So while I'm doing
all of this stuff, right, we're in kind of the other category right now. So while
I'm doing all of this analysis, and I have most of this automated
in a giant ugly shell script that I use, and we'll talk about
recon frameworks in a second. But while I'm doing all this,
it takes a little while for these tools to run. The conglomeration
of tools and techniques we've already talked about takes
anywhere between 5 and 15 minutes,
maybe a little bit more if I have a lot of subdomains
or it's a big project. It'll take to run all of these
tools and give me output. So while I'm doing this,
I do a technique called GitHub dorking
and this has found me many, many great things.
And what you do is
you basically just go to GitHub and type in your domain as a search operator.
So twitch.tv. And you go to GitHub and you type in
twitch.tv and then you just start browsing source code
that has referenced twitch.tv or tesla.com
or whatever, teslamotors.com.
And you can find all kinds of sensitive data that former
employees or current employees
have accidentally put on GitHub.
And so I have built a script
just to build these search queries for me. These dorks, I call them.
And it's right here in a gist. You can grab it.
And it looks for things like my domain twitch.tv that I'm looking at
right now and password or
npmrcauth or dockerconfig
or pemprivate for certificates or
s3config or htpassword or
credentials or bashrcprofiles or sshconfigs.
And so it searches these key terms
along with the domain. And if it comes up
that someone has accidentally
put this on GitHub, it's usually automatically a finding and it'll help me
get into other systems. Let me give you an example here of something that happened in the real
world. I found an admin page for a
site. I couldn't do anything with it.
I fuzzed the form. Great,
didn't work, etc., etc. Then I found
some dude who had basically
posted a password not for that specific site, but it was
related to my domain on GitHub for some other system.
That system was internal, so I couldn't access it.
But then I tried to use that password and username on this admin
portal I had found that I had no luck before. Bam, got in,
stole credit cards, millions of records of data.
It was game over at that point. So this can
help. There's a whole awesome talk here
called GitHub Recon and Sensitive Data Exposure on
Bugcrowd University by the gentleman. It's probably the best
primer on doing this type of dorking to find sensitive
stuff on GitHub that I've ever seen. I highly recommend
you check that out. And then Gwendolyn also wrote a
GitHub search tool that allows you to do this in the command line and
not in the browser, which is...
Okay, so now you have a bunch
of subdomains, and I think I'm running close to time, so I'm going to try
to hurry it up here. You have a bunch
of subdomains here, and now we want to prioritize
which ones we test, right? So what you can do is feed
all of your subdomains to a tool that does screenshotting.
And there are several tools that do
this. Currently, I use Eyewitness. I was using Aquatone.
I go back and forth, but there's four tools here. Aquatone,
HTTP Screenshot, Eyewitness, and WitnessMe.
All are tools that help you take
screenshots, and then some do, like, additional analysis
on your domains that you feed it, but normally
I just use them for screenshotting. It doesn't matter which
one you use. Try them out. See which one you like. I like Eyewitness.
I feed it a list of domains. It will
take those domains, visit them with the headless browser,
take a screenshot of the page, and then I will just look at that
folder and see, okay, which one
of these looks like I want to prioritize first?
Does it look like it redirects to the main site? Well, obviously,
I'm not looking for the main site right now.
It's some kind of back-end admin portal. Okay, I want to prioritize
that. So screenshots can help you prioritize your work.
When you're
given a large list of subdomains like you have now, you can start
to look for some vulnerabilities. One is subdomain takeover.
There's a repo called CanITakeOverXYZ by
edoverflow, and it gives you a list of all of these
services and their fingerprints that
might indicate that you can take over that subdomain. I'm not going to go
crazy into subdomain takeover because it's short on time,
obviously. But you can use a tool to check for
subdomain takeovers. The best one right now is Nuclei
and SubOver, I think, are my two favorites
right now. SubOver was an independent tool,
has been since ported over into the Nuclei framework that
Project Discovery is making. But go check out Nuclei.
It has the most
subdomain takeover checks that I've
seen for any tool. And I would
check that out. And you can run it just across a list of large
domains, and it'll tell you, hey, possible subdomain takeover
at this address. So this is another wide scope tool that I
use in the end part of my recon when I have all these subdomains.
And you can find some volumes just straight off of
using this. Okay, so
we're going to blow through automation real quick. Okay, so when you have
some of the tools and you have a long methodology
like I have, you end up automating it, right? I write a horrible
bash script to do my stuff. It
works for me, but it could be better.
Some of the tools that I use don't do certain things. They're not
threaded. They don't take certain types of
inputs, like list inputs or
glob notation like Nmap does or range notation like
Nmap does. And sometimes I need to feed a tool with another
tool. And so I can't do that.
So Michael Skelton, also known as Kadingo, wrote a tool
called Interlace, which basically wraps around other tools and lets
them do those things. It will thread them. It'll allow you to take different
sources of input. It'll allow you to distribute tools.
Here is an example of a blog written by Hackloop,
which talks about basically threading Nikto, which
Nikto doesn't support inherent threading.
So you can use Interlace
to basically glue together a lot of stuff
that your other tools don't do. Any tool written by
TomNomNom is awesome in automation, right?
HttpProbe, WaybackURLs, Meg.
TomNomNom has several talks out there talking about his tools and how they work.
I use HttpProbe a lot for the glue
between me finding subdomains and you feed
it a lot of subdomains that you have found after your initial analysis.
And then it tells you which ones have actual live listening
web servers associated to them. So that's HttpProbe.
WaybackURLs will find you URLs associated to any...
old URLs associated to any
site that you're currently looking at. Meg is like a directory brute
forcer, but for many hosts. And I use this to find some stuff
too. So all of TomNomNom's tools are just
amazing. Alright, last part. I promise.
Omar, are we okay to finish this last little bit?
Absolutely.
Awesome. Okay, so maybe
recon is not your thing. I definitely run into hackers who are like, this is
the most boring part of assessment to me.
And then I run into other people who are like, yeah, recon
is awesome. It's like my favorite part of the assessment. So it could
be that recon is not really your thing. Finding all these sites
is not your thing. Hacking the sites is more your thing.
And that's cool. There are a lot of new tools out here
these days that basically automate all this
for you. And they're called recon frameworks.
So if you've ever looked at a video game before,
like Diablo, every once in a while,
content creators will make these tiers of builds for Diablo
or something like that. So I've put recon frameworks into a couple
of tiers. C tier, B tier, A tier, and S tier.
C tier are recon frameworks that are built around
scripting up other tools in Bash or Python.
They're step-based. They don't really have a workflow. They only have a few
techniques and they're not really super extensible. But they work
really well. Let me put out a big disclaimer here.
My tools that I use personally on all bug hunts
are C tier tools. And they work
for what I want and they do. And they're great.
So there's nothing bad to say about a C tier tool.
A B tier tool, in my mind, and this is all
very rough, right? These classifications are not
in any way super serious.
But a B tier tool, in my mind, has
some of its own modules it's doing, some of its own sources,
maybe has a GUI, maybe has some
workflow in it where they're bringing data in from the end of the
recon process back to the beginning if they find new stuff.
It has more techniques than a C tier.
But it still runs that point in time and it's still working on flat files
to track the data for recon.
A tier is maybe
writing all of their own modules.
These tools start to have some GUIs, some of them
they run iteratively. So like
Cron or something like that, they'll run at a schedule.
And they start to manage things via database so you can compare
scans and recon scans over time.
And then S tier are kind of the highest level of what's out right now.
They write a lot of their own modules. They have a GUI.
They run iteratively. They manage all their data via database.
They scale across multiple boxes to make the scanning faster.
They send alerts back to the user via email and text
and Slack and whatever when they find
new things. They have some novel techniques that not a lot of other people
are doing. So these are how I classify some of the frameworks. I'm going to show
some of them to you now and you can pick one that works for you if you're not
into this recon stuff. Warning.
I had to put this in here. I don't want to hurt anyone's feelings.
I am scaling some tools and giving them grades. It's not
serious. It's my rough experience. My gut feel.
All these tools are wonderful and I respect the authors of them
so darn much of just putting code out there.
I can't say enough about the people that open source their code
and even C tier. Like I said, my tool is a C tier
tool and it works for me. So this is just subjective
and based off my own experience. Okay. So
some C tier tools that you could look into that are wrapping around
a lot of other existing tools.
One of the ones called ultimate recon here
I like is listed here. And the reason I like
ultimate recon is because it's using the new nuclei
scanner that's out by project discovery and it
implements finding subdomains using a couple
different tools. It will then take all of those. It will
port scan them and then it will run nuclei templates on
them. So it's pretty good. I like it.
All of these different C tier tools
they're all choosing their own tools to wrap around
that they like. So there's no best one really. But
I tend to like ultimate recon in this
one. There's also other good ones in here too.
So check these out. B tier
frameworks. This is lazy recon by Captain Milo.
And here you can see his workflow. Very
similar to what we talked about in this presentation. Amass and sub finder
combine those two. You get a final subdomain count.
Then you start to do some vulnerability analysis with
subjack and core scanner to find some vulnerabilities related
around subdomain takeover and cores vulnerabilities.
Then you start brute forcing. Then
you start port scanning. Then you do some
screenshots. And then you have a final
output. So a lot of stuff we talked about today is
implemented inside of this workflow. So lazy recon is what I like to
point out inside of the B tier here.
A tier, there's a couple.
They all teeter on being S tier, right?
I wonder if I should even really have
like an S tier and an A tier. Usually
the thing that's separating some of these is GUI from the other ones.
So the one I have referenced here on this slide is called find domain.
And find domain does pretty much everything. It does subdomain
discovery and scraping and brute force and all of that
via amass and sub finder and asset finder
and everything. And then it stores it in a database.
It does iterative scanning. It'll text you. If it finds new
domains from its last scan, it'll email you, which you see in the bottom left
hand corner, which is great. The only thing it lacks
is a GUI. It's a command line based tool. So
it kind of teeters on doing a lot of the stuff we want to do.
And in the enterprise version, it does port scanning
and screenshots. And that
version is a for pay version, but it's affordable to bounty
hunters. Some of those are really good subscriptions to pay for.
So find domain, pretty cool.
Here's our S tier framework. Some of them you'll never get to use
because they're commercial products, but I include them here to give people
ideas as to what they should be working towards in their frameworks.
Intrigue.io is an asset discovery tool
written by Jonathan Cran, who's a friend of mine.
He builds Intrigue and he has a
GUI for it and it's a SaaS platform and businesses buy it
to map their attack surface. But it's doing the same techniques that we've talked
about today. The cool thing about Intrigue
is even if you don't pay for the subscription of
the SaaS version that he's selling to businesses,
he open sources almost all of the code for Intrigue.io
so you can run your own hosted version or at least
see what he's doing and his kind of analysis. So Intrigue is pretty cool.
Asset note is another one that's
a B2B type play for recon.
It's a team of former bug hunters who made this awesome platform.
This is probably the gold standard of
asset management or attack surface
management tools. You can see that it's pretty.
It breaks things up by assets that need attention.
It's got a lot of graphing. It does
a lot of custom vulnerability checking.
So asset note is probably, it's S tier, but
it's a high price tag so you're probably not going to use it as a bug hunter
or a red teamer. But it is kind of the gold standard that a lot of the tool
makers should probably work towards.
Spiderfoot is another one that does
a lot of OSINT type domain correlation information.
It does some of the techniques we've talked about
in this presentation. They're adding more all the time. I know the author
is really strictly interested
in adding more bug bounty focused features
in it. And it's got a great graphing library and monitoring
and everything. So Spiderfoot's pretty cool.
The unreleased project discovery, we've talked about project discovery a couple times
in the presentation. That team, their unreleased framework looks
to be a killer. Looks to be awesome. I'm really excited about it
when it comes out. So yeah, you can see here
a dashboard of a project, all your discovered domains
and seeds, what the scan
process is. And then if you dig into any of those, you get screenshots,
port scans, technology identification of the
site, its activity, its changes over time.
So the discovery framework when it comes out will
be pretty cool. Jails is a vulnerability
scanner written by Jesse, JJJ and team.
It is also really good. It is a
GUI flow that wraps around some command line stuff.
It does vulnerability scanning, which, you know,
if you're working with Nessus or, you know, not a lot of red teams are
scanning with Nessus. But if you want to look for some
very pointed stuff, you can use Jails and look at their
kind of CVE checks that they use.
And you can see signatures there for
the Zoho management page and fuel CMS RCE.
So they're looking for some cool volume Jails.
I really like it. Osmedias is
awesome as well. It's very similar to some of the previous
pages we've seen. It'll take all of your
domains. It will do... it'll take all your subdomains.
It will resolve them to IP. It will do port scanning on them.
It'll tell you the technologies.
It will do screenshots. It'll do all kinds of stuff.
So Osmedias is also pretty easy to stand up and really good.
Huntersuite.io, also same kind of idea.
You can see technology parsing, services,
domain discovery, etc.
Bounty.offensive.ai,
same type of deal, right? Includes some vulnerability scanning,
technology fingerprinting, subdomain finding,
and then attack graphing.
Reengine is the one I used last week or a week before.
Reengine is pretty sick.
You can have projects in it that are separate.
It'll do the subdomain scanning. It basically takes the methodology
that I outlined in this presentation and turns it into
a tool, a hosted tool.
And they're making some improvements every
week. So lots of potential on
Reengine, which I highly
suggest checking out.
And then Scout PDP on the SecApps team. They've been working
on tools for well over a decade.
And they have built out Scout, which is a paid offering,
but it's affordable enough for bounty hunters. Same idea, right?
Projects, subdomain enumeration, related
domains, reverse DNS. It supports screenshotting.
It's awesome.
Awesome tool.
And then lastly, Nuclei.
I talked about it very briefly, but anyone who's not using
Nuclei in their bounty scanning right now is at a disadvantage.
They're coming out with some epic templates for CVE
identification.
And you can basically build your own via a YAML file.
And it's a scanner that'll go out and scan all your domains for
vulnerabilities, subdomain takeovers, all kinds of stuff.
So I really have been using Nuclei to great success lately.
And that's it.
That's all I got. So that's the Bug Hunters methodology.
Thanks for giving me the time today. I really appreciate it. Sorry I went over a little bit.
No worries. Thank you so much for the presentation
and amazing support.
Once again, from the bottom of my heart, thank you for supporting
DEF CON and the DEF CON Red Team Village.
And a reminder for everyone here,
please take a look at all the talks and activities that we have
in our website. We have the CTF, of course,
the Cyber Raid contest as well.
Tons and tons of activities throughout this weekend.
The link should be in the description at
the bottom of your stream. And all that stuff.