start	end	text
0	1560	Automation already has.
3680	8200	So at this conference, we don't really need to explain what recon is or why it's good
8200	14180	to do. So let's start with the question of why even automate? Why is there a talk about
14180	20160	automation? Because if you're watching this, you obviously like to hack. So why would you
20160	25900	get away from doing that hacking manually? I think it's a good question. And my best
25900	30280	answer is that you don't actually have to choose between automation and manual hacking.
30520	35920	You can have both. More specifically, what I'm saying is you can use automated recon
35920	42080	to feed your manual hacking. So while you're asleep or gaming or relaxing, you can still
42080	49540	be finding stuff to explore later. So that's the why. That's why I think automation is
49540	55880	cool because it goes hand in hand with manual testing. But this talk is about the how, not
55880	61920	really the why. So one of the things I hear about a lot is breaking down tools or methodologies
61920	67580	or security in general into vulnerability assessment or penetration testing or bounty
67580	73680	hunting and making strong lines between these. But I actually think those distinctions are
73800	77960	a bit arbitrary when it comes to automation. And I actually prefer to abstract security
77960	87920	into questions rather than categories. So I like to break down all my testing into individual,
87920	93800	specific, distinct questions. And I get this idea from the Unix philosophy, which talks
93800	99520	about making each program do one thing well and expecting the output of one thing to become
99520	105500	the input of another. So I try to do the same thing with my automation. So on the right,
105500	110460	you can see a number of questions I might want to know about a target. And I would say
110460	117560	this applies almost every time. And I'm a very web-focused tester. So these are obviously
117560	122820	very web-focused, but it's a giant list of questions. And it also includes network security
122820	126380	as well, but it's a giant list of questions that I pretty much want to know about every
126380	133280	target. So my automation is a way of asking and answering those questions for any arbitrary
133280	141870	target. And that brings us here, where we have specific, discrete questions being answered
141870	149670	by a specific piece of code. So in this case, we have two questions. What are their subdomains?
149670	155310	Which is a question captured in checksubdomains.sh. And which of their subdomains are running
155310	165220	web servers, which is captured in checkwebserver.sh. So that's what I'm going to talk about today,
165460	171040	a way to ask lots of different security questions continuously, and then do fun things with
171040	174740	those answers. And if you want to see what kinds of questions you might want to ask for
174840	180480	a bug bounty, for example, I really recommend you start with a good methodology. And I think
180480	188440	the best explanations of the power of methodologies comes from my best buddy, Jason Haddox, who's
188440	193140	also speaking here, by the way, at the same conference. Jason not only talks about the
193140	197800	steps in his bug hunters methodology series, but he talks about how to show them visually
198460	204380	and maps them out into mind maps. It's really good stuff. And I really think he has the
204380	213050	best content out there around web and recon methodologies. So let's look at a few examples
213050	217450	of these. Let's start off with a simple case of where you know you have an external IP
217450	223430	range. And you want to know what hosts are live in that range. The question there would
223430	229230	be something like for a given IP range, which hosts are live? Or which are likely to be
229230	236250	based on the fact that they're serving common services? That question produces a seed, live
236250	244690	IPs.text, that becomes the input to countless other modules. So before I go more into modules,
244690	249230	I want to actually say something real quick about the level of the code in your automation.
249230	254250	I think this is likely to come up for anyone who thinks about automation or has actually
254250	258950	tried to write their own. I like to think of it as two extremes, with frameworks on
258950	265690	one side and completely custom code on the other. So completely custom code is like you're
265690	271030	writing in C or Go or something, and you're writing your own port scanner or something.
271030	277250	So you're interacting fairly directly with the kernel, low level programming, right?
277250	283030	And with frameworks, you're like calling a masses enumerate function to get a list of
283030	288650	IPs, for example. So I personally prefer to sit right in the middle of those two and write
288650	296010	extremely small Unix-y modules that leverage a low level utility. So for example, I like to do
296010	303570	my ASN and IP range lookups from ipinfo.io. So in my automation, I write small little wrappers
303570	311130	for each of those discrete functions, rather than using a framework. I think that hybrid idea
311810	317370	gives you a great combination of Unix-y like control, of very small things doing very small
317370	324930	things, without needing to rewrite things like curl or nmap or mass scan on your own.
326070	329610	So that's the level. I like to live right there in the middle.
331470	337350	So here's another module that is fundamental to my workflow, which is just getting a full
337350	344830	HTML output for a page. So here I'm using a headless Chromium browser via command line,
344830	349650	so that it gets accepted when you make the query in as many web servers and
349650	355130	defensive systems as possible, because curl is pretty much denied by a lot of things by now.
355650	360110	And then once I have that raw HTML, I can do all sorts of stuff with it locally via
360110	364910	different modules, like getting all the JavaScript files, parsing it to see if this
364910	370410	page might be marked as sensitive, looking for artifacts that might indicate the application or
370410	376410	text stack, looking for fields that might be known to be vulnerable to certain injections,
376830	382290	et cetera. And I have at least a dozen of these just for parsing HTML,
382910	387410	but you have to have the HTML to start with. So this is a good fundamental
388550	391890	module or script or piece of code to start with.
393730	400830	So one thing you always have to do for another module here is flesh out the scope of your target,
400830	407210	which often involves pivoting from one TLD to another TLD that is related, but you don't
407210	413050	necessarily even have the name of the domain, right? So you might be looking at Tesla and you're
413050	417350	going to find other Tesla-related domains, but there are some Tesla-related domains that don't
417350	423070	have Tesla in the domain itself. So one technique I like to use for that is following redirects
423070	430250	to the target domain. And there can be flaws here. You can have other sites or territories
430250	436690	or whatever that link to a different target, but aren't related. So you've got to do other
436690	444510	checking to make sure that doesn't happen, but it is generally a fairly low noise, high signal
445170	450970	method of finding new stuff. And to do this, one of the tools I like to use again is IP info
450970	459250	and also host.io, which is related. And these are both a key part of my entire automation stack,
459250	467730	because again, I'm writing small modules that call IP info and host.io explicitly to get one
467730	472890	particular function and then produce an output from that. So they are how I do a lot of the lower
472890	478630	level tasks, like getting ASNs for a company, getting IP ranges from ASNs, et cetera.
480670	485510	Once again, there are a million tools that you can do this with, but in my opinion,
485510	490990	the key to solid Unix-y automation is having something as low level as possible
492250	498730	and non-abstracted as possible that you trust. And I just happen to use IP info for that.
498730	503970	Some people use Hurricane for that. Other people use some of the DNS services where they have API
503970	510550	access, or there's just a whole bunch of them out there. And this is just for my automation
510550	515050	workflows, which is what we're talking about here. But if I want to go and do some manual exploring,
515510	520310	and just check some stuff out, I'm extremely partial to Amass for that,
520310	526970	especially since they keep adding functionality. It's now an OWASP project. And Jeff Foley,
526970	532170	who runs that project, is just fantastic. So I really love Amass. It's my favorite
532850	541600	all-arounder tool. I really enjoy it. All right. So now you've seen a number of modules,
542260	546580	and you get the workflow for creating modules, right? And you understand that they have to be
546580	554140	small. They have to have discrete output that isn't consumed by another discrete piece of
554140	560140	functionality. So here's what it starts to look like when you want to answer a complex question.
560140	564840	Again, think Unix. The output of one becomes the input of another.
565620	572160	And this is a simplified view of a workflow. So we're going from git company to company.
572800	576960	We're getting domains. We're taking domains, and we're getting ASNs.
576960	582500	We're taking ASNs. We're getting ranges, and we're ending up with ranges.txt.
582860	589340	So this is a simplified view of a workflow. In actuality, you might have multiple submodules
589340	595920	that add sources or do cleanup or validation on another module. So domains.txt, for example,
595920	601780	might have five different modules feeding into it and maybe one or two cleanup mechanisms
601780	606720	that go and prune out noise from there. Make sure no junk is added.
608520	614380	But now take that to the nth degree, right? You can have one piece of output that feeds
614380	620160	10 different modules, and all of those modules on the right can then, in turn, feed each other
621320	626800	or produce their own outputs. And over the last five years, I've created like 50 of these things
626800	634780	for my own use. And the fact that you can just automate them is completely insane. I'm actually
634780	639440	in the process of putting some of these on GitHub. I meant to do that before this talk, but
640080	645900	I should have some up soon. And they'll be the ones I'm least embarrassed to put out there.
645900	648400	Probably do some cleanup before I release them.
650400	653440	And that brings us to the automation piece, right?
653800	660500	So there's lots of fancy ways to do automation, but this is all about, you know, really cheap
660500	666340	Linux box. What can you do with the tools available? And you can just use Cron for
666340	672080	automation. Comes free with Linux. And you can use it not only to run your modules,
672080	676640	but also to send your notification when things are found. You just need to figure
676640	682420	out what needs to finish before other things start and wire that all up. And you could use
682420	688480	code inside the module to make sure one thing is finished before another. And some interesting
688480	696480	stuff that's fairly obvious once you start wiring stuff up. And finally, once all the modules are
696480	702580	running continuously via Cron, so you've got these discrete pieces of code producing discrete
702580	710620	outputs. They're all wired up using Cron. They're running continuously. You can then rig them up to
710620	716900	notify you when they find something. And this is super easy to do via email, Slack, or really
716900	724160	anything with an API. So I'm really partial to Amazon SES for email. You can send tons of emails
724160	729580	with it for like pennies a month. You could do it all from the command line. And of course,
729580	732960	you could even set up your own Slack channel where you monitor your favorite targets or
732960	738260	bounties or whatever. And send yourself a Slack if your automation finds something new.
741690	749670	So another thing to consider is how to collect and maintain and deploy all of this to the internet.
749670	754070	Right? So we have the scripts themselves. You know, the code, the modules, whatever you want
754070	762350	to call them. We have them automated via Cron. And we're now sending alerts with continuous
762350	768870	monitoring that's going out via email or Slack or whatever system. But now the question is, okay,
768870	774330	how do I build a box that does this? Like, you don't want to have it running on your home system,
774330	780350	like shooting out of your home connection. It's a bad idea. So the natural way to do this is to
780350	785170	build yourself a Linux box somewhere and just start hacking on it, right? And you start adding
785170	790630	code and scripts or whatever. And you're pulling down some libraries and some modules and some
790630	795690	third party tools or some open source tools, whatever. You're just, like, linking all this
795690	801250	stuff up. And that works. But the problem is, once you want to replicate that somewhere else,
801250	806650	you have to redo tons of work to make that box identical. So what I started doing a while back
806650	813810	was using Terraform and Ansible combined with GitHub to manage all the code in the configs.
814010	820250	So you have a self-contained directory for a new target that you want to monitor, like,
820770	827610	you know, Verizon Media or Tesla or whatever program it is. So if I think up or hear about
827710	834970	a new technique, I then make that change in the local copy and just redeploy the box using Terraform
834970	841490	and Ansible. Or if I want to monitor a new target, I create a copy of that working box
842890	848930	with the new seeds for the new target. And I just, you know, replicate that and then push it.
848930	855570	And it goes out via Terraform and Ansible. And it builds itself inside of, you know,
856130	863250	DigitalOcean or AWS, wherever you want to build it. And the crazy thing is, you can actually deploy
863810	868550	and as soon as you press go, within a couple of minutes, the box comes up, it comes live.
868910	876810	But because you have configured Cron already, it just starts monitoring, right? It just starts
876810	881330	working automatically. And you immediately have your emails wired up. You immediately have your
881330	887490	Slack wired up. And you can set up the variables such that it just starts working with the exact
887490	893370	correct names of your new target. And it just starts finding all the domains, it starts finding
893370	897830	all the subdomains, it starts pulling all the websites, it starts testing all the ports,
897830	903690	it starts crawling, it starts, you know, doing vulnerability analysis on the sites themselves
903690	908270	for cross-site scripting and RFIs and all the different techniques you want to use.
908270	914850	All the automation just spins up and starts kicking off. And that is all connected to your
914850	922070	alerting. So you can literally just go in, set up a new target locally in Terraform or Ansible,
922070	929230	press go, and then within two or three minutes, you start getting the alerts. And that is like
929230	938360	incredibly powerful. So what I love about doing automation in this way is that when I hear about
938480	943340	a new technique, I don't just say, oh, that's cool. And maybe write it down or mostly forget
943340	948180	about it. If it's really cool and I want to remember it, I make a note that I need to turn
948180	955580	into a module. And then I go in and add that module to Ansible in the local config, right?
955580	962200	So, for example, Jason just posted a thing on Twitter, I think like last week or maybe the week
962200	969300	before, about crawling CVE details, the website, looking for URLs, because they're often talking
969300	973680	about URLs that are sensitive or dangerous. Well, he was like, why don't I just crawl that and make
973780	978120	a list of URLs? I was like, oh, that's super smart. So it's on my list of things to do to go
978120	983760	create a module that does that. And I mean, I wouldn't have thought of that. I've thought of
983760	988740	similar things, but I didn't think of that. And it's now on my list of things to do. And so you
988740	994120	can go directly from a thing that you saw that you hadn't thought of, that someone else had a
994120	999620	cool idea, and you can make a module out of it, which now incorporates that knowledge.
999740	1006780	It's just really frustrating to me to go to a talk or something, to be super excited about all
1006780	1011840	the stuff that you hear. But then two weeks later, you're like, do I even remember any of that?
1012260	1018820	Well, with an automation stack, you can convert your knowledge and your learnings
1018820	1026680	into something tangible and repeatable. So everyone you're seeing on this slide here,
1026680	1032380	everyone you see here is some combination of a hunter, a tester, and a content creator.
1032380	1037760	And you should absolutely be following their work. So I mean, these people are putting out
1038660	1044420	really cool tools. They're putting out YouTube videos. They're helping the community. They're
1044420	1047980	super accessible. You can just ping them on Twitter and be like, hey, what do you think of
1047980	1054600	this technique? Just very accessible, very knowledgeable, and producing great stuff for
1054600	1060040	the community. So you should absolutely be following their work. Constantly learning
1060040	1064580	stuff from them and many others like them in the community. And it's just really good to be
1064580	1072870	tapped into people like this. All right. So that's what I wanted to talk about today.
1073210	1078490	And as a quick summary, so the biggest takeaway from my approach to this is breaking up your
1078490	1085410	testing into discrete commands. And you want to stay as close to a trusted source as possible
1085410	1091230	when you're writing those commands. So avoiding abstraction. You want to make sure your output
1091230	1097940	is extremely well named and clean. And goes into artifacts that can be used by other modules.
1098450	1103770	You want to chain your commands together into a methodology and schedule them running
1103770	1108290	Cron in this case. You can use whatever you want. But Cron's there and it's free.
1109570	1114970	And you want to lock in your configs into repeatable deployments using Axiom or Terraform
1114970	1122910	or Ansible. Axiom is another really cool tool for this. Yeah. Really, really cool stuff. A guy
1122910	1129810	named Ben made this. And it's a deployment automatically into DigitalOcean. And it's
1129810	1135710	similar to what I'm using with Terraform and Ansible. You should check out that project.
1136170	1140170	You want to follow the people I mentioned and stay aware of the newest techniques.
1140530	1145690	They're putting out great stuff. You just need to follow them. Trust me on this. And you'll find
1145690	1150450	other people to follow by following them. And finally, when your automation workflow
1150450	1157190	brings you fruit, go and hack on it manually for fun or profit or whatever.
1159490	1162770	This is how you can get a hold of me. If you want to chat more about this,
1162770	1166890	there's a number of people thinking about this problem right now and some really cool
1166890	1173130	frameworks coming out around it as well. And yeah, if you want to chat about it, just hit me up.
1173470	1178230	And thanks again to the Red Team Village for having me. We'll see you next time.