WEBVTT 00:00.000 --> 00:01.560 Automation already has. 00:03.680 --> 00:08.980 So at this conference, we don't really need to explain what recon is or why it's good to do. 00:09.060 --> 00:12.500 So let's start with the question of why even automate? 00:12.680 --> 00:15.160 Why is there a talk about automation? 00:15.720 --> 00:18.820 Because if you're watching this, you obviously like to hack. 00:18.820 --> 00:22.660 So why would you get away from doing that hacking manually? 00:23.120 --> 00:25.340 I think it's a good question. 00:25.340 --> 00:30.280 And my best answer is that you don't actually have to choose between automation and manual hacking. 00:30.520 --> 00:32.260 You can have both. 00:33.060 --> 00:38.480 More specifically, what I'm saying is you can use automated recon to feed your manual hacking. 00:39.000 --> 00:44.840 So while you're asleep or gaming or relaxing, you can still be finding stuff to explore later. 00:46.080 --> 00:47.420 So that's the why. 00:47.420 --> 00:52.880 That's why I think automation is cool because it goes hand in hand with manual testing. 00:53.240 --> 00:57.360 But this talk is about the how, not really the why. 00:58.100 --> 01:11.080 So one of the things I hear about a lot is breaking down tools or methodologies or security in general into vulnerability assessment or penetration testing or bounty hunting and making strong lines between these. 01:11.840 --> 01:16.020 But I actually think those distinctions are a bit arbitrary when it comes to automation. 01:16.020 --> 01:22.120 And I actually prefer to abstract security into questions rather than categories. 01:22.520 --> 01:30.460 So I like to break down all my testing into individual, specific, distinct questions. 01:30.760 --> 01:40.880 And I get this idea from the Unix philosophy, which talks about making each program do one thing well and expecting the output of one thing to become the input of another. 01:41.440 --> 01:44.100 So I try to do the same thing with my automation. 01:44.660 --> 01:49.100 So on the right, you can see a number of questions I might want to know about a target. 01:49.920 --> 01:52.640 And I would say this applies almost every time. 01:52.640 --> 01:55.840 And I'm a very web-focused tester. 01:55.840 --> 02:01.200 So these are obviously very web-focused, but it's a giant list of questions. 02:01.200 --> 02:07.160 And it also includes network security as well, but it's a giant list of questions that I pretty much want to know about every target. 02:07.460 --> 02:14.180 So my automation is a way of asking and answering those questions for any arbitrary target. 02:16.810 --> 02:24.630 And that brings us here, where we have specific, discrete questions being answered by a specific piece of code. 02:25.790 --> 02:28.290 So in this case, we have two questions. 02:28.510 --> 02:29.670 What are their subdomains? 02:29.670 --> 02:33.150 Which is a question captured in checksubdomains.sh. 02:33.490 --> 02:39.350 And which of their subdomains are running web servers, which is captured in checkwebserver.sh. 02:42.900 --> 02:52.080 So that's what I'm going to talk about today, a way to ask lots of different security questions continuously, and then do fun things with those answers. 02:52.420 --> 02:59.600 And if you want to see what kinds of questions you might want to ask for a bug bounty, for example, I really recommend you start with a good methodology. 02:59.960 --> 03:11.360 And I think the best explanations of the power of methodologies comes from my best buddy, Jason Haddox, who's also speaking here, by the way, at the same conference. 03:11.860 --> 03:21.740 Jason not only talks about the steps in his bug hunters methodology series, but he talks about how to show them visually and maps them out into mind maps. 03:21.740 --> 03:23.240 It's really good stuff. 03:23.240 --> 03:28.100 And I really think he has the best content out there around web and recon methodologies. 03:31.270 --> 03:33.930 So let's look at a few examples of these. 03:33.930 --> 03:38.370 Let's start off with a simple case of where you know you have an external IP range. 03:38.770 --> 03:41.990 And you want to know what hosts are live in that range. 03:42.630 --> 03:47.330 The question there would be something like for a given IP range, which hosts are live? 03:47.330 --> 03:52.190 Or which are likely to be based on the fact that they're serving common services? 03:52.670 --> 04:01.230 That question produces a seed, live IPs.text, that becomes the input to countless other modules. 04:02.850 --> 04:09.230 So before I go more into modules, I want to actually say something real quick about the level of the code in your automation. 04:09.230 --> 04:15.690 I think this is likely to come up for anyone who thinks about automation or has actually tried to write their own. 04:15.990 --> 04:23.250 I like to think of it as two extremes, with frameworks on one side and completely custom code on the other. 04:23.610 --> 04:31.030 So completely custom code is like you're writing in C or Go or something, and you're writing your own port scanner or something. 04:31.030 --> 04:37.250 So you're interacting fairly directly with the kernel, low level programming, right? 04:37.250 --> 04:44.430 And with frameworks, you're like calling a masses enumerate function to get a list of IPs, for example. 04:44.830 --> 04:53.890 So I personally prefer to sit right in the middle of those two and write extremely small Unix-y modules that leverage a low level utility. 04:54.110 --> 05:00.450 So for example, I like to do my ASN and IP range lookups from ipinfo.io. 05:00.730 --> 05:08.410 So in my automation, I write small little wrappers for each of those discrete functions, rather than using a framework. 05:08.990 --> 05:24.930 I think that hybrid idea gives you a great combination of Unix-y like control, of very small things doing very small things, without needing to rewrite things like curl or nmap or mass scan on your own. 05:26.070 --> 05:27.710 So that's the level. 05:27.710 --> 05:29.610 I like to live right there in the middle. 05:31.470 --> 05:40.410 So here's another module that is fundamental to my workflow, which is just getting a full HTML output for a page. 05:40.630 --> 05:55.130 So here I'm using a headless Chromium browser via command line, so that it gets accepted when you make the query in as many web servers and defensive systems as possible, because curl is pretty much denied by a lot of things by now. 05:55.650 --> 06:12.510 And then once I have that raw HTML, I can do all sorts of stuff with it locally via different modules, like getting all the JavaScript files, parsing it to see if this page might be marked as sensitive, looking for artifacts that might indicate the application or text stack, 06:12.510 --> 06:17.730 looking for fields that might be known to be vulnerable to certain injections, et cetera. 06:18.130 --> 06:25.750 And I have at least a dozen of these just for parsing HTML, but you have to have the HTML to start with. 06:25.750 --> 06:31.890 So this is a good fundamental module or script or piece of code to start with. 06:33.730 --> 06:50.630 So one thing you always have to do for another module here is flesh out the scope of your target, which often involves pivoting from one TLD to another TLD that is related, but you don't necessarily even have the name of the domain, right? 06:50.630 --> 06:59.390 So you might be looking at Tesla and you're going to find other Tesla-related domains, but there are some Tesla-related domains that don't have Tesla in the domain itself. 06:59.850 --> 07:04.410 So one technique I like to use for that is following redirects to the target domain. 07:05.390 --> 07:07.130 And there can be flaws here. 07:07.130 --> 07:15.410 You can have other sites or territories or whatever that link to a different target, but aren't related. 07:15.410 --> 07:27.030 So you've got to do other checking to make sure that doesn't happen, but it is generally a fairly low noise, high signal method of finding new stuff. 07:27.510 --> 07:34.530 And to do this, one of the tools I like to use again is IP info and also host.io, which is related. 07:35.270 --> 07:50.610 And these are both a key part of my entire automation stack, because again, I'm writing small modules that call IP info and host.io explicitly to get one particular function and then produce an output from that. 07:50.770 --> 07:58.630 So they are how I do a lot of the lower level tasks, like getting ASNs for a company, getting IP ranges from ASNs, et cetera. 08:00.670 --> 08:15.790 Once again, there are a million tools that you can do this with, but in my opinion, the key to solid Unix-y automation is having something as low level as possible and non-abstracted as possible that you trust. 08:16.090 --> 08:18.730 And I just happen to use IP info for that. 08:18.730 --> 08:20.290 Some people use Hurricane for that. 08:20.290 --> 08:28.370 Other people use some of the DNS services where they have API access, or there's just a whole bunch of them out there. 08:28.690 --> 08:32.590 And this is just for my automation workflows, which is what we're talking about here. 08:32.590 --> 08:43.050 But if I want to go and do some manual exploring, and just check some stuff out, I'm extremely partial to Amass for that, especially since they keep adding functionality. 08:43.050 --> 08:45.650 It's now an OWASP project. 08:46.070 --> 08:49.830 And Jeff Foley, who runs that project, is just fantastic. 08:49.830 --> 08:51.290 So I really love Amass. 08:51.290 --> 08:54.290 It's my favorite all-arounder tool. 08:54.290 --> 08:56.010 I really enjoy it. 08:58.360 --> 08:58.840 All right. 08:58.840 --> 09:05.360 So now you've seen a number of modules, and you get the workflow for creating modules, right? 09:05.360 --> 09:06.980 And you understand that they have to be small. 09:06.980 --> 09:15.240 They have to have discrete output that isn't consumed by another discrete piece of functionality. 09:15.380 --> 09:20.140 So here's what it starts to look like when you want to answer a complex question. 09:20.140 --> 09:22.260 Again, think Unix. 09:22.320 --> 09:24.840 The output of one becomes the input of another. 09:25.620 --> 09:28.480 And this is a simplified view of a workflow. 09:28.480 --> 09:32.160 So we're going from git company to company. 09:32.800 --> 09:34.080 We're getting domains. 09:34.080 --> 09:36.960 We're taking domains, and we're getting ASNs. 09:36.960 --> 09:38.620 We're taking ASNs. 09:38.620 --> 09:42.500 We're getting ranges, and we're ending up with ranges.txt. 09:42.860 --> 09:46.000 So this is a simplified view of a workflow. 09:46.160 --> 09:53.920 In actuality, you might have multiple submodules that add sources or do cleanup or validation on another module. 09:54.220 --> 10:04.800 So domains.txt, for example, might have five different modules feeding into it and maybe one or two cleanup mechanisms that go and prune out noise from there. 10:04.900 --> 10:06.720 Make sure no junk is added. 10:08.520 --> 10:11.960 But now take that to the nth degree, right? 10:11.960 --> 10:23.540 You can have one piece of output that feeds 10 different modules, and all of those modules on the right can then, in turn, feed each other or produce their own outputs. 10:23.660 --> 10:28.600 And over the last five years, I've created like 50 of these things for my own use. 10:28.620 --> 10:33.460 And the fact that you can just automate them is completely insane. 10:33.960 --> 10:37.260 I'm actually in the process of putting some of these on GitHub. 10:37.360 --> 10:42.080 I meant to do that before this talk, but I should have some up soon. 10:42.080 --> 10:45.900 And they'll be the ones I'm least embarrassed to put out there. 10:45.900 --> 10:48.400 Probably do some cleanup before I release them. 10:50.400 --> 10:53.440 And that brings us to the automation piece, right? 10:53.800 --> 11:01.540 So there's lots of fancy ways to do automation, but this is all about, you know, really cheap Linux box. 11:01.540 --> 11:03.640 What can you do with the tools available? 11:04.000 --> 11:07.040 And you can just use Cron for automation. 11:07.040 --> 11:08.860 Comes free with Linux. 11:09.300 --> 11:15.440 And you can use it not only to run your modules, but also to send your notification when things are found. 11:15.900 --> 11:21.800 You just need to figure out what needs to finish before other things start and wire that all up. 11:21.800 --> 11:26.720 And you could use code inside the module to make sure one thing is finished before another. 11:27.100 --> 11:32.380 And some interesting stuff that's fairly obvious once you start wiring stuff up. 11:34.100 --> 11:43.200 And finally, once all the modules are running continuously via Cron, so you've got these discrete pieces of code producing discrete outputs. 11:43.200 --> 11:45.740 They're all wired up using Cron. 11:45.740 --> 11:47.620 They're running continuously. 11:48.800 --> 11:53.260 You can then rig them up to notify you when they find something. 11:53.260 --> 11:58.480 And this is super easy to do via email, Slack, or really anything with an API. 11:58.620 --> 12:02.380 So I'm really partial to Amazon SES for email. 12:02.500 --> 12:06.320 You can send tons of emails with it for like pennies a month. 12:06.320 --> 12:08.460 You could do it all from the command line. 12:09.080 --> 12:14.140 And of course, you could even set up your own Slack channel where you monitor your favorite targets or bounties or whatever. 12:14.480 --> 12:18.260 And send yourself a Slack if your automation finds something new. 12:21.690 --> 12:29.670 So another thing to consider is how to collect and maintain and deploy all of this to the internet. 12:29.670 --> 12:29.950 Right? 12:29.950 --> 12:32.270 So we have the scripts themselves. 12:32.310 --> 12:34.930 You know, the code, the modules, whatever you want to call them. 12:35.170 --> 12:38.850 We have them automated via Cron. 12:39.290 --> 12:47.190 And we're now sending alerts with continuous monitoring that's going out via email or Slack or whatever system. 12:47.730 --> 12:51.090 But now the question is, okay, how do I build a box that does this? 12:51.970 --> 12:56.250 Like, you don't want to have it running on your home system, like shooting out of your home connection. 12:56.250 --> 12:58.210 It's a bad idea. 12:58.510 --> 13:04.270 So the natural way to do this is to build yourself a Linux box somewhere and just start hacking on it, right? 13:04.270 --> 13:07.390 And you start adding code and scripts or whatever. 13:07.530 --> 13:13.470 And you're pulling down some libraries and some modules and some third party tools or some open source tools, whatever. 13:13.470 --> 13:16.650 You're just, like, linking all this stuff up. 13:16.870 --> 13:17.950 And that works. 13:17.950 --> 13:24.430 But the problem is, once you want to replicate that somewhere else, you have to redo tons of work to make that box identical. 13:24.750 --> 13:33.810 So what I started doing a while back was using Terraform and Ansible combined with GitHub to manage all the code in the configs. 13:34.010 --> 13:45.010 So you have a self-contained directory for a new target that you want to monitor, like, you know, Verizon Media or Tesla or whatever program it is. 13:45.370 --> 13:55.910 So if I think up or hear about a new technique, I then make that change in the local copy and just redeploy the box using Terraform and Ansible. 13:55.910 --> 14:05.190 Or if I want to monitor a new target, I create a copy of that working box with the new seeds for the new target. 14:05.190 --> 14:08.930 And I just, you know, replicate that and then push it. 14:08.930 --> 14:11.910 And it goes out via Terraform and Ansible. 14:12.090 --> 14:19.110 And it builds itself inside of, you know, DigitalOcean or AWS, wherever you want to build it. 14:19.550 --> 14:28.550 And the crazy thing is, you can actually deploy and as soon as you press go, within a couple of minutes, the box comes up, it comes live. 14:28.910 --> 14:36.090 But because you have configured Cron already, it just starts monitoring, right? 14:36.090 --> 14:37.910 It just starts working automatically. 14:37.910 --> 14:40.490 And you immediately have your emails wired up. 14:40.490 --> 14:42.870 You immediately have your Slack wired up. 14:43.170 --> 14:50.650 And you can set up the variables such that it just starts working with the exact correct names of your new target. 14:50.710 --> 15:08.270 And it just starts finding all the domains, it starts finding all the subdomains, it starts pulling all the websites, it starts testing all the ports, it starts crawling, it starts, you know, doing vulnerability analysis on the sites themselves for cross-site scripting and RFIs and all the different techniques you want to use. 15:08.270 --> 15:11.930 All the automation just spins up and starts kicking off. 15:12.410 --> 15:15.730 And that is all connected to your alerting. 15:15.910 --> 15:28.010 So you can literally just go in, set up a new target locally in Terraform or Ansible, press go, and then within two or three minutes, you start getting the alerts. 15:28.470 --> 15:30.970 And that is like incredibly powerful. 15:34.100 --> 15:41.060 So what I love about doing automation in this way is that when I hear about a new technique, I don't just say, oh, that's cool. 15:41.100 --> 15:44.200 And maybe write it down or mostly forget about it. 15:44.360 --> 15:49.440 If it's really cool and I want to remember it, I make a note that I need to turn into a module. 15:49.880 --> 15:55.580 And then I go in and add that module to Ansible in the local config, right? 15:55.580 --> 16:11.620 So, for example, Jason just posted a thing on Twitter, I think like last week or maybe the week before, about crawling CVE details, the website, looking for URLs, because they're often talking about URLs that are sensitive or dangerous. 16:11.940 --> 16:14.800 Well, he was like, why don't I just crawl that and make a list of URLs? 16:14.880 --> 16:16.260 I was like, oh, that's super smart. 16:16.260 --> 16:19.780 So it's on my list of things to do to go create a module that does that. 16:20.860 --> 16:22.940 And I mean, I wouldn't have thought of that. 16:22.940 --> 16:25.760 I've thought of similar things, but I didn't think of that. 16:25.760 --> 16:28.200 And it's now on my list of things to do. 16:28.200 --> 16:39.620 And so you can go directly from a thing that you saw that you hadn't thought of, that someone else had a cool idea, and you can make a module out of it, which now incorporates that knowledge. 16:39.740 --> 16:48.140 It's just really frustrating to me to go to a talk or something, to be super excited about all the stuff that you hear. 16:48.500 --> 16:51.840 But then two weeks later, you're like, do I even remember any of that? 16:52.260 --> 17:02.120 Well, with an automation stack, you can convert your knowledge and your learnings into something tangible and repeatable. 17:02.640 --> 17:12.380 So everyone you're seeing on this slide here, everyone you see here is some combination of a hunter, a tester, and a content creator. 17:12.380 --> 17:15.300 And you should absolutely be following their work. 17:15.700 --> 17:20.300 So I mean, these people are putting out really cool tools. 17:20.300 --> 17:22.600 They're putting out YouTube videos. 17:22.600 --> 17:24.220 They're helping the community. 17:24.220 --> 17:25.880 They're super accessible. 17:25.880 --> 17:29.100 You can just ping them on Twitter and be like, hey, what do you think of this technique? 17:29.280 --> 17:35.260 Just very accessible, very knowledgeable, and producing great stuff for the community. 17:35.260 --> 17:39.140 So you should absolutely be following their work. 17:39.140 --> 17:43.280 Constantly learning stuff from them and many others like them in the community. 17:43.280 --> 17:46.520 And it's just really good to be tapped into people like this. 17:49.070 --> 17:49.730 All right. 17:49.730 --> 17:52.870 So that's what I wanted to talk about today. 17:53.210 --> 18:01.830 And as a quick summary, so the biggest takeaway from my approach to this is breaking up your testing into discrete commands. 18:02.650 --> 18:07.270 And you want to stay as close to a trusted source as possible when you're writing those commands. 18:07.270 --> 18:09.110 So avoiding abstraction. 18:09.170 --> 18:14.190 You want to make sure your output is extremely well named and clean. 18:14.250 --> 18:17.940 And goes into artifacts that can be used by other modules. 18:18.450 --> 18:25.050 You want to chain your commands together into a methodology and schedule them running Cron in this case. 18:25.050 --> 18:26.370 You can use whatever you want. 18:26.370 --> 18:28.290 But Cron's there and it's free. 18:29.570 --> 18:36.070 And you want to lock in your configs into repeatable deployments using Axiom or Terraform or Ansible. 18:36.170 --> 18:39.090 Axiom is another really cool tool for this. 18:40.670 --> 18:40.970 Yeah. 18:40.970 --> 18:42.570 Really, really cool stuff. 18:42.570 --> 18:44.530 A guy named Ben made this. 18:44.530 --> 18:49.530 And it's a deployment automatically into DigitalOcean. 18:49.530 --> 18:53.210 And it's similar to what I'm using with Terraform and Ansible. 18:53.330 --> 18:55.710 You should check out that project. 18:56.170 --> 19:00.170 You want to follow the people I mentioned and stay aware of the newest techniques. 19:00.530 --> 19:02.510 They're putting out great stuff. 19:02.510 --> 19:04.270 You just need to follow them. 19:04.270 --> 19:05.270 Trust me on this. 19:05.270 --> 19:07.830 And you'll find other people to follow by following them. 19:08.510 --> 19:17.190 And finally, when your automation workflow brings you fruit, go and hack on it manually for fun or profit or whatever. 19:19.490 --> 19:21.110 This is how you can get a hold of me. 19:21.110 --> 19:29.090 If you want to chat more about this, there's a number of people thinking about this problem right now and some really cool frameworks coming out around it as well. 19:29.890 --> 19:33.130 And yeah, if you want to chat about it, just hit me up. 19:33.470 --> 19:36.270 And thanks again to the Red Team Village for having me. 19:36.970 --> 19:38.230 We'll see you next time.