1
00:00:00,000 --> 00:00:01,560
Automation already has.

2
00:00:03,680 --> 00:00:08,980
So at this conference, we don't really need to explain what recon is or why it's good to do.

3
00:00:09,060 --> 00:00:12,500
So let's start with the question of why even automate?

4
00:00:12,680 --> 00:00:15,160
Why is there a talk about automation?

5
00:00:15,720 --> 00:00:18,820
Because if you're watching this, you obviously like to hack.

6
00:00:18,820 --> 00:00:22,660
So why would you get away from doing that hacking manually?

7
00:00:23,120 --> 00:00:25,340
I think it's a good question.

8
00:00:25,340 --> 00:00:30,280
And my best answer is that you don't actually have to choose between automation and manual hacking.

9
00:00:30,520 --> 00:00:32,260
You can have both.

10
00:00:33,060 --> 00:00:38,480
More specifically, what I'm saying is you can use automated recon to feed your manual hacking.

11
00:00:39,000 --> 00:00:44,840
So while you're asleep or gaming or relaxing, you can still be finding stuff to explore later.

12
00:00:46,080 --> 00:00:47,420
So that's the why.

13
00:00:47,420 --> 00:00:52,880
That's why I think automation is cool because it goes hand in hand with manual testing.

14
00:00:53,240 --> 00:00:57,360
But this talk is about the how, not really the why.

15
00:00:58,100 --> 00:01:11,080
So one of the things I hear about a lot is breaking down tools or methodologies or security in general into vulnerability assessment or penetration testing or bounty hunting and making strong lines between these.

16
00:01:11,840 --> 00:01:16,020
But I actually think those distinctions are a bit arbitrary when it comes to automation.

17
00:01:16,020 --> 00:01:22,120
And I actually prefer to abstract security into questions rather than categories.

18
00:01:22,520 --> 00:01:30,460
So I like to break down all my testing into individual, specific, distinct questions.

19
00:01:30,760 --> 00:01:40,880
And I get this idea from the Unix philosophy, which talks about making each program do one thing well and expecting the output of one thing to become the input of another.

20
00:01:41,440 --> 00:01:44,100
So I try to do the same thing with my automation.

21
00:01:44,660 --> 00:01:49,100
So on the right, you can see a number of questions I might want to know about a target.

22
00:01:49,920 --> 00:01:52,640
And I would say this applies almost every time.

23
00:01:52,640 --> 00:01:55,840
And I'm a very web-focused tester.

24
00:01:55,840 --> 00:02:01,200
So these are obviously very web-focused, but it's a giant list of questions.

25
00:02:01,200 --> 00:02:07,160
And it also includes network security as well, but it's a giant list of questions that I pretty much want to know about every target.

26
00:02:07,460 --> 00:02:14,180
So my automation is a way of asking and answering those questions for any arbitrary target.

27
00:02:16,810 --> 00:02:24,630
And that brings us here, where we have specific, discrete questions being answered by a specific piece of code.

28
00:02:25,790 --> 00:02:28,290
So in this case, we have two questions.

29
00:02:28,510 --> 00:02:29,670
What are their subdomains?

30
00:02:29,670 --> 00:02:33,150
Which is a question captured in checksubdomains.sh.

31
00:02:33,490 --> 00:02:39,350
And which of their subdomains are running web servers, which is captured in checkwebserver.sh.

32
00:02:42,900 --> 00:02:52,080
So that's what I'm going to talk about today, a way to ask lots of different security questions continuously, and then do fun things with those answers.

33
00:02:52,420 --> 00:02:59,600
And if you want to see what kinds of questions you might want to ask for a bug bounty, for example, I really recommend you start with a good methodology.

34
00:02:59,960 --> 00:03:11,360
And I think the best explanations of the power of methodologies comes from my best buddy, Jason Haddox, who's also speaking here, by the way, at the same conference.

35
00:03:11,860 --> 00:03:21,740
Jason not only talks about the steps in his bug hunters methodology series, but he talks about how to show them visually and maps them out into mind maps.

36
00:03:21,740 --> 00:03:23,240
It's really good stuff.

37
00:03:23,240 --> 00:03:28,100
And I really think he has the best content out there around web and recon methodologies.

38
00:03:31,270 --> 00:03:33,930
So let's look at a few examples of these.

39
00:03:33,930 --> 00:03:38,370
Let's start off with a simple case of where you know you have an external IP range.

40
00:03:38,770 --> 00:03:41,990
And you want to know what hosts are live in that range.

41
00:03:42,630 --> 00:03:47,330
The question there would be something like for a given IP range, which hosts are live?

42
00:03:47,330 --> 00:03:52,190
Or which are likely to be based on the fact that they're serving common services?

43
00:03:52,670 --> 00:04:01,230
That question produces a seed, live IPs.text, that becomes the input to countless other modules.

44
00:04:02,850 --> 00:04:09,230
So before I go more into modules, I want to actually say something real quick about the level of the code in your automation.

45
00:04:09,230 --> 00:04:15,690
I think this is likely to come up for anyone who thinks about automation or has actually tried to write their own.

46
00:04:15,990 --> 00:04:23,250
I like to think of it as two extremes, with frameworks on one side and completely custom code on the other.

47
00:04:23,610 --> 00:04:31,030
So completely custom code is like you're writing in C or Go or something, and you're writing your own port scanner or something.

48
00:04:31,030 --> 00:04:37,250
So you're interacting fairly directly with the kernel, low level programming, right?

49
00:04:37,250 --> 00:04:44,430
And with frameworks, you're like calling a masses enumerate function to get a list of IPs, for example.

50
00:04:44,830 --> 00:04:53,890
So I personally prefer to sit right in the middle of those two and write extremely small Unix-y modules that leverage a low level utility.

51
00:04:54,110 --> 00:05:00,450
So for example, I like to do my ASN and IP range lookups from ipinfo.io.

52
00:05:00,730 --> 00:05:08,410
So in my automation, I write small little wrappers for each of those discrete functions, rather than using a framework.

53
00:05:08,990 --> 00:05:24,930
I think that hybrid idea gives you a great combination of Unix-y like control, of very small things doing very small things, without needing to rewrite things like curl or nmap or mass scan on your own.

54
00:05:26,070 --> 00:05:27,710
So that's the level.

55
00:05:27,710 --> 00:05:29,610
I like to live right there in the middle.

56
00:05:31,470 --> 00:05:40,410
So here's another module that is fundamental to my workflow, which is just getting a full HTML output for a page.

57
00:05:40,630 --> 00:05:55,130
So here I'm using a headless Chromium browser via command line, so that it gets accepted when you make the query in as many web servers and defensive systems as possible, because curl is pretty much denied by a lot of things by now.

58
00:05:55,650 --> 00:06:12,510
And then once I have that raw HTML, I can do all sorts of stuff with it locally via different modules, like getting all the JavaScript files, parsing it to see if this page might be marked as sensitive, looking for artifacts that might indicate the application or text stack,

59
00:06:12,510 --> 00:06:17,730
looking for fields that might be known to be vulnerable to certain injections, et cetera.

60
00:06:18,130 --> 00:06:25,750
And I have at least a dozen of these just for parsing HTML, but you have to have the HTML to start with.

61
00:06:25,750 --> 00:06:31,890
So this is a good fundamental module or script or piece of code to start with.

62
00:06:33,730 --> 00:06:50,630
So one thing you always have to do for another module here is flesh out the scope of your target, which often involves pivoting from one TLD to another TLD that is related, but you don't necessarily even have the name of the domain, right?

63
00:06:50,630 --> 00:06:59,390
So you might be looking at Tesla and you're going to find other Tesla-related domains, but there are some Tesla-related domains that don't have Tesla in the domain itself.

64
00:06:59,850 --> 00:07:04,410
So one technique I like to use for that is following redirects to the target domain.

65
00:07:05,390 --> 00:07:07,130
And there can be flaws here.

66
00:07:07,130 --> 00:07:15,410
You can have other sites or territories or whatever that link to a different target, but aren't related.

67
00:07:15,410 --> 00:07:27,030
So you've got to do other checking to make sure that doesn't happen, but it is generally a fairly low noise, high signal method of finding new stuff.

68
00:07:27,510 --> 00:07:34,530
And to do this, one of the tools I like to use again is IP info and also host.io, which is related.

69
00:07:35,270 --> 00:07:50,610
And these are both a key part of my entire automation stack, because again, I'm writing small modules that call IP info and host.io explicitly to get one particular function and then produce an output from that.

70
00:07:50,770 --> 00:07:58,630
So they are how I do a lot of the lower level tasks, like getting ASNs for a company, getting IP ranges from ASNs, et cetera.

71
00:08:00,670 --> 00:08:15,790
Once again, there are a million tools that you can do this with, but in my opinion, the key to solid Unix-y automation is having something as low level as possible and non-abstracted as possible that you trust.

72
00:08:16,090 --> 00:08:18,730
And I just happen to use IP info for that.

73
00:08:18,730 --> 00:08:20,290
Some people use Hurricane for that.

74
00:08:20,290 --> 00:08:28,370
Other people use some of the DNS services where they have API access, or there's just a whole bunch of them out there.

75
00:08:28,690 --> 00:08:32,590
And this is just for my automation workflows, which is what we're talking about here.

76
00:08:32,590 --> 00:08:43,050
But if I want to go and do some manual exploring, and just check some stuff out, I'm extremely partial to Amass for that, especially since they keep adding functionality.

77
00:08:43,050 --> 00:08:45,650
It's now an OWASP project.

78
00:08:46,070 --> 00:08:49,830
And Jeff Foley, who runs that project, is just fantastic.

79
00:08:49,830 --> 00:08:51,290
So I really love Amass.

80
00:08:51,290 --> 00:08:54,290
It's my favorite all-arounder tool.

81
00:08:54,290 --> 00:08:56,010
I really enjoy it.

82
00:08:58,360 --> 00:08:58,840
All right.

83
00:08:58,840 --> 00:09:05,360
So now you've seen a number of modules, and you get the workflow for creating modules, right?

84
00:09:05,360 --> 00:09:06,980
And you understand that they have to be small.

85
00:09:06,980 --> 00:09:15,240
They have to have discrete output that isn't consumed by another discrete piece of functionality.

86
00:09:15,380 --> 00:09:20,140
So here's what it starts to look like when you want to answer a complex question.

87
00:09:20,140 --> 00:09:22,260
Again, think Unix.

88
00:09:22,320 --> 00:09:24,840
The output of one becomes the input of another.

89
00:09:25,620 --> 00:09:28,480
And this is a simplified view of a workflow.

90
00:09:28,480 --> 00:09:32,160
So we're going from git company to company.

91
00:09:32,800 --> 00:09:34,080
We're getting domains.

92
00:09:34,080 --> 00:09:36,960
We're taking domains, and we're getting ASNs.

93
00:09:36,960 --> 00:09:38,620
We're taking ASNs.

94
00:09:38,620 --> 00:09:42,500
We're getting ranges, and we're ending up with ranges.txt.

95
00:09:42,860 --> 00:09:46,000
So this is a simplified view of a workflow.

96
00:09:46,160 --> 00:09:53,920
In actuality, you might have multiple submodules that add sources or do cleanup or validation on another module.

97
00:09:54,220 --> 00:10:04,800
So domains.txt, for example, might have five different modules feeding into it and maybe one or two cleanup mechanisms that go and prune out noise from there.

98
00:10:04,900 --> 00:10:06,720
Make sure no junk is added.

99
00:10:08,520 --> 00:10:11,960
But now take that to the nth degree, right?

100
00:10:11,960 --> 00:10:23,540
You can have one piece of output that feeds 10 different modules, and all of those modules on the right can then, in turn, feed each other or produce their own outputs.

101
00:10:23,660 --> 00:10:28,600
And over the last five years, I've created like 50 of these things for my own use.

102
00:10:28,620 --> 00:10:33,460
And the fact that you can just automate them is completely insane.

103
00:10:33,960 --> 00:10:37,260
I'm actually in the process of putting some of these on GitHub.

104
00:10:37,360 --> 00:10:42,080
I meant to do that before this talk, but I should have some up soon.

105
00:10:42,080 --> 00:10:45,900
And they'll be the ones I'm least embarrassed to put out there.

106
00:10:45,900 --> 00:10:48,400
Probably do some cleanup before I release them.

107
00:10:50,400 --> 00:10:53,440
And that brings us to the automation piece, right?

108
00:10:53,800 --> 00:11:01,540
So there's lots of fancy ways to do automation, but this is all about, you know, really cheap Linux box.

109
00:11:01,540 --> 00:11:03,640
What can you do with the tools available?

110
00:11:04,000 --> 00:11:07,040
And you can just use Cron for automation.

111
00:11:07,040 --> 00:11:08,860
Comes free with Linux.

112
00:11:09,300 --> 00:11:15,440
And you can use it not only to run your modules, but also to send your notification when things are found.

113
00:11:15,900 --> 00:11:21,800
You just need to figure out what needs to finish before other things start and wire that all up.

114
00:11:21,800 --> 00:11:26,720
And you could use code inside the module to make sure one thing is finished before another.

115
00:11:27,100 --> 00:11:32,380
And some interesting stuff that's fairly obvious once you start wiring stuff up.

116
00:11:34,100 --> 00:11:43,200
And finally, once all the modules are running continuously via Cron, so you've got these discrete pieces of code producing discrete outputs.

117
00:11:43,200 --> 00:11:45,740
They're all wired up using Cron.

118
00:11:45,740 --> 00:11:47,620
They're running continuously.

119
00:11:48,800 --> 00:11:53,260
You can then rig them up to notify you when they find something.

120
00:11:53,260 --> 00:11:58,480
And this is super easy to do via email, Slack, or really anything with an API.

121
00:11:58,620 --> 00:12:02,380
So I'm really partial to Amazon SES for email.

122
00:12:02,500 --> 00:12:06,320
You can send tons of emails with it for like pennies a month.

123
00:12:06,320 --> 00:12:08,460
You could do it all from the command line.

124
00:12:09,080 --> 00:12:14,140
And of course, you could even set up your own Slack channel where you monitor your favorite targets or bounties or whatever.

125
00:12:14,480 --> 00:12:18,260
And send yourself a Slack if your automation finds something new.

126
00:12:21,690 --> 00:12:29,670
So another thing to consider is how to collect and maintain and deploy all of this to the internet.

127
00:12:29,670 --> 00:12:29,950
Right?

128
00:12:29,950 --> 00:12:32,270
So we have the scripts themselves.

129
00:12:32,310 --> 00:12:34,930
You know, the code, the modules, whatever you want to call them.

130
00:12:35,170 --> 00:12:38,850
We have them automated via Cron.

131
00:12:39,290 --> 00:12:47,190
And we're now sending alerts with continuous monitoring that's going out via email or Slack or whatever system.

132
00:12:47,730 --> 00:12:51,090
But now the question is, okay, how do I build a box that does this?

133
00:12:51,970 --> 00:12:56,250
Like, you don't want to have it running on your home system, like shooting out of your home connection.

134
00:12:56,250 --> 00:12:58,210
It's a bad idea.

135
00:12:58,510 --> 00:13:04,270
So the natural way to do this is to build yourself a Linux box somewhere and just start hacking on it, right?

136
00:13:04,270 --> 00:13:07,390
And you start adding code and scripts or whatever.

137
00:13:07,530 --> 00:13:13,470
And you're pulling down some libraries and some modules and some third party tools or some open source tools, whatever.

138
00:13:13,470 --> 00:13:16,650
You're just, like, linking all this stuff up.

139
00:13:16,870 --> 00:13:17,950
And that works.

140
00:13:17,950 --> 00:13:24,430
But the problem is, once you want to replicate that somewhere else, you have to redo tons of work to make that box identical.

141
00:13:24,750 --> 00:13:33,810
So what I started doing a while back was using Terraform and Ansible combined with GitHub to manage all the code in the configs.

142
00:13:34,010 --> 00:13:45,010
So you have a self-contained directory for a new target that you want to monitor, like, you know, Verizon Media or Tesla or whatever program it is.

143
00:13:45,370 --> 00:13:55,910
So if I think up or hear about a new technique, I then make that change in the local copy and just redeploy the box using Terraform and Ansible.

144
00:13:55,910 --> 00:14:05,190
Or if I want to monitor a new target, I create a copy of that working box with the new seeds for the new target.

145
00:14:05,190 --> 00:14:08,930
And I just, you know, replicate that and then push it.

146
00:14:08,930 --> 00:14:11,910
And it goes out via Terraform and Ansible.

147
00:14:12,090 --> 00:14:19,110
And it builds itself inside of, you know, DigitalOcean or AWS, wherever you want to build it.

148
00:14:19,550 --> 00:14:28,550
And the crazy thing is, you can actually deploy and as soon as you press go, within a couple of minutes, the box comes up, it comes live.

149
00:14:28,910 --> 00:14:36,090
But because you have configured Cron already, it just starts monitoring, right?

150
00:14:36,090 --> 00:14:37,910
It just starts working automatically.

151
00:14:37,910 --> 00:14:40,490
And you immediately have your emails wired up.

152
00:14:40,490 --> 00:14:42,870
You immediately have your Slack wired up.

153
00:14:43,170 --> 00:14:50,650
And you can set up the variables such that it just starts working with the exact correct names of your new target.

154
00:14:50,710 --> 00:15:08,270
And it just starts finding all the domains, it starts finding all the subdomains, it starts pulling all the websites, it starts testing all the ports, it starts crawling, it starts, you know, doing vulnerability analysis on the sites themselves for cross-site scripting and RFIs and all the different techniques you want to use.

155
00:15:08,270 --> 00:15:11,930
All the automation just spins up and starts kicking off.

156
00:15:12,410 --> 00:15:15,730
And that is all connected to your alerting.

157
00:15:15,910 --> 00:15:28,010
So you can literally just go in, set up a new target locally in Terraform or Ansible, press go, and then within two or three minutes, you start getting the alerts.

158
00:15:28,470 --> 00:15:30,970
And that is like incredibly powerful.

159
00:15:34,100 --> 00:15:41,060
So what I love about doing automation in this way is that when I hear about a new technique, I don't just say, oh, that's cool.

160
00:15:41,100 --> 00:15:44,200
And maybe write it down or mostly forget about it.

161
00:15:44,360 --> 00:15:49,440
If it's really cool and I want to remember it, I make a note that I need to turn into a module.

162
00:15:49,880 --> 00:15:55,580
And then I go in and add that module to Ansible in the local config, right?

163
00:15:55,580 --> 00:16:11,620
So, for example, Jason just posted a thing on Twitter, I think like last week or maybe the week before, about crawling CVE details, the website, looking for URLs, because they're often talking about URLs that are sensitive or dangerous.

164
00:16:11,940 --> 00:16:14,800
Well, he was like, why don't I just crawl that and make a list of URLs?

165
00:16:14,880 --> 00:16:16,260
I was like, oh, that's super smart.

166
00:16:16,260 --> 00:16:19,780
So it's on my list of things to do to go create a module that does that.

167
00:16:20,860 --> 00:16:22,940
And I mean, I wouldn't have thought of that.

168
00:16:22,940 --> 00:16:25,760
I've thought of similar things, but I didn't think of that.

169
00:16:25,760 --> 00:16:28,200
And it's now on my list of things to do.

170
00:16:28,200 --> 00:16:39,620
And so you can go directly from a thing that you saw that you hadn't thought of, that someone else had a cool idea, and you can make a module out of it, which now incorporates that knowledge.

171
00:16:39,740 --> 00:16:48,140
It's just really frustrating to me to go to a talk or something, to be super excited about all the stuff that you hear.

172
00:16:48,500 --> 00:16:51,840
But then two weeks later, you're like, do I even remember any of that?

173
00:16:52,260 --> 00:17:02,120
Well, with an automation stack, you can convert your knowledge and your learnings into something tangible and repeatable.

174
00:17:02,640 --> 00:17:12,380
So everyone you're seeing on this slide here, everyone you see here is some combination of a hunter, a tester, and a content creator.

175
00:17:12,380 --> 00:17:15,300
And you should absolutely be following their work.

176
00:17:15,700 --> 00:17:20,300
So I mean, these people are putting out really cool tools.

177
00:17:20,300 --> 00:17:22,600
They're putting out YouTube videos.

178
00:17:22,600 --> 00:17:24,220
They're helping the community.

179
00:17:24,220 --> 00:17:25,880
They're super accessible.

180
00:17:25,880 --> 00:17:29,100
You can just ping them on Twitter and be like, hey, what do you think of this technique?

181
00:17:29,280 --> 00:17:35,260
Just very accessible, very knowledgeable, and producing great stuff for the community.

182
00:17:35,260 --> 00:17:39,140
So you should absolutely be following their work.

183
00:17:39,140 --> 00:17:43,280
Constantly learning stuff from them and many others like them in the community.

184
00:17:43,280 --> 00:17:46,520
And it's just really good to be tapped into people like this.

185
00:17:49,070 --> 00:17:49,730
All right.

186
00:17:49,730 --> 00:17:52,870
So that's what I wanted to talk about today.

187
00:17:53,210 --> 00:18:01,830
And as a quick summary, so the biggest takeaway from my approach to this is breaking up your testing into discrete commands.

188
00:18:02,650 --> 00:18:07,270
And you want to stay as close to a trusted source as possible when you're writing those commands.

189
00:18:07,270 --> 00:18:09,110
So avoiding abstraction.

190
00:18:09,170 --> 00:18:14,190
You want to make sure your output is extremely well named and clean.

191
00:18:14,250 --> 00:18:17,940
And goes into artifacts that can be used by other modules.

192
00:18:18,450 --> 00:18:25,050
You want to chain your commands together into a methodology and schedule them running Cron in this case.

193
00:18:25,050 --> 00:18:26,370
You can use whatever you want.

194
00:18:26,370 --> 00:18:28,290
But Cron's there and it's free.

195
00:18:29,570 --> 00:18:36,070
And you want to lock in your configs into repeatable deployments using Axiom or Terraform or Ansible.

196
00:18:36,170 --> 00:18:39,090
Axiom is another really cool tool for this.

197
00:18:40,670 --> 00:18:40,970
Yeah.

198
00:18:40,970 --> 00:18:42,570
Really, really cool stuff.

199
00:18:42,570 --> 00:18:44,530
A guy named Ben made this.

200
00:18:44,530 --> 00:18:49,530
And it's a deployment automatically into DigitalOcean.

201
00:18:49,530 --> 00:18:53,210
And it's similar to what I'm using with Terraform and Ansible.

202
00:18:53,330 --> 00:18:55,710
You should check out that project.

203
00:18:56,170 --> 00:19:00,170
You want to follow the people I mentioned and stay aware of the newest techniques.

204
00:19:00,530 --> 00:19:02,510
They're putting out great stuff.

205
00:19:02,510 --> 00:19:04,270
You just need to follow them.

206
00:19:04,270 --> 00:19:05,270
Trust me on this.

207
00:19:05,270 --> 00:19:07,830
And you'll find other people to follow by following them.

208
00:19:08,510 --> 00:19:17,190
And finally, when your automation workflow brings you fruit, go and hack on it manually for fun or profit or whatever.

209
00:19:19,490 --> 00:19:21,110
This is how you can get a hold of me.

210
00:19:21,110 --> 00:19:29,090
If you want to chat more about this, there's a number of people thinking about this problem right now and some really cool frameworks coming out around it as well.

211
00:19:29,890 --> 00:19:33,130
And yeah, if you want to chat about it, just hit me up.

212
00:19:33,470 --> 00:19:36,270
And thanks again to the Red Team Village for having me.

213
00:19:36,970 --> 00:19:38,230
We'll see you next time.