1
00:00:00,000 --> 00:00:02,640
Hello, DEF CON Red Team Village.

2
00:00:02,640 --> 00:00:06,100
I want to thank you today for taking the time to listen to this talk.

3
00:00:06,100 --> 00:00:11,380
I've really been looking forward to it, and I'm excited to be able to share this information with you today.

4
00:00:11,380 --> 00:00:17,160
My talk is titled Combining Notebooks, Datasets, and Cloud for the Ultimate Automation Factory.

5
00:00:17,660 --> 00:00:30,160
And all through this presentation, I really want to challenge you to think about pushing the boundaries of the art of the possible, trying to change how we do things in our day-to-day lives, the manual processes, how can we be more efficient?

6
00:00:30,160 --> 00:00:36,000
How can we operationalize and streamline these activities so that we can be more productive in our lives?

7
00:00:36,000 --> 00:00:45,340
And also, as a Red Teamer, as a Blue Teamer, as a security practitioner, how can we better ourselves and our careers and really elevate ourselves to the next step?

8
00:00:45,740 --> 00:00:50,500
I know it's definitely sad that we're not all together in person at DEF CON.

9
00:00:50,500 --> 00:00:52,800
I know I had everything booked, ready to go.

10
00:00:52,800 --> 00:00:54,820
I was excited, looking forward to it.

11
00:00:54,820 --> 00:01:03,260
But I will say, having this virtual offering is something that's really changed the dynamic of information security and the landscape and the hacking and all of this.

12
00:01:03,260 --> 00:01:19,240
It's actually really set a playing field where a lot of people that didn't have the means or methods to attend Vegas or make it to DEF CON or Black Hat, now it kind of sets the bar where having it online, virtual, free, having direct access through Discord,

13
00:01:19,240 --> 00:01:31,260
through Slack, through different channels and mechanisms, to the industry experts, the people that I've looked up to for years who I've learned from, being able to share the stage with them alongside, it's a pretty awesome time.

14
00:01:31,260 --> 00:01:36,860
The other cool thing is, now I can sit in Discord, I can be ready for questions and answers as we're doing this talk.

15
00:01:36,860 --> 00:01:39,040
So feel free, interactively, to do this.

16
00:01:44,480 --> 00:01:52,740
We won't spend a lot of time in the About Me section, but by day, I'm a security architect, and my main focus is cloud security.

17
00:01:52,740 --> 00:01:57,860
So I get to see a lot of use cases around the new, latest, greatest emerging technologies in cloud.

18
00:01:57,860 --> 00:02:06,680
How are they being leveraged to help businesses and help drive innovation and success and actually decrease operating costs and accelerate delivery.

19
00:02:06,680 --> 00:02:12,200
So I'm taking a lot of those examples, and then I'm combining it with kind of my night job, which is security research.

20
00:02:12,200 --> 00:02:15,980
I mean, security is my, it's my job, but it's also my hobby.

21
00:02:15,980 --> 00:02:24,340
It's what I love to do anytime that I have spare time that's not either working or with family, I'm probably doing security research in some way, shape or form.

22
00:02:24,340 --> 00:02:26,520
So feel free to check out these projects.

23
00:02:26,520 --> 00:02:28,120
I have a blog on Medium.

24
00:02:28,440 --> 00:02:35,660
Any code, any examples, any demos, the slides, everything from this presentation will be available in GitHub.

25
00:02:35,660 --> 00:02:37,160
So you can start checking that out.

26
00:02:37,160 --> 00:02:43,220
If you want to follow along with anything, a lot of the content, the resources, the notebooks that we go through will be available there.

27
00:02:43,220 --> 00:02:48,720
So I encourage you pull it up while I'm talking, do that, check things out, and feel free to ask questions as we go along.

28
00:02:48,720 --> 00:02:51,240
But thanks again, and let's jump right in.

29
00:02:55,800 --> 00:02:57,960
So what are we actually going to cover?

30
00:02:57,960 --> 00:03:04,780
For the agenda today, there's really a couple of things, but I really want to focus on inputs and outputs.

31
00:03:04,780 --> 00:03:05,780
I think it's a mindset.

32
00:03:05,780 --> 00:03:08,780
We always talk about the hacking mindset, the security mindset.

33
00:03:08,960 --> 00:03:13,000
Well, in this case of this talk and the scope, I want you to think about inputs.

34
00:03:13,000 --> 00:03:14,140
What are the objectives?

35
00:03:14,140 --> 00:03:15,140
What are the goals?

36
00:03:15,140 --> 00:03:16,760
What do you put in?

37
00:03:16,820 --> 00:03:18,100
And then what are those outputs?

38
00:03:18,100 --> 00:03:23,400
How can we normalize these outputs so that way you don't have to try to remember how do you do 500 different scripts?

39
00:03:23,400 --> 00:03:25,140
How do you make these tools work?

40
00:03:25,140 --> 00:03:28,340
How do I manually parse through all these outputs and data?

41
00:03:28,360 --> 00:03:35,020
So throughout this, we're going to show kind of modular, repeatable technology, agnostic designs.

42
00:03:35,260 --> 00:03:38,820
And then how do you apply this from a cloud-focused technology?

43
00:03:38,820 --> 00:03:42,420
And we're going to spend our time in the AWS ecosystem.

44
00:03:42,420 --> 00:03:46,140
However, you really could translate this to any of the major cloud providers.

45
00:03:46,240 --> 00:03:48,360
And then also, we're going to provide some solutions.

46
00:03:48,360 --> 00:03:52,380
So how do you tie this all together so that way you can become more efficient?

47
00:03:52,380 --> 00:03:59,160
And I'm hoping that if you watch this talk, you apply these concepts, your key takeaways are I want you to make yourself better.

48
00:03:59,160 --> 00:04:01,160
I want to make the people around you better.

49
00:04:01,160 --> 00:04:04,740
And I'm hoping that overall, we can all work together and make the industry better.

50
00:04:08,000 --> 00:04:10,480
So this is the story of my life.

51
00:04:10,480 --> 00:04:14,620
Thank you, Jason, for sharing these tweets because the timing was perfect.

52
00:04:14,620 --> 00:04:23,880
And I started thinking in terms of, and this is in the scope of Bug Bounty, if you've ever worked on Bug Bounty, you see these different tweets of I just made $5,000.

53
00:04:23,880 --> 00:04:25,580
Or I made a $15,000 bug.

54
00:04:25,580 --> 00:04:27,100
Or I just made $400.

55
00:04:27,980 --> 00:04:33,460
It's kind of, that's how I always feel, is wow, let's just spend a couple hours, let's make a couple thousand bucks.

56
00:04:33,460 --> 00:04:34,620
But that's not the case.

57
00:04:34,620 --> 00:04:48,100
And I'm probably, I don't know if this resonates with any of you, but I'm the person that if you were to ever see me tweet I just made $2,000 on a Bug Bounty, that's because I just probably put in a hundred hours trying to research through reconnaissance,

58
00:04:48,100 --> 00:04:53,720
through tooling, through tests and failures and inputs and report writing just to make that.

59
00:04:53,720 --> 00:04:56,640
And that's what goes on, I think, often behind the scenes that we overlook.

60
00:04:57,040 --> 00:05:02,000
And one of the things is, too, is our most valuable commodity is time.

61
00:05:02,040 --> 00:05:20,420
My biggest challenge is finding the time to be able to do this, to really get in the weeds, to dig in and try to find these red team objectives to relearn, I'm always relearning things or figuring out or looking up or Googling or all the places where I'm trying to relearn all the things I've learned historically.

62
00:05:20,580 --> 00:05:21,700
How can we make it faster?

63
00:05:21,700 --> 00:05:23,360
How can we be more efficient?

64
00:05:23,400 --> 00:05:28,160
So when I have these two or three hours in an evening to spend, I can be focused.

65
00:05:28,160 --> 00:05:29,320
I can look at my targets.

66
00:05:29,320 --> 00:05:31,100
I know exactly what I want to do.

67
00:05:31,100 --> 00:05:38,680
I don't have to spend hours doing the reconnaissance research trying to just make selections on what my targets are because that takes up all my time.

68
00:05:38,680 --> 00:05:41,560
And then the time to value is really zero and net none.

69
00:05:41,580 --> 00:05:43,660
I'm going to focus on how do I be productive?

70
00:05:43,660 --> 00:05:44,860
How do I make money?

71
00:05:44,860 --> 00:05:46,140
And how do I find issues?

72
00:05:46,140 --> 00:05:48,940
And further, what I'm attempting to do.

73
00:05:48,940 --> 00:06:00,600
So that's what really drew me into how do I build this automated ecosystem and take an advantage of the new technologies and new capabilities out there so that I can be the best that I possibly can.

74
00:06:04,120 --> 00:06:13,180
So now let's jump into the architecture of the underlying design of the entire automated ecosystem that we're going to build and talk about through the remainder of this presentation.

75
00:06:13,200 --> 00:06:25,600
We'll have a bunch of demos and code snippets and things that we're going to kind of deep dive in and hoping that you can leverage these as accelerators so they can be agnostic to a specific tool, a specific process.

76
00:06:25,600 --> 00:06:40,460
But as you look at different things that you do and experience from a Red Team perspective or even just from a general information security process or practice that you do in your daily jobs, these are all components that can apply in a lot of different ways,

77
00:06:40,460 --> 00:06:41,780
which I think is really exciting.

78
00:06:41,780 --> 00:06:59,980
And that's a big part of why I feel like rather than just trying to deliver this custom tool that is just a script or something that you run and you routinely run it and it does the same thing every time, what I want to kind of teach is that thinking in terms of how do you build that underlying infrastructure that can support anything that can be thrown at you?

79
00:06:59,980 --> 00:07:11,880
How can you better yourself and understand rather than you kind of move up from knowing just how to run a tool to actually understanding what's going on behind the scenes, how do I piece this together and how do I make it my own?

80
00:07:11,880 --> 00:07:21,020
So I have this system that can be potentially better than others or bring you to the best in the industry or just solve challenges that you have on a day-to-day basis.

81
00:07:21,240 --> 00:07:25,760
So as we look at this architecture, it's really broken down into three key pieces.

82
00:07:25,760 --> 00:07:29,020
And on the left side is the user interaction.

83
00:07:29,140 --> 00:07:35,840
So these are the components of the cloud services that you're going to generally interact with, whether it's an input, whether it's an output.

84
00:07:36,060 --> 00:07:40,480
But in the design that I have that we're going to walk through, it's going to leverage these four pieces.

85
00:07:40,480 --> 00:07:48,000
And we're going to, as we go further, I'm going to give you a quick overview of every one of these components just to explain what it does, where it's used for.

86
00:07:48,040 --> 00:07:54,700
And in the back of your mind, continue to think, how can I reuse this for X function or what I'm trying to do or my daily job?

87
00:07:54,700 --> 00:07:58,620
And how can I convert my manual efforts that I do on a day-to-day basis?

88
00:07:58,620 --> 00:08:04,040
How can I save time, be more efficient, and automate this so that way I can make my life better?

89
00:08:04,960 --> 00:08:11,120
Basically, you can automate yourself and then you can focus more on the latest, greatest technology emerging.

90
00:08:11,120 --> 00:08:20,560
Push your program forward, mature in everything you're doing, and just continue to build and build on top so we're not relearning, we're not doing the same manual tasks over and over.

91
00:08:20,580 --> 00:08:30,680
And this ecosystem will really elevate that so that way you can do and you can build and it's modular and it's really an ecosystem of a lot of microservices in doing that.

92
00:08:30,800 --> 00:08:33,320
The middle layer is your processing and computation.

93
00:08:33,320 --> 00:08:37,960
So you have your user interaction that will pass everything over to your processing and computation.

94
00:08:37,960 --> 00:08:41,440
And this is a pretty big arsenal of different tools.

95
00:08:41,440 --> 00:08:58,780
You have, whether it's big data type, query analytics services, it could be eventually, we're not going to show any of these today, but if you get into machine learning or image recognition and thinking in terms of future opportunities of like you have screenshots of webpages,

96
00:08:58,780 --> 00:09:04,360
maybe you want to process a bunch of images of those really quickly, this would be your processing and computation layer.

97
00:09:04,460 --> 00:09:14,460
And then the last layer, I think is pretty key and it's really important to get this in the mindset as you kind of consider future opportunities, is that data storage layer.

98
00:09:14,540 --> 00:09:21,660
You look at today, a lot of the processes you may have, like if you're doing reconnaissance, you might run five or six different tools.

99
00:09:21,660 --> 00:09:28,380
In those inputs, you're going to submit, maybe it's a domain, maybe it's an ASN, maybe it could be a CIDR range.

100
00:09:28,380 --> 00:09:29,780
It could be various things.

101
00:09:29,780 --> 00:09:30,700
You input that.

102
00:09:30,700 --> 00:09:35,400
Generally, the outputs of your various tools and scripts, they're going to be slightly different.

103
00:09:35,400 --> 00:09:38,320
And that's where that source data bucket comes into play.

104
00:09:38,320 --> 00:09:42,600
So I expect, anticipate, let's pull in 100 tools eventually together.

105
00:09:42,600 --> 00:09:48,000
Let's all put that data, source data into that source data bucket that's just various formats and everything.

106
00:09:48,300 --> 00:09:51,580
And then what we want to do is we want to think in terms of how do we normalize this?

107
00:09:51,580 --> 00:09:53,420
What's the actual objective of this?

108
00:09:53,420 --> 00:10:08,020
Because in a lot of cases, I mean, and we're going to look from kind of a red team bug bounty perspective, but think broader as you think about your actual job, whether you're doing risk management, whether you're doing third party analysis, whether you are a security engineer,

109
00:10:08,020 --> 00:10:11,660
whether you're an architect, any role in this, think about how this can apply.

110
00:10:11,680 --> 00:10:13,240
But we look at our inputs and outputs.

111
00:10:13,240 --> 00:10:16,940
So a lot of our tools, they're either going to output things such as URLs.

112
00:10:16,940 --> 00:10:21,420
It might be parameters for a different web page.

113
00:10:21,420 --> 00:10:24,620
You could have the ASN numbers for network ranges.

114
00:10:24,620 --> 00:10:25,740
You could have CIDR ranges.

115
00:10:25,740 --> 00:10:27,080
You could have domain names.

116
00:10:27,080 --> 00:10:28,140
You could have subdomains.

117
00:10:28,140 --> 00:10:29,580
You could have company names.

118
00:10:29,580 --> 00:10:34,000
You could have certificate assignments, different inputs and outputs.

119
00:10:34,000 --> 00:10:38,040
So those are all of your various outputs that you're generally going to have across your tools.

120
00:10:38,340 --> 00:10:50,180
We need to make these common between that so we can further process this and make this data useful because I know one of the areas that I struggle with the most is parsing through this data of every tool and it just takes so much time.

121
00:10:50,220 --> 00:10:55,980
And that's something that we could, if you do it one time for a tool where you normalize the data, you never have to do it again.

122
00:10:56,100 --> 00:11:04,380
And the cooler part about this is we only need one person in the industry to do it for per tool or start to kind of standardize and build in.

123
00:11:04,480 --> 00:11:08,720
And that's where on my GitHub you'll see Project Straylight.

124
00:11:08,900 --> 00:11:17,040
And my goal is really to just have this running open network of here's accelerators we can do, here's code snippets that have helped me along the way.

125
00:11:17,040 --> 00:11:31,220
And it's maybe not so much a technology or tool that you install, but it's just a bunch of ideas and things that have helped me in my career that over the past 10 to 12 years that I've either had to relearn or do, I'm trying to aggregate it all into this ecosystem,

126
00:11:31,220 --> 00:11:51,340
this architecture that we see of how can I do what I've spent so many hours doing redundantly, manually, and how can I make it better so that way the newer generation, the next people that are going to be experts, leaders in this, can just build on top of what we already have so we're not continuing to rebuild and we can advance further and further and we can win in this industry.

127
00:11:51,340 --> 00:11:53,680
So that's a lot of the intent there.

128
00:11:53,680 --> 00:12:01,240
So as we look at the data storage layers, you basically go from your raw data, which could be the output of any tool, you get to more of a normalized version.

129
00:12:01,580 --> 00:12:05,300
And then you also get into your presentation layer.

130
00:12:05,300 --> 00:12:11,780
And that's when it's your reporting, it's your analysis, it's your metrics, it's the outputs that you want to show for the delivery of your work.

131
00:12:11,780 --> 00:12:14,520
And this could speak to any level depending on how you present it.

132
00:12:14,520 --> 00:12:23,900
So you could do an aggregate of here's all the attacks that we've had over the course of X days, summarized into charts, graphs, and that might be at more of a CISO level.

133
00:12:23,900 --> 00:12:30,620
You could have indicators of compromise or different signatures, different levels, which you would feed into an engineering team.

134
00:12:30,620 --> 00:12:32,340
That could be part of your presentation.

135
00:12:32,340 --> 00:12:47,200
But it's how do you normalize and how do you show this in kind of a formatted way so that way if you have to do monthly vulnerability reports or metrics or showing analysis of how you've further progressed or analyzed websites or targets and you want to compare what you've done,

136
00:12:47,200 --> 00:12:50,340
what you haven't done, that's where that presentation layer comes in.

137
00:12:50,440 --> 00:12:53,420
So we're going to jump in and I think we're going to do, let's do a demo.

138
00:12:53,420 --> 00:12:59,480
And let's kind of show, I'll show from end to end kind of one area where we can go across this of how this ecosystem works.

139
00:12:59,480 --> 00:13:02,280
And then after that, we'll dive in further.

140
00:13:02,280 --> 00:13:05,040
I'm going to show you the accelerators, the pieces, the components.

141
00:13:05,060 --> 00:13:07,860
Keep in mind that all of this is also available in GitHub.

142
00:13:07,860 --> 00:13:09,300
You can access it as we go along.

143
00:13:09,300 --> 00:13:10,040
You can check it out.

144
00:13:10,040 --> 00:13:11,200
Feel free to use it.

145
00:13:13,180 --> 00:13:16,140
Create issues in the GitHub if you find any things.

146
00:13:16,140 --> 00:13:17,360
Reach out to me with questions.

147
00:13:17,360 --> 00:13:23,840
But I really just want to continue to build and teach you how you can kind of think in terms of building your own automated ecosystem.

148
00:13:25,480 --> 00:13:29,640
As you're watching this too, you're probably thinking, why cloud?

149
00:13:29,640 --> 00:13:31,260
I have a really nice home lab.

150
00:13:31,260 --> 00:13:32,620
Do I really need it?

151
00:13:32,620 --> 00:13:39,020
There's three key areas beyond just the amount of what we just walked through in that architecture that you have available to you.

152
00:13:39,020 --> 00:13:43,760
But it's pretty exciting when you think about what cloud can mean to you.

153
00:13:43,760 --> 00:13:45,040
And if you hear...

154
00:13:45,040 --> 00:13:48,220
I mean, the businesses, industries, everybody is adopting it.

155
00:13:48,660 --> 00:13:49,760
That's nothing new.

156
00:13:49,760 --> 00:13:51,160
It's not anything new there.

157
00:13:51,160 --> 00:14:01,980
But I think there's three key parts of cloud that are really exciting as just an individual researcher where you don't need millions of dollars or tens of thousands of dollars of funding behind you to actually make progress.

158
00:14:02,040 --> 00:14:06,200
The first part is the democratized accessibility of datasets.

159
00:14:06,420 --> 00:14:10,240
There's some pretty massive and interesting datasets out there.

160
00:14:10,240 --> 00:14:18,920
The problem is that on our home labs, our home computers, our home servers, a lot of those don't have either the horsepower to accomplish what we need.

161
00:14:18,920 --> 00:14:29,280
We need more memory, more CPU, or it's not close enough to the data where we can't download petabytes of data or terabytes of data in a meaningful way to keep up and be able to analyze and do.

162
00:14:29,280 --> 00:14:32,520
And that's where you kind of run into issues of data gravity.

163
00:14:32,520 --> 00:14:38,040
And you need to build your compute actually closer to the data stores, the data repositories, because it's costly.

164
00:14:38,040 --> 00:14:39,700
It's expensive to move those.

165
00:14:39,700 --> 00:14:42,380
So how can we proceed with that?

166
00:14:44,120 --> 00:14:52,520
The other piece is that if you look across the world, we're facing right now, we're all in quarantine, we're social distancing.

167
00:14:52,540 --> 00:14:52,900
You have...

168
00:14:53,520 --> 00:15:01,720
So take a look at what's going on around the race to find vaccines and solutions to COVID and kind of the data around that in the pieces.

169
00:15:01,820 --> 00:15:10,100
The world is coming around cloud and the latest, greatest computational technologies to solve the world's biggest problems that impact everybody globally.

170
00:15:10,160 --> 00:15:19,480
So what if we can take those pieces that the top scientists in the world, the top in their industries are taking to solve the world's most challenging problems?

171
00:15:19,520 --> 00:15:26,020
I'm sure we can apply that to our daily bug bounty targets or our daily security processes that we need to improve.

172
00:15:26,020 --> 00:15:29,540
But in the same way, we don't have to spend tens of thousands of dollars for this.

173
00:15:29,540 --> 00:15:33,080
The other neat thing about it is it's financially viable.

174
00:15:33,180 --> 00:15:39,100
We can now stand up servers that have 128 gigabytes of memory.

175
00:15:39,100 --> 00:15:42,100
You can do unlimited storage.

176
00:15:42,100 --> 00:15:46,500
And the nice thing is you don't have to pay thousands of dollars to buy this from a capital standpoint.

177
00:15:46,500 --> 00:15:48,240
It's all operating expense.

178
00:15:48,240 --> 00:15:58,360
So as you look at it, if you need to run something that's extremely memory intensive and it's just one operation or a loop, you literally could stand up a massive server for just a couple minutes.

179
00:15:58,360 --> 00:16:03,380
It might cost you $10, $20 for maybe five or ten minutes of processing.

180
00:16:03,620 --> 00:16:14,640
But rather than needing to invest $80,000 in similar hardware equipment, you now can do this in your home office, in your home lab through the cloud capabilities charging.

181
00:16:14,640 --> 00:16:20,120
And that $20, if you think about it in terms of bug bounty investment, that's one P4.

182
00:16:20,120 --> 00:16:22,380
And you have that well covered.

183
00:16:22,380 --> 00:16:24,820
I mean, that's on the low end thinking in terms of that.

184
00:16:24,820 --> 00:16:39,760
But just looking at it from a business perspective, thinking in terms of how can I continue to invest in my capabilities, the technologies, the processes, so I can gain more time and I can make the time that I spend actually manually doing things, evaluating parts,

185
00:16:39,760 --> 00:16:42,160
how can I make it the most beneficial to myself.

186
00:16:47,700 --> 00:16:51,740
So now let's take a look at the various tooling that we can use.

187
00:16:51,740 --> 00:16:53,380
We've looked at the architecture diagram.

188
00:16:53,380 --> 00:17:02,780
I'd like to just give a quick overview of the different components, especially if you're not familiar with the AWS services that are listed on here, kind of what they do, why they're important and how they can apply to you.

189
00:17:02,860 --> 00:17:09,360
So if we go through the list, you saw from the user interaction space the Amazon API Gateway.

190
00:17:09,480 --> 00:17:12,900
So what this does is this is just an entry point into your ecosystem.

191
00:17:12,900 --> 00:17:23,200
And the neat thing about Amazon API Gateway is there's a lot of things you can do with it, but where I've really leveraged it is as a proxy into your Lambda function.

192
00:17:23,200 --> 00:17:32,020
So what you can do is you can do basically a GET request or a POST request through the API Gateway and pass that directly into an AWS Lambda function.

193
00:17:32,160 --> 00:17:44,680
And if you think in terms of what you've seen in the Jupyter notebooks and Python libraries and any functions you run where you pass something, you literally can pass it through a GET request through the API Gateway into AWS Lambda.

194
00:17:44,680 --> 00:17:58,560
So I kind of view that as top tier level of services where you've really normalized your content, you have exception handling, you know that you're ready to completely automate this, let's promote it into a Lambda function, pull it out of your Jupyter notebook,

195
00:17:58,560 --> 00:18:07,580
and then let's put it behind an API Gateway so you can really kick it off and call it ad hoc as you need and really start your ecosystem of automation around there.

196
00:18:08,340 --> 00:18:19,400
Another tool is Amazon SageMaker and it's really a larger framework and compute system where you can do more things like machine learning and training and all of that.

197
00:18:19,400 --> 00:18:22,740
But what I pretty much solely use it for are the Jupyter notebooks.

198
00:18:22,740 --> 00:18:42,140
So you can stand up the Jupyter notebooks, you can run it, you can adjust the size on it so if you need a lot more gigabytes or storage for a duration or a function, a lot of times I'll ramp up and use a larger instance of it for just a few minutes or just to do that process and then I'll turn it down to kind of the standard level.

199
00:18:42,140 --> 00:18:46,600
So Amazon SageMaker is really the tool mechanism for running the Jupyter notebooks.

200
00:18:46,900 --> 00:19:02,060
The static website hosting S3, that's an S3 bucket, you can actually set those up so that they can host web content, make them publicly available so you hit it, you can set an indexed HTML page, you can run it, and you can run websites out of S3 buckets which is really neat.

201
00:19:02,940 --> 00:19:11,860
The Amazon simple notification service, it's a way that you can send push messages to whether it's SMS text, whether it's emails, different things.

202
00:19:11,860 --> 00:19:14,600
So it's really just a notification service that we can do.

203
00:19:14,600 --> 00:19:25,700
In this case, I'll show a code snippet where we can actually send ourselves text so that way we know if there's anything that's going to take a little bit of time or you just want to get notified or have the final URL where you're going to access your data,

204
00:19:25,700 --> 00:19:27,560
you can get texted whenever it's ready.

205
00:19:27,560 --> 00:19:33,420
So you can kind of do bug bounty, reconnaissance, all of this on the fly as you build this system which is pretty cool.

206
00:19:35,140 --> 00:19:43,200
Now as we move into the processing and computation layers, AWS Lambda is really what's used to run code.

207
00:19:43,200 --> 00:19:57,220
So you can set it up, it runs a lot of different things, it's not just Python, but it's Node, and it runs on a serverless compute so really you put the code in there, it will stand up the compute resources and it only runs for a duration.

208
00:19:57,220 --> 00:20:04,700
I think the longest that you can run a Lambda is about 15 minutes so you're charged based on the duration that the computation is running.

209
00:20:04,780 --> 00:20:14,340
So if you think in terms of smaller Python functions that you have ready to automate and scale, that's where you can migrate them to the Lambdas and they can just always be ready to work off of triggers.

210
00:20:15,560 --> 00:20:28,480
The Amazon Elastic Block Storage, a lot of times, S3, there's a little bit of a caveat where S3 you get charged on get requests and put requests and writes, whereas EBS storage you pay for how much storage that you have assigned and allocated.

211
00:20:28,480 --> 00:20:39,680
So depending if I think I'm going to do millions of reads or writes, sometimes I'll do a lot of my testing against Elastic Block Storage and then once I have something that I know is ready to go robust and I want to leverage it in the greater ecosystem,

212
00:20:39,680 --> 00:20:50,020
I'll move it over to our data storage tiers while they're in the refined or the raw data levels to do more normalized and automated operations against.

213
00:20:50,860 --> 00:20:54,540
AWS Glue is a way that you can set up and you can crawl data sets.

214
00:20:54,540 --> 00:21:09,320
So Athena is, you picture that, is like you write your SQL queries and your query language and it will run it and distribute it against data sets and it can run against CSV files, it can run against relational data stores, unstructured data, and all those.

215
00:21:09,320 --> 00:21:14,420
So it's really powerful and if you haven't ever dug into it, AWS Athena, I highly recommend it.

216
00:21:14,420 --> 00:21:23,720
There's a lot of power in it and you can literally just sit there and use the console interface and pretty much ad hoc write queries against big data sets if you decide to as well.

217
00:21:23,960 --> 00:21:29,020
And then Glue is, I started talking, it's what indexes a lot of those data points around it.

218
00:21:29,020 --> 00:21:35,420
So you can create Glue indexes so you don't have to query the entire data set every single time and you can do those.

219
00:21:36,360 --> 00:21:49,340
AWS Step Functions, you won't see anything in the demos today leveraging this, but this is kind of a future capability where I want to leverage where you can actually, it's like setting up workflows and processes and you need X to happen and then once X completes,

220
00:21:49,340 --> 00:22:01,460
you can send the output to this one and do, right now, I'm just leveraging things off of triggers and alarms and Lambda files, but you can really start to use AWS Step Functions as orchestration, as a central core piece.

221
00:22:01,460 --> 00:22:03,840
So something I'm going to dig into more in the future.

222
00:22:04,620 --> 00:22:11,700
AWS Secrets Manager is a place that, so you can move all of your secrets, your API keys, out of your code, put them in Secrets Manager.

223
00:22:11,700 --> 00:22:13,280
It's a quick call out.

224
00:22:13,280 --> 00:22:18,080
You can load them, you can store them, and you can kind of store other parameters in there as you need as well.

225
00:22:18,080 --> 00:22:22,760
You can pull them back into your code so they don't sit as code, but you can use them as variables on the fly.

226
00:22:22,760 --> 00:22:24,200
So I highly recommend digging in there.

227
00:22:24,200 --> 00:22:27,120
There's some accelerators that you'll see that talk to those.

228
00:22:27,760 --> 00:22:37,380
And then on the right side are just all the S3 buckets and you can create multiple S3 buckets and have the different stores and it's just long term data storage for you.

229
00:22:40,180 --> 00:22:42,220
So let's jump in with a demo.

230
00:22:42,540 --> 00:22:51,480
This demo is going to show how we can query Rapid7's Project Sonar, which is essentially it's the 4DNS of the entire internet.

231
00:22:51,480 --> 00:22:54,160
And it's sitting in a dataset within S3.

232
00:22:54,160 --> 00:23:11,200
So rather than try to copy that entire dataset down into our own system, we're going to leverage a Jupyter Notebook and we're going to actually run a distributed query against that external dataset that's hosted in not our own S3 bucket, but the public datasets bucket.

233
00:23:11,380 --> 00:23:14,520
And we're going to get the results back and we're going to process it.

234
00:23:14,520 --> 00:23:16,740
And we're going to do it in just a matter of...

235
00:23:16,740 --> 00:23:26,440
it's probably going to take a couple of minutes as we walk through and talk through the code, but this is literally, I think generally the queries around the 4DNS for a domain wildcard is about 26 seconds.

236
00:23:26,660 --> 00:23:28,740
So let's walk through this.

237
00:23:29,540 --> 00:23:42,380
So on the screen here you can see my Jupyter Notebook and what we'll do is we'll walk through this too because I want you to think in terms of how can I reuse these components, how is this applicable well beyond just this sonar use case.

238
00:23:42,480 --> 00:23:43,720
So let's go ahead and start.

239
00:23:43,720 --> 00:23:47,100
We're going to go ahead and run this first section of code here.

240
00:23:47,100 --> 00:23:54,700
And this is what I talked about earlier too, which is nice is that if you're not an expert at code, you run it, you can break it down into cells, and you can run each cell.

241
00:23:54,700 --> 00:23:56,440
So we're going to run this first cell.

242
00:23:56,520 --> 00:24:00,280
What we just did was you can see this execution ID here.

243
00:24:00,300 --> 00:24:10,080
What we did is we set our domain to Microsoft.com so we're actually querying all the wildcards of the Microsoft domain 4DNS space.

244
00:24:10,080 --> 00:24:13,580
And keep in mind, we are not touching any of Microsoft's systems.

245
00:24:13,580 --> 00:24:19,440
This is purely utilizing the 4DNS dataset and we're getting the results from there.

246
00:24:19,800 --> 00:24:28,700
And then what we've done is we've set some of our variables around our bucket of where we want to store the results, what database we want to query against in Athena, which we've had some...

247
00:24:28,700 --> 00:24:33,120
the setup information is all contained within the GitHub as well so you can set that up.

248
00:24:33,120 --> 00:24:35,340
And then the table we want to query against.

249
00:24:35,580 --> 00:24:38,680
What we do next is we set up our query.

250
00:24:38,680 --> 00:24:53,260
So what this is doing is basically just saying select everything from wildcard microsoft.com and then we select the latest date of the dataset so that way we don't historically go through every single one iteration that we have.

251
00:24:53,880 --> 00:24:56,640
It issues the Athena query.

252
00:24:56,640 --> 00:24:58,400
It takes about 26 seconds to run.

253
00:24:58,400 --> 00:25:00,740
We have the Athena query ID here.

254
00:25:00,740 --> 00:25:04,560
So in this next component, I will actually jump back up.

255
00:25:04,560 --> 00:25:10,500
This run command that I'm highlighting, this is actually a way that you can call other notebooks, which is kind of neat.

256
00:25:10,500 --> 00:25:16,440
So as you think about in terms of you call functions within programming code, this will actually run and load an entire notebook.

257
00:25:16,440 --> 00:25:24,640
So what I've done is I try to take anything that's not tool specific and I put it in kind of a common notebook that you can set a lot of different things to run.

258
00:25:24,640 --> 00:25:35,480
So what this is actually doing is the query Athena function is running out of that other notebook because I've set it up well enough that you can pass in the results bucket.

259
00:25:35,480 --> 00:25:41,300
You just send it the query in the Athena database and it will kick it off so it doesn't matter what tool specific or anything.

260
00:25:41,300 --> 00:25:48,460
It will just run a query that you tell it to run which is why I moved it into that kind of standardized function area.

261
00:25:49,800 --> 00:25:54,040
So what we're going to do is we're going to run this next one and this is going to generate, pull the results back.

262
00:25:54,040 --> 00:25:55,780
So you can see that it's already succeeded.

263
00:25:55,780 --> 00:26:01,500
If we were to run this immediately it would just go in a loop and it would circle every five seconds and check for the results back.

264
00:26:01,640 --> 00:26:08,720
But it's pulled back 18,695 different rows and you can see the types of information we've just pulled back.

265
00:26:08,720 --> 00:26:22,280
So think you have 18,695 different rows basically for DNS entries from Microsoft.com in under 26 seconds with the IP addresses, the C names, A records, MX records, all of those.

266
00:26:23,000 --> 00:26:35,300
And then with that we're going to do this next thing is this is actually going to leverage the MaxMind database and we're going to start figuring out querying for the IP addresses and getting all that information back.

267
00:26:35,300 --> 00:26:41,160
So let's go ahead and run this and what this will do is it creates a data frame which leverages Pandas.

268
00:26:41,160 --> 00:26:46,760
So Pandas is a library within Python that you can run and I completely recommend it.

269
00:26:46,760 --> 00:26:51,680
It is one of the best things that I've ever found in terms of doing metrics and analysis and automation.

270
00:26:51,680 --> 00:26:56,120
But you can see it just processed 18,695 rows in just a few seconds.

271
00:26:56,120 --> 00:27:02,480
And now we have the latitudes, longitudes, countries, and localities all merged to all 18,000 entries.

272
00:27:03,140 --> 00:27:11,660
So what we're going to do now is if you say, hey, I just did all this work, I don't want a chance of losing it even though theoretically you could redo this over about 30 seconds.

273
00:27:11,740 --> 00:27:18,480
But if you want, you can use this command and you can save your data frame off to Excel so you can save it locally.

274
00:27:18,480 --> 00:27:25,700
So if you run this, it will just print and store those 18,000 records into an Excel spreadsheet which you can reload later on which is pretty handy to use.

275
00:27:25,700 --> 00:27:29,860
And if you want to save your artifacts and data points, you can always do that.

276
00:27:31,480 --> 00:27:35,660
So what we'll do now is we're going to run this next command and what this does is it just aggregates.

277
00:27:35,660 --> 00:27:36,800
And you can see how fast that was.

278
00:27:36,800 --> 00:27:45,720
It was almost instantaneous when I pressed the start button and it just grouped up all the different latitudes and longitudes so that we can get ready to plot them on a heat map.

279
00:27:46,040 --> 00:27:51,580
So we know 525 had this latitude, longitude, and it did all that work.

280
00:27:51,580 --> 00:27:52,280
And we can dig in.

281
00:27:52,280 --> 00:28:00,300
This also calls that the other notebook because it's centralized and it's not specific to the project sonar data set.

282
00:28:00,300 --> 00:28:03,480
So you can reuse and redo any of that.

283
00:28:03,660 --> 00:28:05,000
I am going to jump back up, though.

284
00:28:05,000 --> 00:28:07,040
I think this is something that's pretty neat.

285
00:28:07,040 --> 00:28:23,840
And if you think about in terms of how can I reuse some of this, this get location function that I have behind the scenes that just pulled in and merged all this together, what I did, if you're looking, I passed the entire data frame as a part of the function there and I passed the value column,

286
00:28:23,840 --> 00:28:25,480
which is the IP address column.

287
00:28:25,480 --> 00:28:36,180
So if you think about reusability, all I literally did was I just took, if you have an Excel spreadsheet of IP addresses or a CSV or anything like that, you load it into a data frame.

288
00:28:36,180 --> 00:28:37,940
You can pass that entire data frame.

289
00:28:37,940 --> 00:28:38,540
It doesn't matter what it is.

290
00:28:38,540 --> 00:28:39,740
It doesn't matter what the rest of it looks like.

291
00:28:39,740 --> 00:28:51,260
You pass the data frame name and the column that the IP addresses are in or obviously it will fail on if it's not an IP address as it goes through those because they're C names.

292
00:28:51,700 --> 00:28:53,660
And you do that and you can process it.

293
00:28:53,660 --> 00:28:57,660
So anything you have, that's completely reusable and that's just a few lines of code.

294
00:28:57,660 --> 00:28:58,200
It's in GitHub.

295
00:28:58,200 --> 00:29:06,900
You can pull it and I think in terms of just from a cyber operations, a SOC perspective, you could really get a lot of value out of some of that information.

296
00:29:06,900 --> 00:29:16,460
You could look at attackers, web denials, web traffic, who's hitting your web application firewalls and you can do heat maps and different things from an attacker perspective which is pretty cool.

297
00:29:17,660 --> 00:29:18,900
So let's go on down.

298
00:29:18,900 --> 00:29:22,980
We have that information and then we want to now, let's do this.

299
00:29:22,980 --> 00:29:26,140
So this is just a couple lines of code but it's really cool .

300
00:29:26,140 --> 00:29:29,520
So this uses the Google Maps API and you can see how fast that was.

301
00:29:29,520 --> 00:29:46,140
It just plotted the information we have up here for the latitudes and longitudes and created an interactive heat map so we can zoom in on and see what's going on and this shows the concentrations of Microsoft's forward DNS space across the globe.

302
00:29:46,140 --> 00:29:51,500
So you can look and as we zoom in, you can see the different regions they are included in.

303
00:29:52,260 --> 00:30:03,440
So let's, if we were just, let's say, let's tackle this is a big area here, you can zoom in and even though it shows Seattle space, you can go and see that it's not all just in Seattle but they have a few offices.

304
00:30:03,440 --> 00:30:06,540
It looks like in Redman, there's some Seattle and above.

305
00:30:06,540 --> 00:30:10,180
So pretty cool that it's all interactive and you can do that.

306
00:30:10,180 --> 00:30:13,980
And the other neat thing is you're thinking, well, I want to show this to management.

307
00:30:13,980 --> 00:30:17,000
How do I show my managers, my leadership about this?

308
00:30:17,020 --> 00:30:20,980
And I can't have them just opening a Jupyter notebook all the time and doing this.

309
00:30:20,980 --> 00:30:25,580
Well , one thing is you can click the download button and just download the map as an image.

310
00:30:25,580 --> 00:30:42,220
But the other pretty neat thing about all this, and if you think about the architecture of how I have that presentation layer, what this next command does is it takes this map and it actually uploads it to a static S3 bucket that's hosting a static S3 content as a website .

311
00:30:42,380 --> 00:30:53,320
So then I hit this, I uploaded the map file to the website and then I can navigate over to my bucket and if I hit load content right here , it should pull up that map.

312
00:30:54,040 --> 00:31:00,580
And now I can always send any of my leadership, any managers , anybody that wants to see anything , I can run this on an iterative basis while I'm there.

313
00:31:00,580 --> 00:31:03,340
It's on every 24 hours I run this process and do.

314
00:31:03,540 --> 00:31:09,860
And if you think in terms of visually what you want your attack surface to look like for your company, run this against your company every time.

315
00:31:09,860 --> 00:31:14,820
Have this dashboard always going and then you could even trigger on anomalies, you could do different things.

316
00:31:14,900 --> 00:31:20,880
And it's just pretty neat to always see here, here's our coverage from an internet presence that we have there.

317
00:31:20,880 --> 00:31:24,020
So I think there's a lot of value, a lot of opportunity to do in there.

318
00:31:24,020 --> 00:31:36,740
And just the fact that you can literally load it and save the interactive mapping part of it to an HTML file and then have people actually go up, browse to it, navigate it, that's pretty exciting.

319
00:31:41,770 --> 00:31:49,690
So for this next demo, we're going to use, this is actually one of my favorite ones and I'm really excited to continue the build on the capabilities of this.

320
00:31:49,690 --> 00:31:54,710
But this leverages a data set that's publicly available in S3 called Common Crawl.

321
00:31:54,710 --> 00:32:08,490
And what it is, it's a project that's been going on now for, I think it's actually been around for about eight years, where it crawls the internet, searches and indexes and actually downloads the HTML of files and stores them in S3 buckets.

322
00:32:08,490 --> 00:32:13,930
So it's just a massive repository and similar of basically web archives that you can search.

323
00:32:13,930 --> 00:32:17,730
So if you've used Wayback Machine and different components , this is very similar.

324
00:32:17,990 --> 00:32:20,970
And so what we're going to do is we're going to actually search it for a domain.

325
00:32:20,970 --> 00:32:31,110
We're going to pull back the files and it actually can eventually download the files and we can store them in an S3 bucket and then we can run a bunch of different tools around it.

326
00:32:31,110 --> 00:32:44,070
So if we want to do like find URLs or run HackCrawler or anything that's going to go through and does a good job of analyzing HTML, whether it's for vulnerabilities , whether it's for URLs, whether it's for parameters , we can do that against it.

327
00:32:44,070 --> 00:32:45,730
So let's walk through this now.

328
00:32:45,730 --> 00:32:50,050
I'll kind of show you the framework around it and there's a lot that we can build into this capability .

329
00:32:50,050 --> 00:33:03,490
The biggest challenge that I've had so far is just trying to do some of the asynchronous programming against it because a lot of times you have a large site where you're trying to do downloads and actually pulling the files, but as we get more efficient with this,

330
00:33:03,490 --> 00:33:06,550
I'm pretty excited about this and I think it has a lot of promise with it.

331
00:33:07,490 --> 00:33:10,030
So what we're going to do is we have all of our parameters set up.

332
00:33:10,030 --> 00:33:20,470
We're actually going to run this one against the smaller domain for demo purposes because if we have something that has 70,000 websites and pages, it's just not fast enough at this point to download this in a timely fashion.

333
00:33:20,470 --> 00:33:22,890
We would be sitting here for maybe 40 minutes or an hour.

334
00:33:22,890 --> 00:33:28,470
So in this case, we can always speed it up as we go, but we're going to run this against derbycon.com.

335
00:33:28,970 --> 00:33:31,570
So we've kicked off our Athena query.

336
00:33:31,570 --> 00:33:32,670
It's running it now.

337
00:33:32,670 --> 00:33:33,530
It's searching.

338
00:33:33,530 --> 00:33:41,030
It's looking for all of the webpages that this is indexed and cached and copied and saved within the Common Crawl dataset.

339
00:33:41,430 --> 00:33:54,390
So let's go ahead and kick off our run and what this is going to do is it's going to look and it's going to just wait for the query to stop running and then it will pull in and load the results for us into a database.

340
00:33:54,390 --> 00:33:56,910
So we'll let this run for a few minutes.

341
00:33:57,210 --> 00:33:59,410
All right, so it looks like we have our results here.

342
00:33:59,450 --> 00:34:03,510
So it pulled back 579 different websites.

343
00:34:03,510 --> 00:34:20,930
What we want to do is since it's been running over the course of eight years, there's a lot of different versions , iterations, which is actually pretty neat if you want to see how websites have evolved and look for changes or maybe there's a vulnerability or an issue or a comment in there that they moved out and you could see the difference between the different versions.

344
00:34:20,930 --> 00:34:24,870
But in this case, I'm interested more in just the unique users that we have we have.

345
00:34:24,870 --> 00:34:31,990
So we're going to run this next command which is going to sort them and it's going to drop all the duplicates and it's going to keep the latest version of the webpage.

346
00:34:32,010 --> 00:34:33,530
So let's go ahead and run this right now.

347
00:34:33,530 --> 00:34:34,810
And that was pretty instantaneous.

348
00:34:35,330 --> 00:34:37,510
And now we're down to 73 different rows.

349
00:34:37,510 --> 00:34:41,830
So there were a lot of different iterations, versions of the site cached.

350
00:34:42,110 --> 00:34:49,830
And then what this will do is we can run this next command, we can save it off to Excel so that way we can have our URL listing for future analysis, future loading.

351
00:34:51,050 --> 00:35:02,750
And then as we go on, what this will do is if you want to look, this will pull out and only list now our URLs which is what we care about a lot of times, especially if we're trying to scope, we want to do web crawls.

352
00:35:02,750 --> 00:35:08,130
And keep in mind that everything that we're showing today, everything is completely 100% passive.

353
00:35:08,130 --> 00:35:11,170
We are not touching any of these sites, these webpages or anything.

354
00:35:11,170 --> 00:35:15,430
So this is all examples of what you can do, 100% passive reconnaissance.

355
00:35:15,430 --> 00:35:18,990
So we will never touch a system of any of these domains that we've looked at.

356
00:35:18,990 --> 00:35:30,650
And then for next steps then you start, you can start based on scope and allowances and permissions and things like that to get more intrusive as we do further analysis, whether it's from a red teaming perspective or however.

357
00:35:30,850 --> 00:35:44,730
But what we have right now is just a listing of URLs and then what this next big section of kind of a lot of, a lot of different functions and code , I mean it's still for , for the value of what this does, it's, it's not too bad in terms of lines.

358
00:35:44,730 --> 00:35:48,550
But what this is going to do is it's going to go through now of the query results .

359
00:35:48,550 --> 00:35:52,110
So in this case we're dealing with what, like 73 records , so it's not that many .

360
00:35:52,110 --> 00:36:01,110
But if you were to run something, I think I ran, I think I ran one earlier this year against defcon.org and it came back with about 70,000 records.

361
00:36:01,110 --> 00:36:04,950
So some of these, there's a lot of data and content out there and it can take some time.

362
00:36:04,950 --> 00:36:09,910
And this is where, this is not that efficient, not that streamlined, a lot of improvement opportunity here.

363
00:36:09,950 --> 00:36:16,270
But it, it basically searches , it uses a library in Python called Beautiful Soup, which you can parse down HTML .

364
00:36:16,270 --> 00:36:18,050
It's great for HTML processing.

365
00:36:18,050 --> 00:36:21,970
It has some building capabilities to look for comments and URLs and links.

366
00:36:21,990 --> 00:36:23,390
So we're going to start setting this up.

367
00:36:23,390 --> 00:36:33,490
And what, I have three parameters that we can pass in, just mostly out of efficiency, is we can, we can say we want to search the files and generally we're going to always want to do that.

368
00:36:33,990 --> 00:36:35,290
Do we want to write the files?

369
00:36:35,290 --> 00:36:42,970
So what this does is this goes out and it retrieves, well, it's always going to go out and retrieve the files from the S3 buckets, but do we want to write them?

370
00:36:42,970 --> 00:36:47,690
So in this case, we're not going to write them for this demo purpose, but you could write it to an S3 bucket.

371
00:36:47,730 --> 00:36:51,110
Now you have your HTML files that you can run additional tools and things against.

372
00:36:51,110 --> 00:36:52,510
So it's, it's pretty valuable there.

373
00:36:53,150 --> 00:36:54,730
And then we can say how many records.

374
00:36:54,730 --> 00:37:04,370
So if we just want a subset for testing, if we don't want to pull down all 60,000 at once, we could, we can assign how many records and it will loop through and iterate over and download the number of records we say .

375
00:37:04,410 --> 00:37:08,170
And then we, we run this and this function will basically process these .

376
00:37:08,210 --> 00:37:17,830
It's going through, you can see I have a counter on it because a lot of times when you're just waiting and you get bored and don't know where it's at, so we can, you can always clean this up so it doesn't iterate every single count.

377
00:37:17,830 --> 00:37:28,070
But we've gone through the 73 HTML files we've down, we've processed them as byte strings, we've looked for URLs, the comments, and now we can actually save off the different sections.

378
00:37:28,070 --> 00:37:32,190
So we have, we've grabbed all the comments, we've grabbed all the titles, and we've grabbed all the links.

379
00:37:32,410 --> 00:37:36,870
So if I run this next one, this is actually going to show you the, the listing of links.

380
00:37:36,870 --> 00:37:47,830
And what I, what I like about this is that it actually shows you which website that the links came from as well, which I think is really valuable, especially with sometimes when you do crawls, you don't always know, where did I find this link?

381
00:37:47,830 --> 00:37:48,750
Where did this come from?

382
00:37:48,750 --> 00:37:55,390
And in this case, it does all of the mapping, so if you start building like a bigger mind map or something of it, you can, you can have some value there.

383
00:37:55,450 --> 00:38:05,250
And if we wanted to look at something instead, let's say we wanted to look at the, the comments across it, we can do that, and then we can just print it out.

384
00:38:05,250 --> 00:38:09,090
So this will just print out the top, the first 10 comments we have.

385
00:38:09,510 --> 00:38:11,230
Let's just see what this looks like.

386
00:38:12,490 --> 00:38:15,770
And then there's, there's your list of comments that are within the HTML.

387
00:38:15,770 --> 00:38:24,490
So as you get this, you can start then further looking through there of, you know, of, are there any secrets, are there passwords, you can do regex, you can, you can do different things in there, so it's, it's pretty neat.

388
00:38:25,670 --> 00:38:29,410
I guess that's the, that's the extent right now I think we're going to go through.

389
00:38:29,410 --> 00:38:30,910
I started looking into NetworkX.

390
00:38:30,910 --> 00:38:46,090
I think there's some promise there in terms of getting a larger map at scale, but I don't have a lot of that code done yet, but a lot I want to build on, especially with this functionality because I think there's so much value from even an automated perspective where now we can tie in additional tools that are really good at HTML processing.

391
00:38:46,090 --> 00:38:54,870
We can start to grab JavaScript libraries, codes, external references and all that and do, you can really start building a pretty big map of, of your target surface.

392
00:39:00,710 --> 00:39:02,450
Let's jump into another demo.

393
00:39:02,450 --> 00:39:14,810
This one is going to search the autonomous system numbers, which is part of generally your reconnaissance methodology of where you want to find out the different site or network ranges that are assigned to different companies.

394
00:39:14,810 --> 00:39:29,650
So what you typically do is you would enter in the company name or a keyword for that company and it would return the different site or ranges for that so then you can further do your reconnaissance doing whether you do NMAP scans or any type of searches against those IP ranges.

395
00:39:29,750 --> 00:39:38,670
So what this does is this uses the MaxMind databases as well that we talked about earlier with the geolocation data doing that, but this uses their ASN databases.

396
00:39:39,190 --> 00:39:49,730
And so part of this is we'll walk down through it all, but what it does is if you're looking here at this Jupyter Notebook now, we go and these first sections actually just prepare everything.

397
00:39:49,730 --> 00:39:56,990
So if you want to be able to set this up in an environment where it does it completely automated where it will download.

398
00:39:56,990 --> 00:40:09,930
We have the URLs here and you can set this up on your own just by running this notebook so that way you can continue to run tools against it and do and leverage this there because I really want to eliminate all the manual steps possible.

399
00:40:09,930 --> 00:40:14,250
So the last thing I want to do is have to log into their website, get the latest database and do.

400
00:40:14,250 --> 00:40:15,830
So this will walk you through it.

401
00:40:16,170 --> 00:40:22,590
One pretty neat thing about this is that recently they've changed to and you have to have a license key now.

402
00:40:22,590 --> 00:40:28,350
It's still free, but you have to log in, register in their portal, get a license secret to be able to download those databases .

403
00:40:28,350 --> 00:40:31,650
Now , when I share this code , I don't want my license key in there.

404
00:40:31,670 --> 00:40:38,670
That's another benefit of using the cloud and Amazon is that they have a service called Secrets Manager.

405
00:40:38,910 --> 00:40:44,690
And what you can do is actually you can call Secrets Manager from your Python code and pull out secrets in store.

406
00:40:44,690 --> 00:41:00,110
So as you start to get a big list of different API keys and as you talk about across different clouds the cloud, even previously we made a call for the Google Maps API, I pulled the API key that I used for that out of the Secrets Manager within AWS.

407
00:41:00,110 --> 00:41:03,990
So this function here is actually what does it.

408
00:41:03,990 --> 00:41:05,010
It's in the notebooks.

409
00:41:05,010 --> 00:41:07,210
It's in my central notebook because it's reusable.

410
00:41:07,350 --> 00:41:13,290
But you just pass in the secret name, the region name , it loads it into here as a variable so that way you don't have to have it in clear text.

411
00:41:13,290 --> 00:41:17,730
And it's a security best practice and it's definitely safe if you're putting your code into GitHub and things.

412
00:41:17,730 --> 00:41:23,910
So I think as security practitioners anyway, that's just something to get better at because we push our developers to do that.

413
00:41:23,910 --> 00:41:28,910
We push a lot of people that we talk to to do that, but it's hard and it takes extra time.

414
00:41:28,910 --> 00:41:36,250
So here's kind of just an accelerator that you can copy, paste, use it in your Python code, especially if you're working within a cloud ecosystem.

415
00:41:36,650 --> 00:41:43,770
An alternative to that is it's called Pickle and it's a library within Python 2 where you can actually save files locally .

416
00:41:43,770 --> 00:41:49,490
Now they're just binary files that can get loaded back into Pickle so they're not secured, encrypted, anything like that.

417
00:41:49,490 --> 00:41:58,670
But it can at least call it from your operating system rather than pulling it or storing it hard-coded into an environmental variable or within your code.

418
00:41:58,670 --> 00:42:01,630
So another option if you don't want to use Secrets Manager.

419
00:42:02,270 --> 00:42:04,570
So as we go through this, this all gets set up.

420
00:42:04,570 --> 00:42:14,130
I already have all this, but you can see if you put an exclamation mark before any commands, you can actually run Linux commands off of this and SageMaker's running in the back end of a Linux server.

421
00:42:14,130 --> 00:42:18,270
So you can do a lot of that too, which is pretty neat from a flexibility standpoint.

422
00:42:18,270 --> 00:42:24,550
So this manipulates all your data, does a WGET , pulls it all down, goes through that process, does a cleanup.

423
00:42:24,710 --> 00:42:29,490
I hate having files just sitting on my computer that I don't need or want .

424
00:42:29,490 --> 00:42:35,910
So whenever you unzip this , this will do all the cleanup for you, loads it into a directory, and then now you have your files.

425
00:42:36,310 --> 00:42:47,090
So what we're going to do is we're going to actually start on down here because what's neat about this is we can run all these searches.

426
00:42:47,090 --> 00:42:51,590
So we're do this after we have everything set up with just, what is this, like 10 lines of code.

427
00:42:51,590 --> 00:42:54,050
So it's actually pretty straightforward to do this.

428
00:42:54,050 --> 00:42:59,230
And you can start to automate it then by passing in variables, call this from other functions from a reconnaissance standpoint.

429
00:42:59,350 --> 00:43:04,610
I'm going to restart, reset our variables so we don't have any existing output.

430
00:43:04,610 --> 00:43:06,230
So you'll see this cleanup.

431
00:43:06,230 --> 00:43:12,250
Okay , so now what we're going to do is we're going to leverage Microsoft again as our example, and we're going to search their org for their numbers.

432
00:43:12,250 --> 00:43:14,490
So we're going to go ahead and run this.

433
00:43:15,570 --> 00:43:21,290
And what it's doing is it's actually calling out, pulling in the max mine, searching for it, and then it comes back.

434
00:43:21,290 --> 00:43:26,950
And it went through, it looks like the latest row that had Microsoft in was 430,000.

435
00:43:26,950 --> 00:43:32,370
So it went through a considerable amount of data for this, came back , and then it loaded it all into a data frame.

436
00:43:32,370 --> 00:43:37,330
So we have all of the entries for Microsoft's CIDR ranges into there.

437
00:43:37,450 --> 00:43:50,250
So now what we can do is we can write that to a CSV file, or we can start to parse and manipulate and pull it into the next lineup if maybe you want to now kick off NMAP scans automatically, you could leverage this data and do, because it's normalized.

438
00:43:50,910 --> 00:43:56,990
So this actually wrote it to my, back to my recon page that's that static S3 site.

439
00:43:56,990 --> 00:44:00,590
So now I can have a list of here's the output of the different CIDR ranges.

440
00:44:00,590 --> 00:44:03,610
So I can just kind of see what I'm doing on an overview of it.

441
00:44:03,610 --> 00:44:12,090
So it's kind of neat too that collectively you can kind of now have this dashboard that you can watch as you do your own reconnaissance and bug bounty or anything that you're doing.

442
00:44:12,090 --> 00:44:14,010
And it's kind of fun to see that.

443
00:44:14,570 --> 00:44:16,390
So that's, that's really that demo.

444
00:44:16,390 --> 00:44:19,070
And then it actually, we can look at the org too.

445
00:44:19,070 --> 00:44:20,970
So we see it's 8075.

446
00:44:20,970 --> 00:44:25,550
So if we want to search for that org number in case the name's different or anything, we just swap out.

447
00:44:25,550 --> 00:44:34,330
And it actually ran both, but I'm going to comment it out so you can just see the results and we run that against it and we should have the output of the org number pretty quickly too.

448
00:44:34,330 --> 00:44:40,090
So you can see the orgs and it just happens they all have Microsoft in them anyway, but that's just another option you can always do.

449
00:44:48,780 --> 00:44:49,540
All right.

450
00:44:49,540 --> 00:45:03,420
So let's take a look at two more quick demos and one of these and you'll, you'll pretty much hear from me over and over, especially if you worked with me of how much that I don't like to do manual effort in spreadsheets or CSV files and, and from a metrics perspective,

451
00:45:03,420 --> 00:45:05,080
how can I completely automate it?

452
00:45:05,080 --> 00:45:15,700
And the whole basis of this talk is around automating those manual mundane things that just take time away that we should always have almost in real time or at least near real time metrics and data points that we can have.

453
00:45:15,700 --> 00:45:21,380
So I, I hate using Excel spreadsheets and building graphs and charts off of those.

454
00:45:21,380 --> 00:45:34,180
Now I'm fine with updating them and keeping them as kind of how do you maintain different components, but one of the neat things about pandas is literally you can, you can import Excel spreadsheets with literally one single command.

455
00:45:34,540 --> 00:45:38,640
And so when, when you can do that, it's so easy to do.

456
00:45:38,640 --> 00:45:42,280
I figured, of course, we have to be thinking about security awareness.

457
00:45:42,280 --> 00:45:44,080
Security awareness should always be in the back of our mind.

458
00:45:44,080 --> 00:45:49,260
So when we're working with coworkers ourselves, I, so what I did was I took the step and I said, yeah, absolutely.

459
00:45:49,260 --> 00:45:50,560
We need a mask for that.

460
00:45:50,560 --> 00:46:04,220
So as we're working in quarantine, when we're on our Zoom calls with our peers , I created these masks that you can pick up, definitely support, pay the, pay the AWS bills for this, but remind your, remind your coworkers on your Zoom calls that they need to modernize ,

461
00:46:04,220 --> 00:46:06,480
they need to codify and they need to automate .

462
00:46:06,480 --> 00:46:15,000
So I, I recommend pop these on, show that, rock these out as you're, as you're in your Zoom calls and just remind people that we need to we need to move faster.

463
00:46:15,000 --> 00:46:16,020
We need to be more efficient.

464
00:46:16,020 --> 00:46:18,680
We need to automate, codify and move forward.

465
00:46:18,680 --> 00:46:22,700
So as we go, as we go into this, let's, let's check this out.

466
00:46:22,700 --> 00:46:24,700
So we're going to, we're going to do this one.

467
00:46:24,700 --> 00:46:25,400
This is BSIM.

468
00:46:25,400 --> 00:46:27,840
So it's called building security and maturity model.

469
00:46:28,040 --> 00:46:30,900
It's one of the methodologies around application security.

470
00:46:30,900 --> 00:46:37,420
So if you're building an application security program, trying to measure how you're doing, this is one of the models that you can leverage and do.

471
00:46:37,420 --> 00:46:39,500
So all this does is really it imports it.

472
00:46:39,500 --> 00:46:42,780
And this is what I've used to measure programs against over the years.

473
00:46:42,780 --> 00:46:43,740
And I really like it.

474
00:46:43,740 --> 00:46:45,900
I'm a big advocate, big fan of BSIM.

475
00:46:46,300 --> 00:46:50,860
So what we can do is we can actually load this spreadsheet within just a moment.

476
00:46:50,860 --> 00:46:52,200
And this is just a spreadsheet too.

477
00:46:52,200 --> 00:46:58,340
So even though we're working , we've shown massive data sets of terabytes and gigabytes of data we've been processing.

478
00:46:58,340 --> 00:47:02,860
You can use this for just like single megabyte spreadsheets as well and still automate and do.

479
00:47:02,860 --> 00:47:03,820
So this loads it in.

480
00:47:03,940 --> 00:47:07,560
We can see , and this is kind of from a metric scorecard dashboard standpoint.

481
00:47:07,720 --> 00:47:08,800
And it's also in GitHub.

482
00:47:08,800 --> 00:47:12,060
So we can check this out outside of this presentation as well.

483
00:47:12,320 --> 00:47:16,680
And then let's say we want to keep dashboards just for our management, our leadership of the scores.

484
00:47:16,680 --> 00:47:23,360
You literally can just run this command as you update, have your team members update spreadsheets as you complete a project or a program.

485
00:47:23,400 --> 00:47:28,640
And then you can have your spider charts that show kind of where you're at in the maturity loop, where you want to be, where you're at.

486
00:47:28,640 --> 00:47:30,040
And these are kind of interactive.

487
00:47:30,040 --> 00:47:36,940
You can overlay these and they measure based on how BSIM actually measures of how high you are in the quadrants.

488
00:47:36,940 --> 00:47:40,680
So just one cool example that's pretty lightweight.

489
00:47:40,680 --> 00:47:45,500
Here's your output chart of you can see where you're currently at, where your target is, and how you're progressing.

490
00:47:45,500 --> 00:47:49,040
So you can kind of build on this and think about all the different metrics, spreadsheets you have.

491
00:47:49,040 --> 00:47:54,940
That's literally one line of code that you can pull in an Excel spreadsheet into a pandas data frame and manipulate.

492
00:47:55,060 --> 00:47:56,780
And then you never have to do this again .

493
00:47:56,780 --> 00:48:02,920
I mean, that took two seconds to run the latest, greatest metrics around an application security program measurement .

494
00:48:03,720 --> 00:48:08,760
Here's another example just with a CSV of the CVE library .

495
00:48:08,760 --> 00:48:11,620
So it's a data set where you can go actually to MITRE's website.

496
00:48:11,620 --> 00:48:15,740
You can download the historical in a CSV format, all of them.

497
00:48:15,740 --> 00:48:18,720
So I've already kind of taken the approach of that .

498
00:48:18,720 --> 00:48:25,160
We can download the file and we can actually just import it into a data frame.

499
00:48:25,160 --> 00:48:34,380
So these few lines of code here will walk through and it loads up 176,000 different lines of CVE code.

500
00:48:34,380 --> 00:48:34,700
So use.

501
00:48:36,180 --> 00:48:43,480
So if you ever wanted to do analytics around those, trying to look at group by how many were issued per year, you can get this.

502
00:48:43,480 --> 00:48:44,520
You can run it pretty quickly.

503
00:48:44,520 --> 00:48:48,620
So over the past 20 years, you can see how the counts have increased over time.

504
00:48:49,380 --> 00:48:52,740
You can look at which months are busiest with CVs.

505
00:48:52,740 --> 00:48:54,780
Why do you feel like you're always working over the holidays?

506
00:48:54,960 --> 00:49:01,740
You can kind of use this to compare and forecast where you're going to most likely have more patch management resourcing needed in different things.

507
00:49:01,740 --> 00:49:10,900
So kind of cool insights that you can share and kind of make it actually as data-driven background and back you up on why certain things are happening.

508
00:49:10,900 --> 00:49:17,560
And I think as we go and build programs and security, we should always think in terms of data-driven mindset.

509
00:49:18,760 --> 00:49:22,400
The other neat thing is you can load additional libraries.

510
00:49:22,400 --> 00:49:31,560
So this actually loads Bokeh, which is a charting tool, a graphing tool within Python, which is used pretty commonly across the scientific computing community as well.

511
00:49:31,720 --> 00:49:35,800
And you can also write these to interactive websites and things so you can show the dynamics.

512
00:49:35,800 --> 00:49:39,780
So you can start to chart these out, you can graph, you can save them, you can load them, you can do.

513
00:49:39,780 --> 00:49:45,160
So just a couple lines of code, repeatable, reusable, and you can build these that you can just use over and over again.

514
00:49:45,160 --> 00:49:49,700
So tons of efficiency gain, tons of opportunities to do so.

515
00:49:49,700 --> 00:49:52,700
I definitely recommend grasping that.

516
00:49:52,700 --> 00:50:01,380
Don't do manual work in spreadsheets anymore other than maybe just populating data or loading it from an API, but leverage it in code and automate your things.

517
00:50:06,730 --> 00:50:17,110
As we wrap up this presentation, I forgot to mention that as you build this automation, you have services running, it's really easy with the cloud to build your SNS notifications.

518
00:50:17,110 --> 00:50:25,150
So these are simple notification services and you can text yourself, give yourself reminders, let you know where the progress is in any of the processes that you're running.

519
00:50:25,150 --> 00:50:30,310
So definitely a handy snippet of code to really leverage and utilize to your advantage.

520
00:50:30,630 --> 00:50:48,970
The other part that I wanted to mention is we've looked a lot about kind of red teaming techniques and passive reconnaissance in doing this, but what I want you to think about is if you're in an information security program right now working, look at this from a holistic standpoint of how can you help out the other InfoSec domains that you work side by side along,

521
00:50:48,970 --> 00:50:59,170
whether you're in vulnerability risk management, whether you're in third party, whether you're in a cyber fusion center, you're doing security engineering or architecture.

522
00:50:59,170 --> 00:51:15,110
Think about those outputs and inputs and how can you better take the data that you're generating within your department and how can you normalize it and pass it on to make maybe the risk management team help them make better decisions, have better normalized data that they can process and do.

523
00:51:15,110 --> 00:51:27,070
So think about those gaps and I think that's somewhere that you can really accelerate in your career and it's a gap that we have of where we're really good at building out and ensuring individual silos within information security.

524
00:51:27,170 --> 00:51:40,350
But if you can start to show from a career perspective how you can branch and understand kind of what are the value statements , what are the objectives of these different departments within your program, that's where I think you can really accelerate in places that you can really,

525
00:51:40,350 --> 00:51:42,830
I guess, just build and extensify your career.

526
00:51:42,830 --> 00:51:52,370
So I encourage you to take these principles, these platforms, these topics that we've talked about and start applying them to your daily jobs and I think you'll see some great success.

527
00:51:52,370 --> 00:51:54,710
And again, I really appreciate the time today.

528
00:51:54,710 --> 00:51:57,270
I hope that you've taken something away out of this.

529
00:51:57,270 --> 00:51:58,650
Definitely jump into the GitHub.

530
00:51:58,650 --> 00:52:04,510
I'm going to be continuing to build out blog posts and code snippets and accelerators in there.

531
00:52:04,510 --> 00:52:10,170
So in the weeks and months, definitely push me, help drive me to keep that populated.

532
00:52:10,170 --> 00:52:14,690
But I encourage you to definitely reach out if you have any questions as you go.

533
00:52:14,690 --> 00:52:20,070
But I appreciate your time and enjoy the rest of the DEF CON and the Red Team Village.

534
00:52:20,070 --> 00:52:22,990
And thanks again and have a great day.

535
00:52:22,990 --> 00:52:23,350
DR.

536
00:52:23,350 --> 00:52:23,970
MICHAEL www .microsoft.com