Hello, DEF CON Red Team Village.
I want to thank you today for taking the time to listen to this talk.
I've really been looking forward to it,
and I'm excited to be able to share this information with you today.
My talk is titled Combining Notebooks, Datasets, and Cloud
for the Ultimate Automation Factory.
And all through this presentation,
I really want to challenge you to think about
pushing the boundaries of the art of the possible,
trying to change how we do things in our day-to-day lives,
the manual processes, how can we be more efficient?
How can we operationalize and streamline these activities
so that we can be more productive in our lives?
And also, as a Red Teamer, as a Blue Teamer,
as a security practitioner,
how can we better ourselves and our careers
and really elevate ourselves to the next step?
I know it's definitely sad that we're not all together in person at DEF CON.
I know I had everything booked, ready to go.
I was excited, looking forward to it.
But I will say, having this virtual offering
is something that's really changed the dynamic of information security
and the landscape and the hacking and all of this.
It's actually really set a playing field
where a lot of people that didn't have the means or methods
to attend Vegas or make it to DEF CON or Black Hat,
now it kind of sets the bar where having it online, virtual, free,
having direct access through Discord, through Slack,
through different channels and mechanisms,
to the industry experts, the people that I've looked up to for years
who I've learned from, being able to share the stage with them alongside,
it's a pretty awesome time.
The other cool thing is, now I can sit in Discord,
I can be ready for questions and answers as we're doing this talk.
So feel free, interactively, to do this.
We won't spend a lot of time in the About Me section,
but by day, I'm a security architect,
and my main focus is cloud security.
So I get to see a lot of use cases around the new,
latest, greatest emerging technologies in cloud.
How are they being leveraged to help businesses
and help drive innovation and success
and actually decrease operating costs and accelerate delivery.
So I'm taking a lot of those examples,
and then I'm combining it with kind of my night job,
which is security research.
I mean, security is my, it's my job, but it's also my hobby.
It's what I love to do anytime that I have spare time
that's not either working or with family,
I'm probably doing security research in some way, shape or form.
So feel free to check out these projects.
I have a blog on Medium.
Any code, any examples, any demos, the slides,
everything from this presentation will be available in GitHub.
So you can start checking that out.
If you want to follow along with anything,
a lot of the content, the resources, the notebooks
that we go through will be available there.
So I encourage you pull it up while I'm talking,
do that, check things out,
and feel free to ask questions as we go along.
But thanks again, and let's jump right in.
So what are we actually going to cover?
For the agenda today, there's really a couple of things,
but I really want to focus on inputs and outputs.
I think it's a mindset.
We always talk about the hacking mindset, the security mindset.
Well, in this case of this talk and the scope,
I want you to think about inputs.
What are the objectives?
What are the goals?
What do you put in?
And then what are those outputs?
How can we normalize these outputs so that way
you don't have to try to remember
how do you do 500 different scripts?
How do you make these tools work?
How do I manually parse through all these outputs and data?
So throughout this, we're going to show kind of modular,
repeatable technology, agnostic designs.
And then how do you apply this from a cloud-focused technology?
And we're going to spend our time in the AWS ecosystem.
However, you really could translate this to any of the major cloud providers.
And then also, we're going to provide some solutions.
So how do you tie this all together so that way you can become more efficient?
And I'm hoping that if you watch this talk,
you apply these concepts, your key takeaways are
I want you to make yourself better.
I want to make the people around you better.
And I'm hoping that overall, we can all work together
and make the industry better.
So this is the story of my life.
Thank you, Jason, for sharing these tweets
because the timing was perfect.
And I started thinking in terms of,
and this is in the scope of Bug Bounty,
if you've ever worked on Bug Bounty,
you see these different tweets of
I just made $5,000.
Or I made a $15,000 bug.
Or I just made $400.
It's kind of, that's how I always feel,
is wow, let's just spend a couple hours,
let's make a couple thousand bucks.
But that's not the case.
And I'm probably, I don't know if this resonates with any of you,
but I'm the person that if you were to ever see me tweet
I just made $2,000 on a Bug Bounty,
that's because I just probably put in
a hundred hours trying to research
through reconnaissance, through tooling,
through tests and failures and inputs
and report writing just to make that.
And that's what goes on, I think,
often behind the scenes that we overlook.
And one of the things is, too,
is our most valuable commodity is time.
My biggest challenge is finding the time
to be able to do this, to really get in the weeds,
to dig in and try to find these red team objectives
to relearn, I'm always relearning things
or figuring out or looking up or Googling
or all the places where I'm trying to relearn
all the things I've learned historically.
How can we make it faster?
How can we be more efficient?
So when I have these two or three hours
in an evening to spend, I can be focused.
I can look at my targets.
I know exactly what I want to do.
I don't have to spend hours
doing the reconnaissance research
trying to just make selections
on what my targets are
because that takes up all my time.
And then the time to value is really zero
and net none.
I'm going to focus on how do I be productive?
How do I make money?
And how do I find issues?
And further, what I'm attempting to do.
So that's what really drew me into
how do I build this automated ecosystem
and take an advantage of the new technologies
and new capabilities out there
so that I can be the best that I possibly can.
So now let's jump into the architecture
of the underlying design
of the entire automated ecosystem
that we're going to build and talk about
through the remainder of this presentation.
We'll have a bunch of demos
and code snippets and things
that we're going to kind of deep dive in
and hoping that you can leverage these as accelerators
so they can be agnostic to a specific tool,
a specific process.
But as you look at different things
that you do and experience
from a Red Team perspective
or even just from a general
information security process
or practice that you do in your daily jobs,
these are all components that can apply
in a lot of different ways,
which I think is really exciting.
And that's a big part of why I feel like
rather than just trying to deliver
this custom tool that is just a script
or something that you run
and you routinely run it
and it does the same thing every time,
what I want to kind of teach
is that thinking in terms of
how do you build that underlying infrastructure
that can support anything
that can be thrown at you?
How can you better yourself
and understand rather than
you kind of move up from
knowing just how to run a tool
to actually understanding
what's going on behind the scenes,
how do I piece this together
and how do I make it my own?
So I have this system
that can be potentially better than others
or bring you to the best in the industry
or just solve challenges
that you have on a day-to-day basis.
So as we look at this architecture,
it's really broken down into three key pieces.
And on the left side is the user interaction.
So these are the components
of the cloud services
that you're going to generally interact with,
whether it's an input,
whether it's an output.
But in the design that I have
that we're going to walk through,
it's going to leverage these four pieces.
And we're going to, as we go further,
I'm going to give you a quick overview
of every one of these components
just to explain what it does,
where it's used for.
And in the back of your mind,
continue to think,
how can I reuse this for X function
or what I'm trying to do
or my daily job?
And how can I convert my manual efforts
that I do on a day-to-day basis?
How can I save time,
be more efficient,
and automate this
so that way I can make my life better?
Basically, you can automate yourself
and then you can focus more
on the latest, greatest technology emerging.
Push your program forward,
mature in everything you're doing,
and just continue to build and build on top
so we're not relearning,
we're not doing the same manual tasks
over and over.
And this ecosystem will really elevate that
so that way you can do
and you can build
and it's modular
and it's really an ecosystem
of a lot of microservices in doing that.
The middle layer is your processing and computation.
So you have your user interaction
that will pass everything over
to your processing and computation.
And this is a pretty big arsenal
of different tools.
You have, whether it's big data type,
query analytics services,
it could be eventually,
we're not going to show any of these today,
but if you get into machine learning
or image recognition
and thinking in terms of future opportunities
of like you have screenshots of webpages,
maybe you want to process
a bunch of images of those really quickly,
this would be your processing
and computation layer.
And then the last layer,
I think is pretty key
and it's really important
to get this in the mindset
as you kind of consider future opportunities,
is that data storage layer.
You look at today,
a lot of the processes you may have,
like if you're doing reconnaissance,
you might run five or six different tools.
In those inputs,
you're going to submit,
maybe it's a domain,
maybe it's an ASN,
maybe it could be a CIDR range.
It could be various things.
You input that.
Generally, the outputs
of your various tools and scripts,
they're going to be slightly different.
And that's where that source data bucket
comes into play.
So I expect, anticipate,
let's pull in 100 tools eventually together.
Let's all put that data,
source data into that source data bucket
that's just various formats and everything.
And then what we want to do
is we want to think in terms of
how do we normalize this?
What's the actual objective of this?
Because in a lot of cases,
I mean, and we're going to look from
kind of a red team bug bounty perspective,
but think broader
as you think about your actual job,
whether you're doing risk management,
whether you're doing third party analysis,
whether you are a security engineer,
whether you're an architect,
any role in this,
think about how this can apply.
But we look at our inputs and outputs.
So a lot of our tools,
they're either going to output things
such as URLs.
It might be parameters
for a different web page.
You could have the ASN numbers
for network ranges.
You could have CIDR ranges.
You could have domain names.
You could have subdomains.
You could have company names.
You could have certificate assignments,
different inputs and outputs.
So those are all of your various outputs
that you're generally going to have
across your tools.
We need to make these common between that
so we can further process this
and make this data useful
because I know one of the areas
that I struggle with the most
is parsing through this data of every tool
and it just takes so much time.
And that's something that we could,
if you do it one time for a tool
where you normalize the data,
you never have to do it again.
And the cooler part about this is
we only need one person in the industry
to do it for per tool
or start to kind of standardize
and build in.
And that's where on my GitHub
you'll see Project Straylight.
And my goal is really
to just have this running open network
of here's accelerators we can do,
here's code snippets
that have helped me along the way.
And it's maybe not so much a technology
or tool that you install,
but it's just a bunch of ideas
and things that have helped me in my career
that over the past 10 to 12 years
that I've either had to relearn or do,
I'm trying to aggregate it all
into this ecosystem,
this architecture that we see
of how can I do what I've spent
so many hours doing redundantly, manually,
and how can I make it better
so that way the newer generation,
the next people that are going
to be experts, leaders in this,
can just build on top
of what we already have
so we're not continuing to rebuild
and we can advance further and further
and we can win in this industry.
So that's a lot of the intent there.
So as we look at the data storage layers,
you basically go from your raw data,
which could be the output of any tool,
you get to more of a normalized version.
And then you also get
into your presentation layer.
And that's when it's your reporting,
it's your analysis, it's your metrics,
it's the outputs that you want to show
for the delivery of your work.
And this could speak to any level
depending on how you present it.
So you could do an aggregate of
here's all the attacks that we've had
over the course of X days,
summarized into charts, graphs,
and that might be at more of a CISO level.
You could have indicators of compromise
or different signatures,
different levels,
which you would feed into an engineering team.
That could be part of your presentation.
But it's how do you normalize
and how do you show this
in kind of a formatted way
so that way if you have to do
monthly vulnerability reports or metrics
or showing analysis of how you've
further progressed or analyzed websites
or targets and you want to compare
what you've done, what you haven't done,
that's where that presentation layer comes in.
So we're going to jump in
and I think we're going to do,
let's do a demo.
And let's kind of show,
I'll show from end to end
kind of one area
where we can go across this
of how this ecosystem works.
And then after that,
we'll dive in further.
I'm going to show you the accelerators,
the pieces, the components.
Keep in mind that all of this
is also available in GitHub.
You can access it as we go along.
You can check it out.
Feel free to use it.
Create issues in the GitHub
if you find any things.
Reach out to me with questions.
But I really just want to continue
to build and teach you
how you can kind of think
in terms of building
your own automated ecosystem.
As you're watching this too,
you're probably thinking,
why cloud?
I have a really nice home lab.
Do I really need it?
There's three key areas
beyond just the amount
of what we just walked through
in that architecture
that you have available to you.
But it's pretty exciting
when you think about
what cloud can mean to you.
And if you hear...
I mean, the businesses, industries,
everybody is adopting it.
That's nothing new.
It's not anything new there.
But I think there's three key parts
of cloud that are really exciting
as just an individual researcher
where you don't need
millions of dollars
or tens of thousands of dollars
of funding behind you
to actually make progress.
The first part is
the democratized accessibility
of datasets.
There's some pretty massive
and interesting datasets out there.
The problem is that
on our home labs,
our home computers,
our home servers,
a lot of those don't have
either the horsepower
to accomplish what we need.
We need more memory,
more CPU,
or it's not close enough
to the data
where we can't download
petabytes of data
or terabytes of data
in a meaningful way
to keep up
and be able to analyze and do.
And that's where
you kind of run into issues
of data gravity.
And you need to build
your compute actually
closer to the data stores,
the data repositories,
because it's costly.
It's expensive to move those.
So how can we proceed
with that?
The other piece is that
if you look across the world,
we're facing right now,
we're all in quarantine,
we're social distancing.
You have...
So take a look
at what's going on
around the race
to find vaccines
and solutions to COVID
and kind of the data
around that in the pieces.
The world is coming around cloud
and the latest, greatest
computational technologies
to solve the world's
biggest problems
that impact everybody globally.
So what if we can take
those pieces
that the top scientists
in the world,
the top in their industries
are taking to solve
the world's most challenging problems?
I'm sure we can apply that
to our daily bug bounty targets
or our daily security processes
that we need to improve.
But in the same way,
we don't have to spend
tens of thousands of dollars
for this.
The other neat thing about it
is it's financially viable.
We can now stand up servers
that have 128 gigabytes
of memory.
You can do unlimited storage.
And the nice thing is
you don't have to pay
thousands of dollars
to buy this from a capital standpoint.
It's all operating expense.
So as you look at it,
if you need to run something
that's extremely memory intensive
and it's just one operation
or a loop,
you literally could stand up
a massive server
for just a couple minutes.
It might cost you $10, $20
for maybe five or ten minutes
of processing.
But rather than needing
to invest $80,000
in similar hardware equipment,
you now can do this
in your home office,
in your home lab
through the cloud capabilities
charging.
And that $20,
if you think about it
in terms of bug bounty investment,
that's one P4.
And you have that well covered.
I mean, that's on the low end
thinking in terms of that.
But just looking at it
from a business perspective,
thinking in terms of
how can I continue to invest
in my capabilities,
the technologies,
the processes,
so I can gain more time
and I can make the time
that I spend actually manually
doing things,
evaluating parts,
how can I make it
the most beneficial to myself.
So now let's take a look
at the various tooling
that we can use.
We've looked at
the architecture diagram.
I'd like to just give
a quick overview
of the different components,
especially if you're not familiar
with the AWS services
that are listed on here,
kind of what they do,
why they're important
and how they can apply to you.
So if we go through the list,
you saw from the user interaction space
the Amazon API Gateway.
So what this does
is this is just an entry point
into your ecosystem.
And the neat thing
about Amazon API Gateway
is there's a lot of things
you can do with it,
but where I've really leveraged it
is as a proxy
into your Lambda function.
So what you can do
is you can do basically
a GET request
or a POST request
through the API Gateway
and pass that directly
into an AWS Lambda function.
And if you think in terms of
what you've seen
in the Jupyter notebooks
and Python libraries
and any functions you run
where you pass something,
you literally can pass it
through a GET request
through the API Gateway
into AWS Lambda.
So I kind of view that
as top tier level of services
where you've really normalized
your content,
you have exception handling,
you know that you're ready
to completely automate this,
let's promote it
into a Lambda function,
pull it out of your
Jupyter notebook,
and then let's put it
behind an API Gateway
so you can really
kick it off
and call it ad hoc
as you need
and really start
your ecosystem of automation
around there.
Another tool is
Amazon SageMaker
and it's really
a larger framework
and compute system
where you can do more things
like machine learning
and training
and all of that.
But what I pretty much
solely use it for
are the Jupyter notebooks.
So you can stand up
the Jupyter notebooks,
you can run it,
you can adjust the size on it
so if you need a lot more
gigabytes or storage
for a duration
or a function,
a lot of times
I'll ramp up
and use a larger
instance of it
for just a few minutes
or just to do that process
and then I'll turn it down
to kind of the standard level.
So Amazon SageMaker
is really the tool
mechanism for running
the Jupyter notebooks.
The static website
hosting S3,
that's an S3 bucket,
you can actually set those up
so that they can host
web content,
make them publicly available
so you hit it,
you can set an indexed
HTML page,
you can run it,
and you can run websites
out of S3 buckets
which is really neat.
The Amazon simple notification service,
it's a way that you can
send push messages
to whether it's SMS text,
whether it's emails,
different things.
So it's really just
a notification service
that we can do.
In this case,
I'll show a code snippet
where we can actually
send ourselves text
so that way we know
if there's anything
that's going to take
a little bit of time
or you just want to get notified
or have the final URL
where you're going to
access your data,
you can get texted
whenever it's ready.
So you can kind of do
bug bounty, reconnaissance,
all of this on the fly
as you build this system
which is pretty cool.
Now as we move into
the processing and computation layers,
AWS Lambda is really
what's used to run code.
So you can set it up,
it runs a lot of different things,
it's not just Python,
but it's Node,
and it runs on a serverless compute
so really you put
the code in there,
it will stand up
the compute resources
and it only runs
for a duration.
I think the longest
that you can run a Lambda
is about 15 minutes
so you're charged
based on the duration
that the computation
is running.
So if you think in terms
of smaller Python functions
that you have ready
to automate and scale,
that's where you can
migrate them to the Lambdas
and they can just always
be ready to work
off of triggers.
The Amazon Elastic Block Storage,
a lot of times,
S3, there's a little bit
of a caveat
where S3 you get charged
on get requests
and put requests
and writes,
whereas EBS storage
you pay for how much storage
that you have assigned
and allocated.
So depending if I think
I'm going to do millions
of reads or writes,
sometimes I'll do a lot
of my testing against
Elastic Block Storage
and then once I have
something that I know
is ready to go robust
and I want to leverage it
in the greater ecosystem,
I'll move it over
to our data storage tiers
while they're in the refined
or the raw data levels
to do more normalized
and automated operations against.
AWS Glue is a way
that you can set up
and you can crawl data sets.
So Athena is,
you picture that,
is like you write
your SQL queries
and your query language
and it will run it
and distribute it
against data sets
and it can run
against CSV files,
it can run against
relational data stores,
unstructured data,
and all those.
So it's really powerful
and if you haven't ever
dug into it,
AWS Athena,
I highly recommend it.
There's a lot of power in it
and you can literally
just sit there
and use the console interface
and pretty much
ad hoc write queries
against big data sets
if you decide to as well.
And then Glue is,
I started talking,
it's what indexes
a lot of those data points
around it.
So you can create
Glue indexes
so you don't have to
query the entire data set
every single time
and you can do those.
AWS Step Functions,
you won't see anything
in the demos today
leveraging this,
but this is kind of
a future capability
where I want to leverage
where you can actually,
it's like setting up
workflows and processes
and you need X to happen
and then once X completes,
you can send the output
to this one
and do, right now,
I'm just leveraging things
off of triggers
and alarms
and Lambda files,
but you can really start
to use AWS Step Functions
as orchestration,
as a central core piece.
So something I'm going
to dig into more
in the future.
AWS Secrets Manager
is a place that,
so you can move
all of your secrets,
your API keys,
out of your code,
put them in Secrets Manager.
It's a quick call out.
You can load them,
you can store them,
and you can kind of
store other parameters
in there as you need as well.
You can pull them back
into your code
so they don't sit as code,
but you can use them
as variables on the fly.
So I highly recommend
digging in there.
There's some accelerators
that you'll see
that talk to those.
And then on the right side
are just all the S3 buckets
and you can create
multiple S3 buckets
and have the different stores
and it's just long term
data storage for you.
So let's jump in with a demo.
This demo is going to show
how we can query
Rapid7's Project Sonar,
which is essentially
it's the 4DNS
of the entire internet.
And it's sitting
in a dataset
within S3.
So rather than try
to copy that entire dataset
down into our own system,
we're going to leverage
a Jupyter Notebook
and we're going to actually
run a distributed query
against that external dataset
that's hosted in
not our own S3 bucket,
but the public datasets bucket.
And we're going to get
the results back
and we're going to process it.
And we're going to do it
in just a matter of...
it's probably going to take
a couple of minutes
as we walk through
and talk through the code,
but this is literally,
I think generally the queries
around the 4DNS
for a domain wildcard
is about 26 seconds.
So let's walk through this.
So on the screen here
you can see my Jupyter Notebook
and what we'll do is
we'll walk through this too
because I want you to think
in terms of how can I reuse
these components,
how is this applicable
well beyond just this
sonar use case.
So let's go ahead and start.
We're going to go ahead
and run this first section
of code here.
And this is what I talked about
earlier too, which is nice
is that if you're not
an expert at code,
you run it,
you can break it down
into cells,
and you can run each cell.
So we're going to run
this first cell.
What we just did was
you can see this execution ID here.
What we did is we set our domain
to Microsoft.com
so we're actually querying
all the wildcards
of the Microsoft domain
4DNS space.
And keep in mind,
we are not touching
any of Microsoft's systems.
This is purely utilizing
the 4DNS dataset
and we're getting the results
from there.
And then what we've done
is we've set some
of our variables around
our bucket of where
we want to store the results,
what database we want to query
against in Athena,
which we've had some...
the setup information
is all contained
within the GitHub as well
so you can set that up.
And then the table
we want to query against.
What we do next
is we set up our query.
So what this is doing
is basically just saying
select everything
from wildcard
microsoft.com
and then we select
the latest date
of the dataset
so that way we don't
historically go through
every single one iteration
that we have.
It issues the
Athena query.
It takes about 26 seconds
to run.
We have the
Athena query ID here.
So in this next
component,
I will actually
jump back up.
This run command
that I'm highlighting,
this is actually a way
that you can call
other notebooks,
which is kind of neat.
So as you think about
in terms of you call functions
within programming code,
this will actually run
and load an entire
notebook.
So what I've done is
I try to take anything
that's not tool specific
and I put it in kind of
a common notebook
that you can set
a lot of different
things to run.
So what this is
actually doing is
the query Athena
function is running
out of that other
notebook because
I've set it up well
enough that you can
pass in the
results bucket.
You just send it
the query in the
Athena database
and it will kick it off
so it doesn't matter
what tool specific
or anything.
It will just run a query
that you tell it to run
which is why I moved
it into that kind of
standardized function
area.
So what we're going
to do is we're going
to run this next one
and this is going to
generate, pull the
results back.
So you can see that
it's already succeeded.
If we were to run this
immediately it would
just go in a loop
and it would circle
every five seconds
and check for the
results back.
But it's pulled back
18,695 different rows
and you can see the
types of information
we've just pulled
back.
So think you have
18,695 different rows
basically for DNS
entries from
Microsoft.com in
under 26 seconds
with the IP
addresses, the C
names, A records,
MX records, all of
those.
And then with that
we're going to do
this next thing is
this is actually
going to leverage
the MaxMind
database and we're
going to start
figuring out
querying for the
IP addresses and
getting all that
information back.
So let's go ahead
and run this and
what this will do
is it creates a
data frame which
leverages Pandas.
So Pandas is a
library within Python
that you can run
and I completely
recommend it.
It is one of the
best things that I've
ever found in terms
of doing metrics and
analysis and
automation.
But you can see it
just processed 18,695
rows in just a few
seconds.
And now we have the
latitudes, longitudes,
countries, and
localities all
merged to all
18,000 entries.
So what we're
going to do now is
if you say, hey, I
just did all this
work, I don't want a
chance of losing it
even though theoretically
you could redo this
over about 30
seconds.
But if you want, you
can use this command
and you can save your
data frame off to
Excel so you can save
it locally.
So if you run this, it
will just print and
store those 18,000
records into an Excel
spreadsheet which you
can reload later on
which is pretty handy
to use.
And if you want to
save your artifacts and
data points, you can
always do that.
So what we'll do now
is we're going to run
this next command and
what this does is it
just aggregates.
And you can see how
fast that was.
It was almost instantaneous
when I pressed the
start button and it
just grouped up all
the different latitudes
and longitudes so that
we can get ready to
plot them on a heat
map.
So we know 525 had
this latitude,
longitude, and it did
all that work.
And we can dig in.
This also calls that
the other notebook
because it's
centralized and it's
not specific to the
project sonar data
set.
So you can reuse and
redo any of that.
I am going to jump
back up, though.
I think this is
something that's
pretty neat.
And if you think
about in terms of
how can I reuse some
of this, this get
location function that
I have behind the
scenes that just
pulled in and merged
all this together,
what I did, if you're
looking, I passed the
entire data frame as
a part of the function
there and I passed
the value column,
which is the IP
address column.
So if you think about
reusability, all I
literally did was I
just took, if you
have an Excel
spreadsheet of IP
addresses or a CSV
or anything like
that, you load it
into a data frame.
You can pass that
entire data frame.
It doesn't matter
what it is.
It doesn't matter what
the rest of it looks
like.
You pass the data
frame name and the
column that the IP
addresses are in or
obviously it will
fail on if it's not
an IP address as it
goes through those
because they're C
names.
And you do that and
you can process it.
So anything you have,
that's completely
reusable and that's
just a few lines of
code.
It's in GitHub.
You can pull it and
I think in terms of
just from a cyber
operations, a SOC
perspective, you
could really get a
lot of value out of
some of that
information.
You could look at
attackers, web
denials, web
traffic, who's
hitting your web
application firewalls
and you can do heat
maps and different
things from an
attacker perspective
which is pretty cool.
So let's go on down.
We have that
information and then
we want to now, let's
do this.
So this is just a
couple lines of
code but it's really
cool.
So this uses the
Google Maps API and
you can see how fast
that was.
It just plotted the
information we have
up here for the
latitudes and
longitudes and
created an
interactive heat
map so we can
zoom in on and
see what's going
on and this shows
the concentrations
of Microsoft's
forward DNS space
across the globe.
So you can look and
as we zoom in, you
can see the
different regions
they are included
in.
So let's, if we
were just, let's
say, let's tackle
this is a big area
here, you can zoom
in and even though
it shows Seattle
space, you can go
and see that it's
not all just in
Seattle but they
have a few offices.
It looks like in
Redman, there's some
Seattle and above.
So pretty cool that
it's all interactive
and you can do that.
And the other neat
thing is you're
thinking, well, I
want to show this to
management.
How do I show my
managers, my leadership
about this?
And I can't have
them just opening a
Jupyter notebook all
the time and doing
this.
Well, one thing is
you can click the
download button and
just download the
map as an image.
But the other pretty
neat thing about all
this, and if you
think about the
architecture of how
I have that
presentation layer,
what this next
command does is it
takes this map and
it actually uploads
it to a static S3
bucket that's
hosting a static S3
content as a
website.
So then I hit this,
I uploaded the map
file to the website
and then I can
navigate over to my
bucket and if I hit
load content right
here, it should pull
up that map.
And now I can
always send any of
my leadership, any
managers, anybody
that wants to see
anything, I can run
this on an
iterative basis
while I'm
there.
It's on every 24
hours I run this
process and do.
And if you think in
terms of visually what
you want your attack
surface to look like
for your company, run
this against your
company every time.
Have this dashboard
always going and then
you could even trigger
on anomalies, you
could do different
things.
And it's just pretty
neat to always see
here, here's our
coverage from an
internet presence
that we have there.
So I think there's a
lot of value, a lot
of opportunity to do
in there.
And just the fact
that you can
literally load it
and save the
interactive mapping
part of it to an
HTML file and then
have people actually
go up, browse to
it, navigate it,
that's pretty
exciting.
So for this next
demo, we're going to
use, this is actually
one of my favorite
ones and I'm really
excited to continue
the build on the
capabilities of this.
But this leverages a
data set that's
publicly available
in S3 called
Common Crawl.
And what it is, it's
a project that's
been going on now
for, I think it's
actually been around
for about eight
years, where it
crawls the internet,
searches and indexes
and actually downloads
the HTML of files
and stores them in
S3 buckets.
So it's just a
massive repository
and similar of
basically web
archives that you
can search.
So if you've used
Wayback Machine and
different components,
this is very similar.
And so what we're
going to do is we're
going to actually
search it for a
domain.
We're going to pull
back the files and
it actually can
eventually download
the files and we
can store them in
an S3 bucket and
then we can run a
bunch of different
tools around it.
So if we want to
do like find URLs or
run HackCrawler or
anything that's going
to go through and
does a good job of
analyzing HTML,
whether it's for
vulnerabilities,
whether it's for URLs,
whether it's for
parameters, we can do
that against it.
So let's walk through
this now.
I'll kind of show you
the framework around
it and there's a lot
that we can build into
this capability.
The biggest challenge
that I've had so far
is just trying to do
some of the
asynchronous programming
against it because a
lot of times you have
a large site where
you're trying to do
downloads and actually
pulling the files, but
as we get more
efficient with this, I'm
pretty excited about
this and I think it has
a lot of promise with
it.
So what we're going to
do is we have all of
our parameters set up.
We're actually going to
run this one against
the smaller domain for
demo purposes because
if we have something
that has 70,000
websites and pages,
it's just not fast
enough at this point
to download this in
a timely fashion.
We would be sitting
here for maybe 40
minutes or an hour.
So in this case, we
can always speed it up
as we go, but we're
going to run this
against derbycon.com.
So we've kicked off
our Athena query.
It's running it now.
It's searching.
It's looking for all
of the webpages that
this is indexed and
cached and copied and
saved within the
Common Crawl dataset.
So let's go ahead and
kick off our run and
what this is going to
do is it's going to
look and it's going to
just wait for the
query to stop running
and then it will pull
in and load the
results for us into
a database.
So we'll let this run
for a few minutes.
All right, so it looks
like we have our
results here.
So it pulled back 579
different websites.
What we want to do is
since it's been running
over the course of
eight years, there's a
lot of different
versions, iterations,
which is actually
pretty neat if you want
to see how websites
have evolved and look
for changes or maybe
there's a vulnerability
or an issue or a
comment in there that
they moved out and you
could see the difference
between the different
versions.
But in this case, I'm
interested more in just
the unique users that
we have.
So we're going to run
this next command which
is going to sort them
and it's going to drop
all the duplicates and
it's going to keep the
latest version of the
webpage.
So let's go ahead and
run this right now.
And that was pretty
instantaneous.
And now we're down to
73 different rows.
So there were a lot of
different iterations,
versions of the site
cached.
And then what this will
do is we can run this
next command, we can
save it off to Excel so
that way we can have
our URL listing for
future analysis, future
loading.
And then as we go on,
what this will do is
if you want to look,
this will pull out and
only list now our URLs
which is what we care
about a lot of times,
especially if we're
trying to scope, we want
to do web crawls.
And keep in mind that
everything that we're
showing today, everything
is completely 100%
passive.
We are not touching any
of these sites, these
webpages or anything.
So this is all examples
of what you can do,
100% passive reconnaissance.
So we will never touch
a system of any of these
domains that we've
looked at.
And then for next steps
then you start, you can
start based on scope and
allowances and permissions
and things like that to
get more intrusive as we
do further analysis,
whether it's from a
red teaming perspective
or however.
But what we have right
now is just a listing of
URLs and then what this
next big section of kind
of a lot of, a lot of
different functions and
code, I mean it's still
for, for the value of
what this does, it's,
it's not too bad in
terms of lines.
But what this is going
to do is it's going to
go through now of the
query results.
So in this case we're
dealing with what, like
73 records, so it's not
that many.
But if you were to run
something, I think I ran,
I think I ran one earlier
this year against
defcon.org and it came
back with about 70,000
records.
So some of these,
there's a lot of data
and content out there
and it can take some
time.
And this is where, this
is not that efficient,
not that streamlined, a
lot of improvement
opportunity here.
But it, it basically
searches, it uses a
library in Python called
Beautiful Soup, which
you can parse down
HTML.
It's great for HTML
processing.
It has some building
capabilities to look for
comments and URLs and
links.
So we're going to start
setting this up.
And what, I have three
parameters that we can
pass in, just mostly out
of efficiency, is we can,
we can say we want to
search the files and
generally we're going to
always want to do that.
Do we want to write the
files?
So what this does is
this goes out and it
retrieves, well, it's
always going to go out
and retrieve the files
from the S3 buckets, but
do we want to write them?
So in this case, we're
not going to write them
for this demo purpose,
but you could write it
to an S3 bucket.
Now you have your HTML
files that you can run
additional tools and
things against.
So it's, it's pretty
valuable there.
And then we can say how
many records.
So if we just want a
subset for testing, if
we don't want to pull
down all 60,000 at once,
we could, we can assign
how many records and it
will loop through and
iterate over and download
the number of records we
say.
And then we, we run
this and this function
will basically process
these.
It's going through, you
can see I have a
counter on it because a
lot of times when you're
just waiting and you get
bored and don't know
where it's at, so we
can, you can always
clean this up so it
doesn't iterate every
single count.
But we've gone through
the 73 HTML files we've
down, we've processed
them as byte
strings, we've looked
for URLs, the comments,
and now we can actually
save off the different
sections.
So we have, we've
grabbed all the
comments, we've grabbed
all the titles, and we've
grabbed all the links.
So if I run this next
one, this is actually
going to show you the,
the listing of links.
And what I, what I like
about this is that it
actually shows you which
website that the links
came from as well,
which I think is really
valuable, especially
with sometimes when you
do crawls, you don't
always know, where did
I find this link?
Where did this come
from?
And in this case, it
does all of the mapping,
so if you start building
like a bigger mind map
or something of it, you
can, you can have some
value there.
And if we wanted to look
at something instead,
let's say we wanted to
look at the, the
comments across it, we
can do that, and then
we can just print it
out.
So this will just print
out the top, the first
10 comments we have.
Let's just see what
this looks like.
And then there's,
there's your list of
comments that are within
the HTML.
So as you get this,
you can start then
further looking through
there of, you know,
of, are there any
secrets, are there
passwords, you can do
regex, you can, you can
do different things in
there, so it's, it's
pretty neat.
I guess that's the,
that's the extent right
now I think we're going
to go through.
I started looking into
NetworkX.
I think there's some
promise there in terms of
getting a larger map at
scale, but I don't have
a lot of that code done
yet, but a lot I want
to build on, especially
with this functionality
because I think there's
so much value from even
an automated perspective
where now we can tie in
additional tools that
are really good at
HTML processing.
We can start to grab
JavaScript libraries,
codes, external references
and all that and do,
you can really start
building a pretty big
map of, of your target
surface.
Let's jump into another
demo.
This one is going to
search the autonomous
system numbers, which is
part of generally your
reconnaissance methodology
of where you want to
find out the different
site or network ranges
that are assigned to
different companies.
So what you typically do
is you would enter in
the company name or a
keyword for that company
and it would return the
different site or ranges
for that so then you can
further do your
reconnaissance doing
whether you do NMAP
scans or any type of
searches against those
IP ranges.
So what this does is
this uses the MaxMind
databases as well that
we talked about earlier
with the geolocation
data doing that, but
this uses their ASN
databases.
And so part of this is
we'll walk down
through it all, but
what it does is if
you're looking here at
this Jupyter Notebook
now, we go and these
first sections actually
just prepare everything.
So if you want to be
able to set this up in
an environment where it
does it completely
automated where it will
download.
We have the URLs here
and you can set this up
on your own just by
running this notebook so
that way you can continue
to run tools against
it and do and leverage
this there because I
really want to eliminate
all the manual steps
possible.
So the last thing I want
to do is have to log
into their website, get
the latest database and
do.
So this will walk you
through it.
One pretty neat thing
about this is that
recently they've changed
to and you have to have
a license key now.
It's still free, but you
have to log in, register
in their portal, get a
license secret to be able
to download those
databases.
Now, when I share this
code, I don't want my
license key in there.
That's another benefit
of using the cloud and
Amazon is that they have
a service called
Secrets Manager.
And what you can do is
actually you can call
Secrets Manager from
your Python code and
pull out secrets in
store.
So as you start to get
a big list of different
API keys and as you
talk about across
the cloud, even
previously we made a
call for the Google
Maps API, I pulled the
API key that I used
for that out of the
Secrets Manager within
AWS.
So this function here
is actually what does
it.
It's in the notebooks.
It's in my central
notebook because it's
reusable.
But you just pass in
the secret name, the
region name, it loads
it into here as a
variable so that way
you don't have to have
it in clear text.
And it's a security
best practice and it's
definitely safe if
you're putting your
code into GitHub and
things.
So I think as security
practitioners anyway,
that's just something
to get better at
because we push our
developers to do that.
We push a lot of
people that we talk
to to do that, but
it's hard and it
takes extra time.
So here's kind of
just an accelerator
that you can copy,
paste, use it in your
Python code, especially
if you're working
within a cloud
ecosystem.
An alternative to
that is it's called
Pickle and it's a
library within Python
2 where you can
actually save files
locally.
Now they're just
binary files that can
get loaded back into
Pickle so they're not
secured, encrypted,
anything like that.
But it can at least
call it from your
operating system rather
than pulling it or
storing it hard-coded
into an environmental
variable or within
your code.
So another option if
you don't want to use
Secrets Manager.
So as we go through
this, this all gets
set up.
I already have all
this, but you can see
if you put an
exclamation mark
before any commands,
you can actually run
Linux commands off of
this and SageMaker's
running in the back
end of a Linux server.
So you can do a lot
of that too, which is
pretty neat from a
flexibility standpoint.
So this manipulates
all your data, does a
WGET, pulls it all
down, goes through
that process, does a
cleanup.
I hate having files
just sitting on my
computer that I don't
need or want.
So whenever you
unzip this, this will
do all the cleanup for
you, loads it into a
directory, and then
now you have your
files.
So what we're going
to do is we're going
to actually start on
down here because
what's neat about this
is we can run all
We're going to
do this after we have
everything set up with
just, what is this,
like 10 lines of
code.
So it's actually pretty
straightforward to do
this.
And you can start to
automate it then by
passing in variables,
call this from other
functions from a
reconnaissance standpoint.
I'm going to restart,
reset our variables so
we don't have any
existing output.
So you'll see this
cleanup.
Okay, so now what
we're going to do is
we're going to leverage
Microsoft again as our
example, and we're
going to search their
org for their numbers.
So we're going to go
ahead and run this.
And what it's doing is
it's actually calling
out, pulling in the
max mine, searching for
it, and then it comes
back.
And it went through,
it looks like the
latest row that had
Microsoft in was
430,000.
So it went through a
considerable amount of
data for this, came
back, and then it
loaded it all into a
data frame.
So we have all of the
entries for Microsoft's
CIDR ranges into
there.
So now what we can
do is we can write
that to a CSV file,
or we can start to
parse and manipulate
and pull it into the
next lineup if maybe
you want to now kick
off NMAP scans
automatically, you
could leverage this
data and do, because
it's normalized.
So this actually wrote
it to my, back to my
recon page that's
that static S3 site.
So now I can have a
list of here's the
output of the
different CIDR ranges.
So I can just kind of
see what I'm doing on
an overview of it.
So it's kind of neat
too that collectively
you can kind of now
have this dashboard
that you can watch as
you do your own
reconnaissance and
bug bounty or anything
that you're doing.
And it's kind of fun
to see that.
So that's, that's
really that demo.
And then it actually,
we can look at the
org too.
So we see it's 8075.
So if we want to
search for that org
number in case the
name's different or
anything, we just swap
out.
And it actually ran
both, but I'm going to
comment it out so you
can just see the results
and we run that
against it and we
should have the output
of the org number
pretty quickly too.
So you can see the
orgs and it just
happens they all have
Microsoft in them
anyway, but that's
just another option
you can always do.
All right.
So let's take a look
at two more quick
demos and one of
these and you'll,
you'll pretty much
hear from me over
and over, especially
if you worked with
me of how much that
I don't like to do
manual effort in
spreadsheets or CSV
files and, and from
a metrics perspective,
how can I completely
automate it?
And the whole basis
of this talk is around
automating those manual
mundane things that
just take time away
that we should always
have almost in real
time or at least
near real time metrics
and data points that
we can have.
So I, I hate
using Excel
spreadsheets and
building graphs and
charts off of those.
Now I'm fine with
updating them and
keeping them as kind
of how do you
maintain different
components, but one
of the neat things
about pandas is
literally you can, you
can import Excel
spreadsheets with
literally one single
command.
And so when, when
you can do that, it's
so easy to do.
I figured, of course,
we have to be thinking
about security
awareness. Security
awareness should always
be in the back of our
mind. So when we're
working with coworkers
ourselves, I, so what
I did was I took the
step and I said, yeah,
absolutely. We need a
mask for that. So as
we're working in
quarantine, when we're
on our Zoom calls with
our peers, I created
these masks that you
can pick up, definitely
support, pay the, pay
the AWS bills for this,
but remind your,
remind your coworkers
on your Zoom calls
that they need to
modernize, they need
to codify and they
need to automate. So
I, I recommend pop
these on, show that,
rock these out as
you're, as you're in
your Zoom calls and
just remind people
that we need to
we need to move
faster. We need to be
more efficient. We
need to automate,
codify and move
forward. So as we go,
as we go into this,
let's, let's check
this out. So we're
going to, we're going
to do this one. This
is BSIM. So it's
called building
security and maturity
model. It's one of
the methodologies
around application
security. So if
you're building an
application security
program, trying to
measure how you're
doing, this is one of
the models that you
can leverage and do.
So all this does is
really it imports it.
And this is what I've
used to measure
programs against over
the years. And I
really like it. I'm a
big advocate, big fan
of BSIM. So what we
can do is we can
actually load this
spreadsheet within
just a moment. And
this is just a
spreadsheet too. So
even though we're
working, we've shown
massive data sets of
terabytes and
gigabytes of data
we've been processing.
You can use this for
just like single
megabyte spreadsheets
as well and still
automate and do. So
this loads it in. We
can see, and this is
kind of from a metric
scorecard dashboard
standpoint. And it's
also in GitHub. So we
can check this out
outside of this
presentation as well.
And then let's say
we want to keep
dashboards just for
our management, our
leadership of the
scores. You literally
can just run this
command as you update,
have your team members
update spreadsheets as
you complete a project
or a program. And
then you can have
your spider charts that
show kind of where
you're at in the
maturity loop, where
you want to be, where
you're at. And these
are kind of
interactive. You can
overlay these and
they measure based on
how BSIM actually
measures of how high
you are in the
quadrants. So just
one cool example
that's pretty
lightweight. Here's
your output chart of
you can see where
you're currently at,
where your target is,
and how you're
progressing. So you
can kind of build on
this and think about
all the different
metrics, spreadsheets
you have. That's
literally one line of
code that you can
pull in an Excel
spreadsheet into a
pandas data frame and
manipulate. And then
you never have to do
this again. I mean,
that took two
seconds to run the
latest, greatest
metrics around an
application security
program measurement.
Here's another
example just with a
CSV of the CVE
library. So it's a
data set where you
can go actually to
MITRE's website. You
can download the
historical in a
CSV format, all of
them. So I've
already kind of
taken the approach
of that. We can
download the file
and we can actually
just import it into
a data frame. So
these few lines of
code here will
walk through and
it loads up
176,000 different
code that you can
use. So if you ever
wanted to do
analytics around
those, trying to
look at group by
how many were
issued per year,
you can get this.
You can run it
pretty quickly. So
over the past 20
years, you can see
how the counts
have increased
over time. You
can look at which
months are busiest
with CVs. Why do
you feel like you're
always working over
the holidays? You
can kind of use
this to compare and
forecast where you're
going to most
likely have more
patch management
resourcing needed
in different things.
So kind of cool
insights that you
can share and
kind of make it
actually as
data-driven
background and
back you up on
why certain things
are happening. And
I think as we go
and build programs
and security, we
should always think
in terms of
data-driven
mindset. The
other neat thing
is you can load
additional libraries.
So this actually
loads Bokeh,
which is a
charting tool,
a graphing tool
within Python,
which is used
pretty commonly
across the scientific
computing community
as well. And
you can also
write these to
interactive websites
and things so you
can show the
dynamics. So you
can start to chart
these out, you can
graph, you can save
them, you can load
them, you can do.
So just a couple
lines of code,
repeatable, reusable,
and you can build
these that you can
just use over and
over again. So
tons of efficiency
gain, tons of
opportunities to do
so. I definitely
recommend grasping
that. Don't do
manual work in
spreadsheets anymore
other than maybe
just populating data
or loading it from
an API, but leverage
it in code and
automate your things.
As we wrap up
this presentation, I
forgot to mention
that as you build
this automation, you
have services running,
it's really easy with
the cloud to build
your SNS
notifications. So
these are simple
notification services
and you can text
yourself, give
yourself reminders,
let you know where
the progress is in
any of the processes
that you're running.
So definitely a
handy snippet of
code to really
leverage and
utilize to your
advantage. The other
part that I wanted to
mention is we've
looked a lot about
kind of red teaming
techniques and
passive reconnaissance
in doing this, but
what I want you to
think about is if
you're in an
information security
program right now
working, look at this
from a holistic
standpoint of how
can you help out the
other InfoSec domains
that you work side
by side along,
whether you're in
vulnerability risk
management, whether
you're in third
party, whether
you're in a
cyber fusion center,
you're doing security
engineering or
architecture. Think
about those outputs
and inputs and how
can you better take
the data that you're
generating within
your department and
how can you
normalize it and
pass it on to make
maybe the risk
management team help
them make better
decisions, have better
normalized data that
they can process and
do. So think about
those gaps and I
think that's
somewhere that you
can really accelerate
in your career and
it's a gap that we
have of where we're
really good at
building out and
ensuring individual
silos within
information security.
But if you can start
to show from a
career perspective how
you can branch and
understand kind of
what are the value
statements, what are
the objectives of
these different
departments within
your program, that's
where I think you
can really accelerate
in places that you
can really, I guess,
just build and
extensify your career.
So I encourage you
to take these
principles, these
platforms, these
topics that we've
talked about and
start applying them
to your daily jobs
and I think you'll
see some great
success. And again,
I really appreciate
the time today. I
hope that you've
taken something away
out of this. Definitely
jump into the GitHub.
I'm going to be
continuing to build
out blog posts and
code snippets and
accelerators in there.
So in the weeks and
months, definitely
push me, help drive
me to keep that
populated. But I
encourage you to
definitely reach out
if you have any
questions as you go.
But I appreciate
your time and enjoy
the rest of the
DEF CON and the
Red Team Village.
And thanks again
and have a great
day.
DR. MICHAEL
www.microsoft.com