start	end	text
0	2640	Hello, DEF CON Red Team Village.
2640	6100	I want to thank you today for taking the time to listen to this talk.
6100	7820	I've really been looking forward to it,
7820	11380	and I'm excited to be able to share this information with you today.
11380	14740	My talk is titled Combining Notebooks, Datasets, and Cloud
14740	17160	for the Ultimate Automation Factory.
17660	19260	And all through this presentation,
19260	21820	I really want to challenge you to think about
21820	24480	pushing the boundaries of the art of the possible,
24480	27120	trying to change how we do things in our day-to-day lives,
27120	30160	the manual processes, how can we be more efficient?
30160	33080	How can we operationalize and streamline these activities
33080	36000	so that we can be more productive in our lives?
36000	38400	And also, as a Red Teamer, as a Blue Teamer,
38400	40740	as a security practitioner,
40740	42660	how can we better ourselves and our careers
42660	45340	and really elevate ourselves to the next step?
45740	50500	I know it's definitely sad that we're not all together in person at DEF CON.
50500	52800	I know I had everything booked, ready to go.
52800	54820	I was excited, looking forward to it.
54820	57060	But I will say, having this virtual offering
57060	60300	is something that's really changed the dynamic of information security
60300	63260	and the landscape and the hacking and all of this.
63260	66100	It's actually really set a playing field
66100	68820	where a lot of people that didn't have the means or methods
68820	72220	to attend Vegas or make it to DEF CON or Black Hat,
72220	77080	now it kind of sets the bar where having it online, virtual, free,
77080	79960	having direct access through Discord, through Slack,
79960	81860	through different channels and mechanisms,
81860	85180	to the industry experts, the people that I've looked up to for years
85180	88960	who I've learned from, being able to share the stage with them alongside,
88960	91260	it's a pretty awesome time.
91260	93900	The other cool thing is, now I can sit in Discord,
93900	96860	I can be ready for questions and answers as we're doing this talk.
96860	99040	So feel free, interactively, to do this.
104480	107440	We won't spend a lot of time in the About Me section,
107440	110560	but by day, I'm a security architect,
110560	112740	and my main focus is cloud security.
112740	115400	So I get to see a lot of use cases around the new,
115400	117860	latest, greatest emerging technologies in cloud.
117860	119800	How are they being leveraged to help businesses
119800	122220	and help drive innovation and success
122220	126680	and actually decrease operating costs and accelerate delivery.
126680	128660	So I'm taking a lot of those examples,
128660	130920	and then I'm combining it with kind of my night job,
130920	132200	which is security research.
132200	135980	I mean, security is my, it's my job, but it's also my hobby.
135980	138940	It's what I love to do anytime that I have spare time
138940	141060	that's not either working or with family,
141060	144340	I'm probably doing security research in some way, shape or form.
144340	146520	So feel free to check out these projects.
146520	148120	I have a blog on Medium.
148440	152060	Any code, any examples, any demos, the slides,
152060	155660	everything from this presentation will be available in GitHub.
155660	157160	So you can start checking that out.
157160	159240	If you want to follow along with anything,
159680	161600	a lot of the content, the resources, the notebooks
161600	163220	that we go through will be available there.
163220	165340	So I encourage you pull it up while I'm talking,
165340	166460	do that, check things out,
166460	168720	and feel free to ask questions as we go along.
168720	171240	But thanks again, and let's jump right in.
175800	177960	So what are we actually going to cover?
177960	182440	For the agenda today, there's really a couple of things,
182440	184780	but I really want to focus on inputs and outputs.
184780	185780	I think it's a mindset.
185780	188780	We always talk about the hacking mindset, the security mindset.
188960	191320	Well, in this case of this talk and the scope,
191320	193000	I want you to think about inputs.
193000	194140	What are the objectives?
194140	195140	What are the goals?
195140	196760	What do you put in?
196820	198100	And then what are those outputs?
198100	199980	How can we normalize these outputs so that way
199980	201060	you don't have to try to remember
201060	203400	how do you do 500 different scripts?
203400	205140	How do you make these tools work?
205140	208340	How do I manually parse through all these outputs and data?
208360	211840	So throughout this, we're going to show kind of modular,
211840	215020	repeatable technology, agnostic designs.
215260	218820	And then how do you apply this from a cloud-focused technology?
218820	222420	And we're going to spend our time in the AWS ecosystem.
222420	226140	However, you really could translate this to any of the major cloud providers.
226240	228360	And then also, we're going to provide some solutions.
228360	232380	So how do you tie this all together so that way you can become more efficient?
232380	234600	And I'm hoping that if you watch this talk,
234600	237120	you apply these concepts, your key takeaways are
237120	239160	I want you to make yourself better.
239160	241160	I want to make the people around you better.
241160	243260	And I'm hoping that overall, we can all work together
243260	244740	and make the industry better.
248000	250480	So this is the story of my life.
250480	252900	Thank you, Jason, for sharing these tweets
252900	254620	because the timing was perfect.
254620	256520	And I started thinking in terms of,
256520	257880	and this is in the scope of Bug Bounty,
257880	260380	if you've ever worked on Bug Bounty,
260380	261720	you see these different tweets of
261720	263880	I just made $5,000.
263880	265580	Or I made a $15,000 bug.
265580	267100	Or I just made $400.
267980	269980	It's kind of, that's how I always feel,
269980	272000	is wow, let's just spend a couple hours,
272000	273460	let's make a couple thousand bucks.
273460	274620	But that's not the case.
274620	278280	And I'm probably, I don't know if this resonates with any of you,
278280	280860	but I'm the person that if you were to ever see me tweet
280860	283040	I just made $2,000 on a Bug Bounty,
283040	284980	that's because I just probably put in
285340	286900	a hundred hours trying to research
286900	289320	through reconnaissance, through tooling,
289320	291640	through tests and failures and inputs
291640	293720	and report writing just to make that.
293720	294700	And that's what goes on, I think,
294700	296640	often behind the scenes that we overlook.
297040	299080	And one of the things is, too,
299080	302000	is our most valuable commodity is time.
302040	304440	My biggest challenge is finding the time
304440	306440	to be able to do this, to really get in the weeds,
306440	309360	to dig in and try to find these red team objectives
309360	311780	to relearn, I'm always relearning things
311780	313680	or figuring out or looking up or Googling
313680	318480	or all the places where I'm trying to relearn
318480	320420	all the things I've learned historically.
320580	321700	How can we make it faster?
321700	323360	How can we be more efficient?
323400	325240	So when I have these two or three hours
325240	328160	in an evening to spend, I can be focused.
328160	329320	I can look at my targets.
329320	331100	I know exactly what I want to do.
331100	332200	I don't have to spend hours
332200	334260	doing the reconnaissance research
334260	335720	trying to just make selections
335720	336840	on what my targets are
336840	338680	because that takes up all my time.
338680	340800	And then the time to value is really zero
340800	341560	and net none.
341580	343660	I'm going to focus on how do I be productive?
343660	344860	How do I make money?
344860	346140	And how do I find issues?
346140	348940	And further, what I'm attempting to do.
348940	351900	So that's what really drew me into
351900	354060	how do I build this automated ecosystem
354060	356600	and take an advantage of the new technologies
356600	357920	and new capabilities out there
357920	360600	so that I can be the best that I possibly can.
364120	366280	So now let's jump into the architecture
366280	367880	of the underlying design
367880	369740	of the entire automated ecosystem
369740	371500	that we're going to build and talk about
371500	373180	through the remainder of this presentation.
373200	374820	We'll have a bunch of demos
374820	377260	and code snippets and things
377260	378960	that we're going to kind of deep dive in
378960	381360	and hoping that you can leverage these as accelerators
381780	384500	so they can be agnostic to a specific tool,
384580	385600	a specific process.
385600	386800	But as you look at different things
386800	387940	that you do and experience
387940	389580	from a Red Team perspective
389580	391580	or even just from a general
391580	393320	information security process
393320	395800	or practice that you do in your daily jobs,
395800	399020	these are all components that can apply
399020	400460	in a lot of different ways,
400460	401780	which I think is really exciting.
401780	404120	And that's a big part of why I feel like
404120	405700	rather than just trying to deliver
405700	407640	this custom tool that is just a script
407640	408680	or something that you run
408680	410160	and you routinely run it
410160	411900	and it does the same thing every time,
411900	413580	what I want to kind of teach
413580	414980	is that thinking in terms of
414980	417120	how do you build that underlying infrastructure
417120	418400	that can support anything
418400	419980	that can be thrown at you?
419980	421340	How can you better yourself
421340	422560	and understand rather than
422560	424320	you kind of move up from
424320	426000	knowing just how to run a tool
426000	427180	to actually understanding
427180	428860	what's going on behind the scenes,
428860	430060	how do I piece this together
430060	431880	and how do I make it my own?
431880	432940	So I have this system
432940	435080	that can be potentially better than others
435080	437460	or bring you to the best in the industry
437460	438920	or just solve challenges
438920	441020	that you have on a day-to-day basis.
441240	443240	So as we look at this architecture,
443240	445760	it's really broken down into three key pieces.
445760	449020	And on the left side is the user interaction.
449140	450220	So these are the components
450720	451740	of the cloud services
451740	453800	that you're going to generally interact with,
453800	454780	whether it's an input,
454780	455840	whether it's an output.
456060	457700	But in the design that I have
457700	458820	that we're going to walk through,
458820	460480	it's going to leverage these four pieces.
460480	462880	And we're going to, as we go further,
462880	464140	I'm going to give you a quick overview
464140	465320	of every one of these components
465320	466820	just to explain what it does,
466820	468000	where it's used for.
468040	469040	And in the back of your mind,
469040	470040	continue to think,
470040	472580	how can I reuse this for X function
472580	473680	or what I'm trying to do
473680	474700	or my daily job?
474700	477140	And how can I convert my manual efforts
477140	478620	that I do on a day-to-day basis?
478620	479840	How can I save time,
479840	480920	be more efficient,
480920	481900	and automate this
481900	484040	so that way I can make my life better?
484960	486640	Basically, you can automate yourself
486640	488880	and then you can focus more
488880	491120	on the latest, greatest technology emerging.
491120	493060	Push your program forward,
493060	494640	mature in everything you're doing,
494640	496860	and just continue to build and build on top
496860	497960	so we're not relearning,
497960	499460	we're not doing the same manual tasks
499460	500560	over and over.
500580	502680	And this ecosystem will really elevate that
502680	503720	so that way you can do
503720	504360	and you can build
504360	505180	and it's modular
505180	508060	and it's really an ecosystem
508060	510680	of a lot of microservices in doing that.
510800	513320	The middle layer is your processing and computation.
513320	514700	So you have your user interaction
514700	516240	that will pass everything over
516240	517960	to your processing and computation.
517960	520280	And this is a pretty big arsenal
520280	521440	of different tools.
521440	524860	You have, whether it's big data type,
524860	527140	query analytics services,
527140	528540	it could be eventually,
528540	529980	we're not going to show any of these today,
529980	531460	but if you get into machine learning
532440	533880	or image recognition
533880	536760	and thinking in terms of future opportunities
536760	538780	of like you have screenshots of webpages,
538780	539460	maybe you want to process
539680	541660	a bunch of images of those really quickly,
541660	542700	this would be your processing
542700	544360	and computation layer.
544460	545880	And then the last layer,
545880	547000	I think is pretty key
547000	547860	and it's really important
547860	550160	to get this in the mindset
550160	552780	as you kind of consider future opportunities,
552780	554460	is that data storage layer.
554540	555480	You look at today,
555600	557720	a lot of the processes you may have,
557720	559140	like if you're doing reconnaissance,
559140	561660	you might run five or six different tools.
561660	562760	In those inputs,
562760	563760	you're going to submit,
563760	564780	maybe it's a domain,
564780	566080	maybe it's an ASN,
566080	568380	maybe it could be a CIDR range.
568380	569780	It could be various things.
569780	570700	You input that.
570700	571820	Generally, the outputs
571820	573640	of your various tools and scripts,
573640	575400	they're going to be slightly different.
575400	577300	And that's where that source data bucket
577300	578320	comes into play.
578320	580300	So I expect, anticipate,
580300	582600	let's pull in 100 tools eventually together.
582600	583880	Let's all put that data,
583880	585680	source data into that source data bucket
585680	588000	that's just various formats and everything.
588300	589260	And then what we want to do
589260	590380	is we want to think in terms of
590380	591580	how do we normalize this?
591580	593420	What's the actual objective of this?
593420	595060	Because in a lot of cases,
595060	596260	I mean, and we're going to look from
596260	598900	kind of a red team bug bounty perspective,
598900	599820	but think broader
599820	601760	as you think about your actual job,
601760	603540	whether you're doing risk management,
603540	605720	whether you're doing third party analysis,
605720	608020	whether you are a security engineer,
608020	608640	whether you're an architect,
608640	609820	any role in this,
609820	611660	think about how this can apply.
611680	613240	But we look at our inputs and outputs.
613240	614260	So a lot of our tools,
614260	615580	they're either going to output things
615580	616940	such as URLs.
616940	619560	It might be parameters
619560	621420	for a different web page.
621420	623680	You could have the ASN numbers
623680	624620	for network ranges.
624620	625740	You could have CIDR ranges.
625740	627080	You could have domain names.
627080	628140	You could have subdomains.
628140	629580	You could have company names.
629580	632220	You could have certificate assignments,
632220	634000	different inputs and outputs.
634000	635480	So those are all of your various outputs
635480	636840	that you're generally going to have
636840	638040	across your tools.
638340	640540	We need to make these common between that
640540	642040	so we can further process this
642040	643160	and make this data useful
643160	645080	because I know one of the areas
645080	646440	that I struggle with the most
646440	648420	is parsing through this data of every tool
648420	650180	and it just takes so much time.
650220	651780	And that's something that we could,
651780	653340	if you do it one time for a tool
653340	654520	where you normalize the data,
654520	655980	you never have to do it again.
656100	658760	And the cooler part about this is
658760	660740	we only need one person in the industry
660740	661860	to do it for per tool
661860	663320	or start to kind of standardize
663320	664380	and build in.
664480	666360	And that's where on my GitHub
666360	668720	you'll see Project Straylight.
668900	669760	And my goal is really
669760	672660	to just have this running open network
672660	674980	of here's accelerators we can do,
674980	675780	here's code snippets
675780	677040	that have helped me along the way.
677040	679200	And it's maybe not so much a technology
679200	680660	or tool that you install,
680660	682340	but it's just a bunch of ideas
682340	684040	and things that have helped me in my career
684040	686260	that over the past 10 to 12 years
686260	688700	that I've either had to relearn or do,
688700	689980	I'm trying to aggregate it all
689980	691220	into this ecosystem,
691220	692500	this architecture that we see
692500	695060	of how can I do what I've spent
695060	698120	so many hours doing redundantly, manually,
698120	699100	and how can I make it better
699100	701280	so that way the newer generation,
701280	702620	the next people that are going
702620	703920	to be experts, leaders in this,
703920	704940	can just build on top
704940	705720	of what we already have
705720	707260	so we're not continuing to rebuild
707260	709140	and we can advance further and further
709140	711340	and we can win in this industry.
711340	713680	So that's a lot of the intent there.
713680	715440	So as we look at the data storage layers,
715440	717120	you basically go from your raw data,
717120	719100	which could be the output of any tool,
719100	721240	you get to more of a normalized version.
721580	723140	And then you also get
723140	725300	into your presentation layer.
725300	727040	And that's when it's your reporting,
727040	728680	it's your analysis, it's your metrics,
728680	730100	it's the outputs that you want to show
730100	731780	for the delivery of your work.
731780	733180	And this could speak to any level
733180	734520	depending on how you present it.
734520	736360	So you could do an aggregate of
736360	738360	here's all the attacks that we've had
738360	739840	over the course of X days,
739840	741940	summarized into charts, graphs,
741940	743900	and that might be at more of a CISO level.
743900	746240	You could have indicators of compromise
746240	747460	or different signatures,
747460	748020	different levels,
748020	750620	which you would feed into an engineering team.
750620	752340	That could be part of your presentation.
752340	753440	But it's how do you normalize
753920	754880	and how do you show this
754880	756860	in kind of a formatted way
756860	758040	so that way if you have to do
758040	760520	monthly vulnerability reports or metrics
760520	762860	or showing analysis of how you've
762860	764700	further progressed or analyzed websites
764700	766540	or targets and you want to compare
766540	768200	what you've done, what you haven't done,
768200	770340	that's where that presentation layer comes in.
770440	771240	So we're going to jump in
771240	772440	and I think we're going to do,
772440	773420	let's do a demo.
773420	774500	And let's kind of show,
774500	775520	I'll show from end to end
775520	776400	kind of one area
776880	778020	where we can go across this
778020	779480	of how this ecosystem works.
779480	780740	And then after that,
780740	782280	we'll dive in further.
782280	783600	I'm going to show you the accelerators,
783600	785040	the pieces, the components.
785060	786460	Keep in mind that all of this
786460	787860	is also available in GitHub.
787860	789300	You can access it as we go along.
789300	790040	You can check it out.
790040	791200	Feel free to use it.
793180	794400	Create issues in the GitHub
794400	796140	if you find any things.
796140	797360	Reach out to me with questions.
797360	798500	But I really just want to continue
798500	800020	to build and teach you
800020	801060	how you can kind of think
801060	801860	in terms of building
801860	803840	your own automated ecosystem.
805480	806920	As you're watching this too,
806920	808740	you're probably thinking,
808740	809640	why cloud?
809640	811260	I have a really nice home lab.
811260	812620	Do I really need it?
812620	814040	There's three key areas
814040	815280	beyond just the amount
815280	816620	of what we just walked through
816620	817300	in that architecture
817300	819020	that you have available to you.
819020	820780	But it's pretty exciting
820780	821740	when you think about
821740	823760	what cloud can mean to you.
823760	825040	And if you hear...
825040	826840	I mean, the businesses, industries,
826840	828220	everybody is adopting it.
828660	829760	That's nothing new.
829760	831160	It's not anything new there.
831160	832920	But I think there's three key parts
832920	834720	of cloud that are really exciting
834720	836560	as just an individual researcher
836560	837520	where you don't need
837520	838260	millions of dollars
838260	839280	or tens of thousands of dollars
839280	840360	of funding behind you
840360	841980	to actually make progress.
842040	843280	The first part is
843280	844920	the democratized accessibility
844920	846200	of datasets.
846420	847880	There's some pretty massive
847880	850240	and interesting datasets out there.
850240	851300	The problem is that
851300	852420	on our home labs,
852420	853300	our home computers,
853300	854420	our home servers,
854560	855420	a lot of those don't have
855420	856620	either the horsepower
856620	858920	to accomplish what we need.
858920	859860	We need more memory,
859860	860540	more CPU,
860540	861760	or it's not close enough
861760	862220	to the data
862220	863440	where we can't download
863440	864340	petabytes of data
864340	865260	or terabytes of data
865260	866340	in a meaningful way
866340	867480	to keep up
867480	869280	and be able to analyze and do.
869280	869900	And that's where
869900	871240	you kind of run into issues
871240	872520	of data gravity.
872520	873260	And you need to build
873260	874180	your compute actually
874180	875560	closer to the data stores,
875560	876720	the data repositories,
876720	878040	because it's costly.
878040	879700	It's expensive to move those.
879700	881320	So how can we proceed
881320	882380	with that?
884120	885640	The other piece is that
885640	887340	if you look across the world,
887340	888680	we're facing right now,
888680	889880	we're all in quarantine,
889880	892520	we're social distancing.
892540	892900	You have...
893520	894520	So take a look
894520	895260	at what's going on
895260	895940	around the race
895940	897220	to find vaccines
897220	898540	and solutions to COVID
898540	900120	and kind of the data
900120	901720	around that in the pieces.
901820	903580	The world is coming around cloud
903580	904840	and the latest, greatest
904840	905770	computational technologies
906180	907020	to solve the world's
907020	907900	biggest problems
907900	910100	that impact everybody globally.
910160	912320	So what if we can take
912320	913020	those pieces
913020	914620	that the top scientists
914620	915280	in the world,
915280	916220	the top in their industries
916220	917240	are taking to solve
917240	919480	the world's most challenging problems?
919520	920620	I'm sure we can apply that
920620	922680	to our daily bug bounty targets
922680	924580	or our daily security processes
924580	926020	that we need to improve.
926020	926940	But in the same way,
926940	927720	we don't have to spend
927720	928920	tens of thousands of dollars
928920	929540	for this.
929540	931080	The other neat thing about it
931080	933080	is it's financially viable.
933180	935180	We can now stand up servers
935180	937360	that have 128 gigabytes
938140	939100	of memory.
939100	942100	You can do unlimited storage.
942100	943120	And the nice thing is
943120	943900	you don't have to pay
943900	944700	thousands of dollars
944700	946500	to buy this from a capital standpoint.
946500	948240	It's all operating expense.
948240	949560	So as you look at it,
949560	950720	if you need to run something
950720	952160	that's extremely memory intensive
952160	953640	and it's just one operation
953640	954560	or a loop,
954560	955700	you literally could stand up
955860	956760	a massive server
956760	958360	for just a couple minutes.
958360	960320	It might cost you $10, $20
960320	962280	for maybe five or ten minutes
962280	963380	of processing.
963620	964480	But rather than needing
964480	965900	to invest $80,000
965900	967960	in similar hardware equipment,
967960	968900	you now can do this
968900	970760	in your home office,
970760	971540	in your home lab
971540	973940	through the cloud capabilities
973940	974640	charging.
974640	975520	And that $20,
975520	976240	if you think about it
976240	978320	in terms of bug bounty investment,
978320	980120	that's one P4.
980120	982380	And you have that well covered.
982380	983540	I mean, that's on the low end
983540	984820	thinking in terms of that.
984820	985740	But just looking at it
985740	986940	from a business perspective,
986940	987680	thinking in terms of
987680	990360	how can I continue to invest
990360	991680	in my capabilities,
991680	992440	the technologies,
992440	993240	the processes,
993240	994660	so I can gain more time
994660	995820	and I can make the time
995820	997640	that I spend actually manually
997640	998460	doing things,
998460	999760	evaluating parts,
999760	1000420	how can I make it
1000420	1002160	the most beneficial to myself.
1007700	1008800	So now let's take a look
1008800	1010520	at the various tooling
1010520	1011740	that we can use.
1011740	1012160	We've looked at
1012160	1013380	the architecture diagram.
1013380	1014020	I'd like to just give
1014160	1014800	a quick overview
1014800	1015900	of the different components,
1015900	1016900	especially if you're not familiar
1016900	1018200	with the AWS services
1018200	1019200	that are listed on here,
1019200	1020160	kind of what they do,
1020160	1021040	why they're important
1021040	1022780	and how they can apply to you.
1022860	1024520	So if we go through the list,
1024520	1027500	you saw from the user interaction space
1027500	1029360	the Amazon API Gateway.
1029480	1030100	So what this does
1030100	1031460	is this is just an entry point
1031460	1032900	into your ecosystem.
1032900	1034120	And the neat thing
1034120	1035540	about Amazon API Gateway
1035540	1037200	is there's a lot of things
1037820	1038560	you can do with it,
1038560	1040000	but where I've really leveraged it
1040000	1041640	is as a proxy
1041640	1043200	into your Lambda function.
1043200	1043840	So what you can do
1043840	1045100	is you can do basically
1045320	1045880	a GET request
1045880	1046800	or a POST request
1046800	1047940	through the API Gateway
1047940	1049580	and pass that directly
1049580	1052020	into an AWS Lambda function.
1052160	1053400	And if you think in terms of
1053400	1054000	what you've seen
1054000	1055800	in the Jupyter notebooks
1056400	1057640	and Python libraries
1057640	1058860	and any functions you run
1058860	1059960	where you pass something,
1059960	1061100	you literally can pass it
1061100	1062140	through a GET request
1062140	1062920	through the API Gateway
1062920	1064680	into AWS Lambda.
1064680	1065500	So I kind of view that
1065500	1067920	as top tier level of services
1067920	1069240	where you've really normalized
1069240	1070140	your content,
1070140	1071780	you have exception handling,
1071780	1073220	you know that you're ready
1073220	1074640	to completely automate this,
1074640	1075300	let's promote it
1075300	1076640	into a Lambda function,
1076640	1077660	pull it out of your
1077660	1078560	Jupyter notebook,
1078560	1079660	and then let's put it
1079660	1080780	behind an API Gateway
1080780	1081420	so you can really
1081420	1081860	kick it off
1081860	1083140	and call it ad hoc
1083140	1083740	as you need
1083740	1084820	and really start
1084820	1086300	your ecosystem of automation
1086300	1087580	around there.
1088340	1089280	Another tool is
1089280	1090320	Amazon SageMaker
1090320	1091200	and it's really
1092640	1093520	a larger framework
1093520	1095560	and compute system
1095560	1096680	where you can do more things
1096680	1097560	like machine learning
1097560	1098620	and training
1098620	1099400	and all of that.
1099400	1100740	But what I pretty much
1100740	1101620	solely use it for
1101620	1102740	are the Jupyter notebooks.
1102740	1103580	So you can stand up
1103580	1104400	the Jupyter notebooks,
1104400	1105260	you can run it,
1105260	1106640	you can adjust the size on it
1106640	1107800	so if you need a lot more
1107800	1109620	gigabytes or storage
1109620	1110460	for a duration
1110460	1111540	or a function,
1111800	1112280	a lot of times
1112280	1113320	I'll ramp up
1113320	1115560	and use a larger
1115560	1116340	instance of it
1116340	1117500	for just a few minutes
1117500	1118760	or just to do that process
1118760	1120720	and then I'll turn it down
1120720	1122140	to kind of the standard level.
1122140	1123460	So Amazon SageMaker
1123460	1124380	is really the tool
1124380	1125300	mechanism for running
1125300	1126600	the Jupyter notebooks.
1126900	1128280	The static website
1128280	1129200	hosting S3,
1129200	1130240	that's an S3 bucket,
1130240	1131640	you can actually set those up
1131640	1132400	so that they can host
1132400	1133460	web content,
1133460	1134600	make them publicly available
1134600	1135320	so you hit it,
1135320	1136580	you can set an indexed
1136580	1137580	HTML page,
1137580	1138500	you can run it,
1138500	1139360	and you can run websites
1139360	1140280	out of S3 buckets
1140280	1142060	which is really neat.
1142940	1145360	The Amazon simple notification service,
1145360	1146460	it's a way that you can
1146460	1147640	send push messages
1147640	1149820	to whether it's SMS text,
1149820	1151240	whether it's emails,
1151240	1151860	different things.
1151860	1152500	So it's really just
1152660	1153600	a notification service
1153600	1154600	that we can do.
1154600	1155200	In this case,
1155200	1156280	I'll show a code snippet
1156280	1157240	where we can actually
1157240	1158340	send ourselves text
1158340	1159340	so that way we know
1159340	1159900	if there's anything
1159900	1160520	that's going to take
1160640	1161260	a little bit of time
1161260	1162440	or you just want to get notified
1162440	1164360	or have the final URL
1164360	1164940	where you're going to
1164940	1165700	access your data,
1165700	1166420	you can get texted
1166420	1167560	whenever it's ready.
1167560	1168440	So you can kind of do
1168440	1170240	bug bounty, reconnaissance,
1170240	1171260	all of this on the fly
1171260	1172280	as you build this system
1172280	1173420	which is pretty cool.
1175140	1176420	Now as we move into
1176420	1178940	the processing and computation layers,
1178940	1180800	AWS Lambda is really
1180800	1183200	what's used to run code.
1183200	1184480	So you can set it up,
1184480	1185900	it runs a lot of different things,
1185900	1186800	it's not just Python,
1186800	1187660	but it's Node,
1188560	1191500	and it runs on a serverless compute
1191500	1192980	so really you put
1192980	1194000	the code in there,
1194000	1194620	it will stand up
1194620	1195540	the compute resources
1195540	1196500	and it only runs
1196500	1197220	for a duration.
1197220	1198040	I think the longest
1198040	1199140	that you can run a Lambda
1199140	1200340	is about 15 minutes
1200340	1201540	so you're charged
1201540	1202520	based on the duration
1202520	1203560	that the computation
1203560	1204700	is running.
1204780	1205620	So if you think in terms
1205620	1207280	of smaller Python functions
1207280	1208240	that you have ready
1208240	1209760	to automate and scale,
1209760	1210280	that's where you can
1210280	1211420	migrate them to the Lambdas
1211420	1212340	and they can just always
1212340	1213200	be ready to work
1213200	1214340	off of triggers.
1215560	1217560	The Amazon Elastic Block Storage,
1217640	1218620	a lot of times,
1218620	1219880	S3, there's a little bit
1219880	1220380	of a caveat
1220380	1221840	where S3 you get charged
1221840	1222900	on get requests
1222900	1223720	and put requests
1223720	1224580	and writes,
1224580	1225620	whereas EBS storage
1225620	1227020	you pay for how much storage
1227020	1227840	that you have assigned
1227840	1228480	and allocated.
1228480	1229460	So depending if I think
1229460	1230700	I'm going to do millions
1230700	1231760	of reads or writes,
1231760	1232920	sometimes I'll do a lot
1232920	1233760	of my testing against
1233760	1234780	Elastic Block Storage
1234780	1235820	and then once I have
1235820	1236580	something that I know
1236580	1237620	is ready to go robust
1237620	1238560	and I want to leverage it
1238560	1239680	in the greater ecosystem,
1239680	1240320	I'll move it over
1240320	1241580	to our data storage tiers
1241580	1242460	while they're in the refined
1243140	1245100	or the raw data levels
1245100	1246760	to do more normalized
1246760	1250020	and automated operations against.
1250860	1252120	AWS Glue is a way
1252120	1253140	that you can set up
1253140	1254540	and you can crawl data sets.
1254540	1255800	So Athena is,
1255800	1256560	you picture that,
1256560	1257280	is like you write
1257280	1258500	your SQL queries
1258500	1259640	and your query language
1259640	1260780	and it will run it
1260780	1261700	and distribute it
1261700	1262920	against data sets
1262920	1264240	and it can run
1264240	1265840	against CSV files,
1265840	1266480	it can run against
1266480	1267700	relational data stores,
1267700	1268900	unstructured data,
1268900	1269320	and all those.
1269320	1270240	So it's really powerful
1270240	1271420	and if you haven't ever
1271420	1272140	dug into it,
1272140	1273200	AWS Athena,
1273200	1274420	I highly recommend it.
1274420	1275480	There's a lot of power in it
1275480	1276240	and you can literally
1276240	1276960	just sit there
1276960	1278700	and use the console interface
1278700	1279880	and pretty much
1279880	1281120	ad hoc write queries
1281120	1282120	against big data sets
1282120	1283720	if you decide to as well.
1283960	1284880	And then Glue is,
1284880	1286100	I started talking,
1286100	1286960	it's what indexes
1287600	1288520	a lot of those data points
1288520	1289020	around it.
1289020	1289640	So you can create
1289640	1290460	Glue indexes
1290460	1291380	so you don't have to
1291380	1292940	query the entire data set
1292940	1293820	every single time
1293820	1295420	and you can do those.
1296360	1297520	AWS Step Functions,
1297520	1298320	you won't see anything
1298320	1299160	in the demos today
1299160	1300220	leveraging this,
1300220	1300900	but this is kind of
1301040	1301780	a future capability
1301780	1302680	where I want to leverage
1302680	1303900	where you can actually,
1303900	1304760	it's like setting up
1304760	1305920	workflows and processes
1305920	1307980	and you need X to happen
1307980	1309340	and then once X completes,
1309340	1309980	you can send the output
1309980	1310740	to this one
1310740	1311960	and do, right now,
1311960	1312860	I'm just leveraging things
1312860	1313560	off of triggers
1313560	1314260	and alarms
1314260	1316000	and Lambda files,
1316000	1317080	but you can really start
1317080	1318420	to use AWS Step Functions
1318420	1319580	as orchestration,
1319580	1321460	as a central core piece.
1321460	1322180	So something I'm going
1322180	1322900	to dig into more
1322900	1323840	in the future.
1324620	1325700	AWS Secrets Manager
1325700	1326720	is a place that,
1326720	1327560	so you can move
1327560	1328640	all of your secrets,
1328640	1329340	your API keys,
1329340	1330340	out of your code,
1330340	1331700	put them in Secrets Manager.
1331700	1333280	It's a quick call out.
1333280	1334520	You can load them,
1334520	1335380	you can store them,
1335380	1336220	and you can kind of
1336220	1337060	store other parameters
1337060	1338080	in there as you need as well.
1338080	1338580	You can pull them back
1338580	1339160	into your code
1339160	1341040	so they don't sit as code,
1341040	1341580	but you can use them
1341580	1342760	as variables on the fly.
1342760	1343500	So I highly recommend
1343500	1344200	digging in there.
1344200	1345040	There's some accelerators
1345040	1345600	that you'll see
1345600	1347120	that talk to those.
1347760	1349440	And then on the right side
1349440	1350720	are just all the S3 buckets
1350720	1351660	and you can create
1351660	1352580	multiple S3 buckets
1352580	1353660	and have the different stores
1353660	1354800	and it's just long term
1354800	1357380	data storage for you.
1360180	1362220	So let's jump in with a demo.
1362540	1364280	This demo is going to show
1364280	1365460	how we can query
1366080	1367460	Rapid7's Project Sonar,
1367460	1368380	which is essentially
1368380	1369760	it's the 4DNS
1369760	1371480	of the entire internet.
1371480	1372080	And it's sitting
1372080	1372840	in a dataset
1372840	1374160	within S3.
1374160	1374980	So rather than try
1374980	1376400	to copy that entire dataset
1376400	1377980	down into our own system,
1377980	1379160	we're going to leverage
1379360	1380100	a Jupyter Notebook
1380100	1381100	and we're going to actually
1382120	1383600	run a distributed query
1383600	1385200	against that external dataset
1385200	1387400	that's hosted in
1387400	1389260	not our own S3 bucket,
1389260	1391200	but the public datasets bucket.
1391380	1392420	And we're going to get
1392420	1393100	the results back
1393100	1394520	and we're going to process it.
1394520	1395280	And we're going to do it
1395280	1396740	in just a matter of...
1396740	1397620	it's probably going to take
1397720	1398200	a couple of minutes
1398200	1398780	as we walk through
1398780	1399900	and talk through the code,
1399900	1400960	but this is literally,
1400960	1402140	I think generally the queries
1402140	1403160	around the 4DNS
1403160	1404660	for a domain wildcard
1404660	1406440	is about 26 seconds.
1406660	1408740	So let's walk through this.
1409540	1410620	So on the screen here
1410620	1412580	you can see my Jupyter Notebook
1412580	1414440	and what we'll do is
1414440	1415280	we'll walk through this too
1415280	1416160	because I want you to think
1416160	1417420	in terms of how can I reuse
1417420	1418200	these components,
1418200	1419420	how is this applicable
1419420	1421020	well beyond just this
1421020	1422380	sonar use case.
1422480	1423720	So let's go ahead and start.
1423720	1424800	We're going to go ahead
1424800	1426240	and run this first section
1426240	1427100	of code here.
1427100	1428140	And this is what I talked about
1428140	1429220	earlier too, which is nice
1429220	1430400	is that if you're not
1430400	1431480	an expert at code,
1431480	1432120	you run it,
1432120	1432860	you can break it down
1432860	1433620	into cells,
1433620	1434700	and you can run each cell.
1434700	1435260	So we're going to run
1435260	1436440	this first cell.
1436520	1437640	What we just did was
1437640	1440280	you can see this execution ID here.
1440300	1442440	What we did is we set our domain
1443300	1444600	to Microsoft.com
1444600	1445680	so we're actually querying
1445680	1446720	all the wildcards
1446720	1447820	of the Microsoft domain
1449180	1450080	4DNS space.
1450080	1450820	And keep in mind,
1450820	1451720	we are not touching
1451720	1453580	any of Microsoft's systems.
1453580	1455240	This is purely utilizing
1455960	1457340	the 4DNS dataset
1457340	1458580	and we're getting the results
1458580	1459440	from there.
1459800	1460620	And then what we've done
1460620	1461840	is we've set some
1461840	1462960	of our variables around
1462960	1463740	our bucket of where
1463740	1465480	we want to store the results,
1465480	1466720	what database we want to query
1466720	1467740	against in Athena,
1467740	1468700	which we've had some...
1468700	1470040	the setup information
1470040	1470800	is all contained
1470800	1471740	within the GitHub as well
1471740	1473120	so you can set that up.
1473120	1473800	And then the table
1473800	1475340	we want to query against.
1475580	1476520	What we do next
1476520	1478680	is we set up our query.
1478680	1479460	So what this is doing
1479460	1480600	is basically just saying
1482340	1483140	select everything
1483140	1484500	from wildcard
1484500	1486000	microsoft.com
1486000	1487440	and then we select
1487440	1488240	the latest date
1488240	1489180	of the dataset
1489180	1490040	so that way we don't
1490040	1490900	historically go through
1490900	1492160	every single one iteration
1492160	1493260	that we have.
1493880	1495080	It issues the
1495560	1496640	Athena query.
1496640	1498000	It takes about 26 seconds
1498000	1498400	to run.
1498400	1498900	We have the
1498900	1500740	Athena query ID here.
1500740	1501520	So in this next
1502020	1502820	component,
1502820	1503820	I will actually
1503820	1504560	jump back up.
1504560	1505500	This run command
1505500	1506960	that I'm highlighting,
1506960	1508160	this is actually a way
1508160	1508900	that you can call
1508900	1509720	other notebooks,
1509720	1510500	which is kind of neat.
1510500	1511240	So as you think about
1511240	1512480	in terms of you call functions
1512480	1513900	within programming code,
1513900	1514940	this will actually run
1514940	1515860	and load an entire
1515860	1516440	notebook.
1516440	1517040	So what I've done is
1517040	1518060	I try to take anything
1518060	1519280	that's not tool specific
1519740	1521160	and I put it in kind of
1521280	1521980	a common notebook
1521980	1523200	that you can set
1523420	1523860	a lot of different
1523860	1524640	things to run.
1524640	1526220	So what this is
1526220	1527040	actually doing is
1527040	1528220	the query Athena
1528220	1529440	function is running
1529440	1530060	out of that other
1530060	1530840	notebook because
1530840	1532300	I've set it up well
1532300	1532980	enough that you can
1532980	1534020	pass in the
1534020	1535480	results bucket.
1535480	1536220	You just send it
1536220	1537040	the query in the
1537040	1537680	Athena database
1537680	1538780	and it will kick it off
1538780	1539500	so it doesn't matter
1539500	1540760	what tool specific
1540760	1541300	or anything.
1541300	1542340	It will just run a query
1542340	1543320	that you tell it to run
1543900	1545020	which is why I moved
1545020	1545880	it into that kind of
1545880	1546840	standardized function
1547740	1548460	area.
1549800	1550560	So what we're going
1550560	1551100	to do is we're going
1551100	1551900	to run this next one
1551900	1552520	and this is going to
1552520	1553340	generate, pull the
1553340	1554040	results back.
1554040	1554780	So you can see that
1554780	1555780	it's already succeeded.
1555780	1556420	If we were to run this
1556420	1557100	immediately it would
1557100	1557780	just go in a loop
1557780	1558900	and it would circle
1558900	1559760	every five seconds
1559760	1560460	and check for the
1560460	1561500	results back.
1561640	1562920	But it's pulled back
1562920	1565620	18,695 different rows
1565620	1566680	and you can see the
1566680	1567620	types of information
1567620	1568240	we've just pulled
1568240	1568720	back.
1568720	1570020	So think you have
1570020	1572320	18,695 different rows
1572320	1573360	basically for DNS
1573360	1573980	entries from
1573980	1575040	Microsoft.com in
1575040	1576220	under 26 seconds
1576220	1578540	with the IP
1578540	1579400	addresses, the C
1579400	1580480	names, A records,
1580480	1581500	MX records, all of
1581500	1582280	those.
1583000	1584040	And then with that
1584040	1584780	we're going to do
1584780	1585520	this next thing is
1585520	1586500	this is actually
1586500	1587560	going to leverage
1587560	1588260	the MaxMind
1588260	1589260	database and we're
1589260	1589980	going to start
1589980	1591140	figuring out
1591140	1592280	querying for the
1592280	1593760	IP addresses and
1593760	1594500	getting all that
1594500	1595300	information back.
1595300	1595840	So let's go ahead
1595840	1596560	and run this and
1596560	1597180	what this will do
1597180	1598500	is it creates a
1598500	1600480	data frame which
1600480	1601800	leverages Pandas.
1601800	1602080	So Pandas is a
1602080	1603080	library within Python
1603800	1604720	that you can run
1604720	1605980	and I completely
1605980	1606760	recommend it.
1606760	1607520	It is one of the
1607520	1608180	best things that I've
1608180	1609060	ever found in terms
1609060	1610560	of doing metrics and
1610560	1611200	analysis and
1611200	1611680	automation.
1611680	1612200	But you can see it
1612200	1614520	just processed 18,695
1614520	1615500	rows in just a few
1615500	1616120	seconds.
1616120	1617040	And now we have the
1617040	1618340	latitudes, longitudes,
1618340	1618840	countries, and
1618840	1619500	localities all
1619500	1620940	merged to all
1620940	1622480	18,000 entries.
1623140	1623460	So what we're
1623460	1624240	going to do now is
1624240	1625560	if you say, hey, I
1625560	1626220	just did all this
1626220	1627160	work, I don't want a
1627160	1627820	chance of losing it
1627820	1629160	even though theoretically
1629160	1630280	you could redo this
1630280	1630920	over about 30
1630920	1631660	seconds.
1631740	1632740	But if you want, you
1632740	1633620	can use this command
1634000	1635620	and you can save your
1635620	1636860	data frame off to
1636860	1638020	Excel so you can save
1638020	1638480	it locally.
1638480	1639200	So if you run this, it
1639200	1639920	will just print and
1639920	1640840	store those 18,000
1640840	1641720	records into an Excel
1641720	1642480	spreadsheet which you
1642480	1643580	can reload later on
1643580	1644960	which is pretty handy
1644960	1645700	to use.
1645700	1646500	And if you want to
1646500	1647600	save your artifacts and
1647980	1648860	data points, you can
1648860	1649860	always do that.
1651480	1652300	So what we'll do now
1652300	1652840	is we're going to run
1652840	1654020	this next command and
1654020	1654940	what this does is it
1654940	1655660	just aggregates.
1655660	1656200	And you can see how
1656200	1656800	fast that was.
1656800	1657780	It was almost instantaneous
1657780	1658580	when I pressed the
1658580	1659460	start button and it
1659460	1661120	just grouped up all
1661120	1662180	the different latitudes
1662180	1663320	and longitudes so that
1663320	1664220	we can get ready to
1664220	1665100	plot them on a heat
1665100	1665720	map.
1666040	1668260	So we know 525 had
1668260	1669000	this latitude,
1669000	1670560	longitude, and it did
1670560	1671580	all that work.
1671580	1672280	And we can dig in.
1672280	1673420	This also calls that
1674540	1675680	the other notebook
1675680	1676840	because it's
1676840	1677720	centralized and it's
1677720	1678900	not specific to the
1678900	1679860	project sonar data
1679860	1680300	set.
1680300	1682080	So you can reuse and
1682080	1683480	redo any of that.
1683660	1684340	I am going to jump
1684340	1685000	back up, though.
1685000	1685740	I think this is
1685740	1686300	something that's
1686300	1687040	pretty neat.
1687040	1687740	And if you think
1687740	1688560	about in terms of
1688560	1689780	how can I reuse some
1689780	1690920	of this, this get
1690920	1692000	location function that
1692000	1693200	I have behind the
1693200	1693900	scenes that just
1693900	1694900	pulled in and merged
1694900	1696060	all this together,
1696060	1696860	what I did, if you're
1696860	1698360	looking, I passed the
1698360	1700160	entire data frame as
1700380	1701240	a part of the function
1701240	1702900	there and I passed
1702900	1703840	the value column,
1703840	1704420	which is the IP
1704420	1705480	address column.
1705480	1706280	So if you think about
1706280	1707920	reusability, all I
1707920	1709220	literally did was I
1709220	1710220	just took, if you
1710220	1710740	have an Excel
1710740	1711640	spreadsheet of IP
1711640	1713620	addresses or a CSV
1713620	1714480	or anything like
1714480	1715100	that, you load it
1715100	1716180	into a data frame.
1716180	1716900	You can pass that
1716900	1717940	entire data frame.
1717940	1718380	It doesn't matter
1718380	1718540	what it is.
1718540	1718680	It doesn't matter what
1718680	1719380	the rest of it looks
1719380	1719740	like.
1719740	1720380	You pass the data
1720380	1722100	frame name and the
1722100	1723460	column that the IP
1723460	1724580	addresses are in or
1726380	1727240	obviously it will
1727240	1728560	fail on if it's not
1728560	1729400	an IP address as it
1729400	1730040	goes through those
1730040	1730560	because they're C
1730560	1731260	names.
1731700	1732880	And you do that and
1732880	1733660	you can process it.
1733660	1734440	So anything you have,
1734440	1735140	that's completely
1735140	1736060	reusable and that's
1736060	1737300	just a few lines of
1737300	1737660	code.
1737660	1738200	It's in GitHub.
1738200	1738940	You can pull it and
1738940	1740700	I think in terms of
1740700	1742340	just from a cyber
1742340	1743620	operations, a SOC
1743620	1744380	perspective, you
1744380	1745340	could really get a
1745340	1746100	lot of value out of
1746100	1746340	some of that
1746340	1746900	information.
1746900	1747320	You could look at
1747320	1748500	attackers, web
1748500	1749280	denials, web
1749280	1750520	traffic, who's
1750520	1751140	hitting your web
1751140	1752520	application firewalls
1752520	1753280	and you can do heat
1753280	1753900	maps and different
1753900	1754440	things from an
1754440	1755220	attacker perspective
1755220	1756460	which is pretty cool.
1757660	1758900	So let's go on down.
1758900	1759520	We have that
1759520	1760440	information and then
1760440	1762180	we want to now, let's
1762180	1762980	do this.
1762980	1763800	So this is just a
1763800	1764440	couple lines of
1764440	1766020	code but it's really
1766020	1766240	cool.
1766240	1767380	So this uses the
1767380	1768460	Google Maps API and
1768460	1769240	you can see how fast
1769240	1769580	that was.
1769580	1772340	It just plotted the
1773180	1773860	information we have
1773860	1774960	up here for the
1774960	1775160	latitudes and
1775160	1775720	longitudes and
1775720	1776460	created an
1776980	1777460	interactive heat
1777460	1778140	map so we can
1778140	1779760	zoom in on and
1779760	1780600	see what's going
1780600	1781640	on and this shows
1781640	1782540	the concentrations
1782540	1784180	of Microsoft's
1784180	1785280	forward DNS space
1785280	1786140	across the globe.
1786140	1787240	So you can look and
1787240	1788500	as we zoom in, you
1788500	1788920	can see the
1788920	1789580	different regions
1789580	1790760	they are included
1790760	1791500	in.
1792260	1793440	So let's, if we
1793440	1794500	were just, let's
1794500	1795240	say, let's tackle
1795240	1796120	this is a big area
1796120	1797300	here, you can zoom
1797300	1797960	in and even though
1797960	1798660	it shows Seattle
1798660	1800160	space, you can go
1800160	1800900	and see that it's
1800900	1801840	not all just in
1801840	1802440	Seattle but they
1802440	1803440	have a few offices.
1803440	1804000	It looks like in
1804000	1804920	Redman, there's some
1804920	1806540	Seattle and above.
1806540	1807920	So pretty cool that
1807920	1808880	it's all interactive
1808880	1810180	and you can do that.
1810180	1811140	And the other neat
1811140	1811580	thing is you're
1811580	1812820	thinking, well, I
1812820	1813440	want to show this to
1813440	1813980	management.
1813980	1814700	How do I show my
1814700	1815860	managers, my leadership
1815860	1817000	about this?
1817020	1818060	And I can't have
1818060	1819360	them just opening a
1819360	1820460	Jupyter notebook all
1820460	1821100	the time and doing
1821100	1821720	this.
1821720	1822440	Well, one thing is
1822440	1823260	you can click the
1823260	1824000	download button and
1824000	1824940	just download the
1824940	1826300	map as an image.
1826300	1827700	But the other pretty
1827700	1828160	neat thing about all
1828160	1829260	this, and if you
1829260	1830340	think about the
1831440	1832440	architecture of how
1830340	1830760	I have that
1831120	1832220	presentation layer,
1832220	1832720	what this next
1832720	1833700	command does is it
1833700	1835120	takes this map and
1835120	1836460	it actually uploads
1836460	1838000	it to a static S3
1838000	1838880	bucket that's
1838880	1840440	hosting a static S3
1840440	1841740	content as a
1841740	1842500	website.
1842500	1843560	So then I hit this,
1843560	1844940	I uploaded the map
1844940	1846520	file to the website
1846520	1847340	and then I can
1847340	1848360	navigate over to my
1848360	1849480	bucket and if I hit
1849480	1850780	load content right
1850780	1852380	here, it should pull
1852380	1854140	up that map.
1854140	1855180	And now I can
1855180	1856040	always send any of
1856040	1857160	my leadership, any
1857160	1857740	managers, anybody
1857740	1858440	that wants to see
1858440	1859200	anything, I can run
1859200	1859780	this on an
1859780	1860320	iterative basis
1858340	1859340	while I'm
1860320	1860580	there.
1860580	1861500	It's on every 24
1861500	1862280	hours I run this
1862280	1863340	process and do.
1863540	1864100	And if you think in
1864100	1865400	terms of visually what
1865400	1866220	you want your attack
1866220	1866980	surface to look like
1866980	1868200	for your company, run
1868200	1868800	this against your
1868800	1869860	company every time.
1869860	1870580	Have this dashboard
1870580	1871980	always going and then
1871980	1872740	you could even trigger
1872740	1873640	on anomalies, you
1873640	1874120	could do different
1874120	1874820	things.
1874900	1876140	And it's just pretty
1876140	1876860	neat to always see
1876860	1878000	here, here's our
1878560	1879340	coverage from an
1879340	1879940	internet presence
1879940	1880880	that we have there.
1880880	1882160	So I think there's a
1882160	1882880	lot of value, a lot
1882880	1883720	of opportunity to do
1883720	1884360	in there.
1884360	1885040	And just the fact
1885040	1885380	that you can
1885380	1886280	literally load it
1886280	1887660	and save the
1888980	1889860	interactive mapping
1889860	1891480	part of it to an
1891480	1893080	HTML file and then
1893080	1893860	have people actually
1893860	1894700	go up, browse to
1894700	1895540	it, navigate it,
1895540	1895940	that's pretty
1895940	1896740	exciting.
1901770	1902950	So for this next
1902950	1904230	demo, we're going to
1904230	1904970	use, this is actually
1904970	1905590	one of my favorite
1905590	1906630	ones and I'm really
1906630	1907590	excited to continue
1907590	1908310	the build on the
1908310	1909690	capabilities of this.
1909690	1911070	But this leverages a
1911070	1912390	data set that's
1912390	1913050	publicly available
1913050	1913950	in S3 called
1913950	1914870	Common Crawl.
1914870	1916430	And what it is, it's
1916430	1917030	a project that's
1917030	1917630	been going on now
1917630	1918610	for, I think it's
1918610	1919130	actually been around
1919130	1919230	for about eight
1919230	1920170	years, where it
1920170	1922210	crawls the internet,
1922210	1923530	searches and indexes
1923530	1925030	and actually downloads
1925030	1926710	the HTML of files
1926710	1927690	and stores them in
1927690	1928490	S3 buckets.
1928490	1929450	So it's just a
1929450	1931150	massive repository
1931150	1932290	and similar of
1932290	1933050	basically web
1933050	1933510	archives that you
1933510	1934030	can search.
1934030	1934990	So if you've used
1934990	1936270	Wayback Machine and
1936270	1936630	different components,
1936630	1938370	this is very similar.
1938370	1938790	And so what we're
1938790	1939290	going to do is we're
1939290	1940010	going to actually
1940010	1940970	search it for a
1940970	1941170	domain.
1941170	1942450	We're going to pull
1942450	1944370	back the files and
1944370	1945770	it actually can
1945770	1946470	eventually download
1946470	1947310	the files and we
1947310	1948010	can store them in
1948010	1948890	an S3 bucket and
1948890	1950010	then we can run a
1950010	1950590	bunch of different
1950590	1951110	tools around it.
1951110	1951750	So if we want to
1951750	1953390	do like find URLs or
1953390	1954890	run HackCrawler or
1955510	1956130	anything that's going
1956130	1956870	to go through and
1956990	1957830	does a good job of
1957830	1958710	analyzing HTML,
1958710	1959630	whether it's for
1959630	1960110	vulnerabilities,
1960110	1961530	whether it's for URLs,
1961530	1962390	whether it's for
1962390	1963450	parameters, we can do
1963450	1964210	that against it.
1964210	1965450	So let's walk through
1965450	1965870	this now.
1965870	1966810	I'll kind of show you
1966810	1967630	the framework around
1967630	1968450	it and there's a lot
1968450	1969770	that we can build into
1969770	1970670	this capability.
1970670	1971370	The biggest challenge
1971370	1972590	that I've had so far
1972590	1973470	is just trying to do
1973470	1975230	some of the
1975230	1975710	asynchronous programming
1977330	1977730	against it because a
1977730	1977810	lot of times you have
1977810	1978470	a large site where
1978470	1979070	you're trying to do
1979070	1980470	downloads and actually
1980470	1981610	pulling the files, but
1982030	1982790	as we get more
1982790	1983570	efficient with this, I'm
1983570	1984310	pretty excited about
1984310	1985110	this and I think it has
1985210	1986010	a lot of promise with
1986010	1986550	it.
1987490	1988110	So what we're going to
1988110	1989170	do is we have all of
1989170	1990030	our parameters set up.
1990030	1990630	We're actually going to
1990630	1991330	run this one against
1991330	1992250	the smaller domain for
1992250	1993290	demo purposes because
1993290	1994030	if we have something
1994030	1995490	that has 70,000
1995490	1996510	websites and pages,
1996510	1997390	it's just not fast
1997390	1998190	enough at this point
1998190	1999170	to download this in
1999610	2000470	a timely fashion.
2000470	2000990	We would be sitting
2000990	2001930	here for maybe 40
2001930	2002890	minutes or an hour.
2002890	2004410	So in this case, we
2004410	2005270	can always speed it up
2005270	2005910	as we go, but we're
2005910	2006770	going to run this
2006770	2008470	against derbycon.com.
2008970	2010510	So we've kicked off
2010510	2011570	our Athena query.
2011570	2012670	It's running it now.
2012670	2013530	It's searching.
2013530	2014530	It's looking for all
2014530	2015610	of the webpages that
2015610	2016650	this is indexed and
2016650	2017990	cached and copied and
2017990	2019670	saved within the
2019670	2021030	Common Crawl dataset.
2021430	2022270	So let's go ahead and
2022270	2023290	kick off our run and
2023770	2025010	what this is going to
2025010	2025710	do is it's going to
2025710	2027210	look and it's going to
2027210	2029090	just wait for the
2030470	2031730	query to stop running
2031730	2032830	and then it will pull
2032830	2033410	in and load the
2033410	2034230	results for us into
2034350	2034390	a database.
2034390	2035750	So we'll let this run
2035750	2036910	for a few minutes.
2037210	2038090	All right, so it looks
2038090	2038690	like we have our
2038690	2039410	results here.
2039450	2042290	So it pulled back 579
2042290	2043510	different websites.
2043510	2044770	What we want to do is
2045310	2046350	since it's been running
2046350	2047390	over the course of
2047390	2048750	eight years, there's a
2048750	2049510	lot of different
2049510	2050270	versions, iterations,
2050270	2050870	which is actually
2050870	2051510	pretty neat if you want
2051510	2052470	to see how websites
2052470	2053290	have evolved and look
2053290	2054270	for changes or maybe
2054390	2055270	there's a vulnerability
2055270	2057030	or an issue or a
2057030	2057870	comment in there that
2057870	2058950	they moved out and you
2058950	2060290	could see the difference
2060290	2060930	between the different
2060930	2061150	versions.
2061150	2063090	But in this case, I'm
2063090	2064290	interested more in just
2064290	2064370	the unique users that
2064370	2064870	we have.
2064870	2065610	So we're going to run
2065610	2066490	this next command which
2066490	2067330	is going to sort them
2067330	2068730	and it's going to drop
2068730	2069830	all the duplicates and
2069830	2070550	it's going to keep the
2070550	2071410	latest version of the
2071410	2071990	webpage.
2072010	2072610	So let's go ahead and
2072610	2073530	run this right now.
2073530	2073930	And that was pretty
2073930	2074810	instantaneous.
2075330	2076230	And now we're down to
2076230	2077510	73 different rows.
2077510	2078270	So there were a lot of
2078270	2079190	different iterations,
2079190	2081090	versions of the site
2081090	2081830	cached.
2082110	2082770	And then what this will
2082770	2083630	do is we can run this
2083630	2084610	next command, we can
2084610	2085750	save it off to Excel so
2085750	2086490	that way we can have
2086490	2088030	our URL listing for
2088030	2089150	future analysis, future
2089150	2089830	loading.
2091050	2093470	And then as we go on,
2093470	2094350	what this will do is
2094350	2095430	if you want to look,
2095430	2096990	this will pull out and
2096990	2098350	only list now our URLs
2098350	2099270	which is what we care
2099270	2100090	about a lot of times,
2100090	2100590	especially if we're
2100590	2101510	trying to scope, we want
2101510	2102750	to do web crawls.
2102750	2103510	And keep in mind that
2103510	2104370	everything that we're
2104370	2105230	showing today, everything
2105230	2107570	is completely 100%
2107570	2108130	passive.
2108130	2109370	We are not touching any
2109370	2110110	of these sites, these
2110110	2111170	webpages or anything.
2111170	2112610	So this is all examples
2112610	2113490	of what you can do,
2113490	2115430	100% passive reconnaissance.
2115430	2116430	So we will never touch
2116590	2117590	a system of any of these
2117590	2118250	domains that we've
2118250	2118990	looked at.
2118990	2119990	And then for next steps
2119990	2121230	then you start, you can
2121230	2123170	start based on scope and
2123170	2124270	allowances and permissions
2124270	2125190	and things like that to
2125190	2126430	get more intrusive as we
2126430	2128230	do further analysis,
2128230	2128950	whether it's from a
2128950	2129930	red teaming perspective
2129930	2130990	or however.
2130990	2131790	But what we have right
2131790	2133130	now is just a listing of
2133930	2135230	URLs and then what this
2135230	2137190	next big section of kind
2138110	2139130	of a lot of, a lot of
2139130	2140070	different functions and
2140070	2141210	code, I mean it's still
2141210	2142250	for, for the value of
2142250	2143270	what this does, it's,
2143270	2144170	it's not too bad in
2144170	2145230	terms of lines.
2145230	2145890	But what this is going
2145890	2146850	to do is it's going to
2146850	2148390	go through now of the
2148390	2148770	query results.
2148770	2149930	So in this case we're
2149930	2151270	dealing with what, like
2151270	2152010	73 records, so it's not
2152010	2152330	that many.
2152330	2152870	But if you were to run
2152870	2153970	something, I think I ran,
2153970	2155450	I think I ran one earlier
2155450	2156110	this year against
2156110	2159170	defcon.org and it came
2159170	2160650	back with about 70,000
2160650	2161110	records.
2161110	2161870	So some of these,
2161870	2162690	there's a lot of data
2162690	2163570	and content out there
2163570	2164670	and it can take some
2164670	2164950	time.
2164950	2166010	And this is where, this
2166010	2167290	is not that efficient,
2167290	2168510	not that streamlined, a
2168510	2169430	lot of improvement
2169430	2170350	opportunity here.
2170350	2171710	But it, it basically
2171710	2173210	searches, it uses a
2173210	2174170	library in Python called
2174170	2174890	Beautiful Soup, which
2174890	2176270	you can parse down
2176270	2176590	HTML.
2176590	2178050	It's great for HTML
2178050	2178570	processing.
2178570	2180010	It has some building
2180010	2180950	capabilities to look for
2180950	2181970	comments and URLs and
2181990	2182510	links.
2182510	2182850	So we're going to start
2182850	2183390	setting this up.
2183390	2184410	And what, I have three
2184410	2185370	parameters that we can
2185370	2186590	pass in, just mostly out
2186590	2188850	of efficiency, is we can,
2188850	2189950	we can say we want to
2189950	2191230	search the files and
2191230	2192050	generally we're going to
2192050	2193490	always want to do that.
2193990	2195010	Do we want to write the
2195010	2195290	files?
2195290	2196050	So what this does is
2196050	2197050	this goes out and it
2197050	2198710	retrieves, well, it's
2198710	2199370	always going to go out
2199370	2200470	and retrieve the files
2200470	2201750	from the S3 buckets, but
2201750	2202970	do we want to write them?
2202970	2204150	So in this case, we're
2204150	2204770	not going to write them
2204770	2205590	for this demo purpose,
2205590	2206370	but you could write it
2206370	2207690	to an S3 bucket.
2207730	2208590	Now you have your HTML
2208590	2209530	files that you can run
2209530	2210550	additional tools and
2210550	2211110	things against.
2211110	2211730	So it's, it's pretty
2211730	2212510	valuable there.
2213150	2214090	And then we can say how
2214090	2214730	many records.
2214730	2215650	So if we just want a
2215650	2216970	subset for testing, if
2216970	2217630	we don't want to pull
2217630	2218950	down all 60,000 at once,
2218950	2219990	we could, we can assign
2219990	2220930	how many records and it
2220930	2221850	will loop through and
2221850	2223210	iterate over and download
2223210	2224190	the number of records we
2224190	2224770	say.
2224770	2225870	And then we, we run
2225870	2226770	this and this function
2226770	2228070	will basically process
2228070	2228570	these.
2228570	2229590	It's going through, you
2229590	2230410	can see I have a
2230410	2231170	counter on it because a
2231170	2232190	lot of times when you're
2232190	2233170	just waiting and you get
2233170	2233730	bored and don't know
2233730	2235350	where it's at, so we
2235350	2236170	can, you can always
2236170	2236930	clean this up so it
2236930	2237830	doesn't iterate every
2237830	2238130	single count.
2238130	2239550	But we've gone through
2239550	2240210	the 73 HTML files we've
2240210	2241390	down, we've processed
2241390	2241990	them as byte
2241990	2242670	strings, we've looked
2242670	2244490	for URLs, the comments,
2244490	2245330	and now we can actually
2245330	2246570	save off the different
2247490	2248070	sections.
2248070	2248670	So we have, we've
2248670	2249150	grabbed all the
2249150	2250030	comments, we've grabbed
2250030	2251050	all the titles, and we've
2251050	2252190	grabbed all the links.
2252410	2253470	So if I run this next
2253470	2254310	one, this is actually
2254310	2255450	going to show you the,
2255810	2256870	the listing of links.
2256870	2257730	And what I, what I like
2257730	2259050	about this is that it
2259050	2260190	actually shows you which
2261230	2262270	website that the links
2262270	2263190	came from as well,
2263190	2264090	which I think is really
2264090	2264870	valuable, especially
2264870	2265630	with sometimes when you
2265630	2266370	do crawls, you don't
2266370	2267150	always know, where did
2267150	2267830	I find this link?
2267830	2268450	Where did this come
2268450	2268750	from?
2268750	2269750	And in this case, it
2269750	2270610	does all of the mapping,
2270610	2271410	so if you start building
2271410	2272650	like a bigger mind map
2272650	2273470	or something of it, you
2273470	2274470	can, you can have some
2274470	2275390	value there.
2275450	2276430	And if we wanted to look
2276430	2277610	at something instead,
2277610	2278330	let's say we wanted to
2278330	2279750	look at the, the
2280750	2281870	comments across it, we
2281870	2283850	can do that, and then
2283850	2284650	we can just print it
2284650	2285250	out.
2285250	2286210	So this will just print
2286210	2287710	out the top, the first
2287710	2289090	10 comments we have.
2289510	2290190	Let's just see what
2290190	2291230	this looks like.
2292490	2293570	And then there's,
2293570	2294250	there's your list of
2294250	2295110	comments that are within
2295110	2295770	the HTML.
2295770	2296390	So as you get this,
2296390	2297150	you can start then
2297150	2298370	further looking through
2298370	2298730	there of, you know,
2298730	2299450	of, are there any
2299450	2300030	secrets, are there
2300030	2300950	passwords, you can do
2300950	2302230	regex, you can, you can
2302230	2302950	do different things in
2302950	2303690	there, so it's, it's
2303690	2304490	pretty neat.
2305670	2306850	I guess that's the,
2306850	2308170	that's the extent right
2308170	2308950	now I think we're going
2308950	2309410	to go through.
2309410	2310250	I started looking into
2310250	2310910	NetworkX.
2310910	2311510	I think there's some
2311510	2312530	promise there in terms of
2312530	2313910	getting a larger map at
2313910	2314770	scale, but I don't have
2314890	2315650	a lot of that code done
2315650	2317050	yet, but a lot I want
2317050	2317950	to build on, especially
2317950	2318670	with this functionality
2318670	2319370	because I think there's
2319370	2320950	so much value from even
2320950	2322030	an automated perspective
2322030	2323430	where now we can tie in
2323430	2324290	additional tools that
2324290	2324930	are really good at
2324930	2326090	HTML processing.
2326090	2327150	We can start to grab
2327150	2328230	JavaScript libraries,
2328230	2329310	codes, external references
2329310	2330530	and all that and do,
2330530	2331290	you can really start
2331290	2332290	building a pretty big
2332290	2333890	map of, of your target
2333890	2334870	surface.
2340710	2341950	Let's jump into another
2341950	2342450	demo.
2342450	2344270	This one is going to
2344270	2345330	search the autonomous
2345330	2346490	system numbers, which is
2346490	2347970	part of generally your
2347970	2349130	reconnaissance methodology
2349130	2350230	of where you want to
2350230	2351170	find out the different
2351170	2352390	site or network ranges
2352390	2353570	that are assigned to
2353570	2354810	different companies.
2354810	2355790	So what you typically do
2355790	2356870	is you would enter in
2356870	2358390	the company name or a
2358390	2359750	keyword for that company
2359750	2361110	and it would return the
2361110	2362710	different site or ranges
2362710	2363650	for that so then you can
2363650	2364590	further do your
2364590	2365630	reconnaissance doing
2365630	2366710	whether you do NMAP
2366710	2367670	scans or any type of
2367670	2368410	searches against those
2368410	2369650	IP ranges.
2369750	2370610	So what this does is
2370610	2371830	this uses the MaxMind
2371830	2372970	databases as well that
2372970	2373850	we talked about earlier
2373850	2374850	with the geolocation
2374850	2376390	data doing that, but
2376390	2377670	this uses their ASN
2377670	2378670	databases.
2379190	2380610	And so part of this is
2380610	2381550	we'll walk down
2381550	2382810	through it all, but
2382810	2383950	what it does is if
2383950	2384690	you're looking here at
2384690	2385410	this Jupyter Notebook
2385410	2387530	now, we go and these
2387530	2388630	first sections actually
2388630	2389730	just prepare everything.
2389730	2390630	So if you want to be
2390630	2391450	able to set this up in
2391450	2392430	an environment where it
2392430	2393530	does it completely
2395410	2396250	automated where it will
2396250	2396990	download.
2396990	2398290	We have the URLs here
2399090	2400550	and you can set this up
2400550	2401230	on your own just by
2401230	2402170	running this notebook so
2402170	2403010	that way you can continue
2403010	2404150	to run tools against
2404150	2405490	it and do and leverage
2405490	2406970	this there because I
2406970	2408410	really want to eliminate
2408410	2409450	all the manual steps
2409450	2409930	possible.
2409930	2410670	So the last thing I want
2410670	2411750	to do is have to log
2411750	2412650	into their website, get
2412650	2413770	the latest database and
2413770	2414250	do.
2414250	2415030	So this will walk you
2415030	2415830	through it.
2416170	2417690	One pretty neat thing
2417690	2418630	about this is that
2419370	2420550	recently they've changed
2420550	2421570	to and you have to have
2421690	2422590	a license key now.
2422590	2423610	It's still free, but you
2423610	2424530	have to log in, register
2424530	2425530	in their portal, get a
2425530	2427030	license secret to be able
2427030	2428070	to download those
2428070	2428470	databases.
2428470	2429790	Now, when I share this
2429790	2430730	code, I don't want my
2430730	2431870	license key in there.
2431870	2433070	That's another benefit
2433070	2434810	of using the cloud and
2434810	2436310	Amazon is that they have
2436310	2437930	a service called
2437930	2439110	Secrets Manager.
2439110	2440190	And what you can do is
2440190	2441030	actually you can call
2441030	2442270	Secrets Manager from
2442270	2443690	your Python code and
2443690	2444690	pull out secrets in
2444690	2444970	store.
2444970	2446190	So as you start to get
2446190	2448310	a big list of different
2448310	2449510	API keys and as you
2449510	2450530	talk about across
2450530	2450950	the cloud, even
2451530	2452470	previously we made a
2452470	2453430	call for the Google
2453430	2456030	Maps API, I pulled the
2456030	2456910	API key that I used
2456910	2458690	for that out of the
2458690	2460110	Secrets Manager within
2460110	2460930	AWS.
2460930	2463230	So this function here
2463230	2463990	is actually what does
2463990	2464190	it.
2464190	2465110	It's in the notebooks.
2465110	2466270	It's in my central
2466270	2467210	notebook because it's
2467350	2467570	reusable.
2467570	2468510	But you just pass in
2468510	2469370	the secret name, the
2469370	2470210	region name, it loads
2470210	2471570	it into here as a
2471570	2472070	variable so that way
2472070	2472890	you don't have to have
2472890	2473430	it in clear text.
2473550	2474730	And it's a security
2474730	2476090	best practice and it's
2476090	2476390	definitely safe if
2476390	2477270	you're putting your
2477270	2477830	code into GitHub and
2477830	2478070	things.
2478070	2479270	So I think as security
2479270	2480510	practitioners anyway,
2480510	2481290	that's just something
2481290	2482050	to get better at
2482050	2482830	because we push our
2482830	2483910	developers to do that.
2483910	2485370	We push a lot of
2485370	2486090	people that we talk
2486090	2486830	to to do that, but
2486830	2488050	it's hard and it
2488050	2488910	takes extra time.
2488910	2489630	So here's kind of
2489630	2490430	just an accelerator
2490430	2491250	that you can copy,
2491250	2492290	paste, use it in your
2492290	2493190	Python code, especially
2493190	2493770	if you're working
2493770	2495310	within a cloud
2495310	2496250	ecosystem.
2496650	2497750	An alternative to
2497750	2498870	that is it's called
2498870	2500510	Pickle and it's a
2500510	2501530	library within Python
2501530	2502190	2 where you can
2502190	2503670	actually save files
2503670	2503890	locally.
2503890	2504590	Now they're just
2504590	2505710	binary files that can
2505710	2506950	get loaded back into
2506950	2507490	Pickle so they're not
2507490	2508430	secured, encrypted,
2508430	2509490	anything like that.
2509490	2510290	But it can at least
2510290	2511150	call it from your
2511150	2512170	operating system rather
2512170	2514010	than pulling it or
2514010	2515050	storing it hard-coded
2515050	2517150	into an environmental
2517150	2518270	variable or within
2518270	2518670	your code.
2518670	2520010	So another option if
2520010	2520610	you don't want to use
2520610	2521630	Secrets Manager.
2522270	2523210	So as we go through
2523210	2524090	this, this all gets
2524090	2524570	set up.
2524570	2525150	I already have all
2525150	2526050	this, but you can see
2526050	2526990	if you put an
2526990	2527770	exclamation mark
2527770	2529030	before any commands,
2529030	2529690	you can actually run
2529690	2530730	Linux commands off of
2530730	2531930	this and SageMaker's
2531930	2533010	running in the back
2533010	2534130	end of a Linux server.
2534130	2535350	So you can do a lot
2535350	2536110	of that too, which is
2536110	2537110	pretty neat from a
2537110	2538270	flexibility standpoint.
2538270	2539110	So this manipulates
2539650	2541150	all your data, does a
2541150	2542670	WGET, pulls it all
2542670	2543570	down, goes through
2543570	2545090	that process, does a
2545090	2546250	cleanup.
2546250	2547870	I hate having files
2547870	2548670	just sitting on my
2548670	2549390	computer that I don't
2549390	2550110	need or want.
2550110	2550650	So whenever you
2550650	2551490	unzip this, this will
2551490	2552770	do all the cleanup for
2552770	2554330	you, loads it into a
2554330	2554930	directory, and then
2554930	2556670	now you have your
2556670	2556890	files.
2556890	2557610	So what we're going
2557610	2560450	to do is we're going
2560450	2561650	to actually start on
2561650	2562370	down here because
2562370	2566450	what's neat about this
2566450	2567090	is we can run all
2565110	2566110	We're going to
2567090	2568410	do this after we have
2568410	2569330	everything set up with
2569330	2570430	just, what is this,
2570430	2571210	like 10 lines of
2571210	2571590	code.
2571590	2572870	So it's actually pretty
2572870	2573770	straightforward to do
2573770	2574050	this.
2574050	2574710	And you can start to
2574710	2575590	automate it then by
2575590	2576590	passing in variables,
2576590	2577250	call this from other
2577250	2578390	functions from a
2578390	2579570	reconnaissance standpoint.
2579570	2581890	I'm going to restart,
2581890	2582790	reset our variables so
2582790	2584610	we don't have any
2584610	2585310	existing output.
2585310	2586370	So you'll see this
2586370	2587030	cleanup.
2587030	2587450	Okay, so now what
2587450	2588030	we're going to do is
2588030	2589250	we're going to leverage
2589250	2590430	Microsoft again as our
2590430	2591050	example, and we're
2591050	2591870	going to search their
2591870	2592490	org for their numbers.
2592490	2593710	So we're going to go
2593710	2594490	ahead and run this.
2595570	2596670	And what it's doing is
2596670	2597890	it's actually calling
2597890	2598850	out, pulling in the
2598850	2600010	max mine, searching for
2600010	2600790	it, and then it comes
2600790	2601290	back.
2601290	2602970	And it went through,
2602970	2604090	it looks like the
2604090	2605090	latest row that had
2605090	2605810	Microsoft in was
2605810	2606950	430,000.
2606950	2608210	So it went through a
2608210	2608910	considerable amount of
2608910	2610530	data for this, came
2610530	2611230	back, and then it
2611230	2612050	loaded it all into a
2612050	2612470	data frame.
2612470	2613550	So we have all of the
2613550	2616110	entries for Microsoft's
2616110	2617590	CIDR ranges into
2617590	2618130	there.
2618130	2618870	So now what we can
2618870	2620310	do is we can write
2620310	2622070	that to a CSV file,
2622070	2623330	or we can start to
2623330	2623790	parse and manipulate
2623790	2623970	and pull it into the
2623970	2625030	next lineup if maybe
2625030	2626130	you want to now kick
2626130	2627050	off NMAP scans
2627050	2627830	automatically, you
2627830	2628450	could leverage this
2628450	2629290	data and do, because
2629290	2630250	it's normalized.
2630910	2632410	So this actually wrote
2632410	2634150	it to my, back to my
2634150	2635610	recon page that's
2635610	2636990	that static S3 site.
2636990	2638010	So now I can have a
2638010	2639410	list of here's the
2639410	2640010	output of the
2640010	2640650	different CIDR ranges.
2640650	2641310	So I can just kind of
2641310	2642610	see what I'm doing on
2642610	2643710	an overview of it.
2643710	2644630	So it's kind of neat
2644630	2645530	too that collectively
2645530	2646230	you can kind of now
2646230	2647090	have this dashboard
2647090	2647850	that you can watch as
2647850	2648970	you do your own
2649830	2650750	reconnaissance and
2650750	2651610	bug bounty or anything
2651610	2652470	that you're doing.
2652470	2653150	And it's kind of fun
2653150	2654010	to see that.
2654570	2655470	So that's, that's
2655470	2656390	really that demo.
2656390	2657650	And then it actually,
2657650	2658610	we can look at the
2658610	2659070	org too.
2659070	2660970	So we see it's 8075.
2660970	2661490	So if we want to
2661490	2662750	search for that org
2662750	2663490	number in case the
2663490	2664170	name's different or
2664170	2665230	anything, we just swap
2665230	2665550	out.
2665550	2666650	And it actually ran
2666650	2667870	both, but I'm going to
2667870	2668590	comment it out so you
2668590	2669370	can just see the results
2669370	2669890	and we run that
2669890	2670790	against it and we
2670790	2672330	should have the output
2672330	2673210	of the org number
2673210	2674330	pretty quickly too.
2674330	2674990	So you can see the
2674990	2675970	orgs and it just
2675970	2676750	happens they all have
2676750	2677430	Microsoft in them
2677430	2678250	anyway, but that's
2678250	2678990	just another option
2678990	2680090	you can always do.
2688780	2689540	All right.
2689540	2690720	So let's take a look
2690720	2691760	at two more quick
2691760	2693060	demos and one of
2693060	2693680	these and you'll,
2693680	2694660	you'll pretty much
2694660	2695540	hear from me over
2695540	2696260	and over, especially
2696260	2696880	if you worked with
2696880	2698200	me of how much that
2698200	2699080	I don't like to do
2699080	2699860	manual effort in
2699860	2701040	spreadsheets or CSV
2701040	2702520	files and, and from
2702680	2703420	a metrics perspective,
2703420	2704440	how can I completely
2704440	2705080	automate it?
2705080	2705780	And the whole basis
2705780	2706680	of this talk is around
2706680	2707840	automating those manual
2707840	2708820	mundane things that
2708820	2709860	just take time away
2709860	2710880	that we should always
2710880	2711880	have almost in real
2711880	2712740	time or at least
2712740	2713880	near real time metrics
2713880	2714680	and data points that
2714680	2715700	we can have.
2715700	2717660	So I, I hate
2717660	2718460	using Excel
2718460	2719240	spreadsheets and
2719240	2720500	building graphs and
2720500	2721380	charts off of those.
2721380	2722100	Now I'm fine with
2722100	2723220	updating them and
2723220	2724100	keeping them as kind
2724100	2724560	of how do you
2724560	2725580	maintain different
2725580	2727000	components, but one
2727000	2727620	of the neat things
2727620	2728460	about pandas is
2728460	2729640	literally you can, you
2729640	2730700	can import Excel
2731420	2732180	spreadsheets with
2732180	2733540	literally one single
2733540	2734180	command.
2734540	2736820	And so when, when
2736820	2737640	you can do that, it's
2737640	2738640	so easy to do.
2738640	2740120	I figured, of course,
2740120	2741100	we have to be thinking
2741100	2741760	about security
2741760	2742520	awareness. Security
2742520	2743200	awareness should always
2743200	2743900	be in the back of our
2743900	2744400	mind. So when we're
2744400	2745160	working with coworkers
2745160	2747160	ourselves, I, so what
2747160	2747980	I did was I took the
2747980	2748820	step and I said, yeah,
2748820	2750060	absolutely. We need a
2750060	2751040	mask for that. So as
2751040	2751940	we're working in
2751940	2752640	quarantine, when we're
2752640	2753720	on our Zoom calls with
2753720	2755140	our peers, I created
2755140	2755760	these masks that you
2755760	2757100	can pick up, definitely
2757100	2758280	support, pay the, pay
2758280	2760040	the AWS bills for this,
2760040	2761160	but remind your,
2761160	2761980	remind your coworkers
2761980	2763260	on your Zoom calls
2763260	2764220	that they need to
2764220	2765180	modernize, they need
2765180	2766280	to codify and they
2766280	2767340	need to automate. So
2767340	2769100	I, I recommend pop
2769100	2770320	these on, show that,
2770320	2770860	rock these out as
2770860	2771860	you're, as you're in
2771860	2772980	your Zoom calls and
2772980	2773520	just remind people
2773520	2773700	that we need to
2773700	2774500	we need to move
2774500	2775340	faster. We need to be
2775340	2776020	more efficient. We
2776020	2776800	need to automate,
2776800	2778020	codify and move
2778020	2779620	forward. So as we go,
2779620	2780920	as we go into this,
2781300	2782200	let's, let's check
2782200	2783160	this out. So we're
2783160	2784060	going to, we're going
2784060	2784740	to do this one. This
2784740	2786060	is BSIM. So it's
2786060	2786500	called building
2786500	2787340	security and maturity
2787340	2789000	model. It's one of
2789000	2789700	the methodologies
2789700	2790420	around application
2790420	2791260	security. So if
2791260	2791800	you're building an
2791800	2792580	application security
2792580	2793920	program, trying to
2793920	2794520	measure how you're
2794520	2795660	doing, this is one of
2795660	2796340	the models that you
2796340	2797420	can leverage and do.
2797420	2798240	So all this does is
2798240	2799500	really it imports it.
2799500	2800260	And this is what I've
2800260	2801100	used to measure
2801100	2802220	programs against over
2802220	2802820	the years. And I
2802820	2804100	really like it. I'm a
2804100	2805220	big advocate, big fan
2805220	2806820	of BSIM. So what we
2806820	2807660	can do is we can
2807660	2808540	actually load this
2809320	2810000	spreadsheet within
2810440	2811140	just a moment. And
2811140	2811880	this is just a
2811880	2812600	spreadsheet too. So
2812600	2813220	even though we're
2813220	2814320	working, we've shown
2814320	2816320	massive data sets of
2816320	2817320	terabytes and
2817320	2817840	gigabytes of data
2817840	2818720	we've been processing.
2818720	2819980	You can use this for
2819980	2820820	just like single
2820820	2821660	megabyte spreadsheets
2821660	2822620	as well and still
2822620	2823420	automate and do. So
2823420	2824600	this loads it in. We
2824600	2825300	can see, and this is
2825300	2826420	kind of from a metric
2826420	2827560	scorecard dashboard
2827720	2828480	standpoint. And it's
2828480	2829340	also in GitHub. So we
2829340	2829760	can check this out
2829760	2830720	outside of this
2830720	2832060	presentation as well.
2832320	2833140	And then let's say
2833140	2833820	we want to keep
2833820	2834720	dashboards just for
2834720	2835400	our management, our
2835400	2836260	leadership of the
2836260	2837100	scores. You literally
2837100	2837940	can just run this
2837940	2839280	command as you update,
2839280	2840140	have your team members
2840140	2841100	update spreadsheets as
2841100	2842020	you complete a project
2842020	2843580	or a program. And
2843580	2844080	then you can have
2844080	2844960	your spider charts that
2844960	2845620	show kind of where
2845620	2846120	you're at in the
2846120	2847360	maturity loop, where
2847360	2848220	you want to be, where
2848220	2848920	you're at. And these
2848920	2849440	are kind of
2849440	2850280	interactive. You can
2850280	2851920	overlay these and
2851920	2853720	they measure based on
2853720	2854680	how BSIM actually
2854680	2855800	measures of how high
2855800	2856320	you are in the
2856320	2857680	quadrants. So just
2857680	2859540	one cool example
2859540	2860080	that's pretty
2860080	2860840	lightweight. Here's
2860840	2861980	your output chart of
2861980	2862580	you can see where
2862580	2863320	you're currently at,
2863320	2864520	where your target is,
2864520	2864940	and how you're
2864940	2865660	progressing. So you
2865660	2866400	can kind of build on
2866400	2867020	this and think about
2867020	2867460	all the different
2867460	2868260	metrics, spreadsheets
2868260	2869380	you have. That's
2869380	2870380	literally one line of
2870380	2871060	code that you can
2871060	2872180	pull in an Excel
2872180	2873360	spreadsheet into a
2873360	2874940	pandas data frame and
2875060	2875740	manipulate. And then
2875740	2876680	you never have to do
2876680	2877200	this again. I mean,
2877200	2877920	that took two
2877920	2878860	seconds to run the
2878860	2880500	latest, greatest
2880500	2881720	metrics around an
2881720	2882480	application security
2882480	2884160	program measurement.
2884160	2884400	Here's another
2884400	2886440	example just with a
2886440	2888600	CSV of the CVE
2888600	2889240	library. So it's a
2889240	2889880	data set where you
2889880	2891060	can go actually to
2891060	2892400	MITRE's website. You
2892400	2893920	can download the
2893920	2894620	historical in a
2894620	2896060	CSV format, all of
2896060	2897760	them. So I've
2897760	2898440	already kind of
2898440	2899260	taken the approach
2899260	2901980	of that. We can
2901980	2902660	download the file
2902660	2904780	and we can actually
2904780	2905560	just import it into
2905560	2906960	a data frame. So
2906960	2908920	these few lines of
2908920	2910100	code here will
2912000	2912980	walk through and
2912980	2913860	it loads up
2913860	2914380	176,000 different
2909400	2910400	code that you can
2914380	2917000	use. So if you ever
2917000	2917460	wanted to do
2917460	2918240	analytics around
2918240	2919360	those, trying to
2919360	2920200	look at group by
2920200	2921000	how many were
2921000	2922680	issued per year,
2922680	2923480	you can get this.
2923480	2923860	You can run it
2923860	2924600	pretty quickly. So
2924600	2925340	over the past 20
2925340	2926100	years, you can see
2926100	2926980	how the counts
2926980	2927500	have increased
2927500	2929820	over time. You
2929820	2931140	can look at which
2931140	2931960	months are busiest
2931960	2932960	with CVs. Why do
2932960	2933500	you feel like you're
2933500	2934220	always working over
2934220	2935400	the holidays? You
2935400	2936020	can kind of use
2936020	2936860	this to compare and
2936860	2937800	forecast where you're
2937800	2938340	going to most
2938340	2938980	likely have more
2938980	2939580	patch management
2939580	2940820	resourcing needed
2940820	2941740	in different things.
2941740	2942540	So kind of cool
2942540	2943120	insights that you
2943120	2943860	can share and
2943860	2944900	kind of make it
2944900	2945360	actually as
2945360	2946140	data-driven
2946780	2947600	background and
2947600	2948660	back you up on
2948660	2950240	why certain things
2950240	2950940	are happening. And
2950940	2952580	I think as we go
2952580	2953600	and build programs
2953600	2954700	and security, we
2954700	2955320	should always think
2955320	2956180	in terms of
2956180	2956660	data-driven
2956660	2959280	mindset. The
2959280	2959860	other neat thing
2959860	2961580	is you can load
2961580	2962400	additional libraries.
2962400	2962980	So this actually
2962980	2963980	loads Bokeh,
2963980	2965240	which is a
2965240	2965980	charting tool,
2965980	2967220	a graphing tool
2967220	2967960	within Python,
2967960	2968360	which is used
2968820	2969820	pretty commonly
2968360	2969260	across the scientific
2969260	2970660	computing community
2970660	2972020	as well. And
2972020	2972580	you can also
2972580	2973240	write these to
2973240	2974160	interactive websites
2974160	2974740	and things so you
2974740	2975280	can show the
2975280	2975920	dynamics. So you
2975920	2976600	can start to chart
2976600	2977200	these out, you can
2977200	2978020	graph, you can save
2978020	2978720	them, you can load
2978720	2979780	them, you can do.
2979780	2980460	So just a couple
2980460	2981220	lines of code,
2981220	2982360	repeatable, reusable,
2982360	2982940	and you can build
2982940	2983780	these that you can
2983780	2984700	just use over and
2984700	2985280	over again. So
2985280	2986680	tons of efficiency
2986680	2987460	gain, tons of
2987460	2989340	opportunities to do
2989340	2990320	so. I definitely
2991300	2992100	recommend grasping
2992100	2993820	that. Don't do
2993820	2994540	manual work in
2994540	2995180	spreadsheets anymore
2995180	2995820	other than maybe
2995820	2996800	just populating data
2996800	2997520	or loading it from
2997520	2998960	an API, but leverage
2998960	2999960	it in code and
2999960	3001380	automate your things.
3006730	3007810	As we wrap up
3007810	3009270	this presentation, I
3009270	3009970	forgot to mention
3009970	3010930	that as you build
3010930	3011870	this automation, you
3011870	3013150	have services running,
3013150	3014170	it's really easy with
3014170	3015350	the cloud to build
3015350	3016190	your SNS
3016190	3017190	notifications. So
3017190	3017850	these are simple
3017850	3019010	notification services
3019010	3020130	and you can text
3020130	3020810	yourself, give
3020810	3021770	yourself reminders,
3021770	3022510	let you know where
3022510	3023390	the progress is in
3023390	3024330	any of the processes
3024330	3025150	that you're running.
3025150	3026450	So definitely a
3026450	3027330	handy snippet of
3027330	3028210	code to really
3028210	3028470	leverage and
3028470	3029630	utilize to your
3029630	3031230	advantage. The other
3031230	3032070	part that I wanted to
3032070	3032750	mention is we've
3032750	3033610	looked a lot about
3033610	3034650	kind of red teaming
3034650	3035930	techniques and
3035930	3036560	passive reconnaissance
3037070	3038170	in doing this, but
3038170	3038790	what I want you to
3038790	3039530	think about is if
3039530	3040290	you're in an
3040290	3041170	information security
3041170	3042050	program right now
3042050	3043750	working, look at this
3043750	3044490	from a holistic
3044490	3045330	standpoint of how
3045330	3046210	can you help out the
3046210	3047230	other InfoSec domains
3047230	3048050	that you work side
3048050	3048970	by side along,
3048970	3050090	whether you're in
3050090	3051010	vulnerability risk
3051010	3051790	management, whether
3051790	3053510	you're in third
3053510	3054690	party, whether
3054690	3055950	you're in a
3055950	3057090	cyber fusion center,
3057090	3057690	you're doing security
3057690	3058430	engineering or
3058430	3059270	architecture. Think
3059270	3060190	about those outputs
3060190	3061870	and inputs and how
3061870	3062930	can you better take
3062930	3064170	the data that you're
3064170	3064930	generating within
3064930	3066190	your department and
3066190	3066690	how can you
3066690	3067570	normalize it and
3067570	3068730	pass it on to make
3068730	3070070	maybe the risk
3070070	3070990	management team help
3070990	3071710	them make better
3071710	3072930	decisions, have better
3072930	3073890	normalized data that
3073890	3074830	they can process and
3074830	3076030	do. So think about
3076030	3076930	those gaps and I
3076930	3077290	think that's
3077290	3077850	somewhere that you
3077850	3078730	can really accelerate
3078730	3080430	in your career and
3080430	3081170	it's a gap that we
3081170	3081970	have of where we're
3081970	3082670	really good at
3082670	3083570	building out and
3083570	3084830	ensuring individual
3084830	3085710	silos within
3085710	3087070	information security.
3087170	3087850	But if you can start
3087850	3088910	to show from a
3088910	3089870	career perspective how
3089870	3090990	you can branch and
3090990	3091570	understand kind of
3091570	3092550	what are the value
3092550	3092990	statements, what are
3092990	3093850	the objectives of
3093850	3094670	these different
3094670	3095250	departments within
3095250	3096570	your program, that's
3096570	3097110	where I think you
3097110	3098090	can really accelerate
3098630	3099710	in places that you
3099710	3100930	can really, I guess,
3100930	3102070	just build and
3102070	3103850	extensify your career.
3103850	3104610	So I encourage you
3104610	3105370	to take these
3105370	3106210	principles, these
3106210	3106970	platforms, these
3106970	3108190	topics that we've
3108190	3108850	talked about and
3108850	3109650	start applying them
3109650	3110590	to your daily jobs
3110590	3112370	and I think you'll
3112370	3113010	see some great
3113710	3114710	success. And again,
3113010	3113990	I really appreciate
3113990	3114750	the time today. I
3114750	3115650	hope that you've
3115650	3116490	taken something away
3116490	3117590	out of this. Definitely
3117590	3118650	jump into the GitHub.
3118650	3119310	I'm going to be
3119310	3120490	continuing to build
3120490	3122290	out blog posts and
3122290	3123390	code snippets and
3123390	3124510	accelerators in there.
3124510	3125990	So in the weeks and
3125990	3126830	months, definitely
3126830	3128510	push me, help drive
3128510	3129450	me to keep that
3129450	3130630	populated. But I
3130630	3131510	encourage you to
3131510	3132590	definitely reach out
3132590	3133070	if you have any
3133070	3134690	questions as you go.
3134690	3135650	But I appreciate
3135650	3136930	your time and enjoy
3136930	3137990	the rest of the
3137990	3139050	DEF CON and the
3139050	3140070	Red Team Village.
3140070	3141970	And thanks again
3141970	3142930	and have a great
3142930	3142990	day.
3142990	3143630	DR. MICHAEL
3143630	3143970	www.microsoft.com