start	end	text
0	5040	Alright, so the next speaker that we have is Gunner Andrews.
5040	10480	And Gunner is working in the application security space for almost five years now.
10520	14700	He is a hacker who is passionate about software and sharing knowledge.
14700	20980	And is currently working as a senior application and a cloud security engineer at Interwell Health.
21340	26220	And also, he's also a SynecRed team member. So please welcome Gunner Andrews.
26220	35100	Alright, hello?
35840	37740	Thanks for coming, first of all.
37740	42480	So this is my talk, called How I Built Recon to Scale with Serverless Architecture.
42680	46460	So we're going to go over a couple different kinds of serverless, some logistics,
46460	51460	and then there's a bunch of screenshots, workflows, code, all that kind of stuff.
51640	57900	The slides get shared and everything else, so if I'm going too fast, it's all available after.
57900	60840	I just wanted to leave it at that, just so people are interested.
61980	68000	Okay, so again, this is kind of what we're going to go over in order.
68380	73380	So obviously I have to start with the Who Am I slide, kind of like what you just heard.
73720	76000	I do AppSec and CloudSec full-time.
76200	81400	In my off-time, I like to develop stuff, I like to build stuff, like to play video games,
81400	84240	like to bug bounty hunt, and I'm into fitness as well.
86980	89640	So we're going to talk about two different kinds of serverless.
89780	92180	So there's a lot of the word serverless flying around.
92260	95020	One is kind of an idea, one is a framework.
95360	98060	So the first one we're going to talk about is the idea of serverless.
98300	101360	So there's a couple quotes up here from both Red Hat and CloudFlare,
101580	104980	a little bit about what serverless actually is.
105020	109560	So basically, if you read a little bit of those snippets, I'm sorry about the small text,
110200	114960	it basically allows you to build applications, manage applications, deploy, etc.,
114960	118440	without actually having to manage the infrastructure under it.
119040	124900	There are tools, especially in the bug bounty space, that are developed to help with this.
124900	129740	One is called Axiom, that allows you to basically take a workload and spread it over a bunch of servers,
130220	134580	and then it will basically ingest the data, bring it back, and give it all to you as one thing.
134580	138900	There's also a tool called ShadowClone that actually does get into the serverless space,
138900	143080	that basically allows you to do the same thing, but break up a workload over a bunch of Lambda functions.
143080	146200	Or, I think it uses Azure and GCP, too.
146540	151340	But basically, what serverless computing is, is if you take the...
151340	156620	there's a line that what you're responsible for, and then what, let's say, a cloud provider is responsible for.
156620	161700	If you spin up an EC2 instance in AWS, you are responsible for updating that server,
161700	164300	you're responsible for managing what ports are open on that server,
164300	167500	and there's a whole laundry list of things that you're responsible for.
167500	172940	AWS is responsible for the data center. If someone breaks into the data center, you are not responsible.
173000	175940	If someone breaks into the server, you are very responsible.
177160	180760	Serverless takes that line and moves it basically one more step up.
180820	184960	You are now only responsible for what you are deploying on the servers.
184960	189360	That's actually the code you're writing, the software you may be deploying, whatever it may be,
189360	192060	that's now the only thing you are responsible for.
192320	197440	AWS is actually managing the runtime environments for you, the servers in the backend for you,
197440	198840	scaling, everything.
199600	202640	Caveat that we'll talk about later, obviously you have to pay for that,
202640	205000	but we will talk about that a little bit more in a bit.
207950	211370	So there is a lot of these out there, there's a lot of different flavors,
211370	212930	and they do a lot of different functions.
213430	218230	A lot of what we're going to be seeing today is AWS, and that's some of the stuff you see there on the top.
218270	223010	So we're going to talk about AWS Lambda, AWS DynamoDB, Step Functions,
223010	225130	but GCP has this kind of thing too.
225130	229190	One example is Cloud Run, and Azure also has Azure functions,
229190	231910	I'm pretty sure DigitalOcean also has DigitalOcean functions.
231990	235170	There's a lot of it out there, serverless is kind of spreading to everybody,
235170	239850	but the ones that we're going to focus on today in some of the demos and the screenshots is all AWS,
239850	242110	but it is not cloud agnostic.
243010	248470	But this basically allows a user to either package up code in a zip file,
248470	251810	or use Docker files and Docker containers,
251810	255890	and basically take it to your image, or your zip file of code, and just deploy it.
255890	260130	You're basically telling the cloud this is Python, Go, whatever it may be, and it handles the rest.
260130	262870	All you're worried about is updating and managing the actual code base,
262870	264750	or the software you're deploying, and nothing else.
266830	269410	So this is the other serverless.
269490	271410	This is the serverless framework.
272350	274690	This is basically a CLI tool.
274790	276670	If anyone here has heard of Terraform,
276670	282290	it's pretty much a spinoff of Terraform for only serverless products.
282570	286410	It works with all the different clouds, all the main cloud providers,
286410	288470	and it works with their serverless offerings.
288470	291070	It comes with a whole bunch of plugins to do a whole bunch of things.
291070	295210	One of the plugins we'll see is it allows you to deploy a Python Lambda function
295210	299210	with a requirements.txt file, which has been annoying in the past,
299210	301310	if anyone has done Lambda development.
301310	305790	And it handles these laundry lists of languages that I have up there,
305790	307130	plus a lot more.
307530	309730	And the reason why we're going to talk about it is because
309730	313290	there's a lot of different ways to remotely deploy serverless.
313490	317290	AWS has something called SAM that you can use to deploy serverless.
318190	321630	It's not user-friendly 100% of the time.
322310	324830	This, in my argument, is.
324830	326690	And we're going to show a little video walkthrough of it,
326690	330450	but basically it breaks down to these four things.
330450	332450	It's an npm package that you install.
333030	336690	When you type just serverless, it pulls up a whole list of templates
336690	340090	that you can choose, and it will basically do what a lot of other tools do,
340090	343190	and it will make you a new directory and populate that directory with what you need.
343630	346830	For example, with Lambda, a Lambda handler, the YAML file,
346830	349570	you'll need to deploy it and configure it and everything else.
349870	352450	Whenever you're ready to deploy, you just type serverless deploy,
352450	354530	and obviously there's a bunch of flags if you want to configure it,
354530	356450	but serverless deploy will deploy it.
356450	360390	And then you can invoke it right from the command line, and we'll show that.
360530	364150	There's also ways that I'll show you when you invoke the function.
364150	368590	If you want to test your Lambda locally, you do not have to pay for AWS Cloud9.
368610	370530	You do not have to mess around with SAM.
370530	374210	All you do is, instead of serverless invoke, and you're calling your function,
374210	376330	you just add in local, and we'll show that.
376330	378530	But then you're testing your Lambda function locally,
378530	382510	and you're not actually paying to run a bunch of tests.
382730	385450	And you can hand it data, fake data, dummy data, and everything else,
385450	389210	and you can test all your serverless stuff locally before you actually deploy it.
389550	392110	And if you turn out you want to remove something,
392110	396350	serverless remove removes all the backend, as well as the actual Lambda.
396350	399450	So this runs on the CloudFormation stack,
399450	402230	and then uses, obviously, S3 as a backend for CloudFormation.
402270	404150	And that's actually how it will be deployed.
404150	408930	Very similar to SAM, but it puts it all in a nice little YAML format for you
408930	412670	that we're going to go over, and basically puts a CLI wrapper on it.
412670	416870	So I have a video here, and I'm going to probably speed it up,
416870	418370	and skip through some stuff.
419090	422050	But basically, like I said, when you type serverless in the beginning,
422050	424110	it will let you pick from a bunch of templates.
424110	426190	And there's a whole bunch there if you click Other.
426410	429030	But all I did here was pick an Amazon starter one,
429030	432210	which is obviously, of course, your cliche Hello World.
432730	437510	You're naming it, and it will then build you that new directory,
438190	439970	and deploy it out there, right?
441170	443810	You can use the dashboard. The dashboard is a paid thing.
443810	446050	You definitely don't have to use, right?
446390	448570	And we don't want to deploy it.
448570	452770	After we CD into the directory, I'll show you when you LS in here.
452770	458210	This right here is in your directory.
458210	460070	There's two things in the beginning one.
460070	463310	The handler is the basic handler that you would use,
463310	465890	whether using SAM or something else, to deploy something.
465910	468510	And then the serverless.yaml is the actual YAML file
468510	470410	you're using to configure the Lambda function.
470410	472730	So this is how much memory you want to have,
472730	474950	the timeout you want to have, and stuff like that.
474990	477430	But as you'll see here in a second...
477430	478650	Oh, geez.
480070	482530	I even tried to avoid live demos, too.
488040	489080	So let's go.
489820	492040	Okay, so back where we were.
493420	495360	The handler looks a little bit like this.
495360	497740	This is like a super basic handler, right?
498720	502040	You're defining a function called Hello, since the Hello World.
502040	504060	You'll see that show up in the YAML file.
504160	506420	That function is how you'll tell the YAML file,
506420	508300	like, this function is my handler function.
508300	510080	So it's in a file called handler.py,
510080	511440	and the function's called Hello.
511440	513980	So that's key here, because the next thing that's going to come up
513980	515740	is the YAML file.
515900	518260	The YAML file is very small,
518260	520240	because it's just the basic version.
520500	522120	And I'll show you some of mine later,
522120	523460	as we go into the thing.
526180	527400	There we go.
527500	529140	So this is the YAML file.
529140	530780	So you see in the bottom, you're calling functions,
530780	532060	you're naming a function.
532380	533780	The name is called Hello.
533780	535340	And then you're pointing it to the handler.
535340	537400	So you're pointing it to the file that it should go to,
537400	539240	and then dot whatever function.
539400	543800	You can have a bunch of functions in one directory.
543800	545740	You can make one directory per.
545800	546540	All that kind of stuff.
546540	548900	You don't have to put any secrets in here.
548900	550520	This makes it so you can actually package up
550520	552020	Lambda functions in directories,
552020	555000	and use Git or any other type of version control
555000	558380	to actually co-develop serverless architecture.
558380	559020	Right?
562160	562620	So...
562620	569620	In the hallway, and they stopped everybody.
570140	570840	Oh.
572500	573660	Don't fight.
577300	580380	So, this is the part I'm probably going to skip through,
580380	581940	because this is basically just like,
581940	584300	it runs for about 75 seconds.
584740	587000	But when you type serverless deploy,
587000	588380	like I said, it's using CloudFormation
588380	590100	and S3, just like SAM does,
590100	591400	in the back.
591400	593580	And that will basically,
593580	595700	this is now just deploying it to my AWS account.
595700	598200	So the only requirement to get this to work
598200	600100	is you download the npm package,
600100	602000	and you have your AWS credentials configured.
602000	604760	Just like, you know, if you're using the AWS CLI.
604760	605640	It's the same thing.
606440	607940	So really, at the end of the day,
608580	610660	what it is, is it's a CLI wrapper
610660	613240	and YAML wrapper around the AWS CLI,
613240	614760	and basically doing what SAM does.
614760	615840	It just makes it a lot easier
615840	618760	to locally develop, deploy, and update
619560	620660	what you're doing.
620660	622800	So, once it finishes here,
624300	626180	right here, we're going to invoke it.
626180	627500	You invoke it by the function,
627500	629500	which we called hello.
629540	630600	And when you invoke it,
630600	633280	this is actually running on my AWS account.
633280	634540	After it was deployed.
634880	636000	Like I said before, though,
636000	637700	you can do the same exact thing,
637700	639660	and just tell it to invoke locally.
641620	643220	And this is now using
643220	645580	whatever I have on my machine.
645600	646100	Right?
646820	649000	So if you want to test before you deploy,
649000	650320	you can invoke locally,
650320	651720	and then whenever you're ready to remove it,
651720	653900	the last step, since I don't necessarily
653900	656860	need to keep a hello thing on my account,
656860	658960	you serverless remove, and it removes everything.
660380	661620	It's very simple,
661620	663480	it's very easy to use,
663480	665180	especially compared to some of the other stuff,
665180	666640	like SAM, or just using
666640	668160	the AWS console.
668800	671220	So now, of course, we have to talk logistics.
671780	672660	What I mean by that is
672660	674600	we're going to talk about cost and scalability.
676360	677080	So,
677080	679200	I'm going to pre-wreck this by saying
679200	680900	if you want to use serverless
680900	682700	to build your own automation framework for
682700	685000	either red teaming, bug bounty hunting, or blue teaming,
685000	687120	or whatever it may be, and you want to scan
687120	689000	100 million domains every
689000	691680	five minutes, forever,
691680	693720	it's going to cost you money.
694120	694500	There's
694980	697660	no way around it not costing you money.
697740	699080	There is a free tier for a lot of
699080	701120	this stuff, and I will talk about some of the
701120	702960	ways to save money if you want to
702960	705600	maybe make it a little bit less user-friendly,
705600	707500	but a little more cost-saving. We're going to go over
707500	709220	that during the example, but
709640	711660	a little caveat is it does save you money.
711960	714340	But, a decent example,
714340	715560	and I'm sorry, again, I'm sorry for the
715560	717780	small text. I tried to make it as big as I could.
717900	718820	At the top
720460	721260	is doing
721260	723420	10 million DNS resolutions in
723420	725000	Lambda, in estimate.
725200	727560	So if you Google it, or chat GPT,
727560	729260	the average DNS resolution is about
729260	731400	33 milliseconds. So if you
731400	733280	run a little Lambda function that
733280	735240	resolves one domain, which will
735240	737540	happen in hopefully 33 milliseconds,
737540	739880	and you run it 10 million times,
739880	741680	you pay $1.80.
743080	745680	So if you resolve 10 million domains per month,
745680	747240	you paid $1, almost
747240	749320	$2. Right? If you
749320	751320	buy the smallest EC2 box you can
751320	753140	buy, at least on my account,
753140	755920	you pay $3.65 a month.
756660	757460	So,
757460	758800	if you're resolving
758800	761100	10 million domains, you're still cheaper
761100	763060	doing it with Lambda than
763060	764940	doing it in EC2.
765260	767160	Now, as data storage
767160	769120	comes into play, or continuous
769120	771180	scanning, right? If you're
771180	773200	going to try and resolve domains every minute
773200	774860	of every month,
774860	776980	that gets a little bit different. So
777660	779220	some of the advantages of
779220	779720	serverless
780940	782880	are as follows.
782880	784640	I tried to fit it all in.
784720	787320	The one thing to look at is whatever you're trying to build
787320	788940	will have a traffic pattern.
788940	790640	If you're doing continuous scanning
790640	792880	and it's always running and it's always
792880	794920	on all the time, don't use
794920	796820	serverless. This is probably
796820	797940	not for you.
798360	800940	Because you will spend more money using serverless
802760	804760	than you would using server-based. The only
804760	806780	advantage is you would be paying more money
806780	808700	to not have to worry about your infrastructure.
808700	810580	That may be worth it to you, it may not,
810580	812620	but this won't save you money.
812680	814740	If you have spiky workloads,
814740	816560	let's say you want to resolve 10 million domains
816560	818560	once a month,
819200	820540	serverless is for you.
820680	822740	Because you don't have to worry about spinning up and spinning
822740	824640	down servers and forgetting it's there and then paying
824640	826560	for the whole month. You don't have to worry about
826560	828500	where your servers are and how big or how small they are
828500	830000	and if they can handle the load or if you're going to
830470	832520	overload the server. All of
832520	834400	serverless has scalability
834400	836220	basically to unlimited.
836520	838820	Obviously there's benchmarks, but
838820	840220	just quote-unquote unlimited
840220	842560	for whatever workload you want to give it.
842760	844500	It will scale for you and you
844500	847200	only pay for what you use in serverless.
847320	848440	So Lambda for instance,
848440	850480	you pay for the computing resources times the
850480	852600	amount of time you're using them. That's it.
852600	854400	It doesn't matter if you do it one Lambda
854400	856800	at a time, it doesn't matter if you do it 100,000
856800	858480	Lambdas at a time, you're using the
858480	860620	same amount of computing resources, the math works
860620	862820	out the same whether you run it one at a
862820	866120	time or 100,000 at once.
866120	866720	So for
866720	869180	very spiky workloads, serverless
869180	871440	normally will perform better than servers
871440	872580	and it's way easier to manage
872580	873920	and it's way easier to scale.
874420	876660	The other thing like I talked about is the development
876660	878700	maintenance. I do this for fun
878700	880580	and I'm one person. I do not
880580	882780	have a dev team, I do not have an infrastructure team,
882780	884120	I don't have anybody.
885740	886500	This makes it
886500	888180	very easy because all I do
888180	890480	is write code and do what you just
890480	892320	saw, I just deploy it.
892460	893760	And then I don't worry about it.
894380	896560	And we're going to talk about how to schedule it and it'll even run
896560	898320	on its own and all that kind of stuff. You basically
898320	900540	code, it's basically like you're pushing to get
900540	902520	you code, push and you forget. Unless
902520	904540	it breaks. But we'll talk about that
904540	905060	too.
908520	909900	So now... okay good, I'm doing
909900	911860	good on time. So I'm going to go through
911860	913080	my stuff.
914680	915980	A little birdie told me opening
915980	918180	AWS console, a hacker conference, was probably
918180	919240	not a good idea.
919620	921380	So it's all screenshots.
922260	923760	I am not hiding anything.
923760	926200	If you have a question about something or you want to see something,
926200	927720	get in touch with me.
927720	930160	Ask me. I am more than willing to share it with you.
930160	931620	These slides will be available.
931820	934760	All the code is in the slides.
934760	936440	It's just kind of the best way of both worlds
936440	938120	where I'm not showing my AWS
938120	939820	console at a hacker conference
939820	941780	and I can show you guys
941780	943420	how I do things.
944020	945660	So this is
946120	947740	one of the workflows.
947800	949060	This is the one we're going to focus on.
949060	951340	This is a passive enumeration
951340	953880	that I use for bug bounty. There's passive
953880	956080	and active enumeration. This is the passive side
956080	958120	of things. So any bug bounty
958120	958920	people in here, any
959420	961980	all the recon folks in here have heard of
961980	963360	subfinder and amass.
963940	965980	This is basically bundling
965980	967820	those tools up and running them for me against
967820	969440	whatever I want it to run on.
970240	972080	This is called step functions.
972080	974020	In AWS, this basically allows you
974020	975900	to orchestrate both serverless
975900	977260	and non-serverless.
978040	980040	And this basically is a workflow.
980300	981800	So it comes in on the top
981800	984440	to a lambda function that's called passive enumeration.
984680	985920	And then in parallel
985920	987320	it will both delete
987860	990040	what I just processed from the queue, assuming
990040	991860	it's successful, and it will
991860	993960	loop through all my results. And if the domain is
994060	995720	a new domain, it stores it in my database.
995720	998420	Which is going to be DynamoDB, which we'll show in a second.
998600	1000000	After that, there's temporary
1000000	1001900	recon data, because step
1001900	1003960	functions can only handle so much data being passed
1003960	1005900	from step to step. And you use S3
1005900	1007960	just like basically every other AWS tool
1007960	1009820	to handle the large temporary
1009820	1011840	data you're handling in between. And then
1011840	1013860	it all just gets deleted at the end. Because it
1013860	1015400	ends up in the database.
1017620	1019640	So this is just a really quick...
1019640	1021820	Again, this is my configuration I use, but this
1021820	1023860	is just the configuration for a very basic SQS
1023860	1025900	queue. This is basically how everything
1025900	1028040	talks. So I use a PubSub
1028040	1029940	type model for my stuff.
1029940	1031940	And I use SQS queue as that basically
1031940	1033700	broker in the middle. So anything
1033700	1035780	that either will go from workflow
1035780	1038000	to workflow, or that goes from a
1038000	1039900	workflow getting kicked off into a workflow,
1039900	1042060	all happens through SQS queues.
1042060	1044120	SQS queues have a massive
1044120	1046140	free tier. I have never paid for it
1046140	1048140	yet. This is not one of
1048140	1050080	the services where you will pay.
1050260	1052060	If you break the free tier,
1052960	1053840	congratulations.
1054740	1056400	I think it's like 50 million
1057340	1058680	messages or something.
1058940	1060260	So this is how everything starts.
1060260	1062400	Again, I use this for bug bounties, mostly.
1063280	1064120	As well as a little bit
1064120	1065800	of work. So PlayStation
1065800	1068380	is pretty cool. They're giving one away, I think,
1068380	1070140	in the CTF. But PlayStation has
1070220	1072080	a bug bounty program. So I have
1072360	1074360	a separate table in DynamoDB,
1074360	1076360	which is a NoSQL serverless database
1076360	1078060	that scales with you, just like everything else
1078060	1079700	in serverless, on AWS.
1080040	1082220	And this is what every target I have
1082220	1084280	looks like. So I put in the URL
1084280	1086260	of where I would turn in bugs,
1086260	1088640	hopefully, the name of the program,
1088640	1090160	and then a list of domains, whatever their
1090160	1092000	root domains may be. I put any wildcard
1092000	1094100	domains in here that I'm going to do wide scope recon
1094100	1096040	on, and these will automatically get
1096040	1097860	ingested, as we'll see in a moment.
1098760	1099880	Then this program,
1099880	1101460	and this kind of thing, will be
1101900	1103760	cross-searchable across the other tables, which
1103760	1105320	I'll show in a second.
1107280	1107980	This is
1107980	1110100	an example of what it would look like after.
1110300	1112040	So, again, there are other steps
1112040	1113840	to this that I only have 45
1113840	1116180	minutes to go over. Again, if you're interested,
1116180	1117440	we can chat.
1118640	1119760	But it starts out
1119760	1121780	as it finds domains, it'll put them in there
1121780	1123780	with the subdomain, the program, and then
1123780	1125780	an empty list for CNAMEs and IPs that it
1125780	1128020	finds, and then there's more, obviously.
1128020	1129680	This is kind of truncated, but then whether or not
1129680	1131740	it resolves, there's timestamps, everything else.
1131740	1133260	You can add whatever you want here.
1133980	1135660	And this can be updated, and this is
1135660	1137720	one of the tables as a source of truth for
1137720	1139660	results. Now, with
1139660	1141700	this, you can do something in DynamoDB
1141700	1143780	called scanning. It's sort of
1143780	1145880	an intensive process, but it works
1145880	1147780	very well cross-table, and you can do a lot
1147780	1149620	of very cool queries with it.
1149620	1151780	So, in my opinion, I think it's worth it to look
1151780	1153740	into. So, for example,
1153740	1155520	here, if I wanted to go into
1155520	1157660	my domains table of all my
1157660	1159620	recon of any amount of domains, and I
1159620	1161560	just want to find the PlayStation one, it's very
1161560	1163500	easy to just scan my table
1163500	1165580	called domains, and the attribute
1165580	1167860	name is program, and I want it to be equal to PlayStation.
1167860	1169660	And I'll get every domain from all three
1169660	1171360	of those seed domains that
1171360	1173520	PlayStation is a part of. If I just wanted one of those
1173520	1175660	domains, I can set the attribute name to seed name
1175660	1177520	equal to whatever the one domain that I wanted
1177520	1179740	to find was. And you can scan...
1179740	1181480	those filters are, like I said, very
1182320	1183120	intense.
1183180	1185460	You can basically filter against anything you want.
1188400	1189800	So, this is the other thing.
1189800	1191700	So, a lot of bug bounty hunters, especially
1191700	1193620	when you're running automated recon,
1193620	1195260	you want to know when new stuff shows up, right?
1195260	1197840	You want to be the first one to it, all that kind of stuff.
1198500	1199620	DynamoDB makes that very
1199620	1201360	easy, because there's something called a DynamoDB
1201360	1203760	stream that streams...
1203760	1205560	it's basically an event, API,
1205560	1207620	of everything that's happening to your DynamoDB.
1207720	1209380	It's very easy to make a trigger
1209380	1211500	and hook on, just like you see in the thing,
1211500	1212640	to hook on a Lambda function
1213120	1215460	to that trigger. So, anytime a new thing happens to your
1215460	1217340	DynamoDB, there's a Lambda function to
1217340	1219440	ingest it, and you can tell it, only ingest
1219440	1221400	inserts, only ingest updates, whatever
1221400	1223460	it is. So, obviously, for a new domain,
1223460	1225200	it's pretty straightforward.
1225240	1226560	Anytime a new insert
1227360	1229880	goes into my domains table,
1229880	1231240	that means there's a new domain.
1231760	1233780	There's a Lambda function called NewDomainNotification
1233780	1235200	that we're going to show on the screen
1235700	1237580	that will basically ingest that stream's
1237580	1239500	API, grab the new domain that just got
1239500	1241780	added to the database, and shoot it to Slack.
1242540	1243980	So, I don't use...
1243980	1245420	there's not necessarily a tool for that.
1245420	1247500	It's just a POST request to a Slack
1247500	1249300	webhook that we're going to go over.
1250580	1251540	So, this is
1251540	1253160	actually some of the code.
1253240	1255660	Again, I tried to make it as big as I possibly
1255660	1257460	could, so it's split up into
1257460	1259360	chunks. So, this
1259360	1260820	is a Lambda function
1261580	1263220	that actually kicks off
1263220	1265560	running my recon against every
1265560	1268100	domain I have in that domains table I showed you.
1268100	1270100	So, all it is, is I go
1270100	1272300	through the table name that I call targets.
1272340	1273860	I just have one item
1273860	1276340	in the database. It's just one big item.
1277460	1278060	And for that
1278060	1280460	item, there's a bunch of programs.
1280500	1282000	So, I loop through the programs, and for
1282000	1284020	every program, as we saw, we'll have that
1284020	1286060	list of domains. And all
1286060	1288080	I'm doing is queuing those domains
1288080	1289420	up in my SQS queue
1290000	1291440	to the beginning of the workflow.
1292900	1294340	We'll look at this later.
1294340	1296040	This one gets scheduled.
1296040	1297920	So, I don't kick it off. I have it
1297920	1299980	scheduled for, I think, every 10 days, I think,
1299980	1301980	in the picture. So, every 10 days,
1301980	1304020	this gets kicked off. It takes everything
1304020	1305960	out of the programs
1305960	1308240	that I'm hacking on, takes those domains,
1308240	1310060	puts them into the SQS queue, and there's
1310060	1311840	another event that we're going to go over
1311840	1313980	that whenever something gets added to the SQS queue, it
1313980	1315960	automatically kicks off workflow. So, I'm not
1315960	1317400	touching it. I'm not
1318120	1320280	starting and stopping it. You can if you want.
1320800	1322160	But it is...
1322160	1324160	it can go, basically, fully automated.
1324160	1326260	This is the big lambda function that sat
1326260	1328280	at the top of the workflow. This is
1328280	1330040	the passive enumeration runner.
1330180	1332220	So, as you can see, there's a little bit more than when we
1332220	1334480	did the serverless demo. There's the handler.
1334480	1336160	It's the same thing. And there's the serverless
1336160	1338280	YAML. Requirements.txt.
1338280	1339740	Any Python people in here? That's your
1340440	1342180	requirements, right? Now,
1342180	1344320	subfinder and amass are actually just binaries.
1344580	1346060	Those still get packaged up in the
1346060	1348140	zip file and go with it. So, all this
1348140	1350140	is, it's nothing special, as much as I'd
1350140	1352200	like to pretend it is, is it's just Python
1352200	1354780	wrapping around tools that I want to use.
1354900	1356360	That I don't want to run myself
1356360	1357660	over and over.
1358680	1360480	And this is the passive part.
1360680	1362320	The active part looks very similar
1362320	1364240	to this. Again, I don't have enough slides and enough
1364240	1366300	time to go over everything, but the active part
1366300	1368280	would be very similar to this. And you can break
1368280	1369840	up the workload however you want.
1373360	1374400	So this, again,
1374400	1376420	I split it up into chunks. So this
1376420	1378680	is all, this is the whole YAML file
1379640	1380440	to deploy
1380440	1382540	the passive enumeration runner. It's only
1382540	1384460	this long. It's not, if you do this in
1384460	1386760	Terraform, anybody who's done Terraform,
1386760	1388480	it's a lot of lines.
1388480	1390340	It's a lot of lines.
1390580	1392340	And it's very readable
1392340	1394600	in my opinion. So we're telling how much
1394600	1396520	memory size we want, what we want the
1396520	1398500	timeout to be for the lambda function, and then
1398500	1400540	the same thing as before. I'm declaring a function,
1400540	1402480	it's called passive enum runner, I'm pointing
1402480	1404560	it to the handler, and then I guess in this one I add
1404640	1406500	a name and a description. When you want to use
1406620	1408680	a plugin, it's right there at the bottom.
1409220	1410900	So serverless Python requirements.
1410900	1412800	That's the plugin to bring Python
1412800	1414840	requirements with me into the lambda.
1418190	1420590	So this code is broken into chunks.
1420590	1422890	So this is basically my function.
1422890	1424770	From the serverless YAML that we just saw,
1424770	1427110	this is the handler that it's calling.
1427510	1428530	The data that it's
1428530	1430630	going to ingest is from event. That stuff
1430630	1432570	you see at the top is configuring the
1432570	1434470	lambda. I'm adding some config, as
1434470	1436570	people know who run the tools, there's some config files, API
1436570	1438050	keys, you know all the secrets off.
1438890	1440430	And basically I'm using Python
1440430	1442510	threading to run both SubFinder and Amass at the same
1442510	1444530	time. Because being that they're passive,
1444530	1446470	they're pretty low memory,
1446470	1448370	pretty easy to use. So it's
1448370	1450310	just running Amass and SubFinder against
1450310	1452230	whatever domain I give it. It's
1452230	1454410	pulling the results together, and it's saving them
1454410	1456370	to S3. Again, it's saving them to S3
1456370	1458590	because if I run it on Yahoo, and I pull
1458590	1460510	tens of thousands or hundreds of thousands or whatever
1460510	1462490	of domains, that step function
1462490	1464230	that's orchestrating my workflow,
1464230	1466190	as data passes in between it,
1466190	1468570	you can't pass over
1468570	1470630	I think it's like 5 megabytes
1470630	1473030	worth of data in between steps.
1473490	1474810	S3 is what you use
1475350	1476610	because all of the steps
1476610	1478730	will ingest directly from S3.
1478730	1480610	So you can output directly to S3 and at the next
1480610	1482470	step just say, yeah, grab this S3 file
1482470	1484610	from the step before. And that's how
1484610	1486390	you handle... and again,
1486390	1488550	that definitely won't put you over the free tier.
1488550	1490590	I mean, unless you are doing something very crazy, in which
1490590	1491510	case, talk to me.
1493970	1494690	This is...
1494690	1496170	I put this on here just in case
1496170	1497750	anyone's curious how
1497750	1499490	I actually wrap up
1499950	1502770	SubFinder in a mass in Python. It just looks like that.
1502810	1504210	This is the new version of a mass
1504210	1506290	too, so it changed a little bit.
1507450	1508390	But it's basically
1508390	1510250	like that. I'm using my config files.
1510250	1511750	I'm putting in the domains. It's all
1511750	1514170	very simple. It's just a wrapper.
1514350	1516050	Again, there's more than this. I just
1516050	1518170	don't have time. If this interests you,
1519010	1520210	we can talk.
1521430	1522250	So this is
1522250	1524050	the notification lambda. This is the handler
1524050	1526590	for the notification... sorry, the serverless.yaml
1526590	1528030	for the notification lambda.
1528030	1529910	Notice how it looks very similar.
1529990	1532090	Once you do this like twice, it gets
1532090	1533250	pretty quick.
1535270	1536110	And this is
1536110	1538130	the code, right? So all it
1538130	1540070	is, down at the bottom
1540070	1542230	is my handler function.
1542430	1544110	I'm starting a message with new domains
1544110	1546050	found from my
1546050	1547750	records in the event. The event
1547750	1550050	is basically the API data that came from
1550050	1552090	the API. I'm grabbing...
1552650	1554050	it's called keys.
1554770	1556310	And basically all that long string
1556310	1557990	there is doing that appends to the message is
1557990	1560410	just appending all the new domains that were inserted.
1560570	1562270	Once those are done, as long
1562270	1564190	as it counts greater than zero, which means assuming
1564190	1566090	there was an insert, I'm sending
1566250	1568130	a message to Slack. It's a POST request. It's just
1568250	1569550	a POST request to a webhook.
1569830	1574060	So this is what
1574060	1575800	it looks like when it works.
1576960	1578120	The reason why
1578120	1580040	I use step functions is because as you
1580040	1582160	build it, and as you use it,
1582160	1583940	it will track every run for you
1583940	1586300	and it will tell you what worked, what didn't
1586300	1588180	work, the input, the output,
1588180	1590080	how long things ran,
1590080	1592060	all that kind of stuff. So this is an example
1592060	1594020	of one that worked. This was ran against
1594020	1596060	Costco. So you can see the body there
1596660	1598060	was the program, and then
1598060	1600080	I do like three lines just to separate
1600080	1602180	and then the domain that
1602180	1603220	I was trying to call.
1604420	1606240	As you can see, everything is green. That means
1606240	1608260	it works. That's great. It doesn't always
1608260	1609120	work.
1609120	1610280	Right?
1611140	1613320	This is what it looks like. So this was
1613320	1615240	from a while ago when I was first making
1615240	1617380	it. As you saw, my serverless.yaml
1617380	1619160	set the timeout to 900 seconds.
1619220	1621200	That's because I had it at 600 and it wasn't
1621200	1623320	enough, if you can read. Right at the
1623320	1625220	bottom there, it says the task timed out after 600
1625220	1627200	seconds. It tells me the error. Sometimes
1627200	1629020	the error that it gives you right there
1629020	1630880	isn't the best. That's okay
1630880	1633220	because right under the passive enumeration
1633220	1635300	title, that little text that I apologize
1635300	1636840	for again, says log.
1637440	1638780	That link takes you
1638780	1640960	directly to the CloudWatch logs, directly
1640960	1642960	for that invocation of
1642960	1644760	the Lambda. Tells you exactly what
1644760	1646580	happened, exactly each step.
1646580	1648740	If that error
1648740	1651260	doesn't give you exactly what you're looking for,
1651260	1653120	there's a link directly to the CloudWatch
1653120	1654820	log of directly that invocation.
1654820	1656860	Which is why step functions really come
1656860	1658580	in handy as you grow.
1658920	1660880	Again, before I move on, this is
1660880	1662380	where you will spend the money.
1662560	1665000	Probably even more than Lambda. Step functions
1665000	1667500	are either standard or express.
1667680	1668960	Standard, you are
1668960	1671460	charged by state transition.
1671500	1673040	So when you saw that flow chart, every
1673040	1675200	time you go to a new square, you're charged.
1675200	1677140	There is a free tier, it's kind of small,
1677140	1681440	and then it's like .000025 cents per state transition.
1681440	1683080	But, if let's say
1683080	1684960	you have a state to store a domain, for
1684960	1686800	instance, and you're storing 100,000
1686800	1688960	domains, that could be 100,000
1688960	1691480	states. But,
1691480	1692980	the other form
1692980	1694480	of a state function
1695740	1697540	is called express.
1697540	1699240	That, you are not charged
1699240	1700920	by state transition.
1700920	1702620	You're charged similar to Lambda
1702620	1705180	based on how much compute resources you use
1705180	1707200	and for how long. You're not
1707200	1709860	charged by state transition.
1710320	1711100	The kicker
1711100	1713120	to that is express will only
1713120	1714140	run up to five minutes
1714780	1717020	for the whole, whatever you have in an express state
1717020	1719160	function. Standard will run
1719160	1720260	for a year.
1720260	1722060	So, if you have a very large
1722060	1724020	workflow, you kind of have to go standard
1724020	1726140	if it's long running. But, if I'm
1726140	1728320	using a workflow to store domains
1728320	1730280	100,000 times, it should
1730280	1732120	be express. I haven't broken
1732120	1734180	the express free tier yet, I have
1734180	1735700	broken the standard one.
1736320	1738220	Right? The good
1738220	1740060	thing about that is, is when you looked at my
1740060	1742200	workflow, you
1742200	1744320	can take the map
1744320	1746080	element, which is a loop element,
1746080	1748500	and inside of a looping element,
1748500	1750900	people assume you're going to loop a lot.
1751000	1752700	So, you're allowed to take a map element
1752700	1754600	in a standard step function
1754600	1756820	and make just that map element
1756820	1758560	express. So, you're
1758560	1760420	charged by an
1760420	1762500	express state function for that
1762500	1764960	loop. So, when I loop through all those domains,
1764960	1766480	even though the whole workflow is a standard
1766480	1768460	state function, that loop where I'm storing
1768460	1770440	all my domains is charged as express.
1770440	1772320	And that saves a lot of money.
1772360	1774560	You can plug and play state functions into
1774560	1776560	each other. One state function can call
1776560	1778540	another state function, so you can embed
1778540	1780720	them, and you can basically plug and play
1780720	1782460	with that as you will. I like
1782460	1784480	using it for this reason, but if you're trying
1784480	1786460	to save money, you can
1786460	1788700	ditch the state function and just have one lambda
1788700	1790580	trigger another lambda, trigger another, and do its
1790580	1792620	own thing. You don't have
1792620	1794400	to use state functions. You just get
1794400	1796480	much less visibility as you
1796480	1798480	scale. But, you don't have to
1798480	1799240	use it.
1801700	1802580	This is
1802580	1804960	now getting into how I trigger it.
1804960	1807020	All of this happens in event
1807020	1808980	bridge. There's an event bridge pipe
1808980	1811180	that I use in this demo,
1811180	1813040	and there's an event bridge schedule.
1813040	1815140	Schedule is cron, pipe is
1815140	1816080	basically event
1817520	1818920	happens, what you see at the bottom.
1818920	1820920	There's a source and a target. The source
1820920	1822940	in this pipe is my SQS
1822940	1824980	queue. Anytime a message gets added to
1824980	1826900	the SQS queue, it triggers the pipe
1826900	1829280	and the target is the step function.
1830000	1831500	It's very easy.
1831500	1833300	Anytime a message gets sent to the SQS
1833300	1835500	queue, it will either sit in the queue
1835500	1837380	if I have the max amount of
1837380	1839680	concurrent step functions I have set running,
1839680	1841320	and then it will trigger it when it's
1841320	1843820	ready. Or, if there's none running,
1843820	1845280	it will automatically
1845280	1847340	kick off a step function, which will ingest the message
1847340	1849460	from the SQS queue and kick off.
1850000	1851360	The other one that I have
1851360	1853340	in event bridge, again, is called a schedule,
1853340	1855820	which is basically cron in AWS.
1856280	1857340	So, I have a schedule
1857340	1859400	at a rate of 10 days, and you
1859400	1861240	target it at a lambda function.
1861240	1863240	Like I said, that target one right next to
1863240	1865500	the schedule is where you'll pick your lambda.
1866000	1867320	And I pick... so every 10
1867320	1869220	days, I have a target of that full
1869220	1871120	run lambda that we talked about that
1871120	1873160	just goes through my targets table and
1873160	1875240	posts them all to the queue, which will then trigger
1875240	1877160	the pipe on the last slide and
1877160	1879120	ingest it all into step functions. And that happens
1879120	1881260	every 10 days automatically without touching
1881260	1883180	it. You can change that, you can delete it, you can do
1883180	1885040	whatever you want. You can obviously manually
1885040	1887240	kick off and invoke lambdas if you want to
1887240	1889220	and you don't feel comfortable doing it automatic.
1889220	1891280	That is basically what
1891280	1893320	it is. I will say
1893320	1895220	since I'm pretty sure...
1895220	1896840	So I will say before the
1896840	1898920	thank you and the questions, I will say
1898920	1900520	I have personally
1901300	1903160	monitored a couple hundred thousand
1903160	1904880	domains, because I don't do mass
1904880	1907180	recon. I like to do recon on the targets that I'm
1907180	1908760	hacking on at that time.
1909120	1911140	I have monitored fairly
1911140	1913400	continuously a couple hundred thousand domains
1913400	1915380	and kept my price
1915380	1917120	pretty firmly under $50
1918020	1918740	a month.
1918740	1921220	So it's not crazy expensive, it does
1921220	1923260	cost money, and again, it's
1924040	1925240	not... if you scale
1925240	1927260	it, it will scale with you and it
1927260	1929160	will work very good, but
1929160	1930540	you still pay for it.
1931360	1933240	So yeah, that's all I have.
1933240	1934640	It looks like I do have
1934640	1937200	10 or 15 minutes here for questions, because I
1937200	1938800	do have to jet after, but
1938800	1941080	if there's any questions, I would love
1941080	1942680	to answer them.
1942720	1943360	Yeah.
1944860	1946420	Thank you.
1964540	1966620	Yes and no. No, because
1966620	1968520	the Lambda has ephemeral memory that will
1968520	1970780	grow with you. Yes, because there is a
1970780	1972780	limit to that, and you pay
1972780	1974800	for that also, and
1974800	1977000	there is a limit. It's rather
1977000	1980200	large, but there is a limit. Yeah.
1980200	1988510	Sure.
1988510	1989890	Yes, so
1989890	1991870	serverless will set up
1991870	1993290	the IAM roles that go with
1993290	1994570	the Lambdas automatically,
1994570	1996670	and you can configure those either in the YAML
1996670	1998670	or in AWS, but they do get spun
1998670	2000110	up automatically. So if your
2000110	2002590	handler imports
2002590	2004810	the Bottles3 library
2004810	2006250	and uses DynamoDB,
2006250	2008630	it will spin up an IAM role for
2008630	2009930	you that gives you
2010490	2012870	access to DynamoDB to that Lambda.
2016470	2017490	Yes.
2018090	2018950	Yep.
2019650	2020670	Yeah.
2033780	2035480	So when I...
2035480	2037400	And again, the reason why I didn't touch the active one
2037400	2039140	is because it gets a little more interesting.
2039860	2041480	But the active stuff, the way I
2041480	2043480	approach it, is I split it
2043480	2045680	into super teeny chunks that run lightning fast.
2045720	2047500	So that's why I talked about domain resolution
2047500	2049560	because I resolve domains one at a time
2049560	2051240	and they resolve in, like, milliseconds.
2051480	2053440	Because at the end of the day, it doesn't matter.
2053440	2055560	It's all runtime versus memory. So it doesn't
2055560	2057460	matter if I resolve a chunk of 10,000 domains
2057460	2059400	at the same time or one at a time. It's all
2059400	2061440	the same runtime. So
2061440	2063400	the only thing that would really matter, that
2063400	2065480	really matters, is hitting your database
2066160	2067460	a bunch. So you can
2067460	2069580	resolve a bunch and have them all catch
2069580	2071480	at a step in the step function, and then
2071480	2073520	as batches happen, then you can batch
2073520	2075460	update databases. But for the active
2075460	2077460	stuff, I break it into super teeny
2077460	2079440	chunks, so I never really have to worry
2079440	2080420	about a time limit.
2082340	2082920	Yeah.
2094470	2095790	You can do either one.
2096850	2097690	Yep. Yep.
2097690	2099170	You can configure it to do either one.
2100170	2100850	Yeah.
2105250	2105430	...
2106290	2106730	...
2113570	2114010	...
2115490	2115930	...
2115930	2116050	...
2116670	2119270	Absolutely. So serverless, the
2119270	2121770	framework, supports Azure as well.
2121770	2123310	And I think it's called Azure Functions.
2123310	2125430	Again, I'm kind of hiding AWS.
2125690	2127430	But Azure has Azure
2127430	2129410	Functions, and it is, I'm sure
2129410	2131270	it is very similar. It will handle it
2131270	2133150	all for you and deploy it.
2133150	2133530	...
2133530	2135230	Mm-hmm.
2139560	2140920	So I personally
2140920	2143220	talked to AWS before I gave the talk.
2143220	2144780	They said as long as I'm not doing anything
2144780	2146860	naughty, it's allowed. Right?
2147080	2148900	Because it's just public
2148900	2150900	domain. But I did have someone
2150900	2152940	else from AWS tell me to
2152940	2154940	watch out. So it is. And I have heard
2154940	2156820	of people getting banned off the platform. So I've got
2156820	2158580	I was going to say, I've gotten pretty mixed reviews
2159260	2160900	from it. I haven't gotten personally
2160900	2162900	in trouble yet. As well as when
2162900	2165260	Lambda spin up, they run on different IPs.
2165260	2167580	So I guess there's that benefit.
2167580	2169400	I don't really use it for that benefit. And again,
2169400	2170280	I don't
2171740	2173180	use this type of recon
2173180	2175200	to continually berate
2175200	2176620	sites. Right? I
2177260	2178820	do pretty relaxed
2179280	2180540	scanning, if at all.
2181060	2183180	And I don't try and update data
2183180	2185300	every minute on the minute. That kind of stuff.
2185300	2187220	So I feel pretty safe
2187220	2189260	about that. But I mean, if you were using it like that...
2191140	2191760	Yeah.
2191760	2193220	You might get in trouble.
2193220	2194060	Yeah.
2195720	2196140	Yeah.
2197160	2198040	Absolutely.
2200260	2201140	Yeah.
2205220	2206100	Yep.
2209160	2211260	You can. It's all the same thing.
2211280	2212740	I use Lambda because
2212740	2215140	again, when I do active scanning,
2215140	2217760	I just break it up into super tiny chunks.
2218400	2219100	Fargate,
2219100	2220840	literally, you can use the same thing.
2220840	2223220	You would just do the same thing with step functions, but for the longer...
2223220	2225020	If you want longer running
2225020	2226940	jobs, you'd have to use a standard step function,
2226940	2228940	so it can run for up to a year. And then you would just
2228940	2230500	plug Fargate in, and it would be the same thing.
2231320	2232920	Personal preference. If you want to do the exact
2232920	2234900	same thing, but with Fargate,
2234900	2236640	it will work almost
2236640	2237440	identically.
2239440	2239980	Yeah.
2243920	2244880	Good question.
2256480	2257440	There you go.
2258660	2259460	Yeah.
2268080	2268480	Yes
2268480	2269800	and no.
2270760	2272600	I am, like I said, I'm willing to
2272600	2275040	share, and I may,
2275040	2276480	if there's a lot of ask for it, I may
2276480	2278800	just make it easier to put some stuff in a
2278800	2283180	GitHub repo and share it. I'm not against that.
2283180	2284560	Part of it,
2284560	2288220	I won't, just because it's a lot.
2288220	2288760	But again,
2288760	2290620	if you have a question, get in touch
2290620	2292080	with me over Twitter or something like that, and I can.
2292080	2294000	And if there's a lot of interest, I probably will.
2294320	2296580	I guess I didn't have plans for it, but it's not necessarily
2296580	2298400	because I'm
2298820	2300640	secretive or anything. I guess if there was interest,
2300640	2302460	I wouldn't be against it, but I didn't
2302460	2304220	originally have plans to, yeah.
2306000	2306460	Cool.
2306460	2307160	Thanks, guys.