start	end	text
0	2720	your support for supporting the DEF CON community
2720	5000	and the Red Team Village specifically.
5000	6940	We're looking forward to your presentation.
6940	8060	So take it away.
8260	12200	Thank you so much, Omer, for introducing and having me here.
12300	14800	Let me just share the screen and confirm.
15860	17420	Can you see the screen?
20450	21810	Yes, you're all set.
21810	23050	Awesome, perfect.
23190	25630	Thank you everyone for joining me.
25630	29050	And first of all, to just give some context,
29530	32790	thank you so much, DEF CON Red Team Village,
32790	35290	especially in this pandemic times to make it virtual
35290	37770	and people can enjoy these talks
37770	39990	without any physical intervention.
40210	44190	And this talk is about the Kubernetes code.
44190	46650	So disclaimer, basically,
46650	49430	today's we will not see any slides, basically.
49490	51690	And it's mostly, everything is in online.
51690	53030	You can play along with me,
53030	55110	or if you can go back and try yourself,
55110	56850	you can go ahead and try as well.
56850	58430	So there will be no slides,
58430	61510	mostly live demos and the hands-on exercises.
62050	62790	Cool.
62790	65030	So this talk is primarily focusing
65030	67630	on introducing a project recently I have created
67630	69570	called Kubernetes Code,
69570	74150	which is, this is in github.madhuvakula.kubernetescode.
74150	77650	This project was mainly intentionally vulnerable application,
77650	81530	which is to basically try and teach participants,
81530	83410	or like anyone come to the new Kubernetes
83410	86710	to understand how you can assess these Kubernetes clusters
86710	89290	and find vulnerabilities so that you can go back
89290	91430	and try it out in your organizations,
91430	92430	or day-to-day pen tests,
92430	94550	which you find are a team exercises, right?
94550	97210	So I have presented at some of the places
97210	99210	and also I have created a playground.
99210	100310	If you wanted to write now,
100310	103290	try it in your browser and follow along, right?
103670	106770	So a little bit about me, myself, Madhu Akula.
106770	108150	I work at currently Zibia
108150	110470	as a Cloud Native Security Specialist.
110470	112830	And I have been speaking and training
112830	114850	at a bunch of conferences around the world,
114850	117890	like DEF CON Black Hat, USENIX, Orally, and some of them.
118150	120650	And I'm fortunate to find some of the vulnerabilities
120650	122670	long back using my research
123330	125970	in some of the organizations like Microsoft,
125970	128070	LinkedIn, Google, and some of them.
128270	130270	So apart from that, my primary interest
130270	133430	mostly working around security of cloud,
133430	134610	containers, Kubernetes,
134610	136810	and automating the infrastructure part.
136910	139450	So that's a little bit about me.
139450	140650	If you wanted to follow me,
140650	142590	you can follow me at Madhu Akula
142590	144710	in most of the social media stuff.
145610	146510	Cool.
146510	149550	Let's get into what exactly is this Kubernetes code
149550	151650	and what we are trying to learn
151650	153930	like within this next one hour, right?
154010	158490	So to give a little context about Kubernetes code.
158530	160550	So as I said, it is intentionally vulnerable.
160550	162690	I think as most of you have come across
162690	164150	the different types of code,
164150	167110	like web code, or Terra code, or some other codes.
167110	168810	So this is a similar project,
168810	170970	but it is mostly focusing on Kubernetes
171510	174270	to understand what are the known vulnerabilities.
174270	176370	And also we have seen common vulnerabilities
176370	178030	in the real world environments,
178030	180410	like while performing penetration testing,
180410	182150	or like while performing assessments
182150	183990	in Kubernetes environments.
184010	186710	From those we have taken as scenarios.
186710	188690	And these are some of the scenarios.
188690	190410	Let me zoom a bit.
190410	192730	I think maybe easier to see.
192730	193330	Yeah.
193330	196170	So these are some of the scenarios I have created.
196170	199010	And we will see some of the scenarios,
199010	201250	exploitation, and how you can bypass
201250	203230	or gain access to Kubernetes clusters
203970	206110	as an attacker point, right?
206110	208950	And I'm really sorry, and I'm shameful.
209070	211450	I said I will keep updating these slides
211450	213490	at the documentation.
213530	216570	And also I have been working on the defense part also.
216570	219030	Hopefully this month we will see them coming away.
219530	221110	This part I'm going to skip.
222130	224630	So before diving into Kubernetes,
224630	226470	if you are really completely new
226470	229350	and coming to the containers or Kubernetes world,
229350	231570	I highly recommend understanding the technology
231570	233650	because there is a fair amount of gap
233650	235550	in between penetration testing
235550	237570	or finding or assessing clusters
238410	241690	if you don't understand what technology you are doing, right?
241690	243650	So we have seen commonly,
243650	246390	especially in the modern tech infrastructure world,
246390	248510	there is a huge gap for the security team
248510	250070	to understand the technology
250070	253130	before even testing or trying out.
253130	255070	Because once you are able to understand,
255070	256430	you can be able to do your best
256430	258010	or you can be able to possibly find
258010	261330	most of the Kubernetes or any ecosystem.
261550	262890	This is a very funny video
262890	265750	and it's a simple children's guide
265750	268230	to explain what exactly Kubernetes is.
268230	269290	So I would highly recommend
269290	271950	if you want to go through these things.
271950	274570	So to give you a little quick context,
274570	276610	how Kubernetes architecture looks like
276610	278050	at a higher level, right?
278050	280850	So this is again from the wiki comments.
280850	283850	You see a server and client architecture
283850	286890	similar to most of the server client.
287090	290010	So one thing you have to observe here is in Kubernetes,
290010	292050	everything which you make, any operation,
292050	294230	it may be from within the master,
294230	295410	which means server,
295410	297710	within the node, which means worker node.
297710	300110	So whatever the operations which will make,
300110	301390	like it can be from the node,
301390	302410	it can be from the developer
302410	304710	or any user, any service
304710	308390	will go and touch something called API server.
308390	309870	So which is one of the key component
309870	311610	in terms of Kubernetes,
311610	313670	because Kubernetes is completely driven
313670	315670	by API first approach.
315670	318290	So whatever you make all requests or something,
318290	321550	everything you will see going through the API server, right?
321550	324890	And control manager is basically to make the decisions,
324890	326610	whatever the requests you make.
326610	330230	And scheduler is ensuring to place the workloads
330230	333090	or the request which you made.
333090	335330	Suppose if you wanted to serve a Nginx server
335330	337230	with 10 pods, it will ensure that
337230	340570	how many nodes has the capacity to schedule.
340570	342530	So the scheduler job is to figure out that
342530	344190	and place them accordingly.
344310	347090	So etcd is one of the key value store database,
347090	348750	which is used by Kubernetes,
348750	350790	which is as its name suggests, key value.
350790	352270	So it stores the data,
352270	354630	which is coming to the API server
354630	356910	and stores the key value base.
356910	360870	And we will see a little bit in a later coming sections.
361650	363350	So apart from that nodes,
363350	364210	when you come to the node,
364210	367390	kubelet is the talking point to the API server
367390	370730	and communicates within the local node itself.
370730	374450	So whenever you wanted to make a request as a scheduler,
374450	376110	you wanted to schedule in the node,
376110	378370	it will talk to the container runtime,
378370	379510	which is within the node,
379510	382650	like it can be Docker, RunC, etcd.
382650	385810	So then it will go ahead and schedule
385810	388430	based on the workload kind of thing.
388610	392030	So kubeproxy is one of the another key component,
392030	394110	which is kind of like IP tables,
394110	396010	like it will try to provide a communication
396630	399230	across like the nodes as well as a cluster.
399810	403730	So let's go ahead and a little bit go deep,
403730	407010	but I would highly recommend you practice the Kubernetes
407010	409990	before you diving into the assessments
409990	411370	or security testing.
411370	414390	So here are some of the references I have curated.
414390	415830	These are pretty hands-on,
415830	417250	like if you wanted to try,
417250	419970	so you can go ahead and try as a literally playground
419970	423670	and also specific objects if you start while testing,
423670	426190	you can click and understand what exactly it is.
426190	427770	So these references I would,
428370	430670	I think it would be definitely helpful to try.
431130	432090	Cool.
432490	434370	Setting up Kubernetes code.
434610	436870	The Kubernetes code is in design in such a way
436870	438690	it can set up any out of the box.
438690	440810	It can be in your existing cloud provider services
440810	443990	like GKE, EKS, AKS, or digitalization based.
444350	446050	And also your locally provisioned
446050	447750	Minikube or another playground.
447750	451550	To make easy of setup for the people to try out and learn,
451550	453990	I have created something called Katakoda Playground.
453990	455490	If you just go ahead and click this,
455490	457510	it will open up a playground for you,
457510	459950	which is completely free and online.
459950	461790	So you don't need anything to be done.
461790	463450	I think you have to register first time.
463450	466470	So then you can get the complete setup here for you
466950	468290	to practice and learn.
468290	470130	Let me put it in a presentation mode
470130	471590	so that you can see a little better.
471590	472770	Yep. Cool.
473650	474270	Cool.
474330	477950	So I'll walk you through how you can set this up
477950	478790	and practice.
478790	481350	And if you wanted to follow along with me,
481350	483670	you can follow this guide at madhuvakula.com
483670	485470	slash Kubernetes code.
485470	487450	This is the URL, right?
487530	487950	Cool.
487950	489690	And if you wanted to set up your own,
489690	494310	you can just follow this, the documentation.
494310	496830	Basically you need to have a working Kubernetes cluster
496830	498630	set up with admin access.
498630	501570	And also you need a Helm version too,
501570	502890	because in one of the scenario,
502890	505070	we will try to focus on how you can exploit
505070	507990	the existing vulnerability in the older version of Helm.
507990	509370	So that's it.
509370	511390	So if you wanted to set up your own,
511390	513250	let's go ahead and do that as well.
513430	515550	So if you have your own set cluster,
515550	516810	you clone the repository,
516810	520630	which is the git clone of the Kubernetes code.
520710	526050	And then you can go ahead and set up the Kubernetes code.
526390	530650	So it checks if Kubernetes kubectl is available,
530650	532950	Helm 2 is available in your path,
532950	535310	and it'll go ahead and deploy the tiller,
535310	537810	which is the server side component of the Helm,
537810	539910	and a bunch of vulnerable services,
539910	543590	which will be used as practice to exploit
543590	546470	or try practicing Kubernetes security.
547550	551670	So if you are someone following the Katakoda live,
551670	553230	you can use this playground,
553230	555230	which is completely free and online.
555230	556790	You can go ahead and do the same thing
556790	559010	in the playground as well.
559790	562030	So the playground is designed in such a way,
562030	564010	you can go ahead and see here only the guide
564010	566510	in the second, in the specific tab.
566510	569010	And also you can browse those ports and other stuff
569010	571270	within the browser itself.
571270	573490	Even in your mobile, also you can try this.
575050	577730	So it takes a while to set up
577730	581910	the entire Kubernetes code platform.
581910	583630	I think it's almost done.
583630	586970	So it creates a bunch of vulnerable scenarios,
586970	589610	which I have pre-created.
589650	591070	You can go through the code
591070	593550	and you can able to see what exactly it is trying to do,
593550	595010	if you wanted to look at it.
595010	597350	So this is nothing but it checks
597350	600190	if kubectl is existing or Helm 2 existing
600190	603010	and it sets a bunch of emails,
603010	605910	like which is nothing but deployments in the cluster.
607550	608030	Cool.
608030	610410	Now we have the cluster up and running
610410	612170	and we have Kubernetes code set up.
612170	613730	So what's next, right?
613730	616470	So I think it looks like here also it is done.
616550	618970	Ensure you check all the ports are running.
618970	621990	You can go ahead and run kubectl get ports.
621990	624230	It is currently container creating,
624230	625730	which means it's starting.
625810	627950	It takes a while to set up the,
627950	631530	depends on the way you are running the Kubernetes cluster.
631650	634770	So let me go ahead and do this as well here,
634770	637230	kubectl get ports.
637390	639870	I hope you people can see the screen.
639870	642630	I think, let me zoom a little bit as well.
642630	643150	Cool.
643150	644930	So you can see a bunch of them.
644930	646790	Most of the ports are created.
646790	650130	I think just only one port is waiting to get created.
650390	651790	That's awesome.
651830	653370	Now you can see all ports are running
653370	655770	and I think this is intentional.
656150	658150	This is a job and get company.
658350	659990	So if you see the status,
659990	663450	basically your entire Kubernetes code is set up and running.
663450	665970	So now it's time to hack and get into the cluster.
665970	666450	Bye.
667490	668270	Cool.
668270	670890	That's where we go back and refer to the documentation.
670990	673050	So if you look at the scenarios,
673050	675810	so there are a bunch of scenarios we have created,
675810	676770	as I said.
676770	678490	So for the time being,
678490	680630	for the entire today's presentation,
680630	683430	we will not be able to look at all of those scenarios,
683430	685130	but definitely we will be looking
685130	686870	at some of the primary scenarios,
686870	689470	which might be interesting and fun to try out.
689490	692270	And I would highly recommend go back and try out yourself
692270	695610	and check out more other scenarios as well, right?
695610	700610	And in some cases, especially not all of them scenarios,
700610	701710	but in some scenarios,
701710	704030	I try to create a flag in this format,
704030	705910	k8sgoot-goot.
705910	711890	And some MD5s, basically it's a number of 32 characters.
712390	715290	Then if you reach this kind of flag,
715290	718050	basically it means end of the scenarios,
718050	720510	but it may not be found in for all the scenarios,
720510	722550	but I try to keep for certain scenarios
722550	727590	to reduce the effort of like going on beyond post-exploitation.
729010	729870	Cool.
729870	733030	So let's go ahead and start with the scenario one,
733030	735050	sensitive keys in code base.
735050	737390	So I would recommend before doing that,
737390	739010	I think if you are setting up locally
739670	743230	or in the same place, all ports are running,
743230	746070	make sure you run bash access Kubernetes code.
746070	748130	What it does is basically put forward
748130	751870	all the ports or services into your local system
751870	753210	so that you can access locally
753210	757330	without even exposing to the world, which is internet, right?
757330	759030	So now if you go ahead and click this,
759030	760930	basically you can see the homepage,
760930	763230	like which is the Kubernetes code homepage, right?
763230	765910	So this is where you can see when it's good, it's running,
765910	767310	you can follow the guide here.
767670	770010	And also if you want to check out each scenario,
770010	771390	the scenarios are here.
771710	774090	So let's get started with the worst scenario,
774090	776010	which we will look at one of them
776010	778250	is called sensitive keys in code basis.
778530	781530	So the Kubernetes code is designed in such a way,
781530	784570	if you wanted to practice as an attacker or learner,
784570	786070	you can go ahead and try this scenario
786070	789090	by clicking here and starting with attacking.
789290	791230	Or if you are someone who wanted to learn
791230	794350	and practice yourself and follow some kind of guide,
794350	796370	we have created the documentation.
796370	798470	The solutions will be here in the same guide
799230	800710	for each scenario.
800730	802590	Like you can follow them as well here
802590	804470	if you're stuck somewhere, right?
804610	806570	And some scenarios we try to create
806570	808050	two different methodologies.
808050	809990	It can be tagged in different ways.
809990	812830	So we try to document in the same way.
814670	816270	So if you see here,
816270	819090	let's go back and read a little bit about the scenario.
819090	821650	Developers tends to commit sensitive information
821650	823070	to version control systems.
823190	826310	As we are moving towards the ACD and GitHub systems,
826310	828770	we tend to forget identifying sensitive information
828770	830410	in code and commits.
830410	833250	Let's see if we can find something cool here, right?
833250	837130	So by looking at, you can see that some kind of vulnerability
837130	839290	or some kind of mistake has happened.
841210	843150	Now, as you can read this,
843150	844830	this is version control using Git.
844830	847330	So what I can do is I can try as a DartKit
847330	850710	and you can see the default file in the Git will be config.
850710	852950	Now you can see that there is a bare repository
852950	854610	hanging there, right?
854650	857730	And by the way, these vulnerabilities which you see,
858450	860550	I was showing like in the web vulnerabilities,
860550	862170	these are just an entry points
862170	863890	into getting into the cluster.
863890	865030	In real world,
865030	867690	it may not be the same vulnerabilities you find.
867730	869310	Maybe there is a different approach
869310	871130	you find to get in here, right?
871130	872210	That's the thing.
872510	875110	So now I can see that there is a Git config.
875110	876890	Let's go ahead and see how I can clone
876890	877970	this repository locally
877970	880690	so that I can understand this code base better, right?
880690	882890	So if you want to start,
882890	886470	basically you can go back and follow this repository here.
886470	888410	So there is a tool called GitDumper.
888410	889890	It can be any other tool as well.
889890	891250	I just take an example.
891250	894770	So you can try this tool by running a Python3-GitDumper
895330	897130	localhost and save it like this.
897230	899950	Maybe I think I have configured here.
902070	903110	Cool.
904530	905530	GitDumper.
905590	907710	So what I'm trying to do is I'm cloning
907710	912010	the localhost1230.git folder into local system.
912030	914170	So what it is trying is it is going to fetch
914170	918130	all the repository and clone it locally, right?
918130	921590	So now if you go here, kdiskcode.git,
921590	924390	so it is the Git repository which we have.
924390	925490	So if you see here,
925490	928530	we can see there is a Golang bunch of files,
928530	931850	main.co, and a bunch of custom code, right?
931850	934530	And also you can see Git, the folder.
934530	936470	So you can also look for what are the old commits
936470	938190	which are happening, right?
938190	940870	As developers sometimes make mistakes,
940870	942970	by the way, this scenario is not only focused
942970	944650	towards containers or Kubernetes.
944710	947810	In general world, we have seen these kinds of mistakes
947810	950410	in most of the modern software development,
950410	951830	especially in cloud deployments
951830	955110	or any internal deployments, right?
955790	957890	So I can see some of the bunch of commits
957890	961130	as you attack and you can go and check manually,
961130	965410	like Git checkout to specific, the log.check commit.
965410	966650	And also you can see,
966650	972330	oh, you can see an interesting file called env, right?
972350	973330	So there you go.
973330	976810	So you can see there is the access key and secret key
976810	979050	for the AWS has mistakenly committed.
979090	982930	And you can also see there is the kdiskcode flag is there.
982930	984890	So basically this means that you have reached
984890	986610	the end of this scenario,
986610	989730	but in real world you can use this AWS access key,
989730	992070	secret key and get the session token
992070	995910	and you can do post-exploitation and go along beyond.
996230	998890	So these scenarios are just we have taken
998890	1002750	from what we see in real world as a pen testing engagement.
1002750	1005450	And some of them are like in a common scenarios
1005450	1007910	which we see in real world, right?
1007910	1010190	So this may not be possible
1010190	1013130	and this may be possible in different use cases.
1013130	1014770	So feel free to give it a try
1014770	1018130	and understand what exactly we are trying to achieve here.
1018150	1021730	So basically what will happen is in the same use case.
1021810	1023710	So we can try in different method.
1023710	1025790	Like what I'm trying to do is most of the time
1025790	1028110	developers mistakenly build a Docker images
1028610	1033150	which may be committed with the .git files
1033150	1035550	because they don't ignore these files
1035550	1038070	when it is creating into Docker container,
1038070	1039930	maybe Docker ignore or something.
1039930	1043170	So which might end up pushing this data
1043170	1044890	into the containers as well.
1044890	1046930	So looking at these kinds of sensitive information
1047530	1050030	is very useful in the container.
1050030	1051670	Like the another methodology you can try,
1051670	1053030	as I said, explanation.
1053030	1056030	So you can get the pod name, you can exec into the pod.
1056030	1057730	This is like another method.
1058370	1061030	Now we can also use tools like TruffleHog.
1061030	1063670	So TruffleHog, I think most of you know,
1063670	1068450	this is a tool which is basically finds high entropy
1068450	1071570	or sensitive secrets in a Git repositories.
1071570	1073350	So now I'm trying to do TruffleHog
1073350	1076010	and given the current directory
1076010	1078150	is the where Git repository is there.
1078150	1080510	So what it does is it will check through all the commits
1080510	1082870	and finds, is there any high entropy values
1082870	1085870	or like sensitive secrets found
1085870	1089790	based on the regular expressions and return to the studio.
1089930	1091790	So you can include these kinds of tools
1091790	1093030	in your CI, CD systems
1093030	1095190	to detect these kinds of vulnerabilities.
1095190	1096690	But in attacker's point of view,
1096690	1098770	if you find any of the code basis
1098770	1100230	which has Git or something,
1100230	1102590	maybe it's a really definitely good way to check out
1102590	1105010	in especially the container environments,
1105010	1107450	you might end up finding some kind of sensitive approach.
1108410	1110230	Like you can see the same thing.
1111610	1112410	Cool.
1112430	1114550	So this is our first scenario,
1114550	1117630	which we just tried from the Kubernetes code.
1117630	1120270	And similarly, as I said, you can try out here as well.
1120270	1122850	So what you can do is you can look at here
1123410	1127110	and if you wanted to specifically browse specific code,
1127110	1128470	like one, two, three, zero,
1128470	1129930	in the custom code tab,
1129930	1132270	you can just open in a new browser window also.
1132270	1133750	If you press this icon,
1133750	1134870	then you can just say,
1134870	1136710	I wanted to browse one, two, three, zero code.
1136710	1138170	So you can able to browse the same thing
1138170	1141070	and play it in online yourself for free
1141070	1144750	and understand what exactly you can assess
1144750	1146390	or test for Kubernetes security.
1148090	1148890	Cool.
1149170	1152990	So that's kind of one of the simple scenario.
1152990	1155350	So let's go ahead and see what could be
1155350	1156790	the some of the complex scenarios
1156790	1158410	or different kinds of vulnerabilities
1158970	1161950	you see in Kubernetes security assessment.
1162290	1166490	So let's go back to the some maybe a little down,
1166490	1169290	especially like attacking private registries
1169290	1172330	or Kubernetes CS benchmarks or container escape.
1172750	1174490	So let me go and showcase you
1174490	1177430	how container escape to host system works.
1177770	1181510	To browse to this scenario, basically you navigate here.
1182610	1185510	So this is basically one of the pod
1186270	1187850	inside Kubernetes cluster running
1187850	1191910	with some specific privileged access has, right?
1191930	1194330	So to set the context here,
1194330	1197570	most of the times, especially operations teams
1197570	1200630	may tend to require to run specific containers
1200630	1202050	with high privileges,
1202050	1205070	maybe giving gaps is admin or host privileges
1205070	1207570	or privilege context,
1207570	1210150	which is a full admin, sysadmin capabilities.
1210150	1211530	So these are things.
1211530	1213850	And also to just give an example,
1213850	1215930	how we found in a real world,
1215930	1218250	I have seen, especially while we performing
1218430	1220250	a penetration testing engagement.
1220270	1222710	So most of the developers get an access
1222710	1225750	to completely one namespace with a full privilege.
1225750	1229510	So if your clusters doesn't have PSPs,
1229510	1231750	which is nothing but pod security policies,
1231750	1235190	developers can basically has full access to that namespace.
1235190	1236330	Whatever they can do,
1236330	1240910	they can deploy a pod with specific privileges,
1240910	1244250	which they can gain into the whole system access, right?
1244250	1247450	So also this is one of the use case
1247450	1249350	we have taken to simulate or showcase
1249350	1252830	how you can find these kind of container escape
1252830	1255150	and gain access to the whole system, right?
1256310	1256910	Cool.
1256910	1258490	So what we are trying to do here is
1259130	1261770	we can try and look for some bunch of information.
1261770	1264170	Most of the time, look for a printenv.
1264270	1267030	Sorry, I think I typed print.
1269330	1270290	Cool.
1270290	1271590	So most of the time,
1271590	1275690	you might see some kind of stuff here as well in printenv.
1275690	1277830	And also I'm checking for mount.
1277830	1279590	Is there any mounted directories?
1279590	1281450	Now you can see there is a mounted directory
1281450	1286330	from host system and which is into the container,
1286330	1287850	which means basically in the pod,
1287850	1289730	in this Kubernetes case, right?
1289730	1291290	So now what I can do is,
1292930	1294130	I think,
1295690	1299610	let's look at what exactly the host system is doing.
1299610	1302530	So if you see, there is a host system mounted,
1302530	1304850	like which is basically from the host system,
1304850	1306430	to just give an example.
1306510	1309630	So which is nothing but your underlying host system,
1309630	1314230	which is node, is mounted into your pod as a volume, right?
1314230	1318510	So due to the reason we have extra capabilities, right?
1318510	1320710	Like this container has a bunch of extra capabilities
1320710	1321630	as well.
1321630	1324070	So using those capabilities,
1324070	1326610	as an attacker, what we can do is,
1326610	1330590	we can like try to gain some kind of access
1330590	1331670	into host system.
1331670	1334510	What you can do is, we can sage root on the host system
1334510	1338150	because we have capability of the privilege.
1338450	1340790	And now we can see that it is given
1340790	1343390	that the host system sage root.
1343390	1344710	And now what we can do is,
1344710	1346710	we can perform the internal commands,
1346710	1348950	which are specific to system, right?
1348950	1350930	Like now you can see docker ps.
1351370	1354490	So these are nothing but running inside the host system,
1354490	1355970	which is not a container.
1355970	1358890	So you went below one level from the pod
1359310	1360650	to the host system.
1360650	1363250	From there, you are executing commands, right?
1363250	1368210	So the possible attack surface here you can do is,
1368210	1370590	to do post exploitation might be basically
1371130	1373650	using the host system configuration,
1373650	1376890	which is nothing but node level Kubernetes configuration,
1376890	1379930	to go and try attacking other nodes
1379930	1382270	or to gain access to other nodes,
1382730	1385170	which basically you're controlling the entire cluster
1385170	1387370	by passing through, passing through, right?
1387370	1389590	So for example, what you can do is,
1389590	1392090	you can use the configuration of node,
1392090	1396370	which is available in the cat var lib.
1401580	1406560	So you can use the configuration of cat var lib quip config,
1406560	1409000	which has the node level configuration.
1409000	1410440	So you can use that configuration
1410440	1413180	and perform the Kubernetes commands,
1413180	1414700	which is not only available
1414700	1417380	for the existing namespace specific,
1417380	1419720	it can also perform at the node level,
1419720	1422600	whatever the node has the service account,
1422600	1424200	you can able to do that level.
1424400	1427020	To do that, for example, you can do this command.
1429760	1432080	A kubectl command is not available.
1432080	1433860	Let's download kubectl.
1436640	1440260	Let me download this kubectl here.
1442100	1444880	It just takes some time.
1444880	1446440	Hopefully it is faster.
1447400	1448160	Yeah.
1452190	1453590	Yep, there you go.
1453590	1455030	So what we are trying to do is,
1455030	1457470	we are using now kubectl command
1457470	1459690	and passing the kube config flag,
1459690	1461990	which is using the node configuration
1462420	1465410	and getting all like whatever the all resources
1465410	1468610	in namespace of kube system, right?
1468650	1471450	Oops, I'm trying the same thing again.
1478550	1484250	I have to do chmod plus x kubectl.
1486090	1486830	Cool.
1486830	1489470	Now you can see when I run kubectl kube config
1489470	1491210	with the specific configuration
1491710	1494010	and get all in kube system,
1494010	1495870	it will go ahead and execute that command
1495870	1497770	and get all the resources.
1497770	1499270	Now you can see here,
1499270	1500950	it can able to get all the ports
1500950	1503730	which are not restricted to specific namespace,
1503730	1506830	which is the developer or the user has.
1506830	1508530	And you can able to explore more
1508530	1511730	or access more resources and also services.
1511970	1514990	And you can see some of the specific use cases,
1514990	1516990	especially I think replication controllers
1516990	1519430	and other jobs and secrets,
1519430	1521030	you cannot able to access.
1521230	1524230	So the node, whatever the access it has,
1524230	1525410	the node configuration,
1525410	1527130	you can able to perform same level.
1527130	1529210	So as an attacker, you can do post-exploitation,
1529210	1532430	basically deploy another port in a different node
1532430	1535090	and you can able to pass to another node
1535090	1536730	and again get access to that.
1536730	1540070	So for that, you can just look for how many nodes you have,
1540070	1542410	like currently two nodes, maybe you are in this node,
1542410	1544710	then you can deploy another port in the different node
1544710	1546870	and get again access to the different.
1546870	1550450	So you can basically use this kind of hopping methodology
1550850	1554270	and hop on to another node and gain complete cluster access.
1556030	1560190	So this is mostly you will see if an organization
1560190	1563090	or the company which is not running up
1563090	1565090	the port security policies,
1565090	1567890	you can able to perform these kind of attacks
1567890	1571590	or escalations of the container escapes, right?
1571730	1575330	So you might use BSPs as a defense
1575330	1576590	for these kind of vulnerabilities
1576590	1579770	and it would be an easy and simple way
1579770	1582750	to fix this kind of exploitation.
1583990	1585030	Cool.
1585330	1588450	So let's go ahead and see another use case.
1588450	1591970	I think we have some more time to see two other scenarios.
1591970	1594910	Let's take an example of Kubernetes namespace bypass.
1596770	1597850	Okay.
1598690	1602750	I'm going here, Kubernetes namespace bypass.
1602830	1603970	I don't know exactly.
1603970	1604590	Yeah.
1604590	1607410	So by default, Kubernetes uses flat networking scheme.
1607410	1611570	I think this is one of the default feature of Kubernetes.
1611690	1614150	So by default, Kubernetes uses flat networking scheme
1614150	1617770	means I can able to talk to from any port to any port
1617770	1619670	by default, there is no firewall rules
1619670	1622390	which will happen, allow or deny.
1622390	1624050	By default, anything can talk to anything
1624050	1626530	within the cluster IP range, right?
1626530	1629210	So which means that whenever someone,
1629210	1631710	SRE or ops team or like security people says
1631710	1634190	that there is a namespace we have created in the Kubernetes
1634190	1636810	which means it's just a logical boundary.
1636810	1638670	There is no way you literally segregating
1638670	1640430	between the different namespace
1640430	1644010	unless until apply some kind of security policies
1644010	1647350	like NSPs and network security policies, right?
1647350	1650470	So if you are testing a Kubernetes clusters
1650470	1652070	where there was no NSPs
1652070	1654250	and there was no restrictions between the namespace
1654250	1657670	as attacker, you can able to talk to other namespaces
1657670	1659850	and other services within the cluster
1660470	1663070	by using a standard network security tools
1663070	1664350	like which we use day to day,
1664350	1668050	like Nmap or Zmap or a bunch of tools, right?
1668050	1670190	So to just get started with the scenario,
1670190	1672810	I have created something, an example.
1672830	1673930	So what we are trying to do is
1673930	1675830	we are running something called Hacker Container
1675830	1679050	which has again created by me.
1679050	1683210	I have while performing bunch of Kubernetes security testings
1683210	1686230	I thought that there are so many tools we require
1686230	1689070	especially focused on container environment.
1689070	1691110	So I have created curated list of tools
1691850	1693390	which are maybe 50 plus tools
1693390	1695550	which will help you aid in
1695550	1697910	while performing Kubernetes security assessment.
1697910	1700270	So this container has those all tools
1700270	1702470	like while you are performing Kubernetes security assessment
1702470	1705390	and you can use this as like a Swiss Army knife
1705390	1707410	in security assessment, right?
1707410	1710170	And I have seen this has been very valuable
1710170	1711950	while performing assessment.
1712030	1713810	So definitely I'll try to create
1713810	1715610	some kind of documentation out of it.
1715610	1716950	Maybe hopefully you can go back
1716950	1720570	and try out different use cases with Hacker Container.
1720570	1723670	I think it's going and deploying there and cool.
1723670	1724570	So we got a shell.
1724570	1726950	So what we are doing is we are inside a pod
1727590	1729770	in one of the default namespace.
1729770	1731730	Like if you wanted to just see,
1731730	1733350	let's go ahead and run some command
1733350	1735970	called kubectl get pods.
1735970	1738270	Now you can see Hacker Container
1738270	1740610	which is running in a default namespace.
1740670	1743270	And let's go ahead and see what else we can do
1743270	1744770	from the shell, right?
1744770	1747410	So as attacker, you are in one of the pod
1747410	1750530	and you don't know what else you can do from there, right?
1750750	1753090	So as I said, Hacker Container has a bunch of tools
1753090	1755430	like Docker Bench Security, Kubebench, Linus
1755430	1757390	and a bunch of other tools.
1757470	1760590	So one interesting thing you can do especially is
1761730	1764050	looking for the entire cluster range.
1764050	1767610	As I said, by default, Kubernetes uses flat networking scheme.
1767610	1769870	You can look at the entire cluster range, right?
1769870	1771190	So you can do IP route.
1771190	1774890	You can see this is using 10.0.0.0 network range.
1774890	1777310	And also you can look for what is the current IP address
1777310	1780150	and 10.0.0.0.12.
1780150	1783810	So you can scan the entire network and identify services.
1783810	1786470	So one thing I can confirm, I can assure
1786470	1789610	is most of the times internally within the cluster
1789610	1791450	as a microservices or services
1791450	1794290	which are running, many of the developers
1794290	1796370	or operations teams tend to forget
1796370	1799710	or assume that it is only exposed to internal networks.
1799710	1802790	They will never use authentications or authorizations
1802790	1805870	and you might end up seeing some kind of bypasses
1805870	1808950	which is exposed internally itself,
1808950	1812390	especially Mongo services, Elasticsearch instances
1812890	1814690	and also even some microservices
1814690	1817790	because the API gateway terminates at the ingress level
1818190	1820250	which means before exposing into the world
1820250	1821650	like using load balancers,
1821650	1824390	they will terminate at the ingress level.
1824390	1826630	So you might end up seeing some kind of services
1826630	1829250	especially the application services
1829250	1832050	also don't have authentication authorization checks
1832050	1833590	within the clusters.
1833590	1836550	So this is a really great place to explore
1836550	1838450	and do as much recon you can do
1838450	1841330	and you can gain as much as sensitive information
1841330	1842850	or services access.
1843870	1844730	Cool.
1844730	1847710	So what I'm trying to do is here I'm exploring same
1847710	1850430	and what I did is I'm trying to use ZMap
1850430	1852990	which is similar to Nmap.
1852990	1855530	It's trying to do a scan for the entire cluster range
1855530	1857910	which is to do a faster way.
1858010	1861190	And I'm specifically looking for a port of 6379
1861190	1863350	which is a Redis instance.
1863370	1865350	So I assume that there might be a Redis running
1865350	1867610	or like a 9200 you can also try
1867610	1870730	which is Elasticsearch or Mongo or MySQL
1870730	1872810	or any other services you can scan.
1872810	1875010	It's like traditional security will apply here.
1875250	1877090	Let's go ahead and run this ZMap
1877090	1879230	and see what we can see.
1879230	1880950	So, oops.
1881170	1881770	Okay.
1881770	1886290	So we have to have enable the internal network range
1887390	1889730	allow within this list.
1889730	1891710	Let's go ahead and do that.
1893710	1897750	Let's comment this and run again.
1897750	1898930	Sorry.
1900970	1902170	Oh, okay.
1902170	1906290	You have to do another thing which is specify the gateway
1906290	1908210	where this ZMap is starting.
1908370	1910730	Let's go ahead and do that as well.
1911590	1913110	F1G capital.
1913130	1913710	Okay.
1914330	1916410	And what is the gateway?
1918430	1920490	Yeah, this is the MAC address.
1921050	1921990	Cool.
1922450	1923570	So, yeah.
1923570	1924770	So what it is doing,
1924770	1927250	it is trying to scan the entire cluster range
1927610	1929610	for the specific port 6379
1929610	1933650	and results the output if any of the IPs running
1933650	1935450	or services running or what is running
1935450	1937570	with the specific port open,
1937570	1939790	it'll save into the other results.csv.
1939790	1940710	It take a while.
1940710	1944210	I think it's pretty fast in this case,
1944210	1947850	but this will take some more time to go through and scan.
1948550	1950410	But as I said, in general,
1950410	1953750	we have seen in real world a lot of services running,
1953750	1955270	especially Elasticsearch instances
1955270	1958290	where you can look for without authentication
1959270	1961690	and where you can gain access to indices
1961690	1963450	and the log information,
1963450	1965770	especially audit logs and sensitive information
1965770	1968890	and also Mongo instances without any authentication
1969490	1971750	or also the microservices, right?
1971750	1974270	So this is a pretty goldmine if you are inside a pod
1974270	1975750	and without NSPs,
1975750	1978690	network security policies applied within the cluster, right?
1979670	1980610	Cool.
1980610	1982650	So this will take a while.
1982650	1985250	I think 76% and all stuff.
1985530	1985870	Yeah.
1985870	1987070	As I said, similarly,
1987070	1989750	you can try out those vulnerabilities here
1989750	1992850	and follow the guide if you wanted to do as well.
1992850	1996730	And I think it's a simple way to try out
1996730	1999690	this entire Katakoda environment within the browser.
1999690	2003370	You don't need to set up the entire cluster itself locally.
2003410	2004470	And also once you are done,
2004470	2005690	you can just quit the browser.
2005690	2006350	It's all done.
2006350	2008110	You don't need to clean up the entire environment.
2009290	2010650	And another disclaimer,
2010650	2012490	I think I forgot to mention in the beginning,
2012490	2014790	do not try to run or set up this Kubernetes code
2014790	2016810	in your production environments or in your office
2017250	2021290	because it has intentionally vulnerabilities introduced
2021290	2023670	due to the application flaws or something.
2023670	2025990	It is a vulnerable application,
2025990	2029310	which may have gain access to your production
2029310	2031250	or wherever you run these environments.
2031470	2033950	And at least I try to take care of using,
2033950	2036170	exposing the ports within to the localhost,
2036170	2039290	but it's highly not recommended to run in production.
2039390	2040390	Cool.
2040770	2042570	We have ZMap completed.
2042570	2045090	Let's go ahead and see what we got from the results.
2045330	2049000	You can see that there is the IP,
2049000	2052460	which has 6379 port is exposed within the cluster.
2052640	2055280	And let's go ahead and see what we can get out of it.
2055280	2056500	As you know, it's Redis.
2056500	2060540	As I said, this hacker container has built-in tools
2060540	2063720	to look for these kind of common attacks and all stuff.
2063720	2065420	We can use Redis CLI,
2065420	2068060	and I forgot to be honest, how to access.
2068380	2070700	Yeah, I found H and the IP.
2070760	2073300	And now you can see there is another integrated Redis
2073300	2075900	running within the cluster you can able to access
2075900	2078860	because there is no network security policies
2078860	2081620	which is by default in the cluster,
2081620	2085660	means you can able to access without any issues.
2085660	2087140	I mean, so they don't have any allow list
2087140	2089500	or denial list within the cluster.
2089840	2093500	Now you can do enumeration standard like get key.
2093980	2095600	You can see nothing.
2095600	2095920	Okay.
2095960	2096680	Keys.
2096680	2098900	You can see the keys, what are available,
2098900	2103820	and you can see there is a key and get secret stuff.
2104800	2105540	Cool.
2105540	2106740	So you reach the flag.
2106740	2109800	That means you are done with the challenge, right?
2109800	2113840	So this is just an example of try to simulate the scenarios,
2113840	2116240	but definitely you see these kind of vulnerabilities
2116240	2117660	in a real world.
2117660	2119060	Once you end up in a pod,
2119060	2122660	try to look out for more services and try as much as you can
2122660	2126160	and you will get most information out of it.
2126160	2126720	Right.
2126960	2127940	Cool.
2128620	2131140	So let's go back and see another scenario
2131140	2133740	before we conclude the presentation
2133740	2135360	and moving into the next sections.
2136320	2139040	Let's look at one of the common oldest,
2139040	2141080	the vulnerabilities which we have seen
2142000	2144100	exploiting the Helm or Tiller.
2146320	2149480	So Helm is like a package manager for Kubernetes.
2149480	2152440	So how we deploy application, install applications
2152440	2154300	into, suppose you are using Debian-based
2154300	2156800	apt-get install the package name.
2156800	2159980	Similarly, developers or operations teams use Helm,
2159980	2162760	which is a package manager to deploy applications
2162760	2164500	into Kubernetes, right?
2164500	2167740	So they will just do Helm install and the Nginx,
2167740	2170320	it'll go and install the Nginx within the cluster, right?
2170320	2174180	So by default, if you are testing any cluster
2174180	2175980	and which is using version two,
2175980	2177340	which is the older version of Helm
2177340	2180620	and it used to be by default,
2180620	2183340	most people using is Helm two in a previous cases,
2183340	2186460	but they have raised three recently, like someone's back.
2186600	2190060	So by default, Helm version two has RBAC setup
2190060	2192060	with the full cluster admin access.
2192060	2195240	So if you end up seeing these kind of Helm two configurations
2195680	2196660	running within the cluster
2196660	2199020	and there are no network security policies applied
2199020	2201680	and it is running with the default setup,
2201680	2204560	basically using this misconfiguration as an attacker,
2204560	2206600	you can get entire cluster admin access
2206600	2208820	and form the entire cluster, right?
2208820	2210880	Let's go ahead and see how this scenario looks like.
2210880	2213720	I'm going and running a simple pod
2213720	2216400	and executing into that, right?
2217720	2220080	Cool, let's go ahead and run this command.
2220080	2222980	It'll go ahead and spin up another pod,
2222980	2226300	it is called HelmTeller and give you a bash access, right?
2234910	2236890	Yep, so now we are inside a pod
2236890	2241350	and we assume that this cluster has Helm installed
2241350	2244150	and we wanted to see how we can exploit
2244150	2247670	this kind of vulnerability in a real world environment, right?
2247670	2250570	So let's go back to our documentation, right?
2250570	2253390	So by default in Kubernetes,
2253390	2255270	there is a schema which they follow
2255270	2259010	to access or basically it's like a DNS.
2259190	2262550	Similarly, how we use, they use in the Kubernetes.
2262550	2264450	So this is nothing but a service name,
2264450	2268270	this is the namespace and they use either a service name
2268270	2272010	and namespace and a cluster.local.
2272170	2275490	So this is how the structure which is followed, right?
2275490	2278890	So by default, if you have installed the Helm,
2278890	2281750	which is Teller is get deployed in version two,
2281750	2284510	which is nothing but a server side component of Helm.
2284510	2286030	Helm is the client side version
2286030	2287930	where you do Helm install or something.
2287930	2289790	So the Teller is the server side component,
2289790	2291930	which has a full cluster admin access, right?
2291930	2294170	So now we are doing Teller deploy is the service name
2294170	2297530	and it is by default deployed in a kube system.
2297530	2299230	So I'm trying to look for kube system
2299670	2302630	and I'm taking Telnet and see if this port
2302630	2305390	is able to access from the current port.
2305390	2307030	So this is the default port,
2307030	2309630	which is in a by default configurations
2309630	2313730	exposed to 0.0.0, which means if you are in any one
2313730	2315610	of the four port within the cluster,
2315610	2318470	you can able to access the Teller deployment service, right?
2318470	2322350	You can see, looks like you can able to access the Teller.
2322350	2324150	So which means you can able to talk
2324150	2325370	to the Teller deployment service
2325370	2328150	because there were no NSPs and by default,
2328150	2332050	it is exposed to 0.0.0, which is within the cluster, right?
2332050	2334110	And let's see what we can do.
2334150	2338230	So I have also installed a Helm
2338770	2342090	within this same container to talk to the Helm.
2342090	2345010	So in the Helm version two,
2345010	2348130	you can specify the dash dash host flag,
2348130	2351450	which means you can specify which Teller service
2351450	2352730	you wanted to talk to.
2352730	2354830	Now we know the Teller service name
2354830	2357150	as well as the DNS basically,
2357150	2359550	and we are specifying talk to this host
2359550	2361870	and perform these operations, right?
2361870	2364150	Let's go ahead and run this command.
2367030	2369410	I'm trying to run Helm command
2369410	2371050	and specifying the specific service
2371530	2372730	and the port,
2372730	2374810	and I'm just running Helm version command, right?
2374810	2376970	Now you can see the client version,
2376970	2379150	which is Helm, is 2.16.7,
2379150	2382350	and the server version, which is Teller, is 2.16.9.
2382350	2384370	So I can able to talk to the Helm service
2384370	2387050	and I can also see what all charts deployed,
2387050	2388070	Helm charts, right?
2388070	2390690	Helm-less, like you can see metadata DB is deployed.
2390690	2393330	So you can create your own chart and apply as well, right?
2393330	2395850	So what we have done is, as an attacker,
2395850	2397530	simply to showcase the complexity
2397530	2399810	or what you can do as an attacker,
2399810	2401810	let's go ahead and run a simple command.
2402190	2404430	By default, Kubernetes stores,
2404950	2406130	has a service account,
2406130	2408070	which is stored in var lib,
2408070	2410210	service secret,
2418690	2419730	one,
2419730	2421670	yeah, secrets,
2422230	2424770	Kubernetes IOS service account,
2424770	2425570	namespace,
2425570	2427690	because I'm running in default namespace,
2427690	2431470	and this is the token for that specific service account,
2431470	2433010	which is mapped to the current port.
2433010	2435750	So what I'm trying to do now here is,
2435750	2437270	we have created a Helm chart,
2437270	2438470	which is one chart,
2438470	2442290	and this I have taken from the Bitnami repository research,
2442290	2445590	and I have given the reference here,
2445590	2447310	and go and check more resource.
2447310	2449010	What it does is basically,
2449010	2450470	it's a simple Helm chart.
2450470	2452150	What it is do is,
2452150	2453630	it has a bunch of values,
2453630	2455530	which is nothing but namespace is default,
2455530	2457250	and the name of the,
2457250	2458170	is default.
2458170	2460470	Let's look at the templates, how it looks like.
2461810	2462510	Templates,
2463210	2464950	and a cluster role.
2464950	2466550	It has a cluster role,
2466550	2467630	which is nothing but,
2468390	2471910	it gives all API groups in the Kubernetes,
2471910	2473550	for all the resources,
2473550	2474570	and all the verbs,
2474570	2476050	which means you can do get,
2476050	2476650	post,
2476650	2477670	delete,
2477670	2478210	list,
2478210	2479530	all of the resources,
2479530	2482070	for any of the resources in the Kubernetes cluster,
2482070	2484130	any of the kind of API groups,
2484130	2485490	you can perform.
2485550	2487590	So what we are trying to do is,
2487590	2488330	templates,
2488330	2489930	cluster role binding.
2490090	2492630	So we are trying to bind this cluster role,
2492630	2494790	which is entered across the cluster,
2494790	2496210	to the service account,
2496210	2498050	which is taken, if you've seen,
2498050	2499810	which is namespace is default,
2499810	2500630	because we are in the,
2500630	2502910	this pod is currently running in the default namespace,
2502910	2504430	and the service account is default,
2504430	2505750	because you see here,
2505750	2507170	it's the service account is default,
2507170	2508090	when you do,
2508090	2509650	slash var run,
2510310	2511910	secret service account.
2511930	2513150	So what we are trying to do is,
2513150	2514870	we are creating a new cluster role,
2514870	2516670	which gives full cluster access,
2516670	2518770	to perform anything on all resources,
2518770	2520470	and binding that cluster role,
2520470	2522030	to the default service account.
2522030	2524510	So now, as you can't get the secrets,
2524510	2526150	or any operations,
2526150	2528370	within the cluster using the default account,
2528370	2530110	once we deploy this Helm chart,
2530110	2532510	you can do anything as a cluster admin, right?
2532510	2534070	Let's go ahead and run this,
2534070	2535650	deploy this Helm chart.
2535710	2537330	So we are trying to do Helm,
2537330	2539490	and we are using the same Tiller configuration,
2539490	2541210	and deploying the Pawn chart, right?
2541210	2542230	Let's go ahead and run.
2542230	2544690	So it went ahead and deployed the Pawn chart,
2544690	2548130	because Helm has the capability to talk to the Tiller,
2548130	2549910	and Tiller has the full cluster admin access,
2549910	2551710	so it can go ahead and deploy this chart.
2551710	2555130	So now if you go ahead and run kubectl get secrets,
2555130	2556530	you can able to get the secrets
2556530	2558650	of the entire Kubernetes here, right?
2558650	2562430	So you can also get kubectl get pods,
2562430	2564130	you can able to get all the pods,
2564130	2567070	and in all namespaces as well, right?
2567070	2569170	So basically you have full cluster admin access,
2569170	2571310	because currently whatever the service account,
2571310	2573770	which is there in our pod,
2573770	2576790	because due to the reason we have created a cluster role,
2576790	2578870	this cluster admin, sorry,
2578870	2581990	this pod has basically full cluster admin access,
2581990	2583870	which can able to perform any kind of operations
2583870	2585630	on all resources, right?
2585630	2587890	So as a attacker, some of the useful commands
2588730	2594010	is like look for auth, can I create pods?
2594010	2596450	You can check if the current service account,
2596450	2599550	which you own, if you anytime get a service account access,
2599550	2603070	it can able to create pods or something, sorry, create.
2603370	2606230	You can also check delayed pods,
2606690	2608930	it can able to delay pods or something.
2608930	2612430	You can also specify, can I do in namespace of kube system?
2612650	2615970	So this is talk to the API and give you the information.
2616170	2618510	So this command is pretty useful.
2618510	2620190	If you got a service account and you wanted to see
2620190	2621510	what kind of privilege it has,
2621510	2623390	you can make these calls, right?
2623390	2625090	And also some tips, I don't know,
2625090	2627650	we are about to conclude and let me,
2627650	2629750	whenever you make these commands,
2629750	2632670	you can also increase the verbosity of the Kubernetes,
2632670	2634650	you can increase the six or seven,
2634650	2636590	you can able to see each and every request
2636590	2638490	it is making to the Kubernetes cluster
2638490	2640630	and in a verbose way.
2640630	2641790	Like if you increase this,
2641790	2644930	you can able to see much better, even like headers.
2644930	2646610	And if we increase like height,
2646610	2649310	like you can even see like a response body and everything.
2649450	2652910	And if you increase to nine and like little more depth,
2652910	2654150	it will keep going, right?
2654150	2656370	So this is completely API driven.
2656370	2658470	So you can able to play around with the verbose
2658470	2660670	and understand exactly each and every thing
2660670	2662030	which is being answered.
2662110	2664110	And also very useful command is explain.
2664110	2666330	If you wanted to understand a pod,
2666330	2668430	then it will give you more details about it.
2668430	2670590	And I wanted to learn more about pod spec,
2670590	2674090	you can also do kubectl spec.
2674130	2675550	So this is very handy.
2675550	2677650	And while you are performing assessments,
2677650	2679090	you can quickly look at the documentation
2679090	2681410	rather like explain commands, right?
2682030	2683590	Cool, so now you see,
2683590	2686710	due to the simple default misconfiguration
2686710	2687810	of our Helm pillar,
2687810	2690010	we can able to control the entire cluster.
2690050	2693750	And this vulnerability I have seen in so many of the cluster
2693750	2695750	in real-time pentesting assessments
2695750	2699750	due to the administrator doesn't configure properly.
2699790	2703050	And this is default security installation by Helm itself.
2703110	2705910	So the newer version of Helm v3 fixed this vulnerability,
2705910	2707450	so it kind of patched.
2708770	2711230	Cool, so let me go with the last scenario.
2711230	2712950	I think then we will kind of conclude.
2712950	2714570	And this you can try, it's a DIND,
2714570	2716070	Docker in Docker and Exploitation.
2716070	2718910	This you commonly see in CI-CD systems,
2718910	2720330	especially in build pipelines,
2720330	2722310	like GitLab CI-CD or Bitbucket CI-CD
2722310	2725090	or in real-world where they pass the socket,
2725090	2726050	our socket, right?
2726050	2727250	You can go ahead and try.
2727250	2728390	And SSRF vulnerability,
2728390	2730090	especially this is very widely exploited
2730550	2732050	in the cloud infrastructure.
2732050	2734150	But in coming to the Kubernetes context,
2734150	2736890	it is little much, you can get more out of it.
2736890	2738890	Because as I said, within the cluster,
2738890	2742030	there might not be some kind of services authenticated.
2742030	2743510	There might be services which are giving
2743510	2745230	more health information of the nodes
2745230	2747910	as well as the ports or internal services.
2747910	2750770	Using SSRF, you can get more information out of it
2750770	2754170	within the clusters compared to the cloud native,
2754170	2756490	cloud way of exploiting, right?
2757290	2758210	Cool.
2758210	2760890	So let me just complete with the...
2762490	2763930	Yeah, this looks fun.
2763930	2764470	Cool.
2764470	2767010	So this is the use case where basically
2767010	2768330	I'm trying to showcase,
2768330	2770270	if you end up in any of the pod
2770270	2772790	and the developer or operations teams
2772790	2775410	didn't set up any resource consumptions,
2775410	2778370	like you can only consume these many resources, right?
2778370	2781290	So you can able to use utilities,
2781290	2784330	especially like I'm trying to showcase here, stress-ng.
2784890	2787630	So let me showcase a simple example.
2787870	2790190	Let me show in the terminal, maybe.
2790690	2795830	Exit, kubectl get pods.
2796050	2798210	Now you see a bunch of pods are running.
2798210	2800210	So I'm looking for a specific pod.
2800210	2806550	Let's look for that kubectl top pod.
2807290	2810410	So when you do top, it's basically a system like top command.
2810410	2812090	It shows how much CPU resources
2812090	2814750	and memory resources it is using.
2814810	2817350	And if any of the deployment or manifest
2817350	2820310	which doesn't have specifically resource limit setup
2820310	2823450	and the cluster is deployed in like cloud infrastructure,
2823450	2825130	like which has auto-scaling enabled.
2825130	2826390	If you re-hit this limit,
2826390	2827750	it'll go and scale up a new node
2827750	2831190	or deploys like some more nodes into the cluster, right?
2831210	2832990	So that times you can exploit this
2832990	2835250	in an attacker's point of view.
2835250	2836770	Like let's go ahead and run a simple tool.
2836970	2840390	I'm running with two kicks and 30 seconds.
2840530	2842970	So now you can see if you're on the same command
2843590	2848290	from the specific resources of 3MB, 1MB, 2MB,
2848290	2850290	like increasing so much, right?
2850290	2851890	I think it's just taking some time.
2851890	2853570	Let's keep looking.
2865330	2868850	You can see it hits a crazy amount of memory resource limits
2868850	2873450	because there was no resource limit set for this deployment,
2873450	2874930	whatever the hunger check,
2874930	2877130	it will be deployed as part of the Kubernetes code.
2877490	2879710	Attacker can basically consume more CPU resources
2879710	2881870	and memory, which basically cost the billing
2881870	2883850	and very like cost-effective
2883850	2886550	for the company's perspective, right?
2886550	2887230	Billing.
2887230	2888490	And due to the reason,
2888490	2890410	the most of the modern infrastructure is set up
2890410	2891950	in such a way which is auto-scalable
2891950	2896110	and like scalable based on the certain parameters.
2896230	2897870	So you've been able to basically DDoS
2897870	2899990	or gain more resource consumption
2899990	2904030	for the companies and organizations, right?
2904030	2906590	So this is definitely one thing.
2906590	2909350	So before moving in,
2909350	2912550	let me just conclude with the Hacker Container preview.
2912910	2915030	As I said, Hacker Container,
2915030	2919330	let me just showcase maybe here, Hacker Container.
2919330	2921330	So Hacker Container is a similar,
2921330	2924310	another utility I created as part of the assessment.
2924310	2926570	So you can read more about it here, the blog post.
2926570	2928430	So this has a bunch of utilities,
2928430	2929930	which you use day-to-day,
2929930	2932950	like while performing assessments in the Kubernetes cluster.
2932950	2935490	So I have used it extensively,
2935490	2937370	I would say, while performing assessment
2937370	2939390	and it is very useful utility.
2939390	2940710	Let's go ahead and run.
2941210	2942690	Oops, looks like it's already there.
2942690	2946730	Let's exec into the pod, Hacker Container.
2946810	2953110	So we are inside the pod,
2953110	2957210	like you see a bunch of other stuff like default tools.
2957210	2960150	So some of the things like it has is amicontain,
2960150	2962590	so which is again, utility written by Jessica Frazel.
2962590	2965170	She has done quite a lot of work for the containers
2965170	2967610	and security community in the Kubernetes
2968050	2969610	and the container ecosystem.
2969710	2972030	So what it does is it will try to identify
2972030	2973650	what container runtime you are using
2973650	2977290	and do you have shared namespaces from the host system
2977290	2979330	like PID, username space,
2979330	2981450	or is there any app armor or sitcom profile
2981450	2984630	applied to the container to restrict certain capabilities?
2984810	2987390	And is that what all capabilities allowed currently,
2987390	2988670	system calls allowed,
2988670	2991490	and it kind of a container introspection tool.
2991490	2994330	So which will help you to perform while in attacks
2994330	2996150	and security assessments.
2996150	2998810	So not only I think specific to the container Kubernetes,
2998810	3001510	then it's a pretty useful tool to try out, right?
3001510	3003170	So you can also do the same thing
3003170	3006190	with proc self cgroup maybe.
3006190	3007090	Yeah, cool.
3007090	3009470	So you can see this is basically trying to figure out
3009470	3013010	kubeports and you can also do capsh, print.
3013010	3015230	So it will look for what all capabilities are there
3015230	3016750	and similarly.
3016810	3020190	So this is kind of pretty useful tool, the Hacker Container.
3020710	3025130	And also it has Nikto and a bunch of other use cases.
3025130	3028430	I think maybe I try to not document it here.
3028430	3029950	So I'm going to try to see
3029950	3032150	what can I kind of create out of it
3032150	3035470	and how I use in day to day while exploiting.
3035470	3038530	And some use cases I would say is very useful
3038530	3040250	is running benchmarks.
3040250	3041690	If you are inside the clusters
3041690	3044510	and you wanted to just get a glance of view,
3044510	3046230	what kind of vulnerabilities possibly there
3046230	3048210	within the Kubernetes clusters,
3048210	3049610	both in terms of container runtime
3049610	3051310	as well as the cluster level.
3051310	3053570	You can run these CIS benchmarks.
3053630	3056970	Like one is you can just go ahead and run this command.
3057130	3058470	Let's go ahead and run.
3058510	3058890	Yep.
3058890	3060070	So what it does is it,
3060070	3061970	I have created something called daemonset,
3061970	3063850	which means each node,
3063850	3066250	how many ever nodes you have in the cluster,
3066250	3069050	will get one pod running in all the nodes.
3069050	3071550	That is called daemonset in Kubernetes.
3071650	3072730	Let's go ahead and get the node.
3072730	3074730	So you can see two nodes are available.
3074730	3076650	Now you should be able to see two pods running
3076650	3079570	in the Docker Bench because there are two nodes
3079570	3082030	because we are running a daemonset, right?
3082030	3083610	So you can see two pods running.
3083610	3087170	So let's go ahead and exec into one of the pod
3087170	3090950	and see what we can do with Docker Bench Security, right?
3090950	3092870	So Docker Bench Security,
3095650	3097650	I'm execing into the pod.
3098390	3101570	Now you can see Docker Bench Security.
3101570	3104650	Now you can go ahead and run dockerbenchsecurity.css.
3104650	3106710	This is also part of a Hacker Container.
3106710	3108030	You can go ahead and run this.
3108030	3110930	So the way I have set it up,
3110930	3117650	sorry, the scenario manifest is if you can see here,
3119980	3123840	I have created the Docker Bench Security in such a way
3123840	3127480	it access the host PID, IPC network,
3127480	3129740	as well as a bunch of volumes.
3129740	3132260	And it also had extra capability called audit control
3132260	3133820	so that you can access and talk
3133820	3136000	to your underlying Docker Container runtime
3136000	3138040	and you can perform benchmark analysis
3138040	3140460	on possible security issues, right?
3140460	3143240	So this is a pretty simple utility.
3143240	3144720	Just go ahead and run this.
3144720	3145720	Once you are inside cluster,
3145720	3147020	then you will get at least high level posture
3147020	3149400	and use the whatever the found vulnerabilities
3149400	3153840	or the checks as attack paths,
3153840	3156380	or you can use those while you are doing assessments
3156380	3158140	or post-exploitation, right?
3158220	3159840	Once you are inside a node,
3159840	3161240	maybe you can use some kind of vulnerability
3161240	3164220	which is detected here to do some kind of post-exploitation.
3164280	3166360	So you can see it does a bunch of checks
3166360	3168960	like host level configuration checks
3168960	3171920	and specific to like the Docker daemon
3172400	3175740	or like directories and daemon configuration,
3175740	3178020	and as well as the files,
3178020	3179980	which is specific to the container runtime
3179980	3181620	here in this case, Docker,
3181620	3183460	and also container images,
3183460	3185060	whether container running with a root privilege
3185060	3187140	or it has any extra capability set
3187140	3190420	or it doesn't have any seccomp or apparm profiles,
3190420	3192100	these kinds of things, right?
3192960	3196340	Similarly, you can go ahead and run the kubebench as well.
3196810	3200300	You can go ahead and run this kubebench like this.
3201440	3203940	Let me go ahead and run here only, I think.
3204860	3207400	It's kind of taking quite a lot of time.
3209560	3212200	So you can see kubebench I have run.
3212260	3213940	You can give it pods.
3214060	3216740	The kubebench is running here.
3216740	3219880	You can get the logs, iPhone app, and the bench.
3219940	3220960	So it will show.
3220960	3222520	You can see there are 12 checks passed,
3222520	3224520	nine checks failed, and two checks running.
3224520	3225980	So you can see what all the vulnerabilities
3225980	3227780	possibly there in this Kubernetes cluster
3227780	3230020	and what would be the remediations.
3230020	3231320	So you can use these vulnerabilities
3231320	3233360	maybe to do further exploitation
3233360	3235820	and understand what you can do, right?
3235820	3239600	So these tools are very useful just to get a glance of it.
3239600	3242080	And there are a bunch of utilities out there as well,
3242080	3244480	like kubehunter and kubepsychaudit
3244480	3248100	and the different golang tool written
3248100	3249500	by different researchers.
3249580	3253700	And I'll try to document and put it in the same scenarios.
3253700	3255640	So hopefully you can go back and try out yourself.
3256520	3257240	Cool.
3257380	3259420	So there are a bunch of other scenarios.
3259500	3260560	Due to the time constraint,
3260560	3262720	I couldn't complete or showcase you,
3262720	3266280	but feel free to go and check out the Kubernetes Go project.
3266360	3268380	And you should be able to get and follow
3268380	3272580	using the playground of that Katakoda, right?
3272580	3275160	And if you have set it up in your own environment,
3275160	3276840	like in our case, right, my case,
3276840	3278620	so you can go ahead and tear down the cluster.
3278620	3281260	Basically, it will try to delete possibly resources
3281260	3282380	which I have created
3282380	3284860	as part of the Kubernetes Go deployment.
3284860	3287040	But I highly recommend you delete the cluster
3287820	3291640	to just ensure cleanup of the entire environment, right?
3291780	3295820	And I have also showcased and run using a tool called Chekhov
3295820	3297880	to see what all possible vulnerabilities
3298580	3301600	using a CACD as part of pipeline.
3301600	3305100	So it found a bunch of vulnerabilities in the deployments
3305100	3306980	and other things which I have used
3307600	3309900	as part of Kubernetes Go as well.
3309900	3310700	All right.
3311180	3312040	Cool.
3312040	3315140	So that's it pretty much about it.
3315140	3319860	And I would highly recommend having your feedback
3319860	3323580	and understand what exactly you want to look forward.
3323580	3325960	And I'm really looking forward
3325960	3327360	to building more and more scenarios
3327920	3331020	and also adding the defense scenario.
3331020	3334000	I'm really building like most of the scenarios
3334000	3336840	and making them in such a way that how you can practice
3336840	3341220	for the CNCF certified Kubernetes security.
3341220	3342820	So you can use this playground to practice
3342820	3345760	and understand how most of the Kubernetes vulnerabilities
3345760	3347940	you can assess and learn more about it.
3347940	3349340	So feel free to get involved
3349340	3352800	and you can please feel free to provide the feedback
3352800	3356180	and share with your colleagues, teammates, or your friends.
3356180	3358780	I would really appreciate if you can share the feedback
3358780	3361180	and I would love to improve and work on it.
3361180	3363000	And if you see any documentation
3363000	3365540	or anything needs to be improved,
3365540	3367680	please let me know by GitHub issues.
3367680	3369860	And I'm trying to be pretty active,
3369860	3371880	but in the past month I was a little busy.
3371880	3373380	So I kind of work on this,
3373380	3375440	working on defense scenarios as well.
3375440	3377100	Hopefully we will be releasing this month
3377100	3379400	for the defense scenarios to understand
3379400	3381680	and fix the vulnerabilities which you have learned
3381680	3383500	while practicing these attacks
3383500	3386680	and also more attacker scenarios as well.
3387080	3391240	And with that, I think I'm pretty much done
3391240	3392500	with the session.
3392500	3394820	Thank you so much once again for having me
3394820	3396800	and really looking forward.
3397200	3398440	Awesome. Thank you so much.
3398440	3399920	It was an amazing presentation.
3399920	3401140	I learned so much.
3401160	3402960	And as a matter of fact, in Discord,
3402960	3404260	there are a lot of people actually talking
3404260	3405860	about your presentation.
3405940	3408580	And once again, thank you for supporting the communities.
3408580	3411040	Thank you for supporting the DEF CON Red Team Village.
3411360	3412920	We're going to go on a little break,
3412920	3415800	but we invite everybody that will have any further questions
3415800	3418200	and so on to join the Discord channel,
3418200	3420000	which is in the description of the video.