[00:00.000 --> 00:02.720]  your support for supporting the DEF CON community
[00:02.720 --> 00:05.000]  and the Red Team Village specifically.
[00:05.000 --> 00:06.940]  We're looking forward to your presentation.
[00:06.940 --> 00:08.060]  So take it away.
[00:08.260 --> 00:12.200]  Thank you so much, Omer, for introducing and having me here.
[00:12.300 --> 00:14.800]  Let me just share the screen and confirm.
[00:15.860 --> 00:17.420]  Can you see the screen?
[00:20.450 --> 00:21.810]  Yes, you're all set.
[00:21.810 --> 00:23.050]  Awesome, perfect.
[00:23.190 --> 00:25.630]  Thank you everyone for joining me.
[00:25.630 --> 00:29.050]  And first of all, to just give some context,
[00:29.530 --> 00:32.790]  thank you so much, DEF CON Red Team Village,
[00:32.790 --> 00:35.290]  especially in this pandemic times to make it virtual
[00:35.290 --> 00:37.770]  and people can enjoy these talks
[00:37.770 --> 00:39.990]  without any physical intervention.
[00:40.210 --> 00:44.190]  And this talk is about the Kubernetes code.
[00:44.190 --> 00:46.650]  So disclaimer, basically,
[00:46.650 --> 00:49.430]  today's we will not see any slides, basically.
[00:49.490 --> 00:51.690]  And it's mostly, everything is in online.
[00:51.690 --> 00:53.030]  You can play along with me,
[00:53.030 --> 00:55.110]  or if you can go back and try yourself,
[00:55.110 --> 00:56.850]  you can go ahead and try as well.
[00:56.850 --> 00:58.430]  So there will be no slides,
[00:58.430 --> 01:01.510]  mostly live demos and the hands-on exercises.
[01:02.050 --> 01:02.790]  Cool.
[01:02.790 --> 01:05.030]  So this talk is primarily focusing
[01:05.030 --> 01:07.630]  on introducing a project recently I have created
[01:07.630 --> 01:09.570]  called Kubernetes Code,
[01:09.570 --> 01:14.150]  which is, this is in github.madhuvakula.kubernetescode.
[01:14.150 --> 01:17.650]  This project was mainly intentionally vulnerable application,
[01:17.650 --> 01:21.530]  which is to basically try and teach participants,
[01:21.530 --> 01:23.410]  or like anyone come to the new Kubernetes
[01:23.410 --> 01:26.710]  to understand how you can assess these Kubernetes clusters
[01:26.710 --> 01:29.290]  and find vulnerabilities so that you can go back
[01:29.290 --> 01:31.430]  and try it out in your organizations,
[01:31.430 --> 01:32.430]  or day-to-day pen tests,
[01:32.430 --> 01:34.550]  which you find are a team exercises, right?
[01:34.550 --> 01:37.210]  So I have presented at some of the places
[01:37.210 --> 01:39.210]  and also I have created a playground.
[01:39.210 --> 01:40.310]  If you wanted to write now,
[01:40.310 --> 01:43.290]  try it in your browser and follow along, right?
[01:43.670 --> 01:46.770]  So a little bit about me, myself, Madhu Akula.
[01:46.770 --> 01:48.150]  I work at currently Zibia
[01:48.150 --> 01:50.470]  as a Cloud Native Security Specialist.
[01:50.470 --> 01:52.830]  And I have been speaking and training
[01:52.830 --> 01:54.850]  at a bunch of conferences around the world,
[01:54.850 --> 01:57.890]  like DEF CON Black Hat, USENIX, Orally, and some of them.
[01:58.150 --> 02:00.650]  And I'm fortunate to find some of the vulnerabilities
[02:00.650 --> 02:02.670]  long back using my research
[02:03.330 --> 02:05.970]  in some of the organizations like Microsoft,
[02:05.970 --> 02:08.070]  LinkedIn, Google, and some of them.
[02:08.270 --> 02:10.270]  So apart from that, my primary interest
[02:10.270 --> 02:13.430]  mostly working around security of cloud,
[02:13.430 --> 02:14.610]  containers, Kubernetes,
[02:14.610 --> 02:16.810]  and automating the infrastructure part.
[02:16.910 --> 02:19.450]  So that's a little bit about me.
[02:19.450 --> 02:20.650]  If you wanted to follow me,
[02:20.650 --> 02:22.590]  you can follow me at Madhu Akula
[02:22.590 --> 02:24.710]  in most of the social media stuff.
[02:25.610 --> 02:26.510]  Cool.
[02:26.510 --> 02:29.550]  Let's get into what exactly is this Kubernetes code
[02:29.550 --> 02:31.650]  and what we are trying to learn
[02:31.650 --> 02:33.930]  like within this next one hour, right?
[02:34.010 --> 02:38.490]  So to give a little context about Kubernetes code.
[02:38.530 --> 02:40.550]  So as I said, it is intentionally vulnerable.
[02:40.550 --> 02:42.690]  I think as most of you have come across
[02:42.690 --> 02:44.150]  the different types of code,
[02:44.150 --> 02:47.110]  like web code, or Terra code, or some other codes.
[02:47.110 --> 02:48.810]  So this is a similar project,
[02:48.810 --> 02:50.970]  but it is mostly focusing on Kubernetes
[02:51.510 --> 02:54.270]  to understand what are the known vulnerabilities.
[02:54.270 --> 02:56.370]  And also we have seen common vulnerabilities
[02:56.370 --> 02:58.030]  in the real world environments,
[02:58.030 --> 03:00.410]  like while performing penetration testing,
[03:00.410 --> 03:02.150]  or like while performing assessments
[03:02.150 --> 03:03.990]  in Kubernetes environments.
[03:04.010 --> 03:06.710]  From those we have taken as scenarios.
[03:06.710 --> 03:08.690]  And these are some of the scenarios.
[03:08.690 --> 03:10.410]  Let me zoom a bit.
[03:10.410 --> 03:12.730]  I think maybe easier to see.
[03:12.730 --> 03:13.330]  Yeah.
[03:13.330 --> 03:16.170]  So these are some of the scenarios I have created.
[03:16.170 --> 03:19.010]  And we will see some of the scenarios,
[03:19.010 --> 03:21.250]  exploitation, and how you can bypass
[03:21.250 --> 03:23.230]  or gain access to Kubernetes clusters
[03:23.970 --> 03:26.110]  as an attacker point, right?
[03:26.110 --> 03:28.950]  And I'm really sorry, and I'm shameful.
[03:29.070 --> 03:31.450]  I said I will keep updating these slides
[03:31.450 --> 03:33.490]  at the documentation.
[03:33.530 --> 03:36.570]  And also I have been working on the defense part also.
[03:36.570 --> 03:39.030]  Hopefully this month we will see them coming away.
[03:39.530 --> 03:41.110]  This part I'm going to skip.
[03:42.130 --> 03:44.630]  So before diving into Kubernetes,
[03:44.630 --> 03:46.470]  if you are really completely new
[03:46.470 --> 03:49.350]  and coming to the containers or Kubernetes world,
[03:49.350 --> 03:51.570]  I highly recommend understanding the technology
[03:51.570 --> 03:53.650]  because there is a fair amount of gap
[03:53.650 --> 03:55.550]  in between penetration testing
[03:55.550 --> 03:57.570]  or finding or assessing clusters
[03:58.410 --> 04:01.690]  if you don't understand what technology you are doing, right?
[04:01.690 --> 04:03.650]  So we have seen commonly,
[04:03.650 --> 04:06.390]  especially in the modern tech infrastructure world,
[04:06.390 --> 04:08.510]  there is a huge gap for the security team
[04:08.510 --> 04:10.070]  to understand the technology
[04:10.070 --> 04:13.130]  before even testing or trying out.
[04:13.130 --> 04:15.070]  Because once you are able to understand,
[04:15.070 --> 04:16.430]  you can be able to do your best
[04:16.430 --> 04:18.010]  or you can be able to possibly find
[04:18.010 --> 04:21.330]  most of the Kubernetes or any ecosystem.
[04:21.550 --> 04:22.890]  This is a very funny video
[04:22.890 --> 04:25.750]  and it's a simple children's guide
[04:25.750 --> 04:28.230]  to explain what exactly Kubernetes is.
[04:28.230 --> 04:29.290]  So I would highly recommend
[04:29.290 --> 04:31.950]  if you want to go through these things.
[04:31.950 --> 04:34.570]  So to give you a little quick context,
[04:34.570 --> 04:36.610]  how Kubernetes architecture looks like
[04:36.610 --> 04:38.050]  at a higher level, right?
[04:38.050 --> 04:40.850]  So this is again from the wiki comments.
[04:40.850 --> 04:43.850]  You see a server and client architecture
[04:43.850 --> 04:46.890]  similar to most of the server client.
[04:47.090 --> 04:50.010]  So one thing you have to observe here is in Kubernetes,
[04:50.010 --> 04:52.050]  everything which you make, any operation,
[04:52.050 --> 04:54.230]  it may be from within the master,
[04:54.230 --> 04:55.410]  which means server,
[04:55.410 --> 04:57.710]  within the node, which means worker node.
[04:57.710 --> 05:00.110]  So whatever the operations which will make,
[05:00.110 --> 05:01.390]  like it can be from the node,
[05:01.390 --> 05:02.410]  it can be from the developer
[05:02.410 --> 05:04.710]  or any user, any service
[05:04.710 --> 05:08.390]  will go and touch something called API server.
[05:08.390 --> 05:09.870]  So which is one of the key component
[05:09.870 --> 05:11.610]  in terms of Kubernetes,
[05:11.610 --> 05:13.670]  because Kubernetes is completely driven
[05:13.670 --> 05:15.670]  by API first approach.
[05:15.670 --> 05:18.290]  So whatever you make all requests or something,
[05:18.290 --> 05:21.550]  everything you will see going through the API server, right?
[05:21.550 --> 05:24.890]  And control manager is basically to make the decisions,
[05:24.890 --> 05:26.610]  whatever the requests you make.
[05:26.610 --> 05:30.230]  And scheduler is ensuring to place the workloads
[05:30.230 --> 05:33.090]  or the request which you made.
[05:33.090 --> 05:35.330]  Suppose if you wanted to serve a Nginx server
[05:35.330 --> 05:37.230]  with 10 pods, it will ensure that
[05:37.230 --> 05:40.570]  how many nodes has the capacity to schedule.
[05:40.570 --> 05:42.530]  So the scheduler job is to figure out that
[05:42.530 --> 05:44.190]  and place them accordingly.
[05:44.310 --> 05:47.090]  So etcd is one of the key value store database,
[05:47.090 --> 05:48.750]  which is used by Kubernetes,
[05:48.750 --> 05:50.790]  which is as its name suggests, key value.
[05:50.790 --> 05:52.270]  So it stores the data,
[05:52.270 --> 05:54.630]  which is coming to the API server
[05:54.630 --> 05:56.910]  and stores the key value base.
[05:56.910 --> 06:00.870]  And we will see a little bit in a later coming sections.
[06:01.650 --> 06:03.350]  So apart from that nodes,
[06:03.350 --> 06:04.210]  when you come to the node,
[06:04.210 --> 06:07.390]  kubelet is the talking point to the API server
[06:07.390 --> 06:10.730]  and communicates within the local node itself.
[06:10.730 --> 06:14.450]  So whenever you wanted to make a request as a scheduler,
[06:14.450 --> 06:16.110]  you wanted to schedule in the node,
[06:16.110 --> 06:18.370]  it will talk to the container runtime,
[06:18.370 --> 06:19.510]  which is within the node,
[06:19.510 --> 06:22.650]  like it can be Docker, RunC, etcd.
[06:22.650 --> 06:25.810]  So then it will go ahead and schedule
[06:25.810 --> 06:28.430]  based on the workload kind of thing.
[06:28.610 --> 06:32.030]  So kubeproxy is one of the another key component,
[06:32.030 --> 06:34.110]  which is kind of like IP tables,
[06:34.110 --> 06:36.010]  like it will try to provide a communication
[06:36.630 --> 06:39.230]  across like the nodes as well as a cluster.
[06:39.810 --> 06:43.730]  So let's go ahead and a little bit go deep,
[06:43.730 --> 06:47.010]  but I would highly recommend you practice the Kubernetes
[06:47.010 --> 06:49.990]  before you diving into the assessments
[06:49.990 --> 06:51.370]  or security testing.
[06:51.370 --> 06:54.390]  So here are some of the references I have curated.
[06:54.390 --> 06:55.830]  These are pretty hands-on,
[06:55.830 --> 06:57.250]  like if you wanted to try,
[06:57.250 --> 06:59.970]  so you can go ahead and try as a literally playground
[06:59.970 --> 07:03.670]  and also specific objects if you start while testing,
[07:03.670 --> 07:06.190]  you can click and understand what exactly it is.
[07:06.190 --> 07:07.770]  So these references I would,
[07:08.370 --> 07:10.670]  I think it would be definitely helpful to try.
[07:11.130 --> 07:12.090]  Cool.
[07:12.490 --> 07:14.370]  Setting up Kubernetes code.
[07:14.610 --> 07:16.870]  The Kubernetes code is in design in such a way
[07:16.870 --> 07:18.690]  it can set up any out of the box.
[07:18.690 --> 07:20.810]  It can be in your existing cloud provider services
[07:20.810 --> 07:23.990]  like GKE, EKS, AKS, or digitalization based.
[07:24.350 --> 07:26.050]  And also your locally provisioned
[07:26.050 --> 07:27.750]  Minikube or another playground.
[07:27.750 --> 07:31.550]  To make easy of setup for the people to try out and learn,
[07:31.550 --> 07:33.990]  I have created something called Katakoda Playground.
[07:33.990 --> 07:35.490]  If you just go ahead and click this,
[07:35.490 --> 07:37.510]  it will open up a playground for you,
[07:37.510 --> 07:39.950]  which is completely free and online.
[07:39.950 --> 07:41.790]  So you don't need anything to be done.
[07:41.790 --> 07:43.450]  I think you have to register first time.
[07:43.450 --> 07:46.470]  So then you can get the complete setup here for you
[07:46.950 --> 07:48.290]  to practice and learn.
[07:48.290 --> 07:50.130]  Let me put it in a presentation mode
[07:50.130 --> 07:51.590]  so that you can see a little better.
[07:51.590 --> 07:52.770]  Yep. Cool.
[07:53.650 --> 07:54.270]  Cool.
[07:54.330 --> 07:57.950]  So I'll walk you through how you can set this up
[07:57.950 --> 07:58.790]  and practice.
[07:58.790 --> 08:01.350]  And if you wanted to follow along with me,
[08:01.350 --> 08:03.670]  you can follow this guide at madhuvakula.com
[08:03.670 --> 08:05.470]  slash Kubernetes code.
[08:05.470 --> 08:07.450]  This is the URL, right?
[08:07.530 --> 08:07.950]  Cool.
[08:07.950 --> 08:09.690]  And if you wanted to set up your own,
[08:09.690 --> 08:14.310]  you can just follow this, the documentation.
[08:14.310 --> 08:16.830]  Basically you need to have a working Kubernetes cluster
[08:16.830 --> 08:18.630]  set up with admin access.
[08:18.630 --> 08:21.570]  And also you need a Helm version too,
[08:21.570 --> 08:22.890]  because in one of the scenario,
[08:22.890 --> 08:25.070]  we will try to focus on how you can exploit
[08:25.070 --> 08:27.990]  the existing vulnerability in the older version of Helm.
[08:27.990 --> 08:29.370]  So that's it.
[08:29.370 --> 08:31.390]  So if you wanted to set up your own,
[08:31.390 --> 08:33.250]  let's go ahead and do that as well.
[08:33.430 --> 08:35.550]  So if you have your own set cluster,
[08:35.550 --> 08:36.810]  you clone the repository,
[08:36.810 --> 08:40.630]  which is the git clone of the Kubernetes code.
[08:40.710 --> 08:46.050]  And then you can go ahead and set up the Kubernetes code.
[08:46.390 --> 08:50.650]  So it checks if Kubernetes kubectl is available,
[08:50.650 --> 08:52.950]  Helm 2 is available in your path,
[08:52.950 --> 08:55.310]  and it'll go ahead and deploy the tiller,
[08:55.310 --> 08:57.810]  which is the server side component of the Helm,
[08:57.810 --> 08:59.910]  and a bunch of vulnerable services,
[08:59.910 --> 09:03.590]  which will be used as practice to exploit
[09:03.590 --> 09:06.470]  or try practicing Kubernetes security.
[09:07.550 --> 09:11.670]  So if you are someone following the Katakoda live,
[09:11.670 --> 09:13.230]  you can use this playground,
[09:13.230 --> 09:15.230]  which is completely free and online.
[09:15.230 --> 09:16.790]  You can go ahead and do the same thing
[09:16.790 --> 09:19.010]  in the playground as well.
[09:19.790 --> 09:22.030]  So the playground is designed in such a way,
[09:22.030 --> 09:24.010]  you can go ahead and see here only the guide
[09:24.010 --> 09:26.510]  in the second, in the specific tab.
[09:26.510 --> 09:29.010]  And also you can browse those ports and other stuff
[09:29.010 --> 09:31.270]  within the browser itself.
[09:31.270 --> 09:33.490]  Even in your mobile, also you can try this.
[09:35.050 --> 09:37.730]  So it takes a while to set up
[09:37.730 --> 09:41.910]  the entire Kubernetes code platform.
[09:41.910 --> 09:43.630]  I think it's almost done.
[09:43.630 --> 09:46.970]  So it creates a bunch of vulnerable scenarios,
[09:46.970 --> 09:49.610]  which I have pre-created.
[09:49.650 --> 09:51.070]  You can go through the code
[09:51.070 --> 09:53.550]  and you can able to see what exactly it is trying to do,
[09:53.550 --> 09:55.010]  if you wanted to look at it.
[09:55.010 --> 09:57.350]  So this is nothing but it checks
[09:57.350 --> 10:00.190]  if kubectl is existing or Helm 2 existing
[10:00.190 --> 10:03.010]  and it sets a bunch of emails,
[10:03.010 --> 10:05.910]  like which is nothing but deployments in the cluster.
[10:07.550 --> 10:08.030]  Cool.
[10:08.030 --> 10:10.410]  Now we have the cluster up and running
[10:10.410 --> 10:12.170]  and we have Kubernetes code set up.
[10:12.170 --> 10:13.730]  So what's next, right?
[10:13.730 --> 10:16.470]  So I think it looks like here also it is done.
[10:16.550 --> 10:18.970]  Ensure you check all the ports are running.
[10:18.970 --> 10:21.990]  You can go ahead and run kubectl get ports.
[10:21.990 --> 10:24.230]  It is currently container creating,
[10:24.230 --> 10:25.730]  which means it's starting.
[10:25.810 --> 10:27.950]  It takes a while to set up the,
[10:27.950 --> 10:31.530]  depends on the way you are running the Kubernetes cluster.
[10:31.650 --> 10:34.770]  So let me go ahead and do this as well here,
[10:34.770 --> 10:37.230]  kubectl get ports.
[10:37.390 --> 10:39.870]  I hope you people can see the screen.
[10:39.870 --> 10:42.630]  I think, let me zoom a little bit as well.
[10:42.630 --> 10:43.150]  Cool.
[10:43.150 --> 10:44.930]  So you can see a bunch of them.
[10:44.930 --> 10:46.790]  Most of the ports are created.
[10:46.790 --> 10:50.130]  I think just only one port is waiting to get created.
[10:50.390 --> 10:51.790]  That's awesome.
[10:51.830 --> 10:53.370]  Now you can see all ports are running
[10:53.370 --> 10:55.770]  and I think this is intentional.
[10:56.150 --> 10:58.150]  This is a job and get company.
[10:58.350 --> 10:59.990]  So if you see the status,
[10:59.990 --> 11:03.450]  basically your entire Kubernetes code is set up and running.
[11:03.450 --> 11:05.970]  So now it's time to hack and get into the cluster.
[11:05.970 --> 11:06.450]  Bye.
[11:07.490 --> 11:08.270]  Cool.
[11:08.270 --> 11:10.890]  That's where we go back and refer to the documentation.
[11:10.990 --> 11:13.050]  So if you look at the scenarios,
[11:13.050 --> 11:15.810]  so there are a bunch of scenarios we have created,
[11:15.810 --> 11:16.770]  as I said.
[11:16.770 --> 11:18.490]  So for the time being,
[11:18.490 --> 11:20.630]  for the entire today's presentation,
[11:20.630 --> 11:23.430]  we will not be able to look at all of those scenarios,
[11:23.430 --> 11:25.130]  but definitely we will be looking
[11:25.130 --> 11:26.870]  at some of the primary scenarios,
[11:26.870 --> 11:29.470]  which might be interesting and fun to try out.
[11:29.490 --> 11:32.270]  And I would highly recommend go back and try out yourself
[11:32.270 --> 11:35.610]  and check out more other scenarios as well, right?
[11:35.610 --> 11:40.610]  And in some cases, especially not all of them scenarios,
[11:40.610 --> 11:41.710]  but in some scenarios,
[11:41.710 --> 11:44.030]  I try to create a flag in this format,
[11:44.030 --> 11:45.910]  k8sgoot-goot.
[11:45.910 --> 11:51.890]  And some MD5s, basically it's a number of 32 characters.
[11:52.390 --> 11:55.290]  Then if you reach this kind of flag,
[11:55.290 --> 11:58.050]  basically it means end of the scenarios,
[11:58.050 --> 12:00.510]  but it may not be found in for all the scenarios,
[12:00.510 --> 12:02.550]  but I try to keep for certain scenarios
[12:02.550 --> 12:07.590]  to reduce the effort of like going on beyond post-exploitation.
[12:09.010 --> 12:09.870]  Cool.
[12:09.870 --> 12:13.030]  So let's go ahead and start with the scenario one,
[12:13.030 --> 12:15.050]  sensitive keys in code base.
[12:15.050 --> 12:17.390]  So I would recommend before doing that,
[12:17.390 --> 12:19.010]  I think if you are setting up locally
[12:19.670 --> 12:23.230]  or in the same place, all ports are running,
[12:23.230 --> 12:26.070]  make sure you run bash access Kubernetes code.
[12:26.070 --> 12:28.130]  What it does is basically put forward
[12:28.130 --> 12:31.870]  all the ports or services into your local system
[12:31.870 --> 12:33.210]  so that you can access locally
[12:33.210 --> 12:37.330]  without even exposing to the world, which is internet, right?
[12:37.330 --> 12:39.030]  So now if you go ahead and click this,
[12:39.030 --> 12:40.930]  basically you can see the homepage,
[12:40.930 --> 12:43.230]  like which is the Kubernetes code homepage, right?
[12:43.230 --> 12:45.910]  So this is where you can see when it's good, it's running,
[12:45.910 --> 12:47.310]  you can follow the guide here.
[12:47.670 --> 12:50.010]  And also if you want to check out each scenario,
[12:50.010 --> 12:51.390]  the scenarios are here.
[12:51.710 --> 12:54.090]  So let's get started with the worst scenario,
[12:54.090 --> 12:56.010]  which we will look at one of them
[12:56.010 --> 12:58.250]  is called sensitive keys in code basis.
[12:58.530 --> 13:01.530]  So the Kubernetes code is designed in such a way,
[13:01.530 --> 13:04.570]  if you wanted to practice as an attacker or learner,
[13:04.570 --> 13:06.070]  you can go ahead and try this scenario
[13:06.070 --> 13:09.090]  by clicking here and starting with attacking.
[13:09.290 --> 13:11.230]  Or if you are someone who wanted to learn
[13:11.230 --> 13:14.350]  and practice yourself and follow some kind of guide,
[13:14.350 --> 13:16.370]  we have created the documentation.
[13:16.370 --> 13:18.470]  The solutions will be here in the same guide
[13:19.230 --> 13:20.710]  for each scenario.
[13:20.730 --> 13:22.590]  Like you can follow them as well here
[13:22.590 --> 13:24.470]  if you're stuck somewhere, right?
[13:24.610 --> 13:26.570]  And some scenarios we try to create
[13:26.570 --> 13:28.050]  two different methodologies.
[13:28.050 --> 13:29.990]  It can be tagged in different ways.
[13:29.990 --> 13:32.830]  So we try to document in the same way.
[13:34.670 --> 13:36.270]  So if you see here,
[13:36.270 --> 13:39.090]  let's go back and read a little bit about the scenario.
[13:39.090 --> 13:41.650]  Developers tends to commit sensitive information
[13:41.650 --> 13:43.070]  to version control systems.
[13:43.190 --> 13:46.310]  As we are moving towards the ACD and GitHub systems,
[13:46.310 --> 13:48.770]  we tend to forget identifying sensitive information
[13:48.770 --> 13:50.410]  in code and commits.
[13:50.410 --> 13:53.250]  Let's see if we can find something cool here, right?
[13:53.250 --> 13:57.130]  So by looking at, you can see that some kind of vulnerability
[13:57.130 --> 13:59.290]  or some kind of mistake has happened.
[14:01.210 --> 14:03.150]  Now, as you can read this,
[14:03.150 --> 14:04.830]  this is version control using Git.
[14:04.830 --> 14:07.330]  So what I can do is I can try as a DartKit
[14:07.330 --> 14:10.710]  and you can see the default file in the Git will be config.
[14:10.710 --> 14:12.950]  Now you can see that there is a bare repository
[14:12.950 --> 14:14.610]  hanging there, right?
[14:14.650 --> 14:17.730]  And by the way, these vulnerabilities which you see,
[14:18.450 --> 14:20.550]  I was showing like in the web vulnerabilities,
[14:20.550 --> 14:22.170]  these are just an entry points
[14:22.170 --> 14:23.890]  into getting into the cluster.
[14:23.890 --> 14:25.030]  In real world,
[14:25.030 --> 14:27.690]  it may not be the same vulnerabilities you find.
[14:27.730 --> 14:29.310]  Maybe there is a different approach
[14:29.310 --> 14:31.130]  you find to get in here, right?
[14:31.130 --> 14:32.210]  That's the thing.
[14:32.510 --> 14:35.110]  So now I can see that there is a Git config.
[14:35.110 --> 14:36.890]  Let's go ahead and see how I can clone
[14:36.890 --> 14:37.970]  this repository locally
[14:37.970 --> 14:40.690]  so that I can understand this code base better, right?
[14:40.690 --> 14:42.890]  So if you want to start,
[14:42.890 --> 14:46.470]  basically you can go back and follow this repository here.
[14:46.470 --> 14:48.410]  So there is a tool called GitDumper.
[14:48.410 --> 14:49.890]  It can be any other tool as well.
[14:49.890 --> 14:51.250]  I just take an example.
[14:51.250 --> 14:54.770]  So you can try this tool by running a Python3-GitDumper
[14:55.330 --> 14:57.130]  localhost and save it like this.
[14:57.230 --> 14:59.950]  Maybe I think I have configured here.
[15:02.070 --> 15:03.110]  Cool.
[15:04.530 --> 15:05.530]  GitDumper.
[15:05.590 --> 15:07.710]  So what I'm trying to do is I'm cloning
[15:07.710 --> 15:12.010]  the localhost1230.git folder into local system.
[15:12.030 --> 15:14.170]  So what it is trying is it is going to fetch
[15:14.170 --> 15:18.130]  all the repository and clone it locally, right?
[15:18.130 --> 15:21.590]  So now if you go here, kdiskcode.git,
[15:21.590 --> 15:24.390]  so it is the Git repository which we have.
[15:24.390 --> 15:25.490]  So if you see here,
[15:25.490 --> 15:28.530]  we can see there is a Golang bunch of files,
[15:28.530 --> 15:31.850]  main.co, and a bunch of custom code, right?
[15:31.850 --> 15:34.530]  And also you can see Git, the folder.
[15:34.530 --> 15:36.470]  So you can also look for what are the old commits
[15:36.470 --> 15:38.190]  which are happening, right?
[15:38.190 --> 15:40.870]  As developers sometimes make mistakes,
[15:40.870 --> 15:42.970]  by the way, this scenario is not only focused
[15:42.970 --> 15:44.650]  towards containers or Kubernetes.
[15:44.710 --> 15:47.810]  In general world, we have seen these kinds of mistakes
[15:47.810 --> 15:50.410]  in most of the modern software development,
[15:50.410 --> 15:51.830]  especially in cloud deployments
[15:51.830 --> 15:55.110]  or any internal deployments, right?
[15:55.790 --> 15:57.890]  So I can see some of the bunch of commits
[15:57.890 --> 16:01.130]  as you attack and you can go and check manually,
[16:01.130 --> 16:05.410]  like Git checkout to specific, the log.check commit.
[16:05.410 --> 16:06.650]  And also you can see,
[16:06.650 --> 16:12.330]  oh, you can see an interesting file called env, right?
[16:12.350 --> 16:13.330]  So there you go.
[16:13.330 --> 16:16.810]  So you can see there is the access key and secret key
[16:16.810 --> 16:19.050]  for the AWS has mistakenly committed.
[16:19.090 --> 16:22.930]  And you can also see there is the kdiskcode flag is there.
[16:22.930 --> 16:24.890]  So basically this means that you have reached
[16:24.890 --> 16:26.610]  the end of this scenario,
[16:26.610 --> 16:29.730]  but in real world you can use this AWS access key,
[16:29.730 --> 16:32.070]  secret key and get the session token
[16:32.070 --> 16:35.910]  and you can do post-exploitation and go along beyond.
[16:36.230 --> 16:38.890]  So these scenarios are just we have taken
[16:38.890 --> 16:42.750]  from what we see in real world as a pen testing engagement.
[16:42.750 --> 16:45.450]  And some of them are like in a common scenarios
[16:45.450 --> 16:47.910]  which we see in real world, right?
[16:47.910 --> 16:50.190]  So this may not be possible
[16:50.190 --> 16:53.130]  and this may be possible in different use cases.
[16:53.130 --> 16:54.770]  So feel free to give it a try
[16:54.770 --> 16:58.130]  and understand what exactly we are trying to achieve here.
[16:58.150 --> 17:01.730]  So basically what will happen is in the same use case.
[17:01.810 --> 17:03.710]  So we can try in different method.
[17:03.710 --> 17:05.790]  Like what I'm trying to do is most of the time
[17:05.790 --> 17:08.110]  developers mistakenly build a Docker images
[17:08.610 --> 17:13.150]  which may be committed with the .git files
[17:13.150 --> 17:15.550]  because they don't ignore these files
[17:15.550 --> 17:18.070]  when it is creating into Docker container,
[17:18.070 --> 17:19.930]  maybe Docker ignore or something.
[17:19.930 --> 17:23.170]  So which might end up pushing this data
[17:23.170 --> 17:24.890]  into the containers as well.
[17:24.890 --> 17:26.930]  So looking at these kinds of sensitive information
[17:27.530 --> 17:30.030]  is very useful in the container.
[17:30.030 --> 17:31.670]  Like the another methodology you can try,
[17:31.670 --> 17:33.030]  as I said, explanation.
[17:33.030 --> 17:36.030]  So you can get the pod name, you can exec into the pod.
[17:36.030 --> 17:37.730]  This is like another method.
[17:38.370 --> 17:41.030]  Now we can also use tools like TruffleHog.
[17:41.030 --> 17:43.670]  So TruffleHog, I think most of you know,
[17:43.670 --> 17:48.450]  this is a tool which is basically finds high entropy
[17:48.450 --> 17:51.570]  or sensitive secrets in a Git repositories.
[17:51.570 --> 17:53.350]  So now I'm trying to do TruffleHog
[17:53.350 --> 17:56.010]  and given the current directory
[17:56.010 --> 17:58.150]  is the where Git repository is there.
[17:58.150 --> 18:00.510]  So what it does is it will check through all the commits
[18:00.510 --> 18:02.870]  and finds, is there any high entropy values
[18:02.870 --> 18:05.870]  or like sensitive secrets found
[18:05.870 --> 18:09.790]  based on the regular expressions and return to the studio.
[18:09.930 --> 18:11.790]  So you can include these kinds of tools
[18:11.790 --> 18:13.030]  in your CI, CD systems
[18:13.030 --> 18:15.190]  to detect these kinds of vulnerabilities.
[18:15.190 --> 18:16.690]  But in attacker's point of view,
[18:16.690 --> 18:18.770]  if you find any of the code basis
[18:18.770 --> 18:20.230]  which has Git or something,
[18:20.230 --> 18:22.590]  maybe it's a really definitely good way to check out
[18:22.590 --> 18:25.010]  in especially the container environments,
[18:25.010 --> 18:27.450]  you might end up finding some kind of sensitive approach.
[18:28.410 --> 18:30.230]  Like you can see the same thing.
[18:31.610 --> 18:32.410]  Cool.
[18:32.430 --> 18:34.550]  So this is our first scenario,
[18:34.550 --> 18:37.630]  which we just tried from the Kubernetes code.
[18:37.630 --> 18:40.270]  And similarly, as I said, you can try out here as well.
[18:40.270 --> 18:42.850]  So what you can do is you can look at here
[18:43.410 --> 18:47.110]  and if you wanted to specifically browse specific code,
[18:47.110 --> 18:48.470]  like one, two, three, zero,
[18:48.470 --> 18:49.930]  in the custom code tab,
[18:49.930 --> 18:52.270]  you can just open in a new browser window also.
[18:52.270 --> 18:53.750]  If you press this icon,
[18:53.750 --> 18:54.870]  then you can just say,
[18:54.870 --> 18:56.710]  I wanted to browse one, two, three, zero code.
[18:56.710 --> 18:58.170]  So you can able to browse the same thing
[18:58.170 --> 19:01.070]  and play it in online yourself for free
[19:01.070 --> 19:04.750]  and understand what exactly you can assess
[19:04.750 --> 19:06.390]  or test for Kubernetes security.
[19:08.090 --> 19:08.890]  Cool.
[19:09.170 --> 19:12.990]  So that's kind of one of the simple scenario.
[19:12.990 --> 19:15.350]  So let's go ahead and see what could be
[19:15.350 --> 19:16.790]  the some of the complex scenarios
[19:16.790 --> 19:18.410]  or different kinds of vulnerabilities
[19:18.970 --> 19:21.950]  you see in Kubernetes security assessment.
[19:22.290 --> 19:26.490]  So let's go back to the some maybe a little down,
[19:26.490 --> 19:29.290]  especially like attacking private registries
[19:29.290 --> 19:32.330]  or Kubernetes CS benchmarks or container escape.
[19:32.750 --> 19:34.490]  So let me go and showcase you
[19:34.490 --> 19:37.430]  how container escape to host system works.
[19:37.770 --> 19:41.510]  To browse to this scenario, basically you navigate here.
[19:42.610 --> 19:45.510]  So this is basically one of the pod
[19:46.270 --> 19:47.850]  inside Kubernetes cluster running
[19:47.850 --> 19:51.910]  with some specific privileged access has, right?
[19:51.930 --> 19:54.330]  So to set the context here,
[19:54.330 --> 19:57.570]  most of the times, especially operations teams
[19:57.570 --> 20:00.630]  may tend to require to run specific containers
[20:00.630 --> 20:02.050]  with high privileges,
[20:02.050 --> 20:05.070]  maybe giving gaps is admin or host privileges
[20:05.070 --> 20:07.570]  or privilege context,
[20:07.570 --> 20:10.150]  which is a full admin, sysadmin capabilities.
[20:10.150 --> 20:11.530]  So these are things.
[20:11.530 --> 20:13.850]  And also to just give an example,
[20:13.850 --> 20:15.930]  how we found in a real world,
[20:15.930 --> 20:18.250]  I have seen, especially while we performing
[20:18.430 --> 20:20.250]  a penetration testing engagement.
[20:20.270 --> 20:22.710]  So most of the developers get an access
[20:22.710 --> 20:25.750]  to completely one namespace with a full privilege.
[20:25.750 --> 20:29.510]  So if your clusters doesn't have PSPs,
[20:29.510 --> 20:31.750]  which is nothing but pod security policies,
[20:31.750 --> 20:35.190]  developers can basically has full access to that namespace.
[20:35.190 --> 20:36.330]  Whatever they can do,
[20:36.330 --> 20:40.910]  they can deploy a pod with specific privileges,
[20:40.910 --> 20:44.250]  which they can gain into the whole system access, right?
[20:44.250 --> 20:47.450]  So also this is one of the use case
[20:47.450 --> 20:49.350]  we have taken to simulate or showcase
[20:49.350 --> 20:52.830]  how you can find these kind of container escape
[20:52.830 --> 20:55.150]  and gain access to the whole system, right?
[20:56.310 --> 20:56.910]  Cool.
[20:56.910 --> 20:58.490]  So what we are trying to do here is
[20:59.130 --> 21:01.770]  we can try and look for some bunch of information.
[21:01.770 --> 21:04.170]  Most of the time, look for a printenv.
[21:04.270 --> 21:07.030]  Sorry, I think I typed print.
[21:09.330 --> 21:10.290]  Cool.
[21:10.290 --> 21:11.590]  So most of the time,
[21:11.590 --> 21:15.690]  you might see some kind of stuff here as well in printenv.
[21:15.690 --> 21:17.830]  And also I'm checking for mount.
[21:17.830 --> 21:19.590]  Is there any mounted directories?
[21:19.590 --> 21:21.450]  Now you can see there is a mounted directory
[21:21.450 --> 21:26.330]  from host system and which is into the container,
[21:26.330 --> 21:27.850]  which means basically in the pod,
[21:27.850 --> 21:29.730]  in this Kubernetes case, right?
[21:29.730 --> 21:31.290]  So now what I can do is,
[21:32.930 --> 21:34.130]  I think,
[21:35.690 --> 21:39.610]  let's look at what exactly the host system is doing.
[21:39.610 --> 21:42.530]  So if you see, there is a host system mounted,
[21:42.530 --> 21:44.850]  like which is basically from the host system,
[21:44.850 --> 21:46.430]  to just give an example.
[21:46.510 --> 21:49.630]  So which is nothing but your underlying host system,
[21:49.630 --> 21:54.230]  which is node, is mounted into your pod as a volume, right?
[21:54.230 --> 21:58.510]  So due to the reason we have extra capabilities, right?
[21:58.510 --> 22:00.710]  Like this container has a bunch of extra capabilities
[22:00.710 --> 22:01.630]  as well.
[22:01.630 --> 22:04.070]  So using those capabilities,
[22:04.070 --> 22:06.610]  as an attacker, what we can do is,
[22:06.610 --> 22:10.590]  we can like try to gain some kind of access
[22:10.590 --> 22:11.670]  into host system.
[22:11.670 --> 22:14.510]  What you can do is, we can sage root on the host system
[22:14.510 --> 22:18.150]  because we have capability of the privilege.
[22:18.450 --> 22:20.790]  And now we can see that it is given
[22:20.790 --> 22:23.390]  that the host system sage root.
[22:23.390 --> 22:24.710]  And now what we can do is,
[22:24.710 --> 22:26.710]  we can perform the internal commands,
[22:26.710 --> 22:28.950]  which are specific to system, right?
[22:28.950 --> 22:30.930]  Like now you can see docker ps.
[22:31.370 --> 22:34.490]  So these are nothing but running inside the host system,
[22:34.490 --> 22:35.970]  which is not a container.
[22:35.970 --> 22:38.890]  So you went below one level from the pod
[22:39.310 --> 22:40.650]  to the host system.
[22:40.650 --> 22:43.250]  From there, you are executing commands, right?
[22:43.250 --> 22:48.210]  So the possible attack surface here you can do is,
[22:48.210 --> 22:50.590]  to do post exploitation might be basically
[22:51.130 --> 22:53.650]  using the host system configuration,
[22:53.650 --> 22:56.890]  which is nothing but node level Kubernetes configuration,
[22:56.890 --> 22:59.930]  to go and try attacking other nodes
[22:59.930 --> 23:02.270]  or to gain access to other nodes,
[23:02.730 --> 23:05.170]  which basically you're controlling the entire cluster
[23:05.170 --> 23:07.370]  by passing through, passing through, right?
[23:07.370 --> 23:09.590]  So for example, what you can do is,
[23:09.590 --> 23:12.090]  you can use the configuration of node,
[23:12.090 --> 23:16.370]  which is available in the cat var lib.
[23:21.580 --> 23:26.560]  So you can use the configuration of cat var lib quip config,
[23:26.560 --> 23:29.000]  which has the node level configuration.
[23:29.000 --> 23:30.440]  So you can use that configuration
[23:30.440 --> 23:33.180]  and perform the Kubernetes commands,
[23:33.180 --> 23:34.700]  which is not only available
[23:34.700 --> 23:37.380]  for the existing namespace specific,
[23:37.380 --> 23:39.720]  it can also perform at the node level,
[23:39.720 --> 23:42.600]  whatever the node has the service account,
[23:42.600 --> 23:44.200]  you can able to do that level.
[23:44.400 --> 23:47.020]  To do that, for example, you can do this command.
[23:49.760 --> 23:52.080]  A kubectl command is not available.
[23:52.080 --> 23:53.860]  Let's download kubectl.
[23:56.640 --> 24:00.260]  Let me download this kubectl here.
[24:02.100 --> 24:04.880]  It just takes some time.
[24:04.880 --> 24:06.440]  Hopefully it is faster.
[24:07.400 --> 24:08.160]  Yeah.
[24:12.190 --> 24:13.590]  Yep, there you go.
[24:13.590 --> 24:15.030]  So what we are trying to do is,
[24:15.030 --> 24:17.470]  we are using now kubectl command
[24:17.470 --> 24:19.690]  and passing the kube config flag,
[24:19.690 --> 24:21.990]  which is using the node configuration
[24:22.420 --> 24:25.410]  and getting all like whatever the all resources
[24:25.410 --> 24:28.610]  in namespace of kube system, right?
[24:28.650 --> 24:31.450]  Oops, I'm trying the same thing again.
[24:38.550 --> 24:44.250]  I have to do chmod plus x kubectl.
[24:46.090 --> 24:46.830]  Cool.
[24:46.830 --> 24:49.470]  Now you can see when I run kubectl kube config
[24:49.470 --> 24:51.210]  with the specific configuration
[24:51.710 --> 24:54.010]  and get all in kube system,
[24:54.010 --> 24:55.870]  it will go ahead and execute that command
[24:55.870 --> 24:57.770]  and get all the resources.
[24:57.770 --> 24:59.270]  Now you can see here,
[24:59.270 --> 25:00.950]  it can able to get all the ports
[25:00.950 --> 25:03.730]  which are not restricted to specific namespace,
[25:03.730 --> 25:06.830]  which is the developer or the user has.
[25:06.830 --> 25:08.530]  And you can able to explore more
[25:08.530 --> 25:11.730]  or access more resources and also services.
[25:11.970 --> 25:14.990]  And you can see some of the specific use cases,
[25:14.990 --> 25:16.990]  especially I think replication controllers
[25:16.990 --> 25:19.430]  and other jobs and secrets,
[25:19.430 --> 25:21.030]  you cannot able to access.
[25:21.230 --> 25:24.230]  So the node, whatever the access it has,
[25:24.230 --> 25:25.410]  the node configuration,
[25:25.410 --> 25:27.130]  you can able to perform same level.
[25:27.130 --> 25:29.210]  So as an attacker, you can do post-exploitation,
[25:29.210 --> 25:32.430]  basically deploy another port in a different node
[25:32.430 --> 25:35.090]  and you can able to pass to another node
[25:35.090 --> 25:36.730]  and again get access to that.
[25:36.730 --> 25:40.070]  So for that, you can just look for how many nodes you have,
[25:40.070 --> 25:42.410]  like currently two nodes, maybe you are in this node,
[25:42.410 --> 25:44.710]  then you can deploy another port in the different node
[25:44.710 --> 25:46.870]  and get again access to the different.
[25:46.870 --> 25:50.450]  So you can basically use this kind of hopping methodology
[25:50.850 --> 25:54.270]  and hop on to another node and gain complete cluster access.
[25:56.030 --> 26:00.190]  So this is mostly you will see if an organization
[26:00.190 --> 26:03.090]  or the company which is not running up
[26:03.090 --> 26:05.090]  the port security policies,
[26:05.090 --> 26:07.890]  you can able to perform these kind of attacks
[26:07.890 --> 26:11.590]  or escalations of the container escapes, right?
[26:11.730 --> 26:15.330]  So you might use BSPs as a defense
[26:15.330 --> 26:16.590]  for these kind of vulnerabilities
[26:16.590 --> 26:19.770]  and it would be an easy and simple way
[26:19.770 --> 26:22.750]  to fix this kind of exploitation.
[26:23.990 --> 26:25.030]  Cool.
[26:25.330 --> 26:28.450]  So let's go ahead and see another use case.
[26:28.450 --> 26:31.970]  I think we have some more time to see two other scenarios.
[26:31.970 --> 26:34.910]  Let's take an example of Kubernetes namespace bypass.
[26:36.770 --> 26:37.850]  Okay.
[26:38.690 --> 26:42.750]  I'm going here, Kubernetes namespace bypass.
[26:42.830 --> 26:43.970]  I don't know exactly.
[26:43.970 --> 26:44.590]  Yeah.
[26:44.590 --> 26:47.410]  So by default, Kubernetes uses flat networking scheme.
[26:47.410 --> 26:51.570]  I think this is one of the default feature of Kubernetes.
[26:51.690 --> 26:54.150]  So by default, Kubernetes uses flat networking scheme
[26:54.150 --> 26:57.770]  means I can able to talk to from any port to any port
[26:57.770 --> 26:59.670]  by default, there is no firewall rules
[26:59.670 --> 27:02.390]  which will happen, allow or deny.
[27:02.390 --> 27:04.050]  By default, anything can talk to anything
[27:04.050 --> 27:06.530]  within the cluster IP range, right?
[27:06.530 --> 27:09.210]  So which means that whenever someone,
[27:09.210 --> 27:11.710]  SRE or ops team or like security people says
[27:11.710 --> 27:14.190]  that there is a namespace we have created in the Kubernetes
[27:14.190 --> 27:16.810]  which means it's just a logical boundary.
[27:16.810 --> 27:18.670]  There is no way you literally segregating
[27:18.670 --> 27:20.430]  between the different namespace
[27:20.430 --> 27:24.010]  unless until apply some kind of security policies
[27:24.010 --> 27:27.350]  like NSPs and network security policies, right?
[27:27.350 --> 27:30.470]  So if you are testing a Kubernetes clusters
[27:30.470 --> 27:32.070]  where there was no NSPs
[27:32.070 --> 27:34.250]  and there was no restrictions between the namespace
[27:34.250 --> 27:37.670]  as attacker, you can able to talk to other namespaces
[27:37.670 --> 27:39.850]  and other services within the cluster
[27:40.470 --> 27:43.070]  by using a standard network security tools
[27:43.070 --> 27:44.350]  like which we use day to day,
[27:44.350 --> 27:48.050]  like Nmap or Zmap or a bunch of tools, right?
[27:48.050 --> 27:50.190]  So to just get started with the scenario,
[27:50.190 --> 27:52.810]  I have created something, an example.
[27:52.830 --> 27:53.930]  So what we are trying to do is
[27:53.930 --> 27:55.830]  we are running something called Hacker Container
[27:55.830 --> 27:59.050]  which has again created by me.
[27:59.050 --> 28:03.210]  I have while performing bunch of Kubernetes security testings
[28:03.210 --> 28:06.230]  I thought that there are so many tools we require
[28:06.230 --> 28:09.070]  especially focused on container environment.
[28:09.070 --> 28:11.110]  So I have created curated list of tools
[28:11.850 --> 28:13.390]  which are maybe 50 plus tools
[28:13.390 --> 28:15.550]  which will help you aid in
[28:15.550 --> 28:17.910]  while performing Kubernetes security assessment.
[28:17.910 --> 28:20.270]  So this container has those all tools
[28:20.270 --> 28:22.470]  like while you are performing Kubernetes security assessment
[28:22.470 --> 28:25.390]  and you can use this as like a Swiss Army knife
[28:25.390 --> 28:27.410]  in security assessment, right?
[28:27.410 --> 28:30.170]  And I have seen this has been very valuable
[28:30.170 --> 28:31.950]  while performing assessment.
[28:32.030 --> 28:33.810]  So definitely I'll try to create
[28:33.810 --> 28:35.610]  some kind of documentation out of it.
[28:35.610 --> 28:36.950]  Maybe hopefully you can go back
[28:36.950 --> 28:40.570]  and try out different use cases with Hacker Container.
[28:40.570 --> 28:43.670]  I think it's going and deploying there and cool.
[28:43.670 --> 28:44.570]  So we got a shell.
[28:44.570 --> 28:46.950]  So what we are doing is we are inside a pod
[28:47.590 --> 28:49.770]  in one of the default namespace.
[28:49.770 --> 28:51.730]  Like if you wanted to just see,
[28:51.730 --> 28:53.350]  let's go ahead and run some command
[28:53.350 --> 28:55.970]  called kubectl get pods.
[28:55.970 --> 28:58.270]  Now you can see Hacker Container
[28:58.270 --> 29:00.610]  which is running in a default namespace.
[29:00.670 --> 29:03.270]  And let's go ahead and see what else we can do
[29:03.270 --> 29:04.770]  from the shell, right?
[29:04.770 --> 29:07.410]  So as attacker, you are in one of the pod
[29:07.410 --> 29:10.530]  and you don't know what else you can do from there, right?
[29:10.750 --> 29:13.090]  So as I said, Hacker Container has a bunch of tools
[29:13.090 --> 29:15.430]  like Docker Bench Security, Kubebench, Linus
[29:15.430 --> 29:17.390]  and a bunch of other tools.
[29:17.470 --> 29:20.590]  So one interesting thing you can do especially is
[29:21.730 --> 29:24.050]  looking for the entire cluster range.
[29:24.050 --> 29:27.610]  As I said, by default, Kubernetes uses flat networking scheme.
[29:27.610 --> 29:29.870]  You can look at the entire cluster range, right?
[29:29.870 --> 29:31.190]  So you can do IP route.
[29:31.190 --> 29:34.890]  You can see this is using 10.0.0.0 network range.
[29:34.890 --> 29:37.310]  And also you can look for what is the current IP address
[29:37.310 --> 29:40.150]  and 10.0.0.0.12.
[29:40.150 --> 29:43.810]  So you can scan the entire network and identify services.
[29:43.810 --> 29:46.470]  So one thing I can confirm, I can assure
[29:46.470 --> 29:49.610]  is most of the times internally within the cluster
[29:49.610 --> 29:51.450]  as a microservices or services
[29:51.450 --> 29:54.290]  which are running, many of the developers
[29:54.290 --> 29:56.370]  or operations teams tend to forget
[29:56.370 --> 29:59.710]  or assume that it is only exposed to internal networks.
[29:59.710 --> 30:02.790]  They will never use authentications or authorizations
[30:02.790 --> 30:05.870]  and you might end up seeing some kind of bypasses
[30:05.870 --> 30:08.950]  which is exposed internally itself,
[30:08.950 --> 30:12.390]  especially Mongo services, Elasticsearch instances
[30:12.890 --> 30:14.690]  and also even some microservices
[30:14.690 --> 30:17.790]  because the API gateway terminates at the ingress level
[30:18.190 --> 30:20.250]  which means before exposing into the world
[30:20.250 --> 30:21.650]  like using load balancers,
[30:21.650 --> 30:24.390]  they will terminate at the ingress level.
[30:24.390 --> 30:26.630]  So you might end up seeing some kind of services
[30:26.630 --> 30:29.250]  especially the application services
[30:29.250 --> 30:32.050]  also don't have authentication authorization checks
[30:32.050 --> 30:33.590]  within the clusters.
[30:33.590 --> 30:36.550]  So this is a really great place to explore
[30:36.550 --> 30:38.450]  and do as much recon you can do
[30:38.450 --> 30:41.330]  and you can gain as much as sensitive information
[30:41.330 --> 30:42.850]  or services access.
[30:43.870 --> 30:44.730]  Cool.
[30:44.730 --> 30:47.710]  So what I'm trying to do is here I'm exploring same
[30:47.710 --> 30:50.430]  and what I did is I'm trying to use ZMap
[30:50.430 --> 30:52.990]  which is similar to Nmap.
[30:52.990 --> 30:55.530]  It's trying to do a scan for the entire cluster range
[30:55.530 --> 30:57.910]  which is to do a faster way.
[30:58.010 --> 31:01.190]  And I'm specifically looking for a port of 6379
[31:01.190 --> 31:03.350]  which is a Redis instance.
[31:03.370 --> 31:05.350]  So I assume that there might be a Redis running
[31:05.350 --> 31:07.610]  or like a 9200 you can also try
[31:07.610 --> 31:10.730]  which is Elasticsearch or Mongo or MySQL
[31:10.730 --> 31:12.810]  or any other services you can scan.
[31:12.810 --> 31:15.010]  It's like traditional security will apply here.
[31:15.250 --> 31:17.090]  Let's go ahead and run this ZMap
[31:17.090 --> 31:19.230]  and see what we can see.
[31:19.230 --> 31:20.950]  So, oops.
[31:21.170 --> 31:21.770]  Okay.
[31:21.770 --> 31:26.290]  So we have to have enable the internal network range
[31:27.390 --> 31:29.730]  allow within this list.
[31:29.730 --> 31:31.710]  Let's go ahead and do that.
[31:33.710 --> 31:37.750]  Let's comment this and run again.
[31:37.750 --> 31:38.930]  Sorry.
[31:40.970 --> 31:42.170]  Oh, okay.
[31:42.170 --> 31:46.290]  You have to do another thing which is specify the gateway
[31:46.290 --> 31:48.210]  where this ZMap is starting.
[31:48.370 --> 31:50.730]  Let's go ahead and do that as well.
[31:51.590 --> 31:53.110]  F1G capital.
[31:53.130 --> 31:53.710]  Okay.
[31:54.330 --> 31:56.410]  And what is the gateway?
[31:58.430 --> 32:00.490]  Yeah, this is the MAC address.
[32:01.050 --> 32:01.990]  Cool.
[32:02.450 --> 32:03.570]  So, yeah.
[32:03.570 --> 32:04.770]  So what it is doing,
[32:04.770 --> 32:07.250]  it is trying to scan the entire cluster range
[32:07.610 --> 32:09.610]  for the specific port 6379
[32:09.610 --> 32:13.650]  and results the output if any of the IPs running
[32:13.650 --> 32:15.450]  or services running or what is running
[32:15.450 --> 32:17.570]  with the specific port open,
[32:17.570 --> 32:19.790]  it'll save into the other results.csv.
[32:19.790 --> 32:20.710]  It take a while.
[32:20.710 --> 32:24.210]  I think it's pretty fast in this case,
[32:24.210 --> 32:27.850]  but this will take some more time to go through and scan.
[32:28.550 --> 32:30.410]  But as I said, in general,
[32:30.410 --> 32:33.750]  we have seen in real world a lot of services running,
[32:33.750 --> 32:35.270]  especially Elasticsearch instances
[32:35.270 --> 32:38.290]  where you can look for without authentication
[32:39.270 --> 32:41.690]  and where you can gain access to indices
[32:41.690 --> 32:43.450]  and the log information,
[32:43.450 --> 32:45.770]  especially audit logs and sensitive information
[32:45.770 --> 32:48.890]  and also Mongo instances without any authentication
[32:49.490 --> 32:51.750]  or also the microservices, right?
[32:51.750 --> 32:54.270]  So this is a pretty goldmine if you are inside a pod
[32:54.270 --> 32:55.750]  and without NSPs,
[32:55.750 --> 32:58.690]  network security policies applied within the cluster, right?
[32:59.670 --> 33:00.610]  Cool.
[33:00.610 --> 33:02.650]  So this will take a while.
[33:02.650 --> 33:05.250]  I think 76% and all stuff.
[33:05.530 --> 33:05.870]  Yeah.
[33:05.870 --> 33:07.070]  As I said, similarly,
[33:07.070 --> 33:09.750]  you can try out those vulnerabilities here
[33:09.750 --> 33:12.850]  and follow the guide if you wanted to do as well.
[33:12.850 --> 33:16.730]  And I think it's a simple way to try out
[33:16.730 --> 33:19.690]  this entire Katakoda environment within the browser.
[33:19.690 --> 33:23.370]  You don't need to set up the entire cluster itself locally.
[33:23.410 --> 33:24.470]  And also once you are done,
[33:24.470 --> 33:25.690]  you can just quit the browser.
[33:25.690 --> 33:26.350]  It's all done.
[33:26.350 --> 33:28.110]  You don't need to clean up the entire environment.
[33:29.290 --> 33:30.650]  And another disclaimer,
[33:30.650 --> 33:32.490]  I think I forgot to mention in the beginning,
[33:32.490 --> 33:34.790]  do not try to run or set up this Kubernetes code
[33:34.790 --> 33:36.810]  in your production environments or in your office
[33:37.250 --> 33:41.290]  because it has intentionally vulnerabilities introduced
[33:41.290 --> 33:43.670]  due to the application flaws or something.
[33:43.670 --> 33:45.990]  It is a vulnerable application,
[33:45.990 --> 33:49.310]  which may have gain access to your production
[33:49.310 --> 33:51.250]  or wherever you run these environments.
[33:51.470 --> 33:53.950]  And at least I try to take care of using,
[33:53.950 --> 33:56.170]  exposing the ports within to the localhost,
[33:56.170 --> 33:59.290]  but it's highly not recommended to run in production.
[33:59.390 --> 34:00.390]  Cool.
[34:00.770 --> 34:02.570]  We have ZMap completed.
[34:02.570 --> 34:05.090]  Let's go ahead and see what we got from the results.
[34:05.330 --> 34:09.000]  You can see that there is the IP,
[34:09.000 --> 34:12.460]  which has 6379 port is exposed within the cluster.
[34:12.640 --> 34:15.280]  And let's go ahead and see what we can get out of it.
[34:15.280 --> 34:16.500]  As you know, it's Redis.
[34:16.500 --> 34:20.540]  As I said, this hacker container has built-in tools
[34:20.540 --> 34:23.720]  to look for these kind of common attacks and all stuff.
[34:23.720 --> 34:25.420]  We can use Redis CLI,
[34:25.420 --> 34:28.060]  and I forgot to be honest, how to access.
[34:28.380 --> 34:30.700]  Yeah, I found H and the IP.
[34:30.760 --> 34:33.300]  And now you can see there is another integrated Redis
[34:33.300 --> 34:35.900]  running within the cluster you can able to access
[34:35.900 --> 34:38.860]  because there is no network security policies
[34:38.860 --> 34:41.620]  which is by default in the cluster,
[34:41.620 --> 34:45.660]  means you can able to access without any issues.
[34:45.660 --> 34:47.140]  I mean, so they don't have any allow list
[34:47.140 --> 34:49.500]  or denial list within the cluster.
[34:49.840 --> 34:53.500]  Now you can do enumeration standard like get key.
[34:53.980 --> 34:55.600]  You can see nothing.
[34:55.600 --> 34:55.920]  Okay.
[34:55.960 --> 34:56.680]  Keys.
[34:56.680 --> 34:58.900]  You can see the keys, what are available,
[34:58.900 --> 35:03.820]  and you can see there is a key and get secret stuff.
[35:04.800 --> 35:05.540]  Cool.
[35:05.540 --> 35:06.740]  So you reach the flag.
[35:06.740 --> 35:09.800]  That means you are done with the challenge, right?
[35:09.800 --> 35:13.840]  So this is just an example of try to simulate the scenarios,
[35:13.840 --> 35:16.240]  but definitely you see these kind of vulnerabilities
[35:16.240 --> 35:17.660]  in a real world.
[35:17.660 --> 35:19.060]  Once you end up in a pod,
[35:19.060 --> 35:22.660]  try to look out for more services and try as much as you can
[35:22.660 --> 35:26.160]  and you will get most information out of it.
[35:26.160 --> 35:26.720]  Right.
[35:26.960 --> 35:27.940]  Cool.
[35:28.620 --> 35:31.140]  So let's go back and see another scenario
[35:31.140 --> 35:33.740]  before we conclude the presentation
[35:33.740 --> 35:35.360]  and moving into the next sections.
[35:36.320 --> 35:39.040]  Let's look at one of the common oldest,
[35:39.040 --> 35:41.080]  the vulnerabilities which we have seen
[35:42.000 --> 35:44.100]  exploiting the Helm or Tiller.
[35:46.320 --> 35:49.480]  So Helm is like a package manager for Kubernetes.
[35:49.480 --> 35:52.440]  So how we deploy application, install applications
[35:52.440 --> 35:54.300]  into, suppose you are using Debian-based
[35:54.300 --> 35:56.800]  apt-get install the package name.
[35:56.800 --> 35:59.980]  Similarly, developers or operations teams use Helm,
[35:59.980 --> 36:02.760]  which is a package manager to deploy applications
[36:02.760 --> 36:04.500]  into Kubernetes, right?
[36:04.500 --> 36:07.740]  So they will just do Helm install and the Nginx,
[36:07.740 --> 36:10.320]  it'll go and install the Nginx within the cluster, right?
[36:10.320 --> 36:14.180]  So by default, if you are testing any cluster
[36:14.180 --> 36:15.980]  and which is using version two,
[36:15.980 --> 36:17.340]  which is the older version of Helm
[36:17.340 --> 36:20.620]  and it used to be by default,
[36:20.620 --> 36:23.340]  most people using is Helm two in a previous cases,
[36:23.340 --> 36:26.460]  but they have raised three recently, like someone's back.
[36:26.600 --> 36:30.060]  So by default, Helm version two has RBAC setup
[36:30.060 --> 36:32.060]  with the full cluster admin access.
[36:32.060 --> 36:35.240]  So if you end up seeing these kind of Helm two configurations
[36:35.680 --> 36:36.660]  running within the cluster
[36:36.660 --> 36:39.020]  and there are no network security policies applied
[36:39.020 --> 36:41.680]  and it is running with the default setup,
[36:41.680 --> 36:44.560]  basically using this misconfiguration as an attacker,
[36:44.560 --> 36:46.600]  you can get entire cluster admin access
[36:46.600 --> 36:48.820]  and form the entire cluster, right?
[36:48.820 --> 36:50.880]  Let's go ahead and see how this scenario looks like.
[36:50.880 --> 36:53.720]  I'm going and running a simple pod
[36:53.720 --> 36:56.400]  and executing into that, right?
[36:57.720 --> 37:00.080]  Cool, let's go ahead and run this command.
[37:00.080 --> 37:02.980]  It'll go ahead and spin up another pod,
[37:02.980 --> 37:06.300]  it is called HelmTeller and give you a bash access, right?
[37:14.910 --> 37:16.890]  Yep, so now we are inside a pod
[37:16.890 --> 37:21.350]  and we assume that this cluster has Helm installed
[37:21.350 --> 37:24.150]  and we wanted to see how we can exploit
[37:24.150 --> 37:27.670]  this kind of vulnerability in a real world environment, right?
[37:27.670 --> 37:30.570]  So let's go back to our documentation, right?
[37:30.570 --> 37:33.390]  So by default in Kubernetes,
[37:33.390 --> 37:35.270]  there is a schema which they follow
[37:35.270 --> 37:39.010]  to access or basically it's like a DNS.
[37:39.190 --> 37:42.550]  Similarly, how we use, they use in the Kubernetes.
[37:42.550 --> 37:44.450]  So this is nothing but a service name,
[37:44.450 --> 37:48.270]  this is the namespace and they use either a service name
[37:48.270 --> 37:52.010]  and namespace and a cluster.local.
[37:52.170 --> 37:55.490]  So this is how the structure which is followed, right?
[37:55.490 --> 37:58.890]  So by default, if you have installed the Helm,
[37:58.890 --> 38:01.750]  which is Teller is get deployed in version two,
[38:01.750 --> 38:04.510]  which is nothing but a server side component of Helm.
[38:04.510 --> 38:06.030]  Helm is the client side version
[38:06.030 --> 38:07.930]  where you do Helm install or something.
[38:07.930 --> 38:09.790]  So the Teller is the server side component,
[38:09.790 --> 38:11.930]  which has a full cluster admin access, right?
[38:11.930 --> 38:14.170]  So now we are doing Teller deploy is the service name
[38:14.170 --> 38:17.530]  and it is by default deployed in a kube system.
[38:17.530 --> 38:19.230]  So I'm trying to look for kube system
[38:19.670 --> 38:22.630]  and I'm taking Telnet and see if this port
[38:22.630 --> 38:25.390]  is able to access from the current port.
[38:25.390 --> 38:27.030]  So this is the default port,
[38:27.030 --> 38:29.630]  which is in a by default configurations
[38:29.630 --> 38:33.730]  exposed to 0.0.0, which means if you are in any one
[38:33.730 --> 38:35.610]  of the four port within the cluster,
[38:35.610 --> 38:38.470]  you can able to access the Teller deployment service, right?
[38:38.470 --> 38:42.350]  You can see, looks like you can able to access the Teller.
[38:42.350 --> 38:44.150]  So which means you can able to talk
[38:44.150 --> 38:45.370]  to the Teller deployment service
[38:45.370 --> 38:48.150]  because there were no NSPs and by default,
[38:48.150 --> 38:52.050]  it is exposed to 0.0.0, which is within the cluster, right?
[38:52.050 --> 38:54.110]  And let's see what we can do.
[38:54.150 --> 38:58.230]  So I have also installed a Helm
[38:58.770 --> 39:02.090]  within this same container to talk to the Helm.
[39:02.090 --> 39:05.010]  So in the Helm version two,
[39:05.010 --> 39:08.130]  you can specify the dash dash host flag,
[39:08.130 --> 39:11.450]  which means you can specify which Teller service
[39:11.450 --> 39:12.730]  you wanted to talk to.
[39:12.730 --> 39:14.830]  Now we know the Teller service name
[39:14.830 --> 39:17.150]  as well as the DNS basically,
[39:17.150 --> 39:19.550]  and we are specifying talk to this host
[39:19.550 --> 39:21.870]  and perform these operations, right?
[39:21.870 --> 39:24.150]  Let's go ahead and run this command.
[39:27.030 --> 39:29.410]  I'm trying to run Helm command
[39:29.410 --> 39:31.050]  and specifying the specific service
[39:31.530 --> 39:32.730]  and the port,
[39:32.730 --> 39:34.810]  and I'm just running Helm version command, right?
[39:34.810 --> 39:36.970]  Now you can see the client version,
[39:36.970 --> 39:39.150]  which is Helm, is 2.16.7,
[39:39.150 --> 39:42.350]  and the server version, which is Teller, is 2.16.9.
[39:42.350 --> 39:44.370]  So I can able to talk to the Helm service
[39:44.370 --> 39:47.050]  and I can also see what all charts deployed,
[39:47.050 --> 39:48.070]  Helm charts, right?
[39:48.070 --> 39:50.690]  Helm-less, like you can see metadata DB is deployed.
[39:50.690 --> 39:53.330]  So you can create your own chart and apply as well, right?
[39:53.330 --> 39:55.850]  So what we have done is, as an attacker,
[39:55.850 --> 39:57.530]  simply to showcase the complexity
[39:57.530 --> 39:59.810]  or what you can do as an attacker,
[39:59.810 --> 40:01.810]  let's go ahead and run a simple command.
[40:02.190 --> 40:04.430]  By default, Kubernetes stores,
[40:04.950 --> 40:06.130]  has a service account,
[40:06.130 --> 40:08.070]  which is stored in var lib,
[40:08.070 --> 40:10.210]  service secret,
[40:18.690 --> 40:19.730]  one,
[40:19.730 --> 40:21.670]  yeah, secrets,
[40:22.230 --> 40:24.770]  Kubernetes IOS service account,
[40:24.770 --> 40:25.570]  namespace,
[40:25.570 --> 40:27.690]  because I'm running in default namespace,
[40:27.690 --> 40:31.470]  and this is the token for that specific service account,
[40:31.470 --> 40:33.010]  which is mapped to the current port.
[40:33.010 --> 40:35.750]  So what I'm trying to do now here is,
[40:35.750 --> 40:37.270]  we have created a Helm chart,
[40:37.270 --> 40:38.470]  which is one chart,
[40:38.470 --> 40:42.290]  and this I have taken from the Bitnami repository research,
[40:42.290 --> 40:45.590]  and I have given the reference here,
[40:45.590 --> 40:47.310]  and go and check more resource.
[40:47.310 --> 40:49.010]  What it does is basically,
[40:49.010 --> 40:50.470]  it's a simple Helm chart.
[40:50.470 --> 40:52.150]  What it is do is,
[40:52.150 --> 40:53.630]  it has a bunch of values,
[40:53.630 --> 40:55.530]  which is nothing but namespace is default,
[40:55.530 --> 40:57.250]  and the name of the,
[40:57.250 --> 40:58.170]  is default.
[40:58.170 --> 41:00.470]  Let's look at the templates, how it looks like.
[41:01.810 --> 41:02.510]  Templates,
[41:03.210 --> 41:04.950]  and a cluster role.
[41:04.950 --> 41:06.550]  It has a cluster role,
[41:06.550 --> 41:07.630]  which is nothing but,
[41:08.390 --> 41:11.910]  it gives all API groups in the Kubernetes,
[41:11.910 --> 41:13.550]  for all the resources,
[41:13.550 --> 41:14.570]  and all the verbs,
[41:14.570 --> 41:16.050]  which means you can do get,
[41:16.050 --> 41:16.650]  post,
[41:16.650 --> 41:17.670]  delete,
[41:17.670 --> 41:18.210]  list,
[41:18.210 --> 41:19.530]  all of the resources,
[41:19.530 --> 41:22.070]  for any of the resources in the Kubernetes cluster,
[41:22.070 --> 41:24.130]  any of the kind of API groups,
[41:24.130 --> 41:25.490]  you can perform.
[41:25.550 --> 41:27.590]  So what we are trying to do is,
[41:27.590 --> 41:28.330]  templates,
[41:28.330 --> 41:29.930]  cluster role binding.
[41:30.090 --> 41:32.630]  So we are trying to bind this cluster role,
[41:32.630 --> 41:34.790]  which is entered across the cluster,
[41:34.790 --> 41:36.210]  to the service account,
[41:36.210 --> 41:38.050]  which is taken, if you've seen,
[41:38.050 --> 41:39.810]  which is namespace is default,
[41:39.810 --> 41:40.630]  because we are in the,
[41:40.630 --> 41:42.910]  this pod is currently running in the default namespace,
[41:42.910 --> 41:44.430]  and the service account is default,
[41:44.430 --> 41:45.750]  because you see here,
[41:45.750 --> 41:47.170]  it's the service account is default,
[41:47.170 --> 41:48.090]  when you do,
[41:48.090 --> 41:49.650]  slash var run,
[41:50.310 --> 41:51.910]  secret service account.
[41:51.930 --> 41:53.150]  So what we are trying to do is,
[41:53.150 --> 41:54.870]  we are creating a new cluster role,
[41:54.870 --> 41:56.670]  which gives full cluster access,
[41:56.670 --> 41:58.770]  to perform anything on all resources,
[41:58.770 --> 42:00.470]  and binding that cluster role,
[42:00.470 --> 42:02.030]  to the default service account.
[42:02.030 --> 42:04.510]  So now, as you can't get the secrets,
[42:04.510 --> 42:06.150]  or any operations,
[42:06.150 --> 42:08.370]  within the cluster using the default account,
[42:08.370 --> 42:10.110]  once we deploy this Helm chart,
[42:10.110 --> 42:12.510]  you can do anything as a cluster admin, right?
[42:12.510 --> 42:14.070]  Let's go ahead and run this,
[42:14.070 --> 42:15.650]  deploy this Helm chart.
[42:15.710 --> 42:17.330]  So we are trying to do Helm,
[42:17.330 --> 42:19.490]  and we are using the same Tiller configuration,
[42:19.490 --> 42:21.210]  and deploying the Pawn chart, right?
[42:21.210 --> 42:22.230]  Let's go ahead and run.
[42:22.230 --> 42:24.690]  So it went ahead and deployed the Pawn chart,
[42:24.690 --> 42:28.130]  because Helm has the capability to talk to the Tiller,
[42:28.130 --> 42:29.910]  and Tiller has the full cluster admin access,
[42:29.910 --> 42:31.710]  so it can go ahead and deploy this chart.
[42:31.710 --> 42:35.130]  So now if you go ahead and run kubectl get secrets,
[42:35.130 --> 42:36.530]  you can able to get the secrets
[42:36.530 --> 42:38.650]  of the entire Kubernetes here, right?
[42:38.650 --> 42:42.430]  So you can also get kubectl get pods,
[42:42.430 --> 42:44.130]  you can able to get all the pods,
[42:44.130 --> 42:47.070]  and in all namespaces as well, right?
[42:47.070 --> 42:49.170]  So basically you have full cluster admin access,
[42:49.170 --> 42:51.310]  because currently whatever the service account,
[42:51.310 --> 42:53.770]  which is there in our pod,
[42:53.770 --> 42:56.790]  because due to the reason we have created a cluster role,
[42:56.790 --> 42:58.870]  this cluster admin, sorry,
[42:58.870 --> 43:01.990]  this pod has basically full cluster admin access,
[43:01.990 --> 43:03.870]  which can able to perform any kind of operations
[43:03.870 --> 43:05.630]  on all resources, right?
[43:05.630 --> 43:07.890]  So as a attacker, some of the useful commands
[43:08.730 --> 43:14.010]  is like look for auth, can I create pods?
[43:14.010 --> 43:16.450]  You can check if the current service account,
[43:16.450 --> 43:19.550]  which you own, if you anytime get a service account access,
[43:19.550 --> 43:23.070]  it can able to create pods or something, sorry, create.
[43:23.370 --> 43:26.230]  You can also check delayed pods,
[43:26.690 --> 43:28.930]  it can able to delay pods or something.
[43:28.930 --> 43:32.430]  You can also specify, can I do in namespace of kube system?
[43:32.650 --> 43:35.970]  So this is talk to the API and give you the information.
[43:36.170 --> 43:38.510]  So this command is pretty useful.
[43:38.510 --> 43:40.190]  If you got a service account and you wanted to see
[43:40.190 --> 43:41.510]  what kind of privilege it has,
[43:41.510 --> 43:43.390]  you can make these calls, right?
[43:43.390 --> 43:45.090]  And also some tips, I don't know,
[43:45.090 --> 43:47.650]  we are about to conclude and let me,
[43:47.650 --> 43:49.750]  whenever you make these commands,
[43:49.750 --> 43:52.670]  you can also increase the verbosity of the Kubernetes,
[43:52.670 --> 43:54.650]  you can increase the six or seven,
[43:54.650 --> 43:56.590]  you can able to see each and every request
[43:56.590 --> 43:58.490]  it is making to the Kubernetes cluster
[43:58.490 --> 44:00.630]  and in a verbose way.
[44:00.630 --> 44:01.790]  Like if you increase this,
[44:01.790 --> 44:04.930]  you can able to see much better, even like headers.
[44:04.930 --> 44:06.610]  And if we increase like height,
[44:06.610 --> 44:09.310]  like you can even see like a response body and everything.
[44:09.450 --> 44:12.910]  And if you increase to nine and like little more depth,
[44:12.910 --> 44:14.150]  it will keep going, right?
[44:14.150 --> 44:16.370]  So this is completely API driven.
[44:16.370 --> 44:18.470]  So you can able to play around with the verbose
[44:18.470 --> 44:20.670]  and understand exactly each and every thing
[44:20.670 --> 44:22.030]  which is being answered.
[44:22.110 --> 44:24.110]  And also very useful command is explain.
[44:24.110 --> 44:26.330]  If you wanted to understand a pod,
[44:26.330 --> 44:28.430]  then it will give you more details about it.
[44:28.430 --> 44:30.590]  And I wanted to learn more about pod spec,
[44:30.590 --> 44:34.090]  you can also do kubectl spec.
[44:34.130 --> 44:35.550]  So this is very handy.
[44:35.550 --> 44:37.650]  And while you are performing assessments,
[44:37.650 --> 44:39.090]  you can quickly look at the documentation
[44:39.090 --> 44:41.410]  rather like explain commands, right?
[44:42.030 --> 44:43.590]  Cool, so now you see,
[44:43.590 --> 44:46.710]  due to the simple default misconfiguration
[44:46.710 --> 44:47.810]  of our Helm pillar,
[44:47.810 --> 44:50.010]  we can able to control the entire cluster.
[44:50.050 --> 44:53.750]  And this vulnerability I have seen in so many of the cluster
[44:53.750 --> 44:55.750]  in real-time pentesting assessments
[44:55.750 --> 44:59.750]  due to the administrator doesn't configure properly.
[44:59.790 --> 45:03.050]  And this is default security installation by Helm itself.
[45:03.110 --> 45:05.910]  So the newer version of Helm v3 fixed this vulnerability,
[45:05.910 --> 45:07.450]  so it kind of patched.
[45:08.770 --> 45:11.230]  Cool, so let me go with the last scenario.
[45:11.230 --> 45:12.950]  I think then we will kind of conclude.
[45:12.950 --> 45:14.570]  And this you can try, it's a DIND,
[45:14.570 --> 45:16.070]  Docker in Docker and Exploitation.
[45:16.070 --> 45:18.910]  This you commonly see in CI-CD systems,
[45:18.910 --> 45:20.330]  especially in build pipelines,
[45:20.330 --> 45:22.310]  like GitLab CI-CD or Bitbucket CI-CD
[45:22.310 --> 45:25.090]  or in real-world where they pass the socket,
[45:25.090 --> 45:26.050]  our socket, right?
[45:26.050 --> 45:27.250]  You can go ahead and try.
[45:27.250 --> 45:28.390]  And SSRF vulnerability,
[45:28.390 --> 45:30.090]  especially this is very widely exploited
[45:30.550 --> 45:32.050]  in the cloud infrastructure.
[45:32.050 --> 45:34.150]  But in coming to the Kubernetes context,
[45:34.150 --> 45:36.890]  it is little much, you can get more out of it.
[45:36.890 --> 45:38.890]  Because as I said, within the cluster,
[45:38.890 --> 45:42.030]  there might not be some kind of services authenticated.
[45:42.030 --> 45:43.510]  There might be services which are giving
[45:43.510 --> 45:45.230]  more health information of the nodes
[45:45.230 --> 45:47.910]  as well as the ports or internal services.
[45:47.910 --> 45:50.770]  Using SSRF, you can get more information out of it
[45:50.770 --> 45:54.170]  within the clusters compared to the cloud native,
[45:54.170 --> 45:56.490]  cloud way of exploiting, right?
[45:57.290 --> 45:58.210]  Cool.
[45:58.210 --> 46:00.890]  So let me just complete with the...
[46:02.490 --> 46:03.930]  Yeah, this looks fun.
[46:03.930 --> 46:04.470]  Cool.
[46:04.470 --> 46:07.010]  So this is the use case where basically
[46:07.010 --> 46:08.330]  I'm trying to showcase,
[46:08.330 --> 46:10.270]  if you end up in any of the pod
[46:10.270 --> 46:12.790]  and the developer or operations teams
[46:12.790 --> 46:15.410]  didn't set up any resource consumptions,
[46:15.410 --> 46:18.370]  like you can only consume these many resources, right?
[46:18.370 --> 46:21.290]  So you can able to use utilities,
[46:21.290 --> 46:24.330]  especially like I'm trying to showcase here, stress-ng.
[46:24.890 --> 46:27.630]  So let me showcase a simple example.
[46:27.870 --> 46:30.190]  Let me show in the terminal, maybe.
[46:30.690 --> 46:35.830]  Exit, kubectl get pods.
[46:36.050 --> 46:38.210]  Now you see a bunch of pods are running.
[46:38.210 --> 46:40.210]  So I'm looking for a specific pod.
[46:40.210 --> 46:46.550]  Let's look for that kubectl top pod.
[46:47.290 --> 46:50.410]  So when you do top, it's basically a system like top command.
[46:50.410 --> 46:52.090]  It shows how much CPU resources
[46:52.090 --> 46:54.750]  and memory resources it is using.
[46:54.810 --> 46:57.350]  And if any of the deployment or manifest
[46:57.350 --> 47:00.310]  which doesn't have specifically resource limit setup
[47:00.310 --> 47:03.450]  and the cluster is deployed in like cloud infrastructure,
[47:03.450 --> 47:05.130]  like which has auto-scaling enabled.
[47:05.130 --> 47:06.390]  If you re-hit this limit,
[47:06.390 --> 47:07.750]  it'll go and scale up a new node
[47:07.750 --> 47:11.190]  or deploys like some more nodes into the cluster, right?
[47:11.210 --> 47:12.990]  So that times you can exploit this
[47:12.990 --> 47:15.250]  in an attacker's point of view.
[47:15.250 --> 47:16.770]  Like let's go ahead and run a simple tool.
[47:16.970 --> 47:20.390]  I'm running with two kicks and 30 seconds.
[47:20.530 --> 47:22.970]  So now you can see if you're on the same command
[47:23.590 --> 47:28.290]  from the specific resources of 3MB, 1MB, 2MB,
[47:28.290 --> 47:30.290]  like increasing so much, right?
[47:30.290 --> 47:31.890]  I think it's just taking some time.
[47:31.890 --> 47:33.570]  Let's keep looking.
[47:45.330 --> 47:48.850]  You can see it hits a crazy amount of memory resource limits
[47:48.850 --> 47:53.450]  because there was no resource limit set for this deployment,
[47:53.450 --> 47:54.930]  whatever the hunger check,
[47:54.930 --> 47:57.130]  it will be deployed as part of the Kubernetes code.
[47:57.490 --> 47:59.710]  Attacker can basically consume more CPU resources
[47:59.710 --> 48:01.870]  and memory, which basically cost the billing
[48:01.870 --> 48:03.850]  and very like cost-effective
[48:03.850 --> 48:06.550]  for the company's perspective, right?
[48:06.550 --> 48:07.230]  Billing.
[48:07.230 --> 48:08.490]  And due to the reason,
[48:08.490 --> 48:10.410]  the most of the modern infrastructure is set up
[48:10.410 --> 48:11.950]  in such a way which is auto-scalable
[48:11.950 --> 48:16.110]  and like scalable based on the certain parameters.
[48:16.230 --> 48:17.870]  So you've been able to basically DDoS
[48:17.870 --> 48:19.990]  or gain more resource consumption
[48:19.990 --> 48:24.030]  for the companies and organizations, right?
[48:24.030 --> 48:26.590]  So this is definitely one thing.
[48:26.590 --> 48:29.350]  So before moving in,
[48:29.350 --> 48:32.550]  let me just conclude with the Hacker Container preview.
[48:32.910 --> 48:35.030]  As I said, Hacker Container,
[48:35.030 --> 48:39.330]  let me just showcase maybe here, Hacker Container.
[48:39.330 --> 48:41.330]  So Hacker Container is a similar,
[48:41.330 --> 48:44.310]  another utility I created as part of the assessment.
[48:44.310 --> 48:46.570]  So you can read more about it here, the blog post.
[48:46.570 --> 48:48.430]  So this has a bunch of utilities,
[48:48.430 --> 48:49.930]  which you use day-to-day,
[48:49.930 --> 48:52.950]  like while performing assessments in the Kubernetes cluster.
[48:52.950 --> 48:55.490]  So I have used it extensively,
[48:55.490 --> 48:57.370]  I would say, while performing assessment
[48:57.370 --> 48:59.390]  and it is very useful utility.
[48:59.390 --> 49:00.710]  Let's go ahead and run.
[49:01.210 --> 49:02.690]  Oops, looks like it's already there.
[49:02.690 --> 49:06.730]  Let's exec into the pod, Hacker Container.
[49:06.810 --> 49:13.110]  So we are inside the pod,
[49:13.110 --> 49:17.210]  like you see a bunch of other stuff like default tools.
[49:17.210 --> 49:20.150]  So some of the things like it has is amicontain,
[49:20.150 --> 49:22.590]  so which is again, utility written by Jessica Frazel.
[49:22.590 --> 49:25.170]  She has done quite a lot of work for the containers
[49:25.170 --> 49:27.610]  and security community in the Kubernetes
[49:28.050 --> 49:29.610]  and the container ecosystem.
[49:29.710 --> 49:32.030]  So what it does is it will try to identify
[49:32.030 --> 49:33.650]  what container runtime you are using
[49:33.650 --> 49:37.290]  and do you have shared namespaces from the host system
[49:37.290 --> 49:39.330]  like PID, username space,
[49:39.330 --> 49:41.450]  or is there any app armor or sitcom profile
[49:41.450 --> 49:44.630]  applied to the container to restrict certain capabilities?
[49:44.810 --> 49:47.390]  And is that what all capabilities allowed currently,
[49:47.390 --> 49:48.670]  system calls allowed,
[49:48.670 --> 49:51.490]  and it kind of a container introspection tool.
[49:51.490 --> 49:54.330]  So which will help you to perform while in attacks
[49:54.330 --> 49:56.150]  and security assessments.
[49:56.150 --> 49:58.810]  So not only I think specific to the container Kubernetes,
[49:58.810 --> 50:01.510]  then it's a pretty useful tool to try out, right?
[50:01.510 --> 50:03.170]  So you can also do the same thing
[50:03.170 --> 50:06.190]  with proc self cgroup maybe.
[50:06.190 --> 50:07.090]  Yeah, cool.
[50:07.090 --> 50:09.470]  So you can see this is basically trying to figure out
[50:09.470 --> 50:13.010]  kubeports and you can also do capsh, print.
[50:13.010 --> 50:15.230]  So it will look for what all capabilities are there
[50:15.230 --> 50:16.750]  and similarly.
[50:16.810 --> 50:20.190]  So this is kind of pretty useful tool, the Hacker Container.
[50:20.710 --> 50:25.130]  And also it has Nikto and a bunch of other use cases.
[50:25.130 --> 50:28.430]  I think maybe I try to not document it here.
[50:28.430 --> 50:29.950]  So I'm going to try to see
[50:29.950 --> 50:32.150]  what can I kind of create out of it
[50:32.150 --> 50:35.470]  and how I use in day to day while exploiting.
[50:35.470 --> 50:38.530]  And some use cases I would say is very useful
[50:38.530 --> 50:40.250]  is running benchmarks.
[50:40.250 --> 50:41.690]  If you are inside the clusters
[50:41.690 --> 50:44.510]  and you wanted to just get a glance of view,
[50:44.510 --> 50:46.230]  what kind of vulnerabilities possibly there
[50:46.230 --> 50:48.210]  within the Kubernetes clusters,
[50:48.210 --> 50:49.610]  both in terms of container runtime
[50:49.610 --> 50:51.310]  as well as the cluster level.
[50:51.310 --> 50:53.570]  You can run these CIS benchmarks.
[50:53.630 --> 50:56.970]  Like one is you can just go ahead and run this command.
[50:57.130 --> 50:58.470]  Let's go ahead and run.
[50:58.510 --> 50:58.890]  Yep.
[50:58.890 --> 51:00.070]  So what it does is it,
[51:00.070 --> 51:01.970]  I have created something called daemonset,
[51:01.970 --> 51:03.850]  which means each node,
[51:03.850 --> 51:06.250]  how many ever nodes you have in the cluster,
[51:06.250 --> 51:09.050]  will get one pod running in all the nodes.
[51:09.050 --> 51:11.550]  That is called daemonset in Kubernetes.
[51:11.650 --> 51:12.730]  Let's go ahead and get the node.
[51:12.730 --> 51:14.730]  So you can see two nodes are available.
[51:14.730 --> 51:16.650]  Now you should be able to see two pods running
[51:16.650 --> 51:19.570]  in the Docker Bench because there are two nodes
[51:19.570 --> 51:22.030]  because we are running a daemonset, right?
[51:22.030 --> 51:23.610]  So you can see two pods running.
[51:23.610 --> 51:27.170]  So let's go ahead and exec into one of the pod
[51:27.170 --> 51:30.950]  and see what we can do with Docker Bench Security, right?
[51:30.950 --> 51:32.870]  So Docker Bench Security,
[51:35.650 --> 51:37.650]  I'm execing into the pod.
[51:38.390 --> 51:41.570]  Now you can see Docker Bench Security.
[51:41.570 --> 51:44.650]  Now you can go ahead and run dockerbenchsecurity.css.
[51:44.650 --> 51:46.710]  This is also part of a Hacker Container.
[51:46.710 --> 51:48.030]  You can go ahead and run this.
[51:48.030 --> 51:50.930]  So the way I have set it up,
[51:50.930 --> 51:57.650]  sorry, the scenario manifest is if you can see here,
[51:59.980 --> 52:03.840]  I have created the Docker Bench Security in such a way
[52:03.840 --> 52:07.480]  it access the host PID, IPC network,
[52:07.480 --> 52:09.740]  as well as a bunch of volumes.
[52:09.740 --> 52:12.260]  And it also had extra capability called audit control
[52:12.260 --> 52:13.820]  so that you can access and talk
[52:13.820 --> 52:16.000]  to your underlying Docker Container runtime
[52:16.000 --> 52:18.040]  and you can perform benchmark analysis
[52:18.040 --> 52:20.460]  on possible security issues, right?
[52:20.460 --> 52:23.240]  So this is a pretty simple utility.
[52:23.240 --> 52:24.720]  Just go ahead and run this.
[52:24.720 --> 52:25.720]  Once you are inside cluster,
[52:25.720 --> 52:27.020]  then you will get at least high level posture
[52:27.020 --> 52:29.400]  and use the whatever the found vulnerabilities
[52:29.400 --> 52:33.840]  or the checks as attack paths,
[52:33.840 --> 52:36.380]  or you can use those while you are doing assessments
[52:36.380 --> 52:38.140]  or post-exploitation, right?
[52:38.220 --> 52:39.840]  Once you are inside a node,
[52:39.840 --> 52:41.240]  maybe you can use some kind of vulnerability
[52:41.240 --> 52:44.220]  which is detected here to do some kind of post-exploitation.
[52:44.280 --> 52:46.360]  So you can see it does a bunch of checks
[52:46.360 --> 52:48.960]  like host level configuration checks
[52:48.960 --> 52:51.920]  and specific to like the Docker daemon
[52:52.400 --> 52:55.740]  or like directories and daemon configuration,
[52:55.740 --> 52:58.020]  and as well as the files,
[52:58.020 --> 52:59.980]  which is specific to the container runtime
[52:59.980 --> 53:01.620]  here in this case, Docker,
[53:01.620 --> 53:03.460]  and also container images,
[53:03.460 --> 53:05.060]  whether container running with a root privilege
[53:05.060 --> 53:07.140]  or it has any extra capability set
[53:07.140 --> 53:10.420]  or it doesn't have any seccomp or apparm profiles,
[53:10.420 --> 53:12.100]  these kinds of things, right?
[53:12.960 --> 53:16.340]  Similarly, you can go ahead and run the kubebench as well.
[53:16.810 --> 53:20.300]  You can go ahead and run this kubebench like this.
[53:21.440 --> 53:23.940]  Let me go ahead and run here only, I think.
[53:24.860 --> 53:27.400]  It's kind of taking quite a lot of time.
[53:29.560 --> 53:32.200]  So you can see kubebench I have run.
[53:32.260 --> 53:33.940]  You can give it pods.
[53:34.060 --> 53:36.740]  The kubebench is running here.
[53:36.740 --> 53:39.880]  You can get the logs, iPhone app, and the bench.
[53:39.940 --> 53:40.960]  So it will show.
[53:40.960 --> 53:42.520]  You can see there are 12 checks passed,
[53:42.520 --> 53:44.520]  nine checks failed, and two checks running.
[53:44.520 --> 53:45.980]  So you can see what all the vulnerabilities
[53:45.980 --> 53:47.780]  possibly there in this Kubernetes cluster
[53:47.780 --> 53:50.020]  and what would be the remediations.
[53:50.020 --> 53:51.320]  So you can use these vulnerabilities
[53:51.320 --> 53:53.360]  maybe to do further exploitation
[53:53.360 --> 53:55.820]  and understand what you can do, right?
[53:55.820 --> 53:59.600]  So these tools are very useful just to get a glance of it.
[53:59.600 --> 54:02.080]  And there are a bunch of utilities out there as well,
[54:02.080 --> 54:04.480]  like kubehunter and kubepsychaudit
[54:04.480 --> 54:08.100]  and the different golang tool written
[54:08.100 --> 54:09.500]  by different researchers.
[54:09.580 --> 54:13.700]  And I'll try to document and put it in the same scenarios.
[54:13.700 --> 54:15.640]  So hopefully you can go back and try out yourself.
[54:16.520 --> 54:17.240]  Cool.
[54:17.380 --> 54:19.420]  So there are a bunch of other scenarios.
[54:19.500 --> 54:20.560]  Due to the time constraint,
[54:20.560 --> 54:22.720]  I couldn't complete or showcase you,
[54:22.720 --> 54:26.280]  but feel free to go and check out the Kubernetes Go project.
[54:26.360 --> 54:28.380]  And you should be able to get and follow
[54:28.380 --> 54:32.580]  using the playground of that Katakoda, right?
[54:32.580 --> 54:35.160]  And if you have set it up in your own environment,
[54:35.160 --> 54:36.840]  like in our case, right, my case,
[54:36.840 --> 54:38.620]  so you can go ahead and tear down the cluster.
[54:38.620 --> 54:41.260]  Basically, it will try to delete possibly resources
[54:41.260 --> 54:42.380]  which I have created
[54:42.380 --> 54:44.860]  as part of the Kubernetes Go deployment.
[54:44.860 --> 54:47.040]  But I highly recommend you delete the cluster
[54:47.820 --> 54:51.640]  to just ensure cleanup of the entire environment, right?
[54:51.780 --> 54:55.820]  And I have also showcased and run using a tool called Chekhov
[54:55.820 --> 54:57.880]  to see what all possible vulnerabilities
[54:58.580 --> 55:01.600]  using a CACD as part of pipeline.
[55:01.600 --> 55:05.100]  So it found a bunch of vulnerabilities in the deployments
[55:05.100 --> 55:06.980]  and other things which I have used
[55:07.600 --> 55:09.900]  as part of Kubernetes Go as well.
[55:09.900 --> 55:10.700]  All right.
[55:11.180 --> 55:12.040]  Cool.
[55:12.040 --> 55:15.140]  So that's it pretty much about it.
[55:15.140 --> 55:19.860]  And I would highly recommend having your feedback
[55:19.860 --> 55:23.580]  and understand what exactly you want to look forward.
[55:23.580 --> 55:25.960]  And I'm really looking forward
[55:25.960 --> 55:27.360]  to building more and more scenarios
[55:27.920 --> 55:31.020]  and also adding the defense scenario.
[55:31.020 --> 55:34.000]  I'm really building like most of the scenarios
[55:34.000 --> 55:36.840]  and making them in such a way that how you can practice
[55:36.840 --> 55:41.220]  for the CNCF certified Kubernetes security.
[55:41.220 --> 55:42.820]  So you can use this playground to practice
[55:42.820 --> 55:45.760]  and understand how most of the Kubernetes vulnerabilities
[55:45.760 --> 55:47.940]  you can assess and learn more about it.
[55:47.940 --> 55:49.340]  So feel free to get involved
[55:49.340 --> 55:52.800]  and you can please feel free to provide the feedback
[55:52.800 --> 55:56.180]  and share with your colleagues, teammates, or your friends.
[55:56.180 --> 55:58.780]  I would really appreciate if you can share the feedback
[55:58.780 --> 56:01.180]  and I would love to improve and work on it.
[56:01.180 --> 56:03.000]  And if you see any documentation
[56:03.000 --> 56:05.540]  or anything needs to be improved,
[56:05.540 --> 56:07.680]  please let me know by GitHub issues.
[56:07.680 --> 56:09.860]  And I'm trying to be pretty active,
[56:09.860 --> 56:11.880]  but in the past month I was a little busy.
[56:11.880 --> 56:13.380]  So I kind of work on this,
[56:13.380 --> 56:15.440]  working on defense scenarios as well.
[56:15.440 --> 56:17.100]  Hopefully we will be releasing this month
[56:17.100 --> 56:19.400]  for the defense scenarios to understand
[56:19.400 --> 56:21.680]  and fix the vulnerabilities which you have learned
[56:21.680 --> 56:23.500]  while practicing these attacks
[56:23.500 --> 56:26.680]  and also more attacker scenarios as well.
[56:27.080 --> 56:31.240]  And with that, I think I'm pretty much done
[56:31.240 --> 56:32.500]  with the session.
[56:32.500 --> 56:34.820]  Thank you so much once again for having me
[56:34.820 --> 56:36.800]  and really looking forward.
[56:37.200 --> 56:38.440]  Awesome. Thank you so much.
[56:38.440 --> 56:39.920]  It was an amazing presentation.
[56:39.920 --> 56:41.140]  I learned so much.
[56:41.160 --> 56:42.960]  And as a matter of fact, in Discord,
[56:42.960 --> 56:44.260]  there are a lot of people actually talking
[56:44.260 --> 56:45.860]  about your presentation.
[56:45.940 --> 56:48.580]  And once again, thank you for supporting the communities.
[56:48.580 --> 56:51.040]  Thank you for supporting the DEF CON Red Team Village.
[56:51.360 --> 56:52.920]  We're going to go on a little break,
[56:52.920 --> 56:55.800]  but we invite everybody that will have any further questions
[56:55.800 --> 56:58.200]  and so on to join the Discord channel,
[56:58.200 --> 57:00.000]  which is in the description of the video.