WEBVTT 00:00.000 --> 00:05.000 your support for supporting the DEF CON community and the Red Team Village specifically. 00:05.000 --> 00:06.940 We're looking forward to your presentation. 00:06.940 --> 00:08.060 So take it away. 00:08.260 --> 00:12.200 Thank you so much, Omer, for introducing and having me here. 00:12.300 --> 00:14.800 Let me just share the screen and confirm. 00:15.860 --> 00:17.420 Can you see the screen? 00:20.450 --> 00:21.810 Yes, you're all set. 00:21.810 --> 00:23.050 Awesome, perfect. 00:23.190 --> 00:25.630 Thank you everyone for joining me. 00:25.630 --> 00:39.990 And first of all, to just give some context, thank you so much, DEF CON Red Team Village, especially in this pandemic times to make it virtual and people can enjoy these talks without any physical intervention. 00:40.210 --> 00:44.190 And this talk is about the Kubernetes code. 00:44.190 --> 00:49.430 So disclaimer, basically, today's we will not see any slides, basically. 00:49.490 --> 00:51.690 And it's mostly, everything is in online. 00:51.690 --> 00:56.850 You can play along with me, or if you can go back and try yourself, you can go ahead and try as well. 00:56.850 --> 01:01.510 So there will be no slides, mostly live demos and the hands-on exercises. 01:02.050 --> 01:02.790 Cool. 01:02.790 --> 01:14.150 So this talk is primarily focusing on introducing a project recently I have created called Kubernetes Code, which is, this is in github.madhuvakula.kubernetescode. 01:14.150 --> 01:31.430 This project was mainly intentionally vulnerable application, which is to basically try and teach participants, or like anyone come to the new Kubernetes to understand how you can assess these Kubernetes clusters and find vulnerabilities so that you can go back and try it out in your organizations, 01:31.430 --> 01:34.550 or day-to-day pen tests, which you find are a team exercises, right? 01:34.550 --> 01:39.210 So I have presented at some of the places and also I have created a playground. 01:39.210 --> 01:43.290 If you wanted to write now, try it in your browser and follow along, right? 01:43.670 --> 01:46.770 So a little bit about me, myself, Madhu Akula. 01:46.770 --> 01:50.470 I work at currently Zibia as a Cloud Native Security Specialist. 01:50.470 --> 01:57.890 And I have been speaking and training at a bunch of conferences around the world, like DEF CON Black Hat, USENIX, Orally, and some of them. 01:58.150 --> 02:08.070 And I'm fortunate to find some of the vulnerabilities long back using my research in some of the organizations like Microsoft, LinkedIn, Google, and some of them. 02:08.270 --> 02:16.810 So apart from that, my primary interest mostly working around security of cloud, containers, Kubernetes, and automating the infrastructure part. 02:16.910 --> 02:19.450 So that's a little bit about me. 02:19.450 --> 02:24.710 If you wanted to follow me, you can follow me at Madhu Akula in most of the social media stuff. 02:25.610 --> 02:26.510 Cool. 02:26.510 --> 02:33.930 Let's get into what exactly is this Kubernetes code and what we are trying to learn like within this next one hour, right? 02:34.010 --> 02:38.490 So to give a little context about Kubernetes code. 02:38.530 --> 02:40.550 So as I said, it is intentionally vulnerable. 02:40.550 --> 02:47.110 I think as most of you have come across the different types of code, like web code, or Terra code, or some other codes. 02:47.110 --> 02:54.270 So this is a similar project, but it is mostly focusing on Kubernetes to understand what are the known vulnerabilities. 02:54.270 --> 03:03.990 And also we have seen common vulnerabilities in the real world environments, like while performing penetration testing, or like while performing assessments in Kubernetes environments. 03:04.010 --> 03:06.710 From those we have taken as scenarios. 03:06.710 --> 03:08.690 And these are some of the scenarios. 03:08.690 --> 03:10.410 Let me zoom a bit. 03:10.410 --> 03:12.730 I think maybe easier to see. 03:12.730 --> 03:13.330 Yeah. 03:13.330 --> 03:16.170 So these are some of the scenarios I have created. 03:16.170 --> 03:26.110 And we will see some of the scenarios, exploitation, and how you can bypass or gain access to Kubernetes clusters as an attacker point, right? 03:26.110 --> 03:28.950 And I'm really sorry, and I'm shameful. 03:29.070 --> 03:33.490 I said I will keep updating these slides at the documentation. 03:33.530 --> 03:36.570 And also I have been working on the defense part also. 03:36.570 --> 03:39.030 Hopefully this month we will see them coming away. 03:39.530 --> 03:41.110 This part I'm going to skip. 03:42.130 --> 04:01.090 So before diving into Kubernetes, if you are really completely new and coming to the containers or Kubernetes world, I highly recommend understanding the technology because there is a fair amount of gap in between penetration testing or finding or assessing clusters if you don't understand what technology you are doing, 04:01.090 --> 04:01.690 right? 04:01.690 --> 04:13.130 So we have seen commonly, especially in the modern tech infrastructure world, there is a huge gap for the security team to understand the technology before even testing or trying out. 04:13.130 --> 04:21.330 Because once you are able to understand, you can be able to do your best or you can be able to possibly find most of the Kubernetes or any ecosystem. 04:21.550 --> 04:28.230 This is a very funny video and it's a simple children's guide to explain what exactly Kubernetes is. 04:28.230 --> 04:31.950 So I would highly recommend if you want to go through these things. 04:31.950 --> 04:38.050 So to give you a little quick context, how Kubernetes architecture looks like at a higher level, right? 04:38.050 --> 04:40.850 So this is again from the wiki comments. 04:40.850 --> 04:46.890 You see a server and client architecture similar to most of the server client. 04:47.090 --> 04:57.710 So one thing you have to observe here is in Kubernetes, everything which you make, any operation, it may be from within the master, which means server, within the node, which means worker node. 04:57.710 --> 05:08.390 So whatever the operations which will make, like it can be from the node, it can be from the developer or any user, any service will go and touch something called API server. 05:08.390 --> 05:15.670 So which is one of the key component in terms of Kubernetes, because Kubernetes is completely driven by API first approach. 05:15.670 --> 05:21.550 So whatever you make all requests or something, everything you will see going through the API server, right? 05:21.550 --> 05:26.610 And control manager is basically to make the decisions, whatever the requests you make. 05:26.610 --> 05:33.090 And scheduler is ensuring to place the workloads or the request which you made. 05:33.090 --> 05:40.570 Suppose if you wanted to serve a Nginx server with 10 pods, it will ensure that how many nodes has the capacity to schedule. 05:40.570 --> 05:44.190 So the scheduler job is to figure out that and place them accordingly. 05:44.310 --> 05:50.790 So etcd is one of the key value store database, which is used by Kubernetes, which is as its name suggests, key value. 05:50.790 --> 05:56.910 So it stores the data, which is coming to the API server and stores the key value base. 05:56.910 --> 06:00.870 And we will see a little bit in a later coming sections. 06:01.650 --> 06:10.730 So apart from that nodes, when you come to the node, kubelet is the talking point to the API server and communicates within the local node itself. 06:10.730 --> 06:22.650 So whenever you wanted to make a request as a scheduler, you wanted to schedule in the node, it will talk to the container runtime, which is within the node, like it can be Docker, RunC, etcd. 06:22.650 --> 06:28.430 So then it will go ahead and schedule based on the workload kind of thing. 06:28.610 --> 06:39.230 So kubeproxy is one of the another key component, which is kind of like IP tables, like it will try to provide a communication across like the nodes as well as a cluster. 06:39.810 --> 06:51.370 So let's go ahead and a little bit go deep, but I would highly recommend you practice the Kubernetes before you diving into the assessments or security testing. 06:51.370 --> 06:54.390 So here are some of the references I have curated. 06:54.390 --> 07:06.190 These are pretty hands-on, like if you wanted to try, so you can go ahead and try as a literally playground and also specific objects if you start while testing, you can click and understand what exactly it is. 07:06.190 --> 07:10.670 So these references I would, I think it would be definitely helpful to try. 07:11.130 --> 07:12.090 Cool. 07:12.490 --> 07:14.370 Setting up Kubernetes code. 07:14.610 --> 07:18.690 The Kubernetes code is in design in such a way it can set up any out of the box. 07:18.690 --> 07:23.990 It can be in your existing cloud provider services like GKE, EKS, AKS, or digitalization based. 07:24.350 --> 07:27.750 And also your locally provisioned Minikube or another playground. 07:27.750 --> 07:33.990 To make easy of setup for the people to try out and learn, I have created something called Katakoda Playground. 07:33.990 --> 07:39.950 If you just go ahead and click this, it will open up a playground for you, which is completely free and online. 07:39.950 --> 07:41.790 So you don't need anything to be done. 07:41.790 --> 07:43.450 I think you have to register first time. 07:43.450 --> 07:48.290 So then you can get the complete setup here for you to practice and learn. 07:48.290 --> 07:51.590 Let me put it in a presentation mode so that you can see a little better. 07:51.590 --> 07:52.110 Yep. 07:52.170 --> 07:52.770 Cool. 07:53.650 --> 07:54.270 Cool. 07:54.330 --> 07:58.790 So I'll walk you through how you can set this up and practice. 07:58.790 --> 08:05.470 And if you wanted to follow along with me, you can follow this guide at madhuvakula.com slash Kubernetes code. 08:05.470 --> 08:07.450 This is the URL, right? 08:07.530 --> 08:07.950 Cool. 08:07.950 --> 08:14.310 And if you wanted to set up your own, you can just follow this, the documentation. 08:14.310 --> 08:18.630 Basically you need to have a working Kubernetes cluster set up with admin access. 08:18.630 --> 08:27.990 And also you need a Helm version too, because in one of the scenario, we will try to focus on how you can exploit the existing vulnerability in the older version of Helm. 08:27.990 --> 08:29.370 So that's it. 08:29.370 --> 08:33.250 So if you wanted to set up your own, let's go ahead and do that as well. 08:33.430 --> 08:40.630 So if you have your own set cluster, you clone the repository, which is the git clone of the Kubernetes code. 08:40.710 --> 08:46.050 And then you can go ahead and set up the Kubernetes code. 08:46.390 --> 09:06.470 So it checks if Kubernetes kubectl is available, Helm 2 is available in your path, and it'll go ahead and deploy the tiller, which is the server side component of the Helm, and a bunch of vulnerable services, which will be used as practice to exploit or try practicing Kubernetes security. 09:07.550 --> 09:15.230 So if you are someone following the Katakoda live, you can use this playground, which is completely free and online. 09:15.230 --> 09:19.010 You can go ahead and do the same thing in the playground as well. 09:19.790 --> 09:26.510 So the playground is designed in such a way, you can go ahead and see here only the guide in the second, in the specific tab. 09:26.510 --> 09:31.270 And also you can browse those ports and other stuff within the browser itself. 09:31.270 --> 09:33.490 Even in your mobile, also you can try this. 09:35.050 --> 09:41.910 So it takes a while to set up the entire Kubernetes code platform. 09:41.910 --> 09:43.630 I think it's almost done. 09:43.630 --> 09:49.610 So it creates a bunch of vulnerable scenarios, which I have pre-created. 09:49.650 --> 09:55.010 You can go through the code and you can able to see what exactly it is trying to do, if you wanted to look at it. 09:55.010 --> 10:05.910 So this is nothing but it checks if kubectl is existing or Helm 2 existing and it sets a bunch of emails, like which is nothing but deployments in the cluster. 10:07.550 --> 10:08.030 Cool. 10:08.030 --> 10:12.170 Now we have the cluster up and running and we have Kubernetes code set up. 10:12.170 --> 10:13.730 So what's next, right? 10:13.730 --> 10:16.470 So I think it looks like here also it is done. 10:16.550 --> 10:18.970 Ensure you check all the ports are running. 10:18.970 --> 10:21.990 You can go ahead and run kubectl get ports. 10:21.990 --> 10:25.730 It is currently container creating, which means it's starting. 10:25.810 --> 10:31.530 It takes a while to set up the, depends on the way you are running the Kubernetes cluster. 10:31.650 --> 10:37.230 So let me go ahead and do this as well here, kubectl get ports. 10:37.390 --> 10:39.870 I hope you people can see the screen. 10:39.870 --> 10:42.630 I think, let me zoom a little bit as well. 10:42.630 --> 10:43.150 Cool. 10:43.150 --> 10:44.930 So you can see a bunch of them. 10:44.930 --> 10:46.790 Most of the ports are created. 10:46.790 --> 10:50.130 I think just only one port is waiting to get created. 10:50.390 --> 10:51.790 That's awesome. 10:51.830 --> 10:55.770 Now you can see all ports are running and I think this is intentional. 10:56.150 --> 10:58.150 This is a job and get company. 10:58.350 --> 11:03.450 So if you see the status, basically your entire Kubernetes code is set up and running. 11:03.450 --> 11:05.970 So now it's time to hack and get into the cluster. 11:05.970 --> 11:06.450 Bye . 11:07.490 --> 11:08.270 Cool. 11:08.270 --> 11:10.890 That's where we go back and refer to the documentation. 11:10.990 --> 11:16.770 So if you look at the scenarios, so there are a bunch of scenarios we have created, as I said. 11:16.770 --> 11:29.470 So for the time being, for the entire today's presentation, we will not be able to look at all of those scenarios, but definitely we will be looking at some of the primary scenarios, which might be interesting and fun to try out. 11:29.490 --> 11:35.610 And I would highly recommend go back and try out yourself and check out more other scenarios as well, right? 11:35.610 --> 11:45.910 And in some cases, especially not all of them scenarios, but in some scenarios, I try to create a flag in this format, k8sgoot-goot. 11:45.910 --> 11:51.890 And some MD5s, basically it's a number of 32 characters. 11:52.390 --> 12:07.590 Then if you reach this kind of flag, basically it means end of the scenarios, but it may not be found in for all the scenarios, but I try to keep for certain scenarios to reduce the effort of like going on beyond post-exploitation. 12:09.010 --> 12:09.870 Cool. 12:09.870 --> 12:15.050 So let's go ahead and start with the scenario one, sensitive keys in code base. 12:15.050 --> 12:26.070 So I would recommend before doing that, I think if you are setting up locally or in the same place, all ports are running, make sure you run bash access Kubernetes code. 12:26.070 --> 12:37.330 What it does is basically put forward all the ports or services into your local system so that you can access locally without even exposing to the world, which is internet, right? 12:37.330 --> 12:43.230 So now if you go ahead and click this, basically you can see the homepage, like which is the Kubernetes code homepage, right? 12:43.230 --> 12:47.310 So this is where you can see when it's good, it's running, you can follow the guide here. 12:47.670 --> 12:51.390 And also if you want to check out each scenario, the scenarios are here. 12:51.710 --> 12:58.250 So let's get started with the worst scenario, which we will look at one of them is called sensitive keys in code basis. 12:58.530 --> 13:09.090 So the Kubernetes code is designed in such a way, if you wanted to practice as an attacker or learner, you can go ahead and try this scenario by clicking here and starting with attacking. 13:09.290 --> 13:16.370 Or if you are someone who wanted to learn and practice yourself and follow some kind of guide, we have created the documentation. 13:16.370 --> 13:20.710 The solutions will be here in the same guide for each scenario. 13:20.730 --> 13:24.470 Like you can follow them as well here if you're stuck somewhere, right? 13:24.610 --> 13:28.050 And some scenarios we try to create two different methodologies. 13:28.050 --> 13:29.990 It can be tagged in different ways. 13:29.990 --> 13:32.830 So we try to document in the same way. 13:34.670 --> 13:39.090 So if you see here, let's go back and read a little bit about the scenario. 13:39.090 --> 13:43.070 Developers tends to commit sensitive information to version control systems. 13:43.190 --> 13:50.410 As we are moving towards the ACD and GitHub systems, we tend to forget identifying sensitive information in code and commits. 13:50.410 --> 13:53.250 Let's see if we can find something cool here, right? 13:53.250 --> 13:59.290 So by looking at, you can see that some kind of vulnerability or some kind of mistake has happened. 14:01.210 --> 14:04.830 Now, as you can read this, this is version control using Git. 14:04.830 --> 14:10.710 So what I can do is I can try as a DartKit and you can see the default file in the Git will be config. 14:10.710 --> 14:14.610 Now you can see that there is a bare repository hanging there, right? 14:14.650 --> 14:23.890 And by the way, these vulnerabilities which you see, I was showing like in the web vulnerabilities, these are just an entry points into getting into the cluster. 14:23.890 --> 14:27.690 In real world, it may not be the same vulnerabilities you find. 14:27.730 --> 14:31.130 Maybe there is a different approach you find to get in here, right? 14:31.130 --> 14:32.210 That's the thing. 14:32.510 --> 14:35.110 So now I can see that there is a Git config. 14:35.110 --> 14:40.690 Let's go ahead and see how I can clone this repository locally so that I can understand this code base better, right? 14:40.690 --> 14:46.470 So if you want to start, basically you can go back and follow this repository here. 14:46.470 --> 14:48.410 So there is a tool called GitDumper. 14:48.410 --> 14:49.890 It can be any other tool as well. 14:49.890 --> 14:51.250 I just take an example. 14:51.250 --> 14:57.130 So you can try this tool by running a Python3-GitDumper localhost and save it like this. 14:57.230 --> 14:59.950 Maybe I think I have configured here. 15:02.070 --> 15:03.110 Cool. 15:04.530 --> 15:05.530 GitDumper. 15:05.590 --> 15:12.010 So what I'm trying to do is I'm cloning the localhost1230.git folder into local system. 15:12.030 --> 15:18.130 So what it is trying is it is going to fetch all the repository and clone it locally, right? 15:18.130 --> 15:24.390 So now if you go here, kdiskcode.git, so it is the Git repository which we have. 15:24.390 --> 15:31.850 So if you see here, we can see there is a Golang bunch of files, main.co, and a bunch of custom code, right? 15:31.850 --> 15:34.530 And also you can see Git, the folder. 15:34.530 --> 15:38.190 So you can also look for what are the old commits which are happening, right? 15:38.190 --> 15:44.650 As developers sometimes make mistakes, by the way, this scenario is not only focused towards containers or Kubernetes. 15:44.710 --> 15:55.110 In general world, we have seen these kinds of mistakes in most of the modern software development, especially in cloud deployments or any internal deployments, right? 15:55.790 --> 16:05.410 So I can see some of the bunch of commits as you attack and you can go and check manually, like Git checkout to specific, the log.check commit. 16:05.410 --> 16:12.330 And also you can see, oh, you can see an interesting file called env, right? 16:12.350 --> 16:13.330 So there you go. 16:13.330 --> 16:19.050 So you can see there is the access key and secret key for the AWS has mistakenly committed. 16:19.090 --> 16:22.930 And you can also see there is the kdiskcode flag is there. 16:22.930 --> 16:35.910 So basically this means that you have reached the end of this scenario, but in real world you can use this AWS access key, secret key and get the session token and you can do post-exploitation and go along beyond. 16:36.230 --> 16:42.750 So these scenarios are just we have taken from what we see in real world as a pen testing engagement. 16:42.750 --> 16:47.910 And some of them are like in a common scenarios which we see in real world, right? 16:47.910 --> 16:53.130 So this may not be possible and this may be possible in different use cases. 16:53.130 --> 16:58.130 So feel free to give it a try and understand what exactly we are trying to achieve here. 16:58.150 --> 17:01.730 So basically what will happen is in the same use case. 17:01.810 --> 17:03.710 So we can try in different method. 17:03.710 --> 17:19.930 Like what I'm trying to do is most of the time developers mistakenly build a Docker images which may be committed with the .git files because they don't ignore these files when it is creating into Docker container, maybe Docker ignore or something. 17:19.930 --> 17:24.890 So which might end up pushing this data into the containers as well. 17:24.890 --> 17:30.030 So looking at these kinds of sensitive information is very useful in the container. 17:30.030 --> 17:33.030 Like the another methodology you can try, as I said, explanation. 17:33.030 --> 17:36.030 So you can get the pod name, you can exec into the pod. 17:36.030 --> 17:37.730 This is like another method. 17:38.370 --> 17:41.030 Now we can also use tools like TruffleHog. 17:41.030 --> 17:51.570 So TruffleHog, I think most of you know, this is a tool which is basically finds high entropy or sensitive secrets in a Git repositories. 17:51.570 --> 17:58.150 So now I'm trying to do TruffleHog and given the current directory is the where Git repository is there. 17:58.150 --> 18:09.790 So what it does is it will check through all the commits and finds, is there any high entropy values or like sensitive secrets found based on the regular expressions and return to the studio. 18:09.930 --> 18:15.190 So you can include these kinds of tools in your CI, CD systems to detect these kinds of vulnerabilities. 18:15.190 --> 18:27.450 But in attacker's point of view, if you find any of the code basis which has Git or something, maybe it's a really definitely good way to check out in especially the container environments, you might end up finding some kind of sensitive approach. 18:28.410 --> 18:30.230 Like you can see the same thing. 18:31.610 --> 18:32.410 Cool. 18:32.430 --> 18:37.630 So this is our first scenario, which we just tried from the Kubernetes code. 18:37.630 --> 18:40.270 And similarly, as I said, you can try out here as well. 18:40.270 --> 18:52.270 So what you can do is you can look at here and if you wanted to specifically browse specific code, like one, two, three, zero, in the custom code tab, you can just open in a new browser window also. 18:52.270 --> 18:56.710 If you press this icon, then you can just say, I wanted to browse one, two, three, zero code. 18:56.710 --> 19:06.390 So you can able to browse the same thing and play it in online yourself for free and understand what exactly you can assess or test for Kubernetes security. 19:08.090 --> 19:08.890 Cool. 19:09.170 --> 19:12.990 So that's kind of one of the simple scenario. 19:12.990 --> 19:21.950 So let's go ahead and see what could be the some of the complex scenarios or different kinds of vulnerabilities you see in Kubernetes security assessment. 19:22.290 --> 19:32.330 So let's go back to the some maybe a little down, especially like attacking private registries or Kubernetes CS benchmarks or container escape. 19:32.750 --> 19:37.430 So let me go and showcase you how container escape to host system works. 19:37.770 --> 19:41.510 To browse to this scenario, basically you navigate here. 19:42.610 --> 19:51.910 So this is basically one of the pod inside Kubernetes cluster running with some specific privileged access has, right? 19:51.930 --> 20:10.150 So to set the context here, most of the times, especially operations teams may tend to require to run specific containers with high privileges, maybe giving gaps is admin or host privileges or privilege context, which is a full admin, sysadmin capabilities. 20:10.150 --> 20:11.530 So these are things. 20:11.530 --> 20:20.250 And also to just give an example, how we found in a real world, I have seen, especially while we performing a penetration testing engagement. 20:20.270 --> 20:25.750 So most of the developers get an access to completely one namespace with a full privilege. 20:25.750 --> 20:35.190 So if your clusters doesn't have PSPs, which is nothing but pod security policies, developers can basically has full access to that namespace. 20:35.190 --> 20:44.250 Whatever they can do, they can deploy a pod with specific privileges, which they can gain into the whole system access, right? 20:44.250 --> 20:55.150 So also this is one of the use case we have taken to simulate or showcase how you can find these kind of container escape and gain access to the whole system, right? 20:56.310 --> 20:56.910 Cool. 20:56.910 --> 21:01.770 So what we are trying to do here is we can try and look for some bunch of information. 21:01.770 --> 21:04.170 Most of the time, look for a printenv. 21:04.270 --> 21:07.030 Sorry, I think I typed print. 21:09.330 --> 21:10.290 Cool. 21:10.290 --> 21:15.690 So most of the time, you might see some kind of stuff here as well in printenv. 21:15.690 --> 21:17.830 And also I'm checking for mount. 21:17.830 --> 21:19.590 Is there any mounted directories? 21:19.590 --> 21:29.730 Now you can see there is a mounted directory from host system and which is into the container, which means basically in the pod, in this Kubernetes case, right? 21:29.730 --> 21:39.610 So now what I can do is, I think, let's look at what exactly the host system is doing. 21:39.610 --> 21:46.430 So if you see, there is a host system mounted, like which is basically from the host system, to just give an example. 21:46.510 --> 21:54.230 So which is nothing but your underlying host system, which is node, is mounted into your pod as a volume, right? 21:54.230 --> 21:58.510 So due to the reason we have extra capabilities, right? 21:58.510 --> 22:01.630 Like this container has a bunch of extra capabilities as well. 22:01.630 --> 22:11.670 So using those capabilities, as an attacker, what we can do is, we can like try to gain some kind of access into host system. 22:11.670 --> 22:18.150 What you can do is, we can sage root on the host system because we have capability of the privilege. 22:18.450 --> 22:23.390 And now we can see that it is given that the host system sage root. 22:23.390 --> 22:28.950 And now what we can do is, we can perform the internal commands, which are specific to system, right? 22:28.950 --> 22:30.930 Like now you can see docker ps. 22:31.370 --> 22:35.970 So these are nothing but running inside the host system, which is not a container. 22:35.970 --> 22:40.650 So you went below one level from the pod to the host system. 22:40.650 --> 22:43.250 From there, you are executing commands, right? 22:43.250 --> 23:02.270 So the possible attack surface here you can do is, to do post exploitation might be basically using the host system configuration, which is nothing but node level Kubernetes configuration, to go and try attacking other nodes or to gain access to other nodes, 23:02.730 --> 23:07.370 which basically you're controlling the entire cluster by passing through, passing through, right? 23:07.370 --> 23:16.370 So for example, what you can do is, you can use the configuration of node, which is available in the cat var lib. 23:21.580 --> 23:29.000 So you can use the configuration of cat var lib quip config, which has the node level configuration. 23:29.000 --> 23:44.200 So you can use that configuration and perform the Kubernetes commands, which is not only available for the existing namespace specific, it can also perform at the node level, whatever the node has the service account, you can able to do that level. 23:44.400 --> 23:47.020 To do that, for example, you can do this command. 23:49.760 --> 23:52.080 A kubectl command is not available. 23:52.080 --> 23:53.860 Let's download kubectl. 23:56.640 --> 24:00.260 Let me download this kubectl here. 24:02.100 --> 24:04.880 It just takes some time. 24:04.880 --> 24:06.440 Hopefully it is faster. 24:07.400 --> 24:08.160 Yeah. 24:12.190 --> 24:13.590 Yep, there you go. 24:13.590 --> 24:28.610 So what we are trying to do is, we are using now kubectl command and passing the kube config flag, which is using the node configuration and getting all like whatever the all resources in namespace of kube system, right? 24:28.650 --> 24:31.450 Oops, I'm trying the same thing again. 24:38.550 --> 24:44.250 I have to do chmod plus x kubectl. 24:46.090 --> 24:46.830 Cool. 24:46.830 --> 24:57.770 Now you can see when I run kubectl kube config with the specific configuration and get all in kube system, it will go ahead and execute that command and get all the resources. 24:57.770 --> 25:06.830 Now you can see here, it can able to get all the ports which are not restricted to specific namespace, which is the developer or the user has. 25:06.830 --> 25:11.730 And you can able to explore more or access more resources and also services. 25:11.970 --> 25:21.030 And you can see some of the specific use cases, especially I think replication controllers and other jobs and secrets, you cannot able to access. 25:21.230 --> 25:27.130 So the node, whatever the access it has, the node configuration, you can able to perform same level. 25:27.130 --> 25:36.730 So as an attacker, you can do post-exploitation, basically deploy another port in a different node and you can able to pass to another node and again get access to that. 25:36.730 --> 25:46.870 So for that, you can just look for how many nodes you have, like currently two nodes, maybe you are in this node, then you can deploy another port in the different node and get again access to the different. 25:46.870 --> 25:54.270 So you can basically use this kind of hopping methodology and hop on to another node and gain complete cluster access. 25:56.030 --> 26:11.590 So this is mostly you will see if an organization or the company which is not running up the port security policies, you can able to perform these kind of attacks or escalations of the container escapes, right? 26:11.730 --> 26:22.750 So you might use BSPs as a defense for these kind of vulnerabilities and it would be an easy and simple way to fix this kind of exploitation. 26:23.990 --> 26:25.030 Cool. 26:25.330 --> 26:28.450 So let's go ahead and see another use case. 26:28.450 --> 26:31.970 I think we have some more time to see two other scenarios. 26:31.970 --> 26:34.910 Let's take an example of Kubernetes namespace bypass. 26:36.770 --> 26:37.850 Okay. 26:38.690 --> 26:42.750 I'm going here, Kubernetes namespace bypass. 26:42.830 --> 26:43.970 I don't know exactly. 26:43.970 --> 26:44.590 Yeah. 26:44.590 --> 26:47.410 So by default, Kubernetes uses flat networking scheme. 26:47.410 --> 26:51.570 I think this is one of the default feature of Kubernetes. 26:51.690 --> 27:02.390 So by default, Kubernetes uses flat networking scheme means I can able to talk to from any port to any port by default, there is no firewall rules which will happen, allow or deny. 27:02.390 --> 27:06.530 By default, anything can talk to anything within the cluster IP range, right? 27:06.530 --> 27:16.810 So which means that whenever someone, SRE or ops team or like security people says that there is a namespace we have created in the Kubernetes which means it's just a logical boundary. 27:16.810 --> 27:27.350 There is no way you literally segregating between the different namespace unless until apply some kind of security policies like NSPs and network security policies, right? 27:27.350 --> 27:44.350 So if you are testing a Kubernetes clusters where there was no NSPs and there was no restrictions between the namespace as attacker, you can able to talk to other namespaces and other services within the cluster by using a standard network security tools like which we use day to day, 27:44.350 --> 27:48.050 like Nmap or Zmap or a bunch of tools, right? 27:48.050 --> 27:52.810 So to just get started with the scenario, I have created something, an example. 27:52.830 --> 27:59.050 So what we are trying to do is we are running something called Hacker Container which has again created by me. 27:59.050 --> 28:09.070 I have while performing bunch of Kubernetes security testings I thought that there are so many tools we require especially focused on container environment. 28:09.070 --> 28:17.910 So I have created curated list of tools which are maybe 50 plus tools which will help you aid in while performing Kubernetes security assessment. 28:17.910 --> 28:27.410 So this container has those all tools like while you are performing Kubernetes security assessment and you can use this as like a Swiss Army knife in security assessment, right? 28:27.410 --> 28:31.950 And I have seen this has been very valuable while performing assessment. 28:32.030 --> 28:35.610 So definitely I'll try to create some kind of documentation out of it. 28:35.610 --> 28:40.570 Maybe hopefully you can go back and try out different use cases with Hacker Container. 28:40.570 --> 28:43.670 I think it's going and deploying there and cool. 28:43.670 --> 28:44.570 So we got a shell. 28:44.570 --> 28:49.770 So what we are doing is we are inside a pod in one of the default namespace. 28:49.770 --> 28:55.970 Like if you wanted to just see, let's go ahead and run some command called kubectl get pods. 28:55.970 --> 29:00.610 Now you can see Hacker Container which is running in a default namespace. 29:00.670 --> 29:04.770 And let's go ahead and see what else we can do from the shell, right? 29:04.770 --> 29:10.530 So as attacker, you are in one of the pod and you don't know what else you can do from there, right? 29:10.750 --> 29:17.390 So as I said, Hacker Container has a bunch of tools like Docker Bench Security, Kubebench, Linus and a bunch of other tools. 29:17.470 --> 29:24.050 So one interesting thing you can do especially is looking for the entire cluster range. 29:24.050 --> 29:27.610 As I said, by default, Kubernetes uses flat networking scheme. 29:27.610 --> 29:29.870 You can look at the entire cluster range, right? 29:29.870 --> 29:31.190 So you can do IP route. 29:31.190 --> 29:34.890 You can see this is using 10.0.0.0 network range. 29:34.890 --> 29:40.150 And also you can look for what is the current IP address and 10.0.0.0.12. 29:40.150 --> 29:43.810 So you can scan the entire network and identify services. 29:43.810 --> 29:59.710 So one thing I can confirm, I can assure is most of the times internally within the cluster as a microservices or services which are running, many of the developers or operations teams tend to forget or assume that it is only exposed to internal networks. 29:59.710 --> 30:21.650 They will never use authentications or authorizations and you might end up seeing some kind of bypasses which is exposed internally itself, especially Mongo services, Elasticsearch instances and also even some microservices because the API gateway terminates at the ingress level which means before exposing into the world like using load balancers, 30:21.650 --> 30:24.390 they will terminate at the ingress level. 30:24.390 --> 30:33.590 So you might end up seeing some kind of services especially the application services also don't have authentication authorization checks within the clusters. 30:33.590 --> 30:42.850 So this is a really great place to explore and do as much recon you can do and you can gain as much as sensitive information or services access. 30:43.870 --> 30:44.730 Cool. 30:44.730 --> 30:52.990 So what I'm trying to do is here I'm exploring same and what I did is I'm trying to use ZMap which is similar to Nmap. 30:52.990 --> 30:57.910 It's trying to do a scan for the entire cluster range which is to do a faster way. 30:58.010 --> 31:03.350 And I'm specifically looking for a port of 6379 which is a Redis instance. 31:03.370 --> 31:12.810 So I assume that there might be a Redis running or like a 9200 you can also try which is Elasticsearch or Mongo or MySQL or any other services you can scan. 31:12.810 --> 31:15.010 It's like traditional security will apply here. 31:15.250 --> 31:19.230 Let's go ahead and run this ZMap and see what we can see. 31:19.230 --> 31:20.950 So, oops. 31:21.170 --> 31:21.770 Okay. 31:21.770 --> 31:29.730 So we have to have enable the internal network range allow within this list. 31:29.730 --> 31:31.710 Let's go ahead and do that. 31:33.710 --> 31:37.750 Let's comment this and run again. 31:37.750 --> 31:38.930 Sorry. 31:40.970 --> 31:42.170 Oh, okay. 31:42.170 --> 31:48.210 You have to do another thing which is specify the gateway where this ZMap is starting. 31:48.370 --> 31:50.730 Let's go ahead and do that as well. 31:51.590 --> 31:53.110 F1G capital. 31:53.130 --> 31:53.710 Okay. 31:54.330 --> 31:56.410 And what is the gateway? 31:58.430 --> 32:00.490 Yeah, this is the MAC address. 32:01.050 --> 32:01.990 Cool. 32:02.450 --> 32:03.570 So, yeah. 32:03.570 --> 32:19.790 So what it is doing, it is trying to scan the entire cluster range for the specific port 6379 and results the output if any of the IPs running or services running or what is running with the specific port open, it'll save into the other results.csv. 32:19.790 --> 32:20.710 It take a while. 32:20.710 --> 32:27.850 I think it's pretty fast in this case, but this will take some more time to go through and scan. 32:28.550 --> 32:51.250 But as I said, in general, we have seen in real world a lot of services running, especially Elasticsearch instances where you can look for without authentication and where you can gain access to indices and the log information, especially audit logs and sensitive information and also Mongo instances without any authentication or also the microservices, 32:51.250 --> 32:51.750 right? 32:51.750 --> 32:58.690 So this is a pretty goldmine if you are inside a pod and without NSPs, network security policies applied within the cluster, right? 32:59.670 --> 33:00.610 Cool. 33:00.610 --> 33:02.650 So this will take a while. 33:02.650 --> 33:05.250 I think 76% and all stuff. 33:05.530 --> 33:05.870 Yeah. 33:05.870 --> 33:12.850 As I said, similarly, you can try out those vulnerabilities here and follow the guide if you wanted to do as well. 33:12.850 --> 33:19.690 And I think it's a simple way to try out this entire Katakoda environment within the browser. 33:19.690 --> 33:23.370 You don't need to set up the entire cluster itself locally. 33:23.410 --> 33:25.690 And also once you are done, you can just quit the browser. 33:25.690 --> 33:26.350 It's all done. 33:26.350 --> 33:28.110 You don't need to clean up the entire environment. 33:29.290 --> 33:43.670 And another disclaimer, I think I forgot to mention in the beginning, do not try to run or set up this Kubernetes code in your production environments or in your office because it has intentionally vulnerabilities introduced due to the application flaws or something. 33:43.670 --> 33:51.250 It is a vulnerable application, which may have gain access to your production or wherever you run these environments. 33:51.470 --> 33:59.290 And at least I try to take care of using, exposing the ports within to the localhost, but it's highly not recommended to run in production. 33:59.390 --> 34:00.390 Cool. 34:00.770 --> 34:02.570 We have ZMap completed. 34:02.570 --> 34:05.090 Let's go ahead and see what we got from the results. 34:05.330 --> 34:12.460 You can see that there is the IP, which has 6379 port is exposed within the cluster. 34:12.640 --> 34:15.280 And let's go ahead and see what we can get out of it. 34:15.280 --> 34:16.500 As you know, it's Redis. 34:16.500 --> 34:23.720 As I said, this hacker container has built-in tools to look for these kind of common attacks and all stuff. 34:23.720 --> 34:28.060 We can use Redis CLI, and I forgot to be honest, how to access. 34:28.380 --> 34:30.700 Yeah, I found H and the IP. 34:30.760 --> 34:45.660 And now you can see there is another integrated Redis running within the cluster you can able to access because there is no network security policies which is by default in the cluster, means you can able to access without any issues. 34:45.660 --> 34:49.500 I mean, so they don't have any allow list or denial list within the cluster. 34:49.840 --> 34:53.500 Now you can do enumeration standard like get key. 34:53.980 --> 34:55.600 You can see nothing. 34:55.600 --> 34:55.920 Okay. 34:55.960 --> 34:56.680 Keys. 34:56.680 --> 35:03.820 You can see the keys, what are available, and you can see there is a key and get secret stuff. 35:04.800 --> 35:05.540 Cool. 35:05.540 --> 35:06.740 So you reach the flag. 35:06.740 --> 35:09.800 That means you are done with the challenge, right? 35:09.800 --> 35:17.660 So this is just an example of try to simulate the scenarios, but definitely you see these kind of vulnerabilities in a real world. 35:17.660 --> 35:26.160 Once you end up in a pod, try to look out for more services and try as much as you can and you will get most information out of it. 35:26.160 --> 35:26.720 Right. 35:26.960 --> 35:27.940 Cool. 35:28.620 --> 35:35.360 So let's go back and see another scenario before we conclude the presentation and moving into the next sections. 35:36.320 --> 35:44.100 Let's look at one of the common oldest, the vulnerabilities which we have seen exploiting the Helm or Tiller. 35:46.320 --> 35:49.480 So Helm is like a package manager for Kubernetes. 35:49.480 --> 35:56.800 So how we deploy application, install applications into, suppose you are using Debian-based apt-get install the package name. 35:56.800 --> 36:04.500 Similarly, developers or operations teams use Helm, which is a package manager to deploy applications into Kubernetes, right? 36:04.500 --> 36:10.320 So they will just do Helm install and the Nginx, it'll go and install the Nginx within the cluster, right? 36:10.320 --> 36:26.460 So by default, if you are testing any cluster and which is using version two, which is the older version of Helm and it used to be by default, most people using is Helm two in a previous cases, but they have raised three recently, like someone's back. 36:26.600 --> 36:32.060 So by default, Helm version two has RBAC setup with the full cluster admin access. 36:32.060 --> 36:48.280 So if you end up seeing these kind of Helm two configurations running within the cluster and there are no network security policies applied and it is running with the default setup, basically using this misconfiguration as an attacker, you can get entire cluster admin access and form the entire cluster, 36:48.280 --> 36:48.820 right? 36:48.820 --> 36:50.880 Let's go ahead and see how this scenario looks like. 36:50.880 --> 36:56.400 I'm going and running a simple pod and executing into that, right? 36:57.720 --> 37:00.080 Cool, let's go ahead and run this command. 37:00.080 --> 37:06.300 It'll go ahead and spin up another pod, it is called HelmTeller and give you a bash access, right? 37:14.910 --> 37:27.670 Yep, so now we are inside a pod and we assume that this cluster has Helm installed and we wanted to see how we can exploit this kind of vulnerability in a real world environment, right? 37:27.670 --> 37:30.570 So let's go back to our documentation, right? 37:30.570 --> 37:39.010 So by default in Kubernetes, there is a schema which they follow to access or basically it's like a DNS. 37:39.190 --> 37:42.550 Similarly, how we use, they use in the Kubernetes. 37:42.550 --> 37:52.010 So this is nothing but a service name, this is the namespace and they use either a service name and namespace and a cluster.local. 37:52.170 --> 37:55.490 So this is how the structure which is followed, right? 37:55.490 --> 38:04.510 So by default, if you have installed the Helm, which is Teller is get deployed in version two, which is nothing but a server side component of Helm. 38:04.510 --> 38:07.930 Helm is the client side version where you do Helm install or something. 38:07.930 --> 38:11.930 So the Teller is the server side component, which has a full cluster admin access, right? 38:11.930 --> 38:17.530 So now we are doing Teller deploy is the service name and it is by default deployed in a kube system. 38:17.530 --> 38:25.390 So I'm trying to look for kube system and I'm taking Telnet and see if this port is able to access from the current port. 38:25.390 --> 38:38.470 So this is the default port, which is in a by default configurations exposed to 0.0.0, which means if you are in any one of the four port within the cluster, you can able to access the Teller deployment service, right? 38:38.470 --> 38:42.350 You can see, looks like you can able to access the Teller. 38:42.350 --> 38:52.050 So which means you can able to talk to the Teller deployment service because there were no NSPs and by default, it is exposed to 0.0.0, which is within the cluster, right? 38:52.050 --> 38:54.110 And let's see what we can do. 38:54.150 --> 39:02.090 So I have also installed a Helm within this same container to talk to the Helm. 39:02.090 --> 39:12.730 So in the Helm version two, you can specify the dash dash host flag, which means you can specify which Teller service you wanted to talk to. 39:12.730 --> 39:21.870 Now we know the Teller service name as well as the DNS basically, and we are specifying talk to this host and perform these operations, right? 39:21.870 --> 39:24.150 Let's go ahead and run this command. 39:27.030 --> 39:34.810 I'm trying to run Helm command and specifying the specific service and the port, and I'm just running Helm version command, right? 39:34.810 --> 39:42.350 Now you can see the client version, which is Helm, is 2.16.7, and the server version, which is Teller, is 2.16.9. 39:42.350 --> 39:48.070 So I can able to talk to the Helm service and I can also see what all charts deployed, Helm charts, right? 39:48.070 --> 39:50.690 Helm-less, like you can see metadata DB is deployed. 39:50.690 --> 39:53.330 So you can create your own chart and apply as well, right? 39:53.330 --> 40:01.810 So what we have done is, as an attacker, simply to showcase the complexity or what you can do as an attacker, let's go ahead and run a simple command. 40:02.190 --> 40:10.210 By default, Kubernetes stores, has a service account, which is stored in var lib, service secret, 40:18.690 --> 40:33.010 one, yeah, secrets, Kubernetes IOS service account, namespace, because I'm running in default namespace, and this is the token for that specific service account, which is mapped to the current port. 40:33.010 --> 40:47.310 So what I'm trying to do now here is, we have created a Helm chart, which is one chart, and this I have taken from the Bitnami repository research, and I have given the reference here, and go and check more resource. 40:47.310 --> 40:50.470 What it does is basically, it's a simple Helm chart. 40:50.470 --> 40:58.170 What it is do is, it has a bunch of values, which is nothing but namespace is default, and the name of the, is default. 40:58.170 --> 41:00.470 Let's look at the templates, how it looks like. 41:01.810 --> 41:04.950 Templates, and a cluster role. 41:04.950 --> 41:22.070 It has a cluster role, which is nothing but, it gives all API groups in the Kubernetes, for all the resources, and all the verbs, which means you can do get, post, delete, list, all of the resources, for any of the resources in the Kubernetes cluster, 41:22.070 --> 41:25.490 any of the kind of API groups, you can perform. 41:25.550 --> 41:29.930 So what we are trying to do is, templates, cluster role binding. 41:30.090 --> 41:44.430 So we are trying to bind this cluster role, which is entered across the cluster, to the service account, which is taken, if you've seen, which is namespace is default, because we are in the, this pod is currently running in the default namespace, and the service account is default, 41:44.430 --> 41:51.910 because you see here, it's the service account is default, when you do, slash var run, secret service account. 41:51.930 --> 42:02.030 So what we are trying to do is, we are creating a new cluster role, which gives full cluster access, to perform anything on all resources, and binding that cluster role, to the default service account. 42:02.030 --> 42:12.510 So now, as you can't get the secrets, or any operations, within the cluster using the default account, once we deploy this Helm chart, you can do anything as a cluster admin, right? 42:12.510 --> 42:15.650 Let's go ahead and run this, deploy this Helm chart. 42:15.710 --> 42:21.210 So we are trying to do Helm, and we are using the same Tiller configuration, and deploying the Pawn chart, right? 42:21.210 --> 42:22.230 Let's go ahead and run. 42:22.230 --> 42:31.710 So it went ahead and deployed the Pawn chart, because Helm has the capability to talk to the Tiller, and Tiller has the full cluster admin access, so it can go ahead and deploy this chart. 42:31.710 --> 42:38.650 So now if you go ahead and run kubectl get secrets, you can able to get the secrets of the entire Kubernetes here, right? 42:38.650 --> 42:47.070 So you can also get kubectl get pods, you can able to get all the pods, and in all namespaces as well, right? 42:47.070 --> 43:01.990 So basically you have full cluster admin access, because currently whatever the service account, which is there in our pod, because due to the reason we have created a cluster role, this cluster admin, sorry, this pod has basically full cluster admin access, 43:01.990 --> 43:05.630 which can able to perform any kind of operations on all resources, right? 43:05.630 --> 43:14.010 So as a attacker, some of the useful commands is like look for auth, can I create pods? 43:14.010 --> 43:23.070 You can check if the current service account, which you own, if you anytime get a service account access, it can able to create pods or something, sorry, create. 43:23.370 --> 43:28.930 You can also check delayed pods, it can able to delay pods or something. 43:28.930 --> 43:32.430 You can also specify, can I do in namespace of kube system? 43:32.650 --> 43:35.970 So this is talk to the API and give you the information. 43:36.170 --> 43:38.510 So this command is pretty useful. 43:38.510 --> 43:43.390 If you got a service account and you wanted to see what kind of privilege it has, you can make these calls, right? 43:43.390 --> 44:00.630 And also some tips, I don't know, we are about to conclude and let me, whenever you make these commands, you can also increase the verbosity of the Kubernetes, you can increase the six or seven, you can able to see each and every request it is making to the Kubernetes cluster and in a verbose way. 44:00.630 --> 44:04.930 Like if you increase this, you can able to see much better, even like headers. 44:04.930 --> 44:09.310 And if we increase like height, like you can even see like a response body and everything. 44:09.450 --> 44:14.150 And if you increase to nine and like little more depth, it will keep going, right? 44:14.150 --> 44:16.370 So this is completely API driven. 44:16.370 --> 44:22.030 So you can able to play around with the verbose and understand exactly each and every thing which is being answered. 44:22.110 --> 44:24.110 And also very useful command is explain. 44:24.110 --> 44:28.430 If you wanted to understand a pod, then it will give you more details about it. 44:28.430 --> 44:34.090 And I wanted to learn more about pod spec, you can also do kubectl spec. 44:34.130 --> 44:35.550 So this is very handy. 44:35.550 --> 44:41.410 And while you are performing assessments, you can quickly look at the documentation rather like explain commands, right? 44:42.030 --> 44:50.010 Cool, so now you see, due to the simple default misconfiguration of our Helm pillar, we can able to control the entire cluster. 44:50.050 --> 44:59.750 And this vulnerability I have seen in so many of the cluster in real-time pentesting assessments due to the administrator doesn't configure properly. 44:59.790 --> 45:03.050 And this is default security installation by Helm itself. 45:03.110 --> 45:07.450 So the newer version of Helm v3 fixed this vulnerability, so it kind of patched. 45:08.770 --> 45:11.230 Cool, so let me go with the last scenario. 45:11.230 --> 45:12.950 I think then we will kind of conclude. 45:12.950 --> 45:16.070 And this you can try, it's a DIND, Docker in Docker and Exploitation. 45:16.070 --> 45:26.050 This you commonly see in CI-CD systems, especially in build pipelines, like GitLab CI-CD or Bitbucket CI-CD or in real-world where they pass the socket, our socket, right? 45:26.050 --> 45:27.250 You can go ahead and try. 45:27.250 --> 45:32.050 And SSRF vulnerability, especially this is very widely exploited in the cloud infrastructure. 45:32.050 --> 45:36.890 But in coming to the Kubernetes context, it is little much, you can get more out of it. 45:36.890 --> 45:42.030 Because as I said, within the cluster, there might not be some kind of services authenticated. 45:42.030 --> 45:47.910 There might be services which are giving more health information of the nodes as well as the ports or internal services. 45:47.910 --> 45:56.490 Using SSRF, you can get more information out of it within the clusters compared to the cloud native, cloud way of exploiting, right? 45:57.290 --> 45:58.210 Cool. 45:58.210 --> 46:00.890 So let me just complete with the... 46:02.490 --> 46:03.930 Yeah, this looks fun. 46:03.930 --> 46:04.470 Cool. 46:04.470 --> 46:18.370 So this is the use case where basically I'm trying to showcase, if you end up in any of the pod and the developer or operations teams didn't set up any resource consumptions, like you can only consume these many resources, right? 46:18.370 --> 46:24.330 So you can able to use utilities, especially like I'm trying to showcase here, stress-ng. 46:24.890 --> 46:27.630 So let me showcase a simple example. 46:27.870 --> 46:30.190 Let me show in the terminal, maybe. 46:30.690 --> 46:35.830 Exit, kubectl get pods. 46:36.050 --> 46:38.210 Now you see a bunch of pods are running. 46:38.210 --> 46:40.210 So I'm looking for a specific pod. 46:40.210 --> 46:46.550 Let's look for that kubectl top pod. 46:47.290 --> 46:50.410 So when you do top, it's basically a system like top command. 46:50.410 --> 46:54.750 It shows how much CPU resources and memory resources it is using. 46:54.810 --> 47:05.130 And if any of the deployment or manifest which doesn't have specifically resource limit setup and the cluster is deployed in like cloud infrastructure, like which has auto-scaling enabled. 47:05.130 --> 47:11.190 If you re-hit this limit, it'll go and scale up a new node or deploys like some more nodes into the cluster, right? 47:11.210 --> 47:15.250 So that times you can exploit this in an attacker's point of view. 47:15.250 --> 47:16.770 Like let's go ahead and run a simple tool. 47:16.970 --> 47:20.390 I'm running with two kicks and 30 seconds. 47:20.530 --> 47:30.290 So now you can see if you're on the same command from the specific resources of 3MB, 1MB, 2MB, like increasing so much, right? 47:30.290 --> 47:31.890 I think it's just taking some time. 47:31.890 --> 47:33.570 Let's keep looking. 47:45.330 --> 47:57.130 You can see it hits a crazy amount of memory resource limits because there was no resource limit set for this deployment, whatever the hunger check, it will be deployed as part of the Kubernetes code. 47:57.490 --> 48:06.550 Attacker can basically consume more CPU resources and memory, which basically cost the billing and very like cost-effective for the company's perspective, right? 48:06.550 --> 48:07.230 Billing. 48:07.230 --> 48:16.110 And due to the reason, the most of the modern infrastructure is set up in such a way which is auto-scalable and like scalable based on the certain parameters. 48:16.230 --> 48:24.030 So you've been able to basically DDoS or gain more resource consumption for the companies and organizations, right? 48:24.030 --> 48:26.590 So this is definitely one thing. 48:26.590 --> 48:32.550 So before moving in, let me just conclude with the Hacker Container preview. 48:32.910 --> 48:39.330 As I said, Hacker Container, let me just showcase maybe here, Hacker Container. 48:39.330 --> 48:44.310 So Hacker Container is a similar, another utility I created as part of the assessment. 48:44.310 --> 48:46.570 So you can read more about it here, the blog post. 48:46.570 --> 48:52.950 So this has a bunch of utilities, which you use day-to-day, like while performing assessments in the Kubernetes cluster. 48:52.950 --> 48:59.390 So I have used it extensively, I would say, while performing assessment and it is very useful utility. 48:59.390 --> 49:00.710 Let's go ahead and run. 49:01.210 --> 49:02.690 Oops, looks like it's already there. 49:02.690 --> 49:06.730 Let's exec into the pod, Hacker Container. 49:06.810 --> 49:17.210 So we are inside the pod, like you see a bunch of other stuff like default tools. 49:17.210 --> 49:22.590 So some of the things like it has is amicontain, so which is again, utility written by Jessica Frazel. 49:22.590 --> 49:29.610 She has done quite a lot of work for the containers and security community in the Kubernetes and the container ecosystem. 49:29.710 --> 49:44.630 So what it does is it will try to identify what container runtime you are using and do you have shared namespaces from the host system like PID, username space, or is there any app armor or sitcom profile applied to the container to restrict certain capabilities? 49:44.810 --> 49:51.490 And is that what all capabilities allowed currently, system calls allowed, and it kind of a container introspection tool. 49:51.490 --> 49:56.150 So which will help you to perform while in attacks and security assessments. 49:56.150 --> 50:01.510 So not only I think specific to the container Kubernetes, then it's a pretty useful tool to try out, right? 50:01.510 --> 50:06.190 So you can also do the same thing with proc self cgroup maybe. 50:06.190 --> 50:07.090 Yeah, cool. 50:07.090 --> 50:13.010 So you can see this is basically trying to figure out kubeports and you can also do capsh, print. 50:13.010 --> 50:16.750 So it will look for what all capabilities are there and similarly. 50:16.810 --> 50:20.190 So this is kind of pretty useful tool, the Hacker Container. 50:20.710 --> 50:25.130 And also it has Nikto and a bunch of other use cases. 50:25.130 --> 50:28.430 I think maybe I try to not document it here. 50:28.430 --> 50:35.470 So I'm going to try to see what can I kind of create out of it and how I use in day to day while exploiting. 50:35.470 --> 50:40.250 And some use cases I would say is very useful is running benchmarks. 50:40.250 --> 50:51.310 If you are inside the clusters and you wanted to just get a glance of view, what kind of vulnerabilities possibly there within the Kubernetes clusters, both in terms of container runtime as well as the cluster level. 50:51.310 --> 50:53.570 You can run these CIS benchmarks. 50:53.630 --> 50:56.970 Like one is you can just go ahead and run this command. 50:57.130 --> 50:58.470 Let's go ahead and run. 50:58.510 --> 50:58.890 Yep. 50:58.890 --> 51:09.050 So what it does is it, I have created something called daemonset, which means each node, how many ever nodes you have in the cluster, will get one pod running in all the nodes. 51:09.050 --> 51:11.550 That is called daemonset in Kubernetes. 51:11.650 --> 51:12.730 Let's go ahead and get the node. 51:12.730 --> 51:14.730 So you can see two nodes are available. 51:14.730 --> 51:22.030 Now you should be able to see two pods running in the Docker Bench because there are two nodes because we are running a daemonset, right? 51:22.030 --> 51:23.610 So you can see two pods running. 51:23.610 --> 51:30.950 So let's go ahead and exec into one of the pod and see what we can do with Docker Bench Security, right? 51:30.950 --> 51:37.650 So Docker Bench Security, I'm execing into the pod. 51:38.390 --> 51:41.570 Now you can see Docker Bench Security. 51:41.570 --> 51:44.650 Now you can go ahead and run dockerbenchsecurity.css. 51:44.650 --> 51:46.710 This is also part of a Hacker Container. 51:46.710 --> 51:48.030 You can go ahead and run this. 51:48.030 --> 52:09.740 So the way I have set it up, sorry, the scenario manifest is if you can see here, I have created the Docker Bench Security in such a way it access the host PID, IPC network, as well as a bunch of volumes. 52:09.740 --> 52:20.460 And it also had extra capability called audit control so that you can access and talk to your underlying Docker Container runtime and you can perform benchmark analysis on possible security issues, right? 52:20.460 --> 52:23.240 So this is a pretty simple utility. 52:23.240 --> 52:24.720 Just go ahead and run this. 52:24.720 --> 52:38.140 Once you are inside cluster, then you will get at least high level posture and use the whatever the found vulnerabilities or the checks as attack paths, or you can use those while you are doing assessments or post-exploitation, right? 52:38.220 --> 52:44.220 Once you are inside a node, maybe you can use some kind of vulnerability which is detected here to do some kind of post-exploitation. 52:44.280 --> 53:01.620 So you can see it does a bunch of checks like host level configuration checks and specific to like the Docker daemon or like directories and daemon configuration, and as well as the files, which is specific to the container runtime here in this case, Docker, 53:01.620 --> 53:12.100 and also container images, whether container running with a root privilege or it has any extra capability set or it doesn't have any seccomp or apparm profiles, these kinds of things, right? 53:12.960 --> 53:16.340 Similarly, you can go ahead and run the kubebench as well. 53:16.810 --> 53:20.300 You can go ahead and run this kubebench like this. 53:21.440 --> 53:23.940 Let me go ahead and run here only, I think. 53:24.860 --> 53:27.400 It's kind of taking quite a lot of time. 53:29.560 --> 53:32.200 So you can see kubebench I have run. 53:32.260 --> 53:33.940 You can give it pods. 53:34.060 --> 53:36.740 The kubebench is running here. 53:36.740 --> 53:39.880 You can get the logs, iPhone app, and the bench. 53:39.940 --> 53:40.960 So it will show. 53:40.960 --> 53:44.520 You can see there are 12 checks passed, nine checks failed, and two checks running. 53:44.520 --> 53:50.020 So you can see what all the vulnerabilities possibly there in this Kubernetes cluster and what would be the remediations. 53:50.020 --> 53:55.820 So you can use these vulnerabilities maybe to do further exploitation and understand what you can do, right? 53:55.820 --> 53:59.600 So these tools are very useful just to get a glance of it. 53:59.600 --> 54:09.500 And there are a bunch of utilities out there as well, like kubehunter and kubepsychaudit and the different golang tool written by different researchers. 54:09.580 --> 54:13.700 And I'll try to document and put it in the same scenarios. 54:13.700 --> 54:15.640 So hopefully you can go back and try out yourself. 54:16.520 --> 54:17.240 Cool. 54:17.380 --> 54:19.420 So there are a bunch of other scenarios. 54:19.500 --> 54:26.280 Due to the time constraint, I couldn't complete or showcase you, but feel free to go and check out the Kubernetes Go project. 54:26.360 --> 54:32.580 And you should be able to get and follow using the playground of that Katakoda, right? 54:32.580 --> 54:38.620 And if you have set it up in your own environment, like in our case, right, my case, so you can go ahead and tear down the cluster. 54:38.620 --> 54:44.860 Basically, it will try to delete possibly resources which I have created as part of the Kubernetes Go deployment. 54:44.860 --> 54:51.640 But I highly recommend you delete the cluster to just ensure cleanup of the entire environment, right? 54:51.780 --> 55:01.600 And I have also showcased and run using a tool called Chekhov to see what all possible vulnerabilities using a CACD as part of pipeline. 55:01.600 --> 55:09.900 So it found a bunch of vulnerabilities in the deployments and other things which I have used as part of Kubernetes Go as well. 55:09.900 --> 55:10.700 All right. 55:11.180 --> 55:12.040 Cool. 55:12.040 --> 55:15.140 So that's it pretty much about it. 55:15.140 --> 55:23.580 And I would highly recommend having your feedback and understand what exactly you want to look forward. 55:23.580 --> 55:31.020 And I'm really looking forward to building more and more scenarios and also adding the defense scenario. 55:31.020 --> 55:41.220 I'm really building like most of the scenarios and making them in such a way that how you can practice for the CNCF certified Kubernetes security. 55:41.220 --> 55:47.940 So you can use this playground to practice and understand how most of the Kubernetes vulnerabilities you can assess and learn more about it. 55:47.940 --> 55:56.180 So feel free to get involved and you can please feel free to provide the feedback and share with your colleagues, teammates, or your friends. 55:56.180 --> 56:01.180 I would really appreciate if you can share the feedback and I would love to improve and work on it. 56:01.180 --> 56:07.680 And if you see any documentation or anything needs to be improved, please let me know by GitHub issues. 56:07.680 --> 56:11.880 And I'm trying to be pretty active, but in the past month I was a little busy. 56:11.880 --> 56:15.440 So I kind of work on this, working on defense scenarios as well. 56:15.440 --> 56:26.680 Hopefully we will be releasing this month for the defense scenarios to understand and fix the vulnerabilities which you have learned while practicing these attacks and also more attacker scenarios as well. 56:27.080 --> 56:32.500 And with that, I think I'm pretty much done with the session. 56:32.500 --> 56:36.800 Thank you so much once again for having me and really looking forward. 56:37.200 --> 56:37.660 Awesome. 56:37.660 --> 56:38.440 Thank you so much. 56:38.440 --> 56:39.920 It was an amazing presentation. 56:39.920 --> 56:41.140 I learned so much. 56:41.160 --> 56:45.860 And as a matter of fact, in Discord, there are a lot of people actually talking about your presentation. 56:45.940 --> 56:48.580 And once again, thank you for supporting the communities. 56:48.580 --> 56:51.040 Thank you for supporting the DEF CON Red Team Village. 56:51.360 --> 57:00.000 We're going to go on a little break, but we invite everybody that will have any further questions and so on to join the Discord channel, which is in the description of the video.