[00:01.280]Ah, come back here. Why does that keep doing that? [00:05.120] [00:05.600]All right, great. We are here and ready to do securing in, [00:09.560] [00:09.560]path testing, great spaghetti monster, as I like to call it, or [00:13.280] [00:13.280]Kubernetes. This is who I am, Kat Fitzgerald. [00:18.400] [00:18.400]You can find me on Twitter at Rainbow Kat. I've been doing this a long time. [00:22.840] [00:22.840]I'm the CEO of B-Sides Chicago. I was COO for [00:26.580] [00:26.580]the Diana Initiative last year, part of B-Sides Pittsburgh, [00:30.820] [00:30.820]and also my first DEFCON was DEFCON 3. [00:34.900] [00:34.900]How many other people remember that one? There were 16 [00:38.860] [00:39.140]speakers there, if I have to remember that. [00:42.560] [00:42.840]I have an emphasis in the blue team. However, I'm a formal purple teamer [00:46.820] [00:47.160]and all this stuff. And I'm in Pittsburgh now, but moving to [00:50.880] [00:50.880]Kirkland, Washington very, very soon. You will always [00:54.760] [00:54.760]find me sipping Grand Mayan Extra Añejo, or in [00:58.800] [00:58.800]this case, it is Casanoble Añejo. [01:03.740] [01:03.740]So, cheers. Let's kick off DEFCON. Jason [01:06.840] [01:06.840]already did a great job. [01:08.960] [01:11.120]And also, special thanks to the entire Red Team Village, [01:15.240] [01:15.240]all the staff, all the volunteers, everybody for kicking butt [01:18.720] [01:18.720]and making this happen. A lot of my favorite stuff are honey pots [01:22.980] [01:22.980]and refrigerators. On Saturday, I'm doing a talk for [01:26.640] [01:26.640]IoT security, so check that out, too. But this is one of my favorite ones [01:30.920] [01:30.920]because I just finished writing this particular [01:34.300] [01:35.560]presentation just about a month ago, made some changes to it, and so on. [01:39.380] [01:39.380]Remember, please, the views and opinions here are mine and mine [01:43.000] [01:43.000]alone. They do not reflect anything from my employers, past [01:47.000] [01:47.000]or present. Anything that I show you, [01:50.940] [01:50.940]as always, please use with a grain of salt. Don't go breaking [01:54.780] [01:54.780]into things that you shouldn't be breaking into. And those of you with an [01:58.820] [01:58.820]overwhelming fear of the unknown, you'll be happy to know that [02:02.320] [02:02.320]there is no hidden message in the disclaimer if you read it [02:06.720] [02:06.720]backwards. Yes, my jokes are that bad. [02:10.340] [02:10.380]Alright, let's get this thing on the road. So, why are we [02:14.920] [02:14.920]not here? Well, first of all, I'm not going to solve all [02:18.760] [02:18.760]your Kubernetes security woes. I wish I could. [02:22.640] [02:22.640]I wish Kubernetes and containers and everything were secure by default. [02:27.180] [02:27.180]They are not. Neither is the person [02:30.980] [02:30.980]sitting next to you, although, well, that doesn't work anymore because [02:34.520] [02:34.520]nobody's sitting... well, maybe somebody's sitting next to you. You never know. [02:38.520] [02:38.520]You never know. Okay. And common sense went out the window [02:42.880] [02:42.880]decades ago. Think about that. When it comes to [02:46.540] [02:46.540]information security, cyber security, hacking, whatever you want to call it, [02:50.940] [02:50.940]I honestly think we lost common sense quite [02:54.340] [02:54.340]some time ago. Why really are [02:58.560] [02:58.560]we here? Well, Kubernetes, I [03:02.520] [03:02.520]don't know. I still think it's fairly new. It's fun. [03:06.740] [03:06.740]It's crazy. It's a lot of things. [03:10.420] [03:10.420]You can deploy it in clouds. You can deploy it in... [03:14.560] [03:14.560]I have Raspberry Pis sitting down here. I was [03:18.500] [03:18.500]going to be showing that. There have been some demo problems and also some [03:22.280] [03:22.280]timing issues, so I won't be showing my Raspberry Pi cluster today. [03:26.520] [03:26.520]But you can build very extensive Raspberry Pi [03:30.160] [03:30.160]clusters and do all the things that I'm going to talk about today. [03:34.780] [03:35.340]Containers. Containers are not new. [03:38.800] [03:38.800]But people seem to have forgotten that they [03:42.400] [03:42.400]have anything to do with security. They get all [03:46.460] [03:46.460]hyped up on Kubernetes security, and they're forgetting about [03:50.500] [03:50.500]containers. We're going to talk about containers because it's [03:54.240] [03:54.240]important. It's, in fact, very important. And always remember this. [03:58.600] [03:58.600]Security is for everyone. Common [04:02.480] [04:02.480]sense is required. So please [04:06.080] [04:06.080]disregard the previous slide. Throw it out the window. Why? [04:10.340] [04:10.340]Because if we're not using common sense in security, then [04:14.560] [04:14.560]we're definitely not going to get where we need to be. [04:18.220] [04:18.640]All right. Just some quick security facts. We want to [04:22.380] [04:22.380]know this stuff. In 2018 alone, $114 [04:26.640] [04:26.640]billion were spent by companies on [04:30.620] [04:30.620]information security. On fancy software, [04:34.580] [04:34.580]fancy hardware, things with blinky lights and all of that. [04:39.100] [04:39.100]But breaches kept on going up. We're doing [04:42.980] [04:42.980]something wrong here. Security appliances and software [04:47.120] [04:47.120]have vulnerabilities that we're not dealing with or that [04:51.060] [04:51.060]need to be dealt with by the vendors. Kubernetes itself [04:55.320] [04:55.320]has had a few issues. We'll talk about that. But so has Docker. [04:59.700] [04:59.700]So don't forget your containers. Also, I've [05:03.160] [05:03.160]actually had people say to me, we have a very unique security [05:07.400] [05:07.400]architecture. And I'm like, no, you don't. [05:11.460] [05:11.560]We all do the same stuff. So remember, your architecture [05:15.280] [05:15.280]is not unique. I love this quote. If you don't have this [05:19.360] [05:19.360]book, Offensive Countermeasures by John Strand, go out [05:23.500] [05:23.500]and get it. It's a small book with a ton of information. [05:27.220] [05:27.220]So get it. But he said, instead of brilliance, [05:31.280] [05:31.280]we've standardized on mediocrity. And that seems to be [05:35.840] [05:35.840]what information security is about these days. [05:40.100] [05:40.100]And I kind of have issues with that. So let's talk about [05:43.800] [05:43.800]that some more. First of all, let's get a common definition [05:48.040] [05:48.040]of what is a breach. Or maybe we'll call it [05:51.920] [05:51.920]what is an incident. Let's not necessarily call it a breach. [05:56.200] [05:56.200]So most incidents are not caused by zero-day [05:59.520] [05:59.520]fancy stuff that you find on the Internet. Typically, [06:04.620] [06:04.620]they're done because people have made mistakes and so on. [06:08.160] [06:08.160]Most breaches are not coming from your vulnerability scanners, [06:12.140] [06:12.140]especially when it comes to Kubernetes. You're not going to run a vulnerability [06:16.120] [06:16.120]scanner and go, oh, look, fix this one thing, and we're [06:20.360] [06:20.360]done and ready to go. No, it doesn't happen that way. [06:24.120] [06:25.420]Most breaches are going to come from [06:28.000] [06:28.580]configuration issues. This is something I want [06:32.380] [06:32.380]to emphasize. In fact, I'm going to show you some examples how [06:36.160] [06:36.160]configuration issues tend to cause most of the breaches. A close [06:40.380] [06:40.380]second is compromised credentials. [06:44.460] [06:44.500]When I say credentials, I still go out on GitHub. [06:49.040] [06:49.240]Well, maybe not every single day. And I think Jason [06:52.280] [06:52.280]even mentioned it, too, in his previous talk on bug bounties. [06:56.080] [06:56.320]Go out and look on GitHub and search for simple things [07:00.540] [07:00.540]like password. See how many times you find [07:04.520] [07:04.520]the word root. Yes, thousands of them [07:08.520] [07:08.520]are invalid. They don't make any sense. It's probably going to find the keyword [07:12.700] [07:12.700]root or password, and it's not going to get you anywhere. [07:17.140] [07:17.140]And yet I still find where people hard-code credentials [07:20.540] [07:20.540]into variables that say password. And right [07:24.540] [07:24.540]next to it, there's the key. There's the whatever the password [07:28.600] [07:28.600]is. Also, trailing in third [07:32.560] [07:32.560]place for most incidents is overprivileged [07:36.780] [07:36.780]users. This is incredibly important in [07:40.620] [07:40.620]the Kubernetes world. I'm going to show you this in just a second [07:44.480] [07:44.480]or in a couple of slides here. So keep these three things in mind. [07:49.160] [07:49.160]Configuration issues, compromised credentials, and overprivileged [07:52.920] [07:52.920]users. Okay. Kubernetes [07:56.800] [07:56.800]itself, we have to deal with all three [08:00.820] [08:00.820]of these compromises or these types [08:04.720] [08:04.720]of incidents that are going to cause issues in Kubernetes. [08:09.880] [08:09.960]But first, think [08:12.980] [08:12.980]about this. You know, we think about the Capital One hack [08:16.960] [08:16.960]that happened recently. Well, that was a combination [08:20.980] [08:20.980]of compromised credentials and access [08:24.820] [08:24.820]that should not have been granted. We see how [08:29.000] [08:29.000]Tesla, you know, they had poor access credentials [08:32.980] [08:32.980]and so on. Equifax, we all know about that one. [08:37.160] [08:37.160]Go back and check if you don't. You know, Equifax [08:40.580] [08:40.580]was hacked not only by a simple vulnerability [08:44.300] [08:44.820]that had not been patched, but also because they had very [08:48.980] [08:48.980]very poor lateral movement detection on the [08:53.040] [08:53.040]inside. Once they got in with more compromised [08:57.000] [08:57.000]credentials through the vulnerability, they weren't [09:01.240] [09:01.240]detected for quite some time. This is going to be [09:05.180] [09:05.180]critical. And one other thing, I was bored this morning. I decided [09:09.220] [09:09.220]to throw this up here in this slide. Why? Well, I [09:13.140] [09:13.140]put up a new spice rack in my kitchen. I really liked how it looks. [09:17.060] [09:17.060]I took a picture and I added it to the slide. There it is. [09:21.220] [09:21.220]Isn't it a great spice rack? Come on. Alright, you all better [09:25.240] [09:25.240]be laughing. Alright, let's talk about containers. [09:29.460] [09:30.180]Quick review of containers. They are not [09:33.160] [09:33.160]secure by default. Now, I'm really focusing more on [09:37.400] [09:37.400]Docker containers for now because that tends to be [09:40.980] [09:40.980]the norm. And one thing to think about when it comes to [09:45.040] [09:45.040]containers, containers are not like [09:48.760] [09:50.000]a VM. A VM is actually more [09:52.740] [09:52.740]isolated. And this is something that's going to be important coming [09:56.840] [09:56.840]up here. So, a container is a grouping of stuff [10:00.760] [10:00.760]and the stuff that we like to call it are resources. Well, the [10:04.660] [10:04.660]resources are grouped into namespaces. If [10:08.820] [10:08.820]you are new to containers, remember this. Namespaces [10:12.740] [10:12.740]are critical. You group your processes, [10:16.860] [10:16.860]your networks, your users, your ports, your interprocess. All [10:20.980] [10:20.980]of this stuff can be controlled and [10:24.680] [10:24.680]locked down within namespaces. Resources [10:28.960] [10:28.960]are exploitable. I was sitting in another [10:32.840] [10:33.980]web broadcast just the other day [10:36.640] [10:36.640]on containers and I was [10:40.240] [10:41.660]surprised because it was supposed to be a more [10:44.740] [10:44.740]intermediate presentation on containers and [10:48.820] [10:48.820]yet all of the people were asking is there a [10:53.060] [10:53.060]way to prevent process [10:56.740] [10:57.560]abuse such as CPUs and memory and [11:00.960] [11:00.960]so on. And again, this was supposed to be an intermediate [11:04.740] [11:05.560]class on containers so it [11:09.100] [11:09.100]confused me a little bit that people were asking that because that should be something [11:12.860] [11:12.860]that you know from the very beginning that you get into containers. [11:17.740] [11:17.740]Using namespaces and the next [11:20.920] [11:20.920]most important thing is control groups or cgroups. These [11:25.120] [11:25.120]are the limits that you are going to be putting on containers. [11:29.460] [11:29.460]If you don't put limits on them, you're [11:32.820] [11:32.820]going to run into problems. Every time I do a pen test [11:37.180] [11:37.180]of a new application deployed into a container, [11:41.320] [11:41.320]I would say that 7 out of 10 times [11:45.500] [11:45.500]I find that the author of the application [11:49.340] [11:49.340]who has put it in a container has not put any limits. [11:53.220] [11:53.220]I'll get into the application in whatever way possible whether it's [11:57.540] [11:58.140]a SQL injection or an RCE [12:01.180] [12:01.180]or something like that. And once I'm in, I can take over not just the [12:05.400] [12:05.400]container but the entire system that it's on [12:09.040] [12:09.040]because there are no cgroups applied to the container. [12:13.800] [12:13.800]The other thing to remember about containers [12:17.120] [12:17.120]is think of a container as an image or more [12:20.960] [12:20.960]correctly, think of it as a file system snapshot. [12:25.880] [12:26.060]So who decides what goes [12:28.880] [12:28.880]in that image? If you look at this particular picture, [12:34.220] [12:34.220]you'll see, you know, here we have [12:36.900] [12:36.900]namespaces, the control groups that I show. [12:40.200] [12:40.480]It's going to possibly be cherooted if you've set it up [12:44.780] [12:44.780]correctly. And here's the application. [12:48.860] [12:49.400]There's a Docker client that you might use to launch [12:52.840] [12:52.840]this. There's a Docker registry. This is critical [12:57.120] [12:57.120]to the security of your system. If you're [13:00.940] [13:00.940]new to Docker, you may not know this, but Docker Hub was compromised [13:05.320] [13:05.320]two years ago, I believe it was. I've lost track [13:09.400] [13:09.400]of time during COVID. I don't know anymore. [13:12.520] [13:12.860]But you want to have a secure registry where you keep [13:17.160] [13:17.160]your images because if they're somewhere [13:21.000] [13:21.000]that you don't have control, you never know what's going to happen. [13:25.800] [13:25.800]But here's the difference. Notice that we have a host OS [13:29.020] [13:29.020]and hardware down here, unlike a VM. [13:34.280] [13:34.280]In a VM, you would see the entire operating system [13:38.600] [13:38.600]within the VM here. You would not see a shared [13:42.520] [13:42.520]OS. Yes, there's a host OS, but then you also have the guest [13:46.560] [13:46.560]operating systems themselves. And that's where we get into [13:50.580] [13:50.580]how a VM is much more isolated [13:54.580] [13:54.580]and contained than a container is. So [13:58.300] [13:58.300]don't think the word container means that you're more secure. [14:02.540] [14:02.540]It just means that we have some work to do. So that's why [14:06.460] [14:06.460]I say right here in this slide, container isolation. [14:10.120] [14:10.320]Is it really isolated? The simple answer to that is [14:14.740] [14:14.740]no, it is not. So let's talk about some [14:18.500] [14:18.500]container best practices. I should have also said at the very [14:22.700] [14:22.700]beginning of this talk, this is an intro. [14:26.540] [14:26.540]I can't show you everything. I'm trying to get people to start [14:30.360] [14:30.360]thinking about how you secure containers, then [14:34.320] [14:34.320]we're going to jump into how you secure Kubernetes at the [14:38.340] [14:38.340]same time. So remember, we have [14:42.320] [14:42.520]a base image. That means that base image has to come from [14:46.300] [14:46.480]a secure repo. If you do not know what CIS [14:50.000] [14:50.000]is, definitely go to CIS [14:54.200] [14:54.200]or CISecurity.org [14:57.820] [14:58.460]Check it out because you want to use CIS [15:01.860] [15:01.860]benchmarks for locking down your base image. I'm going to [15:05.900] [15:05.900]talk about that a little more coming up. Don't run your images [15:10.020] [15:10.020]as root. By default, if you do not set a user [15:14.040] [15:14.040]name, your container and your application in the container is going [15:17.920] [15:17.920]to run it as root. I have found so many [15:21.480] [15:21.480]who have deployed WordPress applications [15:26.040] [15:26.040]in a Docker container via Kubernetes [15:29.220] [15:29.980]and they don't set the user name and WordPress is running [15:34.000] [15:34.000]as root, so is the database and everything else. So of course, [15:37.960] [15:37.960]once it's compromised, you take over everything. Also, [15:42.400] [15:42.400]visibility. You need monitoring within [15:46.160] [15:46.160]your containers. Your containers put a lot of information [15:49.880] [15:49.880]out and they can log it, but you have to capture that. We're going to talk [15:54.180] [15:54.180]about that in a minute. Also, think of [15:58.220] [15:58.220]it as a domino. If one container is [16:02.140] [16:02.140]compromised, I guarantee you the rest of them will [16:06.280] [16:06.280]fall over like dominoes. [16:09.260] [16:09.880]When you start to get into containers and Kubernetes, you're going to see this. [16:14.520] [16:14.920]Also, we want to set some controls. [16:18.860] [16:18.900]Besides setting a user name that is not [16:22.040] [16:22.040]root, we also set privileges and we set [16:25.940] [16:25.940]certain attributes to those users where it says [16:29.940] [16:29.940]no elevated privileges or no new privileges. [16:33.780] [16:33.780]You don't want user accounts or IDs [16:37.720] [16:37.720]to be able to escalate privileges. And I'll [16:42.020] [16:42.020]show you some examples coming up. This one, my god, this is [16:46.080] [16:46.080]near and dear to my heart, secrets management. I still [16:50.020] [16:50.020]find, I would love to say by a show of hands, [16:54.180] [16:54.180]but I can't do that right here, how many people [16:58.040] [16:59.480]put the secrets [17:02.060] [17:02.060]and tokens and things like that in their Docker [17:05.760] [17:05.760]YAML files so when they then deploy the Docker [17:09.720] [17:09.720]image, if somebody gets a hold of their YAML file, which [17:14.040] [17:14.040]describes the configuration of that container, [17:18.340] [17:18.340]well, the secrets are right in it. I've worked [17:22.340] [17:22.340]with a lot of developers and a lot of engineers and I get [17:26.460] [17:26.460]them, you know, you have to start thinking about putting [17:30.320] [17:30.320]your secrets in another place, whether it's HashiCorp [17:34.360] [17:34.360]Vault, whether it's a cloud secrets management system, [17:38.240] [17:38.240]something. But get them out of the apps and get them out of the [17:42.420] [17:42.420]configuration files. And stop allowing SSH [17:46.480] [17:46.480]into your hosts. When we get into [17:49.740] [17:50.660]Kubernetes, coming up in a couple slides here, [17:54.340] [17:54.340]one of the key constructs of Kubernetes is you have a master node [17:58.600] [17:58.600]or master nodes, and then you have worker nodes. And I [18:02.660] [18:02.660]still find worker nodes with SSH enabled [18:06.500] [18:06.500]on them so people can SSH in and, quote, fix [18:10.380] [18:10.380]things. No, no, you use the [18:14.640] [18:14.640]management tools to fix things and redeploy. Stop [18:18.820] [18:18.820]allowing SSH into your systems. Also, I can't [18:22.940] [18:22.940]stress it enough times, namespace [18:26.880] [18:26.880]limits. Go read up on how to [18:30.840] [18:30.840]control namespaces. That is going to [18:34.820] [18:34.820]be one of your best efforts and best [18:38.800] [18:38.800]practices for containing and controlling and [18:42.760] [18:42.760]securing a container before you distribute it to [18:46.320] [18:47.240]or deploy it via Kubernetes. This is one of [18:50.780] [18:50.780]my favorite tools. It's called Docker Slim. If you've never heard [18:54.800] [18:54.800]of it, if I were to create a container, [18:58.760] [18:58.760]let's say I create a container, it has Ubuntu in it [19:02.340] [19:02.340]or the base of Ubuntu, and then maybe it has WordPress [19:06.260] [19:06.900]running in it for whatever god awful reason that I want to do this. [19:11.020] [19:11.020]Let's say that entire image ends up being [19:15.180] [19:15.180]three to four hundred meg. It's still [19:18.700] [19:18.700]manageable, but if I were to run that through Docker Slim, [19:23.300] [19:23.300]not only will it add some extra configurations [19:27.400] [19:27.400]and namespace limits and so on, but it shrinks it down [19:31.380] [19:31.380]and something that might be three or four hundred megabytes [19:34.840] [19:34.840]deployed into a container will get shrunk down to [19:39.180] [19:39.180]maybe 10, 15, 20 meg. This [19:43.200] [19:43.200]makes it even that much more manageable. It's easier to [19:46.960] [19:46.960]deploy. It's quicker to deploy, especially [19:50.880] [19:50.880]if there are requirements for scaling. [19:55.260] [19:55.260]So think about this. Take a look at this. I also mentioned [19:58.680] [19:58.680]in Discord that my slides will be available in my GitHub [20:03.000] [20:03.000]repo, and I'll get them to the Red Team Village so they can add them [20:07.140] [20:07.140]to the Google Docs location after my talk. [20:12.000] [20:13.260]And always remember [20:15.340] [20:15.340]apply SELinux or AppArmor. One of the things [20:19.200] [20:19.200]that Docker Slim does is it will do that for you. [20:23.120] [20:23.120]It's very easy to enable SELinux or AppArmor [20:27.080] [20:27.080]within a container. If you can do that, you're going to make [20:31.300] [20:31.300]it that much more secure. It's not on by default, but you can set [20:35.220] [20:35.220]it up so it will be. [20:36.960] [20:39.140]And remember the infrastructure, as I put it. [20:44.040] [20:44.640]You [20:45.400] [20:47.120]can't forget that all of this stuff is being deployed on [20:51.180] [20:51.180]an infrastructure of some sort, and that infrastructure better [20:55.160] [20:55.160]be secure. I mentioned SSH. People keep [20:59.180] [20:59.180]allowing SSH into their environment. Be very [21:03.440] [21:03.440]aware that your infrastructure itself should be secure. And I will [21:07.500] [21:07.500]remind you later about that. All right. Here's a quick picture. [21:11.280] [21:11.280]Picture is worth a thousand words, right? If you look at our first example, [21:15.720] [21:15.720]this is our monolithic operating system here. We see [21:19.360] [21:19.360]hardware and operating system. We run apps. Here we see a VM. So we see [21:23.560] [21:23.560]hardware operating system. There's a hypervisor involved. [21:27.300] [21:27.900]And then there's going to be virtual machines. But when you have virtual [21:31.320] [21:31.320]machines, you have additional operating systems, you have additional [21:35.100] [21:35.100]libraries. They're very much isolated. I should have used a different [21:39.480] [21:39.480]picture where there's a big line through it that [21:43.260] [21:43.260]shows, yes, these are isolated and so on. But [21:47.040] [21:47.040]that should not have happened. Nope. Go back. [21:51.380] [21:51.540]Sorry. Okay. But when we look at [21:54.720] [21:55.320]using Docker or Container, we go hardware operating [21:59.300] [21:59.300]system. There is a container runtime. There's no real dividing lines. [22:03.680] [22:03.680]And then we see containers. The only thing that is unique between [22:07.400] [22:07.400]the containers is that bin library, the things [22:11.380] [22:11.380]that are going to run in the application. So there's not a lot of [22:15.340] [22:15.340]isolation there. And people want to [22:19.360] [22:19.360]think that there is. But there is not. So please keep [22:23.460] [22:23.460]that in mind. I do love this [22:27.080] [22:27.080]comic. It came out a couple of years ago. A great [22:30.940] [22:30.940]defense is, you know, we'll inundate them [22:35.040] [22:35.040]with quarterly Kubernetes releases. Yes, they do update [22:39.100] [22:39.100]Kubernetes a lot. That's why you have to pay [22:42.820] [22:42.820]attention to your environment and the tools that you're going to use [22:46.680] [22:46.680]so you can do upgrades to your Kubernetes cluster. So let's bring on [22:51.140] [22:51.140]the orchestrator. Let's talk about Kubernetes. All right. Here [22:55.020] [22:55.020]is Kubernetes 101. In one slide, I'm going to teach you [22:58.840] [22:58.840]at least the basics of how Kubernetes [23:02.780] [23:02.780]works. We have a master node. The master node is [23:07.020] [23:07.020]critical. It manages all of the Kubernetes [23:10.560] [23:10.560]worker nodes. It manages the clusters. It deploys [23:14.520] [23:14.520]pods. Think of a pod as a collection [23:18.460] [23:19.020]of containers that are grouped. [23:23.000] [23:23.000]A pod might be the way that you [23:26.960] [23:26.960]decide, oh, I want to have a pod that supports a particular [23:30.880] [23:30.880]client. Or maybe it's a WordPress instance. And that [23:34.880] [23:34.880]pod has WordPress. It has a container, a [23:39.280] [23:39.280]separate container for the database and so on. So [23:43.140] [23:43.140]the pod is how you can group things that [23:47.320] [23:47.320]are various containers into certain groupings. [23:51.180] [23:51.180]And those groupings are kind of defined by you. But stop and think about [23:55.260] [23:55.260]how you want to deploy pods. It's going to be important. [23:58.920] [23:58.920]A worker node. The node is where all the containers are going to run. [24:03.580] [24:03.580]You have other Kubernetes components that have to [24:07.200] [24:07.200]run there. But the important part is, this is where the containers [24:11.020] [24:11.020]run. If the worker node is not secure [24:14.760] [24:14.760]and not patched, and I can get into that [24:19.020] [24:19.020]underneath the visibility of [24:22.640] [24:22.640]Kubernetes, then Kubernetes has no idea that I've compromised [24:27.000] [24:27.500]a node. I found numerous Kubernetes sites around [24:31.160] [24:31.160]the Internet where there's [24:34.540] [24:34.540]additional applications running on worker nodes because they've been compromised. [24:39.300] [24:39.300]Of course, they're Bitcoin miners. But the master node has no [24:43.080] [24:43.080]idea that it's running on there because they got in via [24:46.980] [24:46.980]SSH and they launched the application, the [24:50.740] [24:50.740]miner, underneath the visibility of Kubernetes. [24:54.900] [24:54.900]So you really want to lock down these worker nodes. A pod, [24:59.100] [24:59.100]I already explained, it's a unit of deployment and it's also how [25:02.900] [25:02.900]you address things from Kubernetes. And a pod [25:07.080] [25:07.080]will have an IP address. So that's important to [25:10.520] [25:10.520]remember. But it's going to contain one or more containers. [25:15.080] [25:15.080]I like to think of pods as, even though it might have [25:18.980] [25:18.980]its own IP address, but think of it like a VLAN in a way. [25:22.840] [25:22.840]Because I've isolated it in one way. Services [25:26.900] [25:26.900]are various things that run within Kubernetes [25:30.200] [25:30.980]that communicate, that balance, that do a lot of work. We're going to look [25:34.920] [25:34.920]at services coming up. There are various components that are [25:38.840] [25:38.840]used. Three that are critical, the API [25:42.760] [25:42.760]server, you're going to have kubelets, and you're also [25:46.640] [25:46.640]etcd. The one concept you should take [25:51.000] [25:51.000]about etcd is this is the database [25:55.040] [25:55.040]that describes and builds everything [25:58.700] [25:58.700]that is your Kubernetes cluster. The etcd [26:02.580] [26:02.580]environment should actually be walled off [26:06.280] [26:06.280]and firewalled and controlled very, very [26:10.560] [26:10.560]securely. If you do that, you will have a much [26:14.580] [26:14.580]more secure Kubernetes cluster. So keep [26:18.720] [26:18.720]that in mind. And you can see it here by looking at a picture. [26:23.040] [26:23.040]So we take all of those concepts that we just talked about [26:26.420] [26:26.420]we see here's an ABI server, here's a controller, [26:31.000] [26:31.000]this is all part, you know, the kubectl... [26:33.660] [26:33.660]Stop doing that. Come on. Kubectl [26:38.640] [26:38.640]is the command you use to control everything. [26:42.640] [26:43.020]And even though this is a disk kind of a representation [26:46.580] [26:46.580]of etcd, I like to say, you know what, this needs to be off [26:50.740] [26:50.740]on its own, firewalled, and separated, and it can be done that way. [26:54.920] [26:54.920]Here's your worker node, and you can see how you might have multiple [26:58.420] [26:58.420]pods on a worker node. This pod might be talking to [27:02.440] [27:02.440]this pod. It doesn't necessarily have to talk to that one, or [27:06.620] [27:06.620]they may not talk at all. These are some of the other services that are running [27:10.720] [27:10.720]the proxies. This is where we are exposed to the internet, [27:14.540] [27:14.540]not over here. Okay? So the only place you're [27:18.500] [27:18.500]exposing Kubernetes is at the worker nodes. [27:22.240] [27:22.780]Now, Kubernetes is not perfect. [27:26.280] [27:26.520]There have been problems. There was... [27:29.980] [27:29.980]I need to update this slide again, because there have been a couple of other [27:34.640] [27:34.640]Kubernetes. These are more for reference. One's a [27:38.900] [27:38.900]little older. This was back... there was the backdoor that involved the [27:42.800] [27:42.800]API server. There were some others. These are only for your reference, [27:46.760] [27:46.760]so you can go look them up and see. And there are [27:51.080] [27:51.080]more. The whole point I'm trying to make here is [27:54.720] [27:55.300]Kubernetes is not secure by default, and it's not [27:59.480] [27:59.480]perfect. Well, let's try to make [28:02.760] [28:02.760]it perfect. Let's take a common Kubernetes cluster [28:06.500] [28:06.500]and say, you know, if we were to threat model this, [28:10.920] [28:10.920]and that is very, very important, first thing [28:14.680] [28:14.680]we want to do is to say, TLS everywhere. Yeah, I know, I have [28:18.800] [28:18.800]too many exclamation points, but I'm trying to make the point. If [28:22.620] [28:23.640]everything within your Kubernetes cluster is using TLS [28:26.820] [28:26.820]and is encrypted, then it's that much [28:30.640] [28:30.640]safer. I still have engineers come to me and say, [28:34.700] [28:34.700]our application is completely internal. Why do we [28:38.620] [28:38.620]need TLS? It's never going to be exposed to the internet. [28:42.680] [28:42.680]I said, really? So what happens if Joe's application [28:46.560] [28:46.560]over here, which is exposed to the internet, is compromised, and then [28:50.680] [28:50.680]they use that as a pivot point to gain access to this environment, [28:55.000] [28:55.000]and then they compromise your system? O [28:58.380] [28:58.380]is typically the response. So I can't say it [29:02.480] [29:02.480]enough, harden the bloody infrastructure. Yes, I'm saying [29:06.440] [29:06.440]it again, but you're running this thing on your infrastructure, [29:11.060] [29:11.060]make sure it's hardened. And very important, [29:14.320] [29:14.320]two concepts. There's RBAC, which is [29:18.180] [29:18.180]regular role-based access control. Set that up. [29:22.780] [29:22.780]Make sure you're doing it right. Disable [29:26.000] [29:26.000]ABAC. ABAC is [29:28.600] [29:30.300]incredibly difficult to manage. You [29:33.960] [29:33.960]will spend days and hours and hours trying [29:38.000] [29:38.000]to do ABAC access controls, [29:41.920] [29:41.920]which is more granular, but it's not worth the effort. [29:46.000] [29:46.000]Make sure you use RBAC and manage it correctly. [29:50.060] [29:50.060]And I'm going to show you a couple of tools coming up [29:53.700] [29:53.700]on how to check it. Also, check out this blog [29:58.000] [29:58.000]entry. I mentioned this previously about [30:01.920] [30:01.920]monitoring. This is one way to audit [30:05.520] [30:05.520]Kubernetes, so keep this in mind. Take a look at that [30:09.560] [30:09.560]blog entry from Wazoo, and it's an excellent [30:13.140] [30:14.780]place to start when it comes to monitoring [30:17.680] [30:17.680]your environment. Also, kubectl, get role [30:21.960] [30:21.960]binding in all your namespaces. You should do this [30:25.960] [30:25.960]periodically in your environment, especially [30:29.280] [30:29.720]if you have engineers deploying their own [30:33.680] [30:33.680]applications. Make sure that all the namespaces [30:37.920] [30:37.920]have good controls in their RBAC accounting. [30:42.300] [30:42.300]Also, use third-party authentication for the API [30:46.040] [30:46.040]server. Again, something like HashiCorp Vault. Yes, I love [30:50.000] [30:50.000]that tool. If you are not familiar with it, get familiar with [30:54.100] [30:54.100]it. It's a great tool, and it integrates very, very well with your [30:58.000] [30:58.000]API server and Kubernetes. [31:00.360] [31:01.680]I already mentioned this. Separate and firewall your etc cluster. Make sure [31:06.120] [31:06.120]it's protected. That is the keys to the kingdom. [31:09.800] [31:09.880]It's the entire configuration of your cluster. And rotate [31:13.700] [31:13.700]encryption keys. Yes, use Vault. And by [31:17.880] [31:18.380]default, turn on security policies in the pods. [31:22.860] [31:22.880]It's some simple lines of configuration [31:25.860] [31:25.860]in the YAML files. It's not hard to do. Or [31:29.440] [31:29.440]use Docker Slim, that tool that I mentioned earlier, because [31:33.920] [31:33.920]that will help you set this up. Alright, let's talk [31:38.000] [31:38.000]about risk. kubectl, get secrets. That [31:41.920] [31:41.920]is the one that I just mentioned where you can look at the secrets [31:45.860] [31:45.860]from pretty much everything. But here's some tools [31:49.840] [31:49.840]that I absolutely love by CyberArk. One is called Kubernetes [31:53.740] [31:53.740]RBAC audit. The other is called KubeScan. [31:58.260] [31:58.260]The RBAC audit does exactly what you think it does. It audits [32:02.100] [32:02.100]all of your RBAC roles. The KubeScan [32:05.560] [32:05.560]will look at all the permissions that are granted from [32:10.000] [32:10.000]the roles. These are two great tools that [32:14.020] [32:14.020]you want to put into your toolbox for securing [32:17.860] [32:17.860]Kubernetes clusters. Because remember at the very beginning, I said [32:22.320] [32:22.320]it's all about configurations. It's the number one [32:26.320] [32:26.320]area that Kubernetes clusters get [32:30.040] [32:30.040]compromised. Also, we want to take a look [32:34.260] [32:34.260]at GitHub. Go out and make sure [32:38.260] [32:38.260]that... take that away... go out and make sure that [32:42.040] [32:42.040]your own configurations are not being [32:46.260] [32:46.260]shared up to GitHub by your engineers accidentally. [32:50.880] [32:51.080]How many times does this happen? It happens. [32:54.340] [32:54.340]I've had it happen by engineers. They don't do it on purpose. [32:58.580] [32:58.580]But they upload code. And in the code are secrets. [33:01.700] [33:01.920]Oops. Oops is not a good thing. [33:05.540] [33:05.760]Also, you need a locked down secure image repo. [33:10.400] [33:10.400]And create gold images. Don't just let your [33:14.380] [33:14.380]engineers go out and do, hey, docker run [33:18.880] [33:18.880]bring down Ubuntu and pull it from the internet. No. [33:22.680] [33:22.680]You need your own repo and registry for your docker [33:26.380] [33:26.380]images. Make sure that happens. And please secure the [33:30.280] [33:30.280]libraries. That's another big one. We also... [33:34.720] [33:34.720]I need an entire slide on just the applications. [33:39.120] [33:39.940]I'm going out on a limb. This is my personal opinion. [33:43.840] [33:43.840]But I want to say that nine times out of ten... okay, [33:47.180] [33:47.180]maybe eight and a half. Eight and a half times out of ten [33:50.740] [33:50.740]Kubernetes clusters and containers are compromised [33:54.600] [33:55.120]because the applications themselves [33:58.180] [33:58.980]were poorly written and not tested. [34:02.860] [34:02.980]And that's what gets me. I have [34:06.900] [34:06.900]environments where I see the engineers create [34:10.720] [34:10.720]an application. They do so much work to lock down [34:14.660] [34:14.660]Kubernetes in the container, but they do a poor [34:18.880] [34:18.880]job and there's a SQL injection vulnerability in the application directly. [34:23.260] [34:23.360]So then the application is compromised [34:26.240] [34:26.900]and then the container is compromised because, oops, [34:30.940] [34:30.940]it was running as root and so on and so on. Remember the dominoes. [34:35.500] [34:35.500]Everything falls over. Also, remember your [34:39.560] [34:39.560]pods. Another common [34:43.420] [34:43.420]mistake that people make when they're new to Kubernetes [34:47.160] [34:47.160]is they don't stop and think, how can I [34:51.600] [34:51.600]group things into pods and secure those pods? [34:56.180] [34:56.180]At least if a container falls [34:59.160] [34:59.160]within a pod, then maybe there's a couple more containers [35:03.380] [35:03.380]in there. Those containers that get compromised are going to be somewhat [35:07.340] [35:07.340]limited within the pod itself. Typically [35:11.720] [35:12.060]a pod is not going to fall over when another pod falls [35:15.440] [35:15.440]over. It's usually containers within the pods. So you want to think [35:19.400] [35:19.400]about grouping things. Also, Jason mentioned [35:23.440] [35:23.440]this in his talk earlier, but showdown. If you have [35:27.460] [35:27.460]never used showdown, get familiar with it. Even [35:31.300] [35:31.300]for Kubernetes. You can go out to showdown. You can start [35:35.400] [35:35.400]searching for Kubernetes. You can search for API clusters [35:39.480] [35:39.480]or API servers. You can search [35:42.640] [35:43.140]for etcd servers. And if you find one of yours, [35:47.540] [35:47.840]oops, but they're out there. [35:51.080] [35:51.080]I did a search this morning for some etcd [35:54.800] [35:56.020]exposure and [35:57.540] [35:59.520]there were over 6,000 [36:01.660] [36:01.660]that I found that were exposed. I don't know how many of those have [36:05.580] [36:05.580]vulnerabilities that would have allowed me to compromise them [36:09.380] [36:09.380]because I was just curious what the number would be. [36:13.440] [36:13.560]It goes up and down on a daily basis, but you should be [36:17.600] [36:17.600]aware of that. So use showdown. Let's talk about [36:21.460] [36:21.460]the configurations. I've said this before. [36:25.860] [36:25.860]Most of the breaches come from configuration issues. [36:29.380] [36:32.890]Use CIS hardening. Center for [36:35.950] [36:35.950]Internet Security. It is an amazing place. It is done by [36:39.950] [36:39.950]volunteers and workers within CIS. You can get [36:43.990] [36:43.990]benchmarks for Docker and you can get benchmarks for Kubernetes. [36:48.670] [36:48.670]You can use these to configure your environment [36:51.790] [36:51.790]so they are indeed hardened. [36:54.750] [36:55.710]Stop ignoring them. I still see people [36:58.630] [36:58.630]that tell me they've never heard of CIS and [37:02.390] [37:02.390]it makes me very sad when I hear that because it is such a great [37:06.850] [37:06.850]resource. The whole point here is [37:10.930] [37:10.930]we have to harden the environment from the [37:14.710] [37:14.710]standpoint of the basic configuration, but don't forget [37:18.750] [37:18.750]the application. Also, I know a lot of people [37:22.930] [37:22.930]have environments that are CICD when it [37:27.010] [37:27.010]comes to Docker and Kubernetes. That's why we have them. It's for [37:30.950] [37:30.950]speed. It's for development. It's for putting things out [37:34.870] [37:34.870]quickly. Okay, great. A lot of people use Jenkins. [37:38.930] [37:39.690]Is there some way to automate [37:42.550] [37:42.550]all of this so at least I can look at my images, [37:47.150] [37:47.150]I can look at my containers, all within the CICD pipeline? [37:51.310] [37:51.310]Yes, there is. There is a great tool called [37:55.350] [37:55.350]Anchor. Anchor, as it says here, is a [37:59.570] [38:00.030]container image scanner plugin. And what it does is [38:03.570] [38:03.910]you put it into the pipeline. It's a plugin for Jenkins. [38:07.710] [38:07.710]And it actually does security scans against the images [38:11.770] [38:11.770]that you're trying to deploy. This is a great way to [38:15.610] [38:15.610]get testing in before you actually [38:19.370] [38:19.370]deploy it to production. So let's talk about some [38:23.350] [38:23.350]additional tools before we get into the pen testing. Because we're still trying to [38:27.450] [38:27.450]lock it down. KubeBench. We want to run KubeBench. [38:31.250] [38:31.250]What it does is it runs against your CIS benchmarks. [38:35.910] [38:35.910]It displays [38:36.790] [38:39.870]a configuration that shows you what has passed, what has failed. I'll show an example [38:43.650] [38:43.650]in a minute. There's also KubeAudit. It's a little more granular [38:47.570] [38:47.570]than KubeSec. And if you run KubeBench [38:51.810] [38:51.810]and KubeAudit, it will give you a report that shows you [38:55.770] [38:55.770]oops, my system is insecure. Here's an example [38:59.210] [38:59.210]of a report. And when we look at it, we see allow [39:03.370] [39:03.370]privilege escalation is not set. There's a very [39:07.730] [39:07.730]first one. Oops, that needs to be turned on. [39:11.870] [39:11.870]Remember, that was the one that I mentioned that allows users [39:15.790] [39:15.790]to do privilege escalation. We don't want that. Oh, crap. [39:20.030] [39:20.030]We have run as non-root is not set. We don't have a read [39:23.930] [39:23.930]only file system set. There are so many [39:27.590] [39:27.590]simple things that can be done. Again, no app [39:31.830] [39:31.830]armor, no security comp. It's not being turned on. [39:35.830] [39:35.830]These are things that are just so easy. And this is from every [39:39.750] [39:39.750]single container within the environment that is [39:43.810] [39:43.810]being deployed under this particular. It's much [39:47.690] [39:47.690]longer. It goes into it. This is just a quick snapshot. [39:51.630] [39:51.630]The whole point here is you run this against your environment [39:55.930] [39:55.930]before you deploy it. And you go, oops, let's go back to our [39:59.710] [39:59.710]configuration file and fix all these things. [40:02.970] [40:03.670]Here are some more tools for you in the environment. Claire, [40:08.010] [40:08.010]another great tool for doing static analysis of [40:11.490] [40:11.490]applications. Claire does, it integrates [40:15.470] [40:15.470]Claire and your Docker registry so it can automate that [40:19.490] [40:19.650]a little bit. Falco, I need to put huge stars next [40:23.650] [40:23.650]to this one because this is actually one of my favorite, favorite [40:27.490] [40:27.490]tools. It was developed years [40:31.490] [40:31.490]ago and then it kind of morphed off and became falco.org. [40:35.810] [40:35.810]It monitors behavior within containers. [40:39.740] [40:39.740]It can detect if a container has been compromised [40:43.100] [40:43.740]because suddenly the container starts acting weird. [40:47.320] [40:47.320]Or there are commands being executed inside the container like [40:51.920] [40:51.920]cat etsy password. Probably not a command [40:55.940] [40:55.940]that should be showing up in one of your containers. [40:58.980] [40:59.220]If anything of this entire talk that I'm [41:03.920] [41:03.920]giving today, you get out of this, go look at falco. [41:07.740] [41:07.740]This is one of the best tools out there. Kubehunter [41:11.280] [41:11.280]is another great tool. We're going to talk about that. [41:15.280] [41:15.540]It's used more so for looking at doing some pen testing. [41:19.640] [41:19.640]We're going to take a look at that in just a second. Alright, ready, set, attack. [41:24.140] [41:24.140]If we were going to pen test an environment, this is an [41:27.700] [41:27.700]example of all the ports and all of the things that are [41:31.520] [41:31.520]open in a Kubernetes cluster. Now, they're [41:35.800] [41:35.800]open, but are they open by firewalls and so on? [41:39.800] [41:39.800]Well, we don't know. The API server [41:43.800] [41:43.800]itself is, yes, it's HTTPS. [41:47.680] [41:47.680]It better damn well be protected. [41:50.140] [41:51.720]etcd on 6666, that should be protected [41:55.920] [41:55.920]wherever it happens to be. This just gives you an idea. All of these [41:59.620] [41:59.620]ports can be changed. These are all the defaults in most cases. [42:03.820] [42:04.360]Just be aware of the ports that are out there and [42:07.600] [42:07.600]exposed that you need to make [42:11.620] [42:11.620]sure are protected. Because as an attacker, what am I [42:15.720] [42:15.720]going to do? Well, one of the first things, even Jason was talking about it in his talk, [42:20.100] [42:20.100]looking at ports, running nmap, running mass scan, and so on. [42:24.360] [42:25.040]This is exactly one of the things that [42:27.380] [42:27.380]Shodan does. It looks for all of these things that are exposed on the [42:31.700] [42:31.700]internet. So please make sure you're aware of all the ports [42:35.700] [42:35.700]within your cluster. Now, I are a hacker. [42:39.920] [42:39.920]See me hack. This is one of the [42:43.720] [42:44.320]ways that I'm going to start to attack [42:47.920] [42:49.000]a Kubernetes cluster. I want to first [42:51.620] [42:51.620]and foremost check access to the API server, make sure that is [42:55.800] [42:55.800]protected. I'm going to be looking at etcd, [42:59.820] [42:59.820]and making sure that's protected. If I find it exposed, [43:03.920] [43:03.920]I'm going to get keys to the kingdom in that case. [43:06.640] [43:06.640]What about the kubelets that are communicating and the command channels [43:11.700] [43:11.700]that are there? They should be read-only, but [43:15.040] [43:15.700]they should not be exposed. There was actually [43:19.420] [43:22.440]a vulnerability exposed [43:23.640] [43:23.640]about this, I want to say two months ago. I'm probably off in that [43:27.640] [43:27.640]sense. But take a look at that. And it allowed, even though it's a [43:31.920] [43:31.920]read-only port and it's supposed to limit what you can see, [43:35.740] [43:35.740]there were some vulnerabilities that allowed for a lot of exposure there. [43:40.500] [43:40.500]Also, container vulnerabilities. [43:44.160] [43:44.160]What about, you know, do you have files [43:47.800] [43:47.800]in that container that the application needs that [43:51.960] [43:51.960]you should be protecting? Well, that's where I would set up Falco and tell [43:55.600] [43:55.600]Falco, hey, there's a special file over here. This application [43:59.440] [43:59.440]is using it. But keep an eye on that. Make sure that the only [44:03.500] [44:03.500]thing that is reading that application is, or excuse me, [44:07.540] [44:07.540]that is reading that file happens to be the [44:10.720] [44:11.600]application. And Falco can monitor that. [44:15.160] [44:15.400]Privilege escalation. Nine times out of ten, once somebody [44:19.160] [44:19.160]compromises an application in a container, they [44:23.380] [44:23.380]elevate privilege because there's some other bugs in there. [44:27.240] [44:27.240]Go back to the Kubotic slide where [44:31.420] [44:31.420]it showed privilege escalation was not [44:35.560] [44:35.560]set. All you have to do is set that configuration [44:39.260] [44:39.260]variable. It's not that hard. Okay? [44:43.760] [44:43.760]Vulnerable applications are critical. [44:46.840] [44:47.360]Can't say it enough times. Especially because there are [44:51.480] [44:51.480]so many people using Kubernetes and Docker to deploy [44:55.840] [44:55.840]WordPress. And WordPress is, of course, one of [44:59.480] [44:59.480]our favorite things to deal with. All right. So your [45:03.500] [45:03.500]final threat model, when it comes to pentesting and [45:07.320] [45:07.320]looking at the environment that is Kubernetes, [45:11.560] [45:12.220]you think of your workload. You think of that [45:15.400] [45:15.400]security within your pods. Can I [45:19.220] [45:19.220]set up my pods so they're indeed secure? [45:23.700] [45:23.700]The container security itself, the configuration [45:27.400] [45:27.400]for that container, the more hooks I put in, [45:31.940] [45:31.940]run as root, no. Privilege escalation, [45:35.620] [45:35.620]no. These are the things that we want to make sure that are [45:39.420] [45:39.420]set. Also, file storage security. [45:43.300] [45:43.300]Maybe there's a particular file system that has to be read right [45:47.440] [45:47.440]but the rest of the file system can be read only. Those are [45:51.380] [45:51.380]other things. The network security. Remember the infrastructure. [45:56.500] [45:56.520]Put firewalls in. Put controls [45:59.300] [45:59.300]in there that detect lateral movement. If [46:03.400] [46:03.400]somebody does compromise one of your applications, [46:07.720] [46:07.720]make sure you have something monitoring. I gave you that [46:11.360] [46:11.360]slide way back about the blog entry from Wazoo. [46:15.680] [46:15.680]They talk about how to audit and monitor [46:19.420] [46:20.000]a Kubernetes cluster using their tool. I love the Wazoo [46:23.440] [46:23.440]manager and agent. Take a look at it if you haven't heard of it [46:27.580] [46:27.580]before. Remember, most [46:31.780] [46:31.780]breaches are not zero day. I can't say it enough times. Everybody's always [46:35.500] [46:35.500]trying to protect against this stuff. They're not fancy. [46:39.600] [46:39.600]Most of these things are being broken into because people [46:43.380] [46:43.380]make silly mistakes. Because remember, common sense. [46:47.540] [46:47.540]Whatever happened to it. The breaches are going to come from configuration [46:51.320] [46:51.320]issues. The kube audit was a very good [46:55.400] [46:55.400]example where it was showing all the problems that would allow someone to [46:59.440] [46:59.440]get in. Compromise credentials. [47:03.180] [47:03.180]Make sure your credentials are stored in a secure place [47:07.260] [47:07.260]somewhere. And don't allow for [47:11.140] [47:11.140]privilege escalation. Kubernetes and Docker [47:14.880] [47:14.880]give you the tools and the configurations to prevent that. [47:18.980] [47:18.980]Just turn them on. And that's what we're not doing. We have [47:23.320] [47:23.320]to turn on the configurations. So your key takeaways [47:27.260] [47:27.260]from today. Common sense. [47:31.140] [47:31.140]Please go back and stop [47:33.880] [47:34.960]and think about where is the exposure? [47:40.240] [47:40.240]Where are my pods exposed? What about my etcd? [47:44.660] [47:45.240]Configuration. That is so critical to any [47:48.200] [47:48.200]Kubernetes. And always remember containers. [47:52.200] [47:52.200]Containers are the linchpin of a Kubernetes [47:55.880] [47:55.880]environment. And if I don't secure my individual containers [48:00.140] [48:00.140]then Kubernetes... I don't care about [48:04.320] [48:04.320]the security of Kubernetes because the container is insecure. [48:08.460] [48:08.460]So stop overthinking this stuff and think about [48:12.720] [48:12.720]security basics. Remember the CIS benchmarks. [48:17.420] [48:17.860]Can't say it enough. Basics. [48:20.540] [48:20.540]I still think as security professionals [48:24.440] [48:24.440]hackers, whatever we want to call ourselves. We have [48:28.480] [48:28.480]forgotten the basics of how to secure an environment. [48:32.340] [48:32.340]And that's why things keep getting broken into. [48:35.780] [48:35.780]Stop doing fancy crap. The fancy crap doesn't work. [48:40.050] [48:41.460]Secure the environment, please. [48:43.940] [48:44.020]Use the CIS benchmarks for the containers, for Kubernetes, [48:48.240] [48:48.240]for the OS that your stuff runs on. If you're going to [48:51.760] [48:51.760]deploy this whole Kubernetes cluster on Ubuntu, then pull up [48:55.760] [48:55.760]the CIS benchmarks for Ubuntu. Or [48:59.280] [48:59.280]Red Hat or CentOS. Whatever you're using. [49:04.040] [49:04.040]Remember, RBAC is your friend. ABAC [49:08.060] [49:08.060]is not. Your infrastructure better be [49:12.080] [49:12.080]secure. Your pods better be secure. [49:15.760] [49:15.760]Containers themselves, the network. Still a lot of people forget [49:20.080] [49:20.080]the network because they forget about TLS everywhere. [49:24.160] [49:26.880]Threat model [49:28.300] [49:28.300]your applications. If you don't take [49:32.000] [49:32.000]the beginning effort to threat model your applications, [49:36.500] [49:36.500]you end up with insecure applications. And all the security [49:40.180] [49:40.180]of your Kubernetes cluster is useless because you have [49:43.580] [49:43.580]insecure applications within your environment. Access controls [49:48.260] [49:48.260]go along with RBAC, but access controls into the [49:51.980] [49:51.980]environment, kind of referencing SSH. [49:56.460] [49:56.920]I found one worker node configuration [49:59.760] [49:59.760]in an environment probably nine months ago. But for some [50:03.960] [50:03.960]reason on the worker nodes, they had FTP enabled on all the worker nodes. [50:08.700] [50:08.820]And my head exploded. [50:12.660] [50:13.180]It allowed me to get into everything, but oh well. [50:16.540] [50:17.380]Yes, did I mention RBAC though? [50:19.920] [50:20.220]Enough times. I can't say it enough. Why? Because [50:23.480] [50:23.480]people tend to ignore it. Remember the tools I mentioned earlier [50:28.000] [50:28.000]about monitoring or auditing the RBAC roles and also [50:32.140] [50:32.140]the permissions. And finally, [50:35.960] [50:35.960]not quite finally, remember to patch everything. [50:39.900] [50:39.900]The infrastructure that goes along with that. [50:43.820] [50:43.860]Logging and auditing and monitoring. I still see people [50:48.080] [50:48.080]set up their environments. There's that blog entry again. [50:52.700] [50:52.700]Please take a look at that. It is well written. [50:56.040] [50:56.040]It's a great tool. You'll learn a lot about Wazoo. I use it on [50:59.960] [50:59.960]all my systems. I deploy a lot of honeypots with [51:04.120] [51:04.120]Kubernetes and containers and things like that. But I monitor them all [51:08.100] [51:08.100]with Wazoo. You can't [51:12.240] [51:12.240]have a secure environment if you're not monitoring that environment. [51:16.540] [51:16.540]That is the biggest thing. [51:18.360] [51:19.860]That brings me to the end. I thank you for your [51:24.080] [51:24.080]attention. There's contact info. There's my Twitter [51:28.180] [51:28.180]account. I will get this to the Red Team [51:32.260] [51:32.260]Village and they can upload it to their environment. But I'll have it on my [51:36.360] [51:39.160]GitHub later [51:40.200] [51:40.200]in an hour or so. I'll update it there. Thank you all very much. [51:44.360] [51:44.360]Hopefully you learned something today. Go out and hack and have some fun. Thanks to [51:48.140] [51:48.140]the Red Team Village and have an awesome DEF CON. [51:51.480] [51:51.480]Awesome. Thank you so much, Kat. Amazing presentation [51:55.840] [51:55.840]as always. Thank you for supporting DEF CON. Thank you for supporting the [51:59.500] [51:59.500]community. Of course, the Red Team Village. For those of you [52:02.700] [52:03.440]that just joined or are looking at [52:06.940] [52:07.340]the presentation on Twitch or YouTube, please look at [52:11.200] [52:11.200]all the activities that we have going on. Go to the website that is in the description of [52:15.480] [52:15.480]the stream. Please also join the conversation in Discord. [52:19.540] [52:19.540]We're going to go on a brief break and the next presentation will start in a few minutes. [52:23.760] [52:23.760]Thank you again and thank you, Kat. [52:25.340]