[00:01.280 --> 00:05.120] Ah, come back here. Why does that keep doing that? [00:05.600 --> 00:09.560] All right, great. We are here and ready to do securing in, [00:09.560 --> 00:13.280] path testing, great spaghetti monster, as I like to call it, or [00:13.280 --> 00:18.400] Kubernetes. This is who I am, Kat Fitzgerald. [00:18.400 --> 00:22.840] You can find me on Twitter at Rainbow Kat. I've been doing this a long time. [00:22.840 --> 00:26.580] I'm the CEO of B-Sides Chicago. I was COO for [00:26.580 --> 00:30.820] the Diana Initiative last year, part of B-Sides Pittsburgh, [00:30.820 --> 00:34.900] and also my first DEFCON was DEFCON 3. [00:34.900 --> 00:38.860] How many other people remember that one? There were 16 [00:39.140 --> 00:42.560] speakers there, if I have to remember that. [00:42.840 --> 00:46.820] I have an emphasis in the blue team. However, I'm a formal purple teamer [00:47.160 --> 00:50.880] and all this stuff. And I'm in Pittsburgh now, but moving to [00:50.880 --> 00:54.760] Kirkland, Washington very, very soon. You will always [00:54.760 --> 00:58.800] find me sipping Grand Mayan Extra Añejo, or in [00:58.800 --> 01:03.740] this case, it is Casanoble Añejo. [01:03.740 --> 01:06.840] So, cheers. Let's kick off DEFCON. Jason [01:06.840 --> 01:08.960] already did a great job. [01:11.120 --> 01:15.240] And also, special thanks to the entire Red Team Village, [01:15.240 --> 01:18.720] all the staff, all the volunteers, everybody for kicking butt [01:18.720 --> 01:22.980] and making this happen. A lot of my favorite stuff are honey pots [01:22.980 --> 01:26.640] and refrigerators. On Saturday, I'm doing a talk for [01:26.640 --> 01:30.920] IoT security, so check that out, too. But this is one of my favorite ones [01:30.920 --> 01:34.300] because I just finished writing this particular [01:35.560 --> 01:39.380] presentation just about a month ago, made some changes to it, and so on. [01:39.380 --> 01:43.000] Remember, please, the views and opinions here are mine and mine [01:43.000 --> 01:47.000] alone. They do not reflect anything from my employers, past [01:47.000 --> 01:50.940] or present. Anything that I show you, [01:50.940 --> 01:54.780] as always, please use with a grain of salt. Don't go breaking [01:54.780 --> 01:58.820] into things that you shouldn't be breaking into. And those of you with an [01:58.820 --> 02:02.320] overwhelming fear of the unknown, you'll be happy to know that [02:02.320 --> 02:06.720] there is no hidden message in the disclaimer if you read it [02:06.720 --> 02:10.340] backwards. Yes, my jokes are that bad. [02:10.380 --> 02:14.920] Alright, let's get this thing on the road. So, why are we [02:14.920 --> 02:18.760] not here? Well, first of all, I'm not going to solve all [02:18.760 --> 02:22.640] your Kubernetes security woes. I wish I could. [02:22.640 --> 02:27.180] I wish Kubernetes and containers and everything were secure by default. [02:27.180 --> 02:30.980] They are not. Neither is the person [02:30.980 --> 02:34.520] sitting next to you, although, well, that doesn't work anymore because [02:34.520 --> 02:38.520] nobody's sitting... well, maybe somebody's sitting next to you. You never know. [02:38.520 --> 02:42.880] You never know. Okay. And common sense went out the window [02:42.880 --> 02:46.540] decades ago. Think about that. When it comes to [02:46.540 --> 02:50.940] information security, cyber security, hacking, whatever you want to call it, [02:50.940 --> 02:54.340] I honestly think we lost common sense quite [02:54.340 --> 02:58.560] some time ago. Why really are [02:58.560 --> 03:02.520] we here? Well, Kubernetes, I [03:02.520 --> 03:06.740] don't know. I still think it's fairly new. It's fun. [03:06.740 --> 03:10.420] It's crazy. It's a lot of things. [03:10.420 --> 03:14.560] You can deploy it in clouds. You can deploy it in... [03:14.560 --> 03:18.500] I have Raspberry Pis sitting down here. I was [03:18.500 --> 03:22.280] going to be showing that. There have been some demo problems and also some [03:22.280 --> 03:26.520] timing issues, so I won't be showing my Raspberry Pi cluster today. [03:26.520 --> 03:30.160] But you can build very extensive Raspberry Pi [03:30.160 --> 03:34.780] clusters and do all the things that I'm going to talk about today. [03:35.340 --> 03:38.800] Containers. Containers are not new. [03:38.800 --> 03:42.400] But people seem to have forgotten that they [03:42.400 --> 03:46.460] have anything to do with security. They get all [03:46.460 --> 03:50.500] hyped up on Kubernetes security, and they're forgetting about [03:50.500 --> 03:54.240] containers. We're going to talk about containers because it's [03:54.240 --> 03:58.600] important. It's, in fact, very important. And always remember this. [03:58.600 --> 04:02.480] Security is for everyone. Common [04:02.480 --> 04:06.080] sense is required. So please [04:06.080 --> 04:10.340] disregard the previous slide. Throw it out the window. Why? [04:10.340 --> 04:14.560] Because if we're not using common sense in security, then [04:14.560 --> 04:18.220] we're definitely not going to get where we need to be. [04:18.640 --> 04:22.380] All right. Just some quick security facts. We want to [04:22.380 --> 04:26.640] know this stuff. In 2018 alone, $114 [04:26.640 --> 04:30.620] billion were spent by companies on [04:30.620 --> 04:34.580] information security. On fancy software, [04:34.580 --> 04:39.100] fancy hardware, things with blinky lights and all of that. [04:39.100 --> 04:42.980] But breaches kept on going up. We're doing [04:42.980 --> 04:47.120] something wrong here. Security appliances and software [04:47.120 --> 04:51.060] have vulnerabilities that we're not dealing with or that [04:51.060 --> 04:55.320] need to be dealt with by the vendors. Kubernetes itself [04:55.320 --> 04:59.700] has had a few issues. We'll talk about that. But so has Docker. [04:59.700 --> 05:03.160] So don't forget your containers. Also, I've [05:03.160 --> 05:07.400] actually had people say to me, we have a very unique security [05:07.400 --> 05:11.460] architecture. And I'm like, no, you don't. [05:11.560 --> 05:15.280] We all do the same stuff. So remember, your architecture [05:15.280 --> 05:19.360] is not unique. I love this quote. If you don't have this [05:19.360 --> 05:23.500] book, Offensive Countermeasures by John Strand, go out [05:23.500 --> 05:27.220] and get it. It's a small book with a ton of information. [05:27.220 --> 05:31.280] So get it. But he said, instead of brilliance, [05:31.280 --> 05:35.840] we've standardized on mediocrity. And that seems to be [05:35.840 --> 05:40.100] what information security is about these days. [05:40.100 --> 05:43.800] And I kind of have issues with that. So let's talk about [05:43.800 --> 05:48.040] that some more. First of all, let's get a common definition [05:48.040 --> 05:51.920] of what is a breach. Or maybe we'll call it [05:51.920 --> 05:56.200] what is an incident. Let's not necessarily call it a breach. [05:56.200 --> 05:59.520] So most incidents are not caused by zero-day [05:59.520 --> 06:04.620] fancy stuff that you find on the Internet. Typically, [06:04.620 --> 06:08.160] they're done because people have made mistakes and so on. [06:08.160 --> 06:12.140] Most breaches are not coming from your vulnerability scanners, [06:12.140 --> 06:16.120] especially when it comes to Kubernetes. You're not going to run a vulnerability [06:16.120 --> 06:20.360] scanner and go, oh, look, fix this one thing, and we're [06:20.360 --> 06:24.120] done and ready to go. No, it doesn't happen that way. [06:25.420 --> 06:28.000] Most breaches are going to come from [06:28.580 --> 06:32.380] configuration issues. This is something I want [06:32.380 --> 06:36.160] to emphasize. In fact, I'm going to show you some examples how [06:36.160 --> 06:40.380] configuration issues tend to cause most of the breaches. A close [06:40.380 --> 06:44.460] second is compromised credentials. [06:44.500 --> 06:49.040] When I say credentials, I still go out on GitHub. [06:49.240 --> 06:52.280] Well, maybe not every single day. And I think Jason [06:52.280 --> 06:56.080] even mentioned it, too, in his previous talk on bug bounties. [06:56.320 --> 07:00.540] Go out and look on GitHub and search for simple things [07:00.540 --> 07:04.520] like password. See how many times you find [07:04.520 --> 07:08.520] the word root. Yes, thousands of them [07:08.520 --> 07:12.700] are invalid. They don't make any sense. It's probably going to find the keyword [07:12.700 --> 07:17.140] root or password, and it's not going to get you anywhere. [07:17.140 --> 07:20.540] And yet I still find where people hard-code credentials [07:20.540 --> 07:24.540] into variables that say password. And right [07:24.540 --> 07:28.600] next to it, there's the key. There's the whatever the password [07:28.600 --> 07:32.560] is. Also, trailing in third [07:32.560 --> 07:36.780] place for most incidents is overprivileged [07:36.780 --> 07:40.620] users. This is incredibly important in [07:40.620 --> 07:44.480] the Kubernetes world. I'm going to show you this in just a second [07:44.480 --> 07:49.160] or in a couple of slides here. So keep these three things in mind. [07:49.160 --> 07:52.920] Configuration issues, compromised credentials, and overprivileged [07:52.920 --> 07:56.800] users. Okay. Kubernetes [07:56.800 --> 08:00.820] itself, we have to deal with all three [08:00.820 --> 08:04.720] of these compromises or these types [08:04.720 --> 08:09.880] of incidents that are going to cause issues in Kubernetes. [08:09.960 --> 08:12.980] But first, think [08:12.980 --> 08:16.960] about this. You know, we think about the Capital One hack [08:16.960 --> 08:20.980] that happened recently. Well, that was a combination [08:20.980 --> 08:24.820] of compromised credentials and access [08:24.820 --> 08:29.000] that should not have been granted. We see how [08:29.000 --> 08:32.980] Tesla, you know, they had poor access credentials [08:32.980 --> 08:37.160] and so on. Equifax, we all know about that one. [08:37.160 --> 08:40.580] Go back and check if you don't. You know, Equifax [08:40.580 --> 08:44.300] was hacked not only by a simple vulnerability [08:44.820 --> 08:48.980] that had not been patched, but also because they had very [08:48.980 --> 08:53.040] very poor lateral movement detection on the [08:53.040 --> 08:57.000] inside. Once they got in with more compromised [08:57.000 --> 09:01.240] credentials through the vulnerability, they weren't [09:01.240 --> 09:05.180] detected for quite some time. This is going to be [09:05.180 --> 09:09.220] critical. And one other thing, I was bored this morning. I decided [09:09.220 --> 09:13.140] to throw this up here in this slide. Why? Well, I [09:13.140 --> 09:17.060] put up a new spice rack in my kitchen. I really liked how it looks. [09:17.060 --> 09:21.220] I took a picture and I added it to the slide. There it is. [09:21.220 --> 09:25.240] Isn't it a great spice rack? Come on. Alright, you all better [09:25.240 --> 09:29.460] be laughing. Alright, let's talk about containers. [09:30.180 --> 09:33.160] Quick review of containers. They are not [09:33.160 --> 09:37.400] secure by default. Now, I'm really focusing more on [09:37.400 --> 09:40.980] Docker containers for now because that tends to be [09:40.980 --> 09:45.040] the norm. And one thing to think about when it comes to [09:45.040 --> 09:48.760] containers, containers are not like [09:50.000 --> 09:52.740] a VM. A VM is actually more [09:52.740 --> 09:56.840] isolated. And this is something that's going to be important coming [09:56.840 --> 10:00.760] up here. So, a container is a grouping of stuff [10:00.760 --> 10:04.660] and the stuff that we like to call it are resources. Well, the [10:04.660 --> 10:08.820] resources are grouped into namespaces. If [10:08.820 --> 10:12.740] you are new to containers, remember this. Namespaces [10:12.740 --> 10:16.860] are critical. You group your processes, [10:16.860 --> 10:20.980] your networks, your users, your ports, your interprocess. All [10:20.980 --> 10:24.680] of this stuff can be controlled and [10:24.680 --> 10:28.960] locked down within namespaces. Resources [10:28.960 --> 10:32.840] are exploitable. I was sitting in another [10:33.980 --> 10:36.640] web broadcast just the other day [10:36.640 --> 10:40.240] on containers and I was [10:41.660 --> 10:44.740] surprised because it was supposed to be a more [10:44.740 --> 10:48.820] intermediate presentation on containers and [10:48.820 --> 10:53.060] yet all of the people were asking is there a [10:53.060 --> 10:56.740] way to prevent process [10:57.560 --> 11:00.960] abuse such as CPUs and memory and [11:00.960 --> 11:04.740] so on. And again, this was supposed to be an intermediate [11:05.560 --> 11:09.100] class on containers so it [11:09.100 --> 11:12.860] confused me a little bit that people were asking that because that should be something [11:12.860 --> 11:17.740] that you know from the very beginning that you get into containers. [11:17.740 --> 11:20.920] Using namespaces and the next [11:20.920 --> 11:25.120] most important thing is control groups or cgroups. These [11:25.120 --> 11:29.460] are the limits that you are going to be putting on containers. [11:29.460 --> 11:32.820] If you don't put limits on them, you're [11:32.820 --> 11:37.180] going to run into problems. Every time I do a pen test [11:37.180 --> 11:41.320] of a new application deployed into a container, [11:41.320 --> 11:45.500] I would say that 7 out of 10 times [11:45.500 --> 11:49.340] I find that the author of the application [11:49.340 --> 11:53.220] who has put it in a container has not put any limits. [11:53.220 --> 11:57.540] I'll get into the application in whatever way possible whether it's [11:58.140 --> 12:01.180] a SQL injection or an RCE [12:01.180 --> 12:05.400] or something like that. And once I'm in, I can take over not just the [12:05.400 --> 12:09.040] container but the entire system that it's on [12:09.040 --> 12:13.800] because there are no cgroups applied to the container. [12:13.800 --> 12:17.120] The other thing to remember about containers [12:17.120 --> 12:20.960] is think of a container as an image or more [12:20.960 --> 12:25.880] correctly, think of it as a file system snapshot. [12:26.060 --> 12:28.880] So who decides what goes [12:28.880 --> 12:34.220] in that image? If you look at this particular picture, [12:34.220 --> 12:36.900] you'll see, you know, here we have [12:36.900 --> 12:40.200] namespaces, the control groups that I show. [12:40.480 --> 12:44.780] It's going to possibly be cherooted if you've set it up [12:44.780 --> 12:48.860] correctly. And here's the application. [12:49.400 --> 12:52.840] There's a Docker client that you might use to launch [12:52.840 --> 12:57.120] this. There's a Docker registry. This is critical [12:57.120 --> 13:00.940] to the security of your system. If you're [13:00.940 --> 13:05.320] new to Docker, you may not know this, but Docker Hub was compromised [13:05.320 --> 13:09.400] two years ago, I believe it was. I've lost track [13:09.400 --> 13:12.520] of time during COVID. I don't know anymore. [13:12.860 --> 13:17.160] But you want to have a secure registry where you keep [13:17.160 --> 13:21.000] your images because if they're somewhere [13:21.000 --> 13:25.800] that you don't have control, you never know what's going to happen. [13:25.800 --> 13:29.020] But here's the difference. Notice that we have a host OS [13:29.020 --> 13:34.280] and hardware down here, unlike a VM. [13:34.280 --> 13:38.600] In a VM, you would see the entire operating system [13:38.600 --> 13:42.520] within the VM here. You would not see a shared [13:42.520 --> 13:46.560] OS. Yes, there's a host OS, but then you also have the guest [13:46.560 --> 13:50.580] operating systems themselves. And that's where we get into [13:50.580 --> 13:54.580] how a VM is much more isolated [13:54.580 --> 13:58.300] and contained than a container is. So [13:58.300 --> 14:02.540] don't think the word container means that you're more secure. [14:02.540 --> 14:06.460] It just means that we have some work to do. So that's why [14:06.460 --> 14:10.120] I say right here in this slide, container isolation. [14:10.320 --> 14:14.740] Is it really isolated? The simple answer to that is [14:14.740 --> 14:18.500] no, it is not. So let's talk about some [14:18.500 --> 14:22.700] container best practices. I should have also said at the very [14:22.700 --> 14:26.540] beginning of this talk, this is an intro. [14:26.540 --> 14:30.360] I can't show you everything. I'm trying to get people to start [14:30.360 --> 14:34.320] thinking about how you secure containers, then [14:34.320 --> 14:38.340] we're going to jump into how you secure Kubernetes at the [14:38.340 --> 14:42.320] same time. So remember, we have [14:42.520 --> 14:46.300] a base image. That means that base image has to come from [14:46.480 --> 14:50.000] a secure repo. If you do not know what CIS [14:50.000 --> 14:54.200] is, definitely go to CIS [14:54.200 --> 14:57.820] or CISecurity.org [14:58.460 --> 15:01.860] Check it out because you want to use CIS [15:01.860 --> 15:05.900] benchmarks for locking down your base image. I'm going to [15:05.900 --> 15:10.020] talk about that a little more coming up. Don't run your images [15:10.020 --> 15:14.040] as root. By default, if you do not set a user [15:14.040 --> 15:17.920] name, your container and your application in the container is going [15:17.920 --> 15:21.480] to run it as root. I have found so many [15:21.480 --> 15:26.040] who have deployed WordPress applications [15:26.040 --> 15:29.220] in a Docker container via Kubernetes [15:29.980 --> 15:34.000] and they don't set the user name and WordPress is running [15:34.000 --> 15:37.960] as root, so is the database and everything else. So of course, [15:37.960 --> 15:42.400] once it's compromised, you take over everything. Also, [15:42.400 --> 15:46.160] visibility. You need monitoring within [15:46.160 --> 15:49.880] your containers. Your containers put a lot of information [15:49.880 --> 15:54.180] out and they can log it, but you have to capture that. We're going to talk [15:54.180 --> 15:58.220] about that in a minute. Also, think of [15:58.220 --> 16:02.140] it as a domino. If one container is [16:02.140 --> 16:06.280] compromised, I guarantee you the rest of them will [16:06.280 --> 16:09.260] fall over like dominoes. [16:09.880 --> 16:14.520] When you start to get into containers and Kubernetes, you're going to see this. [16:14.920 --> 16:18.860] Also, we want to set some controls. [16:18.900 --> 16:22.040] Besides setting a user name that is not [16:22.040 --> 16:25.940] root, we also set privileges and we set [16:25.940 --> 16:29.940] certain attributes to those users where it says [16:29.940 --> 16:33.780] no elevated privileges or no new privileges. [16:33.780 --> 16:37.720] You don't want user accounts or IDs [16:37.720 --> 16:42.020] to be able to escalate privileges. And I'll [16:42.020 --> 16:46.080] show you some examples coming up. This one, my god, this is [16:46.080 --> 16:50.020] near and dear to my heart, secrets management. I still [16:50.020 --> 16:54.180] find, I would love to say by a show of hands, [16:54.180 --> 16:58.040] but I can't do that right here, how many people [16:59.480 --> 17:02.060] put the secrets [17:02.060 --> 17:05.760] and tokens and things like that in their Docker [17:05.760 --> 17:09.720] YAML files so when they then deploy the Docker [17:09.720 --> 17:14.040] image, if somebody gets a hold of their YAML file, which [17:14.040 --> 17:18.340] describes the configuration of that container, [17:18.340 --> 17:22.340] well, the secrets are right in it. I've worked [17:22.340 --> 17:26.460] with a lot of developers and a lot of engineers and I get [17:26.460 --> 17:30.320] them, you know, you have to start thinking about putting [17:30.320 --> 17:34.360] your secrets in another place, whether it's HashiCorp [17:34.360 --> 17:38.240] Vault, whether it's a cloud secrets management system, [17:38.240 --> 17:42.420] something. But get them out of the apps and get them out of the [17:42.420 --> 17:46.480] configuration files. And stop allowing SSH [17:46.480 --> 17:49.740] into your hosts. When we get into [17:50.660 --> 17:54.340] Kubernetes, coming up in a couple slides here, [17:54.340 --> 17:58.600] one of the key constructs of Kubernetes is you have a master node [17:58.600 --> 18:02.660] or master nodes, and then you have worker nodes. And I [18:02.660 --> 18:06.500] still find worker nodes with SSH enabled [18:06.500 --> 18:10.380] on them so people can SSH in and, quote, fix [18:10.380 --> 18:14.640] things. No, no, you use the [18:14.640 --> 18:18.820] management tools to fix things and redeploy. Stop [18:18.820 --> 18:22.940] allowing SSH into your systems. Also, I can't [18:22.940 --> 18:26.880] stress it enough times, namespace [18:26.880 --> 18:30.840] limits. Go read up on how to [18:30.840 --> 18:34.820] control namespaces. That is going to [18:34.820 --> 18:38.800] be one of your best efforts and best [18:38.800 --> 18:42.760] practices for containing and controlling and [18:42.760 --> 18:46.320] securing a container before you distribute it to [18:47.240 --> 18:50.780] or deploy it via Kubernetes. This is one of [18:50.780 --> 18:54.800] my favorite tools. It's called Docker Slim. If you've never heard [18:54.800 --> 18:58.760] of it, if I were to create a container, [18:58.760 --> 19:02.340] let's say I create a container, it has Ubuntu in it [19:02.340 --> 19:06.260] or the base of Ubuntu, and then maybe it has WordPress [19:06.900 --> 19:11.020] running in it for whatever god awful reason that I want to do this. [19:11.020 --> 19:15.180] Let's say that entire image ends up being [19:15.180 --> 19:18.700] three to four hundred meg. It's still [19:18.700 --> 19:23.300] manageable, but if I were to run that through Docker Slim, [19:23.300 --> 19:27.400] not only will it add some extra configurations [19:27.400 --> 19:31.380] and namespace limits and so on, but it shrinks it down [19:31.380 --> 19:34.840] and something that might be three or four hundred megabytes [19:34.840 --> 19:39.180] deployed into a container will get shrunk down to [19:39.180 --> 19:43.200] maybe 10, 15, 20 meg. This [19:43.200 --> 19:46.960] makes it even that much more manageable. It's easier to [19:46.960 --> 19:50.880] deploy. It's quicker to deploy, especially [19:50.880 --> 19:55.260] if there are requirements for scaling. [19:55.260 --> 19:58.680] So think about this. Take a look at this. I also mentioned [19:58.680 --> 20:03.000] in Discord that my slides will be available in my GitHub [20:03.000 --> 20:07.140] repo, and I'll get them to the Red Team Village so they can add them [20:07.140 --> 20:12.000] to the Google Docs location after my talk. [20:13.260 --> 20:15.340] And always remember [20:15.340 --> 20:19.200] apply SELinux or AppArmor. One of the things [20:19.200 --> 20:23.120] that Docker Slim does is it will do that for you. [20:23.120 --> 20:27.080] It's very easy to enable SELinux or AppArmor [20:27.080 --> 20:31.300] within a container. If you can do that, you're going to make [20:31.300 --> 20:35.220] it that much more secure. It's not on by default, but you can set [20:35.220 --> 20:36.960] it up so it will be. [20:39.140 --> 20:44.040] And remember the infrastructure, as I put it. [20:44.640 --> 20:45.400] You [20:47.120 --> 20:51.180] can't forget that all of this stuff is being deployed on [20:51.180 --> 20:55.160] an infrastructure of some sort, and that infrastructure better [20:55.160 --> 20:59.180] be secure. I mentioned SSH. People keep [20:59.180 --> 21:03.440] allowing SSH into their environment. Be very [21:03.440 --> 21:07.500] aware that your infrastructure itself should be secure. And I will [21:07.500 --> 21:11.280] remind you later about that. All right. Here's a quick picture. [21:11.280 --> 21:15.720] Picture is worth a thousand words, right? If you look at our first example, [21:15.720 --> 21:19.360] this is our monolithic operating system here. We see [21:19.360 --> 21:23.560] hardware and operating system. We run apps. Here we see a VM. So we see [21:23.560 --> 21:27.300] hardware operating system. There's a hypervisor involved. [21:27.900 --> 21:31.320] And then there's going to be virtual machines. But when you have virtual [21:31.320 --> 21:35.100] machines, you have additional operating systems, you have additional [21:35.100 --> 21:39.480] libraries. They're very much isolated. I should have used a different [21:39.480 --> 21:43.260] picture where there's a big line through it that [21:43.260 --> 21:47.040] shows, yes, these are isolated and so on. But [21:47.040 --> 21:51.380] that should not have happened. Nope. Go back. [21:51.540 --> 21:54.720] Sorry. Okay. But when we look at [21:55.320 --> 21:59.300] using Docker or Container, we go hardware operating [21:59.300 --> 22:03.680] system. There is a container runtime. There's no real dividing lines. [22:03.680 --> 22:07.400] And then we see containers. The only thing that is unique between [22:07.400 --> 22:11.380] the containers is that bin library, the things [22:11.380 --> 22:15.340] that are going to run in the application. So there's not a lot of [22:15.340 --> 22:19.360] isolation there. And people want to [22:19.360 --> 22:23.460] think that there is. But there is not. So please keep [22:23.460 --> 22:27.080] that in mind. I do love this [22:27.080 --> 22:30.940] comic. It came out a couple of years ago. A great [22:30.940 --> 22:35.040] defense is, you know, we'll inundate them [22:35.040 --> 22:39.100] with quarterly Kubernetes releases. Yes, they do update [22:39.100 --> 22:42.820] Kubernetes a lot. That's why you have to pay [22:42.820 --> 22:46.680] attention to your environment and the tools that you're going to use [22:46.680 --> 22:51.140] so you can do upgrades to your Kubernetes cluster. So let's bring on [22:51.140 --> 22:55.020] the orchestrator. Let's talk about Kubernetes. All right. Here [22:55.020 --> 22:58.840] is Kubernetes 101. In one slide, I'm going to teach you [22:58.840 --> 23:02.780] at least the basics of how Kubernetes [23:02.780 --> 23:07.020] works. We have a master node. The master node is [23:07.020 --> 23:10.560] critical. It manages all of the Kubernetes [23:10.560 --> 23:14.520] worker nodes. It manages the clusters. It deploys [23:14.520 --> 23:18.460] pods. Think of a pod as a collection [23:19.020 --> 23:23.000] of containers that are grouped. [23:23.000 --> 23:26.960] A pod might be the way that you [23:26.960 --> 23:30.880] decide, oh, I want to have a pod that supports a particular [23:30.880 --> 23:34.880] client. Or maybe it's a WordPress instance. And that [23:34.880 --> 23:39.280] pod has WordPress. It has a container, a [23:39.280 --> 23:43.140] separate container for the database and so on. So [23:43.140 --> 23:47.320] the pod is how you can group things that [23:47.320 --> 23:51.180] are various containers into certain groupings. [23:51.180 --> 23:55.260] And those groupings are kind of defined by you. But stop and think about [23:55.260 --> 23:58.920] how you want to deploy pods. It's going to be important. [23:58.920 --> 24:03.580] A worker node. The node is where all the containers are going to run. [24:03.580 --> 24:07.200] You have other Kubernetes components that have to [24:07.200 --> 24:11.020] run there. But the important part is, this is where the containers [24:11.020 --> 24:14.760] run. If the worker node is not secure [24:14.760 --> 24:19.020] and not patched, and I can get into that [24:19.020 --> 24:22.640] underneath the visibility of [24:22.640 --> 24:27.000] Kubernetes, then Kubernetes has no idea that I've compromised [24:27.500 --> 24:31.160] a node. I found numerous Kubernetes sites around [24:31.160 --> 24:34.540] the Internet where there's [24:34.540 --> 24:39.300] additional applications running on worker nodes because they've been compromised. [24:39.300 --> 24:43.080] Of course, they're Bitcoin miners. But the master node has no [24:43.080 --> 24:46.980] idea that it's running on there because they got in via [24:46.980 --> 24:50.740] SSH and they launched the application, the [24:50.740 --> 24:54.900] miner, underneath the visibility of Kubernetes. [24:54.900 --> 24:59.100] So you really want to lock down these worker nodes. A pod, [24:59.100 --> 25:02.900] I already explained, it's a unit of deployment and it's also how [25:02.900 --> 25:07.080] you address things from Kubernetes. And a pod [25:07.080 --> 25:10.520] will have an IP address. So that's important to [25:10.520 --> 25:15.080] remember. But it's going to contain one or more containers. [25:15.080 --> 25:18.980] I like to think of pods as, even though it might have [25:18.980 --> 25:22.840] its own IP address, but think of it like a VLAN in a way. [25:22.840 --> 25:26.900] Because I've isolated it in one way. Services [25:26.900 --> 25:30.200] are various things that run within Kubernetes [25:30.980 --> 25:34.920] that communicate, that balance, that do a lot of work. We're going to look [25:34.920 --> 25:38.840] at services coming up. There are various components that are [25:38.840 --> 25:42.760] used. Three that are critical, the API [25:42.760 --> 25:46.640] server, you're going to have kubelets, and you're also [25:46.640 --> 25:51.000] etcd. The one concept you should take [25:51.000 --> 25:55.040] about etcd is this is the database [25:55.040 --> 25:58.700] that describes and builds everything [25:58.700 --> 26:02.580] that is your Kubernetes cluster. The etcd [26:02.580 --> 26:06.280] environment should actually be walled off [26:06.280 --> 26:10.560] and firewalled and controlled very, very [26:10.560 --> 26:14.580] securely. If you do that, you will have a much [26:14.580 --> 26:18.720] more secure Kubernetes cluster. So keep [26:18.720 --> 26:23.040] that in mind. And you can see it here by looking at a picture. [26:23.040 --> 26:26.420] So we take all of those concepts that we just talked about [26:26.420 --> 26:31.000] we see here's an ABI server, here's a controller, [26:31.000 --> 26:33.660] this is all part, you know, the kubectl... [26:33.660 --> 26:38.640] Stop doing that. Come on. Kubectl [26:38.640 --> 26:42.640] is the command you use to control everything. [26:43.020 --> 26:46.580] And even though this is a disk kind of a representation [26:46.580 --> 26:50.740] of etcd, I like to say, you know what, this needs to be off [26:50.740 --> 26:54.920] on its own, firewalled, and separated, and it can be done that way. [26:54.920 --> 26:58.420] Here's your worker node, and you can see how you might have multiple [26:58.420 --> 27:02.440] pods on a worker node. This pod might be talking to [27:02.440 --> 27:06.620] this pod. It doesn't necessarily have to talk to that one, or [27:06.620 --> 27:10.720] they may not talk at all. These are some of the other services that are running [27:10.720 --> 27:14.540] the proxies. This is where we are exposed to the internet, [27:14.540 --> 27:18.500] not over here. Okay? So the only place you're [27:18.500 --> 27:22.240] exposing Kubernetes is at the worker nodes. [27:22.780 --> 27:26.280] Now, Kubernetes is not perfect. [27:26.520 --> 27:29.980] There have been problems. There was... [27:29.980 --> 27:34.640] I need to update this slide again, because there have been a couple of other [27:34.640 --> 27:38.900] Kubernetes. These are more for reference. One's a [27:38.900 --> 27:42.800] little older. This was back... there was the backdoor that involved the [27:42.800 --> 27:46.760] API server. There were some others. These are only for your reference, [27:46.760 --> 27:51.080] so you can go look them up and see. And there are [27:51.080 --> 27:54.720] more. The whole point I'm trying to make here is [27:55.300 --> 27:59.480] Kubernetes is not secure by default, and it's not [27:59.480 --> 28:02.760] perfect. Well, let's try to make [28:02.760 --> 28:06.500] it perfect. Let's take a common Kubernetes cluster [28:06.500 --> 28:10.920] and say, you know, if we were to threat model this, [28:10.920 --> 28:14.680] and that is very, very important, first thing [28:14.680 --> 28:18.800] we want to do is to say, TLS everywhere. Yeah, I know, I have [28:18.800 --> 28:22.620] too many exclamation points, but I'm trying to make the point. If [28:23.640 --> 28:26.820] everything within your Kubernetes cluster is using TLS [28:26.820 --> 28:30.640] and is encrypted, then it's that much [28:30.640 --> 28:34.700] safer. I still have engineers come to me and say, [28:34.700 --> 28:38.620] our application is completely internal. Why do we [28:38.620 --> 28:42.680] need TLS? It's never going to be exposed to the internet. [28:42.680 --> 28:46.560] I said, really? So what happens if Joe's application [28:46.560 --> 28:50.680] over here, which is exposed to the internet, is compromised, and then [28:50.680 --> 28:55.000] they use that as a pivot point to gain access to this environment, [28:55.000 --> 28:58.380] and then they compromise your system? O [28:58.380 --> 29:02.480] is typically the response. So I can't say it [29:02.480 --> 29:06.440] enough, harden the bloody infrastructure. Yes, I'm saying [29:06.440 --> 29:11.060] it again, but you're running this thing on your infrastructure, [29:11.060 --> 29:14.320] make sure it's hardened. And very important, [29:14.320 --> 29:18.180] two concepts. There's RBAC, which is [29:18.180 --> 29:22.780] regular role-based access control. Set that up. [29:22.780 --> 29:26.000] Make sure you're doing it right. Disable [29:26.000 --> 29:28.600] ABAC. ABAC is [29:30.300 --> 29:33.960] incredibly difficult to manage. You [29:33.960 --> 29:38.000] will spend days and hours and hours trying [29:38.000 --> 29:41.920] to do ABAC access controls, [29:41.920 --> 29:46.000] which is more granular, but it's not worth the effort. [29:46.000 --> 29:50.060] Make sure you use RBAC and manage it correctly. [29:50.060 --> 29:53.700] And I'm going to show you a couple of tools coming up [29:53.700 --> 29:58.000] on how to check it. Also, check out this blog [29:58.000 --> 30:01.920] entry. I mentioned this previously about [30:01.920 --> 30:05.520] monitoring. This is one way to audit [30:05.520 --> 30:09.560] Kubernetes, so keep this in mind. Take a look at that [30:09.560 --> 30:13.140] blog entry from Wazoo, and it's an excellent [30:14.780 --> 30:17.680] place to start when it comes to monitoring [30:17.680 --> 30:21.960] your environment. Also, kubectl, get role [30:21.960 --> 30:25.960] binding in all your namespaces. You should do this [30:25.960 --> 30:29.280] periodically in your environment, especially [30:29.720 --> 30:33.680] if you have engineers deploying their own [30:33.680 --> 30:37.920] applications. Make sure that all the namespaces [30:37.920 --> 30:42.300] have good controls in their RBAC accounting. [30:42.300 --> 30:46.040] Also, use third-party authentication for the API [30:46.040 --> 30:50.000] server. Again, something like HashiCorp Vault. Yes, I love [30:50.000 --> 30:54.100] that tool. If you are not familiar with it, get familiar with [30:54.100 --> 30:58.000] it. It's a great tool, and it integrates very, very well with your [30:58.000 --> 31:00.360] API server and Kubernetes. [31:01.680 --> 31:06.120] I already mentioned this. Separate and firewall your etc cluster. Make sure [31:06.120 --> 31:09.800] it's protected. That is the keys to the kingdom. [31:09.880 --> 31:13.700] It's the entire configuration of your cluster. And rotate [31:13.700 --> 31:17.880] encryption keys. Yes, use Vault. And by [31:18.380 --> 31:22.860] default, turn on security policies in the pods. [31:22.880 --> 31:25.860] It's some simple lines of configuration [31:25.860 --> 31:29.440] in the YAML files. It's not hard to do. Or [31:29.440 --> 31:33.920] use Docker Slim, that tool that I mentioned earlier, because [31:33.920 --> 31:38.000] that will help you set this up. Alright, let's talk [31:38.000 --> 31:41.920] about risk. kubectl, get secrets. That [31:41.920 --> 31:45.860] is the one that I just mentioned where you can look at the secrets [31:45.860 --> 31:49.840] from pretty much everything. But here's some tools [31:49.840 --> 31:53.740] that I absolutely love by CyberArk. One is called Kubernetes [31:53.740 --> 31:58.260] RBAC audit. The other is called KubeScan. [31:58.260 --> 32:02.100] The RBAC audit does exactly what you think it does. It audits [32:02.100 --> 32:05.560] all of your RBAC roles. The KubeScan [32:05.560 --> 32:10.000] will look at all the permissions that are granted from [32:10.000 --> 32:14.020] the roles. These are two great tools that [32:14.020 --> 32:17.860] you want to put into your toolbox for securing [32:17.860 --> 32:22.320] Kubernetes clusters. Because remember at the very beginning, I said [32:22.320 --> 32:26.320] it's all about configurations. It's the number one [32:26.320 --> 32:30.040] area that Kubernetes clusters get [32:30.040 --> 32:34.260] compromised. Also, we want to take a look [32:34.260 --> 32:38.260] at GitHub. Go out and make sure [32:38.260 --> 32:42.040] that... take that away... go out and make sure that [32:42.040 --> 32:46.260] your own configurations are not being [32:46.260 --> 32:50.880] shared up to GitHub by your engineers accidentally. [32:51.080 --> 32:54.340] How many times does this happen? It happens. [32:54.340 --> 32:58.580] I've had it happen by engineers. They don't do it on purpose. [32:58.580 --> 33:01.700] But they upload code. And in the code are secrets. [33:01.920 --> 33:05.540] Oops. Oops is not a good thing. [33:05.760 --> 33:10.400] Also, you need a locked down secure image repo. [33:10.400 --> 33:14.380] And create gold images. Don't just let your [33:14.380 --> 33:18.880] engineers go out and do, hey, docker run [33:18.880 --> 33:22.680] bring down Ubuntu and pull it from the internet. No. [33:22.680 --> 33:26.380] You need your own repo and registry for your docker [33:26.380 --> 33:30.280] images. Make sure that happens. And please secure the [33:30.280 --> 33:34.720] libraries. That's another big one. We also... [33:34.720 --> 33:39.120] I need an entire slide on just the applications. [33:39.940 --> 33:43.840] I'm going out on a limb. This is my personal opinion. [33:43.840 --> 33:47.180] But I want to say that nine times out of ten... okay, [33:47.180 --> 33:50.740] maybe eight and a half. Eight and a half times out of ten [33:50.740 --> 33:54.600] Kubernetes clusters and containers are compromised [33:55.120 --> 33:58.180] because the applications themselves [33:58.980 --> 34:02.860] were poorly written and not tested. [34:02.980 --> 34:06.900] And that's what gets me. I have [34:06.900 --> 34:10.720] environments where I see the engineers create [34:10.720 --> 34:14.660] an application. They do so much work to lock down [34:14.660 --> 34:18.880] Kubernetes in the container, but they do a poor [34:18.880 --> 34:23.260] job and there's a SQL injection vulnerability in the application directly. [34:23.360 --> 34:26.240] So then the application is compromised [34:26.900 --> 34:30.940] and then the container is compromised because, oops, [34:30.940 --> 34:35.500] it was running as root and so on and so on. Remember the dominoes. [34:35.500 --> 34:39.560] Everything falls over. Also, remember your [34:39.560 --> 34:43.420] pods. Another common [34:43.420 --> 34:47.160] mistake that people make when they're new to Kubernetes [34:47.160 --> 34:51.600] is they don't stop and think, how can I [34:51.600 --> 34:56.180] group things into pods and secure those pods? [34:56.180 --> 34:59.160] At least if a container falls [34:59.160 --> 35:03.380] within a pod, then maybe there's a couple more containers [35:03.380 --> 35:07.340] in there. Those containers that get compromised are going to be somewhat [35:07.340 --> 35:11.720] limited within the pod itself. Typically [35:12.060 --> 35:15.440] a pod is not going to fall over when another pod falls [35:15.440 --> 35:19.400] over. It's usually containers within the pods. So you want to think [35:19.400 --> 35:23.440] about grouping things. Also, Jason mentioned [35:23.440 --> 35:27.460] this in his talk earlier, but showdown. If you have [35:27.460 --> 35:31.300] never used showdown, get familiar with it. Even [35:31.300 --> 35:35.400] for Kubernetes. You can go out to showdown. You can start [35:35.400 --> 35:39.480] searching for Kubernetes. You can search for API clusters [35:39.480 --> 35:42.640] or API servers. You can search [35:43.140 --> 35:47.540] for etcd servers. And if you find one of yours, [35:47.840 --> 35:51.080] oops, but they're out there. [35:51.080 --> 35:54.800] I did a search this morning for some etcd [35:56.020 --> 35:57.540] exposure and [35:59.520 --> 36:01.660] there were over 6,000 [36:01.660 --> 36:05.580] that I found that were exposed. I don't know how many of those have [36:05.580 --> 36:09.380] vulnerabilities that would have allowed me to compromise them [36:09.380 --> 36:13.440] because I was just curious what the number would be. [36:13.560 --> 36:17.600] It goes up and down on a daily basis, but you should be [36:17.600 --> 36:21.460] aware of that. So use showdown. Let's talk about [36:21.460 --> 36:25.860] the configurations. I've said this before. [36:25.860 --> 36:29.380] Most of the breaches come from configuration issues. [36:32.890 --> 36:35.950] Use CIS hardening. Center for [36:35.950 --> 36:39.950] Internet Security. It is an amazing place. It is done by [36:39.950 --> 36:43.990] volunteers and workers within CIS. You can get [36:43.990 --> 36:48.670] benchmarks for Docker and you can get benchmarks for Kubernetes. [36:48.670 --> 36:51.790] You can use these to configure your environment [36:51.790 --> 36:54.750] so they are indeed hardened. [36:55.710 --> 36:58.630] Stop ignoring them. I still see people [36:58.630 --> 37:02.390] that tell me they've never heard of CIS and [37:02.390 --> 37:06.850] it makes me very sad when I hear that because it is such a great [37:06.850 --> 37:10.930] resource. The whole point here is [37:10.930 --> 37:14.710] we have to harden the environment from the [37:14.710 --> 37:18.750] standpoint of the basic configuration, but don't forget [37:18.750 --> 37:22.930] the application. Also, I know a lot of people [37:22.930 --> 37:27.010] have environments that are CICD when it [37:27.010 --> 37:30.950] comes to Docker and Kubernetes. That's why we have them. It's for [37:30.950 --> 37:34.870] speed. It's for development. It's for putting things out [37:34.870 --> 37:38.930] quickly. Okay, great. A lot of people use Jenkins. [37:39.690 --> 37:42.550] Is there some way to automate [37:42.550 --> 37:47.150] all of this so at least I can look at my images, [37:47.150 --> 37:51.310] I can look at my containers, all within the CICD pipeline? [37:51.310 --> 37:55.350] Yes, there is. There is a great tool called [37:55.350 --> 37:59.570] Anchor. Anchor, as it says here, is a [38:00.030 --> 38:03.570] container image scanner plugin. And what it does is [38:03.910 --> 38:07.710] you put it into the pipeline. It's a plugin for Jenkins. [38:07.710 --> 38:11.770] And it actually does security scans against the images [38:11.770 --> 38:15.610] that you're trying to deploy. This is a great way to [38:15.610 --> 38:19.370] get testing in before you actually [38:19.370 --> 38:23.350] deploy it to production. So let's talk about some [38:23.350 --> 38:27.450] additional tools before we get into the pen testing. Because we're still trying to [38:27.450 --> 38:31.250] lock it down. KubeBench. We want to run KubeBench. [38:31.250 --> 38:35.910] What it does is it runs against your CIS benchmarks. [38:35.910 --> 38:36.790] It displays [38:39.870 --> 38:43.650] a configuration that shows you what has passed, what has failed. I'll show an example [38:43.650 --> 38:47.570] in a minute. There's also KubeAudit. It's a little more granular [38:47.570 --> 38:51.810] than KubeSec. And if you run KubeBench [38:51.810 --> 38:55.770] and KubeAudit, it will give you a report that shows you [38:55.770 --> 38:59.210] oops, my system is insecure. Here's an example [38:59.210 --> 39:03.370] of a report. And when we look at it, we see allow [39:03.370 --> 39:07.730] privilege escalation is not set. There's a very [39:07.730 --> 39:11.870] first one. Oops, that needs to be turned on. [39:11.870 --> 39:15.790] Remember, that was the one that I mentioned that allows users [39:15.790 --> 39:20.030] to do privilege escalation. We don't want that. Oh, crap. [39:20.030 --> 39:23.930] We have run as non-root is not set. We don't have a read [39:23.930 --> 39:27.590] only file system set. There are so many [39:27.590 --> 39:31.830] simple things that can be done. Again, no app [39:31.830 --> 39:35.830] armor, no security comp. It's not being turned on. [39:35.830 --> 39:39.750] These are things that are just so easy. And this is from every [39:39.750 --> 39:43.810] single container within the environment that is [39:43.810 --> 39:47.690] being deployed under this particular. It's much [39:47.690 --> 39:51.630] longer. It goes into it. This is just a quick snapshot. [39:51.630 --> 39:55.930] The whole point here is you run this against your environment [39:55.930 --> 39:59.710] before you deploy it. And you go, oops, let's go back to our [39:59.710 --> 40:02.970] configuration file and fix all these things. [40:03.670 --> 40:08.010] Here are some more tools for you in the environment. Claire, [40:08.010 --> 40:11.490] another great tool for doing static analysis of [40:11.490 --> 40:15.470] applications. Claire does, it integrates [40:15.470 --> 40:19.490] Claire and your Docker registry so it can automate that [40:19.650 --> 40:23.650] a little bit. Falco, I need to put huge stars next [40:23.650 --> 40:27.490] to this one because this is actually one of my favorite, favorite [40:27.490 --> 40:31.490] tools. It was developed years [40:31.490 --> 40:35.810] ago and then it kind of morphed off and became falco.org. [40:35.810 --> 40:39.740] It monitors behavior within containers. [40:39.740 --> 40:43.100] It can detect if a container has been compromised [40:43.740 --> 40:47.320] because suddenly the container starts acting weird. [40:47.320 --> 40:51.920] Or there are commands being executed inside the container like [40:51.920 --> 40:55.940] cat etsy password. Probably not a command [40:55.940 --> 40:58.980] that should be showing up in one of your containers. [40:59.220 --> 41:03.920] If anything of this entire talk that I'm [41:03.920 --> 41:07.740] giving today, you get out of this, go look at falco. [41:07.740 --> 41:11.280] This is one of the best tools out there. Kubehunter [41:11.280 --> 41:15.280] is another great tool. We're going to talk about that. [41:15.540 --> 41:19.640] It's used more so for looking at doing some pen testing. [41:19.640 --> 41:24.140] We're going to take a look at that in just a second. Alright, ready, set, attack. [41:24.140 --> 41:27.700] If we were going to pen test an environment, this is an [41:27.700 --> 41:31.520] example of all the ports and all of the things that are [41:31.520 --> 41:35.800] open in a Kubernetes cluster. Now, they're [41:35.800 --> 41:39.800] open, but are they open by firewalls and so on? [41:39.800 --> 41:43.800] Well, we don't know. The API server [41:43.800 --> 41:47.680] itself is, yes, it's HTTPS. [41:47.680 --> 41:50.140] It better damn well be protected. [41:51.720 --> 41:55.920] etcd on 6666, that should be protected [41:55.920 --> 41:59.620] wherever it happens to be. This just gives you an idea. All of these [41:59.620 --> 42:03.820] ports can be changed. These are all the defaults in most cases. [42:04.360 --> 42:07.600] Just be aware of the ports that are out there and [42:07.600 --> 42:11.620] exposed that you need to make [42:11.620 --> 42:15.720] sure are protected. Because as an attacker, what am I [42:15.720 --> 42:20.100] going to do? Well, one of the first things, even Jason was talking about it in his talk, [42:20.100 --> 42:24.360] looking at ports, running nmap, running mass scan, and so on. [42:25.040 --> 42:27.380] This is exactly one of the things that [42:27.380 --> 42:31.700] Shodan does. It looks for all of these things that are exposed on the [42:31.700 --> 42:35.700] internet. So please make sure you're aware of all the ports [42:35.700 --> 42:39.920] within your cluster. Now, I are a hacker. [42:39.920 --> 42:43.720] See me hack. This is one of the [42:44.320 --> 42:47.920] ways that I'm going to start to attack [42:49.000 --> 42:51.620] a Kubernetes cluster. I want to first [42:51.620 --> 42:55.800] and foremost check access to the API server, make sure that is [42:55.800 --> 42:59.820] protected. I'm going to be looking at etcd, [42:59.820 --> 43:03.920] and making sure that's protected. If I find it exposed, [43:03.920 --> 43:06.640] I'm going to get keys to the kingdom in that case. [43:06.640 --> 43:11.700] What about the kubelets that are communicating and the command channels [43:11.700 --> 43:15.040] that are there? They should be read-only, but [43:15.700 --> 43:19.420] they should not be exposed. There was actually [43:22.440 --> 43:23.640] a vulnerability exposed [43:23.640 --> 43:27.640] about this, I want to say two months ago. I'm probably off in that [43:27.640 --> 43:31.920] sense. But take a look at that. And it allowed, even though it's a [43:31.920 --> 43:35.740] read-only port and it's supposed to limit what you can see, [43:35.740 --> 43:40.500] there were some vulnerabilities that allowed for a lot of exposure there. [43:40.500 --> 43:44.160] Also, container vulnerabilities. [43:44.160 --> 43:47.800] What about, you know, do you have files [43:47.800 --> 43:51.960] in that container that the application needs that [43:51.960 --> 43:55.600] you should be protecting? Well, that's where I would set up Falco and tell [43:55.600 --> 43:59.440] Falco, hey, there's a special file over here. This application [43:59.440 --> 44:03.500] is using it. But keep an eye on that. Make sure that the only [44:03.500 --> 44:07.540] thing that is reading that application is, or excuse me, [44:07.540 --> 44:10.720] that is reading that file happens to be the [44:11.600 --> 44:15.160] application. And Falco can monitor that. [44:15.400 --> 44:19.160] Privilege escalation. Nine times out of ten, once somebody [44:19.160 --> 44:23.380] compromises an application in a container, they [44:23.380 --> 44:27.240] elevate privilege because there's some other bugs in there. [44:27.240 --> 44:31.420] Go back to the Kubotic slide where [44:31.420 --> 44:35.560] it showed privilege escalation was not [44:35.560 --> 44:39.260] set. All you have to do is set that configuration [44:39.260 --> 44:43.760] variable. It's not that hard. Okay? [44:43.760 --> 44:46.840] Vulnerable applications are critical. [44:47.360 --> 44:51.480] Can't say it enough times. Especially because there are [44:51.480 --> 44:55.840] so many people using Kubernetes and Docker to deploy [44:55.840 --> 44:59.480] WordPress. And WordPress is, of course, one of [44:59.480 --> 45:03.500] our favorite things to deal with. All right. So your [45:03.500 --> 45:07.320] final threat model, when it comes to pentesting and [45:07.320 --> 45:11.560] looking at the environment that is Kubernetes, [45:12.220 --> 45:15.400] you think of your workload. You think of that [45:15.400 --> 45:19.220] security within your pods. Can I [45:19.220 --> 45:23.700] set up my pods so they're indeed secure? [45:23.700 --> 45:27.400] The container security itself, the configuration [45:27.400 --> 45:31.940] for that container, the more hooks I put in, [45:31.940 --> 45:35.620] run as root, no. Privilege escalation, [45:35.620 --> 45:39.420] no. These are the things that we want to make sure that are [45:39.420 --> 45:43.300] set. Also, file storage security. [45:43.300 --> 45:47.440] Maybe there's a particular file system that has to be read right [45:47.440 --> 45:51.380] but the rest of the file system can be read only. Those are [45:51.380 --> 45:56.500] other things. The network security. Remember the infrastructure. [45:56.520 --> 45:59.300] Put firewalls in. Put controls [45:59.300 --> 46:03.400] in there that detect lateral movement. If [46:03.400 --> 46:07.720] somebody does compromise one of your applications, [46:07.720 --> 46:11.360] make sure you have something monitoring. I gave you that [46:11.360 --> 46:15.680] slide way back about the blog entry from Wazoo. [46:15.680 --> 46:19.420] They talk about how to audit and monitor [46:20.000 --> 46:23.440] a Kubernetes cluster using their tool. I love the Wazoo [46:23.440 --> 46:27.580] manager and agent. Take a look at it if you haven't heard of it [46:27.580 --> 46:31.780] before. Remember, most [46:31.780 --> 46:35.500] breaches are not zero day. I can't say it enough times. Everybody's always [46:35.500 --> 46:39.600] trying to protect against this stuff. They're not fancy. [46:39.600 --> 46:43.380] Most of these things are being broken into because people [46:43.380 --> 46:47.540] make silly mistakes. Because remember, common sense. [46:47.540 --> 46:51.320] Whatever happened to it. The breaches are going to come from configuration [46:51.320 --> 46:55.400] issues. The kube audit was a very good [46:55.400 --> 46:59.440] example where it was showing all the problems that would allow someone to [46:59.440 --> 47:03.180] get in. Compromise credentials. [47:03.180 --> 47:07.260] Make sure your credentials are stored in a secure place [47:07.260 --> 47:11.140] somewhere. And don't allow for [47:11.140 --> 47:14.880] privilege escalation. Kubernetes and Docker [47:14.880 --> 47:18.980] give you the tools and the configurations to prevent that. [47:18.980 --> 47:23.320] Just turn them on. And that's what we're not doing. We have [47:23.320 --> 47:27.260] to turn on the configurations. So your key takeaways [47:27.260 --> 47:31.140] from today. Common sense. [47:31.140 --> 47:33.880] Please go back and stop [47:34.960 --> 47:40.240] and think about where is the exposure? [47:40.240 --> 47:44.660] Where are my pods exposed? What about my etcd? [47:45.240 --> 47:48.200] Configuration. That is so critical to any [47:48.200 --> 47:52.200] Kubernetes. And always remember containers. [47:52.200 --> 47:55.880] Containers are the linchpin of a Kubernetes [47:55.880 --> 48:00.140] environment. And if I don't secure my individual containers [48:00.140 --> 48:04.320] then Kubernetes... I don't care about [48:04.320 --> 48:08.460] the security of Kubernetes because the container is insecure. [48:08.460 --> 48:12.720] So stop overthinking this stuff and think about [48:12.720 --> 48:17.420] security basics. Remember the CIS benchmarks. [48:17.860 --> 48:20.540] Can't say it enough. Basics. [48:20.540 --> 48:24.440] I still think as security professionals [48:24.440 --> 48:28.480] hackers, whatever we want to call ourselves. We have [48:28.480 --> 48:32.340] forgotten the basics of how to secure an environment. [48:32.340 --> 48:35.780] And that's why things keep getting broken into. [48:35.780 --> 48:40.050] Stop doing fancy crap. The fancy crap doesn't work. [48:41.460 --> 48:43.940] Secure the environment, please. [48:44.020 --> 48:48.240] Use the CIS benchmarks for the containers, for Kubernetes, [48:48.240 --> 48:51.760] for the OS that your stuff runs on. If you're going to [48:51.760 --> 48:55.760] deploy this whole Kubernetes cluster on Ubuntu, then pull up [48:55.760 --> 48:59.280] the CIS benchmarks for Ubuntu. Or [48:59.280 --> 49:04.040] Red Hat or CentOS. Whatever you're using. [49:04.040 --> 49:08.060] Remember, RBAC is your friend. ABAC [49:08.060 --> 49:12.080] is not. Your infrastructure better be [49:12.080 --> 49:15.760] secure. Your pods better be secure. [49:15.760 --> 49:20.080] Containers themselves, the network. Still a lot of people forget [49:20.080 --> 49:24.160] the network because they forget about TLS everywhere. [49:26.880 --> 49:28.300] Threat model [49:28.300 --> 49:32.000] your applications. If you don't take [49:32.000 --> 49:36.500] the beginning effort to threat model your applications, [49:36.500 --> 49:40.180] you end up with insecure applications. And all the security [49:40.180 --> 49:43.580] of your Kubernetes cluster is useless because you have [49:43.580 --> 49:48.260] insecure applications within your environment. Access controls [49:48.260 --> 49:51.980] go along with RBAC, but access controls into the [49:51.980 --> 49:56.460] environment, kind of referencing SSH. [49:56.920 --> 49:59.760] I found one worker node configuration [49:59.760 --> 50:03.960] in an environment probably nine months ago. But for some [50:03.960 --> 50:08.700] reason on the worker nodes, they had FTP enabled on all the worker nodes. [50:08.820 --> 50:12.660] And my head exploded. [50:13.180 --> 50:16.540] It allowed me to get into everything, but oh well. [50:17.380 --> 50:19.920] Yes, did I mention RBAC though? [50:20.220 --> 50:23.480] Enough times. I can't say it enough. Why? Because [50:23.480 --> 50:28.000] people tend to ignore it. Remember the tools I mentioned earlier [50:28.000 --> 50:32.140] about monitoring or auditing the RBAC roles and also [50:32.140 --> 50:35.960] the permissions. And finally, [50:35.960 --> 50:39.900] not quite finally, remember to patch everything. [50:39.900 --> 50:43.820] The infrastructure that goes along with that. [50:43.860 --> 50:48.080] Logging and auditing and monitoring. I still see people [50:48.080 --> 50:52.700] set up their environments. There's that blog entry again. [50:52.700 --> 50:56.040] Please take a look at that. It is well written. [50:56.040 --> 50:59.960] It's a great tool. You'll learn a lot about Wazoo. I use it on [50:59.960 --> 51:04.120] all my systems. I deploy a lot of honeypots with [51:04.120 --> 51:08.100] Kubernetes and containers and things like that. But I monitor them all [51:08.100 --> 51:12.240] with Wazoo. You can't [51:12.240 --> 51:16.540] have a secure environment if you're not monitoring that environment. [51:16.540 --> 51:18.360] That is the biggest thing. [51:19.860 --> 51:24.080] That brings me to the end. I thank you for your [51:24.080 --> 51:28.180] attention. There's contact info. There's my Twitter [51:28.180 --> 51:32.260] account. I will get this to the Red Team [51:32.260 --> 51:36.360] Village and they can upload it to their environment. But I'll have it on my [51:39.160 --> 51:40.200] GitHub later [51:40.200 --> 51:44.360] in an hour or so. I'll update it there. Thank you all very much. [51:44.360 --> 51:48.140] Hopefully you learned something today. Go out and hack and have some fun. Thanks to [51:48.140 --> 51:51.480] the Red Team Village and have an awesome DEF CON. [51:51.480 --> 51:55.840] Awesome. Thank you so much, Kat. Amazing presentation [51:55.840 --> 51:59.500] as always. Thank you for supporting DEF CON. Thank you for supporting the [51:59.500 --> 52:02.700] community. Of course, the Red Team Village. For those of you [52:03.440 --> 52:06.940] that just joined or are looking at [52:07.340 --> 52:11.200] the presentation on Twitch or YouTube, please look at [52:11.200 --> 52:15.480] all the activities that we have going on. Go to the website that is in the description of [52:15.480 --> 52:19.540] the stream. Please also join the conversation in Discord. [52:19.540 --> 52:23.760] We're going to go on a brief break and the next presentation will start in a few minutes. [52:23.760 --> 52:25.340] Thank you again and thank you, Kat.