WEBVTT 00:01.280 --> 00:03.280 Ah, come back here. 00:03.280 --> 00:05.120 Why does that keep doing that? 00:05.600 --> 00:06.820 All right, great. 00:06.820 --> 00:14.660 We are here and ready to do securing in, path testing, great spaghetti monster, as I like to call it, or Kubernetes. 00:15.540 --> 00:18.400 This is who I am, Kat Fitzgerald. 00:18.400 --> 00:20.940 You can find me on Twitter at Rainbow Kat. 00:20.940 --> 00:22.840 I've been doing this a long time. 00:22.840 --> 00:25.440 I'm the CEO of B-Sides Chicago. 00:25.540 --> 00:34.900 I was COO for the Diana Initiative last year, part of B-Sides Pittsburgh, and also my first DEFCON was DEFCON 3. 00:34.900 --> 00:38.180 How many other people remember that one? 00:38.180 --> 00:42.560 There were 16 speakers there, if I have to remember that. 00:42.840 --> 00:45.360 I have an emphasis in the blue team. 00:45.360 --> 00:48.700 However, I'm a formal purple teamer and all this stuff. 00:48.700 --> 00:53.320 And I'm in Pittsburgh now, but moving to Kirkland, Washington very, very soon. 00:53.320 --> 01:03.740 You will always find me sipping Grand Mayan Extra Añejo, or in this case, it is Casanoble Añejo. 01:03.740 --> 01:04.840 So, cheers. 01:04.840 --> 01:06.620 Let's kick off DEFCON. 01:06.620 --> 01:08.960 Jason already did a great job. 01:11.120 --> 01:20.660 And also, special thanks to the entire Red Team Village, all the staff, all the volunteers, everybody for kicking butt and making this happen. 01:20.660 --> 01:24.540 A lot of my favorite stuff are honey pots and refrigerators. 01:24.540 --> 01:29.240 On Saturday, I'm doing a talk for IoT security, so check that out, too. 01:29.240 --> 01:39.380 But this is one of my favorite ones because I just finished writing this particular presentation just about a month ago, made some changes to it, and so on. 01:39.380 --> 01:43.520 Remember, please, the views and opinions here are mine and mine alone. 01:43.520 --> 01:48.420 They do not reflect anything from my employers, past or present. 01:48.800 --> 01:53.940 Anything that I show you, as always, please use with a grain of salt. 01:53.940 --> 01:57.500 Don't go breaking into things that you shouldn't be breaking into. 01:57.640 --> 02:07.680 And those of you with an overwhelming fear of the unknown, you'll be happy to know that there is no hidden message in the disclaimer if you read it backwards. 02:08.020 --> 02:10.340 Yes, my jokes are that bad. 02:10.380 --> 02:13.000 Alright, let's get this thing on the road. 02:13.600 --> 02:16.320 So, why are we not here? 02:16.700 --> 02:21.160 Well, first of all, I'm not going to solve all your Kubernetes security woes. 02:21.160 --> 02:22.640 I wish I could. 02:22.640 --> 02:27.180 I wish Kubernetes and containers and everything were secure by default. 02:27.180 --> 02:28.760 They are not. 02:29.340 --> 02:37.720 Neither is the person sitting next to you, although, well, that doesn't work anymore because nobody's sitting... well, maybe somebody's sitting next to you. 02:37.720 --> 02:38.520 You never know. 02:38.520 --> 02:39.940 You never know. 02:39.940 --> 02:40.920 Okay. 02:40.920 --> 02:44.360 And common sense went out the window decades ago. 02:44.360 --> 02:45.520 Think about that. 02:45.520 --> 02:57.060 When it comes to information security, cyber security, hacking, whatever you want to call it, I honestly think we lost common sense quite some time ago. 02:57.060 --> 02:59.800 Why really are we here? 03:00.380 --> 03:03.000 Well, Kubernetes, I don't know. 03:03.000 --> 03:05.980 I still think it's fairly new. 03:05.980 --> 03:06.740 It's fun. 03:06.740 --> 03:07.680 It's crazy. 03:07.680 --> 03:10.420 It's a lot of things. 03:10.420 --> 03:12.300 You can deploy it in clouds. 03:12.300 --> 03:14.560 You can deploy it in... 03:14.560 --> 03:18.080 I have Raspberry Pis sitting down here. 03:18.080 --> 03:19.680 I was going to be showing that. 03:19.680 --> 03:26.520 There have been some demo problems and also some timing issues, so I won't be showing my Raspberry Pi cluster today. 03:26.520 --> 03:34.780 But you can build very extensive Raspberry Pi clusters and do all the things that I'm going to talk about today. 03:35.340 --> 03:36.260 Containers. 03:36.360 --> 03:38.800 Containers are not new. 03:38.800 --> 03:45.640 But people seem to have forgotten that they have anything to do with security. 03:45.640 --> 03:51.680 They get all hyped up on Kubernetes security, and they're forgetting about containers. 03:51.680 --> 03:54.900 We're going to talk about containers because it's important. 03:54.900 --> 03:56.980 It's, in fact, very important. 03:56.980 --> 03:58.600 And always remember this. 03:58.600 --> 04:00.740 Security is for everyone. 04:01.940 --> 04:05.140 Common sense is required. 04:05.140 --> 04:08.540 So please disregard the previous slide. 04:08.540 --> 04:09.800 Throw it out the window. 04:09.800 --> 04:10.340 Why? 04:10.340 --> 04:18.220 Because if we're not using common sense in security, then we're definitely not going to get where we need to be. 04:18.640 --> 04:19.220 All right. 04:19.220 --> 04:21.660 Just some quick security facts. 04:21.660 --> 04:23.500 We want to know this stuff. 04:23.500 --> 04:32.540 In 2018 alone, $114 billion were spent by companies on information security. 04:32.540 --> 04:39.100 On fancy software, fancy hardware, things with blinky lights and all of that. 04:39.100 --> 04:41.860 But breaches kept on going up. 04:41.860 --> 04:45.140 We're doing something wrong here. 04:45.140 --> 04:54.100 Security appliances and software have vulnerabilities that we're not dealing with or that need to be dealt with by the vendors. 04:54.100 --> 04:56.700 Kubernetes itself has had a few issues. 04:56.700 --> 04:57.720 We'll talk about that. 04:57.720 --> 04:59.700 But so has Docker. 04:59.700 --> 05:01.960 So don't forget your containers. 05:01.960 --> 05:08.720 Also, I've actually had people say to me, we have a very unique security architecture. 05:08.780 --> 05:11.460 And I'm like, no, you don't. 05:11.560 --> 05:13.760 We all do the same stuff. 05:13.760 --> 05:16.700 So remember, your architecture is not unique. 05:16.700 --> 05:18.080 I love this quote. 05:18.080 --> 05:24.060 If you don't have this book, Offensive Countermeasures by John Strand, go out and get it. 05:24.060 --> 05:27.220 It's a small book with a ton of information. 05:27.220 --> 05:28.560 So get it. 05:28.560 --> 05:33.900 But he said, instead of brilliance, we've standardized on mediocrity. 05:33.900 --> 05:40.100 And that seems to be what information security is about these days. 05:40.100 --> 05:42.840 And I kind of have issues with that. 05:42.840 --> 05:44.840 So let's talk about that some more. 05:44.840 --> 05:49.820 First of all, let's get a common definition of what is a breach. 05:50.240 --> 05:53.440 Or maybe we'll call it what is an incident. 05:53.440 --> 05:56.200 Let's not necessarily call it a breach. 05:56.200 --> 06:03.200 So most incidents are not caused by zero-day fancy stuff that you find on the Internet. 06:03.200 --> 06:08.160 Typically, they're done because people have made mistakes and so on. 06:08.160 --> 06:14.420 Most breaches are not coming from your vulnerability scanners, especially when it comes to Kubernetes. 06:14.420 --> 06:21.780 You're not going to run a vulnerability scanner and go, oh, look, fix this one thing, and we're done and ready to go. 06:21.780 --> 06:24.120 No, it doesn't happen that way. 06:25.420 --> 06:30.680 Most breaches are going to come from configuration issues. 06:30.820 --> 06:33.560 This is something I want to emphasize. 06:33.560 --> 06:39.820 In fact, I'm going to show you some examples how configuration issues tend to cause most of the breaches. 06:39.820 --> 06:44.460 A close second is compromised credentials. 06:44.500 --> 06:49.040 When I say credentials, I still go out on GitHub. 06:49.240 --> 06:51.460 Well, maybe not every single day. 06:51.460 --> 06:56.080 And I think Jason even mentioned it, too, in his previous talk on bug bounties. 06:56.320 --> 07:02.340 Go out and look on GitHub and search for simple things like password. 07:02.480 --> 07:06.120 See how many times you find the word root. 07:06.800 --> 07:09.720 Yes, thousands of them are invalid. 07:09.720 --> 07:11.000 They don't make any sense. 07:11.000 --> 07:17.140 It's probably going to find the keyword root or password, and it's not going to get you anywhere. 07:17.140 --> 07:24.000 And yet I still find where people hard-code credentials into variables that say password. 07:24.000 --> 07:26.500 And right next to it, there's the key. 07:26.500 --> 07:29.620 There's the whatever the password is. 07:30.640 --> 07:37.880 Also, trailing in third place for most incidents is overprivileged users. 07:37.880 --> 07:42.560 This is incredibly important in the Kubernetes world. 07:42.560 --> 07:46.780 I'm going to show you this in just a second or in a couple of slides here. 07:46.780 --> 07:49.160 So keep these three things in mind. 07:49.160 --> 07:54.360 Configuration issues, compromised credentials, and overprivileged users. 07:54.780 --> 07:55.700 Okay. 07:56.080 --> 08:09.880 Kubernetes itself, we have to deal with all three of these compromises or these types of incidents that are going to cause issues in Kubernetes. 08:09.960 --> 08:14.340 But first, think about this. 08:14.340 --> 08:18.900 You know, we think about the Capital One hack that happened recently. 08:18.900 --> 08:27.420 Well, that was a combination of compromised credentials and access that should not have been granted. 08:27.500 --> 08:34.060 We see how Tesla, you know, they had poor access credentials and so on. 08:34.120 --> 08:37.160 Equifax, we all know about that one. 08:37.160 --> 08:39.520 Go back and check if you don't. 08:39.520 --> 08:54.320 You know, Equifax was hacked not only by a simple vulnerability that had not been patched, but also because they had very very poor lateral movement detection on the inside. 08:54.320 --> 09:04.160 Once they got in with more compromised credentials through the vulnerability, they weren't detected for quite some time. 09:04.160 --> 09:06.100 This is going to be critical. 09:06.100 --> 09:08.600 And one other thing, I was bored this morning. 09:08.600 --> 09:11.700 I decided to throw this up here in this slide. 09:11.700 --> 09:12.300 Why? 09:12.300 --> 09:15.520 Well, I put up a new spice rack in my kitchen. 09:15.520 --> 09:17.060 I really liked how it looks. 09:17.060 --> 09:20.100 I took a picture and I added it to the slide. 09:20.260 --> 09:21.220 There it is. 09:21.220 --> 09:23.240 Isn't it a great spice rack? 09:23.240 --> 09:24.280 Come on. 09:24.280 --> 09:26.620 Alright, you all better be laughing. 09:27.040 --> 09:29.460 Alright, let's talk about containers. 09:30.180 --> 09:32.280 Quick review of containers. 09:32.280 --> 09:34.780 They are not secure by default. 09:34.780 --> 09:42.680 Now, I'm really focusing more on Docker containers for now because that tends to be the norm. 09:42.680 --> 09:51.000 And one thing to think about when it comes to containers, containers are not like a VM. 09:51.000 --> 09:53.900 A VM is actually more isolated. 09:53.900 --> 09:57.520 And this is something that's going to be important coming up here. 09:57.520 --> 10:04.300 So, a container is a grouping of stuff and the stuff that we like to call it are resources. 10:04.300 --> 10:08.420 Well, the resources are grouped into namespaces. 10:08.420 --> 10:12.160 If you are new to containers, remember this. 10:12.160 --> 10:14.380 Namespaces are critical. 10:14.400 --> 10:20.480 You group your processes, your networks, your users, your ports, your interprocess. 10:20.480 --> 10:28.160 All of this stuff can be controlled and locked down within namespaces. 10:28.360 --> 10:30.700 Resources are exploitable. 10:30.700 --> 11:01.540 I was sitting in another web broadcast just the other day on containers and I was surprised because it was supposed to be a more intermediate presentation on containers and yet all of the people were asking is there a way to prevent process abuse such as CPUs and memory and so on. 11:01.540 --> 11:17.740 And again, this was supposed to be an intermediate class on containers so it confused me a little bit that people were asking that because that should be something that you know from the very beginning that you get into containers. 11:17.740 --> 11:24.480 Using namespaces and the next most important thing is control groups or cgroups. 11:24.480 --> 11:29.460 These are the limits that you are going to be putting on containers. 11:29.460 --> 11:35.180 If you don't put limits on them, you're going to run into problems. 11:35.180 --> 11:53.220 Every time I do a pen test of a new application deployed into a container, I would say that 7 out of 10 times I find that the author of the application who has put it in a container has not put any limits. 11:53.220 --> 12:02.540 I'll get into the application in whatever way possible whether it's a SQL injection or an RCE or something like that. 12:02.540 --> 12:13.800 And once I'm in, I can take over not just the container but the entire system that it's on because there are no cgroups applied to the container. 12:13.800 --> 12:25.880 The other thing to remember about containers is think of a container as an image or more correctly, think of it as a file system snapshot. 12:26.060 --> 12:30.720 So who decides what goes in that image? 12:30.720 --> 12:40.200 If you look at this particular picture, you'll see, you know, here we have namespaces, the control groups that I show. 12:40.480 --> 12:46.080 It's going to possibly be cherooted if you've set it up correctly. 12:46.180 --> 12:48.860 And here's the application. 12:49.400 --> 12:53.300 There's a Docker client that you might use to launch this. 12:53.300 --> 12:55.820 There's a Docker registry. 12:55.860 --> 13:00.240 This is critical to the security of your system. 13:00.240 --> 13:08.600 If you're new to Docker, you may not know this, but Docker Hub was compromised two years ago, I believe it was. 13:08.600 --> 13:10.900 I've lost track of time during COVID. 13:10.900 --> 13:12.520 I don't know anymore. 13:12.860 --> 13:25.800 But you want to have a secure registry where you keep your images because if they're somewhere that you don't have control, you never know what's going to happen. 13:25.800 --> 13:27.380 But here's the difference. 13:27.380 --> 13:34.280 Notice that we have a host OS and hardware down here, unlike a VM. 13:34.280 --> 13:40.800 In a VM, you would see the entire operating system within the VM here. 13:40.800 --> 13:43.200 You would not see a shared OS. 13:43.200 --> 13:48.480 Yes, there's a host OS, but then you also have the guest operating systems themselves. 13:48.480 --> 13:57.980 And that's where we get into how a VM is much more isolated and contained than a container is. 13:57.980 --> 14:02.540 So don't think the word container means that you're more secure. 14:02.540 --> 14:05.780 It just means that we have some work to do. 14:05.780 --> 14:10.120 So that's why I say right here in this slide, container isolation. 14:10.320 --> 14:12.740 Is it really isolated? 14:12.740 --> 14:17.180 The simple answer to that is no, it is not. 14:17.180 --> 14:20.880 So let's talk about some container best practices. 14:20.880 --> 14:26.540 I should have also said at the very beginning of this talk, this is an intro. 14:26.540 --> 14:28.220 I can't show you everything. 14:28.220 --> 14:39.760 I'm trying to get people to start thinking about how you secure containers, then we're going to jump into how you secure Kubernetes at the same time. 14:40.200 --> 14:43.620 So remember, we have a base image. 14:43.620 --> 14:47.740 That means that base image has to come from a secure repo. 14:47.740 --> 15:05.640 If you do not know what CIS is, definitely go to CIS or CISecurity.org Check it out because you want to use CIS benchmarks for locking down your base image. 15:05.640 --> 15:08.260 I'm going to talk about that a little more coming up. 15:08.480 --> 15:11.060 Don't run your images as root. 15:11.220 --> 15:19.440 By default, if you do not set a user name, your container and your application in the container is going to run it as root. 15:19.440 --> 15:37.260 I have found so many who have deployed WordPress applications in a Docker container via Kubernetes and they don't set the user name and WordPress is running as root, so is the database and everything else. 15:37.260 --> 15:40.980 So of course, once it's compromised, you take over everything. 15:41.420 --> 15:43.560 Also, visibility. 15:43.780 --> 15:47.400 You need monitoring within your containers. 15:47.400 --> 15:53.620 Your containers put a lot of information out and they can log it, but you have to capture that. 15:53.620 --> 15:55.820 We're going to talk about that in a minute. 15:56.020 --> 16:00.020 Also, think of it as a domino. 16:00.020 --> 16:09.260 If one container is compromised, I guarantee you the rest of them will fall over like dominoes. 16:09.880 --> 16:14.520 When you start to get into containers and Kubernetes, you're going to see this. 16:14.920 --> 16:18.860 Also, we want to set some controls. 16:18.900 --> 16:33.780 Besides setting a user name that is not root, we also set privileges and we set certain attributes to those users where it says no elevated privileges or no new privileges. 16:33.780 --> 16:41.780 You don't want user accounts or IDs to be able to escalate privileges. 16:41.780 --> 16:44.040 And I'll show you some examples coming up. 16:44.040 --> 16:49.100 This one, my god, this is near and dear to my heart, secrets management. 16:49.100 --> 17:13.840 I still find, I would love to say by a show of hands, but I can't do that right here, how many people put the secrets and tokens and things like that in their Docker YAML files so when they then deploy the Docker image, if somebody gets a hold of their YAML file, 17:13.840 --> 17:21.240 which describes the configuration of that container, well, the secrets are right in it. 17:21.500 --> 17:39.300 I've worked with a lot of developers and a lot of engineers and I get them, you know, you have to start thinking about putting your secrets in another place, whether it's HashiCorp Vault, whether it's a cloud secrets management system, something. 17:39.300 --> 17:44.240 But get them out of the apps and get them out of the configuration files. 17:44.240 --> 17:48.220 And stop allowing SSH into your hosts. 17:48.220 --> 18:02.160 When we get into Kubernetes, coming up in a couple slides here, one of the key constructs of Kubernetes is you have a master node or master nodes, and then you have worker nodes. 18:02.160 --> 18:11.560 And I still find worker nodes with SSH enabled on them so people can SSH in and, quote, fix things. 18:11.560 --> 18:18.360 No, no, you use the management tools to fix things and redeploy. 18:18.360 --> 18:21.520 Stop allowing SSH into your systems. 18:21.520 --> 18:28.120 Also, I can't stress it enough times, namespace limits. 18:28.120 --> 18:33.800 Go read up on how to control namespaces. 18:33.800 --> 18:50.160 That is going to be one of your best efforts and best practices for containing and controlling and securing a container before you distribute it to or deploy it via Kubernetes. 18:50.160 --> 18:51.880 This is one of my favorite tools. 18:51.880 --> 18:53.700 It's called Docker Slim. 18:53.720 --> 19:11.020 If you've never heard of it, if I were to create a container, let's say I create a container, it has Ubuntu in it or the base of Ubuntu, and then maybe it has WordPress running in it for whatever god awful reason that I want to do this. 19:11.020 --> 19:17.780 Let's say that entire image ends up being three to four hundred meg. 19:17.780 --> 19:40.960 It's still manageable, but if I were to run that through Docker Slim, not only will it add some extra configurations and namespace limits and so on, but it shrinks it down and something that might be three or four hundred megabytes deployed into a container will get shrunk down to maybe 10, 19:40.960 --> 19:42.760 15, 20 meg. 19:42.760 --> 19:46.240 This makes it even that much more manageable. 19:46.240 --> 19:48.240 It's easier to deploy. 19:48.240 --> 19:55.260 It's quicker to deploy, especially if there are requirements for scaling. 19:55.260 --> 19:56.420 So think about this. 19:56.420 --> 19:57.540 Take a look at this. 19:57.540 --> 20:12.000 I also mentioned in Discord that my slides will be available in my GitHub repo, and I'll get them to the Red Team Village so they can add them to the Google Docs location after my talk. 20:13.260 --> 20:18.520 And always remember apply SELinux or AppArmor. 20:18.520 --> 20:23.120 One of the things that Docker Slim does is it will do that for you. 20:23.120 --> 20:28.740 It's very easy to enable SELinux or AppArmor within a container. 20:28.740 --> 20:32.580 If you can do that, you're going to make it that much more secure. 20:32.580 --> 20:36.960 It's not on by default, but you can set it up so it will be. 20:39.140 --> 20:44.040 And remember the infrastructure, as I put it. 20:44.640 --> 20:56.400 You can't forget that all of this stuff is being deployed on an infrastructure of some sort, and that infrastructure better be secure. 20:56.400 --> 20:58.040 I mentioned SSH. 20:58.040 --> 21:02.080 People keep allowing SSH into their environment. 21:02.440 --> 21:07.220 Be very aware that your infrastructure itself should be secure. 21:07.220 --> 21:09.860 And I will remind you later about that. 21:09.860 --> 21:10.180 All right. 21:10.180 --> 21:11.280 Here's a quick picture. 21:11.280 --> 21:13.500 Picture is worth a thousand words, right? 21:13.500 --> 21:18.960 If you look at our first example, this is our monolithic operating system here. 21:18.960 --> 21:20.720 We see hardware and operating system. 21:20.720 --> 21:21.820 We run apps. 21:21.820 --> 21:23.120 Here we see a VM. 21:23.120 --> 21:24.960 So we see hardware operating system. 21:24.960 --> 21:27.300 There's a hypervisor involved. 21:27.900 --> 21:30.440 And then there's going to be virtual machines. 21:30.440 --> 21:36.340 But when you have virtual machines, you have additional operating systems, you have additional libraries. 21:36.460 --> 21:38.500 They're very much isolated. 21:38.500 --> 21:46.820 I should have used a different picture where there's a big line through it that shows, yes, these are isolated and so on. 21:46.820 --> 21:49.780 But that should not have happened. 21:49.940 --> 21:50.300 Nope. 21:50.300 --> 21:51.380 Go back. 21:51.540 --> 21:52.560 Sorry. 21:52.620 --> 21:53.140 Okay. 21:53.140 --> 22:00.040 But when we look at using Docker or Container, we go hardware operating system. 22:00.040 --> 22:01.700 There is a container runtime. 22:01.700 --> 22:03.680 There's no real dividing lines. 22:03.680 --> 22:05.060 And then we see containers. 22:05.060 --> 22:13.800 The only thing that is unique between the containers is that bin library, the things that are going to run in the application. 22:13.800 --> 22:17.180 So there's not a lot of isolation there. 22:17.180 --> 22:21.360 And people want to think that there is. 22:21.360 --> 22:22.500 But there is not. 22:22.500 --> 22:24.660 So please keep that in mind. 22:25.260 --> 22:28.280 I do love this comic. 22:28.280 --> 22:30.500 It came out a couple of years ago. 22:30.500 --> 22:37.440 A great defense is, you know, we'll inundate them with quarterly Kubernetes releases. 22:37.440 --> 22:41.180 Yes, they do update Kubernetes a lot. 22:41.180 --> 22:49.960 That's why you have to pay attention to your environment and the tools that you're going to use so you can do upgrades to your Kubernetes cluster. 22:49.960 --> 22:52.320 So let's bring on the orchestrator. 22:52.320 --> 22:54.040 Let's talk about Kubernetes. 22:54.300 --> 22:54.800 All right. 22:54.800 --> 22:56.720 Here is Kubernetes 101. 22:56.720 --> 23:04.000 In one slide, I'm going to teach you at least the basics of how Kubernetes works. 23:04.000 --> 23:05.680 We have a master node. 23:05.680 --> 23:07.960 The master node is critical. 23:07.960 --> 23:12.000 It manages all of the Kubernetes worker nodes. 23:12.000 --> 23:13.700 It manages the clusters. 23:13.700 --> 23:15.760 It deploys pods. 23:15.760 --> 23:23.000 Think of a pod as a collection of containers that are grouped. 23:23.000 --> 23:31.900 A pod might be the way that you decide, oh, I want to have a pod that supports a particular client. 23:31.900 --> 23:34.500 Or maybe it's a WordPress instance. 23:34.500 --> 23:37.460 And that pod has WordPress. 23:37.460 --> 23:42.820 It has a container, a separate container for the database and so on. 23:42.820 --> 23:51.180 So the pod is how you can group things that are various containers into certain groupings. 23:51.180 --> 23:53.760 And those groupings are kind of defined by you. 23:53.760 --> 23:57.420 But stop and think about how you want to deploy pods. 23:57.420 --> 23:58.920 It's going to be important. 23:58.920 --> 24:00.420 A worker node. 24:00.420 --> 24:03.580 The node is where all the containers are going to run. 24:03.580 --> 24:08.080 You have other Kubernetes components that have to run there. 24:08.080 --> 24:11.940 But the important part is, this is where the containers run. 24:11.940 --> 24:28.020 If the worker node is not secure and not patched, and I can get into that underneath the visibility of Kubernetes, then Kubernetes has no idea that I've compromised a node. 24:28.020 --> 24:39.300 I found numerous Kubernetes sites around the Internet where there's additional applications running on worker nodes because they've been compromised. 24:39.300 --> 24:40.920 Of course, they're Bitcoin miners. 24:40.920 --> 24:54.900 But the master node has no idea that it's running on there because they got in via SSH and they launched the application, the miner, underneath the visibility of Kubernetes. 24:54.900 --> 24:58.420 So you really want to lock down these worker nodes. 24:58.420 --> 25:06.300 A pod, I already explained, it's a unit of deployment and it's also how you address things from Kubernetes. 25:06.300 --> 25:08.880 And a pod will have an IP address. 25:08.880 --> 25:12.000 So that's important to remember. 25:12.100 --> 25:15.080 But it's going to contain one or more containers. 25:15.080 --> 25:22.840 I like to think of pods as, even though it might have its own IP address, but think of it like a VLAN in a way. 25:22.840 --> 25:26.420 Because I've isolated it in one way. 25:26.440 --> 25:34.400 Services are various things that run within Kubernetes that communicate, that balance, that do a lot of work. 25:34.400 --> 25:36.540 We're going to look at services coming up. 25:36.540 --> 25:39.760 There are various components that are used. 25:39.760 --> 25:48.040 Three that are critical, the API server, you're going to have kubelets, and you're also etcd. 25:48.040 --> 26:01.380 The one concept you should take about etcd is this is the database that describes and builds everything that is your Kubernetes cluster. 26:01.380 --> 26:11.560 The etcd environment should actually be walled off and firewalled and controlled very, very securely. 26:11.560 --> 26:17.700 If you do that, you will have a much more secure Kubernetes cluster. 26:17.700 --> 26:19.760 So keep that in mind. 26:19.760 --> 26:23.040 And you can see it here by looking at a picture. 26:23.040 --> 26:33.660 So we take all of those concepts that we just talked about we see here's an ABI server, here's a controller, this is all part, you know, the kubectl... 26:33.660 --> 26:36.140 Stop doing that. 26:37.220 --> 26:38.000 Come on. 26:38.000 --> 26:42.640 Kubectl is the command you use to control everything. 26:43.020 --> 26:54.920 And even though this is a disk kind of a representation of etcd, I like to say, you know what, this needs to be off on its own, firewalled, and separated, and it can be done that way. 26:54.920 --> 27:00.820 Here's your worker node, and you can see how you might have multiple pods on a worker node. 27:00.820 --> 27:03.760 This pod might be talking to this pod. 27:03.760 --> 27:08.400 It doesn't necessarily have to talk to that one, or they may not talk at all. 27:08.400 --> 27:11.780 These are some of the other services that are running the proxies. 27:11.780 --> 27:16.680 This is where we are exposed to the internet, not over here. 27:16.680 --> 27:17.320 Okay? 27:17.320 --> 27:22.240 So the only place you're exposing Kubernetes is at the worker nodes. 27:22.780 --> 27:26.280 Now, Kubernetes is not perfect. 27:26.520 --> 27:28.780 There have been problems. 27:28.780 --> 27:29.980 There was... 27:29.980 --> 27:35.560 I need to update this slide again, because there have been a couple of other Kubernetes. 27:35.660 --> 27:37.600 These are more for reference. 27:37.600 --> 27:39.560 One's a little older. 27:39.560 --> 27:43.860 This was back... there was the backdoor that involved the API server. 27:43.860 --> 27:45.060 There were some others. 27:45.060 --> 27:49.620 These are only for your reference, so you can go look them up and see. 27:49.780 --> 27:51.620 And there are more . 27:51.620 --> 28:00.080 The whole point I'm trying to make here is Kubernetes is not secure by default, and it's not perfect . 28:00.680 --> 28:03.780 Well, let's try to make it perfect. 28:03.900 --> 28:17.460 Let's take a common Kubernetes cluster and say, you know, if we were to threat model this, and that is very, very important, first thing we want to do is to say, TLS everywhere. 28:17.920 --> 28:22.240 Yeah, I know, I have too many exclamation points, but I'm trying to make the point. 28:22.340 --> 28:31.460 If everything within your Kubernetes cluster is using TLS and is encrypted, then it's that much safer. 28:31.460 --> 28:38.000 I still have engineers come to me and say, our application is completely internal. 28:38.000 --> 28:39.740 Why do we need TLS? 28:39.740 --> 28:42.680 It's never going to be exposed to the internet. 28:42.680 --> 28:43.700 I said, really? 28:43.700 --> 28:57.800 So what happens if Joe's application over here, which is exposed to the internet, is compromised, and then they use that as a pivot point to gain access to this environment, and then they compromise your system? 28:58.120 --> 29:01.120 O is typically the response. 29:01.180 --> 29:05.180 So I can't say it enough, harden the bloody infrastructure. 29:05.320 --> 29:12.740 Yes, I'm saying it again, but you're running this thing on your infrastructure, make sure it's hardened. 29:12.740 --> 29:15.860 And very important, two concepts. 29:16.060 --> 29:20.940 There's RBAC, which is regular role-based access control. 29:20.940 --> 29:22.780 Set that up. 29:22.780 --> 29:25.220 Make sure you're doing it right. 29:25.240 --> 29:27.060 Disable ABAC. 29:27.180 --> 29:33.360 ABAC is incredibly difficult to manage. 29:33.600 --> 29:46.000 You will spend days and hours and hours trying to do ABAC access controls, which is more granular, but it's not worth the effort. 29:46.000 --> 29:50.060 Make sure you use RBAC and manage it correctly. 29:50.060 --> 29:55.560 And I'm going to show you a couple of tools coming up on how to check it. 29:55.560 --> 29:59.320 Also, check out this blog entry. 29:59.440 --> 30:03.340 I mentioned this previously about monitoring. 30:03.340 --> 30:08.640 This is one way to audit Kubernetes, so keep this in mind. 30:08.640 --> 30:19.520 Take a look at that blog entry from Wazoo, and it's an excellent place to start when it comes to monitoring your environment. 30:19.520 --> 30:24.560 Also, kubectl, get role binding in all your namespaces. 30:24.560 --> 30:35.040 You should do this periodically in your environment, especially if you have engineers deploying their own applications. 30:35.040 --> 30:42.300 Make sure that all the namespaces have good controls in their RBAC accounting. 30:42.300 --> 30:46.820 Also, use third-party authentication for the API server. 30:46.820 --> 30:49.020 Again, something like HashiCorp Vault. 30:49.020 --> 30:50.780 Yes, I love that tool. 30:50.780 --> 30:54.400 If you are not familiar with it, get familiar with it. 30:54.400 --> 31:00.360 It's a great tool, and it integrates very, very well with your API server and Kubernetes. 31:01.680 --> 31:03.500 I already mentioned this. 31:03.500 --> 31:05.420 Separate and firewall your etc cluster. 31:05.420 --> 31:07.580 Make sure it's protected. 31:07.580 --> 31:09.800 That is the keys to the kingdom. 31:09.880 --> 31:12.860 It's the entire configuration of your cluster. 31:12.860 --> 31:14.760 And rotate encryption keys. 31:14.760 --> 31:16.480 Yes, use Vault. 31:16.920 --> 31:22.860 And by default, turn on security policies in the pods. 31:22.880 --> 31:27.580 It's some simple lines of configuration in the YAML files. 31:27.580 --> 31:29.140 It's not hard to do. 31:29.140 --> 31:36.020 Or use Docker Slim, that tool that I mentioned earlier, because that will help you set this up. 31:37.020 --> 31:39.380 Alright, let's talk about risk. 31:39.620 --> 31:41.420 kubectl, get secrets. 31:41.420 --> 31:48.560 That is the one that I just mentioned where you can look at the secrets from pretty much everything. 31:48.560 --> 31:52.380 But here's some tools that I absolutely love by CyberArk. 31:52.380 --> 31:55.100 One is called Kubernetes RBAC audit. 31:55.100 --> 31:58.260 The other is called KubeScan. 31:58.260 --> 32:01.540 The RBAC audit does exactly what you think it does. 32:01.540 --> 32:04.660 It audits all of your RBAC roles. 32:04.660 --> 32:11.280 The KubeScan will look at all the permissions that are granted from the roles. 32:11.280 --> 32:19.680 These are two great tools that you want to put into your toolbox for securing Kubernetes clusters. 32:19.680 --> 32:24.700 Because remember at the very beginning, I said it's all about configurations. 32:24.700 --> 32:31.400 It's the number one area that Kubernetes clusters get compromised. 32:31.860 --> 32:36.260 Also, we want to take a look at GitHub. 32:36.260 --> 32:50.880 Go out and make sure that... take that away... go out and make sure that your own configurations are not being shared up to GitHub by your engineers accidentally. 32:51.080 --> 32:53.260 How many times does this happen? 32:53.260 --> 32:54.340 It happens. 32:54.340 --> 32:56.460 I've had it happen by engineers. 32:56.460 --> 32:58.580 They don't do it on purpose. 32:58.580 --> 32:59.980 But they upload code. 32:59.980 --> 33:01.700 And in the code are secrets. 33:01.920 --> 33:02.880 Oops. 33:03.080 --> 33:05.540 Oops is not a good thing. 33:05.760 --> 33:10.400 Also, you need a locked down secure image repo. 33:10.400 --> 33:12.640 And create gold images. 33:12.640 --> 33:22.120 Don't just let your engineers go out and do, hey, docker run bring down Ubuntu and pull it from the internet. 33:22.120 --> 33:22.680 No. 33:22.680 --> 33:27.340 You need your own repo and registry for your docker images. 33:27.340 --> 33:28.680 Make sure that happens. 33:28.680 --> 33:31.360 And please secure the libraries. 33:31.360 --> 33:33.000 That's another big one. 33:33.000 --> 33:34.720 We also... 33:34.720 --> 33:39.120 I need an entire slide on just the applications. 33:39.940 --> 33:41.800 I'm going out on a limb. 33:41.800 --> 33:43.840 This is my personal opinion. 33:43.840 --> 33:48.600 But I want to say that nine times out of ten... okay, maybe eight and a half. 33:48.600 --> 34:02.860 Eight and a half times out of ten Kubernetes clusters and containers are compromised because the applications themselves were poorly written and not tested. 34:02.980 --> 34:05.680 And that's what gets me. 34:05.680 --> 34:11.740 I have environments where I see the engineers create an application. 34:11.740 --> 34:23.260 They do so much work to lock down Kubernetes in the container, but they do a poor job and there's a SQL injection vulnerability in the application directly. 34:23.360 --> 34:33.980 So then the application is compromised and then the container is compromised because, oops, it was running as root and so on and so on. 34:33.980 --> 34:35.500 Remember the dominoes. 34:35.500 --> 34:37.860 Everything falls over. 34:38.000 --> 34:41.160 Also, remember your pods. 34:41.960 --> 34:56.180 Another common mistake that people make when they're new to Kubernetes is they don't stop and think, how can I group things into pods and secure those pods? 34:56.180 --> 35:04.220 At least if a container falls within a pod, then maybe there's a couple more containers in there. 35:04.220 --> 35:10.940 Those containers that get compromised are going to be somewhat limited within the pod itself. 35:11.080 --> 35:16.060 Typically a pod is not going to fall over when another pod falls over. 35:16.060 --> 35:18.240 It's usually containers within the pods. 35:18.240 --> 35:21.400 So you want to think about grouping things. 35:21.400 --> 35:26.660 Also, Jason mentioned this in his talk earlier, but showdown. 35:26.660 --> 35:30.940 If you have never used showdown, get familiar with it. 35:30.940 --> 35:32.520 Even for Kubernetes. 35:32.520 --> 35:34.520 You can go out to showdown. 35:34.520 --> 35:37.520 You can start searching for Kubernetes. 35:37.520 --> 35:41.660 You can search for API clusters or API servers. 35:41.660 --> 35:45.480 You can search for etcd servers. 35:45.480 --> 35:51.080 And if you find one of yours, oops, but they're out there. 35:51.080 --> 36:03.560 I did a search this morning for some etcd exposure and there were over 6,000 that I found that were exposed. 36:03.560 --> 36:13.440 I don't know how many of those have vulnerabilities that would have allowed me to compromise them because I was just curious what the number would be. 36:13.560 --> 36:18.760 It goes up and down on a daily basis, but you should be aware of that. 36:18.760 --> 36:20.340 So use showdown. 36:20.640 --> 36:22.880 Let's talk about the configurations. 36:24.400 --> 36:25.860 I've said this before. 36:25.860 --> 36:29.380 Most of the breaches come from configuration issues. 36:32.890 --> 36:35.450 Use CIS hardening. 36:35.450 --> 36:37.110 Center for Internet Security. 36:37.110 --> 36:38.750 It is an amazing place. 36:38.750 --> 36:43.070 It is done by volunteers and workers within CIS. 36:43.070 --> 36:48.670 You can get benchmarks for Docker and you can get benchmarks for Kubernetes. 36:48.670 --> 36:54.750 You can use these to configure your environment so they are indeed hardened. 36:55.710 --> 36:57.390 Stop ignoring them. 36:57.390 --> 37:08.190 I still see people that tell me they've never heard of CIS and it makes me very sad when I hear that because it is such a great resource. 37:08.190 --> 37:20.610 The whole point here is we have to harden the environment from the standpoint of the basic configuration, but don't forget the application. 37:20.650 --> 37:29.370 Also, I know a lot of people have environments that are CICD when it comes to Docker and Kubernetes. 37:29.370 --> 37:30.690 That's why we have them. 37:30.690 --> 37:31.710 It's for speed. 37:31.710 --> 37:32.830 It's for development. 37:32.830 --> 37:35.790 It's for putting things out quickly. 37:35.790 --> 37:36.950 Okay, great. 37:36.950 --> 37:38.930 A lot of people use Jenkins. 37:39.690 --> 37:51.310 Is there some way to automate all of this so at least I can look at my images, I can look at my containers, all within the CICD pipeline? 37:51.310 --> 37:52.750 Yes, there is. 37:52.750 --> 37:56.450 There is a great tool called Anchor. 37:56.450 --> 38:01.990 Anchor, as it says here, is a container image scanner plugin. 38:01.990 --> 38:05.590 And what it does is you put it into the pipeline. 38:05.590 --> 38:07.490 It's a plugin for Jenkins. 38:07.490 --> 38:13.510 And it actually does security scans against the images that you're trying to deploy. 38:13.510 --> 38:21.550 This is a great way to get testing in before you actually deploy it to production. 38:21.970 --> 38:26.170 So let's talk about some additional tools before we get into the pen testing. 38:26.170 --> 38:28.410 Because we're still trying to lock it down. 38:28.430 --> 38:29.270 KubeBench. 38:29.270 --> 38:31.250 We want to run KubeBench. 38:31.250 --> 38:35.910 What it does is it runs against your CIS benchmarks. 38:35.910 --> 38:36.790 It displays 38:39.870 --> 38:42.830 a configuration that shows you what has passed, what has failed. 38:42.830 --> 38:44.670 I'll show an example in a minute. 38:44.670 --> 38:46.210 There's also KubeAudit. 38:46.210 --> 38:49.170 It's a little more granular than KubeSec. 38:49.410 --> 38:58.250 And if you run KubeBench and KubeAudit, it will give you a report that shows you oops, my system is insecure. 38:58.250 --> 39:00.770 Here's an example of a report. 39:00.770 --> 39:06.750 And when we look at it, we see allow privilege escalation is not set. 39:06.750 --> 39:09.130 There's a very first one. 39:09.130 --> 39:11.870 Oops, that needs to be turned on. 39:11.870 --> 39:18.150 Remember, that was the one that I mentioned that allows users to do privilege escalation. 39:18.150 --> 39:19.410 We don't want that. 39:19.410 --> 39:20.030 Oh, crap. 39:20.030 --> 39:22.650 We have run as non-root is not set. 39:22.650 --> 39:26.210 We don't have a read only file system set. 39:26.210 --> 39:30.410 There are so many simple things that can be done. 39:30.410 --> 39:33.970 Again, no app armor, no security comp. 39:33.970 --> 39:35.830 It's not being turned on. 39:35.830 --> 39:38.590 These are things that are just so easy. 39:38.590 --> 39:46.850 And this is from every single container within the environment that is being deployed under this particular. 39:46.850 --> 39:48.870 It's much longer. 39:48.910 --> 39:49.830 It goes into it. 39:49.830 --> 39:51.630 This is just a quick snapshot. 39:51.630 --> 39:57.470 The whole point here is you run this against your environment before you deploy it. 39:57.470 --> 40:02.970 And you go, oops, let's go back to our configuration file and fix all these things. 40:03.670 --> 40:07.090 Here are some more tools for you in the environment. 40:07.090 --> 40:13.030 Claire, another great tool for doing static analysis of applications. 40:13.370 --> 40:20.690 Claire does, it integrates Claire and your Docker registry so it can automate that a little bit. 40:20.690 --> 40:28.650 Falco, I need to put huge stars next to this one because this is actually one of my favorite, favorite tools. 40:29.330 --> 40:35.810 It was developed years ago and then it kind of morphed off and became falco.org. 40:35.810 --> 40:39.740 It monitors behavior within containers. 40:39.740 --> 40:47.320 It can detect if a container has been compromised because suddenly the container starts acting weird. 40:47.320 --> 40:54.140 Or there are commands being executed inside the container like cat etsy password. 40:54.240 --> 40:58.980 Probably not a command that should be showing up in one of your containers. 40:59.220 --> 41:07.740 If anything of this entire talk that I'm giving today, you get out of this, go look at falco. 41:07.740 --> 41:10.480 This is one of the best tools out there. 41:10.760 --> 41:13.860 Kubehunter is another great tool. 41:13.860 --> 41:15.280 We're going to talk about that. 41:15.540 --> 41:19.640 It's used more so for looking at doing some pen testing. 41:19.640 --> 41:21.920 We're going to take a look at that in just a second. 41:22.080 --> 41:24.140 Alright, ready, set, attack. 41:24.140 --> 41:34.480 If we were going to pen test an environment, this is an example of all the ports and all of the things that are open in a Kubernetes cluster. 41:34.480 --> 41:39.800 Now, they're open, but are they open by firewalls and so on? 41:39.800 --> 41:41.320 Well, we don't know. 41:42.160 --> 41:47.680 The API server itself is, yes, it's HTTPS. 41:47.680 --> 41:50.140 It better damn well be protected. 41:51.720 --> 41:57.560 etcd on 6666, that should be protected wherever it happens to be. 41:57.560 --> 41:59.140 This just gives you an idea. 41:59.140 --> 42:01.100 All of these ports can be changed. 42:01.100 --> 42:03.820 These are all the defaults in most cases. 42:04.360 --> 42:13.280 Just be aware of the ports that are out there and exposed that you need to make sure are protected. 42:13.280 --> 42:16.300 Because as an attacker, what am I going to do? 42:16.300 --> 42:24.360 Well, one of the first things, even Jason was talking about it in his talk, looking at ports, running nmap, running mass scan, and so on. 42:25.040 --> 42:28.540 This is exactly one of the things that Shodan does. 42:28.540 --> 42:32.400 It looks for all of these things that are exposed on the internet. 42:32.400 --> 42:37.280 So please make sure you're aware of all the ports within your cluster. 42:37.900 --> 42:39.920 Now, I are a hacker. 42:39.920 --> 42:41.180 See me hack. 42:42.340 --> 42:50.320 This is one of the ways that I'm going to start to attack a Kubernetes cluster. 42:50.320 --> 42:57.540 I want to first and foremost check access to the API server, make sure that is protected. 42:57.540 --> 43:01.880 I'm going to be looking at etcd, and making sure that's protected. 43:01.880 --> 43:06.640 If I find it exposed, I'm going to get keys to the kingdom in that case. 43:06.640 --> 43:12.700 What about the kubelets that are communicating and the command channels that are there? 43:12.700 --> 43:17.720 They should be read-only, but they should not be exposed. 43:18.120 --> 43:19.420 There was actually 43:22.440 --> 43:26.300 a vulnerability exposed about this, I want to say two months ago. 43:26.300 --> 43:28.680 I'm probably off in that sense. 43:28.760 --> 43:30.220 But take a look at that. 43:30.220 --> 43:39.980 And it allowed, even though it's a read -only port and it's supposed to limit what you can see, there were some vulnerabilities that allowed for a lot of exposure there. 43:40.020 --> 43:42.740 Also , container vulnerabilities. 43:43.640 --> 43:53.300 What about, you know, do you have files in that container that the application needs that you should be protecting? 43:53.300 --> 43:58.760 Well, that's where I would set up Falco and tell Falco, hey, there's a special file over here. 43:58.760 --> 44:00.900 This application is using it. 44:00.960 --> 44:02.380 But keep an eye on that. 44:02.380 --> 44:12.700 Make sure that the only thing that is reading that application is, or excuse me, that is reading that file happens to be the application. 44:12.700 --> 44:15.160 And Falco can monitor that. 44:15.400 --> 44:17.160 Privilege escalation. 44:17.160 --> 44:27.240 Nine times out of ten, once somebody compromises an application in a container, they elevate privilege because there's some other bugs in there. 44:27.240 --> 44:36.580 Go back to the Kubotic slide where it showed privilege escalation was not set. 44:36.580 --> 44:40.680 All you have to do is set that configuration variable. 44:40.680 --> 44:42.760 It's not that hard. 44:43.040 --> 44:43.760 Okay? 44:43.760 --> 44:46.840 Vulnerable applications are critical. 44:47.360 --> 44:49.640 Can't say it enough times. 44:49.640 --> 44:57.500 Especially because there are so many people using Kubernetes and Docker to deploy WordPress. 44:57.560 --> 45:02.280 And WordPress is, of course, one of our favorite things to deal with. 45:02.420 --> 45:03.180 All right. 45:03.180 --> 45:14.480 So your final threat model, when it comes to pentesting and looking at the environment that is Kubernetes, you think of your workload. 45:14.480 --> 45:18.080 You think of that security within your pods. 45:18.080 --> 45:23.700 Can I set up my pods so they're indeed secure? 45:23.700 --> 45:33.940 The container security itself, the configuration for that container, the more hooks I put in, run as root, no. 45:33.940 --> 45:36.600 Privilege escalation, no. 45:36.600 --> 45:40.160 These are the things that we want to make sure that are set. 45:40.160 --> 45:43.300 Also, file storage security. 45:43.300 --> 45:51.120 Maybe there's a particular file system that has to be read right but the rest of the file system can be read only. 45:51.120 --> 45:52.660 Those are other things. 45:52.660 --> 45:54.260 The network security. 45:54.260 --> 45:56.500 Remember the infrastructure. 45:56.520 --> 45:58.240 Put firewalls in. 45:58.240 --> 46:03.320 Put controls in there that detect lateral movement. 46:03.320 --> 46:10.640 If somebody does compromise one of your applications, make sure you have something monitoring. 46:10.640 --> 46:15.680 I gave you that slide way back about the blog entry from Wazoo. 46:15.680 --> 46:22.340 They talk about how to audit and monitor a Kubernetes cluster using their tool. 46:22.340 --> 46:25.480 I love the Wazoo manager and agent. 46:25.480 --> 46:28.640 Take a look at it if you haven't heard of it before. 46:29.700 --> 46:33.460 Remember, most breaches are not zero day. 46:33.460 --> 46:34.840 I can't say it enough times. 46:34.840 --> 46:38.040 Everybody's always trying to protect against this stuff. 46:38.040 --> 46:39.600 They're not fancy. 46:39.600 --> 46:45.620 Most of these things are being broken into because people make silly mistakes. 46:45.620 --> 46:47.540 Because remember, common sense. 46:47.540 --> 46:49.040 Whatever happened to it. 46:49.040 --> 46:52.780 The breaches are going to come from configuration issues. 46:52.780 --> 47:00.680 The kube audit was a very good example where it was showing all the problems that would allow someone to get in. 47:01.220 --> 47:03.180 Compromise credentials. 47:03.180 --> 47:08.880 Make sure your credentials are stored in a secure place somewhere. 47:09.200 --> 47:13.020 And don't allow for privilege escalation. 47:13.020 --> 47:18.980 Kubernetes and Docker give you the tools and the configurations to prevent that. 47:18.980 --> 47:20.640 Just turn them on. 47:20.640 --> 47:22.700 And that's what we're not doing. 47:22.700 --> 47:26.160 We have to turn on the configurations. 47:26.160 --> 47:29.160 So your key takeaways from today. 47:29.560 --> 47:31.140 Common sense. 47:31.140 --> 47:40.240 Please go back and stop and think about where is the exposure? 47:40.240 --> 47:42.640 Where are my pods exposed? 47:42.640 --> 47:44.660 What about my etcd? 47:45.240 --> 47:46.260 Configuration. 47:46.260 --> 47:49.240 That is so critical to any Kubernetes. 47:49.240 --> 47:52.200 And always remember containers. 47:52.200 --> 47:57.020 Containers are the linchpin of a Kubernetes environment. 47:57.020 --> 48:08.460 And if I don't secure my individual containers then Kubernetes... I don't care about the security of Kubernetes because the container is insecure. 48:08.460 --> 48:14.480 So stop overthinking this stuff and think about security basics. 48:14.480 --> 48:17.420 Remember the CIS benchmarks. 48:17.860 --> 48:19.680 Can't say it enough. 48:19.680 --> 48:20.540 Basics. 48:20.540 --> 48:27.320 I still think as security professionals hackers, whatever we want to call ourselves. 48:27.420 --> 48:32.340 We have forgotten the basics of how to secure an environment. 48:32.340 --> 48:35.780 And that's why things keep getting broken into. 48:35.780 --> 48:38.200 Stop doing fancy crap. 48:38.200 --> 48:40.050 The fancy crap doesn't work. 48:41.460 --> 48:43.940 Secure the environment, please. 48:44.020 --> 48:51.360 Use the CIS benchmarks for the containers, for Kubernetes, for the OS that your stuff runs on. 48:51.360 --> 48:58.760 If you're going to deploy this whole Kubernetes cluster on Ubuntu, then pull up the CIS benchmarks for Ubuntu. 48:58.760 --> 49:01.740 Or Red Hat or CentOS. 49:01.740 --> 49:04.040 Whatever you're using. 49:04.040 --> 49:07.620 Remember, RBAC is your friend. 49:07.620 --> 49:09.580 ABAC is not. 49:10.100 --> 49:12.880 Your infrastructure better be secure. 49:12.880 --> 49:15.760 Your pods better be secure. 49:15.760 --> 49:18.780 Containers themselves, the network. 49:18.780 --> 49:24.160 Still a lot of people forget the network because they forget about TLS everywhere. 49:26.880 --> 49:30.260 Threat model your applications. 49:30.260 --> 49:39.160 If you don't take the beginning effort to threat model your applications, you end up with insecure applications. 49:39.160 --> 49:46.940 And all the security of your Kubernetes cluster is useless because you have insecure applications within your environment. 49:46.940 --> 49:56.460 Access controls go along with RBAC, but access controls into the environment, kind of referencing SSH. 49:56.920 --> 50:03.080 I found one worker node configuration in an environment probably nine months ago. 50:03.080 --> 50:08.700 But for some reason on the worker nodes, they had FTP enabled on all the worker nodes. 50:08.820 --> 50:12.660 And my head exploded. 50:13.180 --> 50:16.540 It allowed me to get into everything, but oh well. 50:17.380 --> 50:19.920 Yes, did I mention RBAC though? 50:20.220 --> 50:21.360 Enough times. 50:21.360 --> 50:22.640 I can't say it enough. 50:22.640 --> 50:23.100 Why? 50:23.100 --> 50:25.180 Because people tend to ignore it. 50:25.180 --> 50:33.780 Remember the tools I mentioned earlier about monitoring or auditing the RBAC roles and also the permissions. 50:34.700 --> 50:39.900 And finally, not quite finally, remember to patch everything. 50:39.900 --> 50:43.820 The infrastructure that goes along with that. 50:43.860 --> 50:46.760 Logging and auditing and monitoring. 50:46.760 --> 50:49.620 I still see people set up their environments. 50:49.620 --> 50:52.700 There's that blog entry again. 50:52.700 --> 50:54.560 Please take a look at that. 50:54.560 --> 50:56.040 It is well written. 50:56.040 --> 50:57.640 It's a great tool. 50:57.640 --> 50:59.060 You'll learn a lot about Wazoo. 50:59.060 --> 51:01.680 I use it on all my systems. 51:01.880 --> 51:06.700 I deploy a lot of honeypots with Kubernetes and containers and things like that. 51:06.700 --> 51:10.360 But I monitor them all with Wazoo. 51:10.960 --> 51:16.540 You can't have a secure environment if you're not monitoring that environment. 51:16.540 --> 51:18.360 That is the biggest thing. 51:19.860 --> 51:22.220 That brings me to the end. 51:22.400 --> 51:25.100 I thank you for your attention. 51:25.880 --> 51:27.600 There's contact info. 51:27.600 --> 51:28.960 There's my Twitter account. 51:28.960 --> 51:35.340 I will get this to the Red Team Village and they can upload it to their environment. 51:35.340 --> 51:41.880 But I'll have it on my GitHub later in an hour or so. 51:41.880 --> 51:43.300 I'll update it there. 51:43.300 --> 51:44.360 Thank you all very much. 51:44.360 --> 51:45.940 Hopefully you learned something today. 51:45.940 --> 51:47.700 Go out and hack and have some fun. 51:47.700 --> 51:51.480 Thanks to the Red Team Village and have an awesome DEF CON. 51:51.480 --> 51:52.940 Awesome. 51:52.940 --> 51:54.740 Thank you so much, Kat. 51:54.740 --> 51:56.760 Amazing presentation as always. 51:56.940 --> 51:58.600 Thank you for supporting DEF CON. 51:58.600 --> 52:00.220 Thank you for supporting the community. 52:00.220 --> 52:01.780 Of course, the Red Team Village. 52:01.780 --> 52:13.660 For those of you that just joined or are looking at the presentation on Twitch or YouTube, please look at all the activities that we have going on. 52:13.660 --> 52:16.340 Go to the website that is in the description of the stream. 52:16.340 --> 52:19.540 Please also join the conversation in Discord. 52:19.540 --> 52:23.760 We're going to go on a brief break and the next presentation will start in a few minutes. 52:23.760 --> 52:25.340 Thank you again and thank you, Kat.