1 00:00:01,370 --> 00:00:05,750 So, welcome to my talk on Kubernetes called Attacking the Helmsman. 2 00:00:05,910 --> 00:00:07,550 So, as Joan just said, I'm Mohit. 3 00:00:07,550 --> 00:00:09,890 I'm a security consultant at F-Secure. 4 00:00:10,430 --> 00:00:17,970 I'm primarily a pen tester, but I also help our clients with, you know, architectural diagrams and stuff from a security point of view. 5 00:00:18,330 --> 00:00:20,470 So, a little bit about what we're going to be talking about today. 6 00:00:20,470 --> 00:00:36,930 So, I'll start on a bit of background of Kubernetes, and then moving to the management plane, you know, the different components within there, how that gets, you know, affected by it being within the cloud, and then a bunch of different security features within Kubernetes and resources. 7 00:00:38,450 --> 00:00:40,550 So, a little bit about Kubernetes. 8 00:00:40,610 --> 00:00:43,730 So, Kubernetes, at least I've been told, is Greek for helmsman. 9 00:00:44,410 --> 00:00:49,390 When I heard about this, I asked one of my Greek colleagues about this, and he wasn't so sure about that. 10 00:00:49,390 --> 00:00:50,870 At least it didn't click in his head. 11 00:00:51,410 --> 00:00:58,390 Later on, he came across Istio, which also happens to follow that same sort of Greek, you know, naming convention. 12 00:00:58,390 --> 00:01:05,430 And at that point, something clicked, and he was like, no, okay, so it may be a thing, however, it's spelled a little bit differently, and he pronounced it Kee-bee-nee-tees. 13 00:01:05,910 --> 00:01:07,970 I hope I haven't butchered that. 14 00:01:08,550 --> 00:01:12,770 For those of you who don't know, Kubernetes is a container orchestration tool. 15 00:01:12,910 --> 00:01:26,010 It essentially helps you manage a large number of containers over potentially a large number of hosts, and it provides a lot of features to aid in that kind of management, and you providing a desired state, and Kubernetes tries to maintain it. 16 00:01:26,010 --> 00:01:29,470 And recently, it's also started moving into multiple clouds. 17 00:01:29,570 --> 00:01:34,390 I remember, I think, a couple of coupons ago, it was talked quite heavily there. 18 00:01:35,370 --> 00:01:42,170 And Kubernetes is, I think, one of the more popular container orchestration tools I've seen. 19 00:01:42,170 --> 00:01:44,610 At least I haven't seen something that's more popular. 20 00:01:44,830 --> 00:01:45,750 So , yeah. 21 00:01:46,350 --> 00:01:48,490 So a little bit about the management plane. 22 00:01:48,490 --> 00:01:55,050 Now, the top box here is the management plane, and it is essentially what helps manage the cluster from an automation perspective. 23 00:01:55,050 --> 00:01:58,090 So at the beginning, you have the API server. 24 00:01:58,090 --> 00:02:12,490 The API server is effectively a stateless REST-based API that is used for users to interact with, and it's kind of like the front end of the management plane, which stores its data within ETCD. 25 00:02:12,990 --> 00:02:15,290 You also have the controller manager and the scheduler. 26 00:02:15,290 --> 00:02:22,450 The scheduler is the resource that kind of wants new pods, which are the smallest item within Kubernetes. 27 00:02:22,530 --> 00:02:25,030 It's created, and it doesn't have a node assigned. 28 00:02:25,030 --> 00:02:41,490 It'll assign it to a node to deploy to, and the controller manager is kind of like the brains in the background, just as things are happening, monitors their states and deploys things or updates things as needed to make sure it's at where we, or where the users desire the state of the cluster to be. 29 00:02:41,770 --> 00:02:45,390 And then outside the management plane, you have the various nodes. 30 00:02:45,390 --> 00:02:50,970 These are the ones that actually do most of the work, running all the containers for the various workloads. 31 00:02:51,310 --> 00:02:56,230 Typically, each node contains a kubelet, which kind of manages the pods on that node. 32 00:02:56,250 --> 00:03:02,710 There's usually some sort of networking with kubeproxy, which does like network-based routing and configuration. 33 00:03:03,070 --> 00:03:06,430 And of course, you're going to have your container runtime and your base OS. 34 00:03:08,910 --> 00:03:11,170 So digging a little bit more into the API server. 35 00:03:11,170 --> 00:03:17,690 So as I said earlier, this is kind of like the front end of the cluster for people to communicate with. 36 00:03:17,690 --> 00:03:22,050 And it performs some additional features on top of just storing that data. 37 00:03:22,050 --> 00:03:25,130 It says, you know, applying any RBAC policies that may apply. 38 00:03:25,130 --> 00:03:29,990 So, you know, if a user wishes to do something, it would first validate that they are allowed to actually do that. 39 00:03:30,230 --> 00:03:34,990 And also does additional data validation to make sure what it's been given does make sense. 40 00:03:34,990 --> 00:03:39,110 So it's not being asked to deploy, you know, a pod with incorrect specifications or anything. 41 00:03:40,370 --> 00:03:48,290 And once it has this data and is happy with it, it gets stored in the ECD, which is effectively the golden grail of information within the cluster, because it stores pretty much everything. 42 00:03:48,570 --> 00:03:55,930 All the resources that are deployed in the cluster are defined in YAML, which are stored in ETCD. 43 00:03:56,110 --> 00:04:01,730 And therefore, if you have write access or read access to this, you have pretty elevated access to the data within the cluster. 44 00:04:01,770 --> 00:04:08,150 With write access, you know, you could write yourself new roles or policies that would let you have cluster admin access. 45 00:04:08,150 --> 00:04:11,810 And if you have read access, you can just read the secrets for like... 46 00:04:11,810 --> 00:04:14,790 Sorry, not read the secrets, just read various data. 47 00:04:14,850 --> 00:04:18,430 Secrets are a little bit more interesting as they tend to be encrypted in ETCD. 48 00:04:20,050 --> 00:04:24,390 And the way ETCD is built, there's not really an application-level authentication. 49 00:04:24,390 --> 00:04:34,160 So you kind of focus more on whitelisting the source addresses that are permitted to talk to it. 50 00:04:34,160 --> 00:04:40,500 Typically, it's recommended to only whitelist this to the Kube API server, as that's really the thing that needs to talk to it. 51 00:04:42,360 --> 00:04:47,840 As I said earlier, the Kubelet is the component that runs on each node. 52 00:04:47,840 --> 00:05:01,440 And this tends to register itself with the API server to say, you know, I'm a node here, please give me work and have permissions within the cluster so I can access the data I need. 53 00:05:01,440 --> 00:05:09,000 So for the port specifications, it needs to deploy pods and any other data that may need to be deployed alongside it, for example, the secrets for those pods. 54 00:05:09,000 --> 00:05:12,840 And to have this access, it has a server's account to the API server. 55 00:05:12,840 --> 00:05:19,120 And in itself, it exposes an API, typically on port 10.2.50. 56 00:05:19,220 --> 00:05:21,900 And there have been attacks against this in the past as well. 57 00:05:21,900 --> 00:05:33,620 So one of my colleagues, Alexander Keskis, wrote a blog post on this a while back, where he was leveraging anonymous authentication on the Kubelet API then to talk to the API server and get into various pods. 58 00:05:35,700 --> 00:05:46,920 Now, the management plane within a cloud-managed Kubernetes cluster such as EKS or GKE the management plane is simply managed by the cloud provider. 59 00:05:46,980 --> 00:05:50,980 As such, when we're doing pentest engagements, these are typically considered out of scope. 60 00:05:52,140 --> 00:05:55,820 And we kind of trust the cloud provider to manage that appropriately. 61 00:05:55,860 --> 00:05:59,720 However, being in the cloud also adds cloud-specific interactions. 62 00:06:00,060 --> 00:06:22,940 For example, instance metadata raises its ugly head again, where the VMs, if deployed on VMs within the cloud, it would have access to the instance metadata and theoretically pods could communicate with that to get AWS credentials to talk to the rest of the environment if the nodes have it or anything else. 63 00:06:23,180 --> 00:06:27,840 Now, there are various protections for this that are possible, but they're not always in play. 64 00:06:28,580 --> 00:06:33,840 And the fact that it is also in the cloud environment, typically IAM is also integrated with access to the cluster. 65 00:06:34,080 --> 00:06:45,180 So when authenticating to a Kubernetes cluster, which is managed, typically you're using your cloud provider's IAM or CLI to kind of get access and generate your kube-config files so you can authenticate. 66 00:06:48,970 --> 00:06:57,250 So a while back, we were doing an assessment where a couple of our guys noticed that they could talk to the instance metadata within a cluster within GKE. 67 00:06:57,630 --> 00:07:03,730 And within the instance metadata, they found tokens for talking to the kube-api server. 68 00:07:04,030 --> 00:07:09,350 They enumerated the permissions for this and noticed that they were allowed to create CSRs. 69 00:07:09,350 --> 00:07:14,030 And going back to the fact that kubelet self-registered, these CSRs did that. 70 00:07:14,170 --> 00:07:23,110 You could essentially create a CSR to send it to the API server, which when it gets signed is your authentication token as a kubelet to the API server. 71 00:07:23,110 --> 00:07:31,630 So they could use this to start enumerating the different nodes and get the pods, the secrets within and get to cluster admin in that manner. 72 00:07:31,770 --> 00:07:38,930 So effectively from a pod that they can talk to the instance metadata, they escalated themselves quite quickly to cluster admin. 73 00:07:39,350 --> 00:07:45,290 Now back then they just did this manually, but recently I got told of a script that kind of did it. 74 00:07:45,770 --> 00:07:48,510 I can't remember the person's name. 75 00:07:48,570 --> 00:07:49,910 Just a second. 76 00:07:52,250 --> 00:08:03,250 A script by Brad Giesemann that's a quick bash script that kind of does this for you and enumerates that data and, you know, gets you the secrets and stuff within the cluster. 77 00:08:04,770 --> 00:08:07,470 Now Kubernetes is very much API based. 78 00:08:07,470 --> 00:08:10,050 So the API server is pretty much a whole bunch of APIs. 79 00:08:10,110 --> 00:08:13,170 And by design, Kubernetes is modular. 80 00:08:13,230 --> 00:08:26,510 And we're going to see this throughout the presentation where the way that it's set up allows for the easy adding of new resources or configurations so it can work better in people's individuals, you know, configurations. 81 00:08:27,670 --> 00:08:35,870 So the ability to add additional APIs is present and it's also useful for identifying technology sites when treating it in an offensive perspective. 82 00:08:36,110 --> 00:08:44,230 As when new technologies, you know, tend to use creating custom APIs, you can enumerate these APIs and figure out what kind of technologies are in use. 83 00:08:44,450 --> 00:08:50,970 So this is the example output of this for a pretty bog standard cluster with some additional things. 84 00:08:50,970 --> 00:08:57,370 And from this we can quickly enumerate that, you know, it's using Calico for networking, it's using LinkedIn as a service mesh and so on. 85 00:08:57,370 --> 00:09:00,170 So it's a quite useful way to enumerate this information. 86 00:09:03,180 --> 00:09:07,060 So Kubernetes uses RBAC for permission control. 87 00:09:07,380 --> 00:09:09,160 It's generally found to use RBAC. 88 00:09:09,160 --> 00:09:11,540 There are other ways it can be done, such as ABAC. 89 00:09:11,800 --> 00:09:14,860 But most of the time it's RBAC. 90 00:09:15,000 --> 00:09:19,040 And within RBAC there are three main high-level concepts within Kubernetes. 91 00:09:19,180 --> 00:09:21,160 So the first one is the targets. 92 00:09:21,280 --> 00:09:23,080 So these are what the permissions apply to. 93 00:09:23,080 --> 00:09:25,360 So your users, your groups, your service accounts. 94 00:09:25,580 --> 00:09:27,360 And then these get assigned roles. 95 00:09:27,360 --> 00:09:32,480 And the roles grant you those permissions in, you know, the various areas of the cluster. 96 00:09:32,800 --> 00:09:36,940 This can be limited by the API groups, the resource, the verb. 97 00:09:36,940 --> 00:09:40,320 So, you know, whether you can create it or just get it or whatever. 98 00:09:40,320 --> 00:09:42,520 And potentially also the resource name. 99 00:09:42,820 --> 00:09:49,320 And now these are bound to those targets using role bindings, which basically are map roles onto your targets. 100 00:09:50,300 --> 00:09:56,860 Now these RBAC permissions can either apply cluster-wide or within a single namespace. 101 00:09:56,860 --> 00:10:02,920 And the way that kind of gets differentiated is with, you know, the role slash cluster role or role binding slash cluster role binding. 102 00:10:02,920 --> 00:10:07,800 So if it were a cluster role and a cluster role binding, the permissions applied would be cluster-wide. 103 00:10:07,840 --> 00:10:12,940 And all resources that, you know, are allowed by that role would be allowed to that target. 104 00:10:12,940 --> 00:10:17,980 However, if it's, you know, using the role binding, then it's only within a specific namespace. 105 00:10:17,980 --> 00:10:19,660 Those permissions are granted. 106 00:10:21,020 --> 00:10:27,960 Now, of those three, you know, kind of targets I kind of mentioned, the only one really managed by Kubernetes is the service account. 107 00:10:28,040 --> 00:10:30,600 And these are authenticated using Bero tokens. 108 00:10:30,600 --> 00:10:38,480 And they are, you know, fully managed within Kubernetes, typically, with service accounts being created as a resource. 109 00:10:38,480 --> 00:10:41,900 The authentication credentials are mounted to pods. 110 00:10:41,900 --> 00:10:45,980 So they have access to those service accounts, which they can use to talk to the cluster. 111 00:10:45,980 --> 00:10:48,820 Users and groups are not managed by Kubernetes. 112 00:10:48,820 --> 00:11:00,860 And it's typically left to some sort of external service with authentication modules within Kubernetes, providing that kind of access to those authentication mechanisms so it can kind of, you know, authenticate users. 113 00:11:01,240 --> 00:11:06,580 And it kind of trusts those third-party systems to be correct and just follows it. 114 00:11:06,580 --> 00:11:16,980 So, for example, using X.509 certificates, the common name provides the user and the organization, you know, within a certificate, tells Kubernetes what groups it's in. 115 00:11:16,980 --> 00:11:22,420 And it just validates that it's been signed by the appropriate certificate authority and says, yep, that's cool with me. 116 00:11:22,420 --> 00:11:33,300 Or, for example, if you're running a Windows-based environment, it can also use the Active Directory as a kind of source of truth for what users are allowed or not. 117 00:11:35,420 --> 00:11:47,240 So service accounts being, you know, in Kubernetes, when you create a new namespace, which is the way people tend to do segregation, a new default service account is created within that namespace. 118 00:11:47,240 --> 00:11:52,900 And this default service account is automatically mounted into every pod within that namespace by default. 119 00:11:53,500 --> 00:11:57,520 Now, this service account does not have any permissions directly applied to it. 120 00:11:57,520 --> 00:12:01,760 However, if it's in any groups, then it would have those permissions as well. 121 00:12:01,760 --> 00:12:05,960 But then this is typically something you should avoid. 122 00:12:06,080 --> 00:12:17,100 Because if you do assign any permissions to this default service account for one pod, and another pod is also using it, then that other pod would also have those permissions, you know, kind of breaking the principles of least privilege. 123 00:12:18,200 --> 00:12:26,180 So it's typically recommended to disable the auto-mounting of these default service accounts and creating bespoke service accounts for each time you'd need it. 124 00:12:27,700 --> 00:12:35,120 Now, the groups that, you know, the default service account may be in, there are three groups it's in by default. 125 00:12:35,440 --> 00:12:49,200 So that's system authenticated, which is for pretty much every authenticated entity within the cluster, system service accounts, which is the group for all service accounts, and then system service accounts and the name of the namespace is for all service accounts within that particular namespace. 126 00:12:49,260 --> 00:12:54,360 So if any of those three groups have any permissions assigned to them, that default service account will also have it. 127 00:12:54,360 --> 00:12:56,640 There's also a system unauthenticated group. 128 00:12:56,640 --> 00:13:09,520 So if, you know, the cluster allows anonymous authentication, the user, well, not really a user, but the person who has communicating with the cluster will have the same permissions, as would be granted by system unauthenticated. 129 00:13:10,700 --> 00:13:20,500 So there was a time not too long ago where we were on assessment, and the groups for system service accounts are actually over-provisioned, granting cluster admin access. 130 00:13:20,500 --> 00:13:29,440 And therefore, you know, as soon as we compromise a single pod in the cluster, we were able to use that service account, which was mounted into every pod by default. 131 00:13:29,780 --> 00:13:33,880 And we very quickly got cluster admin into the cluster in that manner. 132 00:13:36,380 --> 00:13:41,620 Now, when enumerating permissions, there are a bunch of tools I tend to use when enumerating permissions. 133 00:13:41,660 --> 00:13:46,360 I think the one I use the most is a tool called Rakesh, get hub link on the slides. 134 00:13:46,360 --> 00:14:02,180 And essentially, this takes, you know, your token that you used for authentication or whatever, and it kind of enumerates all the permissions that are available and then gives you a nice tick or a cross in a tabular format showing, you know, what you are and aren't allowed to do, 135 00:14:02,180 --> 00:14:09,320 which, you know, if you've just compromised a single JWT token to authenticate for a service account, it's great to kind of figure out what you can and cannot do. 136 00:14:10,540 --> 00:14:22,800 But then, you know, when doing more of a review of RBAC, there's a tool called RBAC Lookup, which kind of lists through all the subjects, all the targets, the roles and role bindings that are present in the cluster. 137 00:14:22,800 --> 00:14:35,700 So, you know, if we've identified a role that is, you know, considered risky in a way, so for example, it grants you excessive permissions, we can quickly identify what targets have that role assigned to them. 138 00:14:36,920 --> 00:14:55,300 The final tool I'd like to quickly mention is QBScan, which kind of takes the approach a bit more automated, where it automatically tries to find those risky accounts, whether these are just our cluster admin by default or they have permissions that could easily help them get to a cluster admin level of access. 139 00:14:57,620 --> 00:15:03,540 So moving on to networking, there are a variety of things you need to consider with networking. 140 00:15:03,720 --> 00:15:11,880 Now , Kubernetes, as I said earlier, being very modular, the actual networking interface you use is actually kind of up to you. 141 00:15:11,880 --> 00:15:17,080 Kubernetes say that, you know, it should follow the CNI container networking interface kind of specification. 142 00:15:17,220 --> 00:15:19,000 But apart from that, we're Gucci. 143 00:15:19,740 --> 00:15:28,080 So within that, you know, there are a bunch of things we need to kind of look into, like pod-to-service, ingress, pod-to-pod, and the CNI itself. 144 00:15:28,200 --> 00:15:38,480 Now, because CNI is pretty much freedom, there are a lot of CNIs out there, and each of them provide different functionality and features. 145 00:15:39,220 --> 00:15:46,640 So there are, you know, the cloud network-based CNIs, which kind of integrate with your cloud providers. 146 00:15:46,640 --> 00:15:55,100 Networking interfaces like AWS VPC would, you know, kind of put you within the VPCs as opposed to a weird overlay network within the cluster itself. 147 00:15:55,400 --> 00:16:00,820 And then there are others that, you know, kind of provide in-transit encryption or other features. 148 00:16:01,380 --> 00:16:03,020 And there's a whole bunch. 149 00:16:04,580 --> 00:16:09,240 For traffic to get into the cluster, typically, you would find an ingress controller. 150 00:16:09,240 --> 00:16:17,540 Some alternative methods are like a node port, which is just, you know, a high-numbered port, usually above 30,000, where a service is exposed. 151 00:16:17,640 --> 00:16:31,640 But more often than not, you're going to find some sort of ingress controller, where it's typically a pod, such as, you know, Trafic or Nginx or something, that has permissions within the cluster to list the services, which are the way you expose pods in a cluster, 152 00:16:31,640 --> 00:16:41,060 and then gets traffic coming in and then tends to route them around internally in the cluster to the correct location or the correct service. 153 00:16:43,160 --> 00:16:45,880 DNS for resolving within the cluster. 154 00:16:46,480 --> 00:16:55,420 Now, if it's just a pod, it's automatically got a DNS name with the pod's IP address, the name of the namespace, .pod.cluster.local. 155 00:16:55,460 --> 00:17:00,780 However, it's usually not common for people to use the pod's DNS name for resolution. 156 00:17:00,780 --> 00:17:16,780 It's more common not to use a headless service on top of that, where after you've created a pod with the labels appropriate to it, you create a headless service pointing to those pods, and it kind of simplifies that entire process, as it can also do a load balancing between the various pods. 157 00:17:17,580 --> 00:17:22,180 And typically, this is called, you know, the service name.namespace.service.cluster.local. 158 00:17:22,180 --> 00:17:35,600 And because the search space for the DNS queries include namespace.service.cluster.local and service.cluster.local, you can just do a query for service if you're in the same namespace, or if you're in a separate namespace, service.namespace. 159 00:17:36,140 --> 00:17:40,520 There's an additional DNS name that's usually also added, and that's hostname.subdomain. 160 00:17:40,520 --> 00:17:44,000 And the hostname and subdomain is kind of extracted from the pod. 161 00:17:44,000 --> 00:17:54,240 So the pod can specify the hostname and subdomain at which point it will be available using that kind of DNS name, which kind of brings the idea of spoofing into play. 162 00:17:54,260 --> 00:18:08,500 So, for example, if I were to set the subdomain to, say, com and the hostname to google, suddenly I have a DNS queryable name, google.com, which would complete and resolve to an internal IP address. 163 00:18:08,960 --> 00:18:21,320 So if you could, you know, somehow get yourself a google.com certificate, which I doubt you're going to get an external CA to do it, but if you're using some sort of internal CA, it could let you do some interesting attacks in that manner, or manhandling other connections. 164 00:18:23,260 --> 00:18:36,880 Now, the Intel network is typically flat by default, because Kubernetes has this thing of, you know, things should be able to talk to each other to facilitate, you know, microservices-based architecture, which is great from, you know, a developer's point of view, 165 00:18:36,880 --> 00:18:43,020 where they can deploy their, you know, series of pods and have them communicate in a way they need to, and it works. 166 00:18:43,060 --> 00:18:56,080 However, from a security point of view, it adds a lot of issues, because the pods in the cluster aren't only going to be the application pods, they're also going to be, you know, a variety of management pods, etc., which if you can talk to, you can potentially do some dangerous things. 167 00:18:57,000 --> 00:19:02,900 So, to kind of support locking down what kind of access pods have in the network, there are network policies. 168 00:19:03,540 --> 00:19:11,000 Now, network policies, effectively, are kind of like firewall rules, where you can say, you know, these pods are allowed to talk to these pods or these CIDR ranges and nothing else, 169 00:19:14,440 --> 00:19:22,240 and they're only allowed to, you know, get incoming traffic from these locations that are, you know, permitted by the administrators. 170 00:19:23,460 --> 00:19:24,860 So, it's kind of great. 171 00:19:24,880 --> 00:19:29,440 Now, by default, there is no, like, deny rule or anything within network policies. 172 00:19:29,440 --> 00:19:33,920 So, all the network policies are kind of like allow this connection. 173 00:19:34,240 --> 00:19:40,060 So, the way it tends to work is if there are no network policies within the namespace, everything's a flat network by default. 174 00:19:40,060 --> 00:19:48,180 But as soon as you start adding the first network policy, it starts becoming, you need to have a network policy allow that connection before it's permitted. 175 00:19:48,180 --> 00:19:54,400 So, your default deny, instead of being a rule that says deny all connections, is a network policy with zero rules in it. 176 00:19:54,440 --> 00:19:57,080 And then you can kind of add the rules in as you need it. 177 00:19:57,880 --> 00:20:05,640 Now, the Intel network, depending on the CNI you use, but most of them don't allow encryption in transit, or they don't support it. 178 00:20:06,000 --> 00:20:08,800 So, there are this concept of service meshes. 179 00:20:09,480 --> 00:20:24,440 Now, service meshes, they typically provide encryption in transit, but a whole bunch of other stuff like monitoring and other metrics that, you know, your administrators can use to kind of help debug issues in the cluster or just have an oversight of what's actually going on. 180 00:20:24,620 --> 00:20:29,160 And the way this is typically done is through a sidecar container of a pod. 181 00:20:29,160 --> 00:20:31,540 So, a pod is one or more containers. 182 00:20:31,580 --> 00:20:34,220 So, this just would be another container in that pod. 183 00:20:35,340 --> 00:20:39,300 And typically, these are automatically injected into the pod at runtime. 184 00:20:39,300 --> 00:20:46,020 So, when that pod gets created, admission controllers are kind of used to inject those pods. 185 00:20:46,540 --> 00:20:59,500 Now, admission controllers are functionalities within the API server that basically allow, you know, modular-based additional features or checks to be added to that validation process or request as they come in. 186 00:20:59,540 --> 00:21:13,320 So, they can, you know, be used to write custom policies, for example, using OPA, which is a policy as code tool, to provide custom checks that the API server can then use to validate the data that's coming in. 187 00:21:13,980 --> 00:21:16,140 Now, there are two types of admission controls. 188 00:21:16,140 --> 00:21:18,080 There's mutating and validating. 189 00:21:18,440 --> 00:21:22,600 Validating just validates the data and says, you know, this looks all right. 190 00:21:22,600 --> 00:21:23,960 Yeah, I'm happy with this. 191 00:21:24,200 --> 00:21:31,100 Mutating takes it a little bit further, where if it's not happy, it has the option to change the request as it comes in. 192 00:21:31,100 --> 00:21:33,500 So, for example, you have a pod coming in. 193 00:21:33,500 --> 00:21:35,760 It feels like changing something. 194 00:21:35,760 --> 00:21:38,260 It's well within its rights to do so. 195 00:21:39,360 --> 00:21:54,700 So, the overall flow looks something like this, where when a request comes into the API server, first, the API server would, you know, make sure you're allowed to do what you're trying to do in the first place, to roll back and make sure, you know, you are a person or service account. 196 00:21:54,920 --> 00:22:01,100 It would then pass it to the mutating admission controllers to see whether there's any changes they would like to make and if they are happy with it. 197 00:22:01,540 --> 00:22:05,940 Once all the changes are made, make sure it still meets, you know, the object schema validation. 198 00:22:06,160 --> 00:22:08,780 So, make sure it's still a valid resource. 199 00:22:09,000 --> 00:22:13,960 And then finally, passes it to the validating admission controls to make sure there are... 200 00:22:13,960 --> 00:22:27,820 or run all the final checks and make sure it's still a valid piece of thing to save before saving it into etcd, at which point the controller managers and schedulers can start, you know, using that data to make the changes within the cluster as needed. 201 00:22:29,620 --> 00:22:38,720 Now, it is possible to create your own admission controls, but there are a whole bunch of inbuilt admission controllers that are there by default, which can be enabled at the API server. 202 00:22:38,960 --> 00:22:43,640 Now, there are a whole bunch of security admission controls as well, but most of these tend not to be enabled by default. 203 00:22:43,640 --> 00:22:48,780 So, it's usually manually need to be enabled in the API server if it's a self-managed cluster. 204 00:22:49,300 --> 00:22:53,440 So, for example, pod security policy, which we're going to be talking about later, or always pull images. 205 00:22:54,460 --> 00:23:06,440 Always pull images is essentially a mutating controller where as pod specifications come through, it makes sure that the image policy is set to always as opposed to the default of if not present. 206 00:23:06,940 --> 00:23:22,520 This is especially important in like multi-tenant clusters where if it's set to if not present and you have two separate tenants where one has permissions over an image and the other one doesn't, the one that doesn't have permission over that image could use that image without having the relevant permissions. 207 00:23:22,520 --> 00:23:25,020 If that image has been pre-pulled onto the node. 208 00:23:25,200 --> 00:23:36,880 However, if the image pull policy is set to always, it would only allow the deployment of that pod if that tenant could pull that image in the first place, which is kind of a way to prevent them using images they're not permitted to. 209 00:23:37,700 --> 00:23:40,240 And there's a whole bunch of admission controls like this. 210 00:23:40,320 --> 00:23:48,720 In fact, having mutating and validating admission controllers, custom ones, require the inbuilt admission controls for the same thing being enabled. 211 00:23:48,720 --> 00:24:01,140 So there was an assessment we did a while back where we actually ran into OPA as a mutating admission controller and within the network, it was the flat network. 212 00:24:01,140 --> 00:24:04,320 So we could talk to the web front end for it. 213 00:24:04,320 --> 00:24:09,920 And we found it in the cluster because the pod we had compromised had over-provisioned groups. 214 00:24:09,940 --> 00:24:15,120 So we could start reading resources within the cluster because they had read-only access. 215 00:24:15,120 --> 00:24:20,820 We could have also guessed the service name because it was just OPA.OPA, but that's boring. 216 00:24:21,760 --> 00:24:28,780 Now, because it was a mutating admission controller, if we could get policies into that, we could change the specification of pods in the cluster. 217 00:24:28,920 --> 00:24:32,400 And luckily for us, there was no authentication for OPA configured. 218 00:24:33,140 --> 00:24:43,780 So a quick policy pushed up to OPA and waiting for pods meant that we could suddenly start backdooring pods in other areas of the cluster and start moving around the cluster in that manner. 219 00:24:46,000 --> 00:24:51,980 Now, when talking about Kubernetes security, one thing that you kind of can't forget is the fact that they are using containers under the hood. 220 00:24:52,460 --> 00:24:55,480 And therefore, things like container breakouts do apply here. 221 00:24:55,480 --> 00:25:19,240 Now, I won't be going into too much detail about, you know, all the ways that are possible for container breakouts, but a couple of common ones are like, you know, leveraging host volume mounts, where, you know, folders from the host file system have been mounted into the container and you can kind of use those to add like a SSH key if you have someone's home folder or if file run is mounted talking to the Docker socket if you're using Docker or all sorts of things. 222 00:25:19,680 --> 00:25:27,300 There's also, you know, kernel exploits like DirtyCow if, you know, it's an out-of-date kernel version or also overly permissive kernel capability. 223 00:25:27,300 --> 00:25:31,480 So if, you know, the privilege flag has been used, you have the ability to load kernel modules. 224 00:25:31,480 --> 00:25:34,380 So you can kind of load a kernel module, get onto the host that way. 225 00:25:34,380 --> 00:25:48,980 And once you have that container breakout set up, because typically one node deploys model pods on that same node, as soon as you've broken out one pod, you suddenly have access to other pods shared on that node. 226 00:25:49,260 --> 00:25:56,060 Now to help mitigate container breakouts, there's a pod security policy once you have the admission controller enabled for such thing. 227 00:25:56,680 --> 00:25:57,520 Excuse me. 228 00:25:57,520 --> 00:26:08,620 And that kind of helps mitigate a lot of container breakout attacks by limiting what you can actually do with, you know, volumes, your user IDs that you're allowed to run with, the kernel capabilities and so on and so forth. 229 00:26:09,380 --> 00:26:13,680 Now, the way these are used is you would create a PSP or pod security policy. 230 00:26:13,860 --> 00:26:16,340 And then through RBAC, you would assign that to your users. 231 00:26:16,340 --> 00:26:24,240 And those users, when deploying the pod, would have to follow it, you know, one of those PSPs if they... yeah. 232 00:26:25,040 --> 00:26:37,460 Now, one of the things I typically recommend with this is, you know, using the system authenticated group where you create the most restrictive PSP, you know, you can, and then assign that to system authenticator at that point, you know, that should apply cluster-wide. 233 00:26:37,520 --> 00:26:45,120 And then where more permissive pod security policies are required, those can be, you know, assigned when and where they're needed. 234 00:26:48,280 --> 00:27:03,620 On the same level, if you can't, you know, if you can't prevent container breakout for whatever reason, and an attacker somehow does still manages it, you can at least start controlling where pods are within a cluster to kind of mitigate the impact of that compromise. 235 00:27:03,620 --> 00:27:09,900 So let's say in the cluster, there are pods that have, you know, just a standard basic brochureware applications. 236 00:27:09,920 --> 00:27:16,060 There are ones that handle massive business critical data, and all sorts of new management pods. 237 00:27:16,400 --> 00:27:20,980 You can start categorizing nodes to, you know, whole different types of pods. 238 00:27:20,980 --> 00:27:34,000 So you can have a whole bunch of nodes allocated to the brochureware applications, all the stuff you don't really care about, a whole bunch of nodes allocated to, you know, the business sensitive pods that handle very critical data that you really don't want compromised, 239 00:27:34,000 --> 00:27:37,560 another set of nodes from the management plane, and so on and so forth. 240 00:27:37,600 --> 00:27:48,940 And these can kind of be enforced through tents and tolerations, where you can apply these to the nodes and pods, and then a pod is only limited where it can be deployed to based on these restrictions. 241 00:27:49,500 --> 00:27:56,760 And then these can also be applied to the namespace layer, where you can say that all pods within this namespace need to follow these tolerations. 242 00:27:56,860 --> 00:28:05,040 And that way you can kind of, from a cluster administrative perspective, manage what the impact could be of a container breakout should it were to happen.