So, welcome to my talk on Kubernetes called Attacking the Helmsman.
So, as Joan just said, I'm Mohit. I'm a security consultant at F-Secure.
I'm primarily a pen tester, but I also help our clients with, you know,
architectural diagrams and stuff from a security point of view.
So, a little bit about what we're going to be talking about today.
So, I'll start on a bit of background of Kubernetes,
and then moving to the management plane, you know, the different components
within there, how that gets, you know, affected by it being
within the cloud, and then a bunch of different security features
within Kubernetes and resources.
So, a little bit about Kubernetes.
So, Kubernetes, at least I've been told, is Greek for helmsman.
When I heard about this, I asked one of my Greek colleagues about this,
and he wasn't so sure about that. At least it didn't click in his head.
Later on, he came across Istio, which also happens to follow
that same sort of Greek, you know, naming convention.
And at that point, something clicked, and he was like, no, okay,
so it may be a thing, however, it's spelled a little bit differently,
and he pronounced it Kee-bee-nee-tees.
I hope I haven't butchered that.
For those of you who don't know, Kubernetes is a container
orchestration tool. It essentially helps you manage a large number
of containers over potentially a large number of hosts,
and it provides a lot of features to aid in that kind of management,
and you providing a desired state, and Kubernetes tries to maintain it.
And recently, it's also started moving into multiple clouds.
I remember, I think, a couple of coupons ago,
it was talked quite heavily there.
And Kubernetes is, I think, one of the more popular
container orchestration tools I've seen.
At least I haven't seen something that's more popular.
So, yeah.
So a little bit about the management plane.
Now, the top box here is the management plane,
and it is essentially what helps manage the cluster
from an automation perspective.
So at the beginning, you have the API server.
The API server is effectively a stateless REST-based API
that is used for users to interact with,
and it's kind of like the front end of the management plane,
which stores its data within ETCD.
You also have the controller manager and the scheduler.
The scheduler is the resource that kind of wants new pods,
which are the smallest item within Kubernetes.
It's created, and it doesn't have a node assigned.
It'll assign it to a node to deploy to,
and the controller manager is kind of like the brains
in the background, just as things are happening,
monitors their states and deploys things or updates things
as needed to make sure it's at where we,
or where the users desire the state of the cluster to be.
And then outside the management plane,
you have the various nodes.
These are the ones that actually do most of the work,
running all the containers for the various workloads.
Typically, each node contains a kubelet,
which kind of manages the pods on that node.
There's usually some sort of networking with kubeproxy,
which does like network-based routing and configuration.
And of course, you're going to have your container runtime
and your base OS.
So digging a little bit more into the API server.
So as I said earlier, this is kind of like the front end
of the cluster for people to communicate with.
And it performs some additional features on top of just storing that data.
It says, you know, applying any RBAC policies that may apply.
So, you know, if a user wishes to do something,
it would first validate that they are allowed to actually do that.
And also does additional data validation to make sure what it's been given
does make sense.
So it's not being asked to deploy, you know, a pod with incorrect specifications
or anything.
And once it has this data and is happy with it,
it gets stored in the ECD, which is effectively the golden grail
of information within the cluster, because it stores pretty much everything.
All the resources that are deployed in the cluster
are defined in YAML, which are stored in ETCD.
And therefore, if you have write access or read access to this,
you have pretty elevated access to the data within the cluster.
With write access, you know, you could write yourself new roles
or policies that would let you have cluster admin access.
And if you have read access, you can just read the secrets for like...
Sorry, not read the secrets, just read various data.
Secrets are a little bit more interesting as they tend to be encrypted in ETCD.
And the way ETCD is built, there's not really an application-level authentication.
So you kind of focus more on whitelisting the source addresses
that are permitted to talk to it.
Typically, it's recommended to only whitelist this to the Kube API server,
as that's really the thing that needs to talk to it.
As I said earlier, the Kubelet is the component that runs on each node.
And this tends to register itself with the API server to say, you know,
I'm a node here, please give me work and have permissions within the cluster
so I can access the data I need.
So for the port specifications, it needs to deploy pods
and any other data that may need to be deployed alongside it,
for example, the secrets for those pods.
And to have this access, it has a server's account to the API server.
And in itself, it exposes an API, typically on port 10.2.50.
And there have been attacks against this in the past as well.
So one of my colleagues, Alexander Keskis, wrote a blog post on this a while back,
where he was leveraging anonymous authentication on the Kubelet API
then to talk to the API server and get into various pods.
Now, the management plane within a cloud-managed Kubernetes cluster such as EKS or GKE
the management plane is simply managed by the cloud provider.
As such, when we're doing pentest engagements, these are typically considered out of scope.
And we kind of trust the cloud provider to manage that appropriately.
However, being in the cloud also adds cloud-specific interactions.
For example, instance metadata raises its ugly head again, where the VMs,
if deployed on VMs within the cloud, it would have access to the instance metadata
and theoretically pods could communicate with that to get AWS credentials
to talk to the rest of the environment if the nodes have it or anything else.
Now, there are various protections for this that are possible, but they're not always in play.
And the fact that it is also in the cloud environment,
typically IAM is also integrated with access to the cluster.
So when authenticating to a Kubernetes cluster, which is managed,
typically you're using your cloud provider's IAM or CLI to kind of get access
and generate your kube-config files so you can authenticate.
So a while back, we were doing an assessment where a couple of our guys noticed
that they could talk to the instance metadata within a cluster within GKE.
And within the instance metadata, they found tokens for talking to the kube-api server.
They enumerated the permissions for this and noticed that they were allowed to create CSRs.
And going back to the fact that kubelet self-registered, these CSRs did that.
You could essentially create a CSR to send it to the API server,
which when it gets signed is your authentication token as a kubelet to the API server.
So they could use this to start enumerating the different nodes and get the pods,
the secrets within and get to cluster admin in that manner.
So effectively from a pod that they can talk to the instance metadata,
they escalated themselves quite quickly to cluster admin.
Now back then they just did this manually, but recently I got told of a script that kind of did it.
I can't remember the person's name. Just a second.
A script by Brad Giesemann that's a quick bash script that kind of does this for you
and enumerates that data and, you know, gets you the secrets and stuff within the cluster.
Now Kubernetes is very much API based. So the API server is pretty much a whole bunch of APIs.
And by design, Kubernetes is modular.
And we're going to see this throughout the presentation where the way that it's set up
allows for the easy adding of new resources or configurations
so it can work better in people's individuals, you know, configurations.
So the ability to add additional APIs is present and it's also useful for identifying technology sites
when treating it in an offensive perspective.
As when new technologies, you know, tend to use creating custom APIs,
you can enumerate these APIs and figure out what kind of technologies are in use.
So this is the example output of this for a pretty bog standard cluster with some additional things.
And from this we can quickly enumerate that, you know, it's using Calico for networking,
it's using LinkedIn as a service mesh and so on.
So it's a quite useful way to enumerate this information.
So Kubernetes uses RBAC for permission control.
It's generally found to use RBAC. There are other ways it can be done, such as ABAC.
But most of the time it's RBAC.
And within RBAC there are three main high-level concepts within Kubernetes.
So the first one is the targets.
So these are what the permissions apply to.
So your users, your groups, your service accounts.
And then these get assigned roles.
And the roles grant you those permissions in, you know, the various areas of the cluster.
This can be limited by the API groups, the resource, the verb.
So, you know, whether you can create it or just get it or whatever.
And potentially also the resource name.
And now these are bound to those targets using role bindings, which basically are map roles onto your targets.
Now these RBAC permissions can either apply cluster-wide or within a single namespace.
And the way that kind of gets differentiated is with, you know, the role slash cluster role or role binding slash cluster role binding.
So if it were a cluster role and a cluster role binding, the permissions applied would be cluster-wide.
And all resources that, you know, are allowed by that role would be allowed to that target.
However, if it's, you know, using the role binding, then it's only within a specific namespace.
Those permissions are granted.
Now, of those three, you know, kind of targets I kind of mentioned, the only one really managed by Kubernetes is the service account.
And these are authenticated using Bero tokens.
And they are, you know, fully managed within Kubernetes, typically, with service accounts being created as a resource.
The authentication credentials are mounted to pods.
So they have access to those service accounts, which they can use to talk to the cluster.
Users and groups are not managed by Kubernetes.
And it's typically left to some sort of external service with authentication modules within Kubernetes,
providing that kind of access to those authentication mechanisms so it can kind of, you know, authenticate users.
And it kind of trusts those third-party systems to be correct and just follows it.
So, for example, using X.509 certificates, the common name provides the user and the organization, you know,
within a certificate, tells Kubernetes what groups it's in.
And it just validates that it's been signed by the appropriate certificate authority and says, yep, that's cool with me.
Or, for example, if you're running a Windows-based environment, it can also use the Active Directory as a kind of source of truth for what users are allowed or not.
So service accounts being, you know, in Kubernetes, when you create a new namespace, which is the way people tend to do segregation,
a new default service account is created within that namespace.
And this default service account is automatically mounted into every pod within that namespace by default.
Now, this service account does not have any permissions directly applied to it.
However, if it's in any groups, then it would have those permissions as well.
But then this is typically something you should avoid.
Because if you do assign any permissions to this default service account for one pod, and another pod is also using it,
then that other pod would also have those permissions, you know, kind of breaking the principles of least privilege.
So it's typically recommended to disable the auto-mounting of these default service accounts and creating bespoke service accounts for each time you'd need it.
Now, the groups that, you know, the default service account may be in, there are three groups it's in by default.
So that's system authenticated, which is for pretty much every authenticated entity within the cluster,
system service accounts, which is the group for all service accounts,
and then system service accounts and the name of the namespace is for all service accounts within that particular namespace.
So if any of those three groups have any permissions assigned to them, that default service account will also have it.
There's also a system unauthenticated group. So if, you know, the cluster allows anonymous authentication,
the user, well, not really a user, but the person who has communicating with the cluster will have the same permissions,
as would be granted by system unauthenticated.
So there was a time not too long ago where we were on assessment,
and the groups for system service accounts are actually over-provisioned, granting cluster admin access.
And therefore, you know, as soon as we compromise a single pod in the cluster,
we were able to use that service account, which was mounted into every pod by default.
And we very quickly got cluster admin into the cluster in that manner.
Now, when enumerating permissions, there are a bunch of tools I tend to use when enumerating permissions.
I think the one I use the most is a tool called Rakesh, get hub link on the slides.
And essentially, this takes, you know, your token that you used for authentication or whatever,
and it kind of enumerates all the permissions that are available and then gives you a nice tick or a cross
in a tabular format showing, you know, what you are and aren't allowed to do,
which, you know, if you've just compromised a single JWT token to authenticate for a service account,
it's great to kind of figure out what you can and cannot do.
But then, you know, when doing more of a review of RBAC, there's a tool called RBAC Lookup,
which kind of lists through all the subjects, all the targets, the roles and role bindings
that are present in the cluster. So, you know, if we've identified a role that is, you know,
considered risky in a way, so for example, it grants you excessive permissions,
we can quickly identify what targets have that role assigned to them.
The final tool I'd like to quickly mention is QBScan, which kind of takes the approach a bit more automated,
where it automatically tries to find those risky accounts, whether these are just our cluster admin by default
or they have permissions that could easily help them get to a cluster admin level of access.
So moving on to networking, there are a variety of things you need to consider with networking.
Now, Kubernetes, as I said earlier, being very modular, the actual networking interface you use is actually kind of up to you.
Kubernetes say that, you know, it should follow the CNI container networking interface kind of specification.
But apart from that, we're Gucci. So within that, you know, there are a bunch of things we need to kind of look into,
like pod-to-service, ingress, pod-to-pod, and the CNI itself.
Now, because CNI is pretty much freedom, there are a lot of CNIs out there,
and each of them provide different functionality and features.
So there are, you know, the cloud network-based CNIs, which kind of integrate with your cloud providers.
Networking interfaces like AWS VPC would, you know, kind of put you within the VPCs
as opposed to a weird overlay network within the cluster itself.
And then there are others that, you know, kind of provide in-transit encryption or other features.
And there's a whole bunch.
For traffic to get into the cluster, typically, you would find an ingress controller.
Some alternative methods are like a node port, which is just, you know, a high-numbered port,
usually above 30,000, where a service is exposed.
But more often than not, you're going to find some sort of ingress controller,
where it's typically a pod, such as, you know, Trafic or Nginx or something,
that has permissions within the cluster to list the services, which are the way you expose pods in a cluster,
and then gets traffic coming in and then tends to route them around internally in the cluster
to the correct location or the correct service.
DNS for resolving within the cluster.
Now, if it's just a pod, it's automatically got a DNS name with the pod's IP address,
the name of the namespace, .pod.cluster.local.
However, it's usually not common for people to use the pod's DNS name for resolution.
It's more common not to use a headless service on top of that,
where after you've created a pod with the labels appropriate to it,
you create a headless service pointing to those pods, and it kind of simplifies that entire process,
as it can also do a load balancing between the various pods.
And typically, this is called, you know, the service name.namespace.service.cluster.local.
And because the search space for the DNS queries include namespace.service.cluster.local and service.cluster.local,
you can just do a query for service if you're in the same namespace,
or if you're in a separate namespace, service.namespace.
There's an additional DNS name that's usually also added, and that's hostname.subdomain.
And the hostname and subdomain is kind of extracted from the pod.
So the pod can specify the hostname and subdomain at which point it will be available using that kind of DNS name,
which kind of brings the idea of spoofing into play.
So, for example, if I were to set the subdomain to, say, com and the hostname to google,
suddenly I have a DNS queryable name, google.com, which would complete and resolve to an internal IP address.
So if you could, you know, somehow get yourself a google.com certificate, which I doubt you're going to get an external CA to do it,
but if you're using some sort of internal CA, it could let you do some interesting attacks in that manner,
or manhandling other connections.
Now, the Intel network is typically flat by default, because Kubernetes has this thing of, you know,
things should be able to talk to each other to facilitate, you know, microservices-based architecture,
which is great from, you know, a developer's point of view, where they can deploy their, you know,
series of pods and have them communicate in a way they need to, and it works.
However, from a security point of view, it adds a lot of issues,
because the pods in the cluster aren't only going to be the application pods,
they're also going to be, you know, a variety of management pods, etc.,
which if you can talk to, you can potentially do some dangerous things.
So, to kind of support locking down what kind of access pods have in the network,
there are network policies.
Now, network policies, effectively, are kind of like firewall rules,
where you can say, you know, these pods are allowed to talk to these pods or these CIDR ranges and nothing else,
and they're only allowed to, you know, get incoming traffic from these locations that are,
you know, permitted by the administrators.
So, it's kind of great.
Now, by default, there is no, like, deny rule or anything within network policies.
So, all the network policies are kind of like allow this connection.
So, the way it tends to work is if there are no network policies within the namespace,
everything's a flat network by default.
But as soon as you start adding the first network policy, it starts becoming,
you need to have a network policy allow that connection before it's permitted.
So, your default deny, instead of being a rule that says deny all connections,
is a network policy with zero rules in it.
And then you can kind of add the rules in as you need it.
Now, the Intel network, depending on the CNI you use, but most of them don't allow encryption in transit,
or they don't support it.
So, there are this concept of service meshes.
Now, service meshes, they typically provide encryption in transit,
but a whole bunch of other stuff like monitoring and other metrics that, you know,
your administrators can use to kind of help debug issues in the cluster
or just have an oversight of what's actually going on.
And the way this is typically done is through a sidecar container of a pod.
So, a pod is one or more containers.
So, this just would be another container in that pod.
And typically, these are automatically injected into the pod at runtime.
So, when that pod gets created, admission controllers are kind of used to inject those pods.
Now, admission controllers are functionalities within the API server that basically allow,
you know, modular-based additional features or checks to be added to that validation process
or request as they come in.
So, they can, you know, be used to write custom policies, for example, using OPA,
which is a policy as code tool, to provide custom checks that the API server can then use
to validate the data that's coming in.
Now, there are two types of admission controls.
There's mutating and validating.
Validating just validates the data and says, you know, this looks all right.
Yeah, I'm happy with this.
Mutating takes it a little bit further, where if it's not happy,
it has the option to change the request as it comes in.
So, for example, you have a pod coming in.
It feels like changing something.
It's well within its rights to do so.
So, the overall flow looks something like this, where when a request comes into the API server,
first, the API server would, you know, make sure you're allowed to do what you're trying to do in the first place,
to roll back and make sure, you know, you are a person or service account.
It would then pass it to the mutating admission controllers to see whether there's any changes they would like to make
and if they are happy with it.
Once all the changes are made, make sure it still meets, you know, the object schema validation.
So, make sure it's still a valid resource.
And then finally, passes it to the validating admission controls to make sure there are...
or run all the final checks and make sure it's still a valid piece of thing to save before saving it into etcd,
at which point the controller managers and schedulers can start, you know,
using that data to make the changes within the cluster as needed.
Now, it is possible to create your own admission controls,
but there are a whole bunch of inbuilt admission controllers that are there by default,
which can be enabled at the API server.
Now, there are a whole bunch of security admission controls as well,
but most of these tend not to be enabled by default.
So, it's usually manually need to be enabled in the API server if it's a self-managed cluster.
So, for example, pod security policy, which we're going to be talking about later,
or always pull images.
Always pull images is essentially a mutating controller where as pod specifications come through,
it makes sure that the image policy is set to always as opposed to the default of if not present.
This is especially important in like multi-tenant clusters where if it's set to if not present
and you have two separate tenants where one has permissions over an image and the other one doesn't,
the one that doesn't have permission over that image could use that image without having the relevant permissions.
If that image has been pre-pulled onto the node.
However, if the image pull policy is set to always, it would only allow the deployment of that pod
if that tenant could pull that image in the first place, which is kind of a way to prevent them using images
they're not permitted to.
And there's a whole bunch of admission controls like this.
In fact, having mutating and validating admission controllers, custom ones,
require the inbuilt admission controls for the same thing being enabled.
So there was an assessment we did a while back where we actually ran into OPA as a mutating admission controller
and within the network, it was the flat network.
So we could talk to the web front end for it.
And we found it in the cluster because the pod we had compromised had over-provisioned groups.
So we could start reading resources within the cluster because they had read-only access.
We could have also guessed the service name because it was just OPA.OPA, but that's boring.
Now, because it was a mutating admission controller, if we could get policies into that,
we could change the specification of pods in the cluster.
And luckily for us, there was no authentication for OPA configured.
So a quick policy pushed up to OPA and waiting for pods meant that we could suddenly start backdooring pods
in other areas of the cluster and start moving around the cluster in that manner.
Now, when talking about Kubernetes security, one thing that you kind of can't forget is the fact that
they are using containers under the hood.
And therefore, things like container breakouts do apply here.
Now, I won't be going into too much detail about, you know, all the ways that are possible for container breakouts,
but a couple of common ones are like, you know, leveraging host volume mounts,
where, you know, folders from the host file system have been mounted into the container
and you can kind of use those to add like a SSH key if you have someone's home folder
or if file run is mounted talking to the Docker socket if you're using Docker or all sorts of things.
There's also, you know, kernel exploits like DirtyCow if, you know, it's an out-of-date kernel version
or also overly permissive kernel capability.
So if, you know, the privilege flag has been used, you have the ability to load kernel modules.
So you can kind of load a kernel module, get onto the host that way.
And once you have that container breakout set up, because typically one node deploys model pods
on that same node, as soon as you've broken out one pod, you suddenly have access
to other pods shared on that node.
Now to help mitigate container breakouts, there's a pod security policy once you have
the admission controller enabled for such thing.
Excuse me. And that kind of helps mitigate a lot of container breakout attacks
by limiting what you can actually do with, you know, volumes, your user IDs that you're allowed
to run with, the kernel capabilities and so on and so forth.
Now, the way these are used is you would create a PSP or pod security policy.
And then through RBAC, you would assign that to your users.
And those users, when deploying the pod, would have to follow it, you know,
one of those PSPs if they... yeah.
Now, one of the things I typically recommend with this is, you know, using the system
authenticated group where you create the most restrictive PSP, you know, you can,
and then assign that to system authenticator at that point, you know, that should apply
cluster-wide. And then where more permissive pod security policies are required,
those can be, you know, assigned when and where they're needed.
On the same level, if you can't, you know, if you can't prevent container breakout
for whatever reason, and an attacker somehow does still manages it,
you can at least start controlling where pods are within a cluster
to kind of mitigate the impact of that compromise.
So let's say in the cluster, there are pods that have, you know,
just a standard basic brochureware applications.
There are ones that handle massive business critical data,
and all sorts of new management pods.
You can start categorizing nodes to, you know, whole different types of pods.
So you can have a whole bunch of nodes allocated to the brochureware applications,
all the stuff you don't really care about, a whole bunch of nodes allocated to,
you know, the business sensitive pods that handle very critical data that
you really don't want compromised, another set of nodes from the management plane,
and so on and so forth.
And these can kind of be enforced through tents and tolerations,
where you can apply these to the nodes and pods, and then a pod is only limited
where it can be deployed to based on these restrictions.
And then these can also be applied to the namespace layer,
where you can say that all pods within this namespace need to follow these tolerations.
And that way you can kind of, from a cluster administrative perspective,
manage what the impact could be of a container breakout should it were to happen.