All right. Talk's going to be a little about a container escape, those of you that are unaware. Enjoy.
What's up, DEF CON?
Yeah, mice are hot now.
After that, I think we're going to do great.
Yeah.
Okay, it's good.
Awesome.
All right.
I'm Dagan.
Hi, I'm Will Klein.
So we're all here as, you know, we're mostly like SRE, DevOps-y type people setting up clusters.
We're not pen testers.
So, you know, but when we're going through setting up these clusters, we notice some things and kind of repeating patterns of, you know, security issues that aren't being talked about enough.
And that's kind of what we decided to put this talk together for.
You don't have to be a Kubernetes developer to understand what we're talking about here.
But you just got to think like one.
And there's no zero days today here.
This is all.
This is all been disclosed properly.
And we're going to have three different demos that we're going to run through, different types of exploits that we want to tell everyone about.
With that being said, with our demos at the end, I have a GitHub URL.
The demo environment is set up in a local laptop using a make.
And that file is available.
So anybody that wants to can go and just clone that repo.
You'll be able to step through each of the demos we do today yourself on your laptop.
And there's also a step-by-step walkthrough.
All right, let's get started.
WTF is a Kubernetes.
I like to think of Kubernetes as the operating system for the cloud.
It takes anywhere from one to several thousand nodes or individual servers and turns it into a single orchestrated system.
That system can span an entire data center.
It can span multiple data centers and multiple clouds.
It's the ultimate shift left platform as a Kubernetes developer.
Not only are you.
Able to take advantage of containerization to bring your file system and all of your dependencies with your application.
You're also able to declare the the exact way that your application should be instantiated.
You can say, I want to run this command with these args.
You can include your configuration.
You can say, I want five instances of it and I don't want any of those instances running in the same availability zone.
You can go further and declare how much CPU time should be allotted to each of your applications.
How much memory is reserved for your applications.
You can say, this is what I want you to do to monitor the health of the application and how to respond when it becomes unhealthy.
Even how to scale the application.
Kubernetes is an extraordinarily powerful system.
You know, and the other thing that's really nice about Kubernetes is that you basically have, you know, decades of Google's experience and other large cloud providers experience and setting up these large scale systems and they've distilled all that down in a series of APIs to help you at any scale you want.
You know, whether it's like a Raspberry Pi.
Up to.
As big as Google deploy these applications and manage them, you get a lot of lessons learned out of that.
So it's really powerful system for managing your systems.
So today we're going to talk about putting a cluster.
So I want to just quick little thing about what we consider this pony a cluster.
So basically, you know, with all of that power to schedule workloads, spin up new instances, manage the networking, you know, what does pony mean here?
So we're talking about, you know, running your own workloads.
This could be as simple as, you know, Bitcoin miners.
This could be something that you're using.
As more of an implant to pivot to other containers in the cluster, stuff like that.
We're modifying existing workloads.
So that could be if I'm running some application that's managing throughput from user data.
If I can manage to modify the configuration of that pod in a way that makes it vulnerable to further exploitation, that's called pwning, right?
We're going to talk about hijacking traffic.
So one of the tools that we talk about a little bit is Istio, which is like the most robust and interesting man in the middle proxy you've ever seen in your life.
It's incredibly powerful.
And if you can take control of that, you can man in the middle literally anything in the cluster.
It's really powerful.
And then finally, you know, the fourth one here is breaking out so that you're executing code on the host as root.
And, you know, at that point, you control everything.
And, you know, that's the, like, pop calc up on the Windows box and prove you owned it, right?
So we're going to talk about three different attack angles today.
And realistically, this is not a real cluster that we're going to throw all this through.
These vulnerabilities probably could have existed at the same time.
But, like, it's not like we ran into a cluster that had all of these at the same time.
And, again, no zero days today.
All right.
So we're going to start with phase one here.
No more secrets.
So I mentioned earlier Istio is giant man in the middle proxy.
So basically the idea is that we're going to inject something into every time we launch a container and we're going to hook onto it to take all of this traffic through IP tables.
We're going to use MTLS, mutual TLS authentication, so that when you as a developer are writing a service, you don't have to figure out how to make it work.
You just run your pod in a system with Istio and you get the benefit of all that.
And that's kind of where Kubernetes becomes really powerful because we can take all of these lessons learned and hard cryptographic problems and boil it down so the developers don't have to think about it.
It's kind of like just bolt-on end-to-end encryption.
And it works really well most of the time.
One of the other great things about it is this Kiali tool, which is visualization.
So now that you're intercepting all the traffic and watching it through the network, you can really figure out what you're doing and how your network is working.
All right.
Quick shout out to JWTs, pronounced Jot, JSON Web Token.
Obviously, we're not talking that much about it, but I just wanted to bring it up as a concept of when you see this, it's a very common authentication protocol, how these little cookies are made.
Jot.io is a really great website for if you ever get one of those and you have to interpret it.
And the important part here is that it gets HMAC'd at the end.
So as everyone knows, cryptography is perfect and very, very easy to implement.
So we're going to talk a little bit about something we noticed.
You guys ready to hear about it?
You ready to hear about a vulnerability?
Yeah.
So one day, Dagan's setting up Kiali on one of his clusters, and we're bringing it up, bringing it down.
He's tweaking all the parameters in the Helm chart to figure out how to deploy it right.
And after doing that the first time, it stands out the first time, you get that login page, hot damn, got it.
And then the next time, you don't get the login page, and eventually he notices that he's taking these pods up and down, and the login page doesn't refresh.
He doesn't have to log in each time.
So in Kubernetes, pods are generally ephemeral, which means that they're not storing data between sessions.
And there's ways to store persistent data in Kubernetes.
So we started digging into it like, okay, if it's keeping my session alive, where is it storing that information, that cryptography to do the HMAC signing?
So I dug in.
And in Kubernetes, there's a lot of ways that you can persist data is what we're saying.
In this particular case, what I would have expected to see is a Kubernetes secret.
Where, you know, you install Kiali for the first time, it looks for the secret, it doesn't exist, so it creates it.
From then on, it would see that same secret and use that as its signing key for the job.
But there was no secret.
And I can delete the entire namespace, anything associated with it, and I was still logged in.
The only piece of persistence was the cookie that was sitting in my browser.
Which meant there was two possibilities, equally terrifying.
The first is there's no authentication of the cookie whatsoever.
You just simply say, hey, I'm admin.
And Kiali's like, I'll take your word for it.
The other is a hard-coded secret that's being used to sign the key.
So I dug through the code.
And you can see, bottom right corner, Kiali, hello.
Let's see, how are we doing on time?
I got a couple minutes.
So anyway, I went over to JotIO.
And I threw that in.
And I was able to just recreate or fabricate a perfect Kiali.
A perfectly valid Jot that would allow me to access Kiali as an admin.
The reason that that happened is because they had an insecure default.
It could be configured and overwritten in the helm chart to specify.
But nowhere did anybody take the time to say, when you're installing this application, you should definitely do this.
So nowhere did it actually happen.
So let's take a look.
So here's the Kiali login page.
I'll go ahead and refresh it just to show I'm not logged in.
Nothing up my sleeves.
I'm going to come over to JotIO and fabricate a Jot.
The sample on the left isn't signed because there's no secret in here.
So I'll put in Kiali as a secret.
I now have a signed Jot.
Just going to copy that onto the clipboard.
Using developer tools on my browser, I'll add a cookie.
It's just called Kiali hyphen token.
The contents of the cookie are just the Jot.
Which is totally valid, assuming it wasn't already signed by a completely hard-coded secret.
Now when I refresh the page.
Fortunately, from the attacker, Kiali is extraordinarily powerful.
There's a lot that I can do here.
It's a powerful service mesh that allows me to completely rewrite how traffic is managed.
If I want to send 10% of your login traffic to my server instead, I can do that.
If I want to add entirely new ingress points to things that shouldn't be secure or shouldn't be exposed, I can do that.
I can pretty much do anything that I want to do.
So, Dagan, we were sitting at the bar that day and you were telling me the story about this.
One of the things you mentioned was that when you kept reloading the page the next day and the next day, it still stayed up.
How did you make that work?
Well, my talk on security ramifications of time travel was rejected.
But I'll explain.
So here's the code that Kiali uses to authenticate a cookie.
Sorry, a jot.
The line 67 there, it's the second one from the top.
It just says to the library, hey, can you parse this token and if it's valid, give me back the claims.
What the developers didn't understand is that particular library, it didn't care what time it was.
It just said, yes, this is a validly signed token.
Jot.
What's not on there is any attempt to say, well, the expiration date is that in the past.
Wasn't there.
So you could do a replay attack with this, no problem.
So, you know, if we imagine our fictitious cluster here, we've now, you know, we're able to get to this Kiali service.
We're able to forge credentials.
And with that, we can pretty much read anything in the cluster.
And we can write to Istio, which is the networking, you know, throbbing.
The nervous system of this cluster.
And we can reroute that wherever we want.
You know, the default for installing Kiali is not read only.
So at this point, we can manage the middle proxy, literally anything we want.
We can blow up the system.
We can shut it down.
We can, you know, steal credentials.
So at this point, it's safe to say we've pretty much pwned that cluster.
You know, not much left to do there.
So that's all right.
Thanks for coming to our DEF CON talk.
But, Dagan.
What?
We promised them three demos.
So what if?
What if?
I said earlier the default wasn't that it was read only.
Let's say it was read only.
What do we do with read only now?
Okay.
That is admittedly a little bit of a challenge.
But fortunately, this talk doesn't end there.
So let's go back and we'll say we're going to make this harder on ourselves.
And let's say that somebody responsible named Will Klein set up this instance of Kiali.
And so it is in view only mode.
What could we do?
Enter my friend Fleet.
So is anybody out there familiar with Fleet?
With Rancher?
Okay.
So you guys know.
Two victims.
Do you work for Rancher?
Ooh.
Listen up.
So this was responsibly disclosed and it has been resolved.
But Fleet is GitOps.
So for those not familiar with GitOps, it's essentially a really popular and great way,
genuinely a good way of administering cloud infrastructure where you put your configuration,
you put whatever the manifest that you need to deploy into your cluster into a Git repository.
And then you can have multiple clusters pointed at the same Git repository.
And it's an easy way to build at scale.
Also, it's Git.
So it's actually very auditable.
And you can go back and say, hey, Will, you deployed this thing.
Why did you do that?
Right?
There's other instances of GitOps tools for Kubernetes.
Flux is a popular one.
Argo CD is a little bit more mature.
But, anyway, that's Fleet.
Well, and the other thing that's great about GitOps tools like Fleet is you can basically,
you know, everyone's had the situation if you're any sort of cloud admin where you set
something up, it's running good, and then, you know, somebody comes along, says, oop,
needed to expand that disk a little bit, and oop, needed to redirect that and change labels
here.
And as time progresses, you just get farther and farther away from whatever you thought
the truth of your cluster was.
GitOps tools will constantly be thinking that and looking at it and saying, like, nope,
back to what Git says.
Nope.
Back to what Git says.
So you always can quickly understand what's running in your cluster and how it's being
deployed, and that's just generally a really good thing.
It makes life easy.
Makes sense.
Let's talk about logging.
I've been professionally developing for about 15 years, and one of the things that I've
come to learn is that developers either whisper or they shout when it comes to logging.
Unfortunately, in this case, Fleet was one that was more of a shouter.
And as I was digging through the logs one day, in fact, it's even a little bit worse
than that.
When you're running Fleet with multiple clusters, and that's not what we're doing in our demo,
but just so you know how bad this was, if there's an error message, it gets displayed
in the UI within a product called Rancher Dashboard.
And so it's very, very helpful.
And I was working with Fleet, and I tried to deploy something, and it didn't work.
So there was an error message that popped up that said, I tried to get this Git repo,
and I wasn't able to.
The problem was Fleet uses a library called GoGetter from HashiCorp.
And the way that GoGetter works is you provide a URL that tells it everything it needs to
know to go and get whatever file you're asking for.
It can work with Git repositories.
It's a very valuable tool.
But in order to get the Git repository, you need to give credentials.
And so those credentials need to be embedded in the URL.
And when the error comes back and says, I couldn't get this entire URL,
and the credentials are in it, the error message that gets logged includes
the credentials to the Git repository, which we just said you can use to deploy
anything you want to all of your clusters.
So what could we do with something like that?
So as previously mentioned, Kiali is super powerful.
One of the things it lets you do is look at logs of individual workloads.
So I'm going to take a look at the Fleet agent logs.
And oh my goodness, is that a base64 encoded SSH private key?
Fortunately, it's encoded, so nobody knows how to get access to this now.
It's perfectly safe.
This is my actual GitHub private key, by the way.
Yeah, of course.
You're running low on entropy, so you only have one.
Let's see, base64 decode, oh my gosh, it's an actual private key.
So what else could I do with this?
Well, one of the things that the log message told me also is the actual URL
to the Git repo that it was trying to read.
Could I perhaps clone that and see what's there?
Yep, I can.
And it turns out it's a Helm chart,
which I can do some interesting things with.
It turns out the reason that there was an error is
if you look at the branches listed, it's only main.
Main is the only branch that exists.
But this is looking for the develop branch.
So what would happen if I created a develop branch,
put my payload into it, and pushed it back up to the Git repo?
Well, let's find out.
I'll just create a job.
And I'm going to put my malicious payload into that job.
This one's set to just pop a reverse shell.
Apparently I have right permissions.
And now it's a waiting game.
GitOps does its thing, and I have my reverse shell.
So I'm going to go ahead and do that.
And I'm going to go ahead and do that.
So to keep this sort of going,
we're going to add some more contrivances.
Again, Fleet has global permissions.
I could install anything that I wanted at this point.
I truly own this cluster.
But what if, again, somebody named Will Klein set this up
and actually constrained the default cluster role
that Fleet runs under so it can only do some things?
You know, it can deploy a pod, but it can't deploy a cluster role,
and cluster role bindings, which would be pretty smart.
Yeah, so basically we're looking at this cluster now.
If we think about it, we've gone in and we've taken a service
that's generally kind of an available thing.
We've gotten past that.
We kind of locked it down a little bit retroactively there,
but we can see the logs now of all the pods.
We're looking at Fleet.
We managed to get access to the GitOps.
Now, one of the things you're thinking in your head is, like,
who in the world would possibly just use their actual SSH credentials
while setting up their GitOps?
Well, I find this all the time in the real world.
Like, it happens pretty often, especially with GitLab tokens,
where the developer will go in and generate a personal token
with read-write access instead of going in
and creating a deploy token with only read access.
If we only had read access, this would be a lot harder to do,
but there's a lot that you can do with that information.
Once we have write access to the GitOps repo,
now we can push and we can do whatever we want
just like we were the cluster admin.
It's really convenient for cluster admins,
and it's also really convenient for anybody else
who wants to run things on the cluster.
So, really, at this point, it's kind of pwned
because we could literally run anything we want
because the GitOps tool is just, you know,
bobsledding whatever we put into Git into your cluster,
and that's how it works, and it's working as planned.
So what went wrong is the developers at Rancher
were using a library, and they used it properly,
but they didn't fully understand
what contents would be in that error message.
So they got an error, and they logged the error,
which is normally a reasonable thing to do,
but sometimes that can turn into an information disclosure.
It's kind of understandable, to be fair.
When I traced it, the error was passed up
through nine different call stacks,
so it was just like, here's an error, here's an error,
here's an error, and when you get to the very top one,
it was like, I don't know, log it?
That way we know what went wrong.
All right, and so we're going to talk about phase three here,
so let's pretend that we had, in this fictitious cluster,
we have a read-only Kiali,
we have a super locked down RBAC around our GitOps tool,
but let's look around and see what we can find there.
So in this cluster we're working on together,
we were using this tool called Longhorn.
Now, when I said earlier that Kubernetes is this whole API,
really, the API describes how to do all the things
you want to do in your cluster,
like I want to launch a service,
or I want to have an endpoint, right?
But there's actually the API-specific implementations
vary cluster to cluster.
So there's a thing called
the Container Storage Interface.
It represents how is persistent data
going to be stored on the cluster.
You know, normally the pods are completely ephemeral,
so you lose it, you lose it.
Longhorn is a microservice-based storage system
that runs on Kubernetes.
So at this point you have Kubernetes storage
feeding Kubernetes storage.
So it's a pretty neat project in that sense.
It's made by Rancher, and, you know, there's basically,
there's a couple different pods running,
but there are two ones that we want to talk about today.
One of them is an instance manager,
which means that, which is running on every single
node as a daemon set.
And the daemon set says, you know,
when a new auto-scaled node pops up,
I want to launch my pod on that one and do some things.
So it's running everywhere in the cluster,
this daemon set.
And then there's also the Longhorn manager,
which kind of wrangles all of those instance managers
running all the different instances.
So these have a communication between them,
and they, you know, manage the provisioning of storage
and all the different actions.
So as part of our, you know, hardening,
getting things ready, you know,
I pull up Longhorn,
and I'm looking at some of the pod security policies,
PSPs, those are being deprecated,
but that's a talk for KubeCon.
And I'm looking at the default ones,
and I'm just like, wow,
they're basically allowing everything.
You know, there's a requirement there is a PSP,
but you've done everything you can
to disable anything useful in it.
So it's, you know, it's like having the firewall
where everything's left open to all the ports, right?
What's the point of having a firewall?
You're just checking a box that you have one at that point.
So we started going in and trying to figure out
like, okay, what can I narrow this down for?
And I'm going in and out,
and we're launching it and tearing it down,
and just, it's taking forever
because every time I try to take
the least amount of privileges away from it,
it crashes and won't run.
And so we're piecing this apart,
and it's like, well, really,
there is no reasonable PSP around this.
It's doing all sorts of nasty things.
It's bringing in directories
from the root on the host
into the container on the inside,
and then it's also running itself as root,
you know, if you're in a container
and you're mounting all the important directories
inside your container and running as root,
you're not really a container anymore.
You're just a way to run code at that point.
You know, Nullify
has any of the security advantages.
So we're looking at this thinking, okay,
well, at least we can try to make the network layer secure
because, you know, they didn't support any network policy
or MTLS.
So I started digging into that now,
and just more of this weird behavior
where the network policies are really difficult to write
because every time it does something,
it opens up a new port.
And, you know, fun fact that a lot of people don't know
that when you're declaring your pod spec,
you specify which ports you have.
That's more of just road signs and information.
You know, your pod can listen on anything.
That is absolutely not a firewall.
That's not a firewall rule?
Yeah. You know, we've had this conversation
with people where they literally thought
that those were some sort of firewall
that was blocking traffic.
It's really just a convenient shorthand
to help you locate the port.
So, you know, we can't lock it down.
And at some point we started talking about,
like, what is this pod doing
and why is it literally root on every single host?
So we started digging into it a little bit
and thinking about it.
And having had our conversation about the Kiali thing before,
we're like, okay, well,
let's pick this apart and see how it's authenticating
because we started noticing that we weren't seeing
secrets in the cluster, which, you know,
that was the same hint that we had for Kiali
that Kiali was up to no good.
So we're digging around and we're not seeing
anything that's storing the persistent data
but, you know, how would these pods
authenticate each other?
And, like, if it worked with MTLS,
we could kind of pretend that it was doing a good job
because Istio can kind of fill that gap
but we couldn't get it to work with Istio
because of this weird network behavior.
So we're digging around and, sure enough,
we found that they actually weren't
authenticating each other.
It's just a gRPC that's listing on a standard port
that's not declared and away it goes.
Okay.
So, for those who may not be aware,
is a remote procedure call protocol
from Google.
It runs on top of HTTP2.
It uses protocol buffers to encode data.
I love it.
It's a very cool protocol.
You should absolutely be using gRPC.
Like, truthfully.
One of the nice things about it
is you don't have to write the server
and client bits yourself.
You write a protocol or a protofile
that defines this is what my API looks like.
If you're familiar with Swagger, it's a little bit like
that, but then in addition,
gRPC has plug-ins for all sorts
of languages.
So you can run that plug-in
in your compiler and point it at your
protofile and you actually get
the server or the client.
With the server, it expects you to have something that
implements an actual interface that does the thing,
but it gives you all the connective bits in between
and it's really nice.
It's particularly nice if you're an endpoint developer
and you want to support people that are writing in
different languages because you can write in Go,
somebody else can write in Java,
I don't know why they'd write in Java, but they could write in Java
and then they could just run the plug-in
and they get a Java client for your
API.
It's also really nice
because it's easy to read.
So I was really excited
when I looked at Longhorn's repo
and just saw these protofiles in there.
I was like, oh, I can read that. That's simple.
So I started pulling it apart and I saw
this service that was called
CreateProcess and I
thought, well, I hope
that the Longhorn developer
developers have a different meaning
of process than what I have.
So let me look at the parameters
that get passed to this service.
And there was fun little bits in there
like binary
and args.
And I thought, surely
this is
not an API
that allows anybody inside the cluster,
because remember, clusters networking perimeter
is the cluster. It's not at the namespace.
So no
ability to add a network policy,
no ability to use Istio
to do MTLS enforcements,
no authentication.
This can't be the case.
They must be doing authentication somehow.
Now Longhorn Manager does generate the pods itself
and push them out there, so it could be generating
something as an environment variable,
which wouldn't be the most secure thing, but it'd be okay-ish.
No.
No.
It wasn't authenticated at all.
It took me...
I work on the west coast and Will works on the east coast
so I have to wake up
at 5 a.m.
So it was about 5.05 a.m.
and I was looking at this and
I'm awake now and I reached out to Will
and we said, well let's do a POC.
This can't be a thing.
So we started looking at it a little bit
and I said, well,
what can we do with this access?
I mean, it's sandbox, it's a container.
Well, we'll go back to what Will was talking about
about the file systems from the host
that were being mounted.
So there was two.
PROC and DEV.
Not the best two.
And Will said, you know,
in PROC,
PID1, root,
that will always be the host's
root file system.
So I wonder if we could just say something
that's just going to touch something
in host, PROC, 1, root,
let's get fun, Etsy,
whatever, right?
And so because this was written
it took me about 25 minutes
to slop together a real quick POC and go
and run this thing
and Will's sitting there
he's SSH into the node
and he's like, okay, I just did an LS on Etsy
go, I go,
I hit go, he refreshes
and there's a file that says, this should not exist.
Owned by root.
So we took that a little bit further
we developed an exploit
for it, it's called Rustler
it is available on GitHub
it basically allows you to point
Rustler at a target that's running
Longhorn and just say
do this for me please, and it does
every time, it's very reliable.
Now to
Longhorn's credit, again, all the developers
we worked with on this have been really amazing
we reached out to them, they pretty much
immediately understood the problem
and they worked to fix it.
But before they did that
let's take a look at what we were able to do.
...
...
...
Okay, so I'm back on my reverse proxy
from the previous demo
I need to find Longhorn
so I'm going to temporarily
broadcast with netmask, ignore that
and I'm going to run a quick end map
across the subnet that I'm working on
looking for anything listening on 8500
which is what instance manager listens on
it's not a big subnet so it doesn't take long
there's two, I'll grab one
remember, it's running
Ubuntu as the base image
so I have access to all of Ubuntu
2004 that I can run commands
against, so I'm going to start up listening
again, and I'm just going to do a really basic
one liner bash reverse shell
...
...
...
...
...
...
...
Who am I?
I'm Root.
...
...
...
So
going into the proc directory
if I change root to
host proc 1 root
now I'm
root on the host
and for the kube administrators
out there, I can run crycontrol
So, basically at this point I can do whatever I want with your cluster.
If this isn't enough fun for you, Longhorn has essentially two different modes that it's
running in.
There's everything and then there's Longhorn Manager, which Will was talking about before.
Which is creating a lot of pods, it's doing a lot of things.
So it has very permissive RBAC rules, like it's allowed to do a lot of things.
For some reason it's allowed to do those in any namespace it wants.
It's not isolated to the Longhorn system namespace.
I don't know why that's the case, it doesn't seem to be necessary at all, but it is the
case.
Also, at least previously, even though there was two very different security modes of I
need to do a lot of things and I basically need to do nothing, there's one service account.
And that service account is shared across all of the pods.
So at this point, running an instance manager, I am able to run inside that pod, kubectl,
and create pods.
I can do fun things like read any secret that's in the instance.
In the cluster anywhere, create, read, update, delete on almost every type of resource.
Even if I didn't have that, Fleet still got my back.
Because Fleet's permissions are API versions, all of them, resources, all of them, verbs,
all of them.
So anything that I want, I can do.
Yeah.
So basically.
Once you have that little foothold in the cluster that we've gotten, and really this
is just a one way to do it, but a lot of people don't think about, they think about the containers
as a security mechanism, and they assume that once your application's in a container, that's
going to hold back an attacker.
But realistically, because namespaces are not a security boundary, and there's really
nothing that's a firewall in this, Kubernetes makes service discovery really easy, which
is why we were able to find Longhorn pods so quickly, and there's nothing holding us
back from talking to them.
So.
So it's one of those things that when you're deploying a cluster, you need to be thinking
about where are the actual security barriers, and where are there kind of like shadows of
like insert security barrier here, right?
You can write.
Somebody's got this for me, right?
Yeah, someday someone's going to insert a security policy that blocks this, but building
out your cluster, you really need to think about that.
Basically once we're on the host, there's very little that we can't do on that host
as far as getting back to the master nodes or the control plane stuff.
And we don't even really need to do anything on the control plane.
Because we can launch literally any pod we want, including more incredibly vulnerable
pods that mount very dangerous directories into them.
So there's really nothing we can't do in the cluster at that point.
And if there are good pod security policies in place that prevent us from doing that,
it's okay.
We can deploy our own pod security policies that correct that for us.
All right, so some big takeaways.
One thing that we do.
Really want to emphasize is Kubernetes itself is pretty damn great.
The Kubernetes developers are absolutely amazing.
There's a ton of them, and they're some of the most brilliant developers that I've ever
had the pleasure of working with.
But not everything in the Kubernetes ecosystem is written by those Kubernetes developers.
And a lot of times, to be really fair, these projects, they start off as one developer's
idea.
And that turns into a proof of concept.
And then someone goes, cool, ship it.
It's mission critical.
Let's go.
And we don't necessarily rewind and sort of rethink, is it secure?
Is it what we need it to be?
I mean, you have some bullet points there.
One of my things that I like to drive home with people is, good code does not equal good
deployments, right?
One of the advantages of Kubernetes being the ultimate shift left capability is that
developers get to work on deployments.
But there's also people that have been working in system administration and building all
of this cluster stuff out.
A lot of lessons learned about how to have least.
And the developers aren't necessarily thinking about that.
So a lot of times what happens is you'll see a perfectly good project that was actually
implemented pretty well on the coding part.
Then when you go to deploy it with the helm chart, like the NAMI has a lot of this, none
of the defaults are anywhere near secure.
And it's difficult to secure it because of the way things are written.
So just because you have a really secure project, like your MySQL, your Postgres that's up
to date, doesn't mean that the deployment of that system can't shoot you in the foot.
Yeah.
So not that there's any attackers in the audience today, but if there were, what do you do?
You got your hands on a Kubernetes cluster.
Maybe you found an insecure application that lets you get like just a toe in.
How do you pivot?
Understand some of the common mistakes that people make.
What are the assumptions that are just frankly inaccurate in a Kubernetes environment?
So again, going back to service discovery, there is unfortunately commonly a presumption
that networks are like that namespaces are network segmentation.
They're not.
So look for people that are just have completely insecure APIs and don't be surprised if you
find it.
Secrets should exist somewhere.
Somehow information needs to be shared across these different applications that need to
be able to communicate with each other.
If that doesn't exist, ask yourself, how is this being secured?
Because it might not be.
Additionally, GitOps is super fun to play with.
If you can get access to a GitOps repo.
Yeah.
You own not one cluster, but lots and lots of clusters in all likelihood.
So watch for that.
And then finally, for me at least, don't forget to check the logs.
I go back to it.
Some developers whispers, other developers definitely are shouting.
You know, I think one of the things that, you know, I want to wrap up on here is that
Kubernetes is pretty damn great and it has all this great stuff in there, but, you know,
a lot of the bad behaviors that, you know, people do on their local development machines
just has a tendency to get embedded into the deployment systems and then end up through
GitOps into the cluster.
So, you know, in no world is GitOps, you know, I used to joke a lot with people that, like,
SecDevOps or SecDevOps is really just that we took the developers and fired all the ops
team.
Right?
There's a lot of lessons learned from traditional ops that can be applied to through DevOps
type stuff to the deployment of applications on Kubernetes, and we can't let the devs just
take over the whole process.
We forgot to hire an ops team.
Okay.
So, before we go to questions, again, I mentioned that the full demo environment is automated
and available.
These slides are available, too, by the way.
Any of the main tracks, if you go to media.defcon.org, you can get all these materials, the recording,
all that's there.
But, you know, feel free to go there to github.com slash Dagan slash Defcon 30.
I promise to really actually try to support folks for at least the next couple of weeks.
So, if you have problems working through any of this, open initiative.com.
It should be pretty easy for folks to do.
Also, today we talked a lot about what can go wrong in Kubernetes, and if you're interested
in knowing how to protect your clusters from that, we're going to be doing a similar talk
but in, like, the defense side at KubeCon, CloudNativeCon in Detroit in October, so
feel free to meet up with us there.
With that being said, we have, I think, just under ten minutes or so left, so if anybody
has any questions, there's a goon back there with the microphone.
We'll do our best to answer them for you.
Thank you very much.
Yeah.
Thank you.
Anybody with questions?
I guess we covered it all.
All right.
Well, we'll be hanging around for a little bit and then headed out that door, so if anybody
wants to ask us anything directly.
Again, thank you all for your time.
Thank you.
Thanks for coming out.