Ah, come back here. Why does that keep doing that?
All right, great. We are here and ready to do securing in,
path testing, great spaghetti monster, as I like to call it, or
Kubernetes. This is who I am, Kat Fitzgerald.
You can find me on Twitter at Rainbow Kat. I've been doing this a long time.
I'm the CEO of B-Sides Chicago. I was COO for
the Diana Initiative last year, part of B-Sides Pittsburgh,
and also my first DEFCON was DEFCON 3.
How many other people remember that one? There were 16
speakers there, if I have to remember that.
I have an emphasis in the blue team. However, I'm a formal purple teamer
and all this stuff. And I'm in Pittsburgh now, but moving to
Kirkland, Washington very, very soon. You will always
find me sipping Grand Mayan Extra Añejo, or in
this case, it is Casanoble Añejo.
So, cheers. Let's kick off DEFCON. Jason
already did a great job.
And also, special thanks to the entire Red Team Village,
all the staff, all the volunteers, everybody for kicking butt
and making this happen. A lot of my favorite stuff are honey pots
and refrigerators. On Saturday, I'm doing a talk for
IoT security, so check that out, too. But this is one of my favorite ones
because I just finished writing this particular
presentation just about a month ago, made some changes to it, and so on.
Remember, please, the views and opinions here are mine and mine
alone. They do not reflect anything from my employers, past
or present. Anything that I show you,
as always, please use with a grain of salt. Don't go breaking
into things that you shouldn't be breaking into. And those of you with an
overwhelming fear of the unknown, you'll be happy to know that
there is no hidden message in the disclaimer if you read it
backwards. Yes, my jokes are that bad.
Alright, let's get this thing on the road. So, why are we
not here? Well, first of all, I'm not going to solve all
your Kubernetes security woes. I wish I could.
I wish Kubernetes and containers and everything were secure by default.
They are not. Neither is the person
sitting next to you, although, well, that doesn't work anymore because
nobody's sitting... well, maybe somebody's sitting next to you. You never know.
You never know. Okay. And common sense went out the window
decades ago. Think about that. When it comes to
information security, cyber security, hacking, whatever you want to call it,
I honestly think we lost common sense quite
some time ago. Why really are
we here? Well, Kubernetes, I
don't know. I still think it's fairly new. It's fun.
It's crazy. It's a lot of things.
You can deploy it in clouds. You can deploy it in...
I have Raspberry Pis sitting down here. I was
going to be showing that. There have been some demo problems and also some
timing issues, so I won't be showing my Raspberry Pi cluster today.
But you can build very extensive Raspberry Pi
clusters and do all the things that I'm going to talk about today.
Containers. Containers are not new.
But people seem to have forgotten that they
have anything to do with security. They get all
hyped up on Kubernetes security, and they're forgetting about
containers. We're going to talk about containers because it's
important. It's, in fact, very important. And always remember this.
Security is for everyone. Common
sense is required. So please
disregard the previous slide. Throw it out the window. Why?
Because if we're not using common sense in security, then
we're definitely not going to get where we need to be.
All right. Just some quick security facts. We want to
know this stuff. In 2018 alone, $114
billion were spent by companies on
information security. On fancy software,
fancy hardware, things with blinky lights and all of that.
But breaches kept on going up. We're doing
something wrong here. Security appliances and software
have vulnerabilities that we're not dealing with or that
need to be dealt with by the vendors. Kubernetes itself
has had a few issues. We'll talk about that. But so has Docker.
So don't forget your containers. Also, I've
actually had people say to me, we have a very unique security
architecture. And I'm like, no, you don't.
We all do the same stuff. So remember, your architecture
is not unique. I love this quote. If you don't have this
book, Offensive Countermeasures by John Strand, go out
and get it. It's a small book with a ton of information.
So get it. But he said, instead of brilliance,
we've standardized on mediocrity. And that seems to be
what information security is about these days.
And I kind of have issues with that. So let's talk about
that some more. First of all, let's get a common definition
of what is a breach. Or maybe we'll call it
what is an incident. Let's not necessarily call it a breach.
So most incidents are not caused by zero-day
fancy stuff that you find on the Internet. Typically,
they're done because people have made mistakes and so on.
Most breaches are not coming from your vulnerability scanners,
especially when it comes to Kubernetes. You're not going to run a vulnerability
scanner and go, oh, look, fix this one thing, and we're
done and ready to go. No, it doesn't happen that way.
Most breaches are going to come from
configuration issues. This is something I want
to emphasize. In fact, I'm going to show you some examples how
configuration issues tend to cause most of the breaches. A close
second is compromised credentials.
When I say credentials, I still go out on GitHub.
Well, maybe not every single day. And I think Jason
even mentioned it, too, in his previous talk on bug bounties.
Go out and look on GitHub and search for simple things
like password. See how many times you find
the word root. Yes, thousands of them
are invalid. They don't make any sense. It's probably going to find the keyword
root or password, and it's not going to get you anywhere.
And yet I still find where people hard-code credentials
into variables that say password. And right
next to it, there's the key. There's the whatever the password
is. Also, trailing in third
place for most incidents is overprivileged
users. This is incredibly important in
the Kubernetes world. I'm going to show you this in just a second
or in a couple of slides here. So keep these three things in mind.
Configuration issues, compromised credentials, and overprivileged
users. Okay. Kubernetes
itself, we have to deal with all three
of these compromises or these types
of incidents that are going to cause issues in Kubernetes.
But first, think
about this. You know, we think about the Capital One hack
that happened recently. Well, that was a combination
of compromised credentials and access
that should not have been granted. We see how
Tesla, you know, they had poor access credentials
and so on. Equifax, we all know about that one.
Go back and check if you don't. You know, Equifax
was hacked not only by a simple vulnerability
that had not been patched, but also because they had very
very poor lateral movement detection on the
inside. Once they got in with more compromised
credentials through the vulnerability, they weren't
detected for quite some time. This is going to be
critical. And one other thing, I was bored this morning. I decided
to throw this up here in this slide. Why? Well, I
put up a new spice rack in my kitchen. I really liked how it looks.
I took a picture and I added it to the slide. There it is.
Isn't it a great spice rack? Come on. Alright, you all better
be laughing. Alright, let's talk about containers.
Quick review of containers. They are not
secure by default. Now, I'm really focusing more on
Docker containers for now because that tends to be
the norm. And one thing to think about when it comes to
containers, containers are not like
a VM. A VM is actually more
isolated. And this is something that's going to be important coming
up here. So, a container is a grouping of stuff
and the stuff that we like to call it are resources. Well, the
resources are grouped into namespaces. If
you are new to containers, remember this. Namespaces
are critical. You group your processes,
your networks, your users, your ports, your interprocess. All
of this stuff can be controlled and
locked down within namespaces. Resources
are exploitable. I was sitting in another
web broadcast just the other day
on containers and I was
surprised because it was supposed to be a more
intermediate presentation on containers and
yet all of the people were asking is there a
way to prevent process
abuse such as CPUs and memory and
so on. And again, this was supposed to be an intermediate
class on containers so it
confused me a little bit that people were asking that because that should be something
that you know from the very beginning that you get into containers.
Using namespaces and the next
most important thing is control groups or cgroups. These
are the limits that you are going to be putting on containers.
If you don't put limits on them, you're
going to run into problems. Every time I do a pen test
of a new application deployed into a container,
I would say that 7 out of 10 times
I find that the author of the application
who has put it in a container has not put any limits.
I'll get into the application in whatever way possible whether it's
a SQL injection or an RCE
or something like that. And once I'm in, I can take over not just the
container but the entire system that it's on
because there are no cgroups applied to the container.
The other thing to remember about containers
is think of a container as an image or more
correctly, think of it as a file system snapshot.
So who decides what goes
in that image? If you look at this particular picture,
you'll see, you know, here we have
namespaces, the control groups that I show.
It's going to possibly be cherooted if you've set it up
correctly. And here's the application.
There's a Docker client that you might use to launch
this. There's a Docker registry. This is critical
to the security of your system. If you're
new to Docker, you may not know this, but Docker Hub was compromised
two years ago, I believe it was. I've lost track
of time during COVID. I don't know anymore.
But you want to have a secure registry where you keep
your images because if they're somewhere
that you don't have control, you never know what's going to happen.
But here's the difference. Notice that we have a host OS
and hardware down here, unlike a VM.
In a VM, you would see the entire operating system
within the VM here. You would not see a shared
OS. Yes, there's a host OS, but then you also have the guest
operating systems themselves. And that's where we get into
how a VM is much more isolated
and contained than a container is. So
don't think the word container means that you're more secure.
It just means that we have some work to do. So that's why
I say right here in this slide, container isolation.
Is it really isolated? The simple answer to that is
no, it is not. So let's talk about some
container best practices. I should have also said at the very
beginning of this talk, this is an intro.
I can't show you everything. I'm trying to get people to start
thinking about how you secure containers, then
we're going to jump into how you secure Kubernetes at the
same time. So remember, we have
a base image. That means that base image has to come from
a secure repo. If you do not know what CIS
is, definitely go to CIS
or CISecurity.org
Check it out because you want to use CIS
benchmarks for locking down your base image. I'm going to
talk about that a little more coming up. Don't run your images
as root. By default, if you do not set a user
name, your container and your application in the container is going
to run it as root. I have found so many
who have deployed WordPress applications
in a Docker container via Kubernetes
and they don't set the user name and WordPress is running
as root, so is the database and everything else. So of course,
once it's compromised, you take over everything. Also,
visibility. You need monitoring within
your containers. Your containers put a lot of information
out and they can log it, but you have to capture that. We're going to talk
about that in a minute. Also, think of
it as a domino. If one container is
compromised, I guarantee you the rest of them will
fall over like dominoes.
When you start to get into containers and Kubernetes, you're going to see this.
Also, we want to set some controls.
Besides setting a user name that is not
root, we also set privileges and we set
certain attributes to those users where it says
no elevated privileges or no new privileges.
You don't want user accounts or IDs
to be able to escalate privileges. And I'll
show you some examples coming up. This one, my god, this is
near and dear to my heart, secrets management. I still
find, I would love to say by a show of hands,
but I can't do that right here, how many people
put the secrets
and tokens and things like that in their Docker
YAML files so when they then deploy the Docker
image, if somebody gets a hold of their YAML file, which
describes the configuration of that container,
well, the secrets are right in it. I've worked
with a lot of developers and a lot of engineers and I get
them, you know, you have to start thinking about putting
your secrets in another place, whether it's HashiCorp
Vault, whether it's a cloud secrets management system,
something. But get them out of the apps and get them out of the
configuration files. And stop allowing SSH
into your hosts. When we get into
Kubernetes, coming up in a couple slides here,
one of the key constructs of Kubernetes is you have a master node
or master nodes, and then you have worker nodes. And I
still find worker nodes with SSH enabled
on them so people can SSH in and, quote, fix
things. No, no, you use the
management tools to fix things and redeploy. Stop
allowing SSH into your systems. Also, I can't
stress it enough times, namespace
limits. Go read up on how to
control namespaces. That is going to
be one of your best efforts and best
practices for containing and controlling and
securing a container before you distribute it to
or deploy it via Kubernetes. This is one of
my favorite tools. It's called Docker Slim. If you've never heard
of it, if I were to create a container,
let's say I create a container, it has Ubuntu in it
or the base of Ubuntu, and then maybe it has WordPress
running in it for whatever god awful reason that I want to do this.
Let's say that entire image ends up being
three to four hundred meg. It's still
manageable, but if I were to run that through Docker Slim,
not only will it add some extra configurations
and namespace limits and so on, but it shrinks it down
and something that might be three or four hundred megabytes
deployed into a container will get shrunk down to
maybe 10, 15, 20 meg. This
makes it even that much more manageable. It's easier to
deploy. It's quicker to deploy, especially
if there are requirements for scaling.
So think about this. Take a look at this. I also mentioned
in Discord that my slides will be available in my GitHub
repo, and I'll get them to the Red Team Village so they can add them
to the Google Docs location after my talk.
And always remember
apply SELinux or AppArmor. One of the things
that Docker Slim does is it will do that for you.
It's very easy to enable SELinux or AppArmor
within a container. If you can do that, you're going to make
it that much more secure. It's not on by default, but you can set
it up so it will be.
And remember the infrastructure, as I put it.
You
can't forget that all of this stuff is being deployed on
an infrastructure of some sort, and that infrastructure better
be secure. I mentioned SSH. People keep
allowing SSH into their environment. Be very
aware that your infrastructure itself should be secure. And I will
remind you later about that. All right. Here's a quick picture.
Picture is worth a thousand words, right? If you look at our first example,
this is our monolithic operating system here. We see
hardware and operating system. We run apps. Here we see a VM. So we see
hardware operating system. There's a hypervisor involved.
And then there's going to be virtual machines. But when you have virtual
machines, you have additional operating systems, you have additional
libraries. They're very much isolated. I should have used a different
picture where there's a big line through it that
shows, yes, these are isolated and so on. But
that should not have happened. Nope. Go back.
Sorry. Okay. But when we look at
using Docker or Container, we go hardware operating
system. There is a container runtime. There's no real dividing lines.
And then we see containers. The only thing that is unique between
the containers is that bin library, the things
that are going to run in the application. So there's not a lot of
isolation there. And people want to
think that there is. But there is not. So please keep
that in mind. I do love this
comic. It came out a couple of years ago. A great
defense is, you know, we'll inundate them
with quarterly Kubernetes releases. Yes, they do update
Kubernetes a lot. That's why you have to pay
attention to your environment and the tools that you're going to use
so you can do upgrades to your Kubernetes cluster. So let's bring on
the orchestrator. Let's talk about Kubernetes. All right. Here
is Kubernetes 101. In one slide, I'm going to teach you
at least the basics of how Kubernetes
works. We have a master node. The master node is
critical. It manages all of the Kubernetes
worker nodes. It manages the clusters. It deploys
pods. Think of a pod as a collection
of containers that are grouped.
A pod might be the way that you
decide, oh, I want to have a pod that supports a particular
client. Or maybe it's a WordPress instance. And that
pod has WordPress. It has a container, a
separate container for the database and so on. So
the pod is how you can group things that
are various containers into certain groupings.
And those groupings are kind of defined by you. But stop and think about
how you want to deploy pods. It's going to be important.
A worker node. The node is where all the containers are going to run.
You have other Kubernetes components that have to
run there. But the important part is, this is where the containers
run. If the worker node is not secure
and not patched, and I can get into that
underneath the visibility of
Kubernetes, then Kubernetes has no idea that I've compromised
a node. I found numerous Kubernetes sites around
the Internet where there's
additional applications running on worker nodes because they've been compromised.
Of course, they're Bitcoin miners. But the master node has no
idea that it's running on there because they got in via
SSH and they launched the application, the
miner, underneath the visibility of Kubernetes.
So you really want to lock down these worker nodes. A pod,
I already explained, it's a unit of deployment and it's also how
you address things from Kubernetes. And a pod
will have an IP address. So that's important to
remember. But it's going to contain one or more containers.
I like to think of pods as, even though it might have
its own IP address, but think of it like a VLAN in a way.
Because I've isolated it in one way. Services
are various things that run within Kubernetes
that communicate, that balance, that do a lot of work. We're going to look
at services coming up. There are various components that are
used. Three that are critical, the API
server, you're going to have kubelets, and you're also
etcd. The one concept you should take
about etcd is this is the database
that describes and builds everything
that is your Kubernetes cluster. The etcd
environment should actually be walled off
and firewalled and controlled very, very
securely. If you do that, you will have a much
more secure Kubernetes cluster. So keep
that in mind. And you can see it here by looking at a picture.
So we take all of those concepts that we just talked about
we see here's an ABI server, here's a controller,
this is all part, you know, the kubectl...
Stop doing that. Come on. Kubectl
is the command you use to control everything.
And even though this is a disk kind of a representation
of etcd, I like to say, you know what, this needs to be off
on its own, firewalled, and separated, and it can be done that way.
Here's your worker node, and you can see how you might have multiple
pods on a worker node. This pod might be talking to
this pod. It doesn't necessarily have to talk to that one, or
they may not talk at all. These are some of the other services that are running
the proxies. This is where we are exposed to the internet,
not over here. Okay? So the only place you're
exposing Kubernetes is at the worker nodes.
Now, Kubernetes is not perfect.
There have been problems. There was...
I need to update this slide again, because there have been a couple of other
Kubernetes. These are more for reference. One's a
little older. This was back... there was the backdoor that involved the
API server. There were some others. These are only for your reference,
so you can go look them up and see. And there are
more. The whole point I'm trying to make here is
Kubernetes is not secure by default, and it's not
perfect. Well, let's try to make
it perfect. Let's take a common Kubernetes cluster
and say, you know, if we were to threat model this,
and that is very, very important, first thing
we want to do is to say, TLS everywhere. Yeah, I know, I have
too many exclamation points, but I'm trying to make the point. If
everything within your Kubernetes cluster is using TLS
and is encrypted, then it's that much
safer. I still have engineers come to me and say,
our application is completely internal. Why do we
need TLS? It's never going to be exposed to the internet.
I said, really? So what happens if Joe's application
over here, which is exposed to the internet, is compromised, and then
they use that as a pivot point to gain access to this environment,
and then they compromise your system? O
is typically the response. So I can't say it
enough, harden the bloody infrastructure. Yes, I'm saying
it again, but you're running this thing on your infrastructure,
make sure it's hardened. And very important,
two concepts. There's RBAC, which is
regular role-based access control. Set that up.
Make sure you're doing it right. Disable
ABAC. ABAC is
incredibly difficult to manage. You
will spend days and hours and hours trying
to do ABAC access controls,
which is more granular, but it's not worth the effort.
Make sure you use RBAC and manage it correctly.
And I'm going to show you a couple of tools coming up
on how to check it. Also, check out this blog
entry. I mentioned this previously about
monitoring. This is one way to audit
Kubernetes, so keep this in mind. Take a look at that
blog entry from Wazoo, and it's an excellent
place to start when it comes to monitoring
your environment. Also, kubectl, get role
binding in all your namespaces. You should do this
periodically in your environment, especially
if you have engineers deploying their own
applications. Make sure that all the namespaces
have good controls in their RBAC accounting.
Also, use third-party authentication for the API
server. Again, something like HashiCorp Vault. Yes, I love
that tool. If you are not familiar with it, get familiar with
it. It's a great tool, and it integrates very, very well with your
API server and Kubernetes.
I already mentioned this. Separate and firewall your etc cluster. Make sure
it's protected. That is the keys to the kingdom.
It's the entire configuration of your cluster. And rotate
encryption keys. Yes, use Vault. And by
default, turn on security policies in the pods.
It's some simple lines of configuration
in the YAML files. It's not hard to do. Or
use Docker Slim, that tool that I mentioned earlier, because
that will help you set this up. Alright, let's talk
about risk. kubectl, get secrets. That
is the one that I just mentioned where you can look at the secrets
from pretty much everything. But here's some tools
that I absolutely love by CyberArk. One is called Kubernetes
RBAC audit. The other is called KubeScan.
The RBAC audit does exactly what you think it does. It audits
all of your RBAC roles. The KubeScan
will look at all the permissions that are granted from
the roles. These are two great tools that
you want to put into your toolbox for securing
Kubernetes clusters. Because remember at the very beginning, I said
it's all about configurations. It's the number one
area that Kubernetes clusters get
compromised. Also, we want to take a look
at GitHub. Go out and make sure
that... take that away... go out and make sure that
your own configurations are not being
shared up to GitHub by your engineers accidentally.
How many times does this happen? It happens.
I've had it happen by engineers. They don't do it on purpose.
But they upload code. And in the code are secrets.
Oops. Oops is not a good thing.
Also, you need a locked down secure image repo.
And create gold images. Don't just let your
engineers go out and do, hey, docker run
bring down Ubuntu and pull it from the internet. No.
You need your own repo and registry for your docker
images. Make sure that happens. And please secure the
libraries. That's another big one. We also...
I need an entire slide on just the applications.
I'm going out on a limb. This is my personal opinion.
But I want to say that nine times out of ten... okay,
maybe eight and a half. Eight and a half times out of ten
Kubernetes clusters and containers are compromised
because the applications themselves
were poorly written and not tested.
And that's what gets me. I have
environments where I see the engineers create
an application. They do so much work to lock down
Kubernetes in the container, but they do a poor
job and there's a SQL injection vulnerability in the application directly.
So then the application is compromised
and then the container is compromised because, oops,
it was running as root and so on and so on. Remember the dominoes.
Everything falls over. Also, remember your
pods. Another common
mistake that people make when they're new to Kubernetes
is they don't stop and think, how can I
group things into pods and secure those pods?
At least if a container falls
within a pod, then maybe there's a couple more containers
in there. Those containers that get compromised are going to be somewhat
limited within the pod itself. Typically
a pod is not going to fall over when another pod falls
over. It's usually containers within the pods. So you want to think
about grouping things. Also, Jason mentioned
this in his talk earlier, but showdown. If you have
never used showdown, get familiar with it. Even
for Kubernetes. You can go out to showdown. You can start
searching for Kubernetes. You can search for API clusters
or API servers. You can search
for etcd servers. And if you find one of yours,
oops, but they're out there.
I did a search this morning for some etcd
exposure and
there were over 6,000
that I found that were exposed. I don't know how many of those have
vulnerabilities that would have allowed me to compromise them
because I was just curious what the number would be.
It goes up and down on a daily basis, but you should be
aware of that. So use showdown. Let's talk about
the configurations. I've said this before.
Most of the breaches come from configuration issues.
Use CIS hardening. Center for
Internet Security. It is an amazing place. It is done by
volunteers and workers within CIS. You can get
benchmarks for Docker and you can get benchmarks for Kubernetes.
You can use these to configure your environment
so they are indeed hardened.
Stop ignoring them. I still see people
that tell me they've never heard of CIS and
it makes me very sad when I hear that because it is such a great
resource. The whole point here is
we have to harden the environment from the
standpoint of the basic configuration, but don't forget
the application. Also, I know a lot of people
have environments that are CICD when it
comes to Docker and Kubernetes. That's why we have them. It's for
speed. It's for development. It's for putting things out
quickly. Okay, great. A lot of people use Jenkins.
Is there some way to automate
all of this so at least I can look at my images,
I can look at my containers, all within the CICD pipeline?
Yes, there is. There is a great tool called
Anchor. Anchor, as it says here, is a
container image scanner plugin. And what it does is
you put it into the pipeline. It's a plugin for Jenkins.
And it actually does security scans against the images
that you're trying to deploy. This is a great way to
get testing in before you actually
deploy it to production. So let's talk about some
additional tools before we get into the pen testing. Because we're still trying to
lock it down. KubeBench. We want to run KubeBench.
What it does is it runs against your CIS benchmarks.
It displays
a configuration that shows you what has passed, what has failed. I'll show an example
in a minute. There's also KubeAudit. It's a little more granular
than KubeSec. And if you run KubeBench
and KubeAudit, it will give you a report that shows you
oops, my system is insecure. Here's an example
of a report. And when we look at it, we see allow
privilege escalation is not set. There's a very
first one. Oops, that needs to be turned on.
Remember, that was the one that I mentioned that allows users
to do privilege escalation. We don't want that. Oh, crap.
We have run as non-root is not set. We don't have a read
only file system set. There are so many
simple things that can be done. Again, no app
armor, no security comp. It's not being turned on.
These are things that are just so easy. And this is from every
single container within the environment that is
being deployed under this particular. It's much
longer. It goes into it. This is just a quick snapshot.
The whole point here is you run this against your environment
before you deploy it. And you go, oops, let's go back to our
configuration file and fix all these things.
Here are some more tools for you in the environment. Claire,
another great tool for doing static analysis of
applications. Claire does, it integrates
Claire and your Docker registry so it can automate that
a little bit. Falco, I need to put huge stars next
to this one because this is actually one of my favorite, favorite
tools. It was developed years
ago and then it kind of morphed off and became falco.org.
It monitors behavior within containers.
It can detect if a container has been compromised
because suddenly the container starts acting weird.
Or there are commands being executed inside the container like
cat etsy password. Probably not a command
that should be showing up in one of your containers.
If anything of this entire talk that I'm
giving today, you get out of this, go look at falco.
This is one of the best tools out there. Kubehunter
is another great tool. We're going to talk about that.
It's used more so for looking at doing some pen testing.
We're going to take a look at that in just a second. Alright, ready, set, attack.
If we were going to pen test an environment, this is an
example of all the ports and all of the things that are
open in a Kubernetes cluster. Now, they're
open, but are they open by firewalls and so on?
Well, we don't know. The API server
itself is, yes, it's HTTPS.
It better damn well be protected.
etcd on 6666, that should be protected
wherever it happens to be. This just gives you an idea. All of these
ports can be changed. These are all the defaults in most cases.
Just be aware of the ports that are out there and
exposed that you need to make
sure are protected. Because as an attacker, what am I
going to do? Well, one of the first things, even Jason was talking about it in his talk,
looking at ports, running nmap, running mass scan, and so on.
This is exactly one of the things that
Shodan does. It looks for all of these things that are exposed on the
internet. So please make sure you're aware of all the ports
within your cluster. Now, I are a hacker.
See me hack. This is one of the
ways that I'm going to start to attack
a Kubernetes cluster. I want to first
and foremost check access to the API server, make sure that is
protected. I'm going to be looking at etcd,
and making sure that's protected. If I find it exposed,
I'm going to get keys to the kingdom in that case.
What about the kubelets that are communicating and the command channels
that are there? They should be read-only, but
they should not be exposed. There was actually
a vulnerability exposed
about this, I want to say two months ago. I'm probably off in that
sense. But take a look at that. And it allowed, even though it's a
read-only port and it's supposed to limit what you can see,
there were some vulnerabilities that allowed for a lot of exposure there.
Also, container vulnerabilities.
What about, you know, do you have files
in that container that the application needs that
you should be protecting? Well, that's where I would set up Falco and tell
Falco, hey, there's a special file over here. This application
is using it. But keep an eye on that. Make sure that the only
thing that is reading that application is, or excuse me,
that is reading that file happens to be the
application. And Falco can monitor that.
Privilege escalation. Nine times out of ten, once somebody
compromises an application in a container, they
elevate privilege because there's some other bugs in there.
Go back to the Kubotic slide where
it showed privilege escalation was not
set. All you have to do is set that configuration
variable. It's not that hard. Okay?
Vulnerable applications are critical.
Can't say it enough times. Especially because there are
so many people using Kubernetes and Docker to deploy
WordPress. And WordPress is, of course, one of
our favorite things to deal with. All right. So your
final threat model, when it comes to pentesting and
looking at the environment that is Kubernetes,
you think of your workload. You think of that
security within your pods. Can I
set up my pods so they're indeed secure?
The container security itself, the configuration
for that container, the more hooks I put in,
run as root, no. Privilege escalation,
no. These are the things that we want to make sure that are
set. Also, file storage security.
Maybe there's a particular file system that has to be read right
but the rest of the file system can be read only. Those are
other things. The network security. Remember the infrastructure.
Put firewalls in. Put controls
in there that detect lateral movement. If
somebody does compromise one of your applications,
make sure you have something monitoring. I gave you that
slide way back about the blog entry from Wazoo.
They talk about how to audit and monitor
a Kubernetes cluster using their tool. I love the Wazoo
manager and agent. Take a look at it if you haven't heard of it
before. Remember, most
breaches are not zero day. I can't say it enough times. Everybody's always
trying to protect against this stuff. They're not fancy.
Most of these things are being broken into because people
make silly mistakes. Because remember, common sense.
Whatever happened to it. The breaches are going to come from configuration
issues. The kube audit was a very good
example where it was showing all the problems that would allow someone to
get in. Compromise credentials.
Make sure your credentials are stored in a secure place
somewhere. And don't allow for
privilege escalation. Kubernetes and Docker
give you the tools and the configurations to prevent that.
Just turn them on. And that's what we're not doing. We have
to turn on the configurations. So your key takeaways
from today. Common sense.
Please go back and stop
and think about where is the exposure?
Where are my pods exposed? What about my etcd?
Configuration. That is so critical to any
Kubernetes. And always remember containers.
Containers are the linchpin of a Kubernetes
environment. And if I don't secure my individual containers
then Kubernetes... I don't care about
the security of Kubernetes because the container is insecure.
So stop overthinking this stuff and think about
security basics. Remember the CIS benchmarks.
Can't say it enough. Basics.
I still think as security professionals
hackers, whatever we want to call ourselves. We have
forgotten the basics of how to secure an environment.
And that's why things keep getting broken into.
Stop doing fancy crap. The fancy crap doesn't work.
Secure the environment, please.
Use the CIS benchmarks for the containers, for Kubernetes,
for the OS that your stuff runs on. If you're going to
deploy this whole Kubernetes cluster on Ubuntu, then pull up
the CIS benchmarks for Ubuntu. Or
Red Hat or CentOS. Whatever you're using.
Remember, RBAC is your friend. ABAC
is not. Your infrastructure better be
secure. Your pods better be secure.
Containers themselves, the network. Still a lot of people forget
the network because they forget about TLS everywhere.
Threat model
your applications. If you don't take
the beginning effort to threat model your applications,
you end up with insecure applications. And all the security
of your Kubernetes cluster is useless because you have
insecure applications within your environment. Access controls
go along with RBAC, but access controls into the
environment, kind of referencing SSH.
I found one worker node configuration
in an environment probably nine months ago. But for some
reason on the worker nodes, they had FTP enabled on all the worker nodes.
And my head exploded.
It allowed me to get into everything, but oh well.
Yes, did I mention RBAC though?
Enough times. I can't say it enough. Why? Because
people tend to ignore it. Remember the tools I mentioned earlier
about monitoring or auditing the RBAC roles and also
the permissions. And finally,
not quite finally, remember to patch everything.
The infrastructure that goes along with that.
Logging and auditing and monitoring. I still see people
set up their environments. There's that blog entry again.
Please take a look at that. It is well written.
It's a great tool. You'll learn a lot about Wazoo. I use it on
all my systems. I deploy a lot of honeypots with
Kubernetes and containers and things like that. But I monitor them all
with Wazoo. You can't
have a secure environment if you're not monitoring that environment.
That is the biggest thing.
That brings me to the end. I thank you for your
attention. There's contact info. There's my Twitter
account. I will get this to the Red Team
Village and they can upload it to their environment. But I'll have it on my
GitHub later
in an hour or so. I'll update it there. Thank you all very much.
Hopefully you learned something today. Go out and hack and have some fun. Thanks to
the Red Team Village and have an awesome DEF CON.
Awesome. Thank you so much, Kat. Amazing presentation
as always. Thank you for supporting DEF CON. Thank you for supporting the
community. Of course, the Red Team Village. For those of you
that just joined or are looking at
the presentation on Twitch or YouTube, please look at
all the activities that we have going on. Go to the website that is in the description of
the stream. Please also join the conversation in Discord.
We're going to go on a brief break and the next presentation will start in a few minutes.
Thank you again and thank you, Kat.