your support for supporting the DEF CON community
and the Red Team Village specifically.
We're looking forward to your presentation.
So take it away.
Thank you so much, Omer, for introducing and having me here.
Let me just share the screen and confirm.
Can you see the screen?
Yes, you're all set.
Awesome, perfect.
Thank you everyone for joining me.
And first of all, to just give some context,
thank you so much, DEF CON Red Team Village,
especially in this pandemic times to make it virtual
and people can enjoy these talks
without any physical intervention.
And this talk is about the Kubernetes code.
So disclaimer, basically,
today's we will not see any slides, basically.
And it's mostly, everything is in online.
You can play along with me,
or if you can go back and try yourself,
you can go ahead and try as well.
So there will be no slides,
mostly live demos and the hands-on exercises.
Cool.
So this talk is primarily focusing
on introducing a project recently I have created
called Kubernetes Code,
which is, this is in github.madhuvakula.kubernetescode.
This project was mainly intentionally vulnerable application,
which is to basically try and teach participants,
or like anyone come to the new Kubernetes
to understand how you can assess these Kubernetes clusters
and find vulnerabilities so that you can go back
and try it out in your organizations,
or day-to-day pen tests,
which you find are a team exercises, right?
So I have presented at some of the places
and also I have created a playground.
If you wanted to write now,
try it in your browser and follow along, right?
So a little bit about me, myself, Madhu Akula.
I work at currently Zibia
as a Cloud Native Security Specialist.
And I have been speaking and training
at a bunch of conferences around the world,
like DEF CON Black Hat, USENIX, Orally, and some of them.
And I'm fortunate to find some of the vulnerabilities
long back using my research
in some of the organizations like Microsoft,
LinkedIn, Google, and some of them.
So apart from that, my primary interest
mostly working around security of cloud,
containers, Kubernetes,
and automating the infrastructure part.
So that's a little bit about me.
If you wanted to follow me,
you can follow me at Madhu Akula
in most of the social media stuff.
Cool.
Let's get into what exactly is this Kubernetes code
and what we are trying to learn
like within this next one hour, right?
So to give a little context about Kubernetes code.
So as I said, it is intentionally vulnerable.
I think as most of you have come across
the different types of code,
like web code, or Terra code, or some other codes.
So this is a similar project,
but it is mostly focusing on Kubernetes
to understand what are the known vulnerabilities.
And also we have seen common vulnerabilities
in the real world environments,
like while performing penetration testing,
or like while performing assessments
in Kubernetes environments.
From those we have taken as scenarios.
And these are some of the scenarios.
Let me zoom a bit.
I think maybe easier to see.
Yeah.
So these are some of the scenarios I have created.
And we will see some of the scenarios,
exploitation, and how you can bypass
or gain access to Kubernetes clusters
as an attacker point, right?
And I'm really sorry, and I'm shameful.
I said I will keep updating these slides
at the documentation.
And also I have been working on the defense part also.
Hopefully this month we will see them coming away.
This part I'm going to skip.
So before diving into Kubernetes,
if you are really completely new
and coming to the containers or Kubernetes world,
I highly recommend understanding the technology
because there is a fair amount of gap
in between penetration testing
or finding or assessing clusters
if you don't understand what technology you are doing, right?
So we have seen commonly,
especially in the modern tech infrastructure world,
there is a huge gap for the security team
to understand the technology
before even testing or trying out.
Because once you are able to understand,
you can be able to do your best
or you can be able to possibly find
most of the Kubernetes or any ecosystem.
This is a very funny video
and it's a simple children's guide
to explain what exactly Kubernetes is.
So I would highly recommend
if you want to go through these things.
So to give you a little quick context,
how Kubernetes architecture looks like
at a higher level, right?
So this is again from the wiki comments.
You see a server and client architecture
similar to most of the server client.
So one thing you have to observe here is in Kubernetes,
everything which you make, any operation,
it may be from within the master,
which means server,
within the node, which means worker node.
So whatever the operations which will make,
like it can be from the node,
it can be from the developer
or any user, any service
will go and touch something called API server.
So which is one of the key component
in terms of Kubernetes,
because Kubernetes is completely driven
by API first approach.
So whatever you make all requests or something,
everything you will see going through the API server, right?
And control manager is basically to make the decisions,
whatever the requests you make.
And scheduler is ensuring to place the workloads
or the request which you made.
Suppose if you wanted to serve a Nginx server
with 10 pods, it will ensure that
how many nodes has the capacity to schedule.
So the scheduler job is to figure out that
and place them accordingly.
So etcd is one of the key value store database,
which is used by Kubernetes,
which is as its name suggests, key value.
So it stores the data,
which is coming to the API server
and stores the key value base.
And we will see a little bit in a later coming sections.
So apart from that nodes,
when you come to the node,
kubelet is the talking point to the API server
and communicates within the local node itself.
So whenever you wanted to make a request as a scheduler,
you wanted to schedule in the node,
it will talk to the container runtime,
which is within the node,
like it can be Docker, RunC, etcd.
So then it will go ahead and schedule
based on the workload kind of thing.
So kubeproxy is one of the another key component,
which is kind of like IP tables,
like it will try to provide a communication
across like the nodes as well as a cluster.
So let's go ahead and a little bit go deep,
but I would highly recommend you practice the Kubernetes
before you diving into the assessments
or security testing.
So here are some of the references I have curated.
These are pretty hands-on,
like if you wanted to try,
so you can go ahead and try as a literally playground
and also specific objects if you start while testing,
you can click and understand what exactly it is.
So these references I would,
I think it would be definitely helpful to try.
Cool.
Setting up Kubernetes code.
The Kubernetes code is in design in such a way
it can set up any out of the box.
It can be in your existing cloud provider services
like GKE, EKS, AKS, or digitalization based.
And also your locally provisioned
Minikube or another playground.
To make easy of setup for the people to try out and learn,
I have created something called Katakoda Playground.
If you just go ahead and click this,
it will open up a playground for you,
which is completely free and online.
So you don't need anything to be done.
I think you have to register first time.
So then you can get the complete setup here for you
to practice and learn.
Let me put it in a presentation mode
so that you can see a little better.
Yep. Cool.
Cool.
So I'll walk you through how you can set this up
and practice.
And if you wanted to follow along with me,
you can follow this guide at madhuvakula.com
slash Kubernetes code.
This is the URL, right?
Cool.
And if you wanted to set up your own,
you can just follow this, the documentation.
Basically you need to have a working Kubernetes cluster
set up with admin access.
And also you need a Helm version too,
because in one of the scenario,
we will try to focus on how you can exploit
the existing vulnerability in the older version of Helm.
So that's it.
So if you wanted to set up your own,
let's go ahead and do that as well.
So if you have your own set cluster,
you clone the repository,
which is the git clone of the Kubernetes code.
And then you can go ahead and set up the Kubernetes code.
So it checks if Kubernetes kubectl is available,
Helm 2 is available in your path,
and it'll go ahead and deploy the tiller,
which is the server side component of the Helm,
and a bunch of vulnerable services,
which will be used as practice to exploit
or try practicing Kubernetes security.
So if you are someone following the Katakoda live,
you can use this playground,
which is completely free and online.
You can go ahead and do the same thing
in the playground as well.
So the playground is designed in such a way,
you can go ahead and see here only the guide
in the second, in the specific tab.
And also you can browse those ports and other stuff
within the browser itself.
Even in your mobile, also you can try this.
So it takes a while to set up
the entire Kubernetes code platform.
I think it's almost done.
So it creates a bunch of vulnerable scenarios,
which I have pre-created.
You can go through the code
and you can able to see what exactly it is trying to do,
if you wanted to look at it.
So this is nothing but it checks
if kubectl is existing or Helm 2 existing
and it sets a bunch of emails,
like which is nothing but deployments in the cluster.
Cool.
Now we have the cluster up and running
and we have Kubernetes code set up.
So what's next, right?
So I think it looks like here also it is done.
Ensure you check all the ports are running.
You can go ahead and run kubectl get ports.
It is currently container creating,
which means it's starting.
It takes a while to set up the,
depends on the way you are running the Kubernetes cluster.
So let me go ahead and do this as well here,
kubectl get ports.
I hope you people can see the screen.
I think, let me zoom a little bit as well.
Cool.
So you can see a bunch of them.
Most of the ports are created.
I think just only one port is waiting to get created.
That's awesome.
Now you can see all ports are running
and I think this is intentional.
This is a job and get company.
So if you see the status,
basically your entire Kubernetes code is set up and running.
So now it's time to hack and get into the cluster.
Bye.
Cool.
That's where we go back and refer to the documentation.
So if you look at the scenarios,
so there are a bunch of scenarios we have created,
as I said.
So for the time being,
for the entire today's presentation,
we will not be able to look at all of those scenarios,
but definitely we will be looking
at some of the primary scenarios,
which might be interesting and fun to try out.
And I would highly recommend go back and try out yourself
and check out more other scenarios as well, right?
And in some cases, especially not all of them scenarios,
but in some scenarios,
I try to create a flag in this format,
k8sgoot-goot.
And some MD5s, basically it's a number of 32 characters.
Then if you reach this kind of flag,
basically it means end of the scenarios,
but it may not be found in for all the scenarios,
but I try to keep for certain scenarios
to reduce the effort of like going on beyond post-exploitation.
Cool.
So let's go ahead and start with the scenario one,
sensitive keys in code base.
So I would recommend before doing that,
I think if you are setting up locally
or in the same place, all ports are running,
make sure you run bash access Kubernetes code.
What it does is basically put forward
all the ports or services into your local system
so that you can access locally
without even exposing to the world, which is internet, right?
So now if you go ahead and click this,
basically you can see the homepage,
like which is the Kubernetes code homepage, right?
So this is where you can see when it's good, it's running,
you can follow the guide here.
And also if you want to check out each scenario,
the scenarios are here.
So let's get started with the worst scenario,
which we will look at one of them
is called sensitive keys in code basis.
So the Kubernetes code is designed in such a way,
if you wanted to practice as an attacker or learner,
you can go ahead and try this scenario
by clicking here and starting with attacking.
Or if you are someone who wanted to learn
and practice yourself and follow some kind of guide,
we have created the documentation.
The solutions will be here in the same guide
for each scenario.
Like you can follow them as well here
if you're stuck somewhere, right?
And some scenarios we try to create
two different methodologies.
It can be tagged in different ways.
So we try to document in the same way.
So if you see here,
let's go back and read a little bit about the scenario.
Developers tends to commit sensitive information
to version control systems.
As we are moving towards the ACD and GitHub systems,
we tend to forget identifying sensitive information
in code and commits.
Let's see if we can find something cool here, right?
So by looking at, you can see that some kind of vulnerability
or some kind of mistake has happened.
Now, as you can read this,
this is version control using Git.
So what I can do is I can try as a DartKit
and you can see the default file in the Git will be config.
Now you can see that there is a bare repository
hanging there, right?
And by the way, these vulnerabilities which you see,
I was showing like in the web vulnerabilities,
these are just an entry points
into getting into the cluster.
In real world,
it may not be the same vulnerabilities you find.
Maybe there is a different approach
you find to get in here, right?
That's the thing.
So now I can see that there is a Git config.
Let's go ahead and see how I can clone
this repository locally
so that I can understand this code base better, right?
So if you want to start,
basically you can go back and follow this repository here.
So there is a tool called GitDumper.
It can be any other tool as well.
I just take an example.
So you can try this tool by running a Python3-GitDumper
localhost and save it like this.
Maybe I think I have configured here.
Cool.
GitDumper.
So what I'm trying to do is I'm cloning
the localhost1230.git folder into local system.
So what it is trying is it is going to fetch
all the repository and clone it locally, right?
So now if you go here, kdiskcode.git,
so it is the Git repository which we have.
So if you see here,
we can see there is a Golang bunch of files,
main.co, and a bunch of custom code, right?
And also you can see Git, the folder.
So you can also look for what are the old commits
which are happening, right?
As developers sometimes make mistakes,
by the way, this scenario is not only focused
towards containers or Kubernetes.
In general world, we have seen these kinds of mistakes
in most of the modern software development,
especially in cloud deployments
or any internal deployments, right?
So I can see some of the bunch of commits
as you attack and you can go and check manually,
like Git checkout to specific, the log.check commit.
And also you can see,
oh, you can see an interesting file called env, right?
So there you go.
So you can see there is the access key and secret key
for the AWS has mistakenly committed.
And you can also see there is the kdiskcode flag is there.
So basically this means that you have reached
the end of this scenario,
but in real world you can use this AWS access key,
secret key and get the session token
and you can do post-exploitation and go along beyond.
So these scenarios are just we have taken
from what we see in real world as a pen testing engagement.
And some of them are like in a common scenarios
which we see in real world, right?
So this may not be possible
and this may be possible in different use cases.
So feel free to give it a try
and understand what exactly we are trying to achieve here.
So basically what will happen is in the same use case.
So we can try in different method.
Like what I'm trying to do is most of the time
developers mistakenly build a Docker images
which may be committed with the .git files
because they don't ignore these files
when it is creating into Docker container,
maybe Docker ignore or something.
So which might end up pushing this data
into the containers as well.
So looking at these kinds of sensitive information
is very useful in the container.
Like the another methodology you can try,
as I said, explanation.
So you can get the pod name, you can exec into the pod.
This is like another method.
Now we can also use tools like TruffleHog.
So TruffleHog, I think most of you know,
this is a tool which is basically finds high entropy
or sensitive secrets in a Git repositories.
So now I'm trying to do TruffleHog
and given the current directory
is the where Git repository is there.
So what it does is it will check through all the commits
and finds, is there any high entropy values
or like sensitive secrets found
based on the regular expressions and return to the studio.
So you can include these kinds of tools
in your CI, CD systems
to detect these kinds of vulnerabilities.
But in attacker's point of view,
if you find any of the code basis
which has Git or something,
maybe it's a really definitely good way to check out
in especially the container environments,
you might end up finding some kind of sensitive approach.
Like you can see the same thing.
Cool.
So this is our first scenario,
which we just tried from the Kubernetes code.
And similarly, as I said, you can try out here as well.
So what you can do is you can look at here
and if you wanted to specifically browse specific code,
like one, two, three, zero,
in the custom code tab,
you can just open in a new browser window also.
If you press this icon,
then you can just say,
I wanted to browse one, two, three, zero code.
So you can able to browse the same thing
and play it in online yourself for free
and understand what exactly you can assess
or test for Kubernetes security.
Cool.
So that's kind of one of the simple scenario.
So let's go ahead and see what could be
the some of the complex scenarios
or different kinds of vulnerabilities
you see in Kubernetes security assessment.
So let's go back to the some maybe a little down,
especially like attacking private registries
or Kubernetes CS benchmarks or container escape.
So let me go and showcase you
how container escape to host system works.
To browse to this scenario, basically you navigate here.
So this is basically one of the pod
inside Kubernetes cluster running
with some specific privileged access has, right?
So to set the context here,
most of the times, especially operations teams
may tend to require to run specific containers
with high privileges,
maybe giving gaps is admin or host privileges
or privilege context,
which is a full admin, sysadmin capabilities.
So these are things.
And also to just give an example,
how we found in a real world,
I have seen, especially while we performing
a penetration testing engagement.
So most of the developers get an access
to completely one namespace with a full privilege.
So if your clusters doesn't have PSPs,
which is nothing but pod security policies,
developers can basically has full access to that namespace.
Whatever they can do,
they can deploy a pod with specific privileges,
which they can gain into the whole system access, right?
So also this is one of the use case
we have taken to simulate or showcase
how you can find these kind of container escape
and gain access to the whole system, right?
Cool.
So what we are trying to do here is
we can try and look for some bunch of information.
Most of the time, look for a printenv.
Sorry, I think I typed print.
Cool.
So most of the time,
you might see some kind of stuff here as well in printenv.
And also I'm checking for mount.
Is there any mounted directories?
Now you can see there is a mounted directory
from host system and which is into the container,
which means basically in the pod,
in this Kubernetes case, right?
So now what I can do is,
I think,
let's look at what exactly the host system is doing.
So if you see, there is a host system mounted,
like which is basically from the host system,
to just give an example.
So which is nothing but your underlying host system,
which is node, is mounted into your pod as a volume, right?
So due to the reason we have extra capabilities, right?
Like this container has a bunch of extra capabilities
as well.
So using those capabilities,
as an attacker, what we can do is,
we can like try to gain some kind of access
into host system.
What you can do is, we can sage root on the host system
because we have capability of the privilege.
And now we can see that it is given
that the host system sage root.
And now what we can do is,
we can perform the internal commands,
which are specific to system, right?
Like now you can see docker ps.
So these are nothing but running inside the host system,
which is not a container.
So you went below one level from the pod
to the host system.
From there, you are executing commands, right?
So the possible attack surface here you can do is,
to do post exploitation might be basically
using the host system configuration,
which is nothing but node level Kubernetes configuration,
to go and try attacking other nodes
or to gain access to other nodes,
which basically you're controlling the entire cluster
by passing through, passing through, right?
So for example, what you can do is,
you can use the configuration of node,
which is available in the cat var lib.
So you can use the configuration of cat var lib quip config,
which has the node level configuration.
So you can use that configuration
and perform the Kubernetes commands,
which is not only available
for the existing namespace specific,
it can also perform at the node level,
whatever the node has the service account,
you can able to do that level.
To do that, for example, you can do this command.
A kubectl command is not available.
Let's download kubectl.
Let me download this kubectl here.
It just takes some time.
Hopefully it is faster.
Yeah.
Yep, there you go.
So what we are trying to do is,
we are using now kubectl command
and passing the kube config flag,
which is using the node configuration
and getting all like whatever the all resources
in namespace of kube system, right?
Oops, I'm trying the same thing again.
I have to do chmod plus x kubectl.
Cool.
Now you can see when I run kubectl kube config
with the specific configuration
and get all in kube system,
it will go ahead and execute that command
and get all the resources.
Now you can see here,
it can able to get all the ports
which are not restricted to specific namespace,
which is the developer or the user has.
And you can able to explore more
or access more resources and also services.
And you can see some of the specific use cases,
especially I think replication controllers
and other jobs and secrets,
you cannot able to access.
So the node, whatever the access it has,
the node configuration,
you can able to perform same level.
So as an attacker, you can do post-exploitation,
basically deploy another port in a different node
and you can able to pass to another node
and again get access to that.
So for that, you can just look for how many nodes you have,
like currently two nodes, maybe you are in this node,
then you can deploy another port in the different node
and get again access to the different.
So you can basically use this kind of hopping methodology
and hop on to another node and gain complete cluster access.
So this is mostly you will see if an organization
or the company which is not running up
the port security policies,
you can able to perform these kind of attacks
or escalations of the container escapes, right?
So you might use BSPs as a defense
for these kind of vulnerabilities
and it would be an easy and simple way
to fix this kind of exploitation.
Cool.
So let's go ahead and see another use case.
I think we have some more time to see two other scenarios.
Let's take an example of Kubernetes namespace bypass.
Okay.
I'm going here, Kubernetes namespace bypass.
I don't know exactly.
Yeah.
So by default, Kubernetes uses flat networking scheme.
I think this is one of the default feature of Kubernetes.
So by default, Kubernetes uses flat networking scheme
means I can able to talk to from any port to any port
by default, there is no firewall rules
which will happen, allow or deny.
By default, anything can talk to anything
within the cluster IP range, right?
So which means that whenever someone,
SRE or ops team or like security people says
that there is a namespace we have created in the Kubernetes
which means it's just a logical boundary.
There is no way you literally segregating
between the different namespace
unless until apply some kind of security policies
like NSPs and network security policies, right?
So if you are testing a Kubernetes clusters
where there was no NSPs
and there was no restrictions between the namespace
as attacker, you can able to talk to other namespaces
and other services within the cluster
by using a standard network security tools
like which we use day to day,
like Nmap or Zmap or a bunch of tools, right?
So to just get started with the scenario,
I have created something, an example.
So what we are trying to do is
we are running something called Hacker Container
which has again created by me.
I have while performing bunch of Kubernetes security testings
I thought that there are so many tools we require
especially focused on container environment.
So I have created curated list of tools
which are maybe 50 plus tools
which will help you aid in
while performing Kubernetes security assessment.
So this container has those all tools
like while you are performing Kubernetes security assessment
and you can use this as like a Swiss Army knife
in security assessment, right?
And I have seen this has been very valuable
while performing assessment.
So definitely I'll try to create
some kind of documentation out of it.
Maybe hopefully you can go back
and try out different use cases with Hacker Container.
I think it's going and deploying there and cool.
So we got a shell.
So what we are doing is we are inside a pod
in one of the default namespace.
Like if you wanted to just see,
let's go ahead and run some command
called kubectl get pods.
Now you can see Hacker Container
which is running in a default namespace.
And let's go ahead and see what else we can do
from the shell, right?
So as attacker, you are in one of the pod
and you don't know what else you can do from there, right?
So as I said, Hacker Container has a bunch of tools
like Docker Bench Security, Kubebench, Linus
and a bunch of other tools.
So one interesting thing you can do especially is
looking for the entire cluster range.
As I said, by default, Kubernetes uses flat networking scheme.
You can look at the entire cluster range, right?
So you can do IP route.
You can see this is using 10.0.0.0 network range.
And also you can look for what is the current IP address
and 10.0.0.0.12.
So you can scan the entire network and identify services.
So one thing I can confirm, I can assure
is most of the times internally within the cluster
as a microservices or services
which are running, many of the developers
or operations teams tend to forget
or assume that it is only exposed to internal networks.
They will never use authentications or authorizations
and you might end up seeing some kind of bypasses
which is exposed internally itself,
especially Mongo services, Elasticsearch instances
and also even some microservices
because the API gateway terminates at the ingress level
which means before exposing into the world
like using load balancers,
they will terminate at the ingress level.
So you might end up seeing some kind of services
especially the application services
also don't have authentication authorization checks
within the clusters.
So this is a really great place to explore
and do as much recon you can do
and you can gain as much as sensitive information
or services access.
Cool.
So what I'm trying to do is here I'm exploring same
and what I did is I'm trying to use ZMap
which is similar to Nmap.
It's trying to do a scan for the entire cluster range
which is to do a faster way.
And I'm specifically looking for a port of 6379
which is a Redis instance.
So I assume that there might be a Redis running
or like a 9200 you can also try
which is Elasticsearch or Mongo or MySQL
or any other services you can scan.
It's like traditional security will apply here.
Let's go ahead and run this ZMap
and see what we can see.
So, oops.
Okay.
So we have to have enable the internal network range
allow within this list.
Let's go ahead and do that.
Let's comment this and run again.
Sorry.
Oh, okay.
You have to do another thing which is specify the gateway
where this ZMap is starting.
Let's go ahead and do that as well.
F1G capital.
Okay.
And what is the gateway?
Yeah, this is the MAC address.
Cool.
So, yeah.
So what it is doing,
it is trying to scan the entire cluster range
for the specific port 6379
and results the output if any of the IPs running
or services running or what is running
with the specific port open,
it'll save into the other results.csv.
It take a while.
I think it's pretty fast in this case,
but this will take some more time to go through and scan.
But as I said, in general,
we have seen in real world a lot of services running,
especially Elasticsearch instances
where you can look for without authentication
and where you can gain access to indices
and the log information,
especially audit logs and sensitive information
and also Mongo instances without any authentication
or also the microservices, right?
So this is a pretty goldmine if you are inside a pod
and without NSPs,
network security policies applied within the cluster, right?
Cool.
So this will take a while.
I think 76% and all stuff.
Yeah.
As I said, similarly,
you can try out those vulnerabilities here
and follow the guide if you wanted to do as well.
And I think it's a simple way to try out
this entire Katakoda environment within the browser.
You don't need to set up the entire cluster itself locally.
And also once you are done,
you can just quit the browser.
It's all done.
You don't need to clean up the entire environment.
And another disclaimer,
I think I forgot to mention in the beginning,
do not try to run or set up this Kubernetes code
in your production environments or in your office
because it has intentionally vulnerabilities introduced
due to the application flaws or something.
It is a vulnerable application,
which may have gain access to your production
or wherever you run these environments.
And at least I try to take care of using,
exposing the ports within to the localhost,
but it's highly not recommended to run in production.
Cool.
We have ZMap completed.
Let's go ahead and see what we got from the results.
You can see that there is the IP,
which has 6379 port is exposed within the cluster.
And let's go ahead and see what we can get out of it.
As you know, it's Redis.
As I said, this hacker container has built-in tools
to look for these kind of common attacks and all stuff.
We can use Redis CLI,
and I forgot to be honest, how to access.
Yeah, I found H and the IP.
And now you can see there is another integrated Redis
running within the cluster you can able to access
because there is no network security policies
which is by default in the cluster,
means you can able to access without any issues.
I mean, so they don't have any allow list
or denial list within the cluster.
Now you can do enumeration standard like get key.
You can see nothing.
Okay.
Keys.
You can see the keys, what are available,
and you can see there is a key and get secret stuff.
Cool.
So you reach the flag.
That means you are done with the challenge, right?
So this is just an example of try to simulate the scenarios,
but definitely you see these kind of vulnerabilities
in a real world.
Once you end up in a pod,
try to look out for more services and try as much as you can
and you will get most information out of it.
Right.
Cool.
So let's go back and see another scenario
before we conclude the presentation
and moving into the next sections.
Let's look at one of the common oldest,
the vulnerabilities which we have seen
exploiting the Helm or Tiller.
So Helm is like a package manager for Kubernetes.
So how we deploy application, install applications
into, suppose you are using Debian-based
apt-get install the package name.
Similarly, developers or operations teams use Helm,
which is a package manager to deploy applications
into Kubernetes, right?
So they will just do Helm install and the Nginx,
it'll go and install the Nginx within the cluster, right?
So by default, if you are testing any cluster
and which is using version two,
which is the older version of Helm
and it used to be by default,
most people using is Helm two in a previous cases,
but they have raised three recently, like someone's back.
So by default, Helm version two has RBAC setup
with the full cluster admin access.
So if you end up seeing these kind of Helm two configurations
running within the cluster
and there are no network security policies applied
and it is running with the default setup,
basically using this misconfiguration as an attacker,
you can get entire cluster admin access
and form the entire cluster, right?
Let's go ahead and see how this scenario looks like.
I'm going and running a simple pod
and executing into that, right?
Cool, let's go ahead and run this command.
It'll go ahead and spin up another pod,
it is called HelmTeller and give you a bash access, right?
Yep, so now we are inside a pod
and we assume that this cluster has Helm installed
and we wanted to see how we can exploit
this kind of vulnerability in a real world environment, right?
So let's go back to our documentation, right?
So by default in Kubernetes,
there is a schema which they follow
to access or basically it's like a DNS.
Similarly, how we use, they use in the Kubernetes.
So this is nothing but a service name,
this is the namespace and they use either a service name
and namespace and a cluster.local.
So this is how the structure which is followed, right?
So by default, if you have installed the Helm,
which is Teller is get deployed in version two,
which is nothing but a server side component of Helm.
Helm is the client side version
where you do Helm install or something.
So the Teller is the server side component,
which has a full cluster admin access, right?
So now we are doing Teller deploy is the service name
and it is by default deployed in a kube system.
So I'm trying to look for kube system
and I'm taking Telnet and see if this port
is able to access from the current port.
So this is the default port,
which is in a by default configurations
exposed to 0.0.0, which means if you are in any one
of the four port within the cluster,
you can able to access the Teller deployment service, right?
You can see, looks like you can able to access the Teller.
So which means you can able to talk
to the Teller deployment service
because there were no NSPs and by default,
it is exposed to 0.0.0, which is within the cluster, right?
And let's see what we can do.
So I have also installed a Helm
within this same container to talk to the Helm.
So in the Helm version two,
you can specify the dash dash host flag,
which means you can specify which Teller service
you wanted to talk to.
Now we know the Teller service name
as well as the DNS basically,
and we are specifying talk to this host
and perform these operations, right?
Let's go ahead and run this command.
I'm trying to run Helm command
and specifying the specific service
and the port,
and I'm just running Helm version command, right?
Now you can see the client version,
which is Helm, is 2.16.7,
and the server version, which is Teller, is 2.16.9.
So I can able to talk to the Helm service
and I can also see what all charts deployed,
Helm charts, right?
Helm-less, like you can see metadata DB is deployed.
So you can create your own chart and apply as well, right?
So what we have done is, as an attacker,
simply to showcase the complexity
or what you can do as an attacker,
let's go ahead and run a simple command.
By default, Kubernetes stores,
has a service account,
which is stored in var lib,
service secret,
one,
yeah, secrets,
Kubernetes IOS service account,
namespace,
because I'm running in default namespace,
and this is the token for that specific service account,
which is mapped to the current port.
So what I'm trying to do now here is,
we have created a Helm chart,
which is one chart,
and this I have taken from the Bitnami repository research,
and I have given the reference here,
and go and check more resource.
What it does is basically,
it's a simple Helm chart.
What it is do is,
it has a bunch of values,
which is nothing but namespace is default,
and the name of the,
is default.
Let's look at the templates, how it looks like.
Templates,
and a cluster role.
It has a cluster role,
which is nothing but,
it gives all API groups in the Kubernetes,
for all the resources,
and all the verbs,
which means you can do get,
post,
delete,
list,
all of the resources,
for any of the resources in the Kubernetes cluster,
any of the kind of API groups,
you can perform.
So what we are trying to do is,
templates,
cluster role binding.
So we are trying to bind this cluster role,
which is entered across the cluster,
to the service account,
which is taken, if you've seen,
which is namespace is default,
because we are in the,
this pod is currently running in the default namespace,
and the service account is default,
because you see here,
it's the service account is default,
when you do,
slash var run,
secret service account.
So what we are trying to do is,
we are creating a new cluster role,
which gives full cluster access,
to perform anything on all resources,
and binding that cluster role,
to the default service account.
So now, as you can't get the secrets,
or any operations,
within the cluster using the default account,
once we deploy this Helm chart,
you can do anything as a cluster admin, right?
Let's go ahead and run this,
deploy this Helm chart.
So we are trying to do Helm,
and we are using the same Tiller configuration,
and deploying the Pawn chart, right?
Let's go ahead and run.
So it went ahead and deployed the Pawn chart,
because Helm has the capability to talk to the Tiller,
and Tiller has the full cluster admin access,
so it can go ahead and deploy this chart.
So now if you go ahead and run kubectl get secrets,
you can able to get the secrets
of the entire Kubernetes here, right?
So you can also get kubectl get pods,
you can able to get all the pods,
and in all namespaces as well, right?
So basically you have full cluster admin access,
because currently whatever the service account,
which is there in our pod,
because due to the reason we have created a cluster role,
this cluster admin, sorry,
this pod has basically full cluster admin access,
which can able to perform any kind of operations
on all resources, right?
So as a attacker, some of the useful commands
is like look for auth, can I create pods?
You can check if the current service account,
which you own, if you anytime get a service account access,
it can able to create pods or something, sorry, create.
You can also check delayed pods,
it can able to delay pods or something.
You can also specify, can I do in namespace of kube system?
So this is talk to the API and give you the information.
So this command is pretty useful.
If you got a service account and you wanted to see
what kind of privilege it has,
you can make these calls, right?
And also some tips, I don't know,
we are about to conclude and let me,
whenever you make these commands,
you can also increase the verbosity of the Kubernetes,
you can increase the six or seven,
you can able to see each and every request
it is making to the Kubernetes cluster
and in a verbose way.
Like if you increase this,
you can able to see much better, even like headers.
And if we increase like height,
like you can even see like a response body and everything.
And if you increase to nine and like little more depth,
it will keep going, right?
So this is completely API driven.
So you can able to play around with the verbose
and understand exactly each and every thing
which is being answered.
And also very useful command is explain.
If you wanted to understand a pod,
then it will give you more details about it.
And I wanted to learn more about pod spec,
you can also do kubectl spec.
So this is very handy.
And while you are performing assessments,
you can quickly look at the documentation
rather like explain commands, right?
Cool, so now you see,
due to the simple default misconfiguration
of our Helm pillar,
we can able to control the entire cluster.
And this vulnerability I have seen in so many of the cluster
in real-time pentesting assessments
due to the administrator doesn't configure properly.
And this is default security installation by Helm itself.
So the newer version of Helm v3 fixed this vulnerability,
so it kind of patched.
Cool, so let me go with the last scenario.
I think then we will kind of conclude.
And this you can try, it's a DIND,
Docker in Docker and Exploitation.
This you commonly see in CI-CD systems,
especially in build pipelines,
like GitLab CI-CD or Bitbucket CI-CD
or in real-world where they pass the socket,
our socket, right?
You can go ahead and try.
And SSRF vulnerability,
especially this is very widely exploited
in the cloud infrastructure.
But in coming to the Kubernetes context,
it is little much, you can get more out of it.
Because as I said, within the cluster,
there might not be some kind of services authenticated.
There might be services which are giving
more health information of the nodes
as well as the ports or internal services.
Using SSRF, you can get more information out of it
within the clusters compared to the cloud native,
cloud way of exploiting, right?
Cool.
So let me just complete with the...
Yeah, this looks fun.
Cool.
So this is the use case where basically
I'm trying to showcase,
if you end up in any of the pod
and the developer or operations teams
didn't set up any resource consumptions,
like you can only consume these many resources, right?
So you can able to use utilities,
especially like I'm trying to showcase here, stress-ng.
So let me showcase a simple example.
Let me show in the terminal, maybe.
Exit, kubectl get pods.
Now you see a bunch of pods are running.
So I'm looking for a specific pod.
Let's look for that kubectl top pod.
So when you do top, it's basically a system like top command.
It shows how much CPU resources
and memory resources it is using.
And if any of the deployment or manifest
which doesn't have specifically resource limit setup
and the cluster is deployed in like cloud infrastructure,
like which has auto-scaling enabled.
If you re-hit this limit,
it'll go and scale up a new node
or deploys like some more nodes into the cluster, right?
So that times you can exploit this
in an attacker's point of view.
Like let's go ahead and run a simple tool.
I'm running with two kicks and 30 seconds.
So now you can see if you're on the same command
from the specific resources of 3MB, 1MB, 2MB,
like increasing so much, right?
I think it's just taking some time.
Let's keep looking.
You can see it hits a crazy amount of memory resource limits
because there was no resource limit set for this deployment,
whatever the hunger check,
it will be deployed as part of the Kubernetes code.
Attacker can basically consume more CPU resources
and memory, which basically cost the billing
and very like cost-effective
for the company's perspective, right?
Billing.
And due to the reason,
the most of the modern infrastructure is set up
in such a way which is auto-scalable
and like scalable based on the certain parameters.
So you've been able to basically DDoS
or gain more resource consumption
for the companies and organizations, right?
So this is definitely one thing.
So before moving in,
let me just conclude with the Hacker Container preview.
As I said, Hacker Container,
let me just showcase maybe here, Hacker Container.
So Hacker Container is a similar,
another utility I created as part of the assessment.
So you can read more about it here, the blog post.
So this has a bunch of utilities,
which you use day-to-day,
like while performing assessments in the Kubernetes cluster.
So I have used it extensively,
I would say, while performing assessment
and it is very useful utility.
Let's go ahead and run.
Oops, looks like it's already there.
Let's exec into the pod, Hacker Container.
So we are inside the pod,
like you see a bunch of other stuff like default tools.
So some of the things like it has is amicontain,
so which is again, utility written by Jessica Frazel.
She has done quite a lot of work for the containers
and security community in the Kubernetes
and the container ecosystem.
So what it does is it will try to identify
what container runtime you are using
and do you have shared namespaces from the host system
like PID, username space,
or is there any app armor or sitcom profile
applied to the container to restrict certain capabilities?
And is that what all capabilities allowed currently,
system calls allowed,
and it kind of a container introspection tool.
So which will help you to perform while in attacks
and security assessments.
So not only I think specific to the container Kubernetes,
then it's a pretty useful tool to try out, right?
So you can also do the same thing
with proc self cgroup maybe.
Yeah, cool.
So you can see this is basically trying to figure out
kubeports and you can also do capsh, print.
So it will look for what all capabilities are there
and similarly.
So this is kind of pretty useful tool, the Hacker Container.
And also it has Nikto and a bunch of other use cases.
I think maybe I try to not document it here.
So I'm going to try to see
what can I kind of create out of it
and how I use in day to day while exploiting.
And some use cases I would say is very useful
is running benchmarks.
If you are inside the clusters
and you wanted to just get a glance of view,
what kind of vulnerabilities possibly there
within the Kubernetes clusters,
both in terms of container runtime
as well as the cluster level.
You can run these CIS benchmarks.
Like one is you can just go ahead and run this command.
Let's go ahead and run.
Yep.
So what it does is it,
I have created something called daemonset,
which means each node,
how many ever nodes you have in the cluster,
will get one pod running in all the nodes.
That is called daemonset in Kubernetes.
Let's go ahead and get the node.
So you can see two nodes are available.
Now you should be able to see two pods running
in the Docker Bench because there are two nodes
because we are running a daemonset, right?
So you can see two pods running.
So let's go ahead and exec into one of the pod
and see what we can do with Docker Bench Security, right?
So Docker Bench Security,
I'm execing into the pod.
Now you can see Docker Bench Security.
Now you can go ahead and run dockerbenchsecurity.css.
This is also part of a Hacker Container.
You can go ahead and run this.
So the way I have set it up,
sorry, the scenario manifest is if you can see here,
I have created the Docker Bench Security in such a way
it access the host PID, IPC network,
as well as a bunch of volumes.
And it also had extra capability called audit control
so that you can access and talk
to your underlying Docker Container runtime
and you can perform benchmark analysis
on possible security issues, right?
So this is a pretty simple utility.
Just go ahead and run this.
Once you are inside cluster,
then you will get at least high level posture
and use the whatever the found vulnerabilities
or the checks as attack paths,
or you can use those while you are doing assessments
or post-exploitation, right?
Once you are inside a node,
maybe you can use some kind of vulnerability
which is detected here to do some kind of post-exploitation.
So you can see it does a bunch of checks
like host level configuration checks
and specific to like the Docker daemon
or like directories and daemon configuration,
and as well as the files,
which is specific to the container runtime
here in this case, Docker,
and also container images,
whether container running with a root privilege
or it has any extra capability set
or it doesn't have any seccomp or apparm profiles,
these kinds of things, right?
Similarly, you can go ahead and run the kubebench as well.
You can go ahead and run this kubebench like this.
Let me go ahead and run here only, I think.
It's kind of taking quite a lot of time.
So you can see kubebench I have run.
You can give it pods.
The kubebench is running here.
You can get the logs, iPhone app, and the bench.
So it will show.
You can see there are 12 checks passed,
nine checks failed, and two checks running.
So you can see what all the vulnerabilities
possibly there in this Kubernetes cluster
and what would be the remediations.
So you can use these vulnerabilities
maybe to do further exploitation
and understand what you can do, right?
So these tools are very useful just to get a glance of it.
And there are a bunch of utilities out there as well,
like kubehunter and kubepsychaudit
and the different golang tool written
by different researchers.
And I'll try to document and put it in the same scenarios.
So hopefully you can go back and try out yourself.
Cool.
So there are a bunch of other scenarios.
Due to the time constraint,
I couldn't complete or showcase you,
but feel free to go and check out the Kubernetes Go project.
And you should be able to get and follow
using the playground of that Katakoda, right?
And if you have set it up in your own environment,
like in our case, right, my case,
so you can go ahead and tear down the cluster.
Basically, it will try to delete possibly resources
which I have created
as part of the Kubernetes Go deployment.
But I highly recommend you delete the cluster
to just ensure cleanup of the entire environment, right?
And I have also showcased and run using a tool called Chekhov
to see what all possible vulnerabilities
using a CACD as part of pipeline.
So it found a bunch of vulnerabilities in the deployments
and other things which I have used
as part of Kubernetes Go as well.
All right.
Cool.
So that's it pretty much about it.
And I would highly recommend having your feedback
and understand what exactly you want to look forward.
And I'm really looking forward
to building more and more scenarios
and also adding the defense scenario.
I'm really building like most of the scenarios
and making them in such a way that how you can practice
for the CNCF certified Kubernetes security.
So you can use this playground to practice
and understand how most of the Kubernetes vulnerabilities
you can assess and learn more about it.
So feel free to get involved
and you can please feel free to provide the feedback
and share with your colleagues, teammates, or your friends.
I would really appreciate if you can share the feedback
and I would love to improve and work on it.
And if you see any documentation
or anything needs to be improved,
please let me know by GitHub issues.
And I'm trying to be pretty active,
but in the past month I was a little busy.
So I kind of work on this,
working on defense scenarios as well.
Hopefully we will be releasing this month
for the defense scenarios to understand
and fix the vulnerabilities which you have learned
while practicing these attacks
and also more attacker scenarios as well.
And with that, I think I'm pretty much done
with the session.
Thank you so much once again for having me
and really looking forward.
Awesome. Thank you so much.
It was an amazing presentation.
I learned so much.
And as a matter of fact, in Discord,
there are a lot of people actually talking
about your presentation.
And once again, thank you for supporting the communities.
Thank you for supporting the DEF CON Red Team Village.
We're going to go on a little break,
but we invite everybody that will have any further questions
and so on to join the Discord channel,
which is in the description of the video.