start	end	text
1280	5120	Ah, come back here. Why does that keep doing that?
5600	9560	All right, great. We are here and ready to do securing in,
9560	13280	path testing, great spaghetti monster, as I like to call it, or
13280	18400	Kubernetes. This is who I am, Kat Fitzgerald.
18400	22840	You can find me on Twitter at Rainbow Kat. I've been doing this a long time.
22840	26580	I'm the CEO of B-Sides Chicago. I was COO for
26580	30820	the Diana Initiative last year, part of B-Sides Pittsburgh,
30820	34900	and also my first DEFCON was DEFCON 3.
34900	38860	How many other people remember that one? There were 16
39140	42560	speakers there, if I have to remember that.
42840	46820	I have an emphasis in the blue team. However, I'm a formal purple teamer
47160	50880	and all this stuff. And I'm in Pittsburgh now, but moving to
50880	54760	Kirkland, Washington very, very soon. You will always
54760	58800	find me sipping Grand Mayan Extra Añejo, or in
58800	63740	this case, it is Casanoble Añejo.
63740	66840	So, cheers. Let's kick off DEFCON. Jason
66840	68960	already did a great job.
71120	75240	And also, special thanks to the entire Red Team Village,
75240	78720	all the staff, all the volunteers, everybody for kicking butt
78720	82980	and making this happen. A lot of my favorite stuff are honey pots
82980	86640	and refrigerators. On Saturday, I'm doing a talk for
86640	90920	IoT security, so check that out, too. But this is one of my favorite ones
90920	94300	because I just finished writing this particular
95560	99380	presentation just about a month ago, made some changes to it, and so on.
99380	103000	Remember, please, the views and opinions here are mine and mine
103000	107000	alone. They do not reflect anything from my employers, past
107000	110940	or present. Anything that I show you,
110940	114780	as always, please use with a grain of salt. Don't go breaking
114780	118820	into things that you shouldn't be breaking into. And those of you with an
118820	122320	overwhelming fear of the unknown, you'll be happy to know that
122320	126720	there is no hidden message in the disclaimer if you read it
126720	130340	backwards. Yes, my jokes are that bad.
130380	134920	Alright, let's get this thing on the road. So, why are we
134920	138760	not here? Well, first of all, I'm not going to solve all
138760	142640	your Kubernetes security woes. I wish I could.
142640	147180	I wish Kubernetes and containers and everything were secure by default.
147180	150980	They are not. Neither is the person
150980	154520	sitting next to you, although, well, that doesn't work anymore because
154520	158520	nobody's sitting... well, maybe somebody's sitting next to you. You never know.
158520	162880	You never know. Okay. And common sense went out the window
162880	166540	decades ago. Think about that. When it comes to
166540	170940	information security, cyber security, hacking, whatever you want to call it,
170940	174340	I honestly think we lost common sense quite
174340	178560	some time ago. Why really are
178560	182520	we here? Well, Kubernetes, I
182520	186740	don't know. I still think it's fairly new. It's fun.
186740	190420	It's crazy. It's a lot of things.
190420	194560	You can deploy it in clouds. You can deploy it in...
194560	198500	I have Raspberry Pis sitting down here. I was
198500	202280	going to be showing that. There have been some demo problems and also some
202280	206520	timing issues, so I won't be showing my Raspberry Pi cluster today.
206520	210160	But you can build very extensive Raspberry Pi
210160	214780	clusters and do all the things that I'm going to talk about today.
215340	218800	Containers. Containers are not new.
218800	222400	But people seem to have forgotten that they
222400	226460	have anything to do with security. They get all
226460	230500	hyped up on Kubernetes security, and they're forgetting about
230500	234240	containers. We're going to talk about containers because it's
234240	238600	important. It's, in fact, very important. And always remember this.
238600	242480	Security is for everyone. Common
242480	246080	sense is required. So please
246080	250340	disregard the previous slide. Throw it out the window. Why?
250340	254560	Because if we're not using common sense in security, then
254560	258220	we're definitely not going to get where we need to be.
258640	262380	All right. Just some quick security facts. We want to
262380	266640	know this stuff. In 2018 alone, $114
266640	270620	billion were spent by companies on
270620	274580	information security. On fancy software,
274580	279100	fancy hardware, things with blinky lights and all of that.
279100	282980	But breaches kept on going up. We're doing
282980	287120	something wrong here. Security appliances and software
287120	291060	have vulnerabilities that we're not dealing with or that
291060	295320	need to be dealt with by the vendors. Kubernetes itself
295320	299700	has had a few issues. We'll talk about that. But so has Docker.
299700	303160	So don't forget your containers. Also, I've
303160	307400	actually had people say to me, we have a very unique security
307400	311460	architecture. And I'm like, no, you don't.
311560	315280	We all do the same stuff. So remember, your architecture
315280	319360	is not unique. I love this quote. If you don't have this
319360	323500	book, Offensive Countermeasures by John Strand, go out
323500	327220	and get it. It's a small book with a ton of information.
327220	331280	So get it. But he said, instead of brilliance,
331280	335840	we've standardized on mediocrity. And that seems to be
335840	340100	what information security is about these days.
340100	343800	And I kind of have issues with that. So let's talk about
343800	348040	that some more. First of all, let's get a common definition
348040	351920	of what is a breach. Or maybe we'll call it
351920	356200	what is an incident. Let's not necessarily call it a breach.
356200	359520	So most incidents are not caused by zero-day
359520	364620	fancy stuff that you find on the Internet. Typically,
364620	368160	they're done because people have made mistakes and so on.
368160	372140	Most breaches are not coming from your vulnerability scanners,
372140	376120	especially when it comes to Kubernetes. You're not going to run a vulnerability
376120	380360	scanner and go, oh, look, fix this one thing, and we're
380360	384120	done and ready to go. No, it doesn't happen that way.
385420	388000	Most breaches are going to come from
388580	392380	configuration issues. This is something I want
392380	396160	to emphasize. In fact, I'm going to show you some examples how
396160	400380	configuration issues tend to cause most of the breaches. A close
400380	404460	second is compromised credentials.
404500	409040	When I say credentials, I still go out on GitHub.
409240	412280	Well, maybe not every single day. And I think Jason
412280	416080	even mentioned it, too, in his previous talk on bug bounties.
416320	420540	Go out and look on GitHub and search for simple things
420540	424520	like password. See how many times you find
424520	428520	the word root. Yes, thousands of them
428520	432700	are invalid. They don't make any sense. It's probably going to find the keyword
432700	437140	root or password, and it's not going to get you anywhere.
437140	440540	And yet I still find where people hard-code credentials
440540	444540	into variables that say password. And right
444540	448600	next to it, there's the key. There's the whatever the password
448600	452560	is. Also, trailing in third
452560	456780	place for most incidents is overprivileged
456780	460620	users. This is incredibly important in
460620	464480	the Kubernetes world. I'm going to show you this in just a second
464480	469160	or in a couple of slides here. So keep these three things in mind.
469160	472920	Configuration issues, compromised credentials, and overprivileged
472920	476800	users. Okay. Kubernetes
476800	480820	itself, we have to deal with all three
480820	484720	of these compromises or these types
484720	489880	of incidents that are going to cause issues in Kubernetes.
489960	492980	But first, think
492980	496960	about this. You know, we think about the Capital One hack
496960	500980	that happened recently. Well, that was a combination
500980	504820	of compromised credentials and access
504820	509000	that should not have been granted. We see how
509000	512980	Tesla, you know, they had poor access credentials
512980	517160	and so on. Equifax, we all know about that one.
517160	520580	Go back and check if you don't. You know, Equifax
520580	524300	was hacked not only by a simple vulnerability
524820	528980	that had not been patched, but also because they had very
528980	533040	very poor lateral movement detection on the
533040	537000	inside. Once they got in with more compromised
537000	541240	credentials through the vulnerability, they weren't
541240	545180	detected for quite some time. This is going to be
545180	549220	critical. And one other thing, I was bored this morning. I decided
549220	553140	to throw this up here in this slide. Why? Well, I
553140	557060	put up a new spice rack in my kitchen. I really liked how it looks.
557060	561220	I took a picture and I added it to the slide. There it is.
561220	565240	Isn't it a great spice rack? Come on. Alright, you all better
565240	569460	be laughing. Alright, let's talk about containers.
570180	573160	Quick review of containers. They are not
573160	577400	secure by default. Now, I'm really focusing more on
577400	580980	Docker containers for now because that tends to be
580980	585040	the norm. And one thing to think about when it comes to
585040	588760	containers, containers are not like
590000	592740	a VM. A VM is actually more
592740	596840	isolated. And this is something that's going to be important coming
596840	600760	up here. So, a container is a grouping of stuff
600760	604660	and the stuff that we like to call it are resources. Well, the
604660	608820	resources are grouped into namespaces. If
608820	612740	you are new to containers, remember this. Namespaces
612740	616860	are critical. You group your processes,
616860	620980	your networks, your users, your ports, your interprocess. All
620980	624680	of this stuff can be controlled and
624680	628960	locked down within namespaces. Resources
628960	632840	are exploitable. I was sitting in another
633980	636640	web broadcast just the other day
636640	640240	on containers and I was
641660	644740	surprised because it was supposed to be a more
644740	648820	intermediate presentation on containers and
648820	653060	yet all of the people were asking is there a
653060	656740	way to prevent process
657560	660960	abuse such as CPUs and memory and
660960	664740	so on. And again, this was supposed to be an intermediate
665560	669100	class on containers so it
669100	672860	confused me a little bit that people were asking that because that should be something
672860	677740	that you know from the very beginning that you get into containers.
677740	680920	Using namespaces and the next
680920	685120	most important thing is control groups or cgroups. These
685120	689460	are the limits that you are going to be putting on containers.
689460	692820	If you don't put limits on them, you're
692820	697180	going to run into problems. Every time I do a pen test
697180	701320	of a new application deployed into a container,
701320	705500	I would say that 7 out of 10 times
705500	709340	I find that the author of the application
709340	713220	who has put it in a container has not put any limits.
713220	717540	I'll get into the application in whatever way possible whether it's
718140	721180	a SQL injection or an RCE
721180	725400	or something like that. And once I'm in, I can take over not just the
725400	729040	container but the entire system that it's on
729040	733800	because there are no cgroups applied to the container.
733800	737120	The other thing to remember about containers
737120	740960	is think of a container as an image or more
740960	745880	correctly, think of it as a file system snapshot.
746060	748880	So who decides what goes
748880	754220	in that image? If you look at this particular picture,
754220	756900	you'll see, you know, here we have
756900	760200	namespaces, the control groups that I show.
760480	764780	It's going to possibly be cherooted if you've set it up
764780	768860	correctly. And here's the application.
769400	772840	There's a Docker client that you might use to launch
772840	777120	this. There's a Docker registry. This is critical
777120	780940	to the security of your system. If you're
780940	785320	new to Docker, you may not know this, but Docker Hub was compromised
785320	789400	two years ago, I believe it was. I've lost track
789400	792520	of time during COVID. I don't know anymore.
792860	797160	But you want to have a secure registry where you keep
797160	801000	your images because if they're somewhere
801000	805800	that you don't have control, you never know what's going to happen.
805800	809020	But here's the difference. Notice that we have a host OS
809020	814280	and hardware down here, unlike a VM.
814280	818600	In a VM, you would see the entire operating system
818600	822520	within the VM here. You would not see a shared
822520	826560	OS. Yes, there's a host OS, but then you also have the guest
826560	830580	operating systems themselves. And that's where we get into
830580	834580	how a VM is much more isolated
834580	838300	and contained than a container is. So
838300	842540	don't think the word container means that you're more secure.
842540	846460	It just means that we have some work to do. So that's why
846460	850120	I say right here in this slide, container isolation.
850320	854740	Is it really isolated? The simple answer to that is
854740	858500	no, it is not. So let's talk about some
858500	862700	container best practices. I should have also said at the very
862700	866540	beginning of this talk, this is an intro.
866540	870360	I can't show you everything. I'm trying to get people to start
870360	874320	thinking about how you secure containers, then
874320	878340	we're going to jump into how you secure Kubernetes at the
878340	882320	same time. So remember, we have
882520	886300	a base image. That means that base image has to come from
886480	890000	a secure repo. If you do not know what CIS
890000	894200	is, definitely go to CIS
894200	897820	or CISecurity.org
898460	901860	Check it out because you want to use CIS
901860	905900	benchmarks for locking down your base image. I'm going to
905900	910020	talk about that a little more coming up. Don't run your images
910020	914040	as root. By default, if you do not set a user
914040	917920	name, your container and your application in the container is going
917920	921480	to run it as root. I have found so many
921480	926040	who have deployed WordPress applications
926040	929220	in a Docker container via Kubernetes
929980	934000	and they don't set the user name and WordPress is running
934000	937960	as root, so is the database and everything else. So of course,
937960	942400	once it's compromised, you take over everything. Also,
942400	946160	visibility. You need monitoring within
946160	949880	your containers. Your containers put a lot of information
949880	954180	out and they can log it, but you have to capture that. We're going to talk
954180	958220	about that in a minute. Also, think of
958220	962140	it as a domino. If one container is
962140	966280	compromised, I guarantee you the rest of them will
966280	969260	fall over like dominoes.
969880	974520	When you start to get into containers and Kubernetes, you're going to see this.
974920	978860	Also, we want to set some controls.
978900	982040	Besides setting a user name that is not
982040	985940	root, we also set privileges and we set
985940	989940	certain attributes to those users where it says
989940	993780	no elevated privileges or no new privileges.
993780	997720	You don't want user accounts or IDs
997720	1002020	to be able to escalate privileges. And I'll
1002020	1006080	show you some examples coming up. This one, my god, this is
1006080	1010020	near and dear to my heart, secrets management. I still
1010020	1014180	find, I would love to say by a show of hands,
1014180	1018040	but I can't do that right here, how many people
1019480	1022060	put the secrets
1022060	1025760	and tokens and things like that in their Docker
1025760	1029720	YAML files so when they then deploy the Docker
1029720	1034040	image, if somebody gets a hold of their YAML file, which
1034040	1038340	describes the configuration of that container,
1038340	1042340	well, the secrets are right in it. I've worked
1042340	1046460	with a lot of developers and a lot of engineers and I get
1046460	1050320	them, you know, you have to start thinking about putting
1050320	1054360	your secrets in another place, whether it's HashiCorp
1054360	1058240	Vault, whether it's a cloud secrets management system,
1058240	1062420	something. But get them out of the apps and get them out of the
1062420	1066480	configuration files. And stop allowing SSH
1066480	1069740	into your hosts. When we get into
1070660	1074340	Kubernetes, coming up in a couple slides here,
1074340	1078600	one of the key constructs of Kubernetes is you have a master node
1078600	1082660	or master nodes, and then you have worker nodes. And I
1082660	1086500	still find worker nodes with SSH enabled
1086500	1090380	on them so people can SSH in and, quote, fix
1090380	1094640	things. No, no, you use the
1094640	1098820	management tools to fix things and redeploy. Stop
1098820	1102940	allowing SSH into your systems. Also, I can't
1102940	1106880	stress it enough times, namespace
1106880	1110840	limits. Go read up on how to
1110840	1114820	control namespaces. That is going to
1114820	1118800	be one of your best efforts and best
1118800	1122760	practices for containing and controlling and
1122760	1126320	securing a container before you distribute it to
1127240	1130780	or deploy it via Kubernetes. This is one of
1130780	1134800	my favorite tools. It's called Docker Slim. If you've never heard
1134800	1138760	of it, if I were to create a container,
1138760	1142340	let's say I create a container, it has Ubuntu in it
1142340	1146260	or the base of Ubuntu, and then maybe it has WordPress
1146900	1151020	running in it for whatever god awful reason that I want to do this.
1151020	1155180	Let's say that entire image ends up being
1155180	1158700	three to four hundred meg. It's still
1158700	1163300	manageable, but if I were to run that through Docker Slim,
1163300	1167400	not only will it add some extra configurations
1167400	1171380	and namespace limits and so on, but it shrinks it down
1171380	1174840	and something that might be three or four hundred megabytes
1174840	1179180	deployed into a container will get shrunk down to
1179180	1183200	maybe 10, 15, 20 meg. This
1183200	1186960	makes it even that much more manageable. It's easier to
1186960	1190880	deploy. It's quicker to deploy, especially
1190880	1195260	if there are requirements for scaling.
1195260	1198680	So think about this. Take a look at this. I also mentioned
1198680	1203000	in Discord that my slides will be available in my GitHub
1203000	1207140	repo, and I'll get them to the Red Team Village so they can add them
1207140	1212000	to the Google Docs location after my talk.
1213260	1215340	And always remember
1215340	1219200	apply SELinux or AppArmor. One of the things
1219200	1223120	that Docker Slim does is it will do that for you.
1223120	1227080	It's very easy to enable SELinux or AppArmor
1227080	1231300	within a container. If you can do that, you're going to make
1231300	1235220	it that much more secure. It's not on by default, but you can set
1235220	1236960	it up so it will be.
1239140	1244040	And remember the infrastructure, as I put it.
1244640	1245400	You
1247120	1251180	can't forget that all of this stuff is being deployed on
1251180	1255160	an infrastructure of some sort, and that infrastructure better
1255160	1259180	be secure. I mentioned SSH. People keep
1259180	1263440	allowing SSH into their environment. Be very
1263440	1267500	aware that your infrastructure itself should be secure. And I will
1267500	1271280	remind you later about that. All right. Here's a quick picture.
1271280	1275720	Picture is worth a thousand words, right? If you look at our first example,
1275720	1279360	this is our monolithic operating system here. We see
1279360	1283560	hardware and operating system. We run apps. Here we see a VM. So we see
1283560	1287300	hardware operating system. There's a hypervisor involved.
1287900	1291320	And then there's going to be virtual machines. But when you have virtual
1291320	1295100	machines, you have additional operating systems, you have additional
1295100	1299480	libraries. They're very much isolated. I should have used a different
1299480	1303260	picture where there's a big line through it that
1303260	1307040	shows, yes, these are isolated and so on. But
1307040	1311380	that should not have happened. Nope. Go back.
1311540	1314720	Sorry. Okay. But when we look at
1315320	1319300	using Docker or Container, we go hardware operating
1319300	1323680	system. There is a container runtime. There's no real dividing lines.
1323680	1327400	And then we see containers. The only thing that is unique between
1327400	1331380	the containers is that bin library, the things
1331380	1335340	that are going to run in the application. So there's not a lot of
1335340	1339360	isolation there. And people want to
1339360	1343460	think that there is. But there is not. So please keep
1343460	1347080	that in mind. I do love this
1347080	1350940	comic. It came out a couple of years ago. A great
1350940	1355040	defense is, you know, we'll inundate them
1355040	1359100	with quarterly Kubernetes releases. Yes, they do update
1359100	1362820	Kubernetes a lot. That's why you have to pay
1362820	1366680	attention to your environment and the tools that you're going to use
1366680	1371140	so you can do upgrades to your Kubernetes cluster. So let's bring on
1371140	1375020	the orchestrator. Let's talk about Kubernetes. All right. Here
1375020	1378840	is Kubernetes 101. In one slide, I'm going to teach you
1378840	1382780	at least the basics of how Kubernetes
1382780	1387020	works. We have a master node. The master node is
1387020	1390560	critical. It manages all of the Kubernetes
1390560	1394520	worker nodes. It manages the clusters. It deploys
1394520	1398460	pods. Think of a pod as a collection
1399020	1403000	of containers that are grouped.
1403000	1406960	A pod might be the way that you
1406960	1410880	decide, oh, I want to have a pod that supports a particular
1410880	1414880	client. Or maybe it's a WordPress instance. And that
1414880	1419280	pod has WordPress. It has a container, a
1419280	1423140	separate container for the database and so on. So
1423140	1427320	the pod is how you can group things that
1427320	1431180	are various containers into certain groupings.
1431180	1435260	And those groupings are kind of defined by you. But stop and think about
1435260	1438920	how you want to deploy pods. It's going to be important.
1438920	1443580	A worker node. The node is where all the containers are going to run.
1443580	1447200	You have other Kubernetes components that have to
1447200	1451020	run there. But the important part is, this is where the containers
1451020	1454760	run. If the worker node is not secure
1454760	1459020	and not patched, and I can get into that
1459020	1462640	underneath the visibility of
1462640	1467000	Kubernetes, then Kubernetes has no idea that I've compromised
1467500	1471160	a node. I found numerous Kubernetes sites around
1471160	1474540	the Internet where there's
1474540	1479300	additional applications running on worker nodes because they've been compromised.
1479300	1483080	Of course, they're Bitcoin miners. But the master node has no
1483080	1486980	idea that it's running on there because they got in via
1486980	1490740	SSH and they launched the application, the
1490740	1494900	miner, underneath the visibility of Kubernetes.
1494900	1499100	So you really want to lock down these worker nodes. A pod,
1499100	1502900	I already explained, it's a unit of deployment and it's also how
1502900	1507080	you address things from Kubernetes. And a pod
1507080	1510520	will have an IP address. So that's important to
1510520	1515080	remember. But it's going to contain one or more containers.
1515080	1518980	I like to think of pods as, even though it might have
1518980	1522840	its own IP address, but think of it like a VLAN in a way.
1522840	1526900	Because I've isolated it in one way. Services
1526900	1530200	are various things that run within Kubernetes
1530980	1534920	that communicate, that balance, that do a lot of work. We're going to look
1534920	1538840	at services coming up. There are various components that are
1538840	1542760	used. Three that are critical, the API
1542760	1546640	server, you're going to have kubelets, and you're also
1546640	1551000	etcd. The one concept you should take
1551000	1555040	about etcd is this is the database
1555040	1558700	that describes and builds everything
1558700	1562580	that is your Kubernetes cluster. The etcd
1562580	1566280	environment should actually be walled off
1566280	1570560	and firewalled and controlled very, very
1570560	1574580	securely. If you do that, you will have a much
1574580	1578720	more secure Kubernetes cluster. So keep
1578720	1583040	that in mind. And you can see it here by looking at a picture.
1583040	1586420	So we take all of those concepts that we just talked about
1586420	1591000	we see here's an ABI server, here's a controller,
1591000	1593660	this is all part, you know, the kubectl...
1593660	1598640	Stop doing that. Come on. Kubectl
1598640	1602640	is the command you use to control everything.
1603020	1606580	And even though this is a disk kind of a representation
1606580	1610740	of etcd, I like to say, you know what, this needs to be off
1610740	1614920	on its own, firewalled, and separated, and it can be done that way.
1614920	1618420	Here's your worker node, and you can see how you might have multiple
1618420	1622440	pods on a worker node. This pod might be talking to
1622440	1626620	this pod. It doesn't necessarily have to talk to that one, or
1626620	1630720	they may not talk at all. These are some of the other services that are running
1630720	1634540	the proxies. This is where we are exposed to the internet,
1634540	1638500	not over here. Okay? So the only place you're
1638500	1642240	exposing Kubernetes is at the worker nodes.
1642780	1646280	Now, Kubernetes is not perfect.
1646520	1649980	There have been problems. There was...
1649980	1654640	I need to update this slide again, because there have been a couple of other
1654640	1658900	Kubernetes. These are more for reference. One's a
1658900	1662800	little older. This was back... there was the backdoor that involved the
1662800	1666760	API server. There were some others. These are only for your reference,
1666760	1671080	so you can go look them up and see. And there are
1671080	1674720	more. The whole point I'm trying to make here is
1675300	1679480	Kubernetes is not secure by default, and it's not
1679480	1682760	perfect. Well, let's try to make
1682760	1686500	it perfect. Let's take a common Kubernetes cluster
1686500	1690920	and say, you know, if we were to threat model this,
1690920	1694680	and that is very, very important, first thing
1694680	1698800	we want to do is to say, TLS everywhere. Yeah, I know, I have
1698800	1702620	too many exclamation points, but I'm trying to make the point. If
1703640	1706820	everything within your Kubernetes cluster is using TLS
1706820	1710640	and is encrypted, then it's that much
1710640	1714700	safer. I still have engineers come to me and say,
1714700	1718620	our application is completely internal. Why do we
1718620	1722680	need TLS? It's never going to be exposed to the internet.
1722680	1726560	I said, really? So what happens if Joe's application
1726560	1730680	over here, which is exposed to the internet, is compromised, and then
1730680	1735000	they use that as a pivot point to gain access to this environment,
1735000	1738380	and then they compromise your system? O
1738380	1742480	is typically the response. So I can't say it
1742480	1746440	enough, harden the bloody infrastructure. Yes, I'm saying
1746440	1751060	it again, but you're running this thing on your infrastructure,
1751060	1754320	make sure it's hardened. And very important,
1754320	1758180	two concepts. There's RBAC, which is
1758180	1762780	regular role-based access control. Set that up.
1762780	1766000	Make sure you're doing it right. Disable
1766000	1768600	ABAC. ABAC is
1770300	1773960	incredibly difficult to manage. You
1773960	1778000	will spend days and hours and hours trying
1778000	1781920	to do ABAC access controls,
1781920	1786000	which is more granular, but it's not worth the effort.
1786000	1790060	Make sure you use RBAC and manage it correctly.
1790060	1793700	And I'm going to show you a couple of tools coming up
1793700	1798000	on how to check it. Also, check out this blog
1798000	1801920	entry. I mentioned this previously about
1801920	1805520	monitoring. This is one way to audit
1805520	1809560	Kubernetes, so keep this in mind. Take a look at that
1809560	1813140	blog entry from Wazoo, and it's an excellent
1814780	1817680	place to start when it comes to monitoring
1817680	1821960	your environment. Also, kubectl, get role
1821960	1825960	binding in all your namespaces. You should do this
1825960	1829280	periodically in your environment, especially
1829720	1833680	if you have engineers deploying their own
1833680	1837920	applications. Make sure that all the namespaces
1837920	1842300	have good controls in their RBAC accounting.
1842300	1846040	Also, use third-party authentication for the API
1846040	1850000	server. Again, something like HashiCorp Vault. Yes, I love
1850000	1854100	that tool. If you are not familiar with it, get familiar with
1854100	1858000	it. It's a great tool, and it integrates very, very well with your
1858000	1860360	API server and Kubernetes.
1861680	1866120	I already mentioned this. Separate and firewall your etc cluster. Make sure
1866120	1869800	it's protected. That is the keys to the kingdom.
1869880	1873700	It's the entire configuration of your cluster. And rotate
1873700	1877880	encryption keys. Yes, use Vault. And by
1878380	1882860	default, turn on security policies in the pods.
1882880	1885860	It's some simple lines of configuration
1885860	1889440	in the YAML files. It's not hard to do. Or
1889440	1893920	use Docker Slim, that tool that I mentioned earlier, because
1893920	1898000	that will help you set this up. Alright, let's talk
1898000	1901920	about risk. kubectl, get secrets. That
1901920	1905860	is the one that I just mentioned where you can look at the secrets
1905860	1909840	from pretty much everything. But here's some tools
1909840	1913740	that I absolutely love by CyberArk. One is called Kubernetes
1913740	1918260	RBAC audit. The other is called KubeScan.
1918260	1922100	The RBAC audit does exactly what you think it does. It audits
1922100	1925560	all of your RBAC roles. The KubeScan
1925560	1930000	will look at all the permissions that are granted from
1930000	1934020	the roles. These are two great tools that
1934020	1937860	you want to put into your toolbox for securing
1937860	1942320	Kubernetes clusters. Because remember at the very beginning, I said
1942320	1946320	it's all about configurations. It's the number one
1946320	1950040	area that Kubernetes clusters get
1950040	1954260	compromised. Also, we want to take a look
1954260	1958260	at GitHub. Go out and make sure
1958260	1962040	that... take that away... go out and make sure that
1962040	1966260	your own configurations are not being
1966260	1970880	shared up to GitHub by your engineers accidentally.
1971080	1974340	How many times does this happen? It happens.
1974340	1978580	I've had it happen by engineers. They don't do it on purpose.
1978580	1981700	But they upload code. And in the code are secrets.
1981920	1985540	Oops. Oops is not a good thing.
1985760	1990400	Also, you need a locked down secure image repo.
1990400	1994380	And create gold images. Don't just let your
1994380	1998880	engineers go out and do, hey, docker run
1998880	2002680	bring down Ubuntu and pull it from the internet. No.
2002680	2006380	You need your own repo and registry for your docker
2006380	2010280	images. Make sure that happens. And please secure the
2010280	2014720	libraries. That's another big one. We also...
2014720	2019120	I need an entire slide on just the applications.
2019940	2023840	I'm going out on a limb. This is my personal opinion.
2023840	2027180	But I want to say that nine times out of ten... okay,
2027180	2030740	maybe eight and a half. Eight and a half times out of ten
2030740	2034600	Kubernetes clusters and containers are compromised
2035120	2038180	because the applications themselves
2038980	2042860	were poorly written and not tested.
2042980	2046900	And that's what gets me. I have
2046900	2050720	environments where I see the engineers create
2050720	2054660	an application. They do so much work to lock down
2054660	2058880	Kubernetes in the container, but they do a poor
2058880	2063260	job and there's a SQL injection vulnerability in the application directly.
2063360	2066240	So then the application is compromised
2066900	2070940	and then the container is compromised because, oops,
2070940	2075500	it was running as root and so on and so on. Remember the dominoes.
2075500	2079560	Everything falls over. Also, remember your
2079560	2083420	pods. Another common
2083420	2087160	mistake that people make when they're new to Kubernetes
2087160	2091600	is they don't stop and think, how can I
2091600	2096180	group things into pods and secure those pods?
2096180	2099160	At least if a container falls
2099160	2103380	within a pod, then maybe there's a couple more containers
2103380	2107340	in there. Those containers that get compromised are going to be somewhat
2107340	2111720	limited within the pod itself. Typically
2112060	2115440	a pod is not going to fall over when another pod falls
2115440	2119400	over. It's usually containers within the pods. So you want to think
2119400	2123440	about grouping things. Also, Jason mentioned
2123440	2127460	this in his talk earlier, but showdown. If you have
2127460	2131300	never used showdown, get familiar with it. Even
2131300	2135400	for Kubernetes. You can go out to showdown. You can start
2135400	2139480	searching for Kubernetes. You can search for API clusters
2139480	2142640	or API servers. You can search
2143140	2147540	for etcd servers. And if you find one of yours,
2147840	2151080	oops, but they're out there.
2151080	2154800	I did a search this morning for some etcd
2156020	2157540	exposure and
2159520	2161660	there were over 6,000
2161660	2165580	that I found that were exposed. I don't know how many of those have
2165580	2169380	vulnerabilities that would have allowed me to compromise them
2169380	2173440	because I was just curious what the number would be.
2173560	2177600	It goes up and down on a daily basis, but you should be
2177600	2181460	aware of that. So use showdown. Let's talk about
2181460	2185860	the configurations. I've said this before.
2185860	2189380	Most of the breaches come from configuration issues.
2192890	2195950	Use CIS hardening. Center for
2195950	2199950	Internet Security. It is an amazing place. It is done by
2199950	2203990	volunteers and workers within CIS. You can get
2203990	2208670	benchmarks for Docker and you can get benchmarks for Kubernetes.
2208670	2211790	You can use these to configure your environment
2211790	2214750	so they are indeed hardened.
2215710	2218630	Stop ignoring them. I still see people
2218630	2222390	that tell me they've never heard of CIS and
2222390	2226850	it makes me very sad when I hear that because it is such a great
2226850	2230930	resource. The whole point here is
2230930	2234710	we have to harden the environment from the
2234710	2238750	standpoint of the basic configuration, but don't forget
2238750	2242930	the application. Also, I know a lot of people
2242930	2247010	have environments that are CICD when it
2247010	2250950	comes to Docker and Kubernetes. That's why we have them. It's for
2250950	2254870	speed. It's for development. It's for putting things out
2254870	2258930	quickly. Okay, great. A lot of people use Jenkins.
2259690	2262550	Is there some way to automate
2262550	2267150	all of this so at least I can look at my images,
2267150	2271310	I can look at my containers, all within the CICD pipeline?
2271310	2275350	Yes, there is. There is a great tool called
2275350	2279570	Anchor. Anchor, as it says here, is a
2280030	2283570	container image scanner plugin. And what it does is
2283910	2287710	you put it into the pipeline. It's a plugin for Jenkins.
2287710	2291770	And it actually does security scans against the images
2291770	2295610	that you're trying to deploy. This is a great way to
2295610	2299370	get testing in before you actually
2299370	2303350	deploy it to production. So let's talk about some
2303350	2307450	additional tools before we get into the pen testing. Because we're still trying to
2307450	2311250	lock it down. KubeBench. We want to run KubeBench.
2311250	2315910	What it does is it runs against your CIS benchmarks.
2315910	2316790	It displays
2319870	2323650	a configuration that shows you what has passed, what has failed. I'll show an example
2323650	2327570	in a minute. There's also KubeAudit. It's a little more granular
2327570	2331810	than KubeSec. And if you run KubeBench
2331810	2335770	and KubeAudit, it will give you a report that shows you
2335770	2339210	oops, my system is insecure. Here's an example
2339210	2343370	of a report. And when we look at it, we see allow
2343370	2347730	privilege escalation is not set. There's a very
2347730	2351870	first one. Oops, that needs to be turned on.
2351870	2355790	Remember, that was the one that I mentioned that allows users
2355790	2360030	to do privilege escalation. We don't want that. Oh, crap.
2360030	2363930	We have run as non-root is not set. We don't have a read
2363930	2367590	only file system set. There are so many
2367590	2371830	simple things that can be done. Again, no app
2371830	2375830	armor, no security comp. It's not being turned on.
2375830	2379750	These are things that are just so easy. And this is from every
2379750	2383810	single container within the environment that is
2383810	2387690	being deployed under this particular. It's much
2387690	2391630	longer. It goes into it. This is just a quick snapshot.
2391630	2395930	The whole point here is you run this against your environment
2395930	2399710	before you deploy it. And you go, oops, let's go back to our
2399710	2402970	configuration file and fix all these things.
2403670	2408010	Here are some more tools for you in the environment. Claire,
2408010	2411490	another great tool for doing static analysis of
2411490	2415470	applications. Claire does, it integrates
2415470	2419490	Claire and your Docker registry so it can automate that
2419650	2423650	a little bit. Falco, I need to put huge stars next
2423650	2427490	to this one because this is actually one of my favorite, favorite
2427490	2431490	tools. It was developed years
2431490	2435810	ago and then it kind of morphed off and became falco.org.
2435810	2439740	It monitors behavior within containers.
2439740	2443100	It can detect if a container has been compromised
2443740	2447320	because suddenly the container starts acting weird.
2447320	2451920	Or there are commands being executed inside the container like
2451920	2455940	cat etsy password. Probably not a command
2455940	2458980	that should be showing up in one of your containers.
2459220	2463920	If anything of this entire talk that I'm
2463920	2467740	giving today, you get out of this, go look at falco.
2467740	2471280	This is one of the best tools out there. Kubehunter
2471280	2475280	is another great tool. We're going to talk about that.
2475540	2479640	It's used more so for looking at doing some pen testing.
2479640	2484140	We're going to take a look at that in just a second. Alright, ready, set, attack.
2484140	2487700	If we were going to pen test an environment, this is an
2487700	2491520	example of all the ports and all of the things that are
2491520	2495800	open in a Kubernetes cluster. Now, they're
2495800	2499800	open, but are they open by firewalls and so on?
2499800	2503800	Well, we don't know. The API server
2503800	2507680	itself is, yes, it's HTTPS.
2507680	2510140	It better damn well be protected.
2511720	2515920	etcd on 6666, that should be protected
2515920	2519620	wherever it happens to be. This just gives you an idea. All of these
2519620	2523820	ports can be changed. These are all the defaults in most cases.
2524360	2527600	Just be aware of the ports that are out there and
2527600	2531620	exposed that you need to make
2531620	2535720	sure are protected. Because as an attacker, what am I
2535720	2540100	going to do? Well, one of the first things, even Jason was talking about it in his talk,
2540100	2544360	looking at ports, running nmap, running mass scan, and so on.
2545040	2547380	This is exactly one of the things that
2547380	2551700	Shodan does. It looks for all of these things that are exposed on the
2551700	2555700	internet. So please make sure you're aware of all the ports
2555700	2559920	within your cluster. Now, I are a hacker.
2559920	2563720	See me hack. This is one of the
2564320	2567920	ways that I'm going to start to attack
2569000	2571620	a Kubernetes cluster. I want to first
2571620	2575800	and foremost check access to the API server, make sure that is
2575800	2579820	protected. I'm going to be looking at etcd,
2579820	2583920	and making sure that's protected. If I find it exposed,
2583920	2586640	I'm going to get keys to the kingdom in that case.
2586640	2591700	What about the kubelets that are communicating and the command channels
2591700	2595040	that are there? They should be read-only, but
2595700	2599420	they should not be exposed. There was actually
2602440	2603640	a vulnerability exposed
2603640	2607640	about this, I want to say two months ago. I'm probably off in that
2607640	2611920	sense. But take a look at that. And it allowed, even though it's a
2611920	2615740	read-only port and it's supposed to limit what you can see,
2615740	2620500	there were some vulnerabilities that allowed for a lot of exposure there.
2620500	2624160	Also, container vulnerabilities.
2624160	2627800	What about, you know, do you have files
2627800	2631960	in that container that the application needs that
2631960	2635600	you should be protecting? Well, that's where I would set up Falco and tell
2635600	2639440	Falco, hey, there's a special file over here. This application
2639440	2643500	is using it. But keep an eye on that. Make sure that the only
2643500	2647540	thing that is reading that application is, or excuse me,
2647540	2650720	that is reading that file happens to be the
2651600	2655160	application. And Falco can monitor that.
2655400	2659160	Privilege escalation. Nine times out of ten, once somebody
2659160	2663380	compromises an application in a container, they
2663380	2667240	elevate privilege because there's some other bugs in there.
2667240	2671420	Go back to the Kubotic slide where
2671420	2675560	it showed privilege escalation was not
2675560	2679260	set. All you have to do is set that configuration
2679260	2683760	variable. It's not that hard. Okay?
2683760	2686840	Vulnerable applications are critical.
2687360	2691480	Can't say it enough times. Especially because there are
2691480	2695840	so many people using Kubernetes and Docker to deploy
2695840	2699480	WordPress. And WordPress is, of course, one of
2699480	2703500	our favorite things to deal with. All right. So your
2703500	2707320	final threat model, when it comes to pentesting and
2707320	2711560	looking at the environment that is Kubernetes,
2712220	2715400	you think of your workload. You think of that
2715400	2719220	security within your pods. Can I
2719220	2723700	set up my pods so they're indeed secure?
2723700	2727400	The container security itself, the configuration
2727400	2731940	for that container, the more hooks I put in,
2731940	2735620	run as root, no. Privilege escalation,
2735620	2739420	no. These are the things that we want to make sure that are
2739420	2743300	set. Also, file storage security.
2743300	2747440	Maybe there's a particular file system that has to be read right
2747440	2751380	but the rest of the file system can be read only. Those are
2751380	2756500	other things. The network security. Remember the infrastructure.
2756520	2759300	Put firewalls in. Put controls
2759300	2763400	in there that detect lateral movement. If
2763400	2767720	somebody does compromise one of your applications,
2767720	2771360	make sure you have something monitoring. I gave you that
2771360	2775680	slide way back about the blog entry from Wazoo.
2775680	2779420	They talk about how to audit and monitor
2780000	2783440	a Kubernetes cluster using their tool. I love the Wazoo
2783440	2787580	manager and agent. Take a look at it if you haven't heard of it
2787580	2791780	before. Remember, most
2791780	2795500	breaches are not zero day. I can't say it enough times. Everybody's always
2795500	2799600	trying to protect against this stuff. They're not fancy.
2799600	2803380	Most of these things are being broken into because people
2803380	2807540	make silly mistakes. Because remember, common sense.
2807540	2811320	Whatever happened to it. The breaches are going to come from configuration
2811320	2815400	issues. The kube audit was a very good
2815400	2819440	example where it was showing all the problems that would allow someone to
2819440	2823180	get in. Compromise credentials.
2823180	2827260	Make sure your credentials are stored in a secure place
2827260	2831140	somewhere. And don't allow for
2831140	2834880	privilege escalation. Kubernetes and Docker
2834880	2838980	give you the tools and the configurations to prevent that.
2838980	2843320	Just turn them on. And that's what we're not doing. We have
2843320	2847260	to turn on the configurations. So your key takeaways
2847260	2851140	from today. Common sense.
2851140	2853880	Please go back and stop
2854960	2860240	and think about where is the exposure?
2860240	2864660	Where are my pods exposed? What about my etcd?
2865240	2868200	Configuration. That is so critical to any
2868200	2872200	Kubernetes. And always remember containers.
2872200	2875880	Containers are the linchpin of a Kubernetes
2875880	2880140	environment. And if I don't secure my individual containers
2880140	2884320	then Kubernetes... I don't care about
2884320	2888460	the security of Kubernetes because the container is insecure.
2888460	2892720	So stop overthinking this stuff and think about
2892720	2897420	security basics. Remember the CIS benchmarks.
2897860	2900540	Can't say it enough. Basics.
2900540	2904440	I still think as security professionals
2904440	2908480	hackers, whatever we want to call ourselves. We have
2908480	2912340	forgotten the basics of how to secure an environment.
2912340	2915780	And that's why things keep getting broken into.
2915780	2920050	Stop doing fancy crap. The fancy crap doesn't work.
2921460	2923940	Secure the environment, please.
2924020	2928240	Use the CIS benchmarks for the containers, for Kubernetes,
2928240	2931760	for the OS that your stuff runs on. If you're going to
2931760	2935760	deploy this whole Kubernetes cluster on Ubuntu, then pull up
2935760	2939280	the CIS benchmarks for Ubuntu. Or
2939280	2944040	Red Hat or CentOS. Whatever you're using.
2944040	2948060	Remember, RBAC is your friend. ABAC
2948060	2952080	is not. Your infrastructure better be
2952080	2955760	secure. Your pods better be secure.
2955760	2960080	Containers themselves, the network. Still a lot of people forget
2960080	2964160	the network because they forget about TLS everywhere.
2966880	2968300	Threat model
2968300	2972000	your applications. If you don't take
2972000	2976500	the beginning effort to threat model your applications,
2976500	2980180	you end up with insecure applications. And all the security
2980180	2983580	of your Kubernetes cluster is useless because you have
2983580	2988260	insecure applications within your environment. Access controls
2988260	2991980	go along with RBAC, but access controls into the
2991980	2996460	environment, kind of referencing SSH.
2996920	2999760	I found one worker node configuration
2999760	3003960	in an environment probably nine months ago. But for some
3003960	3008700	reason on the worker nodes, they had FTP enabled on all the worker nodes.
3008820	3012660	And my head exploded.
3013180	3016540	It allowed me to get into everything, but oh well.
3017380	3019920	Yes, did I mention RBAC though?
3020220	3023480	Enough times. I can't say it enough. Why? Because
3023480	3028000	people tend to ignore it. Remember the tools I mentioned earlier
3028000	3032140	about monitoring or auditing the RBAC roles and also
3032140	3035960	the permissions. And finally,
3035960	3039900	not quite finally, remember to patch everything.
3039900	3043820	The infrastructure that goes along with that.
3043860	3048080	Logging and auditing and monitoring. I still see people
3048080	3052700	set up their environments. There's that blog entry again.
3052700	3056040	Please take a look at that. It is well written.
3056040	3059960	It's a great tool. You'll learn a lot about Wazoo. I use it on
3059960	3064120	all my systems. I deploy a lot of honeypots with
3064120	3068100	Kubernetes and containers and things like that. But I monitor them all
3068100	3072240	with Wazoo. You can't
3072240	3076540	have a secure environment if you're not monitoring that environment.
3076540	3078360	That is the biggest thing.
3079860	3084080	That brings me to the end. I thank you for your
3084080	3088180	attention. There's contact info. There's my Twitter
3088180	3092260	account. I will get this to the Red Team
3092260	3096360	Village and they can upload it to their environment. But I'll have it on my
3099160	3100200	GitHub later
3100200	3104360	in an hour or so. I'll update it there. Thank you all very much.
3104360	3108140	Hopefully you learned something today. Go out and hack and have some fun. Thanks to
3108140	3111480	the Red Team Village and have an awesome DEF CON.
3111480	3115840	Awesome. Thank you so much, Kat. Amazing presentation
3115840	3119500	as always. Thank you for supporting DEF CON. Thank you for supporting the
3119500	3122700	community. Of course, the Red Team Village. For those of you
3123440	3126940	that just joined or are looking at
3127340	3131200	the presentation on Twitch or YouTube, please look at
3131200	3135480	all the activities that we have going on. Go to the website that is in the description of
3135480	3139540	the stream. Please also join the conversation in Discord.
3139540	3143760	We're going to go on a brief break and the next presentation will start in a few minutes.
3143760	3145340	Thank you again and thank you, Kat.