1
00:00:01,280 --> 00:00:03,280
Ah, come back here.

2
00:00:03,280 --> 00:00:05,120
Why does that keep doing that?

3
00:00:05,600 --> 00:00:06,820
All right, great.

4
00:00:06,820 --> 00:00:14,660
We are here and ready to do securing in, path testing, great spaghetti monster, as I like to call it, or Kubernetes.

5
00:00:15,540 --> 00:00:18,400
This is who I am, Kat Fitzgerald.

6
00:00:18,400 --> 00:00:20,940
You can find me on Twitter at Rainbow Kat.

7
00:00:20,940 --> 00:00:22,840
I've been doing this a long time.

8
00:00:22,840 --> 00:00:25,440
I'm the CEO of B-Sides Chicago.

9
00:00:25,540 --> 00:00:34,900
I was COO for the Diana Initiative last year, part of B-Sides Pittsburgh, and also my first DEFCON was DEFCON 3.

10
00:00:34,900 --> 00:00:38,180
How many other people remember that one?

11
00:00:38,180 --> 00:00:42,560
There were 16 speakers there, if I have to remember that.

12
00:00:42,840 --> 00:00:45,360
I have an emphasis in the blue team.

13
00:00:45,360 --> 00:00:48,700
However, I'm a formal purple teamer and all this stuff.

14
00:00:48,700 --> 00:00:53,320
And I'm in Pittsburgh now, but moving to Kirkland, Washington very, very soon.

15
00:00:53,320 --> 00:01:03,740
You will always find me sipping Grand Mayan Extra Añejo, or in this case, it is Casanoble Añejo.

16
00:01:03,740 --> 00:01:04,840
So, cheers.

17
00:01:04,840 --> 00:01:06,620
Let's kick off DEFCON.

18
00:01:06,620 --> 00:01:08,960
Jason already did a great job.

19
00:01:11,120 --> 00:01:20,660
And also, special thanks to the entire Red Team Village, all the staff, all the volunteers, everybody for kicking butt and making this happen.

20
00:01:20,660 --> 00:01:24,540
A lot of my favorite stuff are honey pots and refrigerators.

21
00:01:24,540 --> 00:01:29,240
On Saturday, I'm doing a talk for IoT security, so check that out, too.

22
00:01:29,240 --> 00:01:39,380
But this is one of my favorite ones because I just finished writing this particular presentation just about a month ago, made some changes to it, and so on.

23
00:01:39,380 --> 00:01:43,520
Remember, please, the views and opinions here are mine and mine alone.

24
00:01:43,520 --> 00:01:48,420
They do not reflect anything from my employers, past or present.

25
00:01:48,800 --> 00:01:53,940
Anything that I show you, as always, please use with a grain of salt.

26
00:01:53,940 --> 00:01:57,500
Don't go breaking into things that you shouldn't be breaking into.

27
00:01:57,640 --> 00:02:07,680
And those of you with an overwhelming fear of the unknown, you'll be happy to know that there is no hidden message in the disclaimer if you read it backwards.

28
00:02:08,020 --> 00:02:10,340
Yes, my jokes are that bad.

29
00:02:10,380 --> 00:02:13,000
Alright, let's get this thing on the road.

30
00:02:13,600 --> 00:02:16,320
So, why are we not here?

31
00:02:16,700 --> 00:02:21,160
Well, first of all, I'm not going to solve all your Kubernetes security woes.

32
00:02:21,160 --> 00:02:22,640
I wish I could.

33
00:02:22,640 --> 00:02:27,180
I wish Kubernetes and containers and everything were secure by default.

34
00:02:27,180 --> 00:02:28,760
They are not.

35
00:02:29,340 --> 00:02:37,720
Neither is the person sitting next to you, although, well, that doesn't work anymore because nobody's sitting... well, maybe somebody's sitting next to you.

36
00:02:37,720 --> 00:02:38,520
You never know.

37
00:02:38,520 --> 00:02:39,940
You never know.

38
00:02:39,940 --> 00:02:40,920
Okay.

39
00:02:40,920 --> 00:02:44,360
And common sense went out the window decades ago.

40
00:02:44,360 --> 00:02:45,520
Think about that.

41
00:02:45,520 --> 00:02:57,060
When it comes to information security, cyber security, hacking, whatever you want to call it, I honestly think we lost common sense quite some time ago.

42
00:02:57,060 --> 00:02:59,800
Why really are we here?

43
00:03:00,380 --> 00:03:03,000
Well, Kubernetes, I don't know.

44
00:03:03,000 --> 00:03:05,980
I still think it's fairly new.

45
00:03:05,980 --> 00:03:06,740
It's fun.

46
00:03:06,740 --> 00:03:07,680
It's crazy.

47
00:03:07,680 --> 00:03:10,420
It's a lot of things.

48
00:03:10,420 --> 00:03:12,300
You can deploy it in clouds.

49
00:03:12,300 --> 00:03:14,560
You can deploy it in...

50
00:03:14,560 --> 00:03:18,080
I have Raspberry Pis sitting down here.

51
00:03:18,080 --> 00:03:19,680
I was going to be showing that.

52
00:03:19,680 --> 00:03:26,520
There have been some demo problems and also some timing issues, so I won't be showing my Raspberry Pi cluster today.

53
00:03:26,520 --> 00:03:34,780
But you can build very extensive Raspberry Pi clusters and do all the things that I'm going to talk about today.

54
00:03:35,340 --> 00:03:36,260
Containers.

55
00:03:36,360 --> 00:03:38,800
Containers are not new.

56
00:03:38,800 --> 00:03:45,640
But people seem to have forgotten that they have anything to do with security.

57
00:03:45,640 --> 00:03:51,680
They get all hyped up on Kubernetes security, and they're forgetting about containers.

58
00:03:51,680 --> 00:03:54,900
We're going to talk about containers because it's important.

59
00:03:54,900 --> 00:03:56,980
It's, in fact, very important.

60
00:03:56,980 --> 00:03:58,600
And always remember this.

61
00:03:58,600 --> 00:04:00,740
Security is for everyone.

62
00:04:01,940 --> 00:04:05,140
Common sense is required.

63
00:04:05,140 --> 00:04:08,540
So please disregard the previous slide.

64
00:04:08,540 --> 00:04:09,800
Throw it out the window.

65
00:04:09,800 --> 00:04:10,340
Why?

66
00:04:10,340 --> 00:04:18,220
Because if we're not using common sense in security, then we're definitely not going to get where we need to be.

67
00:04:18,640 --> 00:04:19,220
All right.

68
00:04:19,220 --> 00:04:21,660
Just some quick security facts.

69
00:04:21,660 --> 00:04:23,500
We want to know this stuff.

70
00:04:23,500 --> 00:04:32,540
In 2018 alone, $114 billion were spent by companies on information security.

71
00:04:32,540 --> 00:04:39,100
On fancy software, fancy hardware, things with blinky lights and all of that.

72
00:04:39,100 --> 00:04:41,860
But breaches kept on going up.

73
00:04:41,860 --> 00:04:45,140
We're doing something wrong here.

74
00:04:45,140 --> 00:04:54,100
Security appliances and software have vulnerabilities that we're not dealing with or that need to be dealt with by the vendors.

75
00:04:54,100 --> 00:04:56,700
Kubernetes itself has had a few issues.

76
00:04:56,700 --> 00:04:57,720
We'll talk about that.

77
00:04:57,720 --> 00:04:59,700
But so has Docker.

78
00:04:59,700 --> 00:05:01,960
So don't forget your containers.

79
00:05:01,960 --> 00:05:08,720
Also, I've actually had people say to me, we have a very unique security architecture.

80
00:05:08,780 --> 00:05:11,460
And I'm like, no, you don't.

81
00:05:11,560 --> 00:05:13,760
We all do the same stuff.

82
00:05:13,760 --> 00:05:16,700
So remember, your architecture is not unique.

83
00:05:16,700 --> 00:05:18,080
I love this quote.

84
00:05:18,080 --> 00:05:24,060
If you don't have this book, Offensive Countermeasures by John Strand, go out and get it.

85
00:05:24,060 --> 00:05:27,220
It's a small book with a ton of information.

86
00:05:27,220 --> 00:05:28,560
So get it.

87
00:05:28,560 --> 00:05:33,900
But he said, instead of brilliance, we've standardized on mediocrity.

88
00:05:33,900 --> 00:05:40,100
And that seems to be what information security is about these days.

89
00:05:40,100 --> 00:05:42,840
And I kind of have issues with that.

90
00:05:42,840 --> 00:05:44,840
So let's talk about that some more.

91
00:05:44,840 --> 00:05:49,820
First of all, let's get a common definition of what is a breach.

92
00:05:50,240 --> 00:05:53,440
Or maybe we'll call it what is an incident.

93
00:05:53,440 --> 00:05:56,200
Let's not necessarily call it a breach.

94
00:05:56,200 --> 00:06:03,200
So most incidents are not caused by zero-day fancy stuff that you find on the Internet.

95
00:06:03,200 --> 00:06:08,160
Typically, they're done because people have made mistakes and so on.

96
00:06:08,160 --> 00:06:14,420
Most breaches are not coming from your vulnerability scanners, especially when it comes to Kubernetes.

97
00:06:14,420 --> 00:06:21,780
You're not going to run a vulnerability scanner and go, oh, look, fix this one thing, and we're done and ready to go.

98
00:06:21,780 --> 00:06:24,120
No, it doesn't happen that way.

99
00:06:25,420 --> 00:06:30,680
Most breaches are going to come from configuration issues.

100
00:06:30,820 --> 00:06:33,560
This is something I want to emphasize.

101
00:06:33,560 --> 00:06:39,820
In fact, I'm going to show you some examples how configuration issues tend to cause most of the breaches.

102
00:06:39,820 --> 00:06:44,460
A close second is compromised credentials.

103
00:06:44,500 --> 00:06:49,040
When I say credentials, I still go out on GitHub.

104
00:06:49,240 --> 00:06:51,460
Well, maybe not every single day.

105
00:06:51,460 --> 00:06:56,080
And I think Jason even mentioned it, too, in his previous talk on bug bounties.

106
00:06:56,320 --> 00:07:02,340
Go out and look on GitHub and search for simple things like password.

107
00:07:02,480 --> 00:07:06,120
See how many times you find the word root.

108
00:07:06,800 --> 00:07:09,720
Yes, thousands of them are invalid.

109
00:07:09,720 --> 00:07:11,000
They don't make any sense.

110
00:07:11,000 --> 00:07:17,140
It's probably going to find the keyword root or password, and it's not going to get you anywhere.

111
00:07:17,140 --> 00:07:24,000
And yet I still find where people hard-code credentials into variables that say password.

112
00:07:24,000 --> 00:07:26,500
And right next to it, there's the key.

113
00:07:26,500 --> 00:07:29,620
There's the whatever the password is.

114
00:07:30,640 --> 00:07:37,880
Also, trailing in third place for most incidents is overprivileged users.

115
00:07:37,880 --> 00:07:42,560
This is incredibly important in the Kubernetes world.

116
00:07:42,560 --> 00:07:46,780
I'm going to show you this in just a second or in a couple of slides here.

117
00:07:46,780 --> 00:07:49,160
So keep these three things in mind.

118
00:07:49,160 --> 00:07:54,360
Configuration issues, compromised credentials, and overprivileged users.

119
00:07:54,780 --> 00:07:55,700
Okay.

120
00:07:56,080 --> 00:08:09,880
Kubernetes itself, we have to deal with all three of these compromises or these types of incidents that are going to cause issues in Kubernetes.

121
00:08:09,960 --> 00:08:14,340
But first, think about this.

122
00:08:14,340 --> 00:08:18,900
You know, we think about the Capital One hack that happened recently.

123
00:08:18,900 --> 00:08:27,420
Well, that was a combination of compromised credentials and access that should not have been granted.

124
00:08:27,500 --> 00:08:34,060
We see how Tesla, you know, they had poor access credentials and so on.

125
00:08:34,120 --> 00:08:37,160
Equifax, we all know about that one.

126
00:08:37,160 --> 00:08:39,520
Go back and check if you don't.

127
00:08:39,520 --> 00:08:54,320
You know, Equifax was hacked not only by a simple vulnerability that had not been patched, but also because they had very very poor lateral movement detection on the inside.

128
00:08:54,320 --> 00:09:04,160
Once they got in with more compromised credentials through the vulnerability, they weren't detected for quite some time.

129
00:09:04,160 --> 00:09:06,100
This is going to be critical.

130
00:09:06,100 --> 00:09:08,600
And one other thing, I was bored this morning.

131
00:09:08,600 --> 00:09:11,700
I decided to throw this up here in this slide.

132
00:09:11,700 --> 00:09:12,300
Why?

133
00:09:12,300 --> 00:09:15,520
Well, I put up a new spice rack in my kitchen.

134
00:09:15,520 --> 00:09:17,060
I really liked how it looks.

135
00:09:17,060 --> 00:09:20,100
I took a picture and I added it to the slide.

136
00:09:20,260 --> 00:09:21,220
There it is.

137
00:09:21,220 --> 00:09:23,240
Isn't it a great spice rack?

138
00:09:23,240 --> 00:09:24,280
Come on.

139
00:09:24,280 --> 00:09:26,620
Alright, you all better be laughing.

140
00:09:27,040 --> 00:09:29,460
Alright, let's talk about containers.

141
00:09:30,180 --> 00:09:32,280
Quick review of containers.

142
00:09:32,280 --> 00:09:34,780
They are not secure by default.

143
00:09:34,780 --> 00:09:42,680
Now, I'm really focusing more on Docker containers for now because that tends to be the norm.

144
00:09:42,680 --> 00:09:51,000
And one thing to think about when it comes to containers, containers are not like a VM.

145
00:09:51,000 --> 00:09:53,900
A VM is actually more isolated.

146
00:09:53,900 --> 00:09:57,520
And this is something that's going to be important coming up here.

147
00:09:57,520 --> 00:10:04,300
So, a container is a grouping of stuff and the stuff that we like to call it are resources.

148
00:10:04,300 --> 00:10:08,420
Well, the resources are grouped into namespaces.

149
00:10:08,420 --> 00:10:12,160
If you are new to containers, remember this.

150
00:10:12,160 --> 00:10:14,380
Namespaces are critical.

151
00:10:14,400 --> 00:10:20,480
You group your processes, your networks, your users, your ports, your interprocess.

152
00:10:20,480 --> 00:10:28,160
All of this stuff can be controlled and locked down within namespaces.

153
00:10:28,360 --> 00:10:30,700
Resources are exploitable.

154
00:10:30,700 --> 00:11:01,540
I was sitting in another web broadcast just the other day on containers and I was surprised because it was supposed to be a more intermediate presentation on containers and yet all of the people were asking is there a way to prevent process abuse such as CPUs and memory and so on.

155
00:11:01,540 --> 00:11:17,740
And again, this was supposed to be an intermediate class on containers so it confused me a little bit that people were asking that because that should be something that you know from the very beginning that you get into containers.

156
00:11:17,740 --> 00:11:24,480
Using namespaces and the next most important thing is control groups or cgroups.

157
00:11:24,480 --> 00:11:29,460
These are the limits that you are going to be putting on containers.

158
00:11:29,460 --> 00:11:35,180
If you don't put limits on them, you're going to run into problems.

159
00:11:35,180 --> 00:11:53,220
Every time I do a pen test of a new application deployed into a container, I would say that 7 out of 10 times I find that the author of the application who has put it in a container has not put any limits.

160
00:11:53,220 --> 00:12:02,540
I'll get into the application in whatever way possible whether it's a SQL injection or an RCE or something like that.

161
00:12:02,540 --> 00:12:13,800
And once I'm in, I can take over not just the container but the entire system that it's on because there are no cgroups applied to the container.

162
00:12:13,800 --> 00:12:25,880
The other thing to remember about containers is think of a container as an image or more correctly, think of it as a file system snapshot.

163
00:12:26,060 --> 00:12:30,720
So who decides what goes in that image?

164
00:12:30,720 --> 00:12:40,200
If you look at this particular picture, you'll see, you know, here we have namespaces, the control groups that I show.

165
00:12:40,480 --> 00:12:46,080
It's going to possibly be cherooted if you've set it up correctly.

166
00:12:46,180 --> 00:12:48,860
And here's the application.

167
00:12:49,400 --> 00:12:53,300
There's a Docker client that you might use to launch this.

168
00:12:53,300 --> 00:12:55,820
There's a Docker registry.

169
00:12:55,860 --> 00:13:00,240
This is critical to the security of your system.

170
00:13:00,240 --> 00:13:08,600
If you're new to Docker, you may not know this, but Docker Hub was compromised two years ago, I believe it was.

171
00:13:08,600 --> 00:13:10,900
I've lost track of time during COVID.

172
00:13:10,900 --> 00:13:12,520
I don't know anymore.

173
00:13:12,860 --> 00:13:25,800
But you want to have a secure registry where you keep your images because if they're somewhere that you don't have control, you never know what's going to happen.

174
00:13:25,800 --> 00:13:27,380
But here's the difference.

175
00:13:27,380 --> 00:13:34,280
Notice that we have a host OS and hardware down here, unlike a VM.

176
00:13:34,280 --> 00:13:40,800
In a VM, you would see the entire operating system within the VM here.

177
00:13:40,800 --> 00:13:43,200
You would not see a shared OS.

178
00:13:43,200 --> 00:13:48,480
Yes, there's a host OS, but then you also have the guest operating systems themselves.

179
00:13:48,480 --> 00:13:57,980
And that's where we get into how a VM is much more isolated and contained than a container is.

180
00:13:57,980 --> 00:14:02,540
So don't think the word container means that you're more secure.

181
00:14:02,540 --> 00:14:05,780
It just means that we have some work to do.

182
00:14:05,780 --> 00:14:10,120
So that's why I say right here in this slide, container isolation.

183
00:14:10,320 --> 00:14:12,740
Is it really isolated?

184
00:14:12,740 --> 00:14:17,180
The simple answer to that is no, it is not.

185
00:14:17,180 --> 00:14:20,880
So let's talk about some container best practices.

186
00:14:20,880 --> 00:14:26,540
I should have also said at the very beginning of this talk, this is an intro.

187
00:14:26,540 --> 00:14:28,220
I can't show you everything.

188
00:14:28,220 --> 00:14:39,760
I'm trying to get people to start thinking about how you secure containers, then we're going to jump into how you secure Kubernetes at the same time.

189
00:14:40,200 --> 00:14:43,620
So remember, we have a base image.

190
00:14:43,620 --> 00:14:47,740
That means that base image has to come from a secure repo.

191
00:14:47,740 --> 00:15:05,640
If you do not know what CIS is, definitely go to CIS or CISecurity.org Check it out because you want to use CIS benchmarks for locking down your base image.

192
00:15:05,640 --> 00:15:08,260
I'm going to talk about that a little more coming up.

193
00:15:08,480 --> 00:15:11,060
Don't run your images as root.

194
00:15:11,220 --> 00:15:19,440
By default, if you do not set a user name, your container and your application in the container is going to run it as root.

195
00:15:19,440 --> 00:15:37,260
I have found so many who have deployed WordPress applications in a Docker container via Kubernetes and they don't set the user name and WordPress is running as root, so is the database and everything else.

196
00:15:37,260 --> 00:15:40,980
So of course, once it's compromised, you take over everything.

197
00:15:41,420 --> 00:15:43,560
Also, visibility.

198
00:15:43,780 --> 00:15:47,400
You need monitoring within your containers.

199
00:15:47,400 --> 00:15:53,620
Your containers put a lot of information out and they can log it, but you have to capture that.

200
00:15:53,620 --> 00:15:55,820
We're going to talk about that in a minute.

201
00:15:56,020 --> 00:16:00,020
Also, think of it as a domino.

202
00:16:00,020 --> 00:16:09,260
If one container is compromised, I guarantee you the rest of them will fall over like dominoes.

203
00:16:09,880 --> 00:16:14,520
When you start to get into containers and Kubernetes, you're going to see this.

204
00:16:14,920 --> 00:16:18,860
Also, we want to set some controls.

205
00:16:18,900 --> 00:16:33,780
Besides setting a user name that is not root, we also set privileges and we set certain attributes to those users where it says no elevated privileges or no new privileges.

206
00:16:33,780 --> 00:16:41,780
You don't want user accounts or IDs to be able to escalate privileges.

207
00:16:41,780 --> 00:16:44,040
And I'll show you some examples coming up.

208
00:16:44,040 --> 00:16:49,100
This one, my god, this is near and dear to my heart, secrets management.

209
00:16:49,100 --> 00:17:13,840
I still find, I would love to say by a show of hands, but I can't do that right here, how many people put the secrets and tokens and things like that in their Docker YAML files so when they then deploy the Docker image, if somebody gets a hold of their YAML file,

210
00:17:13,840 --> 00:17:21,240
which describes the configuration of that container, well, the secrets are right in it.

211
00:17:21,500 --> 00:17:39,300
I've worked with a lot of developers and a lot of engineers and I get them, you know, you have to start thinking about putting your secrets in another place, whether it's HashiCorp Vault, whether it's a cloud secrets management system, something.

212
00:17:39,300 --> 00:17:44,240
But get them out of the apps and get them out of the configuration files.

213
00:17:44,240 --> 00:17:48,220
And stop allowing SSH into your hosts.

214
00:17:48,220 --> 00:18:02,160
When we get into Kubernetes, coming up in a couple slides here, one of the key constructs of Kubernetes is you have a master node or master nodes, and then you have worker nodes.

215
00:18:02,160 --> 00:18:11,560
And I still find worker nodes with SSH enabled on them so people can SSH in and, quote, fix things.

216
00:18:11,560 --> 00:18:18,360
No, no, you use the management tools to fix things and redeploy.

217
00:18:18,360 --> 00:18:21,520
Stop allowing SSH into your systems.

218
00:18:21,520 --> 00:18:28,120
Also, I can't stress it enough times, namespace limits.

219
00:18:28,120 --> 00:18:33,800
Go read up on how to control namespaces.

220
00:18:33,800 --> 00:18:50,160
That is going to be one of your best efforts and best practices for containing and controlling and securing a container before you distribute it to or deploy it via Kubernetes.

221
00:18:50,160 --> 00:18:51,880
This is one of my favorite tools.

222
00:18:51,880 --> 00:18:53,700
It's called Docker Slim.

223
00:18:53,720 --> 00:19:11,020
If you've never heard of it, if I were to create a container, let's say I create a container, it has Ubuntu in it or the base of Ubuntu, and then maybe it has WordPress running in it for whatever god awful reason that I want to do this.

224
00:19:11,020 --> 00:19:17,780
Let's say that entire image ends up being three to four hundred meg.

225
00:19:17,780 --> 00:19:40,960
It's still manageable, but if I were to run that through Docker Slim, not only will it add some extra configurations and namespace limits and so on, but it shrinks it down and something that might be three or four hundred megabytes deployed into a container will get shrunk down to maybe 10,

226
00:19:40,960 --> 00:19:42,760
15, 20 meg.

227
00:19:42,760 --> 00:19:46,240
This makes it even that much more manageable.

228
00:19:46,240 --> 00:19:48,240
It's easier to deploy.

229
00:19:48,240 --> 00:19:55,260
It's quicker to deploy, especially if there are requirements for scaling.

230
00:19:55,260 --> 00:19:56,420
So think about this.

231
00:19:56,420 --> 00:19:57,540
Take a look at this.

232
00:19:57,540 --> 00:20:12,000
I also mentioned in Discord that my slides will be available in my GitHub repo, and I'll get them to the Red Team Village so they can add them to the Google Docs location after my talk.

233
00:20:13,260 --> 00:20:18,520
And always remember apply SELinux or AppArmor.

234
00:20:18,520 --> 00:20:23,120
One of the things that Docker Slim does is it will do that for you.

235
00:20:23,120 --> 00:20:28,740
It's very easy to enable SELinux or AppArmor within a container.

236
00:20:28,740 --> 00:20:32,580
If you can do that, you're going to make it that much more secure.

237
00:20:32,580 --> 00:20:36,960
It's not on by default, but you can set it up so it will be.

238
00:20:39,140 --> 00:20:44,040
And remember the infrastructure, as I put it.

239
00:20:44,640 --> 00:20:56,400
You can't forget that all of this stuff is being deployed on an infrastructure of some sort, and that infrastructure better be secure.

240
00:20:56,400 --> 00:20:58,040
I mentioned SSH.

241
00:20:58,040 --> 00:21:02,080
People keep allowing SSH into their environment.

242
00:21:02,440 --> 00:21:07,220
Be very aware that your infrastructure itself should be secure.

243
00:21:07,220 --> 00:21:09,860
And I will remind you later about that.

244
00:21:09,860 --> 00:21:10,180
All right.

245
00:21:10,180 --> 00:21:11,280
Here's a quick picture.

246
00:21:11,280 --> 00:21:13,500
Picture is worth a thousand words, right?

247
00:21:13,500 --> 00:21:18,960
If you look at our first example, this is our monolithic operating system here.

248
00:21:18,960 --> 00:21:20,720
We see hardware and operating system.

249
00:21:20,720 --> 00:21:21,820
We run apps.

250
00:21:21,820 --> 00:21:23,120
Here we see a VM.

251
00:21:23,120 --> 00:21:24,960
So we see hardware operating system.

252
00:21:24,960 --> 00:21:27,300
There's a hypervisor involved.

253
00:21:27,900 --> 00:21:30,440
And then there's going to be virtual machines.

254
00:21:30,440 --> 00:21:36,340
But when you have virtual machines, you have additional operating systems, you have additional libraries.

255
00:21:36,460 --> 00:21:38,500
They're very much isolated.

256
00:21:38,500 --> 00:21:46,820
I should have used a different picture where there's a big line through it that shows, yes, these are isolated and so on.

257
00:21:46,820 --> 00:21:49,780
But that should not have happened.

258
00:21:49,940 --> 00:21:50,300
Nope.

259
00:21:50,300 --> 00:21:51,380
Go back.

260
00:21:51,540 --> 00:21:52,560
Sorry.

261
00:21:52,620 --> 00:21:53,140
Okay.

262
00:21:53,140 --> 00:22:00,040
But when we look at using Docker or Container, we go hardware operating system.

263
00:22:00,040 --> 00:22:01,700
There is a container runtime.

264
00:22:01,700 --> 00:22:03,680
There's no real dividing lines.

265
00:22:03,680 --> 00:22:05,060
And then we see containers.

266
00:22:05,060 --> 00:22:13,800
The only thing that is unique between the containers is that bin library, the things that are going to run in the application.

267
00:22:13,800 --> 00:22:17,180
So there's not a lot of isolation there.

268
00:22:17,180 --> 00:22:21,360
And people want to think that there is.

269
00:22:21,360 --> 00:22:22,500
But there is not.

270
00:22:22,500 --> 00:22:24,660
So please keep that in mind.

271
00:22:25,260 --> 00:22:28,280
I do love this comic.

272
00:22:28,280 --> 00:22:30,500
It came out a couple of years ago.

273
00:22:30,500 --> 00:22:37,440
A great defense is, you know, we'll inundate them with quarterly Kubernetes releases.

274
00:22:37,440 --> 00:22:41,180
Yes, they do update Kubernetes a lot.

275
00:22:41,180 --> 00:22:49,960
That's why you have to pay attention to your environment and the tools that you're going to use so you can do upgrades to your Kubernetes cluster.

276
00:22:49,960 --> 00:22:52,320
So let's bring on the orchestrator.

277
00:22:52,320 --> 00:22:54,040
Let's talk about Kubernetes.

278
00:22:54,300 --> 00:22:54,800
All right.

279
00:22:54,800 --> 00:22:56,720
Here is Kubernetes 101.

280
00:22:56,720 --> 00:23:04,000
In one slide, I'm going to teach you at least the basics of how Kubernetes works.

281
00:23:04,000 --> 00:23:05,680
We have a master node.

282
00:23:05,680 --> 00:23:07,960
The master node is critical.

283
00:23:07,960 --> 00:23:12,000
It manages all of the Kubernetes worker nodes.

284
00:23:12,000 --> 00:23:13,700
It manages the clusters.

285
00:23:13,700 --> 00:23:15,760
It deploys pods.

286
00:23:15,760 --> 00:23:23,000
Think of a pod as a collection of containers that are grouped.

287
00:23:23,000 --> 00:23:31,900
A pod might be the way that you decide, oh, I want to have a pod that supports a particular client.

288
00:23:31,900 --> 00:23:34,500
Or maybe it's a WordPress instance.

289
00:23:34,500 --> 00:23:37,460
And that pod has WordPress.

290
00:23:37,460 --> 00:23:42,820
It has a container, a separate container for the database and so on.

291
00:23:42,820 --> 00:23:51,180
So the pod is how you can group things that are various containers into certain groupings.

292
00:23:51,180 --> 00:23:53,760
And those groupings are kind of defined by you.

293
00:23:53,760 --> 00:23:57,420
But stop and think about how you want to deploy pods.

294
00:23:57,420 --> 00:23:58,920
It's going to be important.

295
00:23:58,920 --> 00:24:00,420
A worker node.

296
00:24:00,420 --> 00:24:03,580
The node is where all the containers are going to run.

297
00:24:03,580 --> 00:24:08,080
You have other Kubernetes components that have to run there.

298
00:24:08,080 --> 00:24:11,940
But the important part is, this is where the containers run.

299
00:24:11,940 --> 00:24:28,020
If the worker node is not secure and not patched, and I can get into that underneath the visibility of Kubernetes, then Kubernetes has no idea that I've compromised a node.

300
00:24:28,020 --> 00:24:39,300
I found numerous Kubernetes sites around the Internet where there's additional applications running on worker nodes because they've been compromised.

301
00:24:39,300 --> 00:24:40,920
Of course, they're Bitcoin miners.

302
00:24:40,920 --> 00:24:54,900
But the master node has no idea that it's running on there because they got in via SSH and they launched the application, the miner, underneath the visibility of Kubernetes.

303
00:24:54,900 --> 00:24:58,420
So you really want to lock down these worker nodes.

304
00:24:58,420 --> 00:25:06,300
A pod, I already explained, it's a unit of deployment and it's also how you address things from Kubernetes.

305
00:25:06,300 --> 00:25:08,880
And a pod will have an IP address.

306
00:25:08,880 --> 00:25:12,000
So that's important to remember.

307
00:25:12,100 --> 00:25:15,080
But it's going to contain one or more containers.

308
00:25:15,080 --> 00:25:22,840
I like to think of pods as, even though it might have its own IP address, but think of it like a VLAN in a way.

309
00:25:22,840 --> 00:25:26,420
Because I've isolated it in one way.

310
00:25:26,440 --> 00:25:34,400
Services are various things that run within Kubernetes that communicate, that balance, that do a lot of work.

311
00:25:34,400 --> 00:25:36,540
We're going to look at services coming up.

312
00:25:36,540 --> 00:25:39,760
There are various components that are used.

313
00:25:39,760 --> 00:25:48,040
Three that are critical, the API server, you're going to have kubelets, and you're also etcd.

314
00:25:48,040 --> 00:26:01,380
The one concept you should take about etcd is this is the database that describes and builds everything that is your Kubernetes cluster.

315
00:26:01,380 --> 00:26:11,560
The etcd environment should actually be walled off and firewalled and controlled very, very securely.

316
00:26:11,560 --> 00:26:17,700
If you do that, you will have a much more secure Kubernetes cluster.

317
00:26:17,700 --> 00:26:19,760
So keep that in mind.

318
00:26:19,760 --> 00:26:23,040
And you can see it here by looking at a picture.

319
00:26:23,040 --> 00:26:33,660
So we take all of those concepts that we just talked about we see here's an ABI server, here's a controller, this is all part, you know, the kubectl...

320
00:26:33,660 --> 00:26:36,140
Stop doing that.

321
00:26:37,220 --> 00:26:38,000
Come on.

322
00:26:38,000 --> 00:26:42,640
Kubectl is the command you use to control everything.

323
00:26:43,020 --> 00:26:54,920
And even though this is a disk kind of a representation of etcd, I like to say, you know what, this needs to be off on its own, firewalled, and separated, and it can be done that way.

324
00:26:54,920 --> 00:27:00,820
Here's your worker node, and you can see how you might have multiple pods on a worker node.

325
00:27:00,820 --> 00:27:03,760
This pod might be talking to this pod.

326
00:27:03,760 --> 00:27:08,400
It doesn't necessarily have to talk to that one, or they may not talk at all.

327
00:27:08,400 --> 00:27:11,780
These are some of the other services that are running the proxies.

328
00:27:11,780 --> 00:27:16,680
This is where we are exposed to the internet, not over here.

329
00:27:16,680 --> 00:27:17,320
Okay?

330
00:27:17,320 --> 00:27:22,240
So the only place you're exposing Kubernetes is at the worker nodes.

331
00:27:22,780 --> 00:27:26,280
Now, Kubernetes is not perfect.

332
00:27:26,520 --> 00:27:28,780
There have been problems.

333
00:27:28,780 --> 00:27:29,980
There was...

334
00:27:29,980 --> 00:27:35,560
I need to update this slide again, because there have been a couple of other Kubernetes.

335
00:27:35,660 --> 00:27:37,600
These are more for reference.

336
00:27:37,600 --> 00:27:39,560
One's a little older.

337
00:27:39,560 --> 00:27:43,860
This was back... there was the backdoor that involved the API server.

338
00:27:43,860 --> 00:27:45,060
There were some others.

339
00:27:45,060 --> 00:27:49,620
These are only for your reference, so you can go look them up and see.

340
00:27:49,780 --> 00:27:51,620
And there are more .

341
00:27:51,620 --> 00:28:00,080
The whole point I'm trying to make here is Kubernetes is not secure by default, and it's not perfect .

342
00:28:00,680 --> 00:28:03,780
Well, let's try to make it perfect.

343
00:28:03,900 --> 00:28:17,460
Let's take a common Kubernetes cluster and say, you know, if we were to threat model this, and that is very, very important, first thing we want to do is to say, TLS everywhere.

344
00:28:17,920 --> 00:28:22,240
Yeah, I know, I have too many exclamation points, but I'm trying to make the point.

345
00:28:22,340 --> 00:28:31,460
If everything within your Kubernetes cluster is using TLS and is encrypted, then it's that much safer.

346
00:28:31,460 --> 00:28:38,000
I still have engineers come to me and say, our application is completely internal.

347
00:28:38,000 --> 00:28:39,740
Why do we need TLS?

348
00:28:39,740 --> 00:28:42,680
It's never going to be exposed to the internet.

349
00:28:42,680 --> 00:28:43,700
I said, really?

350
00:28:43,700 --> 00:28:57,800
So what happens if Joe's application over here, which is exposed to the internet, is compromised, and then they use that as a pivot point to gain access to this environment, and then they compromise your system?

351
00:28:58,120 --> 00:29:01,120
O is typically the response.

352
00:29:01,180 --> 00:29:05,180
So I can't say it enough, harden the bloody infrastructure.

353
00:29:05,320 --> 00:29:12,740
Yes, I'm saying it again, but you're running this thing on your infrastructure, make sure it's hardened.

354
00:29:12,740 --> 00:29:15,860
And very important, two concepts.

355
00:29:16,060 --> 00:29:20,940
There's RBAC, which is regular role-based access control.

356
00:29:20,940 --> 00:29:22,780
Set that up.

357
00:29:22,780 --> 00:29:25,220
Make sure you're doing it right.

358
00:29:25,240 --> 00:29:27,060
Disable ABAC.

359
00:29:27,180 --> 00:29:33,360
ABAC is incredibly difficult to manage.

360
00:29:33,600 --> 00:29:46,000
You will spend days and hours and hours trying to do ABAC access controls, which is more granular, but it's not worth the effort.

361
00:29:46,000 --> 00:29:50,060
Make sure you use RBAC and manage it correctly.

362
00:29:50,060 --> 00:29:55,560
And I'm going to show you a couple of tools coming up on how to check it.

363
00:29:55,560 --> 00:29:59,320
Also, check out this blog entry.

364
00:29:59,440 --> 00:30:03,340
I mentioned this previously about monitoring.

365
00:30:03,340 --> 00:30:08,640
This is one way to audit Kubernetes, so keep this in mind.

366
00:30:08,640 --> 00:30:19,520
Take a look at that blog entry from Wazoo, and it's an excellent place to start when it comes to monitoring your environment.

367
00:30:19,520 --> 00:30:24,560
Also, kubectl, get role binding in all your namespaces.

368
00:30:24,560 --> 00:30:35,040
You should do this periodically in your environment, especially if you have engineers deploying their own applications.

369
00:30:35,040 --> 00:30:42,300
Make sure that all the namespaces have good controls in their RBAC accounting.

370
00:30:42,300 --> 00:30:46,820
Also, use third-party authentication for the API server.

371
00:30:46,820 --> 00:30:49,020
Again, something like HashiCorp Vault.

372
00:30:49,020 --> 00:30:50,780
Yes, I love that tool.

373
00:30:50,780 --> 00:30:54,400
If you are not familiar with it, get familiar with it.

374
00:30:54,400 --> 00:31:00,360
It's a great tool, and it integrates very, very well with your API server and Kubernetes.

375
00:31:01,680 --> 00:31:03,500
I already mentioned this.

376
00:31:03,500 --> 00:31:05,420
Separate and firewall your etc cluster.

377
00:31:05,420 --> 00:31:07,580
Make sure it's protected.

378
00:31:07,580 --> 00:31:09,800
That is the keys to the kingdom.

379
00:31:09,880 --> 00:31:12,860
It's the entire configuration of your cluster.

380
00:31:12,860 --> 00:31:14,760
And rotate encryption keys.

381
00:31:14,760 --> 00:31:16,480
Yes, use Vault.

382
00:31:16,920 --> 00:31:22,860
And by default, turn on security policies in the pods.

383
00:31:22,880 --> 00:31:27,580
It's some simple lines of configuration in the YAML files.

384
00:31:27,580 --> 00:31:29,140
It's not hard to do.

385
00:31:29,140 --> 00:31:36,020
Or use Docker Slim, that tool that I mentioned earlier, because that will help you set this up.

386
00:31:37,020 --> 00:31:39,380
Alright, let's talk about risk.

387
00:31:39,620 --> 00:31:41,420
kubectl, get secrets.

388
00:31:41,420 --> 00:31:48,560
That is the one that I just mentioned where you can look at the secrets from pretty much everything.

389
00:31:48,560 --> 00:31:52,380
But here's some tools that I absolutely love by CyberArk.

390
00:31:52,380 --> 00:31:55,100
One is called Kubernetes RBAC audit.

391
00:31:55,100 --> 00:31:58,260
The other is called KubeScan.

392
00:31:58,260 --> 00:32:01,540
The RBAC audit does exactly what you think it does.

393
00:32:01,540 --> 00:32:04,660
It audits all of your RBAC roles.

394
00:32:04,660 --> 00:32:11,280
The KubeScan will look at all the permissions that are granted from the roles.

395
00:32:11,280 --> 00:32:19,680
These are two great tools that you want to put into your toolbox for securing Kubernetes clusters.

396
00:32:19,680 --> 00:32:24,700
Because remember at the very beginning, I said it's all about configurations.

397
00:32:24,700 --> 00:32:31,400
It's the number one area that Kubernetes clusters get compromised.

398
00:32:31,860 --> 00:32:36,260
Also, we want to take a look at GitHub.

399
00:32:36,260 --> 00:32:50,880
Go out and make sure that... take that away... go out and make sure that your own configurations are not being shared up to GitHub by your engineers accidentally.

400
00:32:51,080 --> 00:32:53,260
How many times does this happen?

401
00:32:53,260 --> 00:32:54,340
It happens.

402
00:32:54,340 --> 00:32:56,460
I've had it happen by engineers.

403
00:32:56,460 --> 00:32:58,580
They don't do it on purpose.

404
00:32:58,580 --> 00:32:59,980
But they upload code.

405
00:32:59,980 --> 00:33:01,700
And in the code are secrets.

406
00:33:01,920 --> 00:33:02,880
Oops.

407
00:33:03,080 --> 00:33:05,540
Oops is not a good thing.

408
00:33:05,760 --> 00:33:10,400
Also, you need a locked down secure image repo.

409
00:33:10,400 --> 00:33:12,640
And create gold images.

410
00:33:12,640 --> 00:33:22,120
Don't just let your engineers go out and do, hey, docker run bring down Ubuntu and pull it from the internet.

411
00:33:22,120 --> 00:33:22,680
No.

412
00:33:22,680 --> 00:33:27,340
You need your own repo and registry for your docker images.

413
00:33:27,340 --> 00:33:28,680
Make sure that happens.

414
00:33:28,680 --> 00:33:31,360
And please secure the libraries.

415
00:33:31,360 --> 00:33:33,000
That's another big one.

416
00:33:33,000 --> 00:33:34,720
We also...

417
00:33:34,720 --> 00:33:39,120
I need an entire slide on just the applications.

418
00:33:39,940 --> 00:33:41,800
I'm going out on a limb.

419
00:33:41,800 --> 00:33:43,840
This is my personal opinion.

420
00:33:43,840 --> 00:33:48,600
But I want to say that nine times out of ten... okay, maybe eight and a half.

421
00:33:48,600 --> 00:34:02,860
Eight and a half times out of ten Kubernetes clusters and containers are compromised because the applications themselves were poorly written and not tested.

422
00:34:02,980 --> 00:34:05,680
And that's what gets me.

423
00:34:05,680 --> 00:34:11,740
I have environments where I see the engineers create an application.

424
00:34:11,740 --> 00:34:23,260
They do so much work to lock down Kubernetes in the container, but they do a poor job and there's a SQL injection vulnerability in the application directly.

425
00:34:23,360 --> 00:34:33,980
So then the application is compromised and then the container is compromised because, oops, it was running as root and so on and so on.

426
00:34:33,980 --> 00:34:35,500
Remember the dominoes.

427
00:34:35,500 --> 00:34:37,860
Everything falls over.

428
00:34:38,000 --> 00:34:41,160
Also, remember your pods.

429
00:34:41,960 --> 00:34:56,180
Another common mistake that people make when they're new to Kubernetes is they don't stop and think, how can I group things into pods and secure those pods?

430
00:34:56,180 --> 00:35:04,220
At least if a container falls within a pod, then maybe there's a couple more containers in there.

431
00:35:04,220 --> 00:35:10,940
Those containers that get compromised are going to be somewhat limited within the pod itself.

432
00:35:11,080 --> 00:35:16,060
Typically a pod is not going to fall over when another pod falls over.

433
00:35:16,060 --> 00:35:18,240
It's usually containers within the pods.

434
00:35:18,240 --> 00:35:21,400
So you want to think about grouping things.

435
00:35:21,400 --> 00:35:26,660
Also, Jason mentioned this in his talk earlier, but showdown.

436
00:35:26,660 --> 00:35:30,940
If you have never used showdown, get familiar with it.

437
00:35:30,940 --> 00:35:32,520
Even for Kubernetes.

438
00:35:32,520 --> 00:35:34,520
You can go out to showdown.

439
00:35:34,520 --> 00:35:37,520
You can start searching for Kubernetes.

440
00:35:37,520 --> 00:35:41,660
You can search for API clusters or API servers.

441
00:35:41,660 --> 00:35:45,480
You can search for etcd servers.

442
00:35:45,480 --> 00:35:51,080
And if you find one of yours, oops, but they're out there.

443
00:35:51,080 --> 00:36:03,560
I did a search this morning for some etcd exposure and there were over 6,000 that I found that were exposed.

444
00:36:03,560 --> 00:36:13,440
I don't know how many of those have vulnerabilities that would have allowed me to compromise them because I was just curious what the number would be.

445
00:36:13,560 --> 00:36:18,760
It goes up and down on a daily basis, but you should be aware of that.

446
00:36:18,760 --> 00:36:20,340
So use showdown.

447
00:36:20,640 --> 00:36:22,880
Let's talk about the configurations.

448
00:36:24,400 --> 00:36:25,860
I've said this before.

449
00:36:25,860 --> 00:36:29,380
Most of the breaches come from configuration issues.

450
00:36:32,890 --> 00:36:35,450
Use CIS hardening.

451
00:36:35,450 --> 00:36:37,110
Center for Internet Security.

452
00:36:37,110 --> 00:36:38,750
It is an amazing place.

453
00:36:38,750 --> 00:36:43,070
It is done by volunteers and workers within CIS.

454
00:36:43,070 --> 00:36:48,670
You can get benchmarks for Docker and you can get benchmarks for Kubernetes.

455
00:36:48,670 --> 00:36:54,750
You can use these to configure your environment so they are indeed hardened.

456
00:36:55,710 --> 00:36:57,390
Stop ignoring them.

457
00:36:57,390 --> 00:37:08,190
I still see people that tell me they've never heard of CIS and it makes me very sad when I hear that because it is such a great resource.

458
00:37:08,190 --> 00:37:20,610
The whole point here is we have to harden the environment from the standpoint of the basic configuration, but don't forget the application.

459
00:37:20,650 --> 00:37:29,370
Also, I know a lot of people have environments that are CICD when it comes to Docker and Kubernetes.

460
00:37:29,370 --> 00:37:30,690
That's why we have them.

461
00:37:30,690 --> 00:37:31,710
It's for speed.

462
00:37:31,710 --> 00:37:32,830
It's for development.

463
00:37:32,830 --> 00:37:35,790
It's for putting things out quickly.

464
00:37:35,790 --> 00:37:36,950
Okay, great.

465
00:37:36,950 --> 00:37:38,930
A lot of people use Jenkins.

466
00:37:39,690 --> 00:37:51,310
Is there some way to automate all of this so at least I can look at my images, I can look at my containers, all within the CICD pipeline?

467
00:37:51,310 --> 00:37:52,750
Yes, there is.

468
00:37:52,750 --> 00:37:56,450
There is a great tool called Anchor.

469
00:37:56,450 --> 00:38:01,990
Anchor, as it says here, is a container image scanner plugin.

470
00:38:01,990 --> 00:38:05,590
And what it does is you put it into the pipeline.

471
00:38:05,590 --> 00:38:07,490
It's a plugin for Jenkins.

472
00:38:07,490 --> 00:38:13,510
And it actually does security scans against the images that you're trying to deploy.

473
00:38:13,510 --> 00:38:21,550
This is a great way to get testing in before you actually deploy it to production.

474
00:38:21,970 --> 00:38:26,170
So let's talk about some additional tools before we get into the pen testing.

475
00:38:26,170 --> 00:38:28,410
Because we're still trying to lock it down.

476
00:38:28,430 --> 00:38:29,270
KubeBench.

477
00:38:29,270 --> 00:38:31,250
We want to run KubeBench.

478
00:38:31,250 --> 00:38:35,910
What it does is it runs against your CIS benchmarks.

479
00:38:35,910 --> 00:38:36,790
It displays

480
00:38:39,870 --> 00:38:42,830
a configuration that shows you what has passed, what has failed.

481
00:38:42,830 --> 00:38:44,670
I'll show an example in a minute.

482
00:38:44,670 --> 00:38:46,210
There's also KubeAudit.

483
00:38:46,210 --> 00:38:49,170
It's a little more granular than KubeSec.

484
00:38:49,410 --> 00:38:58,250
And if you run KubeBench and KubeAudit, it will give you a report that shows you oops, my system is insecure.

485
00:38:58,250 --> 00:39:00,770
Here's an example of a report.

486
00:39:00,770 --> 00:39:06,750
And when we look at it, we see allow privilege escalation is not set.

487
00:39:06,750 --> 00:39:09,130
There's a very first one.

488
00:39:09,130 --> 00:39:11,870
Oops, that needs to be turned on.

489
00:39:11,870 --> 00:39:18,150
Remember, that was the one that I mentioned that allows users to do privilege escalation.

490
00:39:18,150 --> 00:39:19,410
We don't want that.

491
00:39:19,410 --> 00:39:20,030
Oh, crap.

492
00:39:20,030 --> 00:39:22,650
We have run as non-root is not set.

493
00:39:22,650 --> 00:39:26,210
We don't have a read only file system set.

494
00:39:26,210 --> 00:39:30,410
There are so many simple things that can be done.

495
00:39:30,410 --> 00:39:33,970
Again, no app armor, no security comp.

496
00:39:33,970 --> 00:39:35,830
It's not being turned on.

497
00:39:35,830 --> 00:39:38,590
These are things that are just so easy.

498
00:39:38,590 --> 00:39:46,850
And this is from every single container within the environment that is being deployed under this particular.

499
00:39:46,850 --> 00:39:48,870
It's much longer.

500
00:39:48,910 --> 00:39:49,830
It goes into it.

501
00:39:49,830 --> 00:39:51,630
This is just a quick snapshot.

502
00:39:51,630 --> 00:39:57,470
The whole point here is you run this against your environment before you deploy it.

503
00:39:57,470 --> 00:40:02,970
And you go, oops, let's go back to our configuration file and fix all these things.

504
00:40:03,670 --> 00:40:07,090
Here are some more tools for you in the environment.

505
00:40:07,090 --> 00:40:13,030
Claire, another great tool for doing static analysis of applications.

506
00:40:13,370 --> 00:40:20,690
Claire does, it integrates Claire and your Docker registry so it can automate that a little bit.

507
00:40:20,690 --> 00:40:28,650
Falco, I need to put huge stars next to this one because this is actually one of my favorite, favorite tools.

508
00:40:29,330 --> 00:40:35,810
It was developed years ago and then it kind of morphed off and became falco.org.

509
00:40:35,810 --> 00:40:39,740
It monitors behavior within containers.

510
00:40:39,740 --> 00:40:47,320
It can detect if a container has been compromised because suddenly the container starts acting weird.

511
00:40:47,320 --> 00:40:54,140
Or there are commands being executed inside the container like cat etsy password.

512
00:40:54,240 --> 00:40:58,980
Probably not a command that should be showing up in one of your containers.

513
00:40:59,220 --> 00:41:07,740
If anything of this entire talk that I'm giving today, you get out of this, go look at falco.

514
00:41:07,740 --> 00:41:10,480
This is one of the best tools out there.

515
00:41:10,760 --> 00:41:13,860
Kubehunter is another great tool.

516
00:41:13,860 --> 00:41:15,280
We're going to talk about that.

517
00:41:15,540 --> 00:41:19,640
It's used more so for looking at doing some pen testing.

518
00:41:19,640 --> 00:41:21,920
We're going to take a look at that in just a second.

519
00:41:22,080 --> 00:41:24,140
Alright, ready, set, attack.

520
00:41:24,140 --> 00:41:34,480
If we were going to pen test an environment, this is an example of all the ports and all of the things that are open in a Kubernetes cluster.

521
00:41:34,480 --> 00:41:39,800
Now, they're open, but are they open by firewalls and so on?

522
00:41:39,800 --> 00:41:41,320
Well, we don't know.

523
00:41:42,160 --> 00:41:47,680
The API server itself is, yes, it's HTTPS.

524
00:41:47,680 --> 00:41:50,140
It better damn well be protected.

525
00:41:51,720 --> 00:41:57,560
etcd on 6666, that should be protected wherever it happens to be.

526
00:41:57,560 --> 00:41:59,140
This just gives you an idea.

527
00:41:59,140 --> 00:42:01,100
All of these ports can be changed.

528
00:42:01,100 --> 00:42:03,820
These are all the defaults in most cases.

529
00:42:04,360 --> 00:42:13,280
Just be aware of the ports that are out there and exposed that you need to make sure are protected.

530
00:42:13,280 --> 00:42:16,300
Because as an attacker, what am I going to do?

531
00:42:16,300 --> 00:42:24,360
Well, one of the first things, even Jason was talking about it in his talk, looking at ports, running nmap, running mass scan, and so on.

532
00:42:25,040 --> 00:42:28,540
This is exactly one of the things that Shodan does.

533
00:42:28,540 --> 00:42:32,400
It looks for all of these things that are exposed on the internet.

534
00:42:32,400 --> 00:42:37,280
So please make sure you're aware of all the ports within your cluster.

535
00:42:37,900 --> 00:42:39,920
Now, I are a hacker.

536
00:42:39,920 --> 00:42:41,180
See me hack.

537
00:42:42,340 --> 00:42:50,320
This is one of the ways that I'm going to start to attack a Kubernetes cluster.

538
00:42:50,320 --> 00:42:57,540
I want to first and foremost check access to the API server, make sure that is protected.

539
00:42:57,540 --> 00:43:01,880
I'm going to be looking at etcd, and making sure that's protected.

540
00:43:01,880 --> 00:43:06,640
If I find it exposed, I'm going to get keys to the kingdom in that case.

541
00:43:06,640 --> 00:43:12,700
What about the kubelets that are communicating and the command channels that are there?

542
00:43:12,700 --> 00:43:17,720
They should be read-only, but they should not be exposed.

543
00:43:18,120 --> 00:43:19,420
There was actually

544
00:43:22,440 --> 00:43:26,300
a vulnerability exposed about this, I want to say two months ago.

545
00:43:26,300 --> 00:43:28,680
I'm probably off in that sense.

546
00:43:28,760 --> 00:43:30,220
But take a look at that.

547
00:43:30,220 --> 00:43:39,980
And it allowed, even though it's a read -only port and it's supposed to limit what you can see, there were some vulnerabilities that allowed for a lot of exposure there.

548
00:43:40,020 --> 00:43:42,740
Also , container vulnerabilities.

549
00:43:43,640 --> 00:43:53,300
What about, you know, do you have files in that container that the application needs that you should be protecting?

550
00:43:53,300 --> 00:43:58,760
Well, that's where I would set up Falco and tell Falco, hey, there's a special file over here.

551
00:43:58,760 --> 00:44:00,900
This application is using it.

552
00:44:00,960 --> 00:44:02,380
But keep an eye on that.

553
00:44:02,380 --> 00:44:12,700
Make sure that the only thing that is reading that application is, or excuse me, that is reading that file happens to be the application.

554
00:44:12,700 --> 00:44:15,160
And Falco can monitor that.

555
00:44:15,400 --> 00:44:17,160
Privilege escalation.

556
00:44:17,160 --> 00:44:27,240
Nine times out of ten, once somebody compromises an application in a container, they elevate privilege because there's some other bugs in there.

557
00:44:27,240 --> 00:44:36,580
Go back to the Kubotic slide where it showed privilege escalation was not set.

558
00:44:36,580 --> 00:44:40,680
All you have to do is set that configuration variable.

559
00:44:40,680 --> 00:44:42,760
It's not that hard.

560
00:44:43,040 --> 00:44:43,760
Okay?

561
00:44:43,760 --> 00:44:46,840
Vulnerable applications are critical.

562
00:44:47,360 --> 00:44:49,640
Can't say it enough times.

563
00:44:49,640 --> 00:44:57,500
Especially because there are so many people using Kubernetes and Docker to deploy WordPress.

564
00:44:57,560 --> 00:45:02,280
And WordPress is, of course, one of our favorite things to deal with.

565
00:45:02,420 --> 00:45:03,180
All right.

566
00:45:03,180 --> 00:45:14,480
So your final threat model, when it comes to pentesting and looking at the environment that is Kubernetes, you think of your workload.

567
00:45:14,480 --> 00:45:18,080
You think of that security within your pods.

568
00:45:18,080 --> 00:45:23,700
Can I set up my pods so they're indeed secure?

569
00:45:23,700 --> 00:45:33,940
The container security itself, the configuration for that container, the more hooks I put in, run as root, no.

570
00:45:33,940 --> 00:45:36,600
Privilege escalation, no.

571
00:45:36,600 --> 00:45:40,160
These are the things that we want to make sure that are set.

572
00:45:40,160 --> 00:45:43,300
Also, file storage security.

573
00:45:43,300 --> 00:45:51,120
Maybe there's a particular file system that has to be read right but the rest of the file system can be read only.

574
00:45:51,120 --> 00:45:52,660
Those are other things.

575
00:45:52,660 --> 00:45:54,260
The network security.

576
00:45:54,260 --> 00:45:56,500
Remember the infrastructure.

577
00:45:56,520 --> 00:45:58,240
Put firewalls in.

578
00:45:58,240 --> 00:46:03,320
Put controls in there that detect lateral movement.

579
00:46:03,320 --> 00:46:10,640
If somebody does compromise one of your applications, make sure you have something monitoring.

580
00:46:10,640 --> 00:46:15,680
I gave you that slide way back about the blog entry from Wazoo.

581
00:46:15,680 --> 00:46:22,340
They talk about how to audit and monitor a Kubernetes cluster using their tool.

582
00:46:22,340 --> 00:46:25,480
I love the Wazoo manager and agent.

583
00:46:25,480 --> 00:46:28,640
Take a look at it if you haven't heard of it before.

584
00:46:29,700 --> 00:46:33,460
Remember, most breaches are not zero day.

585
00:46:33,460 --> 00:46:34,840
I can't say it enough times.

586
00:46:34,840 --> 00:46:38,040
Everybody's always trying to protect against this stuff.

587
00:46:38,040 --> 00:46:39,600
They're not fancy.

588
00:46:39,600 --> 00:46:45,620
Most of these things are being broken into because people make silly mistakes.

589
00:46:45,620 --> 00:46:47,540
Because remember, common sense.

590
00:46:47,540 --> 00:46:49,040
Whatever happened to it.

591
00:46:49,040 --> 00:46:52,780
The breaches are going to come from configuration issues.

592
00:46:52,780 --> 00:47:00,680
The kube audit was a very good example where it was showing all the problems that would allow someone to get in.

593
00:47:01,220 --> 00:47:03,180
Compromise credentials.

594
00:47:03,180 --> 00:47:08,880
Make sure your credentials are stored in a secure place somewhere.

595
00:47:09,200 --> 00:47:13,020
And don't allow for privilege escalation.

596
00:47:13,020 --> 00:47:18,980
Kubernetes and Docker give you the tools and the configurations to prevent that.

597
00:47:18,980 --> 00:47:20,640
Just turn them on.

598
00:47:20,640 --> 00:47:22,700
And that's what we're not doing.

599
00:47:22,700 --> 00:47:26,160
We have to turn on the configurations.

600
00:47:26,160 --> 00:47:29,160
So your key takeaways from today.

601
00:47:29,560 --> 00:47:31,140
Common sense.

602
00:47:31,140 --> 00:47:40,240
Please go back and stop and think about where is the exposure?

603
00:47:40,240 --> 00:47:42,640
Where are my pods exposed?

604
00:47:42,640 --> 00:47:44,660
What about my etcd?

605
00:47:45,240 --> 00:47:46,260
Configuration.

606
00:47:46,260 --> 00:47:49,240
That is so critical to any Kubernetes.

607
00:47:49,240 --> 00:47:52,200
And always remember containers.

608
00:47:52,200 --> 00:47:57,020
Containers are the linchpin of a Kubernetes environment.

609
00:47:57,020 --> 00:48:08,460
And if I don't secure my individual containers then Kubernetes... I don't care about the security of Kubernetes because the container is insecure.

610
00:48:08,460 --> 00:48:14,480
So stop overthinking this stuff and think about security basics.

611
00:48:14,480 --> 00:48:17,420
Remember the CIS benchmarks.

612
00:48:17,860 --> 00:48:19,680
Can't say it enough.

613
00:48:19,680 --> 00:48:20,540
Basics.

614
00:48:20,540 --> 00:48:27,320
I still think as security professionals hackers, whatever we want to call ourselves.

615
00:48:27,420 --> 00:48:32,340
We have forgotten the basics of how to secure an environment.

616
00:48:32,340 --> 00:48:35,780
And that's why things keep getting broken into.

617
00:48:35,780 --> 00:48:38,200
Stop doing fancy crap.

618
00:48:38,200 --> 00:48:40,050
The fancy crap doesn't work.

619
00:48:41,460 --> 00:48:43,940
Secure the environment, please.

620
00:48:44,020 --> 00:48:51,360
Use the CIS benchmarks for the containers, for Kubernetes, for the OS that your stuff runs on.

621
00:48:51,360 --> 00:48:58,760
If you're going to deploy this whole Kubernetes cluster on Ubuntu, then pull up the CIS benchmarks for Ubuntu.

622
00:48:58,760 --> 00:49:01,740
Or Red Hat or CentOS.

623
00:49:01,740 --> 00:49:04,040
Whatever you're using.

624
00:49:04,040 --> 00:49:07,620
Remember, RBAC is your friend.

625
00:49:07,620 --> 00:49:09,580
ABAC is not.

626
00:49:10,100 --> 00:49:12,880
Your infrastructure better be secure.

627
00:49:12,880 --> 00:49:15,760
Your pods better be secure.

628
00:49:15,760 --> 00:49:18,780
Containers themselves, the network.

629
00:49:18,780 --> 00:49:24,160
Still a lot of people forget the network because they forget about TLS everywhere.

630
00:49:26,880 --> 00:49:30,260
Threat model your applications.

631
00:49:30,260 --> 00:49:39,160
If you don't take the beginning effort to threat model your applications, you end up with insecure applications.

632
00:49:39,160 --> 00:49:46,940
And all the security of your Kubernetes cluster is useless because you have insecure applications within your environment.

633
00:49:46,940 --> 00:49:56,460
Access controls go along with RBAC, but access controls into the environment, kind of referencing SSH.

634
00:49:56,920 --> 00:50:03,080
I found one worker node configuration in an environment probably nine months ago.

635
00:50:03,080 --> 00:50:08,700
But for some reason on the worker nodes, they had FTP enabled on all the worker nodes.

636
00:50:08,820 --> 00:50:12,660
And my head exploded.

637
00:50:13,180 --> 00:50:16,540
It allowed me to get into everything, but oh well.

638
00:50:17,380 --> 00:50:19,920
Yes, did I mention RBAC though?

639
00:50:20,220 --> 00:50:21,360
Enough times.

640
00:50:21,360 --> 00:50:22,640
I can't say it enough.

641
00:50:22,640 --> 00:50:23,100
Why?

642
00:50:23,100 --> 00:50:25,180
Because people tend to ignore it.

643
00:50:25,180 --> 00:50:33,780
Remember the tools I mentioned earlier about monitoring or auditing the RBAC roles and also the permissions.

644
00:50:34,700 --> 00:50:39,900
And finally, not quite finally, remember to patch everything.

645
00:50:39,900 --> 00:50:43,820
The infrastructure that goes along with that.

646
00:50:43,860 --> 00:50:46,760
Logging and auditing and monitoring.

647
00:50:46,760 --> 00:50:49,620
I still see people set up their environments.

648
00:50:49,620 --> 00:50:52,700
There's that blog entry again.

649
00:50:52,700 --> 00:50:54,560
Please take a look at that.

650
00:50:54,560 --> 00:50:56,040
It is well written.

651
00:50:56,040 --> 00:50:57,640
It's a great tool.

652
00:50:57,640 --> 00:50:59,060
You'll learn a lot about Wazoo.

653
00:50:59,060 --> 00:51:01,680
I use it on all my systems.

654
00:51:01,880 --> 00:51:06,700
I deploy a lot of honeypots with Kubernetes and containers and things like that.

655
00:51:06,700 --> 00:51:10,360
But I monitor them all with Wazoo.

656
00:51:10,960 --> 00:51:16,540
You can't have a secure environment if you're not monitoring that environment.

657
00:51:16,540 --> 00:51:18,360
That is the biggest thing.

658
00:51:19,860 --> 00:51:22,220
That brings me to the end.

659
00:51:22,400 --> 00:51:25,100
I thank you for your attention.

660
00:51:25,880 --> 00:51:27,600
There's contact info.

661
00:51:27,600 --> 00:51:28,960
There's my Twitter account.

662
00:51:28,960 --> 00:51:35,340
I will get this to the Red Team Village and they can upload it to their environment.

663
00:51:35,340 --> 00:51:41,880
But I'll have it on my GitHub later in an hour or so.

664
00:51:41,880 --> 00:51:43,300
I'll update it there.

665
00:51:43,300 --> 00:51:44,360
Thank you all very much.

666
00:51:44,360 --> 00:51:45,940
Hopefully you learned something today.

667
00:51:45,940 --> 00:51:47,700
Go out and hack and have some fun.

668
00:51:47,700 --> 00:51:51,480
Thanks to the Red Team Village and have an awesome DEF CON.

669
00:51:51,480 --> 00:51:52,940
Awesome.

670
00:51:52,940 --> 00:51:54,740
Thank you so much, Kat.

671
00:51:54,740 --> 00:51:56,760
Amazing presentation as always.

672
00:51:56,940 --> 00:51:58,600
Thank you for supporting DEF CON.

673
00:51:58,600 --> 00:52:00,220
Thank you for supporting the community.

674
00:52:00,220 --> 00:52:01,780
Of course, the Red Team Village.

675
00:52:01,780 --> 00:52:13,660
For those of you that just joined or are looking at the presentation on Twitch or YouTube, please look at all the activities that we have going on.

676
00:52:13,660 --> 00:52:16,340
Go to the website that is in the description of the stream.

677
00:52:16,340 --> 00:52:19,540
Please also join the conversation in Discord.

678
00:52:19,540 --> 00:52:23,760
We're going to go on a brief break and the next presentation will start in a few minutes.

679
00:52:23,760 --> 00:52:25,340
Thank you again and thank you, Kat.