1
00:00:00,000 --> 00:00:05,000
your support for supporting the DEF CON community and the Red Team Village specifically.

2
00:00:05,000 --> 00:00:06,940
We're looking forward to your presentation.

3
00:00:06,940 --> 00:00:08,060
So take it away.

4
00:00:08,260 --> 00:00:12,200
Thank you so much, Omer, for introducing and having me here.

5
00:00:12,300 --> 00:00:14,800
Let me just share the screen and confirm.

6
00:00:15,860 --> 00:00:17,420
Can you see the screen?

7
00:00:20,450 --> 00:00:21,810
Yes, you're all set.

8
00:00:21,810 --> 00:00:23,050
Awesome, perfect.

9
00:00:23,190 --> 00:00:25,630
Thank you everyone for joining me.

10
00:00:25,630 --> 00:00:39,990
And first of all, to just give some context, thank you so much, DEF CON Red Team Village, especially in this pandemic times to make it virtual and people can enjoy these talks without any physical intervention.

11
00:00:40,210 --> 00:00:44,190
And this talk is about the Kubernetes code.

12
00:00:44,190 --> 00:00:49,430
So disclaimer, basically, today's we will not see any slides, basically.

13
00:00:49,490 --> 00:00:51,690
And it's mostly, everything is in online.

14
00:00:51,690 --> 00:00:56,850
You can play along with me, or if you can go back and try yourself, you can go ahead and try as well.

15
00:00:56,850 --> 00:01:01,510
So there will be no slides, mostly live demos and the hands-on exercises.

16
00:01:02,050 --> 00:01:02,790
Cool.

17
00:01:02,790 --> 00:01:14,150
So this talk is primarily focusing on introducing a project recently I have created called Kubernetes Code, which is, this is in github.madhuvakula.kubernetescode.

18
00:01:14,150 --> 00:01:31,430
This project was mainly intentionally vulnerable application, which is to basically try and teach participants, or like anyone come to the new Kubernetes to understand how you can assess these Kubernetes clusters and find vulnerabilities so that you can go back and try it out in your organizations,

19
00:01:31,430 --> 00:01:34,550
or day-to-day pen tests, which you find are a team exercises, right?

20
00:01:34,550 --> 00:01:39,210
So I have presented at some of the places and also I have created a playground.

21
00:01:39,210 --> 00:01:43,290
If you wanted to write now, try it in your browser and follow along, right?

22
00:01:43,670 --> 00:01:46,770
So a little bit about me, myself, Madhu Akula.

23
00:01:46,770 --> 00:01:50,470
I work at currently Zibia as a Cloud Native Security Specialist.

24
00:01:50,470 --> 00:01:57,890
And I have been speaking and training at a bunch of conferences around the world, like DEF CON Black Hat, USENIX, Orally, and some of them.

25
00:01:58,150 --> 00:02:08,070
And I'm fortunate to find some of the vulnerabilities long back using my research in some of the organizations like Microsoft, LinkedIn, Google, and some of them.

26
00:02:08,270 --> 00:02:16,810
So apart from that, my primary interest mostly working around security of cloud, containers, Kubernetes, and automating the infrastructure part.

27
00:02:16,910 --> 00:02:19,450
So that's a little bit about me.

28
00:02:19,450 --> 00:02:24,710
If you wanted to follow me, you can follow me at Madhu Akula in most of the social media stuff.

29
00:02:25,610 --> 00:02:26,510
Cool.

30
00:02:26,510 --> 00:02:33,930
Let's get into what exactly is this Kubernetes code and what we are trying to learn like within this next one hour, right?

31
00:02:34,010 --> 00:02:38,490
So to give a little context about Kubernetes code.

32
00:02:38,530 --> 00:02:40,550
So as I said, it is intentionally vulnerable.

33
00:02:40,550 --> 00:02:47,110
I think as most of you have come across the different types of code, like web code, or Terra code, or some other codes.

34
00:02:47,110 --> 00:02:54,270
So this is a similar project, but it is mostly focusing on Kubernetes to understand what are the known vulnerabilities.

35
00:02:54,270 --> 00:03:03,990
And also we have seen common vulnerabilities in the real world environments, like while performing penetration testing, or like while performing assessments in Kubernetes environments.

36
00:03:04,010 --> 00:03:06,710
From those we have taken as scenarios.

37
00:03:06,710 --> 00:03:08,690
And these are some of the scenarios.

38
00:03:08,690 --> 00:03:10,410
Let me zoom a bit.

39
00:03:10,410 --> 00:03:12,730
I think maybe easier to see.

40
00:03:12,730 --> 00:03:13,330
Yeah.

41
00:03:13,330 --> 00:03:16,170
So these are some of the scenarios I have created.

42
00:03:16,170 --> 00:03:26,110
And we will see some of the scenarios, exploitation, and how you can bypass or gain access to Kubernetes clusters as an attacker point, right?

43
00:03:26,110 --> 00:03:28,950
And I'm really sorry, and I'm shameful.

44
00:03:29,070 --> 00:03:33,490
I said I will keep updating these slides at the documentation.

45
00:03:33,530 --> 00:03:36,570
And also I have been working on the defense part also.

46
00:03:36,570 --> 00:03:39,030
Hopefully this month we will see them coming away.

47
00:03:39,530 --> 00:03:41,110
This part I'm going to skip.

48
00:03:42,130 --> 00:04:01,090
So before diving into Kubernetes, if you are really completely new and coming to the containers or Kubernetes world, I highly recommend understanding the technology because there is a fair amount of gap in between penetration testing or finding or assessing clusters if you don't understand what technology you are doing,

49
00:04:01,090 --> 00:04:01,690
right?

50
00:04:01,690 --> 00:04:13,130
So we have seen commonly, especially in the modern tech infrastructure world, there is a huge gap for the security team to understand the technology before even testing or trying out.

51
00:04:13,130 --> 00:04:21,330
Because once you are able to understand, you can be able to do your best or you can be able to possibly find most of the Kubernetes or any ecosystem.

52
00:04:21,550 --> 00:04:28,230
This is a very funny video and it's a simple children's guide to explain what exactly Kubernetes is.

53
00:04:28,230 --> 00:04:31,950
So I would highly recommend if you want to go through these things.

54
00:04:31,950 --> 00:04:38,050
So to give you a little quick context, how Kubernetes architecture looks like at a higher level, right?

55
00:04:38,050 --> 00:04:40,850
So this is again from the wiki comments.

56
00:04:40,850 --> 00:04:46,890
You see a server and client architecture similar to most of the server client.

57
00:04:47,090 --> 00:04:57,710
So one thing you have to observe here is in Kubernetes, everything which you make, any operation, it may be from within the master, which means server, within the node, which means worker node.

58
00:04:57,710 --> 00:05:08,390
So whatever the operations which will make, like it can be from the node, it can be from the developer or any user, any service will go and touch something called API server.

59
00:05:08,390 --> 00:05:15,670
So which is one of the key component in terms of Kubernetes, because Kubernetes is completely driven by API first approach.

60
00:05:15,670 --> 00:05:21,550
So whatever you make all requests or something, everything you will see going through the API server, right?

61
00:05:21,550 --> 00:05:26,610
And control manager is basically to make the decisions, whatever the requests you make.

62
00:05:26,610 --> 00:05:33,090
And scheduler is ensuring to place the workloads or the request which you made.

63
00:05:33,090 --> 00:05:40,570
Suppose if you wanted to serve a Nginx server with 10 pods, it will ensure that how many nodes has the capacity to schedule.

64
00:05:40,570 --> 00:05:44,190
So the scheduler job is to figure out that and place them accordingly.

65
00:05:44,310 --> 00:05:50,790
So etcd is one of the key value store database, which is used by Kubernetes, which is as its name suggests, key value.

66
00:05:50,790 --> 00:05:56,910
So it stores the data, which is coming to the API server and stores the key value base.

67
00:05:56,910 --> 00:06:00,870
And we will see a little bit in a later coming sections.

68
00:06:01,650 --> 00:06:10,730
So apart from that nodes, when you come to the node, kubelet is the talking point to the API server and communicates within the local node itself.

69
00:06:10,730 --> 00:06:22,650
So whenever you wanted to make a request as a scheduler, you wanted to schedule in the node, it will talk to the container runtime, which is within the node, like it can be Docker, RunC, etcd.

70
00:06:22,650 --> 00:06:28,430
So then it will go ahead and schedule based on the workload kind of thing.

71
00:06:28,610 --> 00:06:39,230
So kubeproxy is one of the another key component, which is kind of like IP tables, like it will try to provide a communication across like the nodes as well as a cluster.

72
00:06:39,810 --> 00:06:51,370
So let's go ahead and a little bit go deep, but I would highly recommend you practice the Kubernetes before you diving into the assessments or security testing.

73
00:06:51,370 --> 00:06:54,390
So here are some of the references I have curated.

74
00:06:54,390 --> 00:07:06,190
These are pretty hands-on, like if you wanted to try, so you can go ahead and try as a literally playground and also specific objects if you start while testing, you can click and understand what exactly it is.

75
00:07:06,190 --> 00:07:10,670
So these references I would, I think it would be definitely helpful to try.

76
00:07:11,130 --> 00:07:12,090
Cool.

77
00:07:12,490 --> 00:07:14,370
Setting up Kubernetes code.

78
00:07:14,610 --> 00:07:18,690
The Kubernetes code is in design in such a way it can set up any out of the box.

79
00:07:18,690 --> 00:07:23,990
It can be in your existing cloud provider services like GKE, EKS, AKS, or digitalization based.

80
00:07:24,350 --> 00:07:27,750
And also your locally provisioned Minikube or another playground.

81
00:07:27,750 --> 00:07:33,990
To make easy of setup for the people to try out and learn, I have created something called Katakoda Playground.

82
00:07:33,990 --> 00:07:39,950
If you just go ahead and click this, it will open up a playground for you, which is completely free and online.

83
00:07:39,950 --> 00:07:41,790
So you don't need anything to be done.

84
00:07:41,790 --> 00:07:43,450
I think you have to register first time.

85
00:07:43,450 --> 00:07:48,290
So then you can get the complete setup here for you to practice and learn.

86
00:07:48,290 --> 00:07:51,590
Let me put it in a presentation mode so that you can see a little better.

87
00:07:51,590 --> 00:07:52,110
Yep.

88
00:07:52,170 --> 00:07:52,770
Cool.

89
00:07:53,650 --> 00:07:54,270
Cool.

90
00:07:54,330 --> 00:07:58,790
So I'll walk you through how you can set this up and practice.

91
00:07:58,790 --> 00:08:05,470
And if you wanted to follow along with me, you can follow this guide at madhuvakula.com slash Kubernetes code.

92
00:08:05,470 --> 00:08:07,450
This is the URL, right?

93
00:08:07,530 --> 00:08:07,950
Cool.

94
00:08:07,950 --> 00:08:14,310
And if you wanted to set up your own, you can just follow this, the documentation.

95
00:08:14,310 --> 00:08:18,630
Basically you need to have a working Kubernetes cluster set up with admin access.

96
00:08:18,630 --> 00:08:27,990
And also you need a Helm version too, because in one of the scenario, we will try to focus on how you can exploit the existing vulnerability in the older version of Helm.

97
00:08:27,990 --> 00:08:29,370
So that's it.

98
00:08:29,370 --> 00:08:33,250
So if you wanted to set up your own, let's go ahead and do that as well.

99
00:08:33,430 --> 00:08:40,630
So if you have your own set cluster, you clone the repository, which is the git clone of the Kubernetes code.

100
00:08:40,710 --> 00:08:46,050
And then you can go ahead and set up the Kubernetes code.

101
00:08:46,390 --> 00:09:06,470
So it checks if Kubernetes kubectl is available, Helm 2 is available in your path, and it'll go ahead and deploy the tiller, which is the server side component of the Helm, and a bunch of vulnerable services, which will be used as practice to exploit or try practicing Kubernetes security.

102
00:09:07,550 --> 00:09:15,230
So if you are someone following the Katakoda live, you can use this playground, which is completely free and online.

103
00:09:15,230 --> 00:09:19,010
You can go ahead and do the same thing in the playground as well.

104
00:09:19,790 --> 00:09:26,510
So the playground is designed in such a way, you can go ahead and see here only the guide in the second, in the specific tab.

105
00:09:26,510 --> 00:09:31,270
And also you can browse those ports and other stuff within the browser itself.

106
00:09:31,270 --> 00:09:33,490
Even in your mobile, also you can try this.

107
00:09:35,050 --> 00:09:41,910
So it takes a while to set up the entire Kubernetes code platform.

108
00:09:41,910 --> 00:09:43,630
I think it's almost done.

109
00:09:43,630 --> 00:09:49,610
So it creates a bunch of vulnerable scenarios, which I have pre-created.

110
00:09:49,650 --> 00:09:55,010
You can go through the code and you can able to see what exactly it is trying to do, if you wanted to look at it.

111
00:09:55,010 --> 00:10:05,910
So this is nothing but it checks if kubectl is existing or Helm 2 existing and it sets a bunch of emails, like which is nothing but deployments in the cluster.

112
00:10:07,550 --> 00:10:08,030
Cool.

113
00:10:08,030 --> 00:10:12,170
Now we have the cluster up and running and we have Kubernetes code set up.

114
00:10:12,170 --> 00:10:13,730
So what's next, right?

115
00:10:13,730 --> 00:10:16,470
So I think it looks like here also it is done.

116
00:10:16,550 --> 00:10:18,970
Ensure you check all the ports are running.

117
00:10:18,970 --> 00:10:21,990
You can go ahead and run kubectl get ports.

118
00:10:21,990 --> 00:10:25,730
It is currently container creating, which means it's starting.

119
00:10:25,810 --> 00:10:31,530
It takes a while to set up the, depends on the way you are running the Kubernetes cluster.

120
00:10:31,650 --> 00:10:37,230
So let me go ahead and do this as well here, kubectl get ports.

121
00:10:37,390 --> 00:10:39,870
I hope you people can see the screen.

122
00:10:39,870 --> 00:10:42,630
I think, let me zoom a little bit as well.

123
00:10:42,630 --> 00:10:43,150
Cool.

124
00:10:43,150 --> 00:10:44,930
So you can see a bunch of them.

125
00:10:44,930 --> 00:10:46,790
Most of the ports are created.

126
00:10:46,790 --> 00:10:50,130
I think just only one port is waiting to get created.

127
00:10:50,390 --> 00:10:51,790
That's awesome.

128
00:10:51,830 --> 00:10:55,770
Now you can see all ports are running and I think this is intentional.

129
00:10:56,150 --> 00:10:58,150
This is a job and get company.

130
00:10:58,350 --> 00:11:03,450
So if you see the status, basically your entire Kubernetes code is set up and running.

131
00:11:03,450 --> 00:11:05,970
So now it's time to hack and get into the cluster.

132
00:11:05,970 --> 00:11:06,450
Bye .

133
00:11:07,490 --> 00:11:08,270
Cool.

134
00:11:08,270 --> 00:11:10,890
That's where we go back and refer to the documentation.

135
00:11:10,990 --> 00:11:16,770
So if you look at the scenarios, so there are a bunch of scenarios we have created, as I said.

136
00:11:16,770 --> 00:11:29,470
So for the time being, for the entire today's presentation, we will not be able to look at all of those scenarios, but definitely we will be looking at some of the primary scenarios, which might be interesting and fun to try out.

137
00:11:29,490 --> 00:11:35,610
And I would highly recommend go back and try out yourself and check out more other scenarios as well, right?

138
00:11:35,610 --> 00:11:45,910
And in some cases, especially not all of them scenarios, but in some scenarios, I try to create a flag in this format, k8sgoot-goot.

139
00:11:45,910 --> 00:11:51,890
And some MD5s, basically it's a number of 32 characters.

140
00:11:52,390 --> 00:12:07,590
Then if you reach this kind of flag, basically it means end of the scenarios, but it may not be found in for all the scenarios, but I try to keep for certain scenarios to reduce the effort of like going on beyond post-exploitation.

141
00:12:09,010 --> 00:12:09,870
Cool.

142
00:12:09,870 --> 00:12:15,050
So let's go ahead and start with the scenario one, sensitive keys in code base.

143
00:12:15,050 --> 00:12:26,070
So I would recommend before doing that, I think if you are setting up locally or in the same place, all ports are running, make sure you run bash access Kubernetes code.

144
00:12:26,070 --> 00:12:37,330
What it does is basically put forward all the ports or services into your local system so that you can access locally without even exposing to the world, which is internet, right?

145
00:12:37,330 --> 00:12:43,230
So now if you go ahead and click this, basically you can see the homepage, like which is the Kubernetes code homepage, right?

146
00:12:43,230 --> 00:12:47,310
So this is where you can see when it's good, it's running, you can follow the guide here.

147
00:12:47,670 --> 00:12:51,390
And also if you want to check out each scenario, the scenarios are here.

148
00:12:51,710 --> 00:12:58,250
So let's get started with the worst scenario, which we will look at one of them is called sensitive keys in code basis.

149
00:12:58,530 --> 00:13:09,090
So the Kubernetes code is designed in such a way, if you wanted to practice as an attacker or learner, you can go ahead and try this scenario by clicking here and starting with attacking.

150
00:13:09,290 --> 00:13:16,370
Or if you are someone who wanted to learn and practice yourself and follow some kind of guide, we have created the documentation.

151
00:13:16,370 --> 00:13:20,710
The solutions will be here in the same guide for each scenario.

152
00:13:20,730 --> 00:13:24,470
Like you can follow them as well here if you're stuck somewhere, right?

153
00:13:24,610 --> 00:13:28,050
And some scenarios we try to create two different methodologies.

154
00:13:28,050 --> 00:13:29,990
It can be tagged in different ways.

155
00:13:29,990 --> 00:13:32,830
So we try to document in the same way.

156
00:13:34,670 --> 00:13:39,090
So if you see here, let's go back and read a little bit about the scenario.

157
00:13:39,090 --> 00:13:43,070
Developers tends to commit sensitive information to version control systems.

158
00:13:43,190 --> 00:13:50,410
As we are moving towards the ACD and GitHub systems, we tend to forget identifying sensitive information in code and commits.

159
00:13:50,410 --> 00:13:53,250
Let's see if we can find something cool here, right?

160
00:13:53,250 --> 00:13:59,290
So by looking at, you can see that some kind of vulnerability or some kind of mistake has happened.

161
00:14:01,210 --> 00:14:04,830
Now, as you can read this, this is version control using Git.

162
00:14:04,830 --> 00:14:10,710
So what I can do is I can try as a DartKit and you can see the default file in the Git will be config.

163
00:14:10,710 --> 00:14:14,610
Now you can see that there is a bare repository hanging there, right?

164
00:14:14,650 --> 00:14:23,890
And by the way, these vulnerabilities which you see, I was showing like in the web vulnerabilities, these are just an entry points into getting into the cluster.

165
00:14:23,890 --> 00:14:27,690
In real world, it may not be the same vulnerabilities you find.

166
00:14:27,730 --> 00:14:31,130
Maybe there is a different approach you find to get in here, right?

167
00:14:31,130 --> 00:14:32,210
That's the thing.

168
00:14:32,510 --> 00:14:35,110
So now I can see that there is a Git config.

169
00:14:35,110 --> 00:14:40,690
Let's go ahead and see how I can clone this repository locally so that I can understand this code base better, right?

170
00:14:40,690 --> 00:14:46,470
So if you want to start, basically you can go back and follow this repository here.

171
00:14:46,470 --> 00:14:48,410
So there is a tool called GitDumper.

172
00:14:48,410 --> 00:14:49,890
It can be any other tool as well.

173
00:14:49,890 --> 00:14:51,250
I just take an example.

174
00:14:51,250 --> 00:14:57,130
So you can try this tool by running a Python3-GitDumper localhost and save it like this.

175
00:14:57,230 --> 00:14:59,950
Maybe I think I have configured here.

176
00:15:02,070 --> 00:15:03,110
Cool.

177
00:15:04,530 --> 00:15:05,530
GitDumper.

178
00:15:05,590 --> 00:15:12,010
So what I'm trying to do is I'm cloning the localhost1230.git folder into local system.

179
00:15:12,030 --> 00:15:18,130
So what it is trying is it is going to fetch all the repository and clone it locally, right?

180
00:15:18,130 --> 00:15:24,390
So now if you go here, kdiskcode.git, so it is the Git repository which we have.

181
00:15:24,390 --> 00:15:31,850
So if you see here, we can see there is a Golang bunch of files, main.co, and a bunch of custom code, right?

182
00:15:31,850 --> 00:15:34,530
And also you can see Git, the folder.

183
00:15:34,530 --> 00:15:38,190
So you can also look for what are the old commits which are happening, right?

184
00:15:38,190 --> 00:15:44,650
As developers sometimes make mistakes, by the way, this scenario is not only focused towards containers or Kubernetes.

185
00:15:44,710 --> 00:15:55,110
In general world, we have seen these kinds of mistakes in most of the modern software development, especially in cloud deployments or any internal deployments, right?

186
00:15:55,790 --> 00:16:05,410
So I can see some of the bunch of commits as you attack and you can go and check manually, like Git checkout to specific, the log.check commit.

187
00:16:05,410 --> 00:16:12,330
And also you can see, oh, you can see an interesting file called env, right?

188
00:16:12,350 --> 00:16:13,330
So there you go.

189
00:16:13,330 --> 00:16:19,050
So you can see there is the access key and secret key for the AWS has mistakenly committed.

190
00:16:19,090 --> 00:16:22,930
And you can also see there is the kdiskcode flag is there.

191
00:16:22,930 --> 00:16:35,910
So basically this means that you have reached the end of this scenario, but in real world you can use this AWS access key, secret key and get the session token and you can do post-exploitation and go along beyond.

192
00:16:36,230 --> 00:16:42,750
So these scenarios are just we have taken from what we see in real world as a pen testing engagement.

193
00:16:42,750 --> 00:16:47,910
And some of them are like in a common scenarios which we see in real world, right?

194
00:16:47,910 --> 00:16:53,130
So this may not be possible and this may be possible in different use cases.

195
00:16:53,130 --> 00:16:58,130
So feel free to give it a try and understand what exactly we are trying to achieve here.

196
00:16:58,150 --> 00:17:01,730
So basically what will happen is in the same use case.

197
00:17:01,810 --> 00:17:03,710
So we can try in different method.

198
00:17:03,710 --> 00:17:19,930
Like what I'm trying to do is most of the time developers mistakenly build a Docker images which may be committed with the .git files because they don't ignore these files when it is creating into Docker container, maybe Docker ignore or something.

199
00:17:19,930 --> 00:17:24,890
So which might end up pushing this data into the containers as well.

200
00:17:24,890 --> 00:17:30,030
So looking at these kinds of sensitive information is very useful in the container.

201
00:17:30,030 --> 00:17:33,030
Like the another methodology you can try, as I said, explanation.

202
00:17:33,030 --> 00:17:36,030
So you can get the pod name, you can exec into the pod.

203
00:17:36,030 --> 00:17:37,730
This is like another method.

204
00:17:38,370 --> 00:17:41,030
Now we can also use tools like TruffleHog.

205
00:17:41,030 --> 00:17:51,570
So TruffleHog, I think most of you know, this is a tool which is basically finds high entropy or sensitive secrets in a Git repositories.

206
00:17:51,570 --> 00:17:58,150
So now I'm trying to do TruffleHog and given the current directory is the where Git repository is there.

207
00:17:58,150 --> 00:18:09,790
So what it does is it will check through all the commits and finds, is there any high entropy values or like sensitive secrets found based on the regular expressions and return to the studio.

208
00:18:09,930 --> 00:18:15,190
So you can include these kinds of tools in your CI, CD systems to detect these kinds of vulnerabilities.

209
00:18:15,190 --> 00:18:27,450
But in attacker's point of view, if you find any of the code basis which has Git or something, maybe it's a really definitely good way to check out in especially the container environments, you might end up finding some kind of sensitive approach.

210
00:18:28,410 --> 00:18:30,230
Like you can see the same thing.

211
00:18:31,610 --> 00:18:32,410
Cool.

212
00:18:32,430 --> 00:18:37,630
So this is our first scenario, which we just tried from the Kubernetes code.

213
00:18:37,630 --> 00:18:40,270
And similarly, as I said, you can try out here as well.

214
00:18:40,270 --> 00:18:52,270
So what you can do is you can look at here and if you wanted to specifically browse specific code, like one, two, three, zero, in the custom code tab, you can just open in a new browser window also.

215
00:18:52,270 --> 00:18:56,710
If you press this icon, then you can just say, I wanted to browse one, two, three, zero code.

216
00:18:56,710 --> 00:19:06,390
So you can able to browse the same thing and play it in online yourself for free and understand what exactly you can assess or test for Kubernetes security.

217
00:19:08,090 --> 00:19:08,890
Cool.

218
00:19:09,170 --> 00:19:12,990
So that's kind of one of the simple scenario.

219
00:19:12,990 --> 00:19:21,950
So let's go ahead and see what could be the some of the complex scenarios or different kinds of vulnerabilities you see in Kubernetes security assessment.

220
00:19:22,290 --> 00:19:32,330
So let's go back to the some maybe a little down, especially like attacking private registries or Kubernetes CS benchmarks or container escape.

221
00:19:32,750 --> 00:19:37,430
So let me go and showcase you how container escape to host system works.

222
00:19:37,770 --> 00:19:41,510
To browse to this scenario, basically you navigate here.

223
00:19:42,610 --> 00:19:51,910
So this is basically one of the pod inside Kubernetes cluster running with some specific privileged access has, right?

224
00:19:51,930 --> 00:20:10,150
So to set the context here, most of the times, especially operations teams may tend to require to run specific containers with high privileges, maybe giving gaps is admin or host privileges or privilege context, which is a full admin, sysadmin capabilities.

225
00:20:10,150 --> 00:20:11,530
So these are things.

226
00:20:11,530 --> 00:20:20,250
And also to just give an example, how we found in a real world, I have seen, especially while we performing a penetration testing engagement.

227
00:20:20,270 --> 00:20:25,750
So most of the developers get an access to completely one namespace with a full privilege.

228
00:20:25,750 --> 00:20:35,190
So if your clusters doesn't have PSPs, which is nothing but pod security policies, developers can basically has full access to that namespace.

229
00:20:35,190 --> 00:20:44,250
Whatever they can do, they can deploy a pod with specific privileges, which they can gain into the whole system access, right?

230
00:20:44,250 --> 00:20:55,150
So also this is one of the use case we have taken to simulate or showcase how you can find these kind of container escape and gain access to the whole system, right?

231
00:20:56,310 --> 00:20:56,910
Cool.

232
00:20:56,910 --> 00:21:01,770
So what we are trying to do here is we can try and look for some bunch of information.

233
00:21:01,770 --> 00:21:04,170
Most of the time, look for a printenv.

234
00:21:04,270 --> 00:21:07,030
Sorry, I think I typed print.

235
00:21:09,330 --> 00:21:10,290
Cool.

236
00:21:10,290 --> 00:21:15,690
So most of the time, you might see some kind of stuff here as well in printenv.

237
00:21:15,690 --> 00:21:17,830
And also I'm checking for mount.

238
00:21:17,830 --> 00:21:19,590
Is there any mounted directories?

239
00:21:19,590 --> 00:21:29,730
Now you can see there is a mounted directory from host system and which is into the container, which means basically in the pod, in this Kubernetes case, right?

240
00:21:29,730 --> 00:21:39,610
So now what I can do is, I think, let's look at what exactly the host system is doing.

241
00:21:39,610 --> 00:21:46,430
So if you see, there is a host system mounted, like which is basically from the host system, to just give an example.

242
00:21:46,510 --> 00:21:54,230
So which is nothing but your underlying host system, which is node, is mounted into your pod as a volume, right?

243
00:21:54,230 --> 00:21:58,510
So due to the reason we have extra capabilities, right?

244
00:21:58,510 --> 00:22:01,630
Like this container has a bunch of extra capabilities as well.

245
00:22:01,630 --> 00:22:11,670
So using those capabilities, as an attacker, what we can do is, we can like try to gain some kind of access into host system.

246
00:22:11,670 --> 00:22:18,150
What you can do is, we can sage root on the host system because we have capability of the privilege.

247
00:22:18,450 --> 00:22:23,390
And now we can see that it is given that the host system sage root.

248
00:22:23,390 --> 00:22:28,950
And now what we can do is, we can perform the internal commands, which are specific to system, right?

249
00:22:28,950 --> 00:22:30,930
Like now you can see docker ps.

250
00:22:31,370 --> 00:22:35,970
So these are nothing but running inside the host system, which is not a container.

251
00:22:35,970 --> 00:22:40,650
So you went below one level from the pod to the host system.

252
00:22:40,650 --> 00:22:43,250
From there, you are executing commands, right?

253
00:22:43,250 --> 00:23:02,270
So the possible attack surface here you can do is, to do post exploitation might be basically using the host system configuration, which is nothing but node level Kubernetes configuration, to go and try attacking other nodes or to gain access to other nodes,

254
00:23:02,730 --> 00:23:07,370
which basically you're controlling the entire cluster by passing through, passing through, right?

255
00:23:07,370 --> 00:23:16,370
So for example, what you can do is, you can use the configuration of node, which is available in the cat var lib.

256
00:23:21,580 --> 00:23:29,000
So you can use the configuration of cat var lib quip config, which has the node level configuration.

257
00:23:29,000 --> 00:23:44,200
So you can use that configuration and perform the Kubernetes commands, which is not only available for the existing namespace specific, it can also perform at the node level, whatever the node has the service account, you can able to do that level.

258
00:23:44,400 --> 00:23:47,020
To do that, for example, you can do this command.

259
00:23:49,760 --> 00:23:52,080
A kubectl command is not available.

260
00:23:52,080 --> 00:23:53,860
Let's download kubectl.

261
00:23:56,640 --> 00:24:00,260
Let me download this kubectl here.

262
00:24:02,100 --> 00:24:04,880
It just takes some time.

263
00:24:04,880 --> 00:24:06,440
Hopefully it is faster.

264
00:24:07,400 --> 00:24:08,160
Yeah.

265
00:24:12,190 --> 00:24:13,590
Yep, there you go.

266
00:24:13,590 --> 00:24:28,610
So what we are trying to do is, we are using now kubectl command and passing the kube config flag, which is using the node configuration and getting all like whatever the all resources in namespace of kube system, right?

267
00:24:28,650 --> 00:24:31,450
Oops, I'm trying the same thing again.

268
00:24:38,550 --> 00:24:44,250
I have to do chmod plus x kubectl.

269
00:24:46,090 --> 00:24:46,830
Cool.

270
00:24:46,830 --> 00:24:57,770
Now you can see when I run kubectl kube config with the specific configuration and get all in kube system, it will go ahead and execute that command and get all the resources.

271
00:24:57,770 --> 00:25:06,830
Now you can see here, it can able to get all the ports which are not restricted to specific namespace, which is the developer or the user has.

272
00:25:06,830 --> 00:25:11,730
And you can able to explore more or access more resources and also services.

273
00:25:11,970 --> 00:25:21,030
And you can see some of the specific use cases, especially I think replication controllers and other jobs and secrets, you cannot able to access.

274
00:25:21,230 --> 00:25:27,130
So the node, whatever the access it has, the node configuration, you can able to perform same level.

275
00:25:27,130 --> 00:25:36,730
So as an attacker, you can do post-exploitation, basically deploy another port in a different node and you can able to pass to another node and again get access to that.

276
00:25:36,730 --> 00:25:46,870
So for that, you can just look for how many nodes you have, like currently two nodes, maybe you are in this node, then you can deploy another port in the different node and get again access to the different.

277
00:25:46,870 --> 00:25:54,270
So you can basically use this kind of hopping methodology and hop on to another node and gain complete cluster access.

278
00:25:56,030 --> 00:26:11,590
So this is mostly you will see if an organization or the company which is not running up the port security policies, you can able to perform these kind of attacks or escalations of the container escapes, right?

279
00:26:11,730 --> 00:26:22,750
So you might use BSPs as a defense for these kind of vulnerabilities and it would be an easy and simple way to fix this kind of exploitation.

280
00:26:23,990 --> 00:26:25,030
Cool.

281
00:26:25,330 --> 00:26:28,450
So let's go ahead and see another use case.

282
00:26:28,450 --> 00:26:31,970
I think we have some more time to see two other scenarios.

283
00:26:31,970 --> 00:26:34,910
Let's take an example of Kubernetes namespace bypass.

284
00:26:36,770 --> 00:26:37,850
Okay.

285
00:26:38,690 --> 00:26:42,750
I'm going here, Kubernetes namespace bypass.

286
00:26:42,830 --> 00:26:43,970
I don't know exactly.

287
00:26:43,970 --> 00:26:44,590
Yeah.

288
00:26:44,590 --> 00:26:47,410
So by default, Kubernetes uses flat networking scheme.

289
00:26:47,410 --> 00:26:51,570
I think this is one of the default feature of Kubernetes.

290
00:26:51,690 --> 00:27:02,390
So by default, Kubernetes uses flat networking scheme means I can able to talk to from any port to any port by default, there is no firewall rules which will happen, allow or deny.

291
00:27:02,390 --> 00:27:06,530
By default, anything can talk to anything within the cluster IP range, right?

292
00:27:06,530 --> 00:27:16,810
So which means that whenever someone, SRE or ops team or like security people says that there is a namespace we have created in the Kubernetes which means it's just a logical boundary.

293
00:27:16,810 --> 00:27:27,350
There is no way you literally segregating between the different namespace unless until apply some kind of security policies like NSPs and network security policies, right?

294
00:27:27,350 --> 00:27:44,350
So if you are testing a Kubernetes clusters where there was no NSPs and there was no restrictions between the namespace as attacker, you can able to talk to other namespaces and other services within the cluster by using a standard network security tools like which we use day to day,

295
00:27:44,350 --> 00:27:48,050
like Nmap or Zmap or a bunch of tools, right?

296
00:27:48,050 --> 00:27:52,810
So to just get started with the scenario, I have created something, an example.

297
00:27:52,830 --> 00:27:59,050
So what we are trying to do is we are running something called Hacker Container which has again created by me.

298
00:27:59,050 --> 00:28:09,070
I have while performing bunch of Kubernetes security testings I thought that there are so many tools we require especially focused on container environment.

299
00:28:09,070 --> 00:28:17,910
So I have created curated list of tools which are maybe 50 plus tools which will help you aid in while performing Kubernetes security assessment.

300
00:28:17,910 --> 00:28:27,410
So this container has those all tools like while you are performing Kubernetes security assessment and you can use this as like a Swiss Army knife in security assessment, right?

301
00:28:27,410 --> 00:28:31,950
And I have seen this has been very valuable while performing assessment.

302
00:28:32,030 --> 00:28:35,610
So definitely I'll try to create some kind of documentation out of it.

303
00:28:35,610 --> 00:28:40,570
Maybe hopefully you can go back and try out different use cases with Hacker Container.

304
00:28:40,570 --> 00:28:43,670
I think it's going and deploying there and cool.

305
00:28:43,670 --> 00:28:44,570
So we got a shell.

306
00:28:44,570 --> 00:28:49,770
So what we are doing is we are inside a pod in one of the default namespace.

307
00:28:49,770 --> 00:28:55,970
Like if you wanted to just see, let's go ahead and run some command called kubectl get pods.

308
00:28:55,970 --> 00:29:00,610
Now you can see Hacker Container which is running in a default namespace.

309
00:29:00,670 --> 00:29:04,770
And let's go ahead and see what else we can do from the shell, right?

310
00:29:04,770 --> 00:29:10,530
So as attacker, you are in one of the pod and you don't know what else you can do from there, right?

311
00:29:10,750 --> 00:29:17,390
So as I said, Hacker Container has a bunch of tools like Docker Bench Security, Kubebench, Linus and a bunch of other tools.

312
00:29:17,470 --> 00:29:24,050
So one interesting thing you can do especially is looking for the entire cluster range.

313
00:29:24,050 --> 00:29:27,610
As I said, by default, Kubernetes uses flat networking scheme.

314
00:29:27,610 --> 00:29:29,870
You can look at the entire cluster range, right?

315
00:29:29,870 --> 00:29:31,190
So you can do IP route.

316
00:29:31,190 --> 00:29:34,890
You can see this is using 10.0.0.0 network range.

317
00:29:34,890 --> 00:29:40,150
And also you can look for what is the current IP address and 10.0.0.0.12.

318
00:29:40,150 --> 00:29:43,810
So you can scan the entire network and identify services.

319
00:29:43,810 --> 00:29:59,710
So one thing I can confirm, I can assure is most of the times internally within the cluster as a microservices or services which are running, many of the developers or operations teams tend to forget or assume that it is only exposed to internal networks.

320
00:29:59,710 --> 00:30:21,650
They will never use authentications or authorizations and you might end up seeing some kind of bypasses which is exposed internally itself, especially Mongo services, Elasticsearch instances and also even some microservices because the API gateway terminates at the ingress level which means before exposing into the world like using load balancers,

321
00:30:21,650 --> 00:30:24,390
they will terminate at the ingress level.

322
00:30:24,390 --> 00:30:33,590
So you might end up seeing some kind of services especially the application services also don't have authentication authorization checks within the clusters.

323
00:30:33,590 --> 00:30:42,850
So this is a really great place to explore and do as much recon you can do and you can gain as much as sensitive information or services access.

324
00:30:43,870 --> 00:30:44,730
Cool.

325
00:30:44,730 --> 00:30:52,990
So what I'm trying to do is here I'm exploring same and what I did is I'm trying to use ZMap which is similar to Nmap.

326
00:30:52,990 --> 00:30:57,910
It's trying to do a scan for the entire cluster range which is to do a faster way.

327
00:30:58,010 --> 00:31:03,350
And I'm specifically looking for a port of 6379 which is a Redis instance.

328
00:31:03,370 --> 00:31:12,810
So I assume that there might be a Redis running or like a 9200 you can also try which is Elasticsearch or Mongo or MySQL or any other services you can scan.

329
00:31:12,810 --> 00:31:15,010
It's like traditional security will apply here.

330
00:31:15,250 --> 00:31:19,230
Let's go ahead and run this ZMap and see what we can see.

331
00:31:19,230 --> 00:31:20,950
So, oops.

332
00:31:21,170 --> 00:31:21,770
Okay.

333
00:31:21,770 --> 00:31:29,730
So we have to have enable the internal network range allow within this list.

334
00:31:29,730 --> 00:31:31,710
Let's go ahead and do that.

335
00:31:33,710 --> 00:31:37,750
Let's comment this and run again.

336
00:31:37,750 --> 00:31:38,930
Sorry.

337
00:31:40,970 --> 00:31:42,170
Oh, okay.

338
00:31:42,170 --> 00:31:48,210
You have to do another thing which is specify the gateway where this ZMap is starting.

339
00:31:48,370 --> 00:31:50,730
Let's go ahead and do that as well.

340
00:31:51,590 --> 00:31:53,110
F1G capital.

341
00:31:53,130 --> 00:31:53,710
Okay.

342
00:31:54,330 --> 00:31:56,410
And what is the gateway?

343
00:31:58,430 --> 00:32:00,490
Yeah, this is the MAC address.

344
00:32:01,050 --> 00:32:01,990
Cool.

345
00:32:02,450 --> 00:32:03,570
So, yeah.

346
00:32:03,570 --> 00:32:19,790
So what it is doing, it is trying to scan the entire cluster range for the specific port 6379 and results the output if any of the IPs running or services running or what is running with the specific port open, it'll save into the other results.csv.

347
00:32:19,790 --> 00:32:20,710
It take a while.

348
00:32:20,710 --> 00:32:27,850
I think it's pretty fast in this case, but this will take some more time to go through and scan.

349
00:32:28,550 --> 00:32:51,250
But as I said, in general, we have seen in real world a lot of services running, especially Elasticsearch instances where you can look for without authentication and where you can gain access to indices and the log information, especially audit logs and sensitive information and also Mongo instances without any authentication or also the microservices,

350
00:32:51,250 --> 00:32:51,750
right?

351
00:32:51,750 --> 00:32:58,690
So this is a pretty goldmine if you are inside a pod and without NSPs, network security policies applied within the cluster, right?

352
00:32:59,670 --> 00:33:00,610
Cool.

353
00:33:00,610 --> 00:33:02,650
So this will take a while.

354
00:33:02,650 --> 00:33:05,250
I think 76% and all stuff.

355
00:33:05,530 --> 00:33:05,870
Yeah.

356
00:33:05,870 --> 00:33:12,850
As I said, similarly, you can try out those vulnerabilities here and follow the guide if you wanted to do as well.

357
00:33:12,850 --> 00:33:19,690
And I think it's a simple way to try out this entire Katakoda environment within the browser.

358
00:33:19,690 --> 00:33:23,370
You don't need to set up the entire cluster itself locally.

359
00:33:23,410 --> 00:33:25,690
And also once you are done, you can just quit the browser.

360
00:33:25,690 --> 00:33:26,350
It's all done.

361
00:33:26,350 --> 00:33:28,110
You don't need to clean up the entire environment.

362
00:33:29,290 --> 00:33:43,670
And another disclaimer, I think I forgot to mention in the beginning, do not try to run or set up this Kubernetes code in your production environments or in your office because it has intentionally vulnerabilities introduced due to the application flaws or something.

363
00:33:43,670 --> 00:33:51,250
It is a vulnerable application, which may have gain access to your production or wherever you run these environments.

364
00:33:51,470 --> 00:33:59,290
And at least I try to take care of using, exposing the ports within to the localhost, but it's highly not recommended to run in production.

365
00:33:59,390 --> 00:34:00,390
Cool.

366
00:34:00,770 --> 00:34:02,570
We have ZMap completed.

367
00:34:02,570 --> 00:34:05,090
Let's go ahead and see what we got from the results.

368
00:34:05,330 --> 00:34:12,460
You can see that there is the IP, which has 6379 port is exposed within the cluster.

369
00:34:12,640 --> 00:34:15,280
And let's go ahead and see what we can get out of it.

370
00:34:15,280 --> 00:34:16,500
As you know, it's Redis.

371
00:34:16,500 --> 00:34:23,720
As I said, this hacker container has built-in tools to look for these kind of common attacks and all stuff.

372
00:34:23,720 --> 00:34:28,060
We can use Redis CLI, and I forgot to be honest, how to access.

373
00:34:28,380 --> 00:34:30,700
Yeah, I found H and the IP.

374
00:34:30,760 --> 00:34:45,660
And now you can see there is another integrated Redis running within the cluster you can able to access because there is no network security policies which is by default in the cluster, means you can able to access without any issues.

375
00:34:45,660 --> 00:34:49,500
I mean, so they don't have any allow list or denial list within the cluster.

376
00:34:49,840 --> 00:34:53,500
Now you can do enumeration standard like get key.

377
00:34:53,980 --> 00:34:55,600
You can see nothing.

378
00:34:55,600 --> 00:34:55,920
Okay.

379
00:34:55,960 --> 00:34:56,680
Keys.

380
00:34:56,680 --> 00:35:03,820
You can see the keys, what are available, and you can see there is a key and get secret stuff.

381
00:35:04,800 --> 00:35:05,540
Cool.

382
00:35:05,540 --> 00:35:06,740
So you reach the flag.

383
00:35:06,740 --> 00:35:09,800
That means you are done with the challenge, right?

384
00:35:09,800 --> 00:35:17,660
So this is just an example of try to simulate the scenarios, but definitely you see these kind of vulnerabilities in a real world.

385
00:35:17,660 --> 00:35:26,160
Once you end up in a pod, try to look out for more services and try as much as you can and you will get most information out of it.

386
00:35:26,160 --> 00:35:26,720
Right.

387
00:35:26,960 --> 00:35:27,940
Cool.

388
00:35:28,620 --> 00:35:35,360
So let's go back and see another scenario before we conclude the presentation and moving into the next sections.

389
00:35:36,320 --> 00:35:44,100
Let's look at one of the common oldest, the vulnerabilities which we have seen exploiting the Helm or Tiller.

390
00:35:46,320 --> 00:35:49,480
So Helm is like a package manager for Kubernetes.

391
00:35:49,480 --> 00:35:56,800
So how we deploy application, install applications into, suppose you are using Debian-based apt-get install the package name.

392
00:35:56,800 --> 00:36:04,500
Similarly, developers or operations teams use Helm, which is a package manager to deploy applications into Kubernetes, right?

393
00:36:04,500 --> 00:36:10,320
So they will just do Helm install and the Nginx, it'll go and install the Nginx within the cluster, right?

394
00:36:10,320 --> 00:36:26,460
So by default, if you are testing any cluster and which is using version two, which is the older version of Helm and it used to be by default, most people using is Helm two in a previous cases, but they have raised three recently, like someone's back.

395
00:36:26,600 --> 00:36:32,060
So by default, Helm version two has RBAC setup with the full cluster admin access.

396
00:36:32,060 --> 00:36:48,280
So if you end up seeing these kind of Helm two configurations running within the cluster and there are no network security policies applied and it is running with the default setup, basically using this misconfiguration as an attacker, you can get entire cluster admin access and form the entire cluster,

397
00:36:48,280 --> 00:36:48,820
right?

398
00:36:48,820 --> 00:36:50,880
Let's go ahead and see how this scenario looks like.

399
00:36:50,880 --> 00:36:56,400
I'm going and running a simple pod and executing into that, right?

400
00:36:57,720 --> 00:37:00,080
Cool, let's go ahead and run this command.

401
00:37:00,080 --> 00:37:06,300
It'll go ahead and spin up another pod, it is called HelmTeller and give you a bash access, right?

402
00:37:14,910 --> 00:37:27,670
Yep, so now we are inside a pod and we assume that this cluster has Helm installed and we wanted to see how we can exploit this kind of vulnerability in a real world environment, right?

403
00:37:27,670 --> 00:37:30,570
So let's go back to our documentation, right?

404
00:37:30,570 --> 00:37:39,010
So by default in Kubernetes, there is a schema which they follow to access or basically it's like a DNS.

405
00:37:39,190 --> 00:37:42,550
Similarly, how we use, they use in the Kubernetes.

406
00:37:42,550 --> 00:37:52,010
So this is nothing but a service name, this is the namespace and they use either a service name and namespace and a cluster.local.

407
00:37:52,170 --> 00:37:55,490
So this is how the structure which is followed, right?

408
00:37:55,490 --> 00:38:04,510
So by default, if you have installed the Helm, which is Teller is get deployed in version two, which is nothing but a server side component of Helm.

409
00:38:04,510 --> 00:38:07,930
Helm is the client side version where you do Helm install or something.

410
00:38:07,930 --> 00:38:11,930
So the Teller is the server side component, which has a full cluster admin access, right?

411
00:38:11,930 --> 00:38:17,530
So now we are doing Teller deploy is the service name and it is by default deployed in a kube system.

412
00:38:17,530 --> 00:38:25,390
So I'm trying to look for kube system and I'm taking Telnet and see if this port is able to access from the current port.

413
00:38:25,390 --> 00:38:38,470
So this is the default port, which is in a by default configurations exposed to 0.0.0, which means if you are in any one of the four port within the cluster, you can able to access the Teller deployment service, right?

414
00:38:38,470 --> 00:38:42,350
You can see, looks like you can able to access the Teller.

415
00:38:42,350 --> 00:38:52,050
So which means you can able to talk to the Teller deployment service because there were no NSPs and by default, it is exposed to 0.0.0, which is within the cluster, right?

416
00:38:52,050 --> 00:38:54,110
And let's see what we can do.

417
00:38:54,150 --> 00:39:02,090
So I have also installed a Helm within this same container to talk to the Helm.

418
00:39:02,090 --> 00:39:12,730
So in the Helm version two, you can specify the dash dash host flag, which means you can specify which Teller service you wanted to talk to.

419
00:39:12,730 --> 00:39:21,870
Now we know the Teller service name as well as the DNS basically, and we are specifying talk to this host and perform these operations, right?

420
00:39:21,870 --> 00:39:24,150
Let's go ahead and run this command.

421
00:39:27,030 --> 00:39:34,810
I'm trying to run Helm command and specifying the specific service and the port, and I'm just running Helm version command, right?

422
00:39:34,810 --> 00:39:42,350
Now you can see the client version, which is Helm, is 2.16.7, and the server version, which is Teller, is 2.16.9.

423
00:39:42,350 --> 00:39:48,070
So I can able to talk to the Helm service and I can also see what all charts deployed, Helm charts, right?

424
00:39:48,070 --> 00:39:50,690
Helm-less, like you can see metadata DB is deployed.

425
00:39:50,690 --> 00:39:53,330
So you can create your own chart and apply as well, right?

426
00:39:53,330 --> 00:40:01,810
So what we have done is, as an attacker, simply to showcase the complexity or what you can do as an attacker, let's go ahead and run a simple command.

427
00:40:02,190 --> 00:40:10,210
By default, Kubernetes stores, has a service account, which is stored in var lib, service secret,

428
00:40:18,690 --> 00:40:33,010
one, yeah, secrets, Kubernetes IOS service account, namespace, because I'm running in default namespace, and this is the token for that specific service account, which is mapped to the current port.

429
00:40:33,010 --> 00:40:47,310
So what I'm trying to do now here is, we have created a Helm chart, which is one chart, and this I have taken from the Bitnami repository research, and I have given the reference here, and go and check more resource.

430
00:40:47,310 --> 00:40:50,470
What it does is basically, it's a simple Helm chart.

431
00:40:50,470 --> 00:40:58,170
What it is do is, it has a bunch of values, which is nothing but namespace is default, and the name of the, is default.

432
00:40:58,170 --> 00:41:00,470
Let's look at the templates, how it looks like.

433
00:41:01,810 --> 00:41:04,950
Templates, and a cluster role.

434
00:41:04,950 --> 00:41:22,070
It has a cluster role, which is nothing but, it gives all API groups in the Kubernetes, for all the resources, and all the verbs, which means you can do get, post, delete, list, all of the resources, for any of the resources in the Kubernetes cluster,

435
00:41:22,070 --> 00:41:25,490
any of the kind of API groups, you can perform.

436
00:41:25,550 --> 00:41:29,930
So what we are trying to do is, templates, cluster role binding.

437
00:41:30,090 --> 00:41:44,430
So we are trying to bind this cluster role, which is entered across the cluster, to the service account, which is taken, if you've seen, which is namespace is default, because we are in the, this pod is currently running in the default namespace, and the service account is default,

438
00:41:44,430 --> 00:41:51,910
because you see here, it's the service account is default, when you do, slash var run, secret service account.

439
00:41:51,930 --> 00:42:02,030
So what we are trying to do is, we are creating a new cluster role, which gives full cluster access, to perform anything on all resources, and binding that cluster role, to the default service account.

440
00:42:02,030 --> 00:42:12,510
So now, as you can't get the secrets, or any operations, within the cluster using the default account, once we deploy this Helm chart, you can do anything as a cluster admin, right?

441
00:42:12,510 --> 00:42:15,650
Let's go ahead and run this, deploy this Helm chart.

442
00:42:15,710 --> 00:42:21,210
So we are trying to do Helm, and we are using the same Tiller configuration, and deploying the Pawn chart, right?

443
00:42:21,210 --> 00:42:22,230
Let's go ahead and run.

444
00:42:22,230 --> 00:42:31,710
So it went ahead and deployed the Pawn chart, because Helm has the capability to talk to the Tiller, and Tiller has the full cluster admin access, so it can go ahead and deploy this chart.

445
00:42:31,710 --> 00:42:38,650
So now if you go ahead and run kubectl get secrets, you can able to get the secrets of the entire Kubernetes here, right?

446
00:42:38,650 --> 00:42:47,070
So you can also get kubectl get pods, you can able to get all the pods, and in all namespaces as well, right?

447
00:42:47,070 --> 00:43:01,990
So basically you have full cluster admin access, because currently whatever the service account, which is there in our pod, because due to the reason we have created a cluster role, this cluster admin, sorry, this pod has basically full cluster admin access,

448
00:43:01,990 --> 00:43:05,630
which can able to perform any kind of operations on all resources, right?

449
00:43:05,630 --> 00:43:14,010
So as a attacker, some of the useful commands is like look for auth, can I create pods?

450
00:43:14,010 --> 00:43:23,070
You can check if the current service account, which you own, if you anytime get a service account access, it can able to create pods or something, sorry, create.

451
00:43:23,370 --> 00:43:28,930
You can also check delayed pods, it can able to delay pods or something.

452
00:43:28,930 --> 00:43:32,430
You can also specify, can I do in namespace of kube system?

453
00:43:32,650 --> 00:43:35,970
So this is talk to the API and give you the information.

454
00:43:36,170 --> 00:43:38,510
So this command is pretty useful.

455
00:43:38,510 --> 00:43:43,390
If you got a service account and you wanted to see what kind of privilege it has, you can make these calls, right?

456
00:43:43,390 --> 00:44:00,630
And also some tips, I don't know, we are about to conclude and let me, whenever you make these commands, you can also increase the verbosity of the Kubernetes, you can increase the six or seven, you can able to see each and every request it is making to the Kubernetes cluster and in a verbose way.

457
00:44:00,630 --> 00:44:04,930
Like if you increase this, you can able to see much better, even like headers.

458
00:44:04,930 --> 00:44:09,310
And if we increase like height, like you can even see like a response body and everything.

459
00:44:09,450 --> 00:44:14,150
And if you increase to nine and like little more depth, it will keep going, right?

460
00:44:14,150 --> 00:44:16,370
So this is completely API driven.

461
00:44:16,370 --> 00:44:22,030
So you can able to play around with the verbose and understand exactly each and every thing which is being answered.

462
00:44:22,110 --> 00:44:24,110
And also very useful command is explain.

463
00:44:24,110 --> 00:44:28,430
If you wanted to understand a pod, then it will give you more details about it.

464
00:44:28,430 --> 00:44:34,090
And I wanted to learn more about pod spec, you can also do kubectl spec.

465
00:44:34,130 --> 00:44:35,550
So this is very handy.

466
00:44:35,550 --> 00:44:41,410
And while you are performing assessments, you can quickly look at the documentation rather like explain commands, right?

467
00:44:42,030 --> 00:44:50,010
Cool, so now you see, due to the simple default misconfiguration of our Helm pillar, we can able to control the entire cluster.

468
00:44:50,050 --> 00:44:59,750
And this vulnerability I have seen in so many of the cluster in real-time pentesting assessments due to the administrator doesn't configure properly.

469
00:44:59,790 --> 00:45:03,050
And this is default security installation by Helm itself.

470
00:45:03,110 --> 00:45:07,450
So the newer version of Helm v3 fixed this vulnerability, so it kind of patched.

471
00:45:08,770 --> 00:45:11,230
Cool, so let me go with the last scenario.

472
00:45:11,230 --> 00:45:12,950
I think then we will kind of conclude.

473
00:45:12,950 --> 00:45:16,070
And this you can try, it's a DIND, Docker in Docker and Exploitation.

474
00:45:16,070 --> 00:45:26,050
This you commonly see in CI-CD systems, especially in build pipelines, like GitLab CI-CD or Bitbucket CI-CD or in real-world where they pass the socket, our socket, right?

475
00:45:26,050 --> 00:45:27,250
You can go ahead and try.

476
00:45:27,250 --> 00:45:32,050
And SSRF vulnerability, especially this is very widely exploited in the cloud infrastructure.

477
00:45:32,050 --> 00:45:36,890
But in coming to the Kubernetes context, it is little much, you can get more out of it.

478
00:45:36,890 --> 00:45:42,030
Because as I said, within the cluster, there might not be some kind of services authenticated.

479
00:45:42,030 --> 00:45:47,910
There might be services which are giving more health information of the nodes as well as the ports or internal services.

480
00:45:47,910 --> 00:45:56,490
Using SSRF, you can get more information out of it within the clusters compared to the cloud native, cloud way of exploiting, right?

481
00:45:57,290 --> 00:45:58,210
Cool.

482
00:45:58,210 --> 00:46:00,890
So let me just complete with the...

483
00:46:02,490 --> 00:46:03,930
Yeah, this looks fun.

484
00:46:03,930 --> 00:46:04,470
Cool.

485
00:46:04,470 --> 00:46:18,370
So this is the use case where basically I'm trying to showcase, if you end up in any of the pod and the developer or operations teams didn't set up any resource consumptions, like you can only consume these many resources, right?

486
00:46:18,370 --> 00:46:24,330
So you can able to use utilities, especially like I'm trying to showcase here, stress-ng.

487
00:46:24,890 --> 00:46:27,630
So let me showcase a simple example.

488
00:46:27,870 --> 00:46:30,190
Let me show in the terminal, maybe.

489
00:46:30,690 --> 00:46:35,830
Exit, kubectl get pods.

490
00:46:36,050 --> 00:46:38,210
Now you see a bunch of pods are running.

491
00:46:38,210 --> 00:46:40,210
So I'm looking for a specific pod.

492
00:46:40,210 --> 00:46:46,550
Let's look for that kubectl top pod.

493
00:46:47,290 --> 00:46:50,410
So when you do top, it's basically a system like top command.

494
00:46:50,410 --> 00:46:54,750
It shows how much CPU resources and memory resources it is using.

495
00:46:54,810 --> 00:47:05,130
And if any of the deployment or manifest which doesn't have specifically resource limit setup and the cluster is deployed in like cloud infrastructure, like which has auto-scaling enabled.

496
00:47:05,130 --> 00:47:11,190
If you re-hit this limit, it'll go and scale up a new node or deploys like some more nodes into the cluster, right?

497
00:47:11,210 --> 00:47:15,250
So that times you can exploit this in an attacker's point of view.

498
00:47:15,250 --> 00:47:16,770
Like let's go ahead and run a simple tool.

499
00:47:16,970 --> 00:47:20,390
I'm running with two kicks and 30 seconds.

500
00:47:20,530 --> 00:47:30,290
So now you can see if you're on the same command from the specific resources of 3MB, 1MB, 2MB, like increasing so much, right?

501
00:47:30,290 --> 00:47:31,890
I think it's just taking some time.

502
00:47:31,890 --> 00:47:33,570
Let's keep looking.

503
00:47:45,330 --> 00:47:57,130
You can see it hits a crazy amount of memory resource limits because there was no resource limit set for this deployment, whatever the hunger check, it will be deployed as part of the Kubernetes code.

504
00:47:57,490 --> 00:48:06,550
Attacker can basically consume more CPU resources and memory, which basically cost the billing and very like cost-effective for the company's perspective, right?

505
00:48:06,550 --> 00:48:07,230
Billing.

506
00:48:07,230 --> 00:48:16,110
And due to the reason, the most of the modern infrastructure is set up in such a way which is auto-scalable and like scalable based on the certain parameters.

507
00:48:16,230 --> 00:48:24,030
So you've been able to basically DDoS or gain more resource consumption for the companies and organizations, right?

508
00:48:24,030 --> 00:48:26,590
So this is definitely one thing.

509
00:48:26,590 --> 00:48:32,550
So before moving in, let me just conclude with the Hacker Container preview.

510
00:48:32,910 --> 00:48:39,330
As I said, Hacker Container, let me just showcase maybe here, Hacker Container.

511
00:48:39,330 --> 00:48:44,310
So Hacker Container is a similar, another utility I created as part of the assessment.

512
00:48:44,310 --> 00:48:46,570
So you can read more about it here, the blog post.

513
00:48:46,570 --> 00:48:52,950
So this has a bunch of utilities, which you use day-to-day, like while performing assessments in the Kubernetes cluster.

514
00:48:52,950 --> 00:48:59,390
So I have used it extensively, I would say, while performing assessment and it is very useful utility.

515
00:48:59,390 --> 00:49:00,710
Let's go ahead and run.

516
00:49:01,210 --> 00:49:02,690
Oops, looks like it's already there.

517
00:49:02,690 --> 00:49:06,730
Let's exec into the pod, Hacker Container.

518
00:49:06,810 --> 00:49:17,210
So we are inside the pod, like you see a bunch of other stuff like default tools.

519
00:49:17,210 --> 00:49:22,590
So some of the things like it has is amicontain, so which is again, utility written by Jessica Frazel.

520
00:49:22,590 --> 00:49:29,610
She has done quite a lot of work for the containers and security community in the Kubernetes and the container ecosystem.

521
00:49:29,710 --> 00:49:44,630
So what it does is it will try to identify what container runtime you are using and do you have shared namespaces from the host system like PID, username space, or is there any app armor or sitcom profile applied to the container to restrict certain capabilities?

522
00:49:44,810 --> 00:49:51,490
And is that what all capabilities allowed currently, system calls allowed, and it kind of a container introspection tool.

523
00:49:51,490 --> 00:49:56,150
So which will help you to perform while in attacks and security assessments.

524
00:49:56,150 --> 00:50:01,510
So not only I think specific to the container Kubernetes, then it's a pretty useful tool to try out, right?

525
00:50:01,510 --> 00:50:06,190
So you can also do the same thing with proc self cgroup maybe.

526
00:50:06,190 --> 00:50:07,090
Yeah, cool.

527
00:50:07,090 --> 00:50:13,010
So you can see this is basically trying to figure out kubeports and you can also do capsh, print.

528
00:50:13,010 --> 00:50:16,750
So it will look for what all capabilities are there and similarly.

529
00:50:16,810 --> 00:50:20,190
So this is kind of pretty useful tool, the Hacker Container.

530
00:50:20,710 --> 00:50:25,130
And also it has Nikto and a bunch of other use cases.

531
00:50:25,130 --> 00:50:28,430
I think maybe I try to not document it here.

532
00:50:28,430 --> 00:50:35,470
So I'm going to try to see what can I kind of create out of it and how I use in day to day while exploiting.

533
00:50:35,470 --> 00:50:40,250
And some use cases I would say is very useful is running benchmarks.

534
00:50:40,250 --> 00:50:51,310
If you are inside the clusters and you wanted to just get a glance of view, what kind of vulnerabilities possibly there within the Kubernetes clusters, both in terms of container runtime as well as the cluster level.

535
00:50:51,310 --> 00:50:53,570
You can run these CIS benchmarks.

536
00:50:53,630 --> 00:50:56,970
Like one is you can just go ahead and run this command.

537
00:50:57,130 --> 00:50:58,470
Let's go ahead and run.

538
00:50:58,510 --> 00:50:58,890
Yep.

539
00:50:58,890 --> 00:51:09,050
So what it does is it, I have created something called daemonset, which means each node, how many ever nodes you have in the cluster, will get one pod running in all the nodes.

540
00:51:09,050 --> 00:51:11,550
That is called daemonset in Kubernetes.

541
00:51:11,650 --> 00:51:12,730
Let's go ahead and get the node.

542
00:51:12,730 --> 00:51:14,730
So you can see two nodes are available.

543
00:51:14,730 --> 00:51:22,030
Now you should be able to see two pods running in the Docker Bench because there are two nodes because we are running a daemonset, right?

544
00:51:22,030 --> 00:51:23,610
So you can see two pods running.

545
00:51:23,610 --> 00:51:30,950
So let's go ahead and exec into one of the pod and see what we can do with Docker Bench Security, right?

546
00:51:30,950 --> 00:51:37,650
So Docker Bench Security, I'm execing into the pod.

547
00:51:38,390 --> 00:51:41,570
Now you can see Docker Bench Security.

548
00:51:41,570 --> 00:51:44,650
Now you can go ahead and run dockerbenchsecurity.css.

549
00:51:44,650 --> 00:51:46,710
This is also part of a Hacker Container.

550
00:51:46,710 --> 00:51:48,030
You can go ahead and run this.

551
00:51:48,030 --> 00:52:09,740
So the way I have set it up, sorry, the scenario manifest is if you can see here, I have created the Docker Bench Security in such a way it access the host PID, IPC network, as well as a bunch of volumes.

552
00:52:09,740 --> 00:52:20,460
And it also had extra capability called audit control so that you can access and talk to your underlying Docker Container runtime and you can perform benchmark analysis on possible security issues, right?

553
00:52:20,460 --> 00:52:23,240
So this is a pretty simple utility.

554
00:52:23,240 --> 00:52:24,720
Just go ahead and run this.

555
00:52:24,720 --> 00:52:38,140
Once you are inside cluster, then you will get at least high level posture and use the whatever the found vulnerabilities or the checks as attack paths, or you can use those while you are doing assessments or post-exploitation, right?

556
00:52:38,220 --> 00:52:44,220
Once you are inside a node, maybe you can use some kind of vulnerability which is detected here to do some kind of post-exploitation.

557
00:52:44,280 --> 00:53:01,620
So you can see it does a bunch of checks like host level configuration checks and specific to like the Docker daemon or like directories and daemon configuration, and as well as the files, which is specific to the container runtime here in this case, Docker,

558
00:53:01,620 --> 00:53:12,100
and also container images, whether container running with a root privilege or it has any extra capability set or it doesn't have any seccomp or apparm profiles, these kinds of things, right?

559
00:53:12,960 --> 00:53:16,340
Similarly, you can go ahead and run the kubebench as well.

560
00:53:16,810 --> 00:53:20,300
You can go ahead and run this kubebench like this.

561
00:53:21,440 --> 00:53:23,940
Let me go ahead and run here only, I think.

562
00:53:24,860 --> 00:53:27,400
It's kind of taking quite a lot of time.

563
00:53:29,560 --> 00:53:32,200
So you can see kubebench I have run.

564
00:53:32,260 --> 00:53:33,940
You can give it pods.

565
00:53:34,060 --> 00:53:36,740
The kubebench is running here.

566
00:53:36,740 --> 00:53:39,880
You can get the logs, iPhone app, and the bench.

567
00:53:39,940 --> 00:53:40,960
So it will show.

568
00:53:40,960 --> 00:53:44,520
You can see there are 12 checks passed, nine checks failed, and two checks running.

569
00:53:44,520 --> 00:53:50,020
So you can see what all the vulnerabilities possibly there in this Kubernetes cluster and what would be the remediations.

570
00:53:50,020 --> 00:53:55,820
So you can use these vulnerabilities maybe to do further exploitation and understand what you can do, right?

571
00:53:55,820 --> 00:53:59,600
So these tools are very useful just to get a glance of it.

572
00:53:59,600 --> 00:54:09,500
And there are a bunch of utilities out there as well, like kubehunter and kubepsychaudit and the different golang tool written by different researchers.

573
00:54:09,580 --> 00:54:13,700
And I'll try to document and put it in the same scenarios.

574
00:54:13,700 --> 00:54:15,640
So hopefully you can go back and try out yourself.

575
00:54:16,520 --> 00:54:17,240
Cool.

576
00:54:17,380 --> 00:54:19,420
So there are a bunch of other scenarios.

577
00:54:19,500 --> 00:54:26,280
Due to the time constraint, I couldn't complete or showcase you, but feel free to go and check out the Kubernetes Go project.

578
00:54:26,360 --> 00:54:32,580
And you should be able to get and follow using the playground of that Katakoda, right?

579
00:54:32,580 --> 00:54:38,620
And if you have set it up in your own environment, like in our case, right, my case, so you can go ahead and tear down the cluster.

580
00:54:38,620 --> 00:54:44,860
Basically, it will try to delete possibly resources which I have created as part of the Kubernetes Go deployment.

581
00:54:44,860 --> 00:54:51,640
But I highly recommend you delete the cluster to just ensure cleanup of the entire environment, right?

582
00:54:51,780 --> 00:55:01,600
And I have also showcased and run using a tool called Chekhov to see what all possible vulnerabilities using a CACD as part of pipeline.

583
00:55:01,600 --> 00:55:09,900
So it found a bunch of vulnerabilities in the deployments and other things which I have used as part of Kubernetes Go as well.

584
00:55:09,900 --> 00:55:10,700
All right.

585
00:55:11,180 --> 00:55:12,040
Cool.

586
00:55:12,040 --> 00:55:15,140
So that's it pretty much about it.

587
00:55:15,140 --> 00:55:23,580
And I would highly recommend having your feedback and understand what exactly you want to look forward.

588
00:55:23,580 --> 00:55:31,020
And I'm really looking forward to building more and more scenarios and also adding the defense scenario.

589
00:55:31,020 --> 00:55:41,220
I'm really building like most of the scenarios and making them in such a way that how you can practice for the CNCF certified Kubernetes security.

590
00:55:41,220 --> 00:55:47,940
So you can use this playground to practice and understand how most of the Kubernetes vulnerabilities you can assess and learn more about it.

591
00:55:47,940 --> 00:55:56,180
So feel free to get involved and you can please feel free to provide the feedback and share with your colleagues, teammates, or your friends.

592
00:55:56,180 --> 00:56:01,180
I would really appreciate if you can share the feedback and I would love to improve and work on it.

593
00:56:01,180 --> 00:56:07,680
And if you see any documentation or anything needs to be improved, please let me know by GitHub issues.

594
00:56:07,680 --> 00:56:11,880
And I'm trying to be pretty active, but in the past month I was a little busy.

595
00:56:11,880 --> 00:56:15,440
So I kind of work on this, working on defense scenarios as well.

596
00:56:15,440 --> 00:56:26,680
Hopefully we will be releasing this month for the defense scenarios to understand and fix the vulnerabilities which you have learned while practicing these attacks and also more attacker scenarios as well.

597
00:56:27,080 --> 00:56:32,500
And with that, I think I'm pretty much done with the session.

598
00:56:32,500 --> 00:56:36,800
Thank you so much once again for having me and really looking forward.

599
00:56:37,200 --> 00:56:37,660
Awesome.

600
00:56:37,660 --> 00:56:38,440
Thank you so much.

601
00:56:38,440 --> 00:56:39,920
It was an amazing presentation.

602
00:56:39,920 --> 00:56:41,140
I learned so much.

603
00:56:41,160 --> 00:56:45,860
And as a matter of fact, in Discord, there are a lot of people actually talking about your presentation.

604
00:56:45,940 --> 00:56:48,580
And once again, thank you for supporting the communities.

605
00:56:48,580 --> 00:56:51,040
Thank you for supporting the DEF CON Red Team Village.

606
00:56:51,360 --> 00:57:00,000
We're going to go on a little break, but we invite everybody that will have any further questions and so on to join the Discord channel, which is in the description of the video.