1
00:00:31,450 --> 00:00:32,870
Hey there!

2
00:00:32,870 --> 00:00:33,990
Welcome everyone.

3
00:00:34,070 --> 00:00:45,610
We are kicking off the Q&A session here with Ayub, who spoke about Only Takes a Spark Popping a Shell on 1000 Nodes, which was pretty awesome, pretty cool.

4
00:00:45,610 --> 00:00:49,070
A good way of thinking about how to scale up everything.

5
00:00:49,510 --> 00:00:53,450
And, you know, not just hacking one system, but how can you use that to really spread out.

6
00:00:53,890 --> 00:00:57,950
So, really, I heard this is also your first time speaking at DEF CON.

7
00:00:57,950 --> 00:00:59,030
Is that correct?

8
00:01:00,890 --> 00:01:02,050
Exactly, yeah.

9
00:01:02,490 --> 00:01:03,270
So, we have...

10
00:01:03,270 --> 00:01:06,030
I was actually going to give it... yeah, go ahead.

11
00:01:06,090 --> 00:01:11,410
No, yeah, so I was just going to say we have a tradition here at DEF CON, which we call Shoot the Noob.

12
00:01:11,430 --> 00:01:16,910
And so, really, it's just the tradition of, you know, hey, taking a shot on stage for, you know, your first thing.

13
00:01:16,930 --> 00:01:18,310
It's not always liquor, right?

14
00:01:18,310 --> 00:01:19,790
So, people mistake that.

15
00:01:19,790 --> 00:01:22,730
But, really, just want to do that with you now.

16
00:01:22,730 --> 00:01:25,950
So, I want to welcome you in the traditional DEF CON fashion.

17
00:01:26,170 --> 00:01:27,590
So, cheers!

18
00:01:28,210 --> 00:01:28,970
Oh, boy.

19
00:01:28,970 --> 00:01:29,930
Okay, okay, okay.

20
00:01:29,930 --> 00:01:31,130
Oh, you got to pour it first.

21
00:01:31,130 --> 00:01:31,770
Wait for it.

22
00:01:32,490 --> 00:01:33,990
I got to pour it first.

23
00:01:33,990 --> 00:01:34,810
You had to prep.

24
00:01:34,810 --> 00:01:35,290
There you go.

25
00:01:35,290 --> 00:01:35,870
Okay.

26
00:01:36,050 --> 00:01:37,110
There we go.

27
00:01:37,790 --> 00:01:38,410
Cheers, guys.

28
00:01:38,410 --> 00:01:38,930
Cheers.

29
00:01:43,470 --> 00:01:44,190
Awesome.

30
00:01:45,430 --> 00:01:50,090
So, now that you've been officially indoctrinated to the DEF CON and DEF CON safe mode.

31
00:01:51,150 --> 00:01:54,870
Yeah, so, one of the first things, questions I'd ask, right?

32
00:01:54,870 --> 00:01:56,970
So, right, you recorded the talk a little bit ago.

33
00:01:56,970 --> 00:01:58,810
What have you been working on since?

34
00:01:58,810 --> 00:02:00,990
Anything new that you've kind of come up with?

35
00:02:00,990 --> 00:02:02,510
Or, you know, anything you found?

36
00:02:02,510 --> 00:02:05,450
Anything, you know, that's changed since the talk was given?

37
00:02:07,350 --> 00:02:09,490
It has nothing to do with Spark, actually.

38
00:02:09,490 --> 00:02:12,450
I was working on Spark, like, a few months ago now.

39
00:02:12,690 --> 00:02:16,050
But since then, I completely switched subjects and topics.

40
00:02:16,050 --> 00:02:18,630
I've been working on AWS a little bit.

41
00:02:18,970 --> 00:02:24,150
I did, actually, a tool that reflectively loads DLLs and executables in memory.

42
00:02:24,150 --> 00:02:26,010
But I wrote it using Golang.

43
00:02:26,010 --> 00:02:29,250
So, I completely abandoned that area of research.

44
00:02:29,250 --> 00:02:31,650
And I completely switched over to something completely different.

45
00:02:31,650 --> 00:02:33,990
So , we'll see what comes up next.

46
00:02:34,130 --> 00:02:35,230
Good for you.

47
00:02:35,230 --> 00:02:36,630
That's a pretty good way to go.

48
00:02:36,750 --> 00:02:39,270
You get some recognition for one thing.

49
00:02:39,270 --> 00:02:42,530
And then you move off to something totally cool, totally new.

50
00:02:42,650 --> 00:02:43,530
Excellent.

51
00:02:43,930 --> 00:02:47,190
Yeah, definitely like this idea of discovering new stuff.

52
00:02:47,710 --> 00:02:48,570
Excellent.

53
00:02:48,690 --> 00:02:51,610
So, is that a thing you find yourself doing often in your own research?

54
00:02:51,610 --> 00:02:57,530
Is that you want to get a nice breadth of experience in a lot of different things?

55
00:02:57,530 --> 00:03:03,990
Or how deep do you go before you feel like you're comfortable with what you've learned?

56
00:03:06,150 --> 00:03:07,570
I think I can...

57
00:03:07,570 --> 00:03:14,130
Basically, my idea is go as deep as you start to understand how it works, basically.

58
00:03:14,310 --> 00:03:16,630
Before, I was looking into mainframes.

59
00:03:16,630 --> 00:03:29,210
And that took me, I think, two years of going through these 900-page documents published by IBM and these obscure systems and obscure forums actually talking about it.

60
00:03:29,210 --> 00:03:32,270
So, I did the same thing with mainframes.

61
00:03:32,270 --> 00:03:33,610
And then I let it go.

62
00:03:34,110 --> 00:03:37,190
And I switched over to other areas.

63
00:03:37,190 --> 00:03:39,950
And during that work, I think Spark came up.

64
00:03:39,950 --> 00:03:42,090
And I was like, what the hell is this thing?

65
00:03:42,190 --> 00:03:43,050
And I dug into it.

66
00:03:43,050 --> 00:03:45,950
And I found there was not much research was being done on it.

67
00:03:45,950 --> 00:03:47,970
And I thought, well, you know what would be fun?

68
00:03:48,870 --> 00:03:49,870
So, I actually dug into it.

69
00:03:49,870 --> 00:03:51,770
And that's how it all started, really.

70
00:03:51,890 --> 00:03:56,550
It was just going after the next shiny thing, trying to understand how it works.

71
00:03:56,550 --> 00:04:00,990
And once you understand how it works, you try to bend its rules to do whatever you want with it.

72
00:04:01,210 --> 00:04:06,450
And hopefully, write a tool about it, give a talk about it, and then move on to the next stuff.

73
00:04:07,050 --> 00:04:08,010
How deep?

74
00:04:08,210 --> 00:04:09,970
Just enough to understand it, really.

75
00:04:11,210 --> 00:04:12,030
That's the goal.

76
00:04:12,030 --> 00:04:17,850
So, what was it about Spark, really, other than the fact that it was new and there wasn't a lot of research?

77
00:04:17,850 --> 00:04:21,950
I mean, what was it that really dragged you into this one?

78
00:04:21,950 --> 00:04:26,970
Because it's an interesting system, as you showed us in your talk.

79
00:04:27,250 --> 00:04:34,770
So, what was the thing that made you decide, okay, fine, this is where I'm going to spend the next year of my life?

80
00:04:34,770 --> 00:04:36,190
Yeah, and why not continue?

81
00:04:36,190 --> 00:04:38,850
All right.

82
00:04:38,850 --> 00:04:43,870
So, I was working on the offensive side, and then I switched to the blue side.

83
00:04:44,310 --> 00:04:47,010
And I was helping a company secure their systems, etc.

84
00:04:47,010 --> 00:04:51,330
And they were all on these new shiny platforms, if you will.

85
00:04:51,330 --> 00:04:55,210
Everything was on AWS, multi-region, CICD.

86
00:04:55,210 --> 00:04:57,390
I mean, they didn't click a single button.

87
00:04:57,390 --> 00:05:00,750
They pushed everything to code, and everything was deployed and scaled and stuff.

88
00:05:00,750 --> 00:05:02,670
Very, very sexy stuff.

89
00:05:02,670 --> 00:05:07,130
So, I was trying to help them secure that stuff.

90
00:05:07,370 --> 00:05:16,070
And after six or seven months, we thought we did a pretty good job locking pretty much everything that was supposed to be locked.

91
00:05:16,070 --> 00:05:19,450
There was no Windows, by the way, so that's why we came to this state.

92
00:05:19,450 --> 00:05:25,370
But anyway, and then I was talking with this data scientist, just trying to understand what they do.

93
00:05:25,770 --> 00:05:28,770
And he mentioned something about Spark.

94
00:05:28,770 --> 00:05:30,450
And I was like, what is that?

95
00:05:30,450 --> 00:05:34,910
And he told me, oh, it's something that we use to make calculations and parse data.

96
00:05:35,430 --> 00:05:37,150
I mean, what do you mean, parse data?

97
00:05:37,650 --> 00:05:38,950
How many machines do you have?

98
00:05:38,950 --> 00:05:43,270
And he's like, oh, actually, it's like, I have 200 machines.

99
00:05:43,270 --> 00:05:44,510
It's like, what do you mean, you have?

100
00:05:44,510 --> 00:05:51,050
It's like, oh, I spawned a cluster of 200 machines, and that guy over there spawned a cluster of 500 machines, etc., etc.

101
00:05:51,050 --> 00:05:55,350
So, basically, the company had like thousands of machines running sporadically, if you will.

102
00:05:55,350 --> 00:05:59,210
But still, and I was like, okay, well, how do you walk me through it, please?

103
00:05:59,210 --> 00:06:04,470
And he showed me how he launched the job and how he did some calculations, and I was like, oh, my God.

104
00:06:05,810 --> 00:06:10,230
And that's when, where the all interest Spark came in, basically.

105
00:06:11,030 --> 00:06:11,850
That's awesome.

106
00:06:11,850 --> 00:06:13,450
That's a great place to start.

107
00:06:13,450 --> 00:06:19,410
Yeah, you start seeing, oh, a couple of commands, and all of a sudden, there's this many machines doing something.

108
00:06:20,070 --> 00:06:21,730
I want to know that, too.

109
00:06:21,730 --> 00:06:29,810
He was writing Python code on a Jupyter notebook, where, you know, there was an authentication, obviously.

110
00:06:30,250 --> 00:06:36,610
And he was writing code, and it executed over multiple machines, and I was like, dude, this is amazing.

111
00:06:38,330 --> 00:06:42,670
And so I dug into it, and yeah, that's what it all sounded like.

112
00:06:42,670 --> 00:06:49,890
All right, so it's a pretty specific setup that you are talking about in your presentation.

113
00:06:49,890 --> 00:06:56,010
How common is the setup that you're speaking about?

114
00:06:56,010 --> 00:07:02,310
And how many other ways are there to configure this that might have more research opportunities?

115
00:07:02,610 --> 00:07:12,710
Well, the thing is, basically, a Spark cluster can be set up in very different ways.

116
00:07:13,590 --> 00:07:23,910
The cluster manager, so basically the process, the component that is responsible for linking applications to workers, that one can be replaceable.

117
00:07:24,070 --> 00:07:27,850
Not whatever you want, but you can put many different other components.

118
00:07:27,850 --> 00:07:37,730
In my talk, I only briefly demonstrated when it's Spark itself that is doing the orchestration.

119
00:07:37,730 --> 00:07:44,290
But you can have other setups where it's actually Yarn, a product from the new framework, that is doing all this orchestration.

120
00:07:44,910 --> 00:07:48,730
And I think in 3.0, you can even have Kubernetes doing the work.

121
00:07:48,730 --> 00:07:51,610
So you have all your pods coming up to do all the work.

122
00:07:51,610 --> 00:07:53,610
So it's much sexier.

123
00:07:53,610 --> 00:08:01,230
And the one that I showed, the one that bypasses authentication, that one only works on Spark standalone mode.

124
00:08:01,230 --> 00:08:04,010
So when it's actually Spark that's doing all the stuff.

125
00:08:04,010 --> 00:08:08,530
If you're having Yarn in front, it's a completely different story.

126
00:08:08,530 --> 00:08:12,710
It's a completely different protocol that's been going on.

127
00:08:12,750 --> 00:08:16,510
That's some Hadoop shit that's too much.

128
00:08:16,610 --> 00:08:18,260
That's a completely different beast.

129
00:08:18,610 --> 00:08:20,610
It listens on a different port, etc.

130
00:08:20,610 --> 00:08:23,310
The tool that I released, Sparky, handles it.

131
00:08:23,310 --> 00:08:26,770
But I didn't go much deeper into it.

132
00:08:26,770 --> 00:08:31,190
So there's definitely some area of research there to go into.

133
00:08:31,190 --> 00:08:36,530
And that's the default mode when you're using AWS Managed Service called EMR.

134
00:08:36,530 --> 00:08:42,690
So if you're using EMR and you spawn a cluster using their service, it will by default use this Yarn mode.

135
00:08:42,690 --> 00:08:48,150
So it will spin up Yarn and then the work will be done by Spark Worker.

136
00:08:49,030 --> 00:08:52,450
How much is its widespread ratio?

137
00:08:52,830 --> 00:08:58,890
From the studies that I saw, it was around 50-50, maybe 40-60, depending on which website you see.

138
00:09:00,690 --> 00:09:06,590
Basically, the more traditional websites, the companies that had Hadoop before, will stick to Yarn.

139
00:09:06,590 --> 00:09:16,110
Whereas the new ones that have the luxury of starting a cluster of data mining from scratch, will maybe opt more for a Spark standalone cluster.

140
00:09:17,330 --> 00:09:18,730
Okay, that makes sense.

141
00:09:18,730 --> 00:09:27,790
And 50-50 means that there's stuff out there, both more to study elsewhere and more to try and hit with the research that you've done.

142
00:09:27,790 --> 00:09:29,210
So, interesting.

143
00:09:30,070 --> 00:09:30,910
Yeah, I think so.

144
00:09:30,910 --> 00:09:33,650
I would say that...

145
00:09:33,650 --> 00:09:34,910
Here's the thing.

146
00:09:35,070 --> 00:09:43,770
The idemposite community is so much focused on Windows, I find, that there's so many other great stuff to talk about and to research.

147
00:09:44,330 --> 00:09:51,150
In Windows, if you want to make a breakthrough, you have to go through 20 years of past research to try to find something new.

148
00:09:51,150 --> 00:09:55,650
Whereas in this big, shiny, new technology, well, it's right there.

149
00:09:55,650 --> 00:09:58,890
Buffer overflows of 1999, it's right there.

150
00:09:58,890 --> 00:10:02,610
So , there's much opportunity there to be taken.

151
00:10:03,650 --> 00:10:04,930
That's awesome.

152
00:10:04,970 --> 00:10:06,710
Yeah, and I'll ask a question too.

153
00:10:06,710 --> 00:10:10,730
You mentioned how much more efficient Spark is over Hadoop.

154
00:10:11,210 --> 00:10:16,110
Would you say companies that are still heavily relying on Hadoop are behind the times?

155
00:10:16,110 --> 00:10:18,970
Should they be looking at Spark, or is there better security there?

156
00:10:18,970 --> 00:10:20,910
What's your take?

157
00:10:21,590 --> 00:10:22,670
Oh my god.

158
00:10:22,670 --> 00:10:27,990
Well, Hadoop has that thing, well, it actually handles Kerberos authentication.

159
00:10:28,510 --> 00:10:31,870
Now, whether that's a good thing or not, that's up for debate.

160
00:10:31,870 --> 00:10:34,370
But , no, not really.

161
00:10:34,450 --> 00:10:40,010
My mantra, really, I work in Blue Team, so this is Blue Team talking.

162
00:10:40,010 --> 00:10:44,310
Basically, production should be boring, to paraphrase what other people were saying.

163
00:10:44,310 --> 00:10:45,310
Production should be boring.

164
00:10:45,310 --> 00:10:53,310
So, if you have a Hadoop cluster of 3,000 machines that's doing the work, and it's fine, and everything is fine, then, by all means, continue.

165
00:10:53,310 --> 00:10:55,690
If you have a mainframe to do what you need to do.

166
00:10:56,190 --> 00:11:01,770
I've seen a post, actually, that emulates what Spark does, but only using shell commands.

167
00:11:02,990 --> 00:11:04,130
It works.

168
00:11:04,130 --> 00:11:05,330
It's much cheaper.

169
00:11:05,330 --> 00:11:06,810
It runs on a single machine.

170
00:11:07,190 --> 00:11:09,870
And you don't have all the partitioning and the shuffling.

171
00:11:09,870 --> 00:11:11,370
You don't have all the network latencies.

172
00:11:11,370 --> 00:11:20,050
You don't have all that crappy stuff that makes it very, very, very slow, in comparison to keeping everything in memory and just working on a big slice of an object.

173
00:11:20,050 --> 00:11:30,830
So, I mean, hey, whatever works for the company, with that particular set of talent and that particular set of circumstances and data, I mean, go for it.

174
00:11:31,030 --> 00:11:32,770
That's my opinion, yeah.

175
00:11:32,770 --> 00:11:33,890
Yeah, that's awesome.

176
00:11:33,890 --> 00:11:36,290
Yeah, and you mentioned, so you have that Blue Team mindset.

177
00:11:36,370 --> 00:11:40,830
I mean, so, I guess, how would you try to detect what you were doing during these attacks?

178
00:11:40,830 --> 00:11:48,350
Like, are there any particular tidbits you try to give to a company that's trying to secure their Spark instance, other than making sure it's patched?

179
00:11:49,210 --> 00:11:50,730
Very interesting question.

180
00:11:51,310 --> 00:11:52,510
Very interesting.

181
00:11:53,350 --> 00:11:54,530
Here's the thing.

182
00:11:56,210 --> 00:11:58,150
I never thought about that.

183
00:12:00,290 --> 00:12:03,630
That's how de-correlated the two worlds are for me.

184
00:12:03,630 --> 00:12:08,430
It's like, I do Blue Team in the morning and I do Red Team at night, and that's how, again, they are apart.

185
00:12:08,430 --> 00:12:13,170
So, I never thought about, hey, it would be good to release some Yara rules to detect this stuff.

186
00:12:13,170 --> 00:12:15,230
Never crossed my mind.

187
00:12:16,790 --> 00:12:18,890
Future research, there you go.

188
00:12:22,710 --> 00:12:28,390
No, but it's, to me, it's easier to patch.

189
00:12:28,990 --> 00:12:33,730
I know that some, well, a lot of people will not be able to patch.

190
00:12:34,610 --> 00:12:37,070
To me, the first thing you try is you try to patch.

191
00:12:37,070 --> 00:12:40,530
If you can't patch, then you want to detect, then you want to mitigate it somehow.

192
00:12:40,530 --> 00:12:50,590
So, you need to isolate it either through network firewall rules or do something like that to, you know, basically isolate it from the rest of the reducer exposure, if you will.

193
00:12:50,590 --> 00:12:55,770
Now, if you can't do that either, well, you got to detect it somehow.

194
00:12:55,770 --> 00:12:57,330
How would you detect it?

195
00:13:01,580 --> 00:13:07,220
If you have some advanced correlation in place, I think you can detect it.

196
00:13:07,220 --> 00:13:13,660
I'm talking about specifically the exploit of replaying that single theorized object that triggers command execution.

197
00:13:13,660 --> 00:13:15,460
I'm talking specifically about that.

198
00:13:15,460 --> 00:13:30,260
One way to detect it is that, unlike other Spark interactions, it's, like, in that interaction specifically, you only send one command instead of the whole charade of, hey, Spark, hello, which version are you running, master?

199
00:13:30,260 --> 00:13:32,180
Yes, I'm registering the application.

200
00:13:32,180 --> 00:13:33,340
Okay, here are the workers.

201
00:13:33,340 --> 00:13:36,920
You know, you have, like, 80 messages going around.

202
00:13:36,920 --> 00:13:39,200
But instead, in that exploit, you only have one message.

203
00:13:39,200 --> 00:13:55,460
So, if you can, if you have a tool that's advanced enough to make these kinds of correlations say that, oh, this IP address only or this session only initiated that single communication, that single packet, then maybe it's suspicious because it didn't follow up with all the stuff.

204
00:13:56,580 --> 00:13:58,140
That could give you a hint.

205
00:13:58,140 --> 00:14:04,260
You can also try the on out of memory errors that are on your machines.

206
00:14:04,260 --> 00:14:08,180
That may be a bit noisy, but that could get you going.

207
00:14:09,780 --> 00:14:17,660
But then again, a hacker could find another way to trigger the execution because I only gave out an example using the on out of memory error.

208
00:14:17,900 --> 00:14:26,600
Another way... yeah, these are the two main things to look for off the top of my head, basically.

209
00:14:27,160 --> 00:14:27,760
Sure.

210
00:14:27,760 --> 00:14:32,160
We got a follow-up question from PC Squared, it appears.

211
00:14:32,160 --> 00:14:33,300
Really well done talk.

212
00:14:33,300 --> 00:14:37,340
It looks like you did the magic on the Spark stand alone, like we were mentioning there.

213
00:14:37,440 --> 00:14:40,020
Did you get it working with Yarn 2?

214
00:14:40,320 --> 00:14:41,880
Not a big deal.

215
00:14:42,700 --> 00:14:46,400
Not a big deal, if not, but I imagine it's possible.

216
00:14:46,500 --> 00:14:49,960
Also, 100% production should be boring.

217
00:14:51,540 --> 00:14:52,460
Amazing.

218
00:14:53,720 --> 00:14:54,720
Oh, my God.

219
00:14:54,720 --> 00:14:55,360
Okay.

220
00:14:55,460 --> 00:14:56,980
So, here's the real story.

221
00:14:56,980 --> 00:15:00,900
So, I found this thing on Spark, and I was like, oh, my God, this is amazing.

222
00:15:00,900 --> 00:15:01,660
I feel so great.

223
00:15:01,660 --> 00:15:02,020
You know what?

224
00:15:02,020 --> 00:15:02,840
I'm going to follow up.

225
00:15:02,840 --> 00:15:04,520
I'm going to see how it works on Yarn.

226
00:15:04,520 --> 00:15:08,860
Then I pulled up Yarn, did the installation, made it work on Sparky, right?

227
00:15:08,860 --> 00:15:10,240
So, let's look up some runes.

228
00:15:10,240 --> 00:15:13,580
So, I turned up Wireshark, and I saw the traffic.

229
00:15:13,800 --> 00:15:17,760
It was on Wireshark, and I was like, oh, my God, I don't want to touch that.

230
00:15:18,760 --> 00:15:20,240
That's how foreign it was.

231
00:15:20,240 --> 00:15:21,660
That's how...

232
00:15:21,660 --> 00:15:24,800
Yarn is more difficult than Spark stand alone.

233
00:15:25,480 --> 00:15:27,460
It's really, really more difficult.

234
00:15:27,460 --> 00:15:29,180
And I looked at it, and I was like, fuck it.

235
00:15:29,180 --> 00:15:30,040
I don't want to look into it.

236
00:15:30,040 --> 00:15:35,060
That's really what went through my mind when I tried to do that on Yarn.

237
00:15:35,340 --> 00:15:40,880
Now, if you look for a similar vulnerability, I couldn't trigger it.

238
00:15:40,880 --> 00:15:45,100
Like, I was playing around with Yarn a little bit, but I couldn't trigger the same vulnerability.

239
00:15:45,460 --> 00:15:47,940
Does that mean that it's not vulnerable in some way?

240
00:15:47,940 --> 00:15:48,680
Of course not.

241
00:15:48,680 --> 00:15:54,820
But by that time, I was like, oh, Yarn is just too much for me.

242
00:15:55,340 --> 00:15:57,700
I'm not going to waste my time on it.

243
00:15:57,700 --> 00:15:59,200
I'm going to move on to other stuff.

244
00:15:59,760 --> 00:16:02,440
Maybe if you feel a connection with Yarn, you can dig into it.

245
00:16:02,460 --> 00:16:03,740
Please do so.

246
00:16:04,000 --> 00:16:06,440
Because like I said, there's nothing out there on Yarn.

247
00:16:06,440 --> 00:16:07,440
I didn't find anything.

248
00:16:07,440 --> 00:16:11,720
So, probably there are some stuff to be sought after in Yarn.

249
00:16:13,080 --> 00:16:15,120
Well, that makes a lot of sense.

250
00:16:15,120 --> 00:16:35,100
So, if somebody decides that they do want to pursue this, pursue the Yarn side, and would like to ask you more questions, are you available for either consultations or for answering questions that people come up with when they're doing this on their own?

251
00:16:36,220 --> 00:16:37,140
Yeah, of course.

252
00:16:37,140 --> 00:16:39,460
I mean, yeah, definitely.

253
00:16:39,620 --> 00:16:41,360
Like, my Twitter is open.

254
00:16:41,360 --> 00:16:48,380
Right at the end of this, we'll have you post any contact information you'd like for people to have access to in the TrackOne channel.

255
00:16:48,380 --> 00:16:52,400
And you can be there for people.

256
00:16:53,800 --> 00:16:54,940
Yeah, definitely.

257
00:16:54,940 --> 00:17:00,620
I mean, when you look at this stuff, like when you look at the Spark source code, it's beautifully written, by the way.

258
00:17:00,620 --> 00:17:02,520
But it's written in Scala.

259
00:17:02,520 --> 00:17:03,840
Some of it is written in Scala.

260
00:17:03,840 --> 00:17:05,640
And it can be very daunting.

261
00:17:05,640 --> 00:17:08,520
The syntax is weird.

262
00:17:08,740 --> 00:17:20,080
And when you look at information on the web that talks about Spark, I think one blog post or two maybe that are dedicated to the internals of Spark.

263
00:17:20,080 --> 00:17:27,640
Everybody talks about how shuffling works and how it partitions data and stuff, but nobody talks about how it works inside.

264
00:17:29,580 --> 00:17:34,840
So yeah, there's a lot of things to figure out and some simple stuff when you explain them.

265
00:17:34,840 --> 00:17:42,220
They look simple, but to get that information, it takes a little bit of time because nobody talks about those internals.

266
00:17:43,000 --> 00:17:45,540
So yeah, if you have any questions, please do not hesitate.

267
00:17:45,900 --> 00:17:49,840
Just hit me on a DM or just ping me or whatever.

268
00:17:52,520 --> 00:17:55,400
Yeah, it's one of the things I thought was interesting too.

269
00:17:55,400 --> 00:18:00,140
So through what you discovered, I was like, oh man, I wonder if he's going to report this to Apache.

270
00:18:00,360 --> 00:18:02,260
And then like, oh, look, he did.

271
00:18:02,560 --> 00:18:04,600
So I guess, how did that interaction go?

272
00:18:04,600 --> 00:18:06,240
How long did it take him to fix it?

273
00:18:06,240 --> 00:18:07,680
Was it a good interaction?

274
00:18:07,720 --> 00:18:10,920
Tips for other people having to reach out to companies like that?

275
00:18:12,060 --> 00:18:13,020
Okay.

276
00:18:14,960 --> 00:18:19,000
I sent them the phone, I think on 24th of December or something like that.

277
00:18:19,000 --> 00:18:20,660
Okay, very bad of me.

278
00:18:22,600 --> 00:18:29,660
Then I got a response from the Spark security team, and they said, yeah, okay, we'll look into it, etc.

279
00:18:29,660 --> 00:18:34,360
But whatever, and I didn't receive a response later.

280
00:18:34,740 --> 00:18:37,740
And then my talk at Troopers got canceled.

281
00:18:38,540 --> 00:18:42,680
And I think one week before I decided to, you know, because I was rehearsing a little bit.

282
00:18:42,680 --> 00:18:46,700
So I said, you know what, let me just write them again to see where they are on this.

283
00:18:46,900 --> 00:18:52,740
And I got a response from the same guy saying, oh, you know what, we looked it over, and we discarded it because it's not interesting.

284
00:18:53,220 --> 00:18:55,240
I was like, okay.

285
00:18:57,120 --> 00:18:58,380
Let me rephrase.

286
00:18:58,380 --> 00:19:01,260
Oh, and when I sent them, I sent them an email with a proof of concept.

287
00:19:01,260 --> 00:19:04,500
It's like a Python code that actually executes the code fully.

288
00:19:04,500 --> 00:19:06,620
So proof of concept, it's there.

289
00:19:06,620 --> 00:19:08,760
So it was really detailed email.

290
00:19:10,100 --> 00:19:13,860
And yeah, so the guy said that they discarded it and I didn't understand.

291
00:19:13,860 --> 00:19:17,360
So I wrote a second email saying, are you sure you want to do that?

292
00:19:17,360 --> 00:19:23,800
Because you're right in the documentation that when authentication is enabled between this and this component, etc.

293
00:19:24,360 --> 00:19:28,880
So you're kind of breaking this trust by allowing this vulnerability to go on, etc, etc, etc.

294
00:19:29,400 --> 00:19:31,520
And then I got a response saying, oh, my God, I'm so sorry.

295
00:19:31,520 --> 00:19:32,540
Yeah, we made a mistake.

296
00:19:32,540 --> 00:19:36,760
We thought you were talking about the RPC endpoint that we already got a report for.

297
00:19:36,760 --> 00:19:40,100
Now, indeed, this is a dangerous vulnerability, etc.

298
00:19:40,100 --> 00:19:41,280
I'm going to put everything.

299
00:19:41,420 --> 00:19:44,420
And they, you know, I mean, it happens.

300
00:19:44,420 --> 00:19:46,660
I do try it myself and sometimes I get it wrong.

301
00:19:46,660 --> 00:19:48,140
So this is normal.

302
00:19:48,760 --> 00:19:50,840
And then they went on to fix it.

303
00:19:51,000 --> 00:19:54,960
So, all in all, it took, I think, eight months.

304
00:19:54,980 --> 00:19:59,680
But since they acknowledged that it was a vulnerability, I think it only took like three months.

305
00:20:00,200 --> 00:20:00,620
Gotcha.

306
00:20:00,620 --> 00:20:02,140
So persistence was key.

307
00:20:02,940 --> 00:20:03,980
Persistence was key.

308
00:20:03,980 --> 00:20:08,720
Well, I mean, my goal was not to get it out there, to just publish it.

309
00:20:08,720 --> 00:20:11,000
And I didn't release the tool or I didn't even talk about it.

310
00:20:11,000 --> 00:20:17,100
I think I talked about it to two people who were very expert in Spark to validate that I was not saying bullshit.

311
00:20:17,560 --> 00:20:19,720
And I asked them to not disclose it.

312
00:20:20,240 --> 00:20:21,240
That's about it.

313
00:20:21,240 --> 00:20:23,680
Because my intent was not to just, you know, release that.

314
00:20:23,680 --> 00:20:25,460
I am pro full disclosure.

315
00:20:26,600 --> 00:20:29,460
But everybody should make their own choices and it depends on my mood.

316
00:20:29,460 --> 00:20:33,740
So for that one, I decided, you know what, let's just keep it on the low key anyway.

317
00:20:34,340 --> 00:20:44,180
And, yeah, and I think three weeks after disclosing it, they released a complete correction.

318
00:20:44,240 --> 00:20:48,500
Because it was not impacting only one function, it was impacting also two others.

319
00:20:48,500 --> 00:20:49,960
So they rewrote a class, etc.

320
00:20:49,960 --> 00:20:51,400
So we worked on the fix.

321
00:20:51,880 --> 00:20:57,640
Well, I worked on the origin of the fix and then they took it over because more competent than me.

322
00:20:57,640 --> 00:21:02,700
And then the thing that was really long is the release process.

323
00:21:02,700 --> 00:21:04,980
They had the fix three weeks after I reported it.

324
00:21:04,980 --> 00:21:14,200
But since like, let's say, beginning of April, they waited until July on the other version 2.4.6 to actually release the fix.

325
00:21:15,020 --> 00:21:18,960
So that's the part that was long and not actually fixing it, just releasing it.

326
00:21:19,220 --> 00:21:20,220
But, yeah.

327
00:21:20,420 --> 00:21:23,460
It's out there and we're all more secure for your efforts.

328
00:21:23,460 --> 00:21:27,300
So it's, you know, better than having no one looked in that area before.

329
00:21:28,680 --> 00:21:30,340
Yeah, yeah, definitely.

330
00:21:30,340 --> 00:21:31,900
And now we're talking about that.

331
00:21:31,900 --> 00:21:37,640
There was some interesting point during basically this whole Spark adventure.

332
00:21:37,640 --> 00:21:46,980
And I think I briefly touched upon it during the talk is that I was so, like, I spent like a couple of hours, even days, trying to understand how the RPC stuff works.

333
00:21:46,980 --> 00:21:52,200
And only to find out a couple of days later that somebody already published that vault.

334
00:21:52,920 --> 00:21:55,940
So, yeah, that was very interesting.

335
00:21:55,940 --> 00:21:56,880
I don't know if it happened.

336
00:21:56,880 --> 00:21:58,300
I think it happens to some people.

337
00:21:58,300 --> 00:22:09,680
But basically, you're just so stuck in, like, you find yourself drawn into that code base and trying to understand how it works when all you have to do is actually Google the right keyword to get the actual exploit.

338
00:22:11,000 --> 00:22:14,700
And when you do it, you feel a sense of frustration.

339
00:22:16,080 --> 00:22:18,540
But it's part of the game.

340
00:22:18,820 --> 00:22:32,020
And that's pretty good advice for all of us who are into finding vulnerabilities and figuring out how to report them is that enumeration step goes on the software that you're working on.

341
00:22:32,020 --> 00:22:35,920
And you have to read the state of the art of what people know.

342
00:22:35,920 --> 00:22:39,060
So that's a good thing to reinforce.

343
00:22:39,680 --> 00:22:40,440
Exactly.

344
00:22:43,220 --> 00:22:44,060
Awesome.

345
00:22:44,060 --> 00:22:48,920
So tell us a little bit more about the next thing that you're moving on to.

346
00:22:48,920 --> 00:22:52,640
It sounds like you already have this idea of what you're researching next.

347
00:22:54,920 --> 00:22:58,620
Actually, I wrote down a small plan.

348
00:22:58,620 --> 00:23:03,600
Not a plan, but bullet points of what I want to do later.

349
00:23:04,120 --> 00:23:10,580
I just released this tool called Reflect PE, which reflectively loads an assembly in a PE object.

350
00:23:10,580 --> 00:23:15,680
And shout out to name right, please.

351
00:23:19,860 --> 00:23:20,580
Rob?

352
00:23:22,740 --> 00:23:26,180
If needed, you can post it in the track one channel at the end.

353
00:23:26,180 --> 00:23:27,000
I will.

354
00:23:27,000 --> 00:23:27,920
I will post it.

355
00:23:27,920 --> 00:23:28,160
Okay.

356
00:23:28,160 --> 00:23:29,540
So his name is Rob Knopp.

357
00:23:29,540 --> 00:23:30,960
Well, he goes by the handle of Rob Knopp.

358
00:23:30,960 --> 00:23:31,460
Sorry.

359
00:23:31,980 --> 00:23:37,760
But yeah, I wrote a tool called Reflect PE, which reflectively loads a PE executable in memory.

360
00:23:38,720 --> 00:23:44,640
And I borrowed his basically tool to reflectively load also PE assemblies.

361
00:23:44,640 --> 00:23:46,000
So Rob Knopp, great tool.

362
00:23:47,100 --> 00:23:48,780
And why did I do that?

363
00:23:48,780 --> 00:23:50,700
So this is what I'm working on right now.

364
00:23:50,700 --> 00:24:05,680
And why did I do it is very, I find interesting, is that when you look at the same code to do the same thing, but in PowerShell and other languages, is that it's a huge script of like 2000 lines of code.

365
00:24:05,680 --> 00:24:07,100
No tests whatsoever.

366
00:24:07,100 --> 00:24:07,960
And you just launch it.

367
00:24:07,960 --> 00:24:09,740
And if it doesn't work, well, too bad.

368
00:24:09,740 --> 00:24:12,600
You can't debug a 2000 line PowerShell script.

369
00:24:12,720 --> 00:24:20,660
So I decided to write it from scratch using Golang and using like, you know, testing and good dev practices, etc.

370
00:24:20,820 --> 00:24:26,620
So that anybody could just look at the code and understand what's going on and where are the steps and where it didn't work, etc.

371
00:24:26,620 --> 00:24:28,360
So this is what I'm really working on.

372
00:24:28,360 --> 00:24:34,020
I'm trying to make this tool that will actually be easy to understand for people who just get into it.

373
00:24:34,020 --> 00:24:39,720
And just want to understand how reflectively loading something works on Windows.

374
00:24:40,460 --> 00:24:45,080
So that's the thing that taking most of my time right now.

375
00:24:45,080 --> 00:24:47,880
And my next big thing is Kubernetes.

376
00:24:47,880 --> 00:24:49,860
I really want to get into it.

377
00:24:50,280 --> 00:25:01,300
It is all of the talks that I saw, like 90% of them talk about how to abuse a Kubernetes set up with no authentication, no airbox, no nothing.

378
00:25:01,300 --> 00:25:14,900
I want to see what is there to exploit, you know, once you have all these security hardens in place, is it possible to bypass, I don't know, namespace isolation, is it possible to bypass team control, etc.

379
00:25:14,900 --> 00:25:27,080
I want to really dig into it because there's, again, like I said, I really look out for these niche topics where there is not much talk, where there is not much tools available.

380
00:25:27,080 --> 00:25:32,060
And try to focus on them and hopefully something gets out of this research.

381
00:25:33,600 --> 00:25:34,380
That's awesome.

382
00:25:34,380 --> 00:25:35,160
That's awesome.

383
00:25:35,160 --> 00:25:38,440
Yeah, so we're kind of wrapping up here at the end of the time.

384
00:25:38,440 --> 00:25:43,120
Is there anything else you want to share with the group or again, how to reach out to you?

385
00:25:43,120 --> 00:25:49,060
Again, we can post your contact information, but anywhere you want to point anyone to before we call it an end here?

386
00:25:51,840 --> 00:26:00,820
No, nothing except that, basically, thank you to the Spark team, first of all, because honestly, amazing.

387
00:26:00,820 --> 00:26:11,360
Like when you look at the code base, you understand how talented these people are and it's really a pleasure and a privilege to actually have interacted with some of you, especially some of you that I saw on stage talking about it.

388
00:26:11,360 --> 00:26:14,500
So I was pretty excited, though I tried to hide it a little bit.

389
00:26:14,620 --> 00:26:16,480
Yeah, so thank you all.

390
00:26:17,080 --> 00:26:28,840
But again, like for people who want to basically go into InfoSec and break into it, give talks, do research, etc.

391
00:26:29,000 --> 00:26:41,120
My only advice is basically look out for all these small niche topics that are basically abandoned by everybody and go deep until you feel you understand it and then just go crazy with it.

392
00:26:41,120 --> 00:26:43,740
And then most surely something will pop out.

393
00:26:44,160 --> 00:26:45,200
Awesome, awesome.

394
00:26:45,200 --> 00:26:46,420
That's great advice.

395
00:26:46,620 --> 00:26:47,860
I appreciate that.

396
00:26:47,940 --> 00:26:57,100
Yeah, and I was going to say just one more thing, basically, was getting to, you know, since you just actually did a live demo, right?

397
00:26:57,120 --> 00:27:03,960
One thing you're missing out on not having the live DEF CON crowd is a round of applause for having a demo work on stage.

398
00:27:03,960 --> 00:27:11,260
So since you actually did kind of roll that out, I just wanted to give you a nice round of applause and get that in the Discord chat, too.

399
00:27:12,320 --> 00:27:15,560
Thank you for getting that and getting it working and everything.

400
00:27:16,060 --> 00:27:17,260
Thank you very much.

401
00:27:17,920 --> 00:27:18,720
Cool.

402
00:27:18,920 --> 00:27:20,600
So I think that's it for us.

403
00:27:20,600 --> 00:27:22,900
So again, thank you again for joining us.

404
00:27:22,900 --> 00:27:27,000
And thanks for the talk and look forward to what you have coming up next.

405
00:27:27,000 --> 00:27:28,600
And hopefully see you next year at DEF CON.

406
00:27:28,980 --> 00:27:30,120
Thank you very much.

407
00:27:30,120 --> 00:27:31,240
Thank you.

408
00:27:31,240 --> 00:27:32,540
Bye-bye.