1
00:00:00,150 --> 00:00:02,190
- And without any further ado talking

2
00:00:02,190 --> 00:00:03,723
about backdooring pickles.

3
00:00:07,011 --> 00:00:08,760
(audience applauding)

4
00:00:08,760 --> 00:00:09,593
- Thank you.

5
00:00:14,560 --> 00:00:16,440
So to start off,

6
00:00:16,440 --> 00:00:17,790
I work for NVIDIAs AI Red Team.

7
00:00:17,790 --> 00:00:20,580
They let me do this research
or letting me share it

8
00:00:20,580 --> 00:00:21,840
but everything is my own opinion

9
00:00:21,840 --> 00:00:24,930
and not an official
statement from the company.

10
00:00:24,930 --> 00:00:27,900
But yeah, let's talk
about backdooring pickles.

11
00:00:27,900 --> 00:00:30,420
So, 11 years ago, Mark did a talk

12
00:00:30,420 --> 00:00:33,720
at BlackHat called, Sour Pickles.

13
00:00:33,720 --> 00:00:35,670
And in it he talks about how to do

14
00:00:35,670 --> 00:00:38,133
desterilization attacks against pickles.

15
00:00:39,270 --> 00:00:42,000
Not a whole lot's changed from
then, since then, back then.

16
00:00:42,000 --> 00:00:44,010
11 years ago, there was warning message,

17
00:00:44,010 --> 00:00:48,180
it said pickles shouldn't be
used from an untrusted source.

18
00:00:48,180 --> 00:00:49,860
Makes sense 'cause it is code,

19
00:00:49,860 --> 00:00:51,120
pickles are code at the end of the day,

20
00:00:51,120 --> 00:00:53,340
it's a assembly language.

21
00:00:53,340 --> 00:00:54,870
And the attacks that he talked

22
00:00:54,870 --> 00:00:57,990
about were predominantly
desterilization attacks.

23
00:00:57,990 --> 00:00:59,310
So, this is something along the lines

24
00:00:59,310 --> 00:01:01,170
where a web server gives you a cookie,

25
00:01:01,170 --> 00:01:02,250
that cookie's a pickle,

26
00:01:02,250 --> 00:01:04,110
you can replace it with a pickle you craft

27
00:01:04,110 --> 00:01:07,953
in order to get a shell on the
web server when they load it.

28
00:01:09,510 --> 00:01:12,183
So yeah, not a whole lot
changed, as I mentioned.

29
00:01:13,140 --> 00:01:15,120
Pickles still have this warning message,

30
00:01:15,120 --> 00:01:16,260
it's a little more of verbose,

31
00:01:16,260 --> 00:01:17,880
gives you a few more
ideas on what you can do,

32
00:01:17,880 --> 00:01:19,410
but the end of the day, it's still,

33
00:01:19,410 --> 00:01:20,940
if it's coming from an untrusted source

34
00:01:20,940 --> 00:01:24,090
you shouldn't load it
'cause they are still code,

35
00:01:24,090 --> 00:01:25,830
that part hasn't changed at all.

36
00:01:25,830 --> 00:01:28,650
But what has changed is in
2016 Machine Learning Library

37
00:01:28,650 --> 00:01:31,680
started to be released that used pickles

38
00:01:31,680 --> 00:01:34,740
as the way that they save their models.

39
00:01:34,740 --> 00:01:36,540
And so this creates
something along the lines

40
00:01:36,540 --> 00:01:39,480
of a macro in a Word document
or in an Office document

41
00:01:39,480 --> 00:01:42,000
where you can send it to someone

42
00:01:42,000 --> 00:01:44,640
and if they load it then you
get a show on their system

43
00:01:44,640 --> 00:01:47,103
and do whatever you want with it.

44
00:01:48,540 --> 00:01:49,650
Yeah.

45
00:01:49,650 --> 00:01:51,270
So, I've mentioned models.

46
00:01:51,270 --> 00:01:52,870
Just really quick, what is that?

47
00:01:53,790 --> 00:01:55,650
Models are used in artificial intelligence

48
00:01:55,650 --> 00:01:57,300
and machine learning,

49
00:01:57,300 --> 00:01:58,770
for what I'm talking
about they're combination

50
00:01:58,770 --> 00:01:59,820
of layers and weights,

51
00:01:59,820 --> 00:02:01,350
that probably has very little meaning

52
00:02:01,350 --> 00:02:04,290
to most of you as well.
So, what is a layer?

53
00:02:04,290 --> 00:02:06,060
A layer is the mathematical equation

54
00:02:06,060 --> 00:02:08,610
that lets you give it
one input, like an image.

55
00:02:08,610 --> 00:02:12,660
And it goes through this
equation and gives you a label

56
00:02:12,660 --> 00:02:16,410
like cat, dog, chicken,
ostrich, whatever it is.

57
00:02:16,410 --> 00:02:19,050
And these layers we represent as code

58
00:02:19,050 --> 00:02:23,490
in Python specifically
for the Python frameworks.

59
00:02:23,490 --> 00:02:26,250
The weights are the
coefficient for that equation,

60
00:02:26,250 --> 00:02:27,780
it's really just the data that's trained.

61
00:02:27,780 --> 00:02:28,860
So when we're training models,

62
00:02:28,860 --> 00:02:31,230
this is what we're training,
we're training the weights

63
00:02:31,230 --> 00:02:32,970
and that can just be viewed as data.

64
00:02:32,970 --> 00:02:34,170
And because it's a combination

65
00:02:34,170 --> 00:02:35,943
of a bit of code and a ton of data,

66
00:02:37,080 --> 00:02:38,490
it makes sense to store it in pickles

67
00:02:38,490 --> 00:02:39,810
as long as you ignore the fact

68
00:02:39,810 --> 00:02:42,030
that researchers are
gonna be distributing it

69
00:02:42,030 --> 00:02:43,893
and sharing it as to do research,

70
00:02:44,970 --> 00:02:46,980
and this is exactly the opposite

71
00:02:46,980 --> 00:02:49,740
of what that warning
message said we should do.

72
00:02:49,740 --> 00:02:50,970
Just another little note,

73
00:02:50,970 --> 00:02:53,190
a lot of the frameworks
tend to save a model

74
00:02:53,190 --> 00:02:55,800
as multiple pickles in a single file,

75
00:02:55,800 --> 00:02:57,230
which if you do anything with it

76
00:02:57,230 --> 00:02:59,883
it can get confusing at first, but yeah.

77
00:03:01,740 --> 00:03:03,630
So, how do you make a malicious pickle?

78
00:03:03,630 --> 00:03:07,710
This is just randomly, like
just arbitrarily malicious.

79
00:03:07,710 --> 00:03:10,980
It doesn't do anything except
for be malicious, sorry.

80
00:03:10,980 --> 00:03:13,890
This is an example from a blog from 2011.

81
00:03:13,890 --> 00:03:18,070
And what it says is, when
you pickle up exploit

82
00:03:19,380 --> 00:03:20,970
this is how I want you to recreate exploit

83
00:03:20,970 --> 00:03:22,920
when you depickle it later, you depickle,

84
00:03:22,920 --> 00:03:26,790
you recreate it by calling
subprocess.Popen with bin/sh.

85
00:03:26,790 --> 00:03:29,040
Now obviously that's not
gonna create an exploit class

86
00:03:29,040 --> 00:03:30,540
it will just be an exploit,

87
00:03:30,540 --> 00:03:33,030
in this case to give you
a shell on a web server.

88
00:03:33,030 --> 00:03:35,223
So, yeah.

89
00:03:36,300 --> 00:03:37,440
That's not what we wanna do though

90
00:03:37,440 --> 00:03:39,540
because I wanted to find
a way to place a backdoor

91
00:03:39,540 --> 00:03:42,630
into a model that still works
exactly as it's supposed to

92
00:03:42,630 --> 00:03:44,430
and gives you a shell as well.

93
00:03:44,430 --> 00:03:48,150
So, this is the simplified
version of the attack

94
00:03:48,150 --> 00:03:49,920
if all you want is the shell,

95
00:03:49,920 --> 00:03:52,290
you don't care about it working still.

96
00:03:52,290 --> 00:03:53,640
And it gives us a nice example

97
00:03:53,640 --> 00:03:56,130
to talk a little bit more about pickles.

98
00:03:56,130 --> 00:03:58,830
So, this output on the screen is thanks to

99
00:03:58,830 --> 00:04:01,290
a library built into
Python called pickletools

100
00:04:01,290 --> 00:04:04,230
or the library's, the
module's pickletools,

101
00:04:04,230 --> 00:04:05,880
and it's built in and it has a function

102
00:04:05,880 --> 00:04:10,380
called dis, for disassemble
and it produces this output.

103
00:04:10,380 --> 00:04:12,550
The first box in the output is

104
00:04:13,440 --> 00:04:16,260
getting the function
pointer to subprocess.Popen,

105
00:04:16,260 --> 00:04:19,920
it writes subprocess and
Popen onto the stack.

106
00:04:19,920 --> 00:04:21,540
And then it calls stack global

107
00:04:21,540 --> 00:04:23,220
which takes those two pointers,

108
00:04:23,220 --> 00:04:25,740
collapses it all down
into a single pointer

109
00:04:25,740 --> 00:04:27,993
that points to the function.

110
00:04:29,190 --> 00:04:30,150
After that in box two,

111
00:04:30,150 --> 00:04:31,980
we mark the beginning of
all of our parameters,

112
00:04:31,980 --> 00:04:33,750
write them onto the stack as well.

113
00:04:33,750 --> 00:04:35,220
Call tuple, which again,

114
00:04:35,220 --> 00:04:37,350
collapses them all down into one pointer

115
00:04:37,350 --> 00:04:39,600
that is all of the parameters.

116
00:04:39,600 --> 00:04:41,070
And then the last instruction's

117
00:04:41,070 --> 00:04:42,090
really the most important,

118
00:04:42,090 --> 00:04:45,030
it's reduce and that calls the
function with the parameters,

119
00:04:45,030 --> 00:04:47,010
and so it takes the two
pointers on the stack,

120
00:04:47,010 --> 00:04:50,520
pops 'em off, throws 'em
away and puts onto the stack

121
00:04:50,520 --> 00:04:54,393
just the one pointer to the return value.

122
00:04:57,060 --> 00:05:00,090
So, now that that's
understood a little bit more,

123
00:05:00,090 --> 00:05:02,610
how do we make a pickle more malicious?

124
00:05:02,610 --> 00:05:04,170
This really cool tool called fickling,

125
00:05:04,170 --> 00:05:07,020
it was released last
year in the AI Village,

126
00:05:07,020 --> 00:05:08,670
and it's made by Trail of Bits.

127
00:05:08,670 --> 00:05:09,960
It's really good.

128
00:05:09,960 --> 00:05:11,640
It allows you to inject code

129
00:05:11,640 --> 00:05:14,250
into the beginning of a
pickle or the end of a pickle,

130
00:05:14,250 --> 00:05:17,460
and it attempts to detect
if there's malice within it.

131
00:05:17,460 --> 00:05:19,140
Unfortunately for me,

132
00:05:19,140 --> 00:05:21,900
it's a little more
complicated than required

133
00:05:21,900 --> 00:05:24,420
and so it ended up being
easier for me to write

134
00:05:24,420 --> 00:05:26,950
a solution that only injects in

135
00:05:28,050 --> 00:05:29,730
instead of trying to help patch

136
00:05:29,730 --> 00:05:32,190
some of the bugs with how it worked.

137
00:05:32,190 --> 00:05:33,180
And then the second issue,

138
00:05:33,180 --> 00:05:35,310
which would've been easy
to fix if that was it,

139
00:05:35,310 --> 00:05:37,380
is that it can only
inject into the beginning

140
00:05:37,380 --> 00:05:38,670
unless you wanna replace the output

141
00:05:38,670 --> 00:05:40,970
in which case it can
only inject into the end.

142
00:05:43,890 --> 00:05:46,050
So, why do I say it's a
little more complicated?

143
00:05:46,050 --> 00:05:48,360
So, as we've covered,

144
00:05:48,360 --> 00:05:50,040
pickles, you can't just load 'em

145
00:05:50,040 --> 00:05:51,690
because you'll get a shell on the system

146
00:05:51,690 --> 00:05:54,330
or you could have somebody doing that.

147
00:05:54,330 --> 00:05:58,200
So they use a symbolic interpreter
so that it won't happen.

148
00:05:58,200 --> 00:05:59,670
And that works really well

149
00:05:59,670 --> 00:06:02,700
but you have to reimplement
everything about pickles

150
00:06:02,700 --> 00:06:04,560
in order for that to work.

151
00:06:04,560 --> 00:06:05,850
And I kept running into bugs

152
00:06:05,850 --> 00:06:10,170
and ultimately it just became
easier to do something else.

153
00:06:10,170 --> 00:06:11,883
And when I was trying to patch it,

154
00:06:13,110 --> 00:06:14,670
the tool they created is what led me

155
00:06:14,670 --> 00:06:16,020
down all this research in the first place,

156
00:06:16,020 --> 00:06:18,070
so I really can't give 'em enough credit.

157
00:06:19,110 --> 00:06:21,773
I found it was easier to use
something else, unfortunately.

158
00:06:23,250 --> 00:06:26,250
Now why can it only
inject into the beginning?

159
00:06:26,250 --> 00:06:29,010
As I mentioned before,
when you reduce down.

160
00:06:29,010 --> 00:06:31,770
So, on the top of the
slide in the blue box

161
00:06:31,770 --> 00:06:34,080
is the code that fickling adds,

162
00:06:34,080 --> 00:06:37,320
in this case it's exacting print hi,

163
00:06:37,320 --> 00:06:39,543
nothing fancy, but it
works for the example.

164
00:06:40,710 --> 00:06:43,200
So when that all runs, it
leaves one pointer on the stack

165
00:06:43,200 --> 00:06:45,647
and in the beginning of a
pickle that doesn't matter

166
00:06:45,647 --> 00:06:47,130
'cause it all collapses down,

167
00:06:47,130 --> 00:06:50,280
creates an object and
that object's returned

168
00:06:50,280 --> 00:06:51,960
and you have one value left on the stack

169
00:06:51,960 --> 00:06:55,080
that as the error message
on the bottom says,

170
00:06:55,080 --> 00:06:57,540
it's not empty so it's technically wrong

171
00:06:57,540 --> 00:06:59,840
but it doesn't matter
'cause everything works.

172
00:07:01,170 --> 00:07:03,780
But if you put that
same code anywhere else

173
00:07:03,780 --> 00:07:06,000
it's going to alter the stack in a way

174
00:07:06,000 --> 00:07:07,850
where the instructions that follow it

175
00:07:08,730 --> 00:07:09,563
don't work correctly

176
00:07:09,563 --> 00:07:11,190
because the stack is effectively corrupted

177
00:07:11,190 --> 00:07:12,690
for what they're trying to do.

178
00:07:13,950 --> 00:07:15,180
And I wanted to be able to

179
00:07:15,180 --> 00:07:17,523
just add code arbitrarily into a pickle.

180
00:07:21,030 --> 00:07:23,040
So, just a few more things about pickles

181
00:07:23,040 --> 00:07:24,740
and then we can get to the attack.

182
00:07:25,890 --> 00:07:27,450
Pickles are an instruction type,

183
00:07:27,450 --> 00:07:29,820
instruction set not a file type.

184
00:07:29,820 --> 00:07:31,530
So for detection this starts to matter

185
00:07:31,530 --> 00:07:34,470
because a pickle doesn't
have any magic bites

186
00:07:34,470 --> 00:07:37,800
that you can detect to
identify it as a pickle.

187
00:07:37,800 --> 00:07:39,570
There's no specific file extension

188
00:07:39,570 --> 00:07:44,220
because PI Torch is .pth and
not everything is about pkl,

189
00:07:44,220 --> 00:07:45,780
it can be anything.

190
00:07:45,780 --> 00:07:48,060
NumPy like NPZ sometimes has pickles

191
00:07:48,060 --> 00:07:48,893
and sometimes it doesn't

192
00:07:48,893 --> 00:07:53,160
and so you don't have just
one file you can look at.

193
00:07:53,160 --> 00:07:55,890
Also there's no forking
or conditional logic

194
00:07:55,890 --> 00:07:57,900
which makes it so that we
can inject in the middle

195
00:07:57,900 --> 00:08:00,540
'cause we don't have to worry
about it jumping around,

196
00:08:00,540 --> 00:08:01,890
bite offsets don't matter,

197
00:08:01,890 --> 00:08:04,240
you can just add code in
and it should be fine.

198
00:08:05,970 --> 00:08:08,190
As we've shown, you can
import any Python callable.

199
00:08:08,190 --> 00:08:12,780
Subprocess isn't imported
when we exploited it earlier,

200
00:08:12,780 --> 00:08:15,000
pickle actually imports
the subprocess module

201
00:08:15,000 --> 00:08:16,470
and gets the function for you.

202
00:08:16,470 --> 00:08:19,440
So, if it's installed on
the system you can use it.

203
00:08:19,440 --> 00:08:21,420
And there's an instruction called POP

204
00:08:21,420 --> 00:08:23,270
that takes the value and discards it.

205
00:08:24,780 --> 00:08:28,530
So, I keep saying that I
had a simpler solution.

206
00:08:28,530 --> 00:08:30,690
On the right side of this
screen are the 50 lines of code

207
00:08:30,690 --> 00:08:33,483
that I use to inject into any pickle.

208
00:08:34,350 --> 00:08:37,000
And 50 lines of code was
easier than writing patches.

209
00:08:38,010 --> 00:08:40,170
And so when I wrote this
I had a few requirements.

210
00:08:40,170 --> 00:08:41,910
I wanted it to not be obvious to the user

211
00:08:41,910 --> 00:08:43,650
or intrusion detection system

212
00:08:43,650 --> 00:08:45,903
because I don't wanna get caught.

213
00:08:47,580 --> 00:08:49,080
I wanted to parse the
pickle without loading it

214
00:08:49,080 --> 00:08:50,130
but at the same time I don't wanna use

215
00:08:50,130 --> 00:08:51,930
a symbolic interpreter

216
00:08:51,930 --> 00:08:54,000
because I saw how that
can get complicated.

217
00:08:54,000 --> 00:08:56,940
And I want to inject into
an arbitrary location,

218
00:08:56,940 --> 00:09:00,360
mostly that's just to help
support the first point

219
00:09:00,360 --> 00:09:03,273
but also because I can, it
makes it harder to detect, so.

220
00:09:05,490 --> 00:09:06,390
All right, so how do we avoid

221
00:09:06,390 --> 00:09:08,883
being obvious to a user and an IDS?

222
00:09:10,110 --> 00:09:11,280
Well, we'd spin off our own thread

223
00:09:11,280 --> 00:09:12,300
so it hides in the background.

224
00:09:12,300 --> 00:09:14,370
We don't wanna stall the pickle,

225
00:09:14,370 --> 00:09:15,840
we want it to load and everything to work

226
00:09:15,840 --> 00:09:17,730
exactly as the user expects.

227
00:09:17,730 --> 00:09:18,630
If they're loading up a model

228
00:09:18,630 --> 00:09:20,490
and it's supposed to
generate a picture of a cat

229
00:09:20,490 --> 00:09:21,540
they need to see a cat

230
00:09:21,540 --> 00:09:23,190
otherwise they're gonna get suspicious.

231
00:09:23,190 --> 00:09:25,290
So, we have our thread
running in the background

232
00:09:25,290 --> 00:09:28,053
that is running our agent
to call back to our C2.

233
00:09:30,810 --> 00:09:32,520
Also size isn't a concern at all,

234
00:09:32,520 --> 00:09:36,630
most models are hundreds of
megabytes or more like gigs,

235
00:09:36,630 --> 00:09:38,040
maybe more than that.

236
00:09:38,040 --> 00:09:39,960
So adding one meg or one kilobyte,

237
00:09:39,960 --> 00:09:42,570
nobody's gonna notice most likely,

238
00:09:42,570 --> 00:09:44,770
that's my assumption,
if I'm wrong so be it.

239
00:09:45,810 --> 00:09:47,130
But I do use Zlib compression

240
00:09:47,130 --> 00:09:50,220
and that's not to actually
make things smaller.

241
00:09:50,220 --> 00:09:52,440
It's because Base64 makes
a really long string

242
00:09:52,440 --> 00:09:53,273
that you're adding into the file,

243
00:09:53,273 --> 00:09:55,980
and if you run strings
and you see a long string,

244
00:09:55,980 --> 00:09:57,150
you might get suspicious.

245
00:09:57,150 --> 00:10:00,033
Zlib makes binary data, so it
was just a simple solution.

246
00:10:00,900 --> 00:10:03,360
And then finally, as long
as we don't launch MimiKatz

247
00:10:03,360 --> 00:10:06,270
or anything that's just
super malicious and obvious,

248
00:10:06,270 --> 00:10:08,520
and gonna get detected should be fine,

249
00:10:08,520 --> 00:10:09,620
shouldn't be detected.

250
00:10:12,570 --> 00:10:13,530
So, how do we load the pickle

251
00:10:13,530 --> 00:10:15,903
without using a symbolic interpreter?

252
00:10:17,430 --> 00:10:19,260
So, the output I showed before,

253
00:10:19,260 --> 00:10:24,030
there were some numbers on the
left side of the disassembly,

254
00:10:24,030 --> 00:10:26,220
those are the bite offset into the pickle.

255
00:10:26,220 --> 00:10:28,260
And because every instruction,

256
00:10:28,260 --> 00:10:29,610
there's no jumping,
there's nothing like that

257
00:10:29,610 --> 00:10:31,380
that we have to worry about.

258
00:10:31,380 --> 00:10:33,600
We can make those bite offset change,

259
00:10:33,600 --> 00:10:35,760
make those bite offsets change

260
00:10:35,760 --> 00:10:37,380
and nothing's gonna go wrong,

261
00:10:37,380 --> 00:10:38,880
it'll still work the same

262
00:10:38,880 --> 00:10:42,033
so long as the stack is
exactly as it's supposed to be.

263
00:10:42,930 --> 00:10:46,230
So, what we do is we
pick an arbitrary line

264
00:10:46,230 --> 00:10:48,090
from this disassembly, grab the number,

265
00:10:48,090 --> 00:10:50,250
write that many bites
into the output file,

266
00:10:50,250 --> 00:10:52,380
write our malicious instructions
into the output file

267
00:10:52,380 --> 00:10:54,330
and then write the remainder
into the output file.

268
00:10:54,330 --> 00:10:57,060
And you have a pickle
that'll run your instructions

269
00:10:57,060 --> 00:10:59,130
so long as you're not,

270
00:10:59,130 --> 00:11:01,320
so long as the stack is exactly as it was

271
00:11:01,320 --> 00:11:02,870
when your instructions started.

272
00:11:07,170 --> 00:11:09,840
So, how do we avoid leaving
a trace in the stack?

273
00:11:09,840 --> 00:11:12,600
Well, I only use pickle
instructions that alter the stack.

274
00:11:12,600 --> 00:11:14,070
If you know anything
about pickle instructions

275
00:11:14,070 --> 00:11:16,500
you might be wondering why I
don't mention anything else

276
00:11:16,500 --> 00:11:18,577
and that's 'cause I just don't use it.

277
00:11:18,577 --> 00:11:20,280
It does exist, if you wanna know about it

278
00:11:20,280 --> 00:11:23,520
Mark's talk covers everything,
and it's really good.

279
00:11:23,520 --> 00:11:26,760
But for our examples,
we only use the stack

280
00:11:26,760 --> 00:11:28,860
and so, that's all covered.

281
00:11:28,860 --> 00:11:30,090
And because we only use the stack

282
00:11:30,090 --> 00:11:31,350
that's all we have to clean up.

283
00:11:31,350 --> 00:11:34,710
And so, at the end we
add a POP instruction

284
00:11:34,710 --> 00:11:36,210
and it pops off whatever,

285
00:11:36,210 --> 00:11:37,950
well, in this case, we're
adding a single value

286
00:11:37,950 --> 00:11:39,750
to the stack, pops that one value off

287
00:11:39,750 --> 00:11:42,240
and everything's exactly
as it was initially

288
00:11:42,240 --> 00:11:44,513
so everything works exactly
as it's supposed to.

289
00:11:46,410 --> 00:11:48,540
So, this is a bit of an example.

290
00:11:48,540 --> 00:11:49,590
Here, I'm downloading a pickle

291
00:11:49,590 --> 00:11:53,670
that I've injected an agent
into for the Mythic C2

292
00:11:53,670 --> 00:11:57,780
and I'm loading the pickle
and it takes a second,

293
00:11:57,780 --> 00:11:59,370
but I'll get a callback.

294
00:11:59,370 --> 00:12:01,290
This callback is running in the background

295
00:12:01,290 --> 00:12:03,540
while the GAN is still loading.

296
00:12:03,540 --> 00:12:06,720
When the GAN finishes loading,
you see a picture of a cat.

297
00:12:06,720 --> 00:12:09,510
I can tell it to start
running calc, which I did.

298
00:12:09,510 --> 00:12:11,760
And when the agent calls
back, Calc will start

299
00:12:11,760 --> 00:12:13,620
even though I'm able to look at

300
00:12:13,620 --> 00:12:16,560
a bunch of different pictures
of cats while it's working

301
00:12:16,560 --> 00:12:18,510
and Calc pops and, yeah.

302
00:12:18,510 --> 00:12:20,460
In a real attack, obviously
you're not gonna POP Calc

303
00:12:20,460 --> 00:12:21,780
'cause, you know, that's obvious.

304
00:12:21,780 --> 00:12:24,360
But you have an agent
running on their system,

305
00:12:24,360 --> 00:12:25,950
you can do whatever you want with it.

306
00:12:25,950 --> 00:12:27,540
The hard part about the attack

307
00:12:27,540 --> 00:12:29,940
is figuring out how to inject the code

308
00:12:29,940 --> 00:12:32,550
into a malicious agent or
inject the malicious code

309
00:12:32,550 --> 00:12:34,323
into an existing pickle.

310
00:12:35,280 --> 00:12:37,380
The way that you go about doing this,

311
00:12:37,380 --> 00:12:38,370
it's really up to you I guess,

312
00:12:38,370 --> 00:12:40,830
but there's three ways
I see it working out.

313
00:12:40,830 --> 00:12:42,570
There's the supply chain attacks.

314
00:12:42,570 --> 00:12:46,560
If someone's distributing a
bunch of pickles on a website

315
00:12:46,560 --> 00:12:48,960
or however, and you can modify them

316
00:12:48,960 --> 00:12:51,150
then you can get a shell
on all of their customers

317
00:12:51,150 --> 00:12:53,310
assuming that they don't detect it.

318
00:12:53,310 --> 00:12:54,660
There's also watering hole attacks.

319
00:12:54,660 --> 00:12:57,600
If a bunch of data scientists
have a shared Network Share,

320
00:12:57,600 --> 00:13:00,960
they keep passing
pickles or pickled models

321
00:13:00,960 --> 00:13:02,850
back and forth between each other

322
00:13:02,850 --> 00:13:04,260
and you have access to that as well.

323
00:13:04,260 --> 00:13:05,460
You can modify it,

324
00:13:05,460 --> 00:13:06,720
and the next one of them

325
00:13:06,720 --> 00:13:08,550
that loads the one that you modified

326
00:13:08,550 --> 00:13:11,490
will spawn a show on their
system and you have access.

327
00:13:11,490 --> 00:13:13,860
And then finally there's phishing.

328
00:13:13,860 --> 00:13:15,390
Instead of targeting a business person

329
00:13:15,390 --> 00:13:16,890
you can send an awesome research paper

330
00:13:16,890 --> 00:13:19,080
with a real model that really works

331
00:13:19,080 --> 00:13:22,710
that's actually the
research to a data scientist

332
00:13:22,710 --> 00:13:24,840
and when they go and load it up

333
00:13:24,840 --> 00:13:26,940
you can get a show on
their system as well.

334
00:13:26,940 --> 00:13:29,910
And since research is done by
a bunch of different countries

335
00:13:29,910 --> 00:13:30,840
not always who are friends,

336
00:13:30,840 --> 00:13:35,043
but are often sharing data
that's a bit of a concern.

337
00:13:36,390 --> 00:13:37,830
And then after that you
just wait for a callback

338
00:13:37,830 --> 00:13:39,280
and do what you want with it.

339
00:13:44,490 --> 00:13:47,253
So, what can we do about this?

340
00:13:48,480 --> 00:13:50,640
Well, antivirus software
doesn't work great

341
00:13:50,640 --> 00:13:51,693
for detecting this.

342
00:13:53,070 --> 00:13:54,810
Pickles aren't like an unfile type.

343
00:13:54,810 --> 00:13:57,180
A coworker of mine wrote a YARA rule,

344
00:13:57,180 --> 00:13:59,700
but because it's just
a full instruction set,

345
00:13:59,700 --> 00:14:01,620
you can do anything you want with it.

346
00:14:01,620 --> 00:14:06,210
You can make the code do
anything you want, completely.

347
00:14:06,210 --> 00:14:07,380
You can load any library,

348
00:14:07,380 --> 00:14:11,460
so it's essentially detecting
an arbitrary Python file

349
00:14:11,460 --> 00:14:12,690
except written in a different form

350
00:14:12,690 --> 00:14:14,340
that's binary and harder to read.

351
00:14:15,240 --> 00:14:17,070
So, the YARA rules that we wrote

352
00:14:17,070 --> 00:14:20,430
are pretty easy to bypass if
you know what you're doing.

353
00:14:20,430 --> 00:14:21,930
Maybe there's a way to make 'em better,

354
00:14:21,930 --> 00:14:24,060
I mean, there's probably
ways to make 'em better

355
00:14:24,060 --> 00:14:25,560
but I still don't think
you're ever gonna be able

356
00:14:25,560 --> 00:14:26,910
to completely detect every way

357
00:14:26,910 --> 00:14:29,013
you can make a pickle malicious.

358
00:14:30,930 --> 00:14:32,610
So, all you can really do is verify

359
00:14:32,610 --> 00:14:34,110
that the integrity of the file's exactly

360
00:14:34,110 --> 00:14:35,670
as it was when you created it.

361
00:14:35,670 --> 00:14:38,610
But that doesn't help anyone
who's getting a file from you

362
00:14:38,610 --> 00:14:40,510
unless you have a way to share a hash,

363
00:14:42,360 --> 00:14:44,783
that's more secure than the
way you share the pickle,

364
00:14:45,780 --> 00:14:47,520
which maybe you do, and so you can do that

365
00:14:47,520 --> 00:14:50,760
but you sort of run into an issue of

366
00:14:50,760 --> 00:14:53,063
how does anyone know that
it hasn't been modified?

367
00:14:54,090 --> 00:14:56,613
Fickling has a feature for check safety,

368
00:14:57,840 --> 00:14:59,640
but as I mentioned there's
a lot of different ways

369
00:14:59,640 --> 00:15:01,140
you can make something malicious.

370
00:15:01,140 --> 00:15:03,210
And so the last line of their output is,

371
00:15:03,210 --> 00:15:06,570
a don't unpickle this file if
it's from an untrusted source,

372
00:15:06,570 --> 00:15:08,310
because it can still be malicious.

373
00:15:08,310 --> 00:15:10,860
The examples I make aren't
things they thought of

374
00:15:10,860 --> 00:15:12,693
so it doesn't detect them.

375
00:15:17,040 --> 00:15:18,540
So, what can you do if you have a pickle

376
00:15:18,540 --> 00:15:21,690
that comes from a source
you don't 100% trust?

377
00:15:21,690 --> 00:15:24,690
My recommendation is that
you do not load them,

378
00:15:24,690 --> 00:15:25,640
you just shouldn't.

379
00:15:26,850 --> 00:15:28,080
If you really want to,

380
00:15:28,080 --> 00:15:30,120
there's people on the
internet that have examples

381
00:15:30,120 --> 00:15:33,723
of how you can create a
list of approved functions,

382
00:15:35,190 --> 00:15:37,860
only those functions can
be loaded by the pickle.

383
00:15:37,860 --> 00:15:39,810
But at that point you're
creating a Python jail

384
00:15:39,810 --> 00:15:40,800
and there's a lot of CTFs

385
00:15:40,800 --> 00:15:42,750
where we've seen people
breaking out of Python jails,

386
00:15:42,750 --> 00:15:43,980
and a lot of issues that we've seen

387
00:15:43,980 --> 00:15:46,020
in Python jails over the years.

388
00:15:46,020 --> 00:15:48,450
So, I wouldn't recommend that.

389
00:15:48,450 --> 00:15:51,000
But you'll find that
option on the internet

390
00:15:51,000 --> 00:15:52,653
if you really want to try it.

391
00:15:54,660 --> 00:15:55,493
Yeah.

392
00:15:56,790 --> 00:15:59,310
So, what do I recommend?

393
00:15:59,310 --> 00:16:02,370
Well, if you're creating something new

394
00:16:02,370 --> 00:16:04,920
then you can release your layers as code.

395
00:16:04,920 --> 00:16:08,850
Pickles don't really hide
what you created in any way,

396
00:16:08,850 --> 00:16:09,780
in any meaningful way,

397
00:16:09,780 --> 00:16:12,450
it can all be extracted
by anyone who wants to.

398
00:16:12,450 --> 00:16:14,610
So you can release your layers,

399
00:16:14,610 --> 00:16:16,800
all the code that you write
as actual Python code,

400
00:16:16,800 --> 00:16:18,930
whoever is gonna iterate off of it

401
00:16:18,930 --> 00:16:20,340
will have an easier time doing it

402
00:16:20,340 --> 00:16:22,530
and they won't be vulnerable

403
00:16:22,530 --> 00:16:23,760
'cause they can look at the code

404
00:16:23,760 --> 00:16:26,220
and make sure it's not
doing something sketchy.

405
00:16:26,220 --> 00:16:28,410
And then you release your
weights as binary data.

406
00:16:28,410 --> 00:16:30,360
If you wanted to do this in Py Torch,

407
00:16:30,360 --> 00:16:32,070
on the right hand side of
the screen is some code,

408
00:16:32,070 --> 00:16:33,930
which I'm also releasing at the end

409
00:16:33,930 --> 00:16:36,840
and that code will for Py Torch

410
00:16:36,840 --> 00:16:38,160
save the state of the weights

411
00:16:38,160 --> 00:16:43,160
and ride it out to a NumPy
log for you and load it.

412
00:16:43,620 --> 00:16:44,940
And if you look closely,

413
00:16:44,940 --> 00:16:47,850
you see the np-lib NPYO,

414
00:16:47,850 --> 00:16:50,100
like the super deep function
inside the NP Library

415
00:16:50,100 --> 00:16:51,750
that you're not really supposed to call,

416
00:16:51,750 --> 00:16:54,720
that's because the default
one will fall back to pickles

417
00:16:54,720 --> 00:16:57,900
if you have something in
it that makes it want to.

418
00:16:57,900 --> 00:17:00,900
So, it's not really a good function to use

419
00:17:00,900 --> 00:17:02,673
but you can, it exists.

420
00:17:05,430 --> 00:17:07,560
And then if, you know, this
has been around for six years

421
00:17:07,560 --> 00:17:09,312
so there's a lo of
pickles that already exist

422
00:17:09,312 --> 00:17:11,940
that are malicious or well not malicious,

423
00:17:11,940 --> 00:17:14,130
that could be made to be malicious.

424
00:17:14,130 --> 00:17:17,910
And it's sort of unrealistic
for me to expect them

425
00:17:17,910 --> 00:17:20,520
to just delete all of them and start over,

426
00:17:20,520 --> 00:17:23,040
as much as that would
be the safest solution.

427
00:17:23,040 --> 00:17:24,660
So if you have some of those,

428
00:17:24,660 --> 00:17:26,670
then you need to protect 'em

429
00:17:26,670 --> 00:17:27,930
like they're unsigned executables

430
00:17:27,930 --> 00:17:30,240
because effectively that's what they are.

431
00:17:30,240 --> 00:17:31,830
That means verifying
that they're integrity

432
00:17:31,830 --> 00:17:33,933
is exactly what you it to be.

433
00:17:35,160 --> 00:17:37,980
If you're offering people
the ability to download it,

434
00:17:37,980 --> 00:17:39,990
needs to be over an encrypted channel

435
00:17:39,990 --> 00:17:42,900
because otherwise attacker in the middle

436
00:17:42,900 --> 00:17:44,520
can make it whatever they want it

437
00:17:44,520 --> 00:17:46,350
and get a show on their system.

438
00:17:46,350 --> 00:17:48,060
And if an adversary gets access

439
00:17:48,060 --> 00:17:50,430
and could have access, modified it

440
00:17:50,430 --> 00:17:52,733
and you have no way to
verify that they haven't,

441
00:17:53,640 --> 00:17:54,990
you sort of just have to delete it all

442
00:17:54,990 --> 00:17:58,260
and recreate them which sort of sucks,

443
00:17:58,260 --> 00:17:59,613
so don't let that happen.

444
00:18:05,130 --> 00:18:07,410
And some of you that know
about machine learning models

445
00:18:07,410 --> 00:18:09,030
might be thinking like,
okay, I don't use pickles,

446
00:18:09,030 --> 00:18:11,850
I use ONNX or TorchScript
or something else

447
00:18:11,850 --> 00:18:14,130
that still supports arbitrary layers,

448
00:18:14,130 --> 00:18:17,943
and arbitrary layers are gonna
be in some way still code.

449
00:18:18,960 --> 00:18:21,150
So most likely, they're
also still vulnerable

450
00:18:21,150 --> 00:18:23,310
just is gonna take more work to do it

451
00:18:23,310 --> 00:18:25,260
because there's not 10 years of research

452
00:18:25,260 --> 00:18:27,810
showing you how to just
add malicious code into it.

453
00:18:28,770 --> 00:18:32,340
For example, ONNX already has
a POC, thanks to (indistinct).

454
00:18:32,340 --> 00:18:34,230
I'm not sure if you can
read the link at the bottom,

455
00:18:34,230 --> 00:18:36,330
that's his GitHub where
he has a proof of concept

456
00:18:36,330 --> 00:18:40,923
on how to add exploitable
code into an ONNX file.

457
00:18:42,690 --> 00:18:45,600
If you're someone that wants
to do this kind of research,

458
00:18:45,600 --> 00:18:48,030
ONNX and all the other ways to save models

459
00:18:48,030 --> 00:18:50,190
could be something interesting to look at

460
00:18:50,190 --> 00:18:52,290
to add malicious code into as well

461
00:18:52,290 --> 00:18:54,450
'cause I'm pretty sure it's possible

462
00:18:54,450 --> 00:18:56,313
even though I haven't tried myself.

463
00:19:00,030 --> 00:19:01,170
So yeah.

464
00:19:01,170 --> 00:19:03,600
If anyone wants to see the injection code

465
00:19:03,600 --> 00:19:04,980
I have it on my GitHub.

466
00:19:04,980 --> 00:19:07,770
If anyone uses Mythic C2, I made a wrapper

467
00:19:07,770 --> 00:19:09,840
so all you have to do is upload

468
00:19:09,840 --> 00:19:11,010
the pickle you want to inject into,

469
00:19:11,010 --> 00:19:13,710
it'll inject a Mythic Medusa agent into it

470
00:19:13,710 --> 00:19:15,600
and you can put that where you want

471
00:19:15,600 --> 00:19:17,820
and wait for a callback.

472
00:19:17,820 --> 00:19:20,280
And yeah, if you have any questions,

473
00:19:20,280 --> 00:19:23,400
you can ask but also hit me up on Twitter

474
00:19:23,400 --> 00:19:24,950
and I'll do my best to respond.

475
00:19:26,965 --> 00:19:30,132
(audience applauding)