1
00:00:00,450 --> 00:00:03,019
- DEF CON 30. Who's having a good time?

2
00:00:03,019 --> 00:00:04,110
- [Attendees] Yeah.
(attendees clapping hands)

3
00:00:04,110 --> 00:00:05,280
- Whoa. You guys.

4
00:00:05,280 --> 00:00:06,690
Awesome. Awesome. That's great.

5
00:00:06,690 --> 00:00:09,120
So, this is Rex and Junyuan.

6
00:00:09,120 --> 00:00:11,070
So, let's give a big DEF CON 30 welcome.

7
00:00:11,070 --> 00:00:12,180
Come one.
(attendees clapping hands)

8
00:00:12,180 --> 00:00:16,123
Yeah.
(attendees clapping hands)

9
00:00:19,720 --> 00:00:20,553
- Okay.

10
00:00:20,553 --> 00:00:22,710
Thank you, everyone,
for coming to our talk

11
00:00:22,710 --> 00:00:25,260
about a bypassing system called Tracing.

12
00:00:25,260 --> 00:00:26,250
My name is Rex Guo.

13
00:00:26,250 --> 00:00:28,250
And this is my co-speaker, Junyuan Zeng.

14
00:00:29,190 --> 00:00:32,430
So, we have a lot to cover in this talk.

15
00:00:32,430 --> 00:00:34,230
So, I will just jump over this slide,

16
00:00:34,230 --> 00:00:35,703
you can look up us online.

17
00:00:36,930 --> 00:00:40,950
Okay, so imagine a sophisticated attacker

18
00:00:40,950 --> 00:00:44,310
compromised your Linux
production environment.

19
00:00:44,310 --> 00:00:46,650
He launches a Log4Shell exploit

20
00:00:46,650 --> 00:00:49,233
and then a file reverse
shell back to his machine.

21
00:00:51,420 --> 00:00:53,250
Then he discovered that machine is running

22
00:00:53,250 --> 00:00:54,660
a vulnerable version of sudo,

23
00:00:54,660 --> 00:00:57,153
so he elevated privilege on the box.

24
00:00:58,860 --> 00:01:01,050
And then he read the /etc/shadow file

25
00:01:01,050 --> 00:01:03,723
to see if there's any
interesting hashes or cracks.

26
00:01:05,280 --> 00:01:08,250
He also discovered that
he can do ssh hijacking

27
00:01:08,250 --> 00:01:10,230
to move into a second
machine by reading into

28
00:01:10,230 --> 00:01:12,720
the process environment variable.

29
00:01:12,720 --> 00:01:15,090
So, he moved to the second machine.

30
00:01:15,090 --> 00:01:17,490
And as he is celebrating,

31
00:01:17,490 --> 00:01:20,310
he discovered that his access was gone.

32
00:01:20,310 --> 00:01:23,553
So, he fired the RC
again, there's no luck.

33
00:01:25,020 --> 00:01:27,770
Quickly he discovered his
access was completed blocked.

34
00:01:29,850 --> 00:01:32,580
Now, let's take a look at
the other side of the story.

35
00:01:32,580 --> 00:01:34,710
While everything is happening,

36
00:01:34,710 --> 00:01:38,227
your security engineer
received a sequence of alert

37
00:01:38,227 --> 00:01:41,640
from the cutting edge
security monitoring software.

38
00:01:41,640 --> 00:01:44,670
And this software is able
to monitor the system calls,

39
00:01:44,670 --> 00:01:48,090
and also process information
for the application.

40
00:01:48,090 --> 00:01:48,923
So, for example,

41
00:01:48,923 --> 00:01:51,000
when the attacker
executed a reverse shell,

42
00:01:51,000 --> 00:01:52,350
there will be a connect system call,

43
00:01:52,350 --> 00:01:53,430
and maybe others,

44
00:01:53,430 --> 00:01:56,070
depending on the reverse shell technique.

45
00:01:56,070 --> 00:01:59,040
When the attacker read
the /etc/shadow file,

46
00:01:59,040 --> 00:02:00,963
there will be an open at system call.

47
00:02:01,920 --> 00:02:04,920
Essentially any non-trivial action

48
00:02:04,920 --> 00:02:06,600
that the application perform

49
00:02:06,600 --> 00:02:08,350
will need to involve a system call.

50
00:02:09,900 --> 00:02:12,510
So, how do you use the
system call information

51
00:02:12,510 --> 00:02:13,683
to detect threats?

52
00:02:15,570 --> 00:02:18,570
So, here we show a really,
really simple example.

53
00:02:18,570 --> 00:02:20,610
Let me explain what this is.

54
00:02:20,610 --> 00:02:23,700
Here's a rule that try to
detect untrusted program

55
00:02:23,700 --> 00:02:25,173
reading the /etc/shadow file.

56
00:02:26,220 --> 00:02:28,350
And let me explain the rule.

57
00:02:28,350 --> 00:02:29,183
It's saying,

58
00:02:29,183 --> 00:02:32,400
"If the system call is open or open at,

59
00:02:32,400 --> 00:02:34,650
and has a read permission,

60
00:02:34,650 --> 00:02:38,430
and the file name points
to /etc/shadow file,

61
00:02:38,430 --> 00:02:42,270
and the program is not in
the allowlist of programs,

62
00:02:42,270 --> 00:02:44,343
that will fire an alert.

63
00:02:45,690 --> 00:02:46,770
Now, you can probably see

64
00:02:46,770 --> 00:02:51,770
that you can build much
more complex rules,

65
00:02:52,110 --> 00:02:53,580
and even machine learning models,

66
00:02:53,580 --> 00:02:56,160
on top of this dataset
with one system call

67
00:02:56,160 --> 00:02:57,693
or even multiple system calls.

68
00:02:58,680 --> 00:03:00,360
But they all rely on the fact

69
00:03:00,360 --> 00:03:02,850
that the system call monitoring software

70
00:03:02,850 --> 00:03:04,953
is able to extract the data correctly.

71
00:03:07,230 --> 00:03:08,520
So, in this talk,

72
00:03:08,520 --> 00:03:10,200
we are going to dive very deep

73
00:03:10,200 --> 00:03:12,960
into these system call
tracing technologies.

74
00:03:12,960 --> 00:03:14,580
And then we'll talk
about the vulnerabilities

75
00:03:14,580 --> 00:03:18,780
that we discover that allow
us to bypass the tracing.

76
00:03:18,780 --> 00:03:22,230
And then we'll conclude with
mitigations and takeaways.

77
00:03:22,230 --> 00:03:24,190
With that, I will hand over to Junyuan.

78
00:03:26,520 --> 00:03:27,353
- Thank you, Rex.

79
00:03:27,353 --> 00:03:28,433
So, as Rex mentioned,

80
00:03:28,433 --> 00:03:32,070
syscall tracing is very
important to detect any threats.

81
00:03:32,070 --> 00:03:34,110
So, this diagram can give you an overview

82
00:03:34,110 --> 00:03:35,820
of syscall tracing.

83
00:03:35,820 --> 00:03:38,310
Basically, the syscall
tracing include two parts.

84
00:03:38,310 --> 00:03:40,890
One is the hooks for syscall interception,

85
00:03:40,890 --> 00:03:42,453
and also the tracing programs.

86
00:03:43,530 --> 00:03:44,363
So, when the application

87
00:03:44,363 --> 00:03:46,197
issues a system call into the kernel,

88
00:03:46,197 --> 00:03:48,540
the system call code path is executed.

89
00:03:48,540 --> 00:03:51,870
If there are any hooks
placed inside the code path,

90
00:03:51,870 --> 00:03:53,970
the attached tracing
program will be triggered

91
00:03:53,970 --> 00:03:55,740
to collect system call data

92
00:03:55,740 --> 00:03:59,100
and send those information
to the monitor agent

93
00:03:59,100 --> 00:04:00,123
to detect threats.

94
00:04:00,990 --> 00:04:03,149
So, the tracing program can be implemented

95
00:04:03,149 --> 00:04:06,090
in the kernel space, as
shown in the left diagram,

96
00:04:06,090 --> 00:04:08,700
or implemented as a user space program

97
00:04:08,700 --> 00:04:10,260
as part of the monitor agent,

98
00:04:10,260 --> 00:04:11,860
which is show on the right side.

99
00:04:14,040 --> 00:04:16,350
So, the program used to
collect syscall data,

100
00:04:16,350 --> 00:04:17,910
like syscall arguments,

101
00:04:17,910 --> 00:04:19,950
is called a tracing program.

102
00:04:19,950 --> 00:04:21,960
This program can be
attached to different hooks,

103
00:04:21,960 --> 00:04:24,153
like tracepoint, kprobe, ptrace.

104
00:04:25,590 --> 00:04:27,570
So, we can directly leverage

105
00:04:27,570 --> 00:04:30,930
the Linux native mechanisms
as tracing programs,

106
00:04:30,930 --> 00:04:32,790
or implement your own tracing programs,

107
00:04:32,790 --> 00:04:36,303
as kernel modules, eBPF
probes or user space programs.

108
00:04:38,760 --> 00:04:41,250
So, the first kind of hooks
for a syscall interception

109
00:04:41,250 --> 00:04:42,690
is called tracepoint,

110
00:04:42,690 --> 00:04:44,980
basically it's the kernel static hook

111
00:04:46,050 --> 00:04:47,280
for a syscall interception.

112
00:04:47,280 --> 00:04:49,029
The Linux kernel provides

113
00:04:49,029 --> 00:04:51,150
sys_enter and sys_exit tracepoint.

114
00:04:51,150 --> 00:04:53,880
So, if we attach the tracing
program to the tracepoint,

115
00:04:53,880 --> 00:04:56,820
the function call of trace
sys_enter and trace sys_exit

116
00:04:56,820 --> 00:05:00,090
will use the same parameters
to trigger the tracing program.

117
00:05:00,090 --> 00:05:02,400
The first parameter is called regs,

118
00:05:02,400 --> 00:05:04,830
that's saving the syscall arguments.

119
00:05:04,830 --> 00:05:07,140
And the second is called ID,

120
00:05:07,140 --> 00:05:08,733
which saves the syscall number.

121
00:05:09,930 --> 00:05:11,670
Tracepoint provides a low overhead,

122
00:05:11,670 --> 00:05:14,370
but it only provides static
system call interceptions.

123
00:05:16,320 --> 00:05:18,390
You can also use the dynamic approach,

124
00:05:18,390 --> 00:05:19,223
like kprobe,

125
00:05:19,223 --> 00:05:21,150
to intercept system calls.

126
00:05:21,150 --> 00:05:22,050
So, using kprobe,

127
00:05:22,050 --> 00:05:23,910
you can register the tracing program

128
00:05:23,910 --> 00:05:27,090
on almost any instructions
in the kernel code path.

129
00:05:27,090 --> 00:05:29,190
Like, basically, it's a syscall code path.

130
00:05:30,150 --> 00:05:31,917
So, when the instruction's executed,

131
00:05:31,917 --> 00:05:34,053
the tracing program will be caught.

132
00:05:34,980 --> 00:05:36,593
Compared to tracepoint,

133
00:05:36,593 --> 00:05:38,433
kprobe is a dynamic approach,

134
00:05:39,300 --> 00:05:41,460
but you need to know exactly

135
00:05:41,460 --> 00:05:44,130
how data is placed on
the stack or register

136
00:05:44,130 --> 00:05:46,053
in order to get useful information.

137
00:05:48,390 --> 00:05:50,427
So ptrace provides a user space solution

138
00:05:50,427 --> 00:05:52,890
for syscall interception.

139
00:05:52,890 --> 00:05:54,003
Similar to the tracepoint,

140
00:05:54,003 --> 00:05:57,183
it's a static hook in the
syscall enter and exit.

141
00:05:58,710 --> 00:06:00,390
Using ptrace, you don't need to implement

142
00:06:00,390 --> 00:06:03,480
any kernel programs as a tracing program,

143
00:06:03,480 --> 00:06:06,213
but only user space program needed.

144
00:06:07,260 --> 00:06:09,720
Compared to the previous tool's approach,

145
00:06:09,720 --> 00:06:11,733
ptrace's performance overhead is high.

146
00:06:12,870 --> 00:06:14,310
But for optimization,

147
00:06:14,310 --> 00:06:16,050
you can combine it with the seccomp

148
00:06:16,050 --> 00:06:18,303
as syscall filtering
for better performance.

149
00:06:19,890 --> 00:06:22,050
There other approaches
for syscall interception,

150
00:06:22,050 --> 00:06:23,520
like LD_PRELOAD.

151
00:06:23,520 --> 00:06:26,010
This approach actually is easy to bypass

152
00:06:26,010 --> 00:06:29,253
if we use a seccomp call
to trigger the system call.

153
00:06:31,590 --> 00:06:33,000
So, many of you have probably heard

154
00:06:33,000 --> 00:06:35,670
about the cloud workload
protection products.

155
00:06:35,670 --> 00:06:37,170
This product usually provides

156
00:06:37,170 --> 00:06:39,753
advanced threat detection
based on syscall tracing.

157
00:06:41,206 --> 00:06:43,680
There are different
kinds of cloud workloads,

158
00:06:43,680 --> 00:06:45,900
such as virtual machines,

159
00:06:45,900 --> 00:06:48,270
containers on customer-managed VMs,

160
00:06:48,270 --> 00:06:51,510
serverless containers, and others.

161
00:06:51,510 --> 00:06:53,160
So, for serverless containers,

162
00:06:53,160 --> 00:06:55,500
it's usually allocated and maintained

163
00:06:55,500 --> 00:06:57,900
by the cloud providers on demand.

164
00:06:57,900 --> 00:07:00,423
So, they usually have
no access to the host.

165
00:07:03,120 --> 00:07:04,170
This table summarizes

166
00:07:04,170 --> 00:07:05,850
how different syscall tracing techniques

167
00:07:05,850 --> 00:07:08,880
can be applied for
different cloud workloads.

168
00:07:08,880 --> 00:07:09,713
For virtual machine,

169
00:07:09,713 --> 00:07:12,633
we can use any kinds of
hooks and tracing programs.

170
00:07:13,530 --> 00:07:17,220
Tools we can leverage include
Falco eBPF/kernel Module

171
00:07:17,220 --> 00:07:18,810
and Falco pdig

172
00:07:18,810 --> 00:07:21,110
We'll talk about this
tools later in our talk.

173
00:07:22,200 --> 00:07:25,740
For containers that are
on customer-managed VM,

174
00:07:25,740 --> 00:07:28,740
we can have the same
options as virtual machines

175
00:07:28,740 --> 00:07:30,933
as long as we get enough capabilities.

176
00:07:32,490 --> 00:07:35,180
For serverless containers,
as we mentioned,

177
00:07:35,180 --> 00:07:37,470
it has no access to the host.

178
00:07:37,470 --> 00:07:40,770
So, we can only use
ptrace as hooking points

179
00:07:40,770 --> 00:07:44,370
and implement a tracing
program in user space.

180
00:07:44,370 --> 00:07:47,910
Instead of Falco kernel
Module and eBPF programs,

181
00:07:47,910 --> 00:07:49,533
we can only use the Falco pdig.

182
00:07:52,440 --> 00:07:55,350
Falco use similar techniques
to a trace system call.

183
00:07:55,350 --> 00:07:57,573
It's an open source project in CNCF.

184
00:07:58,860 --> 00:07:59,850
It's very popular.

185
00:07:59,850 --> 00:08:01,290
And in kernel space,

186
00:08:01,290 --> 00:08:04,110
it supports kernel Module and eBPF probe

187
00:08:04,110 --> 00:08:05,103
using tracepoint.

188
00:08:05,940 --> 00:08:09,453
In user space, pdig is
developed based on ptrace.

189
00:08:10,890 --> 00:08:15,630
So, Falco pdig is dedicated
for system call tracing

190
00:08:15,630 --> 00:08:17,223
of service workloads.

191
00:08:19,530 --> 00:08:23,160
So, we did not evaluate other
security monitor agents,

192
00:08:23,160 --> 00:08:26,130
but we believe the popularity of Falco

193
00:08:26,130 --> 00:08:27,750
represent an implementation

194
00:08:27,750 --> 00:08:30,450
that is widely accepted by the community.

195
00:08:30,450 --> 00:08:33,660
Unfortunately, this kind of implementation

196
00:08:33,660 --> 00:08:36,900
for syscall tracing is
vulnerable to a TOCTOU issue.

197
00:08:36,900 --> 00:08:39,450
That is, Time_Of_Check, Time_Of_Use.

198
00:08:39,450 --> 00:08:41,790
Let's take the connect
system call, for example.

199
00:08:41,790 --> 00:08:43,040
The second argument
for connect system call

200
00:08:43,040 --> 00:08:45,900
is called user V address.

201
00:08:45,900 --> 00:08:49,470
That is a user pointer pointing
to the user space buffer

202
00:08:49,470 --> 00:08:51,330
called socket address.

203
00:08:51,330 --> 00:08:53,067
So, during Time_Of_Check,
the tracing program

204
00:08:53,067 --> 00:08:56,040
dereferences this user space pointer

205
00:08:56,040 --> 00:08:57,960
to get the socket address.

206
00:08:57,960 --> 00:08:59,400
And during Time_Of_Use,

207
00:08:59,400 --> 00:09:03,420
the kernel also dereferences
the same user space pointer

208
00:09:03,420 --> 00:09:05,640
to also get the socket address.

209
00:09:05,640 --> 00:09:08,610
But between Time_Of_Check and Time_Of_Use,

210
00:09:08,610 --> 00:09:12,330
the user memory pointed
by the user V address

211
00:09:12,330 --> 00:09:14,820
is vulnerable to be
changed by the attacker

212
00:09:14,820 --> 00:09:16,320
from the user space.

213
00:09:16,320 --> 00:09:17,730
So, in this case,

214
00:09:17,730 --> 00:09:19,560
the socket address can be different

215
00:09:19,560 --> 00:09:22,020
between Time_Of_Check and Time_Of_Use,

216
00:09:22,020 --> 00:09:23,283
causing a TOCTOU issue.

217
00:09:25,200 --> 00:09:27,690
So, let's dive into the
connect system call,

218
00:09:27,690 --> 00:09:30,600
which can help you
understand this TOCTOU issue.

219
00:09:30,600 --> 00:09:31,860
So, when the application issued

220
00:09:31,860 --> 00:09:33,843
a connect system call into the kernel,

221
00:09:34,830 --> 00:09:36,750
the system call handler will check

222
00:09:36,750 --> 00:09:39,300
if any tracing programs are attached

223
00:09:39,300 --> 00:09:42,420
to the static hook at
the system call enter,

224
00:09:42,420 --> 00:09:44,410
like ptrace, seccomp,
sys_enter and tracepoint.

225
00:09:46,050 --> 00:09:47,280
If this is true,

226
00:09:47,280 --> 00:09:49,030
the tracing program will be caught.

227
00:09:50,970 --> 00:09:52,200
And after that,

228
00:09:52,200 --> 00:09:54,510
the handler will look at the syscall table

229
00:09:54,510 --> 00:09:56,670
and jump to connect system call

230
00:09:56,670 --> 00:09:58,520
to create a connection on the socket.

231
00:10:00,510 --> 00:10:03,030
Before returning to the user space,

232
00:10:03,030 --> 00:10:05,700
the handler will again
check if any tracing program

233
00:10:05,700 --> 00:10:09,360
attached to the static
hook at syscall exit,

234
00:10:09,360 --> 00:10:12,240
like ptrace sys-exit tracepoint.

235
00:10:12,240 --> 00:10:13,860
Similarly, if this is true,

236
00:10:13,860 --> 00:10:15,633
the tracing program will be caught.

237
00:10:17,400 --> 00:10:19,140
So, as I mentioned earlier,

238
00:10:19,140 --> 00:10:21,210
the second argument
for connect system call

239
00:10:21,210 --> 00:10:24,690
is user pointer pointing
to the user space,

240
00:10:24,690 --> 00:10:27,510
or socket address in the user space.

241
00:10:27,510 --> 00:10:29,580
This pointer is propagated

242
00:10:29,580 --> 00:10:31,560
into the connect system code path

243
00:10:31,560 --> 00:10:34,020
and assigned to different
kernel variables,

244
00:10:34,020 --> 00:10:35,420
which is highlighted in red.

245
00:10:37,677 --> 00:10:39,900
The kernel calls
move_address_to_kernel function

246
00:10:39,900 --> 00:10:43,290
to make a copy of socket
address from user space

247
00:10:43,290 --> 00:10:46,080
to the kernel buffer call address,

248
00:10:46,080 --> 00:10:47,530
that is highlighted in green.

249
00:10:49,710 --> 00:10:52,020
After that, the kernel will
call the internal function,

250
00:10:52,020 --> 00:10:53,460
sys_connect_file function,

251
00:10:53,460 --> 00:10:56,530
to create the connection on the socket

252
00:10:57,420 --> 00:11:00,090
based on the kernel buffer address.

253
00:11:00,090 --> 00:11:02,400
And this is the Time_Of_use
by the Linux kernel

254
00:11:02,400 --> 00:11:04,773
for connect system call arguments.

255
00:11:07,140 --> 00:11:09,810
So, before the memory copy function,

256
00:11:09,810 --> 00:11:11,970
the kernel buffer is not created.

257
00:11:11,970 --> 00:11:13,980
So, the user pointer is the only place

258
00:11:13,980 --> 00:11:16,890
we can get reference and
also get the socket address

259
00:11:16,890 --> 00:11:18,213
from the user space.

260
00:11:20,010 --> 00:11:21,720
In this case, during Time_Of_Check,

261
00:11:21,720 --> 00:11:23,400
if we attach the tracing program

262
00:11:23,400 --> 00:11:26,640
to the static hook at system call enter,

263
00:11:26,640 --> 00:11:30,150
or to any places before
the memory copy function

264
00:11:30,150 --> 00:11:32,253
using a dynamic approach like kprobe,

265
00:11:33,107 --> 00:11:35,280
the tracing programs have no other option,

266
00:11:35,280 --> 00:11:37,830
but need to dereference the user pointer

267
00:11:37,830 --> 00:11:39,273
to get the socket address.

268
00:11:41,430 --> 00:11:43,677
After the memory copy function,

269
00:11:43,677 --> 00:11:45,660
the kernel buffer is created

270
00:11:45,660 --> 00:11:47,880
with copies of the socket address

271
00:11:47,880 --> 00:11:48,903
from user the space.

272
00:11:50,250 --> 00:11:52,470
However, in this case,

273
00:11:52,470 --> 00:11:54,690
this TOCTOU issue may still exist.

274
00:11:54,690 --> 00:11:57,720
So, think about if we
attach a tracing program

275
00:11:57,720 --> 00:12:00,330
to the static hook system called exit,

276
00:12:00,330 --> 00:12:02,673
like sys_exit tracepoint or ptrace.

277
00:12:03,913 --> 00:12:06,210
The tracing program may still dereference

278
00:12:06,210 --> 00:12:09,003
the user pointer to
get the socket address.

279
00:12:12,240 --> 00:12:16,620
Falco pdig use ptrace for
syscall enter and exit,

280
00:12:16,620 --> 00:12:19,470
but it only use the
seccomp syscall filtering

281
00:12:19,470 --> 00:12:20,430
as system call enter,

282
00:12:20,430 --> 00:12:23,433
since the seccomp is not
available for syscall exit.

283
00:12:24,510 --> 00:12:27,837
For Falco version 0.31.1,

284
00:12:27,837 --> 00:12:30,720
the kernel module and eBPF implementation

285
00:12:30,720 --> 00:12:33,200
only use the sys_exit tracepoint.

286
00:12:34,980 --> 00:12:37,593
So, hopefully you get an
idea about the TOCTOU issue

287
00:12:37,593 --> 00:12:38,761
for syscall tracing.

288
00:12:38,761 --> 00:12:40,140
Next, I'll hand over to Rex,

289
00:12:40,140 --> 00:12:41,740
to talk about the vulnerability.

290
00:12:46,129 --> 00:12:46,962
(Rex clears throat)

291
00:12:46,962 --> 00:12:48,720
- Okay, so the example that we see

292
00:12:48,720 --> 00:12:51,210
is on kernel version 5.7,

293
00:12:51,210 --> 00:12:52,500
and on connect system call.

294
00:12:52,500 --> 00:12:55,860
But really, the observation is true

295
00:12:55,860 --> 00:12:57,603
across all the kernel versions.

296
00:12:58,770 --> 00:13:01,290
Because we confirmed with
the kernel developers

297
00:13:01,290 --> 00:13:05,694
these TOCTOU windows exist
since the day that tracepoint

298
00:13:05,694 --> 00:13:06,527
and ptrace were introduced,

299
00:13:06,527 --> 00:13:08,640
because they were originally designed

300
00:13:08,640 --> 00:13:12,060
for performance and debugging purposes.

301
00:13:12,060 --> 00:13:15,390
So, in order to do secure tracing,

302
00:13:15,390 --> 00:13:18,690
what they recommend is the
software needs to monitor

303
00:13:18,690 --> 00:13:21,183
the kernel memory instead.

304
00:13:23,190 --> 00:13:25,920
So, report this issue to Falco.

305
00:13:25,920 --> 00:13:28,827
Basically, the issue is they
used sys_exit tracepoint,

306
00:13:28,827 --> 00:13:30,797
and they also used sys_exit ptrace.

307
00:13:32,381 --> 00:13:37,320
And there's a, you know,
version older than 0.31.1,

308
00:13:37,320 --> 00:13:38,153
it's impacted.

309
00:13:38,153 --> 00:13:40,590
And if you happen to use
the commercial version,

310
00:13:40,590 --> 00:13:42,940
you may want to check
what version is impacted.

311
00:13:44,220 --> 00:13:46,020
So, we reported this issue in December,

312
00:13:46,020 --> 00:13:48,873
and the issues is mitigated in March.

313
00:13:50,040 --> 00:13:52,350
The mitigation that was deployed

314
00:13:52,350 --> 00:13:57,350
is for the Linux security
module or the eBPF version.

315
00:13:58,260 --> 00:14:02,880
They check sys_enter and
sys_exit system call arguments

316
00:14:02,880 --> 00:14:05,520
for a selected of system calls.

317
00:14:05,520 --> 00:14:07,803
And they also do the same for Falco pdig.

318
00:14:11,610 --> 00:14:14,790
In terms of how many
system calls are impacted.

319
00:14:14,790 --> 00:14:18,990
So, we analyzed the open
source rules in the Falco repo,

320
00:14:18,990 --> 00:14:23,370
and majority of them are
impacted, with two exceptions.

321
00:14:23,370 --> 00:14:25,470
One is the execve system call.

322
00:14:25,470 --> 00:14:28,050
The reason for that is
in their implementation,

323
00:14:28,050 --> 00:14:29,340
when they trace the execve,

324
00:14:29,340 --> 00:14:31,590
they actually read the
kernel memory instead.

325
00:14:32,850 --> 00:14:34,890
The other one is the
send to and send message

326
00:14:34,890 --> 00:14:36,450
system call.

327
00:14:36,450 --> 00:14:38,580
We haven't found a reliable way

328
00:14:38,580 --> 00:14:40,740
to really block the system call.

329
00:14:40,740 --> 00:14:42,397
We'll talk about what blocking means

330
00:14:42,397 --> 00:14:44,850
in the next few slides.

331
00:14:44,850 --> 00:14:47,130
But keep in mind that
it's very heavy to monitor

332
00:14:47,130 --> 00:14:48,660
send to and send message,

333
00:14:48,660 --> 00:14:51,123
and this typically limits its usage.

334
00:14:54,150 --> 00:14:56,490
Okay, so hopefully everybody understands

335
00:14:56,490 --> 00:14:58,110
the vulnerability at this point.

336
00:14:58,110 --> 00:15:01,020
It's fairly, relatively simple.

337
00:15:01,020 --> 00:15:03,900
But let's talk about how to exploit it.

338
00:15:03,900 --> 00:15:05,370
Now, in this...

339
00:15:05,370 --> 00:15:08,400
To exploit this tracing,

340
00:15:08,400 --> 00:15:10,440
we don't want to acquire
any additional privilege

341
00:15:10,440 --> 00:15:11,820
or capabilities, right.

342
00:15:11,820 --> 00:15:13,080
Basically, the attacker should be able to

343
00:15:13,080 --> 00:15:16,683
evade detection, any user,
any privilege or capabilities.

344
00:15:17,850 --> 00:15:20,070
We want to have some level of control

345
00:15:20,070 --> 00:15:21,840
on the time to inject the delay.

346
00:15:21,840 --> 00:15:25,020
And we'll talk more about
what I mean by that later.

347
00:15:25,020 --> 00:15:26,970
We also need to inject enough delay,

348
00:15:26,970 --> 00:15:29,790
such that when we override the memory,

349
00:15:29,790 --> 00:15:33,360
the machine has enough time to propagate

350
00:15:33,360 --> 00:15:35,673
the overwritten data to the whole machine.

351
00:15:37,320 --> 00:15:38,490
The last thing is,

352
00:15:38,490 --> 00:15:41,313
we want this exploit
to be a 100% reliable,

353
00:15:42,600 --> 00:15:44,940
because if the attacker
being detected once,

354
00:15:44,940 --> 00:15:47,793
then the entire operation may be at risk.

355
00:15:50,640 --> 00:15:54,780
So, this leads us to two
exploitation strategies.

356
00:15:54,780 --> 00:15:56,640
We'll just quickly briefly overview

357
00:15:56,640 --> 00:15:59,010
the exploit strategy number one,

358
00:15:59,010 --> 00:16:01,680
which is what we did in DEF CON last year.

359
00:16:01,680 --> 00:16:03,120
And from there,

360
00:16:03,120 --> 00:16:06,603
you will see how we reached
exploit strategy number two.

361
00:16:07,740 --> 00:16:09,300
So, last year we discovered

362
00:16:09,300 --> 00:16:12,690
that you can actually inject
a relatively small amount

363
00:16:12,690 --> 00:16:16,200
of delay by using cross core interrupt.

364
00:16:16,200 --> 00:16:17,940
And because the amount of delay we inject

365
00:16:17,940 --> 00:16:19,380
is relatively small,

366
00:16:19,380 --> 00:16:22,620
we have to precisely
orchestrate the whole process.

367
00:16:22,620 --> 00:16:25,830
What I mean by that is have
to inject at the precise time,

368
00:16:25,830 --> 00:16:27,450
precisely override it,

369
00:16:27,450 --> 00:16:30,003
and also synchronize using
some special techniques.

370
00:16:31,350 --> 00:16:33,990
One of the parameters that
we use in the whole exploit

371
00:16:33,990 --> 00:16:35,553
is Userfaultfd system call.

372
00:16:36,630 --> 00:16:39,510
Now, although this technique is powerful,

373
00:16:39,510 --> 00:16:41,460
it has some limitations.

374
00:16:41,460 --> 00:16:42,750
So, for example,

375
00:16:42,750 --> 00:16:45,240
if you use docker container,

376
00:16:45,240 --> 00:16:49,290
and you enable the docker
default seccomp profile,

377
00:16:49,290 --> 00:16:51,440
docker will actually
block the system call.

378
00:16:52,932 --> 00:16:54,663
The other thing is,

379
00:16:55,950 --> 00:16:58,710
most cloud workloads
don't use the system call.

380
00:16:58,710 --> 00:17:01,590
So, the usage of the system call itself

381
00:17:01,590 --> 00:17:03,990
indicate some kind of anomaly.

382
00:17:03,990 --> 00:17:05,760
And this is actually the mitigation

383
00:17:05,760 --> 00:17:07,140
deployed by Falco last year.

384
00:17:07,140 --> 00:17:08,550
They implemented a detection rule

385
00:17:08,550 --> 00:17:10,593
to detect the usage of the system call.

386
00:17:12,150 --> 00:17:14,610
Right, so last year we
went back, and we think,

387
00:17:14,610 --> 00:17:17,760
you know, "Can we
actually develop something

388
00:17:17,760 --> 00:17:20,067
that doesn't rely on
using the system call?"

389
00:17:22,200 --> 00:17:24,033
So, we come up with the scenario,

390
00:17:26,467 --> 00:17:28,980
"What if you can actually inject a delay

391
00:17:28,980 --> 00:17:30,813
that is really, really long,

392
00:17:31,920 --> 00:17:33,870
then you don't have to
orchestrate everything

393
00:17:33,870 --> 00:17:36,420
in the precise timing," right?

394
00:17:36,420 --> 00:17:37,800
It sounds very simple,

395
00:17:37,800 --> 00:17:39,600
but the question is, how to do this?

396
00:17:43,140 --> 00:17:45,390
And we actually find out
there are two ways to do this.

397
00:17:45,390 --> 00:17:50,130
One is to use the blocking
condition in system call.

398
00:17:50,130 --> 00:17:52,710
This can be used to attack sys_exit.

399
00:17:52,710 --> 00:17:56,580
The other one is to insert
additional seccomp rules.

400
00:17:56,580 --> 00:17:58,430
This can be used to attack sys_enter.

401
00:17:59,850 --> 00:18:03,390
So, what do we mean by
blocking the system call?

402
00:18:03,390 --> 00:18:06,900
If you think about the underlying
mechanism of system call,

403
00:18:06,900 --> 00:18:09,840
it's basically the kernel
interacting with resources

404
00:18:09,840 --> 00:18:11,703
on behalf of the user space program.

405
00:18:12,720 --> 00:18:17,220
Now, a lot of these
resources are IOL devices,

406
00:18:17,220 --> 00:18:20,100
and the kernel will need
to wait for the response

407
00:18:20,100 --> 00:18:23,733
from the device before
you need to proceed.

408
00:18:25,350 --> 00:18:27,200
So, let's look at a concrete example.

409
00:18:28,320 --> 00:18:30,150
This is the connect system call.

410
00:18:30,150 --> 00:18:32,310
And in this diagram,
there are two machines.

411
00:18:32,310 --> 00:18:34,560
There's a client, there's a server.

412
00:18:34,560 --> 00:18:36,180
Now, let's say the client is monitored

413
00:18:36,180 --> 00:18:38,190
by tracing software.

414
00:18:38,190 --> 00:18:39,480
The client wants to talk to the server,

415
00:18:39,480 --> 00:18:42,210
so he issue a connect system call.

416
00:18:42,210 --> 00:18:44,700
When the kernel start executing,

417
00:18:44,700 --> 00:18:47,310
it will notify the
underlying networking stack,

418
00:18:47,310 --> 00:18:48,690
and they will send a SYN packet,

419
00:18:48,690 --> 00:18:51,090
the server return the SYN ACK

420
00:18:51,090 --> 00:18:53,160
and then the client return the ACK.

421
00:18:53,160 --> 00:18:55,203
Then the system call will return,

422
00:18:55,203 --> 00:18:58,320
tracepoint ptrace will read the arguments

423
00:18:58,320 --> 00:19:01,440
before the system call exit.

424
00:19:01,440 --> 00:19:02,580
Right?

425
00:19:02,580 --> 00:19:03,480
Now, we may be wondering

426
00:19:03,480 --> 00:19:05,820
if this was computer networking 101,

427
00:19:05,820 --> 00:19:07,233
what can go wrong here?

428
00:19:10,380 --> 00:19:12,030
Now, let's imagine this scenario.

429
00:19:13,951 --> 00:19:17,400
Many times when the attacker
compromise a machine,

430
00:19:17,400 --> 00:19:18,930
or the workload,

431
00:19:18,930 --> 00:19:21,960
they will try to talk to the
command and control server.

432
00:19:21,960 --> 00:19:24,150
This is a very practical setting.

433
00:19:24,150 --> 00:19:25,350
So, in this case,

434
00:19:25,350 --> 00:19:27,903
the attacker also
controls the server side.

435
00:19:28,920 --> 00:19:31,470
So, let's look at the example in detail.

436
00:19:31,470 --> 00:19:35,910
Now, in the diagram, there's
a client and a server.

437
00:19:35,910 --> 00:19:39,060
On the client, it's monitored
by the tracing software.

438
00:19:39,060 --> 00:19:41,910
And then the attacker
create a system call thread.

439
00:19:41,910 --> 00:19:43,100
The system call thread

440
00:19:43,100 --> 00:19:45,753
is first going to create
an override thread.

441
00:19:46,680 --> 00:19:47,820
And then the system call thread

442
00:19:47,820 --> 00:19:49,410
call the connect system call

443
00:19:49,410 --> 00:19:52,980
with a memory page that
points to the malicious IP

444
00:19:52,980 --> 00:19:54,063
it wants to talk to.

445
00:19:55,920 --> 00:19:57,990
Then the kernel will send the SYN packet

446
00:19:57,990 --> 00:19:59,253
over to the server.

447
00:20:00,660 --> 00:20:02,310
Now, remember the attacker
control the server,

448
00:20:02,310 --> 00:20:03,307
so the attacker's gonna to say,

449
00:20:03,307 --> 00:20:05,157
"Hey, I wanna drop the SYN packet."

450
00:20:06,780 --> 00:20:07,680
Then what happens?

451
00:20:08,940 --> 00:20:11,433
The client will resend another SYN packet.

452
00:20:12,930 --> 00:20:15,990
And then the server can
drop the packet again.

453
00:20:15,990 --> 00:20:17,540
Then the client will try again.

454
00:20:18,630 --> 00:20:20,520
So, every time the client retry,

455
00:20:20,520 --> 00:20:24,990
there's a delay controlled by
the TCP congestion algorithm.

456
00:20:24,990 --> 00:20:26,340
Without going to too much detail,

457
00:20:26,340 --> 00:20:27,480
roughly you can think about it

458
00:20:27,480 --> 00:20:29,943
as an exponential delay for every retry.

459
00:20:31,950 --> 00:20:33,570
While this retry is happening,

460
00:20:33,570 --> 00:20:35,280
the override thread is going to override

461
00:20:35,280 --> 00:20:38,343
that memory page with
the benign IP address.

462
00:20:39,930 --> 00:20:42,210
And then we have enough time

463
00:20:42,210 --> 00:20:43,740
for that data to propagate

464
00:20:43,740 --> 00:20:46,380
to all the memory copies on the system.

465
00:20:46,380 --> 00:20:48,427
And then finally the server will say,

466
00:20:48,427 --> 00:20:50,787
"Okay, here's your SYN ACK packet."

467
00:20:52,322 --> 00:20:54,570
And then tracepoint and ptrace

468
00:20:54,570 --> 00:20:56,763
will read out the benign IP address.

469
00:20:57,990 --> 00:20:59,740
And then the system call will exit.

470
00:21:02,670 --> 00:21:06,813
So, to show this works, we
will show you two demos.

471
00:21:08,490 --> 00:21:13,080
First, we'll show a demo
on VM container level.

472
00:21:13,080 --> 00:21:16,511
So, in this demo there will be...

473
00:21:16,511 --> 00:21:18,750
First, you are looking at the server side.

474
00:21:18,750 --> 00:21:23,433
Now, notice the server IP
address ending with 176.

475
00:21:25,350 --> 00:21:26,183
Okay?

476
00:21:26,183 --> 00:21:28,665
And then we fire up this SCP program,

477
00:21:28,665 --> 00:21:32,310
looking for specific
packet on a specific port.

478
00:21:32,310 --> 00:21:34,500
And then we also fire
up a listening server

479
00:21:34,500 --> 00:21:35,450
on the server side.

480
00:21:36,510 --> 00:21:37,560
And then on the client side,

481
00:21:37,560 --> 00:21:40,140
we run the Falco software.

482
00:21:40,140 --> 00:21:42,813
We also run the Wireshark program.

483
00:21:43,860 --> 00:21:44,793
Then we run the exploit.

484
00:21:44,793 --> 00:21:46,410
It's basically a TCP client

485
00:21:46,410 --> 00:21:48,000
that try to connect to the server.

486
00:21:48,000 --> 00:21:50,750
You can see the connection
is successfully established.

487
00:21:53,910 --> 00:21:56,220
And then it's sending some chat message.

488
00:21:56,220 --> 00:21:58,500
Now, you see the server
received the chat message,

489
00:21:58,500 --> 00:21:59,900
it's sending something back.

490
00:22:00,900 --> 00:22:01,733
Right?

491
00:22:01,733 --> 00:22:03,903
Everything is working,
the connection is there.

492
00:22:06,240 --> 00:22:11,240
But now if you look at the
IP address reported by Falco,

493
00:22:11,820 --> 00:22:15,270
it's actually saying 1.1.1.1.

494
00:22:15,270 --> 00:22:17,580
But if you look at the Wireshark,

495
00:22:17,580 --> 00:22:18,810
it's saying the first SYN pack,

496
00:22:18,810 --> 00:22:20,400
two SYN packets are dropped,

497
00:22:20,400 --> 00:22:22,380
and the IP address that is communicating

498
00:22:22,380 --> 00:22:24,090
is actually the server,

499
00:22:24,090 --> 00:22:25,893
the real server IP address.

500
00:22:27,330 --> 00:22:29,793
So, this indicates a successful bypass.

501
00:22:35,050 --> 00:22:36,930
Okay, now in the second demo,

502
00:22:36,930 --> 00:22:39,080
we're gonna show you
the bypass on Fargate.

503
00:22:41,940 --> 00:22:43,020
So, similar here,

504
00:22:43,020 --> 00:22:44,670
first we are showing the server side.

505
00:22:44,670 --> 00:22:47,553
The server IP ending with 163.

506
00:22:48,990 --> 00:22:51,420
And we're going to run the XDP program

507
00:22:51,420 --> 00:22:53,313
that's going to drop the SYN packet.

508
00:22:59,880 --> 00:23:02,433
And then we're going to
start a listening server.

509
00:23:05,970 --> 00:23:09,510
So, here's the Fargate container,

510
00:23:09,510 --> 00:23:13,323
it's going to run the pdig
with our attack program.

511
00:23:16,890 --> 00:23:19,640
So, notice that pdig reports
the connect system call

512
00:23:19,640 --> 00:23:22,110
is actually connecting to 1.1.1,

513
00:23:22,110 --> 00:23:24,990
which again indicates
it's not able to identify

514
00:23:24,990 --> 00:23:26,730
the real IP address.

515
00:23:26,730 --> 00:23:27,563
Right?

516
00:23:27,563 --> 00:23:28,440
You see the client server connected.

517
00:23:28,440 --> 00:23:29,553
They start chatting.

518
00:23:32,040 --> 00:23:34,677
But pdig reports, "Hey,
we're connecting to 1.1.1."

519
00:23:49,611 --> 00:23:51,339
Okay, so at this point
you're probably wondering,

520
00:23:51,339 --> 00:23:53,460
you know, this is for connect system call,

521
00:23:53,460 --> 00:23:55,500
what about the other system calls?

522
00:23:55,500 --> 00:23:58,050
So, we find out that the
entire class of file system

523
00:23:58,050 --> 00:23:59,463
system calls are impacted.

524
00:24:00,870 --> 00:24:05,280
Also, if other system calls
that relies on file system

525
00:24:05,280 --> 00:24:07,110
to perform some task are...

526
00:24:07,110 --> 00:24:08,130
They are also impacted.

527
00:24:08,130 --> 00:24:11,940
So, one example is the execve
and the execveat system call,

528
00:24:11,940 --> 00:24:14,310
because when you execute a binary,

529
00:24:14,310 --> 00:24:18,000
first it's going to the
disk to fetch the binary.

530
00:24:18,000 --> 00:24:20,250
Next, Junyuan will talk
about how we bypass

531
00:24:20,250 --> 00:24:21,993
the file system system calls.

532
00:24:24,270 --> 00:24:27,727
- So, before I talk about
how to bypass open access

533
00:24:27,727 --> 00:24:30,630
system call tracing, so
let me introduce FUSE.

534
00:24:30,630 --> 00:24:33,540
So, FUSE stands for user
space file system framework.

535
00:24:33,540 --> 00:24:35,820
It's usually included
in the kernel Module,

536
00:24:35,820 --> 00:24:39,300
used as a library and
also a remote utility.

537
00:24:39,300 --> 00:24:40,830
So, in cloud scenario,

538
00:24:40,830 --> 00:24:44,340
FUSE is used as the remote storage FUSE.

539
00:24:44,340 --> 00:24:48,030
So, with such FUSE we can
mount the remote objects

540
00:24:48,030 --> 00:24:51,630
as local file system and
also access the remote file

541
00:24:51,630 --> 00:24:52,473
as local file.

542
00:24:53,310 --> 00:24:55,530
Since it is the user space file system,

543
00:24:55,530 --> 00:24:59,370
so it provide faster
involvement or development,

544
00:24:59,370 --> 00:25:01,533
and it usually do not panic the kernel.

545
00:25:02,550 --> 00:25:05,250
So, remote storage FUSE is widely used.

546
00:25:05,250 --> 00:25:08,220
So, here is the list of examples.

547
00:25:08,220 --> 00:25:09,840
From this example we can see

548
00:25:09,840 --> 00:25:12,303
the major cloud providers
has their own FUSE.

549
00:25:14,760 --> 00:25:16,320
So, this is the general architecture

550
00:25:16,320 --> 00:25:18,690
for remote storage FUSE.

551
00:25:18,690 --> 00:25:21,510
If the user space application or container

552
00:25:21,510 --> 00:25:25,170
wants to open the remote
file, usually FUSE,

553
00:25:25,170 --> 00:25:28,120
basically what it does is the
same as opening a local file.

554
00:25:29,866 --> 00:25:32,580
It will issue an open
or open at system call

555
00:25:32,580 --> 00:25:34,380
into the kernel.

556
00:25:34,380 --> 00:25:39,380
And the request will be
routed from the VFS layer

557
00:25:39,450 --> 00:25:40,740
the FUSE kernel driver,

558
00:25:40,740 --> 00:25:45,183
and then to the user space file
system Daemon, like gcsfuse.

559
00:25:46,511 --> 00:25:49,980
The Daemon will send the
request to the remote storage,

560
00:25:49,980 --> 00:25:54,240
like GC storage or AWS s3.

561
00:25:54,240 --> 00:25:55,980
Once the response is back,

562
00:25:55,980 --> 00:25:58,980
it will send back to the
user space application

563
00:25:58,980 --> 00:26:00,280
through the original path.

564
00:26:01,170 --> 00:26:02,430
So, one thing I need to mention here

565
00:26:02,430 --> 00:26:06,330
is the delay between the
client and remote server

566
00:26:06,330 --> 00:26:10,350
is much longer than the
syscall is in the client.

567
00:26:10,350 --> 00:26:12,870
So, basically you can
leverage the long delay

568
00:26:12,870 --> 00:26:15,243
to bypass the open at source tracing.

569
00:26:17,520 --> 00:26:19,440
So, let's see how it works.

570
00:26:19,440 --> 00:26:21,390
So, we have the malicious client

571
00:26:21,390 --> 00:26:23,463
that is monitored by the tracing program.

572
00:26:24,946 --> 00:26:27,900
The syscall thread is
going to open a remote file

573
00:26:27,900 --> 00:26:29,073
called malicious file.

574
00:26:30,240 --> 00:26:32,400
Basically, it will issue
a open at system call

575
00:26:32,400 --> 00:26:34,350
into the kernel with the path name

576
00:26:34,350 --> 00:26:38,160
pointing to the malicious
file in user space.

577
00:26:38,160 --> 00:26:40,380
Since the file is stored remotely,

578
00:26:40,380 --> 00:26:43,907
so the open at request will be
routed from the kernel space

579
00:26:43,907 --> 00:26:47,373
to the user space and then
reach the remote server.

580
00:26:48,360 --> 00:26:50,340
Before the response is back,

581
00:26:50,340 --> 00:26:51,930
the override thread can jump in

582
00:26:51,930 --> 00:26:56,060
and override the path name pointed...

583
00:26:57,270 --> 00:26:59,430
Override the memory pointed by path name

584
00:26:59,430 --> 00:27:01,683
from malicious file name
to benign file name.

585
00:27:02,940 --> 00:27:05,760
And again, because the
delay is much longer

586
00:27:05,760 --> 00:27:07,530
than the syscall itself,

587
00:27:07,530 --> 00:27:10,650
so the CPU have enough
time to propagate a change

588
00:27:10,650 --> 00:27:13,413
to different copies of
memory and registers.

589
00:27:14,700 --> 00:27:17,430
So, after the response back,

590
00:27:17,430 --> 00:27:19,533
and before returning to the user space,

591
00:27:20,520 --> 00:27:23,880
the tracing program can
use the sys_exit tracepoint

592
00:27:23,880 --> 00:27:27,570
to get the arguments from the system call.

593
00:27:27,570 --> 00:27:30,000
Unfortunately, this
argument has been changed

594
00:27:30,000 --> 00:27:32,283
from malicious file name
to benign file name.

595
00:27:33,180 --> 00:27:35,547
In this case, we can successfully bypass

596
00:27:35,547 --> 00:27:37,233
the open at system call tracing.

597
00:27:38,790 --> 00:27:40,953
Let me show you how it works in a demo.

598
00:27:44,855 --> 00:27:47,605
(mouse clicking)

599
00:27:48,480 --> 00:27:50,640
So, we have a console on the left.

600
00:27:50,640 --> 00:27:54,930
So, for the Google Cloud storage bucket.

601
00:27:54,930 --> 00:27:57,843
And we deployed a Falco
agent in GKE cluster.

602
00:27:59,100 --> 00:28:00,700
And we log into one of the pods.

603
00:28:04,230 --> 00:28:06,570
It shows the process inside the container,

604
00:28:06,570 --> 00:28:09,380
so we have a Falco agent running,

605
00:28:09,380 --> 00:28:11,640
We have a GCS FUSE Daemon running,

606
00:28:11,640 --> 00:28:14,010
which mount the Google
Cloud storage bucket

607
00:28:14,010 --> 00:28:15,933
into the local folder, MNT.

608
00:28:16,980 --> 00:28:19,800
And then we check the logs for the Falco.

609
00:28:19,800 --> 00:28:22,110
So there are one event generated,

610
00:28:22,110 --> 00:28:23,760
because we just log into the pod.

611
00:28:27,570 --> 00:28:30,810
And then we check the
folder, MNT, so it's empty,

612
00:28:30,810 --> 00:28:33,543
which mean Google storage bucket is empty.

613
00:28:37,830 --> 00:28:40,967
And then we're trying to open the file

614
00:28:40,967 --> 00:28:44,583
called malicious file in MNT folder.

615
00:28:53,319 --> 00:28:54,630
Yeah, then we check the folder,

616
00:28:54,630 --> 00:28:57,120
so the malicious file is created

617
00:28:57,120 --> 00:28:59,643
in the remote storage in Google cloud.

618
00:29:03,420 --> 00:29:04,920
We check the Falco logs again.

619
00:29:04,920 --> 00:29:08,190
So, there are one event generated,

620
00:29:08,190 --> 00:29:10,320
because we're using an open at system call

621
00:29:10,320 --> 00:29:13,503
to open the malicious
file in the MNT folder.

622
00:29:15,270 --> 00:29:17,073
We remove the malicious file,

623
00:29:23,043 --> 00:29:26,370
which will be removed
from the remote server.

624
00:29:26,370 --> 00:29:28,773
So, you run the attack, as just mentioned,

625
00:29:32,610 --> 00:29:33,900
and you check the MNT folder.

626
00:29:33,900 --> 00:29:37,383
So, the malicious file is
created by the attack code.

627
00:29:40,590 --> 00:29:43,680
And then you check your log for the Falco.

628
00:29:43,680 --> 00:29:45,660
So, there are no new events generated,

629
00:29:45,660 --> 00:29:47,703
which means our bypass succeeded.

630
00:29:48,810 --> 00:29:49,970
Next, I hand over to Rex,

631
00:29:49,970 --> 00:29:51,990
to talk more about the exploits,

632
00:29:51,990 --> 00:29:52,983
and also conclude.

633
00:30:03,310 --> 00:30:05,540
- Okay, so, you know,
the previous gave you

634
00:30:05,540 --> 00:30:08,160
an example on networking and file system.

635
00:30:08,160 --> 00:30:10,140
Basically, these are examples

636
00:30:10,140 --> 00:30:11,790
to demonstrate that how you can trigger

637
00:30:11,790 --> 00:30:13,950
the blocking condition in system call.

638
00:30:13,950 --> 00:30:15,870
Now, you know, this doesn't exhaust

639
00:30:15,870 --> 00:30:16,950
all the blocking condition.

640
00:30:16,950 --> 00:30:19,440
There are many other
ways that you can trigger

641
00:30:19,440 --> 00:30:22,110
blocking conditions on
differences system calls.

642
00:30:22,110 --> 00:30:26,393
But we're going to talk about how to use

643
00:30:26,393 --> 00:30:29,403
seccomp to bypass syscall
tracing at sys_enter.

644
00:30:31,890 --> 00:30:35,550
So, if you recall the slide earlier,

645
00:30:35,550 --> 00:30:38,238
when the system called execute,

646
00:30:38,238 --> 00:30:40,207
it's first going to hit ptrace,

647
00:30:40,207 --> 00:30:41,790
and then going to hit seccomp,

648
00:30:41,790 --> 00:30:43,320
and then a bunch of other stuff.

649
00:30:43,320 --> 00:30:46,699
Then the real system call
is going to dereference

650
00:30:46,699 --> 00:30:47,880
the user space memory.

651
00:30:47,880 --> 00:30:48,713
Right?

652
00:30:48,713 --> 00:30:49,546
Now, the question is,

653
00:30:49,546 --> 00:30:52,500
can we use seccomp to
bypass the ptrace tracing

654
00:30:52,500 --> 00:30:53,577
at sys_enter?

655
00:30:56,820 --> 00:30:59,160
Before I dive into the exploit,

656
00:30:59,160 --> 00:31:01,830
just a brief recap about what seccomp is.

657
00:31:01,830 --> 00:31:06,690
So, seccomp is a kernel
mechanism to filter system calls.

658
00:31:06,690 --> 00:31:09,810
You can build, you know,
modern sandboxes on top of it,

659
00:31:09,810 --> 00:31:12,660
and it allow the developer
to filter system calls,

660
00:31:12,660 --> 00:31:15,060
and even based on their arguments.

661
00:31:15,060 --> 00:31:18,240
Now, if you have complex seccomp rules,

662
00:31:18,240 --> 00:31:20,220
this quickly become harder to compute,

663
00:31:20,220 --> 00:31:22,560
and takes more time to compute.

664
00:31:22,560 --> 00:31:25,170
Also, one caveat to keep in mind,

665
00:31:25,170 --> 00:31:27,000
which will be useful later,

666
00:31:27,000 --> 00:31:29,820
is the first inserted seccomp rule

667
00:31:29,820 --> 00:31:31,530
will get evaluated the last.

668
00:31:31,530 --> 00:31:34,563
So, it's evaluated in a reverse order.

669
00:31:37,920 --> 00:31:42,597
Okay, so here's how to
bypass the ptrace sys_enter.

670
00:31:43,650 --> 00:31:44,880
The idea is very simple.

671
00:31:44,880 --> 00:31:48,030
Now, what you're seeing here
is just a single machine.

672
00:31:48,030 --> 00:31:50,910
The attacker runs the system call thread.

673
00:31:50,910 --> 00:31:53,580
It's going to insert a
bunch of seccomp rules.

674
00:31:53,580 --> 00:31:55,580
And then it creates the override thread.

675
00:31:56,730 --> 00:31:58,800
And then it's going to call a system call.

676
00:31:58,800 --> 00:31:59,633
Right?

677
00:31:59,633 --> 00:32:01,140
So, in this diagram we use create at,

678
00:32:01,140 --> 00:32:03,990
but really this works for any system call.

679
00:32:03,990 --> 00:32:05,670
So, in the create system call

680
00:32:05,670 --> 00:32:07,350
I patch in the path name

681
00:32:07,350 --> 00:32:10,320
that points to malicious
file, sorry, benign file.

682
00:32:10,320 --> 00:32:13,923
And ptrace happily read
the benign file argument.

683
00:32:14,940 --> 00:32:18,000
Then the seccomp check starts.

684
00:32:18,000 --> 00:32:19,080
And then the override thread.

685
00:32:19,080 --> 00:32:20,510
At the same time,

686
00:32:20,510 --> 00:32:24,780
it's overriding the memory
with the malicious file.

687
00:32:25,830 --> 00:32:28,680
Because the seccomp take quite
a bit of time to compute,

688
00:32:28,680 --> 00:32:30,870
due to the number of
rules that we inserted,

689
00:32:30,870 --> 00:32:34,830
then the system has
enough time to propagate

690
00:32:34,830 --> 00:32:37,560
this benign file to all the memory copies.

691
00:32:37,560 --> 00:32:40,399
And then finally, seccomp compute finish,

692
00:32:40,399 --> 00:32:41,250
the kernel reads the argument,

693
00:32:41,250 --> 00:32:43,803
and kernel actually create
a malicious file for us.

694
00:32:47,970 --> 00:32:51,464
Okay, so there is one interesting caveat

695
00:32:51,464 --> 00:32:52,880
related to ptrace.

696
00:32:52,880 --> 00:32:56,100
It is, you can use ptrace in plain mode.

697
00:32:56,100 --> 00:32:57,720
You can also use ptrace in the mode called

698
00:32:57,720 --> 00:32:59,523
ptrace plus seccomp redirect.

699
00:33:00,630 --> 00:33:04,230
What that means is, when the tracer...

700
00:33:04,230 --> 00:33:07,170
If the tracer start the application,

701
00:33:07,170 --> 00:33:09,240
then the tracer can say,

702
00:33:09,240 --> 00:33:13,020
any system call that
the application calls,

703
00:33:13,020 --> 00:33:15,420
please redirect back to the tracer.

704
00:33:15,420 --> 00:33:19,230
So, the tracer sometimes
insert a seccomp rule first

705
00:33:19,230 --> 00:33:23,280
and that rule says as long
as you hit the seccomp rule,

706
00:33:23,280 --> 00:33:25,473
you redirect execution back to the tracer.

707
00:33:26,550 --> 00:33:30,930
So, if the tracer is able to
insert the first seccomp rule,

708
00:33:30,930 --> 00:33:32,880
then no matter how many seccomp rules

709
00:33:32,880 --> 00:33:34,470
that the application insert,

710
00:33:34,470 --> 00:33:36,300
it's not going to impact the timing window

711
00:33:36,300 --> 00:33:40,020
between when the tracer reads
the system call argument

712
00:33:40,020 --> 00:33:41,340
and when the kernel reads it.

713
00:33:41,340 --> 00:33:42,173
Right?

714
00:33:42,173 --> 00:33:44,763
So, in this case, it's not
vulnerable to the attack.

715
00:33:45,600 --> 00:33:49,140
But if you are attacking
to run the application,

716
00:33:49,140 --> 00:33:52,170
then you have no control
on how many seccomp rules

717
00:33:52,170 --> 00:33:54,423
has been inserted before
the tracer attached.

718
00:33:58,990 --> 00:34:03,540
Okay, so we have covered
a lot of different issues,

719
00:34:03,540 --> 00:34:05,640
and also how to exploit them.

720
00:34:05,640 --> 00:34:08,040
So, just to summarize these issues,

721
00:34:08,040 --> 00:34:11,373
and also how to mitigate
different scenarios.

722
00:34:12,780 --> 00:34:16,620
So, first thing is ptrace at sys_enter.

723
00:34:16,620 --> 00:34:18,628
Now, in order to mitigate this,

724
00:34:18,628 --> 00:34:22,470
you can use ptrace plus seccomp redirect,

725
00:34:22,470 --> 00:34:25,320
but you need to start
the application as well.

726
00:34:25,320 --> 00:34:27,510
Now, if you need to attach
to a running application,

727
00:34:27,510 --> 00:34:30,000
you may want to inspect
what seccomp filter

728
00:34:30,000 --> 00:34:31,270
has already been inserted

729
00:34:32,250 --> 00:34:34,053
to see if there's any anomaly.

730
00:34:35,190 --> 00:34:37,440
Otherwise, you can also
insert a seccomp redirect,

731
00:34:37,440 --> 00:34:40,200
but then that means all
the previous seccomp rules

732
00:34:40,200 --> 00:34:42,540
inserted by the application are ignored.

733
00:34:42,540 --> 00:34:43,713
So, it won't be useful.

734
00:34:46,260 --> 00:34:49,770
The second thing is
tracepoint at sys_enter.

735
00:34:49,770 --> 00:34:51,840
Although the TOCTOU issue exists,

736
00:34:51,840 --> 00:34:54,603
but we haven't found a
reliable way to exploit it.

737
00:34:56,190 --> 00:34:58,293
The next one is tracepoint at sys_exit.

738
00:34:59,130 --> 00:35:00,300
Now, this is a vulnerable,

739
00:35:00,300 --> 00:35:02,280
to both the attack in this talk,

740
00:35:02,280 --> 00:35:05,015
and also our research last year.

741
00:35:05,015 --> 00:35:07,860
In order to mitigate this, one option is

742
00:35:07,860 --> 00:35:10,950
to compare the sys_enter
and sys_exit tracepoint data

743
00:35:10,950 --> 00:35:15,030
to see if the argument has been tampered.

744
00:35:15,030 --> 00:35:17,680
Of course, this will incur
some performance overhead.

745
00:35:20,070 --> 00:35:22,950
And the other option is to use kprobe,

746
00:35:22,950 --> 00:35:25,563
which I'll soon talk
about in the last row.

747
00:35:27,150 --> 00:35:29,670
So, next one is ptrace at sys_exit.

748
00:35:29,670 --> 00:35:33,720
Now, it's vulnerable to the
same attack as the previous one.

749
00:35:33,720 --> 00:35:35,430
In order to mitigate this,

750
00:35:35,430 --> 00:35:37,972
one will need to deploy
all the mitigations

751
00:35:37,972 --> 00:35:40,402
at ptrace at sys_enter and then compare

752
00:35:40,402 --> 00:35:45,033
the ptrace at sys_enter
and sys_exit results.

753
00:35:46,388 --> 00:35:47,221
The last thing is,

754
00:35:47,221 --> 00:35:51,090
kprobe and other internal
kernel functions.

755
00:35:51,090 --> 00:35:53,190
Whether this is vulnerable really depends

756
00:35:53,190 --> 00:35:56,223
on which kernel function is being probed.

757
00:35:57,960 --> 00:35:58,793
But in general,

758
00:35:58,793 --> 00:36:02,610
if you want to avoid the TOCTOU issues,

759
00:36:02,610 --> 00:36:05,280
specific on system call monitoring,

760
00:36:05,280 --> 00:36:08,823
you can take the Linux
security module interface.

761
00:36:09,840 --> 00:36:14,002
Some newer kernels also
allow you to do BPF-LSM,

762
00:36:14,002 --> 00:36:19,002
but this depends on how
wide range of kernel

763
00:36:19,830 --> 00:36:20,880
you need to support.

764
00:36:20,880 --> 00:36:23,490
Because the Linux
security module interface

765
00:36:23,490 --> 00:36:27,000
have different support across
different kernel versions.

766
00:36:27,000 --> 00:36:32,000
And if the, you know,
specific kernel interface

767
00:36:32,130 --> 00:36:36,150
you happen to inspect is not in RSM,

768
00:36:36,150 --> 00:36:38,550
then you quickly get into the complexity

769
00:36:38,550 --> 00:36:40,410
of figuring out whether that exists

770
00:36:40,410 --> 00:36:42,653
in all the kernel versions
you want to support.

771
00:36:45,810 --> 00:36:47,883
Okay, so to conclude.

772
00:36:48,780 --> 00:36:53,712
First, if we zoom into
this research issue,

773
00:36:53,712 --> 00:36:57,210
the first thing is we showed
that the kernel tracing

774
00:36:57,210 --> 00:37:00,420
can be bypassed reliably
in many different ways.

775
00:37:00,420 --> 00:37:04,320
So, if you happen to use a
tool that does similar things,

776
00:37:04,320 --> 00:37:07,350
you may want to check if your
tool is vulnerable to this.

777
00:37:07,350 --> 00:37:10,397
Because we've only evaluated
the open source solution,

778
00:37:10,397 --> 00:37:13,443
we didn't evaluate any
proprietary software.

779
00:37:15,180 --> 00:37:16,620
The second thing to keep in mind

780
00:37:16,620 --> 00:37:18,300
is the mitigation is complicated,

781
00:37:18,300 --> 00:37:22,230
it depends on what, you know,
form factors that you use,

782
00:37:22,230 --> 00:37:24,870
and also what kernel
versions that the software

783
00:37:24,870 --> 00:37:25,860
needs to support.

784
00:37:25,860 --> 00:37:28,560
So, you want to double
check that the mitigation

785
00:37:28,560 --> 00:37:32,700
was actually implemented in
the way that you want it to be.

786
00:37:32,700 --> 00:37:33,533
Right?

787
00:37:34,922 --> 00:37:36,780
And then, if I zoom out

788
00:37:36,780 --> 00:37:40,200
from this particular issue a little bit,

789
00:37:40,200 --> 00:37:42,420
if we think about the bigger picture,

790
00:37:42,420 --> 00:37:46,530
if your security team is able to correlate

791
00:37:46,530 --> 00:37:49,950
different data sources and
have a comprehensive view

792
00:37:49,950 --> 00:37:51,150
about your environment,

793
00:37:51,150 --> 00:37:55,143
then it suddenly make the
evasion complexity much higher.

794
00:37:56,520 --> 00:37:57,570
Last, but not least,

795
00:37:57,570 --> 00:38:00,540
this is something that
we do lot at Lacework,

796
00:38:00,540 --> 00:38:05,247
is that we understand, you know, we...

797
00:38:05,247 --> 00:38:08,040
You should understand
that your environment,

798
00:38:08,040 --> 00:38:08,873
what is normal,

799
00:38:08,873 --> 00:38:11,610
what is intended to be
wrong in your environment.

800
00:38:11,610 --> 00:38:14,790
If you know the baseline
of your environment,

801
00:38:14,790 --> 00:38:16,020
then even if the attacker

802
00:38:16,020 --> 00:38:19,110
is able to override these arguments,

803
00:38:19,110 --> 00:38:21,656
they're basically being
constrained to the value

804
00:38:21,656 --> 00:38:26,656
that is fit into the normal expectation.

805
00:38:26,850 --> 00:38:27,683
Right?

806
00:38:29,880 --> 00:38:31,650
Okay, before we take the Q&A,

807
00:38:31,650 --> 00:38:35,280
I just wanna give a quick shout
out to the following folks.

808
00:38:35,280 --> 00:38:39,060
So, Joe helped with the kernel
and security discussion.

809
00:38:39,060 --> 00:38:42,000
A lot of my Lacework labs colleagues

810
00:38:42,000 --> 00:38:44,280
give great feedback on this work.

811
00:38:44,280 --> 00:38:46,620
Also, I wanna thank the
Falco open source team

812
00:38:46,620 --> 00:38:50,670
for a very collaborative
disclosure process.

813
00:38:50,670 --> 00:38:53,493
And also John Dixon for
helping the presentation.

814
00:38:56,250 --> 00:38:59,160
So, before finally we take the Q&A,

815
00:38:59,160 --> 00:39:00,840
if you have any other question

816
00:39:00,840 --> 00:39:02,910
about cloud security in your environment,

817
00:39:02,910 --> 00:39:05,550
or if you want to advance
your career in cloud security,

818
00:39:05,550 --> 00:39:08,613
I'm very happy to talk
to you guys offline.

819
00:39:09,660 --> 00:39:11,463
With that, we're ready for Q&A.

820
00:39:13,870 --> 00:39:17,453
(attendees clapping hands)

821
00:39:26,280 --> 00:39:28,500
So, anyone who want to ask a question,

822
00:39:28,500 --> 00:39:29,490
there's a mic here.

823
00:39:29,490 --> 00:39:34,490
And Steve will help to walk to you guys.

824
00:39:40,500 --> 00:39:41,580
Hi.

825
00:39:41,580 --> 00:39:42,413
- [Attendee] Can you hear me?

826
00:39:42,413 --> 00:39:43,246
- Yeah.
- Okay.

827
00:39:44,190 --> 00:39:46,710
In some of your screen captures,

828
00:39:46,710 --> 00:39:48,600
you had, kind of, your
exploits from last year,

829
00:39:48,600 --> 00:39:51,060
they were Phantom V1.

830
00:39:51,060 --> 00:39:53,220
These are Phantom V3.

831
00:39:53,220 --> 00:39:55,113
What was Phantom V2?

832
00:39:56,280 --> 00:39:57,270
- Oh yeah. Thank you.

833
00:39:57,270 --> 00:39:58,103
Thank you, for that.

834
00:39:58,103 --> 00:40:02,541
So, certainly there's too
many attacks that we publish.

835
00:40:02,541 --> 00:40:06,993
So, Phantom V2, at the time we
called it semantic confusion.

836
00:40:08,040 --> 00:40:11,280
So, for the folks who's not familiar with,

837
00:40:11,280 --> 00:40:13,680
that's also a bypass on
the system call tracing.

838
00:40:14,670 --> 00:40:15,543
The idea is this,

839
00:40:17,340 --> 00:40:20,190
when you trace file search an object,

840
00:40:20,190 --> 00:40:21,633
for example etc/shadow,

841
00:40:22,620 --> 00:40:24,300
if it is an open at,

842
00:40:24,300 --> 00:40:26,697
then the rule is basically detecting

843
00:40:26,697 --> 00:40:28,200
whether the path is open at.

844
00:40:28,200 --> 00:40:29,790
But what if you create a link

845
00:40:29,790 --> 00:40:32,580
that is the linking to the file object,

846
00:40:32,580 --> 00:40:34,200
and then you open the link?

847
00:40:34,200 --> 00:40:35,280
Then the tracing software

848
00:40:35,280 --> 00:40:36,720
will not be able to interpret it,

849
00:40:36,720 --> 00:40:38,820
but the kernel will be
able to interpret it.

850
00:40:38,820 --> 00:40:40,830
So, there's a confusion
between the tracing software

851
00:40:40,830 --> 00:40:41,663
and the kernel.

852
00:41:01,018 --> 00:41:01,920
All right.

853
00:41:01,920 --> 00:41:04,042
Thank you, everyone, for coming.

854
00:41:04,042 --> 00:41:06,508
(attendees clapping hands)