Defcon 30. Who's having a good time? Whoa. You guys, awesome, awesome. That's great. So,
this talk is by Rex and Jun Yuan. So, let's give a big Defcon 30 welcome. Come on. Yeah.
Okay. Thank you, everyone, for coming to our talk about bypassing system call tracing. My
pleasure. We have a lot to cover in this talk. So, I will just jump over this slide. You can look
up us online. Okay. So, imagine sophisticated attacker compromise your Linux production
environment. He launches a lock for shell exploit and then fire reverse shell back to his
machine. Then he discovered that the machine is running a vulnerable version of pseudo. So, he
elevated privilege on the box. And then he read the
shadow file to see if there's any interesting hashes or crack. He also discovered that he can do
SSH hijacking to move into the second machine by reading to the process environment variable. So,
he moved to the second machine. And as he is celebrating, he discovered that his access was
gone. So, he fired the RC again. There's no luck. Quickly, he discovered his access was completed
block.
Let's take a look at the other side of the story. While everything is happening, your security
engineer received a sequence of alert from the cutting edge security monitoring software. And
this software is able to monitor the system calls and also processing information for the
application. So, for example, when the attacker executed reverse shell, there will be a connect
system call. And maybe others, depending on the reverse shell technique. When the attacker read
the SC shadow file, there will be an open. Open. Open. Open. Open. Open. Open. Open. Open. Open.
We just ignore it in that system call. Essentially any non-trivial action will involve a system call.
How do we use the system call information to detect threats? Here's a really, really
simple example. Let me explain what this is. Here's a root that try to detect untrusted programs
reading the SEO file. Let me explain the rule. It is saying that if the system call is exhausted,
is open or open at, and has a read permission, and the file name points to at cshadow file,
and the program is not in the allow list of programs, that will fire an alert.
Now you can probably see that you can build much more complex rules and even machine learning
models on top of this data set with one system call, even multiple system calls.
But they all rely on the fact that the system call monitoring software is able to extract
the data correctly.
So in this talk, we were going to dive very deep into these system call tracing technologies,
and then we'll talk about the vulnerabilities that we discover that allow us to bypass the
tracing.
And then we'll conclude with mitigations and takeaways.
With that, I will hand over to Junyuan.
Thank you, Rex.
So as Rex mentioned, system call tracing is very important to detect any threats.
So this diagram can give you an overview of system call tracing.
Basically the system call tracing includes two parts.
One is the hooks for system call interception, and also the tracing programs.
So when the application issues a system call into the kernel, the system call code pass
is executed.
If any hooks are placed inside the code pass, the attached tracing program will be triggered
to collect system call data, and send those information to the system.
So the tracing program can be implemented in the kernel space, as shown on the left
diagram, or implemented as a user space program, as part of the monitoring agent, which is
shown on the right side.
So the program used to collect system call data, like system call arguments, is called
tracing program.
This program can be attached to different hooks, like TracePoint, KPro, KeyTrace.
So we can directly relate the data to the system call.
So the first kind of hooks for system call interception is called TracePoint, basically
is the kernel static hook.
For system call interception, the Linux kernel provides sysenter and sysexit TracePoint.
So if we attach the tracing program to the TracePoint, the function call of tracestenter
and tracestexit will use the system code.
So we can use the same parameters to trigger the tracing program.
The first parameter is called rex, that's saving the system call arguments.
And the second is called id, which saves the system call number.
TracePoint provides low overhead, but it only provides static system call interception.
You can also use the dynamic approach, like KPro, to intercept system call.
So using KPro, you can register the tracing program on almost any structures in the kernel
code path.
Basically, it's the syscall code path.
So when the instruction gets computed, the tracing program will be called.
Compared to TracePoint, KPro is the dynamic approach.
But you need to know exactly how data is placed on the stack or register in order to get the
useful information.
So KeyTrace provides a user space solution for system call interception.
Similar to the TracePoint is static hook in the system call enter and exit.
Using KeyTrace, you don't need to implement any kernel programs as a tracing program,
but only user space programs are needed.
Compared to the previous tool approach, KeyTrace's performance overhead is high.
But for optimization, you can combine it with the SATCOM as a syscall filtering for
better performance.
The other approach for system call interception, like LDPreload, this approach actually is
easy to bypass.
If we use assembly code, it's very easy to bypass.
You just need to use the code to trigger the system call.
So many of you have probably heard about the Cloud Workload Protection products.
These products usually provide advanced threat detection based on syscall tracing.
There are different kinds of cloud workloads, such as virtual machine containers on customer
managed VMs, service containers, and others.
So for service containers, it's usually allocated and maintained by the cloud providers on
demand.
So it usually has no access to the host.
This table summarizes how different syscall tracing techniques can be applied for different
cloud workloads.
For virtual machines, we can use any kinds of hooks and tracing programs.
Tools we can leverage include the Fargo eBPF, kernel module, and Fargo PDIC.
We'll talk about these tools later in our talk.
For containers that are on customer managed VMs, we can have the same options.
As virtual machines, as long as we get enough capabilities.
For service containers, as I mentioned, it has no access to the host.
So we can only use ptrace as hooking points and implement tracing program in user space.
Instead of Fargo kernel module and eBPF programs, we can only use the Fargo PDIC.
Fargo used a similar technique to trace system call.
It's an open source project.
It's very popular, and in kernel space, it supports kernel module and eBPF Pro using
TracePoint.
In user space, PDIC is developed based on ptrace.
So Fargo PDIC is dedicated for syscall tracing of service workloads.
So we did not evaluate other security monitoring agents, but we believe the popularity of Fargo
PDIC represents an implementation that is widely accepted by the community.
Unfortunately, this kind of implementation for syscall tracing is vulnerable to a total
issue that is time of check, time of use.
Let's take the connect system call, for example.
The second argument for connect system call is called user Vaddress.
That is a user pointer pointing to the user space buffer called socket address.
So during time of check, the tracing program will reference this user space pointer to
to get the socket address.
And during time of use, the kernel also
could reference the same user space pointer
to also get the socket address.
But between time of check and time of use,
the user memory pointed by the user V address
is vulnerable to be changed by the attacker
from the user space.
So in this case, the socket address can be different
between time of check and time of use,
causing a total issue.
So let's dive into the connect system call,
which can help you understand this total issue.
So when the application issues a connect system call
into the kernel, the system call handler will check
if any tracing programs are attached to the static hook
at the system call enter, like ptrace, setcon,
sysender, tracepoint.
If this is true, the tracing program will be caught.
And after that, the handler
will look up the syscall table
and jump to connect system call
to create a connection on the socket.
Before returning to the user space,
the handler will again check if any tracing program
attached to the static hook at syscall exit,
like ptrace, sys exit tracepoint.
Similarly, if this is true,
the tracing program will be caught.
So as I mentioned earlier, the second argument
for connect system call is user pointer.
Pointing to the user space,
or socket address in the user space.
This pointer is propagated into the connect system call pass
and assigned to different kernel variables,
which is highlighted in red.
The kernel calls move address to kernel function
to make a copy of socket address from user space
to the kernel buffer, called address,
that is highlighted in green.
After that, the kernel will call the internal function, six,
and connect file function to create a connection
on the socket based on the kernel buffer address.
And this is time of use by the Linux kernel
for connect system call arguments.
So before the memory copy function,
the kernel buffer is not created.
So the user pointer is the only place we can give reference
and also get the socket address from the user space.
In this case, during time of check,
if we attach the tracing program
to the static hook at system call enter,
or to any places before the memory copy function,
using dynamic approach like kpro,
the tracing programs have no other option
but need to reverse the user pointer
to get the socket address.
After the memory copy function,
the kernel buffer is created,
is copied of the socket address from user space.
However,
in this case, this total issue may still exist.
So think about if we attach the tracing program
to the static hook at system call exit,
like this exit trace point or p-trace.
The tracing program may still be able to reference
the user pointer to get the socket address.
Funcopedic use p-trace for system call enter and exit,
but it only use the setcon system call filtering
as system call enter.
Since the setcon is not available for system call exit.
For file conversion older than 0.31.1,
the kernel module and EBPF implementation
only use the six exit point.
So hopefully you get some idea about the total issue
for system call tracing.
Next, I will hand over to Rex,
talk about the vulnerability.
Okay, so the example that we see is on kernel version 2.0.
5.7 and connect system call, but really the observation is true across all the kernel
versions because we confirm with the kernel developers these top two windows exist since
the day that TracePoint and P-Trace are introduced because they were originally designed for
performance and debugging purposes.
So in order to do secure tracing, what they recommend is the software need to monitor
the kernel memory instead.
So report this issue to Falco.
Basically the issue is they use the SysExit TracePoint and they also use the SysExit P-Trace
and there's a version older than 0.31.1 is impacted and if you happen to use the commercial
version, you may want to check what version is impacted.
So we report this issue in December and the issue is mitigated immediately.
In March.
The mitigation that was deployed is for the Linux security module or the eBPF version.
They check the SysEnter and SysExit system call arguments for selective set of system
calls.
And they also do the same for Falco PDIG.
In terms of how many system calls are impacted, so we analyzed open source rules in the Falco
and the majority of them are impacted with two exceptions.
One is the exact V system call.
The reason for that is in their implementation, when they trace the exact V, they actually
read the kernel memory instead.
The other one is the send to and send message system call.
We haven't found a reliable way to really block the system call.
We'll talk about what blocking means in the next few slides.
But keep in mind that it's very heavy to monitor send to and send message.
So we're going to talk a little bit more about that in the next few slides.
So the first one is the
OK, so hopefully everybody understands the vulnerability at this point.
It's fairly, relatively simple.
But let's talk about how to exploit it.
Now to exploit this tracing, we don't want to acquire any additional privilege or capabilities.
Basically, attackers should be able to evade the detection at any user, any privilege or
capabilities.
We want to have some level of control on the time to inject the delay, and we'll talk
more about what I mean by that later.
We also need to inject enough delay, such that when we override the memory, the machine
has enough time to propagate the overwritten data to the whole machine.
The last thing is we want this exploit to be 100% reliable, because if the attacker
being detected once, then it's in.
The entire operation may be at risk.
So this leads us to two exploitation strategies.
We'll just quickly brief overview the exploit strategy number one, which is what we did
in DEF CON last year.
And from there, you will see how we reach exploit strategy number two.
So last year, we discovered that you can actually inject a relatively small amount of delay
by using cross-core interrupt.
And because the amount of delay we inject is relatively small, we have to precisely
orchestrate the whole process.
What I mean by that is have to inject at the precise time, precisely override it, and
also synchronize using some special techniques.
One of the primitives that we use in the whole exploit is user for FD system call.
Now although this technique is powerful, it has some limitations.
So for example, if you use Docker container.
And.
Enable the Docker defaults that come profile, Docker will actually block the system call.
The other thing is that most cloud workloads don't use the system call.
So the usage of the system call itself indicates some kind of anomaly.
And this is actually the mitigation deployed by Falco last year.
They implemented detection rule to detect the usage of the system call.
So last year we went back.
We think.
You know.
Can we actually develop something that doesn't rely on using the system call.
So we come up with a scenario.
What if you can actually inject a delay that is really, really long.
Then you don't have to orchestrate everything in the precise timing.
It sounds very simple.
But the question is how to do this.
And.
We actually find out there are two ways to do this.
One is to use the blocking condition in system call.
This can be used to attack sys exit.
The other one is to insert additional second rules.
This can be used to attack sys enter.
So what do I mean by blocking the system call.
If you think about the underlying mechanism of system call.
It's basically the kernel interacting with resources on behalf of the user space program.
Now.
A lot of these resources are I.O.
Devices and the kernel will need to wait for the response from the device before you need
to proceed.
So let's look at a concrete example.
This is the connect system call.
In this diagram there are two machines.
There's a client.
There's a server.
Now let's say the client is monitored by a tracing software.
The client wants to talk to the server.
So he issues a connect system call.
When the kernel is connected.
When the kernel starts executing, it will notify the underlying networking stack and
then it will send a sim packet, server return a sim act, and then client return an act.
Then the system call will return, trace point p-trace, will read the arguments before the
system call exit.
Now you may be wondering this is computer networking 101.
What can go wrong here?
Now let's imagine this scenario.
Okay.
Well, many times when the attacker compromised a machine or the workload, they would try
to talk to the command and control server.
This is a very practical setting.
So in this case, the attacker also controls the server side.
So let's look at the example in detail.
Now in the diagram, there's a client and a server.
On the client, it's monitored by the tracing software.
And then the attacker created a system call threat.
The system call threat.
The threat is first going to create an override threat.
And then the system call threat called the connect system call with a memory page that
points to the malicious IP it wants to talk to.
Then the kernel will send the sim packet over to the server.
Now remember the attacker controlled the server.
So the attacker can say, hey, I want to drop the sim packet.
Then what happened?
The client will resend another sim packet.
Okay.
And then the server can drop the packet again.
Then the client will try again.
So every time the client retries, there's a delay controlled by the TCP congestion algorithm.
Without going into too much detail, roughly you can think about it as an exponential delay
for every retry.
While this retry is happening, the override threat is going to override that memory page
with the benign IP address.
And then we have enough time for that.
We have enough data to propagate to all the memory copies on the system.
And then finally servers say, okay, here's your sim packet.
And then the trace point and p-trace will read out the benign IP address.
And then the system call will exit.
So to show this works, we'll show you two demos.
First we'll show a demo on VM container level.
So in this demo, first you're looking at the server side.
Now notice the server IP address ending with 176.
Okay.
And then we fire up this XDP program.
This XDP program looking for a specific sim packet
on a specific port.
And then we also fire up a listening server on the server side.
And then on the client side, we run the Falco software.
We also run the Wireshark program.
Then we run the exploit.
It's basically a TCP client that try to connect to a server.
You see the connection successfully established.
And then it's sending some chat message.
Now you see the server received the chat message.
It's sending something back.
Right, everything's working.
Connection is there.
But now if you look at the IP address reported by Falco,
it's actually saying, okay, this is the IP address.
This is the server 1.1.1.1.
But if you look at the Wireshark,
it's saying the first two sim packet are dropped.
And the IP address that is communicating
is actually the server, the real server IP address.
So this indicates a successful bypass.
Okay, now in the second demo,
we're gonna show you the bypass on Fargate.
So similar here, first we're showing the server side.
The server IP ending was 163.
And we're going to run the XDP program
that's going to drop the sim packet.
And then we're going to start a listening server.
So here's the Fargate container.
It's going to run the PDIC with our attack program.
So notice that PDIC report that the connect system call
is actually connecting to 1.1.1,
which again indicates it's not able to identify
the real IP address.
Right, you see the client server connected.
They start chatting.
But PDIC report, hey, we're connecting to 1.1.1.
Just for yourinals are problem.
Okay.
Okay, so at this point you're probably wondering,
this is for connect system call.
What about the other system calls?
We find out that the entire class of file system
and system calls are impacted.
Also, if other standing calls that relies on file system
to perform some task are also impacted.
So one example.
is the exact V and exact V at system call.
Because when you execute a binary,
first it's going to the disk to fetch the binary.
Next, Junyuan will talk about how we bypass
the file system calls.
So before I talk about how to bypass
open access and call tracing,
so let me introduce FUSE.
So FUSE stands for User Space File System Bring Work.
It usually include kernel module,
users of the library, and also mount utility.
So in cloud scenario, FUSE is used
as the remote storage FUSE.
So with such FUSE we can mount the remote objects
as local file system and also access
the remote file as local file.
Since it's a user space file system,
so it provide faster enrollment or development.
And it usually do not panic the kernel.
So remote storage FUSE is widely used.
So here is the list of example.
From this example you can see the major cloud provider
has their own FUSE.
So this is the general architecture
for remote storage FUSE.
If the user space application or container
wants to open the remote file, use the FUSE.
Basically what it does is the same as opening a local file.
It will issue an open or open access and call
into the kernel.
The kernel will be routed from the VFS layer
to the FUSE kernel driver and then to the user space file
system daemon like GCS FUSE.
The daemon will send a request to the remote storage
like GCS storage or AWS S3.
Once the response is back, it will send back
to the user space application through the original path.
So one thing I need to mention here is the delay between the
remote server and remote server is much longer
than the syscall in the client.
So basically you can leverage the long delay
to bypass the open access tracing.
So let's see how it works.
So we have the malicious client that is monitored
by the tracing program.
The syscall thread is going to open a remote file
called malicious file.
Basically it will issue an open access and call
into the kernel with the path name pointing to the
malicious file in user space.
Since the file is stored remotely, so the open at request
will be routed from the kernel space to the user space
and then reach to the remote server.
Before the response is back, the override thread
can jump in and override the path name pointed,
override the memory pointed by path name
from malicious file name to benign file name.
And again, because the delay is much longer than the normal
syscall itself, so the CPU have enough time to propagate
a change to different copies of memory and registers.
So what after the response is back and before returning
to the user space, the tracing program can use the
six exit trace point to get the arguments
from the system call.
Unfortunately, this argument has been changed
from malicious file name to benign file name.
In this case, it can successfully bypass the equalizer,
bypass the Open Assistant Core Tracing.
Let me show you how it works in a demo.
So we have a console on the left for the Google Cloud Storage
bucket.
And we deployed a FACO agent in GKE cluster.
And we log into one of the pod.
We check the process inside the container.
So we have a FACO agent running.
We have a GCS Fuse Demon running,
which mount the Google Cloud Storage bucket
into the local folder, MNT.
And then we check the logs for the FACO.
So they are one event generated, because we just
log into the pod.
And then we check the folder, MNT, so it's empty.
Which means Google Storage bucket is empty.
And then we're trying to open the file,
open a file called malicious file in MNT folder.
Yeah, then we check the folder.
So the malicious file is created in the remote storage
in Google Cloud.
We check the FACO logs again.
So they're one event generated.
Because we're using open access and call
to open the malicious file in the MNT folder.
We remove the malicious file, which
will be removed from the remote server.
So you run the attack, as I just mentioned.
And we check the MNT folder.
So the malicious file is created by the attack code.
And then we check the log for the FACO.
So there are no new event generated,
which means our bypass has succeeded.
Next, I hand over to Rex to talk more about the exploits
and also conclude.
OK, so the previous example, networking and file system.
Basically, these are examples to demonstrate
how you can trigger the blocking condition in system call.
Now, this doesn't exhaust all the blocking condition.
There are many other ways that you
can trigger blocking condition on different system calls.
But we're going to talk about how
to use SETCOM to bypass this call tracing as this enter.
OK.
So if you recall the slide earlier,
when the system call executes, it's
first going to hit p-trees, and then going to hit SETCOM,
and then a bunch of other stuff.
Then the real system call is going
to use the reference to user space memory, right?
Now, the question is, can we use SETCOM
to bypass the p-trees tracing as the center?
Before I dive into the exploit, just a brief recap.
About what SETCOM is.
So SETCOM is a kernel mechanism to filter system calls.
You can build more than a sandbox on top of it.
And you allow the developer to filter system calls,
and even based on their arguments.
Now, if you have complex SETCOM rules,
this quickly becomes harder to compute,
and takes more time to compute.
Also, one caveat to keep in mind,
which will be useful later, is the first inserted SETCOM rule
will get evaluated at last.
So it's evaluated in the reverse order.
OK, so here's how to bypass the p-trees SET center.
The idea is very simple.
Now, what you're seeing here is just a single machine.
The attacker runs the system call thread.
First, it's going to insert a bunch of SETCOM rules.
And then it creates the override thread.
And then it's going to call a system call, right?
So in this diagram, we use create at.
But really, this works for any system call.
So in the create system call, it passes in the path name
that points to malicious file, sorry, benign file.
And p-trees happily read the benign file argument.
Then the SETCOM check starts.
And then the override thread at the same time
is overriding the memory with the malicious file.
Because the SETCOM take quite a bit of time to compute,
due to the number of rules that we inserted,
then the system has enough time to propagate
this benign file to all the memory copies.
And then finally, SETCOM compute finish.
Kernel reads the argument.
Kernel actually create a malicious file for us.
OK, so there's one interesting caveat related to p-trees,
is you can use p-trees in play mode.
You can also p-trees in the mode called
of ptrace plus seccomp-redirect.
What that means is,
when the tracer, if the tracer start the application,
then the tracer can say,
any system call that the application calls,
please redirect back to the tracer.
So tracer is going to insert a seccomp rule first.
And that rule says, as long as you hit the seccomp rule,
you redirect execution back to the tracer.
So if the tracer is able to insert the first seccomp rule,
then no matter how many seccomp rule
that the application inserts,
it's not going to impact the timing window
between when the tracer reads the system call argument
and when the kernel reads it.
So in this case, it's not vulnerable to the attack.
But if you are attaching to a running application,
then you have no control on how many seccomp rule
has been inserted before the tracer attach.
Okay, so we have covered a lot of different issues,
and also how to exploit them.
So just to summarize these issues,
and also how to mitigate different scenarios.
So first thing is ptrace access enter.
Now in order to mitigate this,
you can use ptrace plus seccomp-redirect,
but you need to start the application as well.
Now if you need to attach to a running application,
you may want to inspect what seccomp filter
has already been inserted to see if there's any anomaly.
Otherwise, you can also insert a seccomp-redirect,
but then that means all the previous seccomp rule
inserted by the application are ignored.
So it won't be used for it.
The second thing is tracepoint access enter.
Although the talk-to issue exists,
but we haven't found out a reliable way to exploit it.
The next one is tracepoint access exit.
Now this is vulnerable to both the attack in this talk,
and also our research last year.
In order to mitigate this,
one option is to compare the sysenter
and sysexit tracepoint data
to see if the argument has been tampered.
Of course, this will incur some performance overhead.
And the other option is to use kpro,
which I will soon talk about in the last, uh,
So next one is ptrace access exit.
Now it's vulnerable to the same attack as the previous one.
In order to mitigate this,
one will need to deploy all the mitigations
at ptrace access enter,
and then compare the ptrace sysenter and sysexit results.
Last thing is kprobe on other kernel internal functions.
Whether this is vulnerable really depends on which kernel
functional is being probe,
um, but in general, if you want to avoid
the doctor issues,
specifically on systematha problem monitoring,
uh, you can pick the linux security module interface.
Some newer kernels also allow you to do intdpf rsm,
but this depends on how wide range of sindh
of kernel you need to support.
Because the linux security module interface has different support, um, uh, in-d片 of of kernel sysnet hoody
different kernel versions, and if the specific kernel interface you happen to inspect is
not in RSM, then you quickly get into the complexity of figuring out whether that exists
in all the kernel versions you want to support.
Okay, so to conclude, first, if we zoom into this research issue, the first thing is we
showed that the kernel tracing can be bypassed reliably in many different ways.
So if you happen to use a tool that does similar things, you may want to check if your tool
is vulnerable to this, because we've only evaluated the open source solution.
We didn't have a open source solution.
any value any proprietary software.
The second thing to keep in mind is
the mitigation is complicated.
Depends on what form factors that you use
and also what kernel versions
that the software need to support.
So you want to double check that the mitigation
was actually implemented in the way
that you want it to be.
And then if I zoom out from this particular issue
a little bit, if we think about the bigger picture,
if your security team is able to correlate
different data sources and have a comprehensive view
about your environment, then it suddenly make
the evasion complexity much higher.
Last but not least, this is something that we do a lot
at Lacework, is that we understand,
you should understand that your environment
what is normal, what is intended to be wrong
in your environment.
If you know the baseline of your environment,
then even if the attacker is able to override
these arguments, they're basically being constrained
to the value that is fit into the normal expectation.
Okay, before we take the Q and A,
I just want to give a quick shout out
to the following folks.
So Joe helped with the kernel and security
discussion, a lot of my Lacework labs colleagues
give great feedback on this work.
Also I want to thank the Falco open source team
for a very collaborative disclosure process.
And also John Dixon for helping the presentation.
So before finally take the Q and A,
if you have any other questions about cloud security
in your environment, or if you want to advance
your career in cloud security, I'm very happy to talk to you
guys offline.
With that, we're ready for Q and A.
So anyone who want to ask a question,
there's a mic here, and Steve will help to walk to you guys.
Hi, can you hear me?
Yeah.
In some of your screen captures,
you had kind of your exploits from last year.
They were Phantom V1, these are Phantom V3.
What was Phantom V2?
Oh yeah, thank you, thank you for that.
So certainly there's too many attacks that we publish.
So Phantom V2, at the time we called it semantic confusion.
So it's also, for the folks who's not familiar with that,
that's also a bypass on the system called tracing.
The idea is this.
When you trace file system object,
for example the At Sea Shadow,
if it is an open ad, then the rule is basically
detecting whether the path is open ad,
but what if you create a link, that is a linking
to the file object, and then you open the link.
Then the tracing software will not be able to interpret it,
it, but the kernel will be able to interpret it. So there's a confusion between the
freezing software and the kernel. Alright, thank you everyone for coming.
Thank you.