Anyway, I'll speak today. Steve, let's give him a big welcome.
All right. My name is Stephen Eccles. Today, I will be presenting on S-Trace, which is a D-Trace on Windows re-implementation.
My name is Stephen Eccles. I'm a reverse engineer on Flare. I also do malware research and development.
I have experience with open source development, specifically a focus on hooking.
I'm the author of the open source library, Polyhook. I know multiple languages, and I have experience as a system architect and an embedded device software developer.
Outside of computers, I like to use Slackbox.
Electric mountain board, and I play a lot of video games.
So today, we're going to be covering this talk in three different sections.
We're going to start with what exactly is D-Trace and why is it interesting.
We're going to be covering how to use Microsoft's port, as well as some of the shortcomings of the system.
We're then going to be talking about the internals of D-Trace.
We're going to cover the architecture, how kernel registration works,
documenting some of these features, and then we're going to be talking about the internals of D-Trace.
Kernel tracing interfaces, and then how we can insert ourselves into the boot process that D-Trace follows.
And then covering some of the neater tricks that D-Trace uses when it interacts with the Windows kernel.
And then finally, I'm going to introduce S-Trace, my re-implementation.
I'm going to talk about some of the differences to Microsoft's port.
We're going to go on a tangent about how we can write drivers in Rust,
and then really follow that rabbit hole by adding a WebAssembly interpreter to the Windows kernel.
We're then going to talk about why I ditched Rust for C++ when it comes to drivers,
and then loading DLLs in the Windows kernel,
and finally, how we can do our own symbolicated call stack traces.
So, what exactly is D-Trace?
D-Trace is a dynamic tracing system originally developed by Sun Microsystems,
and then that was acquired by Oracle, which was then turned into the Open D-Trace effort
in 2016.
D-Trace allows users to write specific scripts, which handle different probe events,
and within these user scripts, arbitrary actions can be performed,
and the events can be logged or handled in any way that they want.
These user scripts execute inside of a VM within the D-Trace.sys driver,
so they execute entirely within kernel mode.
Multiple different types of probes,
are supported by this system.
STP type probes, which is going to be the topic of my talk exclusively,
I'm not going to cover any of the other probe types.
These are essentially system call hooks.
There are other probe types supported,
FPT, PID, and ETW.
FPT and PID rely on the Windows kernel hypervisor,
and I'll talk about why I'm not covering these on a later slide.
The ETW probe types, they could be extended.
My system could be extended to support these,
but I just did not perform that research.
On the right, you can see an example of what a D-Trace script actually looks like.
This example will simply print out which file is being opened
by this NT create file, or NT open file system call.
I want to point out some built-in routines.
The copy in routine is the way in this system that you access wider system memory.
The copy in routine is the way in this system that you access wider system memory.
Because this executes within a VM,
any time you do a memory access,
you need to call this built-in routine
to copy the memory into the VM and perform your operations.
I also want to point out UStack at the bottom.
This is a built-in routine within D-Trace,
which allows you to do a symbolicated call stack trace
from the kernel level down through and including user mode callers.
And then finally, the LKD built-in at the bottom in this return probe.
That will perform a live kernel dump.
I'm gonna be talking about the internals of all these routines
on a later slide and how they work
and why they're a bit interesting.
But I just want to point out
those three interesting built-in routines.
We're gonna cover some more advanced examples
of D-Trace scripts here.
And this is going to start covering some of the pain points
and why I did this research.
So to start, take a look at the left.
The self keyword is a bit interesting.
I'm gonna cover this on a later slide about why,
but essentially this is,
kernel thread local storage,
user scripts can save thread local data.
In this case, I'm using that to get around a issue
within the D-Trace architecture.
Essentially these entry and return probes,
they execute before and after the system service routine,
which means they can execute potentially
before the system routine has paged in any memory.
So within return probes,
if you want to access this backing memory,
you actually have to save a pointer
to the memory in your entry probe,
and then you can't access it
until your return probe in many cases.
On the right side,
we can see an example of trying to print out a enumeration,
the enumeration flags.
In general, the D-Trace language is quite limited.
So anytime you want to do anything somewhat complex,
your code ends up being quite verbose.
And I'll talk about some issues
with this scripting language in general.
So here's an example of one of the bigger issues
with this language, and that's the fact
that it doesn't have user-defined functions.
So if you want to do complicated things
or build utility routines or anything like that,
it's really quite a struggle.
The best way to get around that, that I found,
is essentially you have to invoke the C preprocessor
over top of these D scripts.
This ends up working out because the D language
is similar enough to C that the preprocessor can
handle it.
So here's a utility routine that will just copy in
the backing memory behind a Unicode string
so you can print out that Unicode string.
So you can see this is quite a simple operation,
but we're already using crazy hacks
like invoking the C preprocessor over these D scripts
to build a semblance of a working environment
that is usable for us.
So in summary, some shortcomings of the D-Trace language.
There's no user-defined functions at all.
There's no loops.
If you want to build any kind of
somewhat complicated control flow structure
like nested ifs or anything like that,
you will absolutely run into cases
where it miscompiles into invalid instruction sequences
and causes faults.
It's quite annoying.
There's also some, what I consider to be,
system design issues with this architecture.
You have extremely limited side effects
within these callback routines when probes fire.
There's a built-in called system,
which lets you execute arbitrary,
arbitrary commands similar to like cmd.exe.
Unfortunately, it only supports static strings.
So if you build a dynamic string like dump.exe
with the process name,
you can't pass that to this to invoke that.
So you can't really do a whole lot
other than logging in many cases.
There's issues with memory accessing due to paging,
like I mentioned earlier.
And then the D-Trace Microsoft port at least
doesn't have the preprocessor.
The Unix variant of D-Trace does have a preprocessor.
Microsoft support simply does not have that.
The biggest issue is why string support
and string support in general is extremely lacking.
You can't convert like a Y string
to a narrow string in this language.
You can't implement that yourself
because there are no user-defined loops
and other utility routines like looking for substrings
or any other somewhat complex string manipulation routines.
A lot of them just don't exist,
which is
sort of a bigger issue,
which is that the developer experience,
the tooling, the IDEs,
all of that is completely lacking.
You essentially write user scripts in notepad
and then you ship them off to the driver
and you check if it works that way.
So being frustrated with this process,
this is why I started this research
into how can I create my own version?
The kernel tracing interfaces were extremely powerful
and I thought the system was awesome,
but I just simply could not express the logic.
I wanted within the provided language.
So in this section,
we're gonna be covering the high-level architecture
of the DTrace system,
how it boots and how it talks to the kernel.
I'm gonna talk about some really neat tricks that it uses
or provides via those built-ins
and then I'm gonna document these tracing interfaces
so that we can build our own implementation.
So the architecture of DTrace.
It originally started out with a kernel extension driver
called trace-extent.sys.
This talked directly with the Windows code.
The kernel to provide support
for all these different probe types like PID, FPT, et cetera.
That then communicated hands-in-hand
with the DTrace.sys driver,
which contained this D language VM,
which executes these user scripts
to handle those probe events.
Those user scripts could then log or generate output,
which would be queued up within the driver
and then read back out by the user mode component,
DTrace.exe at a later time.
The DTrace.exe user mode component,
also acts like a symbol server for the driver.
So the driver contains in kernel memory like a symbol cache
and that's used or consulted
when the DTrace built-in use stack is executed.
That symbol cache exists so that any stack output
that needs to be symbolicated
can immediately be done so in kernel mode
before it goes back to user mode.
And this is much faster
than constantly querying a user mode component.
In the 20H1 update of Windows, this architecture,
was changed.
The traceExtent.sys kernel extension driver
was completely removed and that functionality
was subsumed by the DTrace.sys driver.
So in the modern architecture of DTrace,
DTrace.sys is itself a kernel extension
and it directly communicates with the NT kernel.
But all of the components of this system
are exactly the same as they used to be.
And if the wiring diagram on MSDN was updated in this way,
it would be exactly correct.
So we understand the boot process, or sorry,
we understand the architecture of DTrace.
Let's cover the boot process.
In order to understand this,
we have to actually start at the Windows bootloader,
winload.exe.
The first thing that it's going to do is
it's going to check if the boot flag DTrace is set or not,
which can be set by bcdedit.
If it is set, then it's going to add
the API schema set extension,
extms winntoschtrace to the system schema set map.
And I'll cover this on the next slide a little bit,
so if you're not familiar with this mechanism, don't fret.
After it has added this schema set entry
to the system schema set map,
winload will then eventually begin to load the NT kernel.
The NT kernel has an import for this API trace init sys
from this schema set entry.
During the import process,
this import will be eventually resolved
to be an import of DTrace.sys,
and DTrace.sys will be loaded.
And then this import trace in its system
will be invoked by the NT kernel
during phase one kernel initialization.
It's a bit interesting when this happens,
because it actually happens before patch guard initializes
during system boot.
And this is because DTrace fills in
some kernel callback tables,
which are patch guard protected.
So in order for this to work out,
it has to fill those out before patch guard is initialized.
The call chain for that is ke-init-system,
invokes ke-init-dynamic-trace-sys,
which then invokes trace-init-system.
So I mentioned this API schema set system.
What are those?
The API set mechanism is essentially,
you have these PEs, these API set PEs,
which are essentially just aliases
for what are called host or implementing PEs.
In the case of DTrace,
I've shown the sections right here.
This is actually the API set DLL
that comes with the DTrace installer.
Looking at the sections,
we see this section called dot API set.
This is a binary hash map,
which essentially just contains a map
that points these virtual names to the host names,
the implementing names.
And we'll see on the next slide,
a image of parsing this particular one.
For DTrace, it's actually a bit more advanced
than just like a normal API set DLL.
It's a special API schema set extension DLL.
These can be added by setting this particular registry path.
The name fields of these registry entries can be anything.
They just delineate like a new extension,
but the file name path of this entry
must point to one of these pseudo PEs.
And then during the boot process,
when one of these is considered this,
these aliases will be resolved to whatever host PE.
So in the case of DTrace,
ext ms-win-entos-trace is literally just replaced
with DTrace.sys for any kernel imports.
So here's a zero one zero editor template
running over this dot API set binary section.
There's public research on this binary format.
So thank you to the people who have documented that.
It was very helpful in my research.
And this script can be found online.
Looking at this, we can see that DTrace.sys
simply points to ext ms-win-entos-trace.
And because this is a API set extension,
this actually ends up being flipped
from how you might parse
other API set DLLs in a more conventional API set DLL.
This would be the other way around.
The ext ms-win-entos-trace would point at DTrace.sys.
So you read this like DTrace.sys
implements ext ms-win-entos-trace's interface.
So we need to talk about
the signing requirements of the system.
So DTrace.sys, even though it is
a dynamic schema set extension,
it is considered to be effectively part of the Windows
kernel.
So it's under the same signing requirements
as any other Windows kernel component.
It must be signed by a root Microsoft code signing
certificate.
The schema set DLL itself is also under those same signing
requirements.
So you might immediately have the question of, OK,
if we're building a re-implementation,
how the heck are we going to pass these signing
requirements?
And it's a two-step process.
First off, we can reuse the Microsoft-provided API set DLL.
The only requirement from that is,
is that our driver must also be named DTrace.sys.
But this is fine, because it's just a name.
The content could be anything.
The more unfortunate thing is that we do
have to disable driver signature enforcement at boot time
in order to pass this signing requirement when the kernel goes
to import DTrace.sys.
And this is due to the architecture
of this kernel extension API schema set system.
It is not enough to just sign this yourself
with a valid code signing certificate or enable kernel test
signing or even more advanced features
like custom kernel signers, which
lets you load drivers that are self-signed against the UEFI
platform root key.
These will not work.
It must really be signed with a Microsoft code signing
certificate.
And to anyone from Microsoft who might listen to this talk,
please consider changing this architecture.
It would be very nice if kernel extensions, or at least
the DTrace kernel extension, could be loaded or signed
by non-Microsoft parties.
OK.
The kernel interfaces for this are fantastic,
but this requirement is very limiting.
In particular, if the custom kernel signers feature
could be extended so that kernel extensions self-signed
against the platform UEFI root key could be allowed,
this would be great.
In the modern time of secure boot and things like this,
giving back system ownership to the users
would be quite fantastic.
And it shouldn't really violate any of the security organizations
or the architecture of the system,
because you can replace the platform UEFI root key
with your own, which verifies system ownership.
So the loading process of DTrace.
We understand that now.
The system is going to call TraceInitSystem.
We've passed all the signing checks.
What is DTrace going to do?
And how is it going to actually register and fill
out these callback tables?
When TraceInitSystem is invoked, the first thing
that's going to happen is it's going to fill out the table
KI dynamic tracing.
It's going to trace call outs with pointers
to handle each one of these different types of probe events.
After that, it's going to pass a pointer
to this special memory access routine, Trace Access Memory,
to the NT kernel.
The NT kernel is then going to record the start
and the end boundaries of this routine
by looking up that function's x64 unwind table entry.
And I'll talk about why later.
After this, the NT kernel then passes a pointer
to KI dynamic trace context to DTrace.set.
This is a bit interesting because this structure
gives offsets into the K thread for different fields
for different versions of Windows.
And the DTrace.sys driver uses these offsets
to access these fields to perform different kinds
of tracing events.
It also provides pointers to different APIs
to set and remove these trace points.
I will cover at least one or two of these offsets
and how they're used by the driver on a later slide.
Okay, so we've loaded, we've passed signing,
we've registered with the kernel.
What can we do with these probes
once we are fully initialized with the system?
So my talk's about system call probes,
so let's talk about those.
When you do a transition from user mode to kernel mode,
that is handled by the kernel routine KISystemCall64.
This is executed for all user mode system calls
and also for ZW style kernel calls,
which go through KISystemCall internal.
Within KISystemCall64, there is special code
to handle this dynamic tracing system.
It works by first checking if dynamic tracing is enabled.
And if it is, then it will take a snapshot of the registers
and the stack slots that contain arguments.
It will then do a lookup into a tracing tree
and check if there is a entry or an exit probe
registered for the current executing system service routine.
If there is an entry or an exit probe registered,
then that will be invoked either before or directly
after the system service routine is executed.
And these are inline, so if you delay execution
within your entry probe, the system service routine
will not execute until you return from that.
The KITrackSystemCallEntry and KITrackSystemCallExit
are the routines that are invoked for that,
which then dispatch to SDP-style routines.
And these SDP-style routines are what is contained
within the KIDynamicTraceCallouts structure,
which is why that registration process happens at all.
These SDP routines are in that patchcard-protected table,
so if you try to patch these without following
that registration process, you will blue screen.
You might have the question of, okay,
if we have a single routine handling entry and exit
for all of our different system service routines,
how do we know which system service routine
is currently being dispatched with that routine?
And the answer to that is, when you set a trace point
with KESetSystemServiceCallback, you create an association
between the system service name and a thing called a probe ID.
The probe ID is just an arbitrary magic value,
which is essentially just like a handle
for this tracing system, and you as the one
who set a tracing probe have to remember the association
between that name and that probe type, but this probe ID gets
passed to your callback in entry and exit,
and you can resolve which system service
is executing that way.
Okay, so we're gonna cover some neat tricks
that DTrace uses when it talks to the Windows kernel
at this point.
I mentioned that trace access memory routine
that DTrace passes to the kernel.
This is a safe way to read potentially arbitrarily
inaccessible or paged out memory while you're at
a very high RQL without blue screening the system.
This is a whiteboard notification that the Microsoft team
added to the MMAccessFault handler.
When during the registration process when they,
when they pass this pointer to this memory access routine,
the start and the end bounds are recorded,
and then if a memory access fault occurs later,
the system will try to determine the caller.
The caller is determined by calling this
KIIsTraceAccessMemory function, which will check
if dynamic tracing is enabled.
If it is enabled then it will say to the cluster,
It will check if the faulting address is within the bounds of this special memory access function,
and also that the IRQL is set to 15.
The 15 is because Dtrace itself, these user scripts, they execute at this IRQL.
This is part of the VM.
When they enter the VM, they set it really high, and then when they exit, they set it
back.
I don't know why they do this.
It's not a requirement of the kernel interface, but it is a requirement of this particular
access fault system.
If the caller is determined to be Dtrace, then MM access fault will instead just return
a status access violation code rather than bug checking your system.
So if you've ever tried to do system call hooking yourself manually, you can appreciate
how useful it is to have a built-in function that can sort of like safely do this for you.
I pointed out the self keyword earlier in the Dtrace scripts.
This is a proper implementation of the concept of kernel thread local storage.
In order to support this, Microsoft added a field called tracing private data to the
Kthread structure.
This is the width of one pointer.
It's currently set as an array, but it only has one entry.
And the KI dynamic trace context structure that the kernel passes to the Dtrace driver
actually gives the size of this array as well as the offset to it within the Kthread structure.
So you can always know where it is on all versions of modern Windows, even though it
is like technically undocumented.
Thank you.
And you can set this to just be an arbitrary data value.
And it's only used by the Dtrace system.
So if you don't have Dtrace installed, this field should not be filled out.
One of the other neat tricks that Dtrace has is the ability to invoke the same APIs
that your kernel debugger will use to generate a system dump.
It will use the live kernel dump feature of Windows, which uses the undocumented API debug
or capture live kernel dump with this particular type definition.
This generates system dumps without having a system debugger attached.
And these are standard.dump files that can be loaded in WinDebug.
And the arguments, bug check code, P1, P2, P3, are what ends up being shown within the
dump if you load it in WinDebug.
This generates dumps in the C colon Windows live kernel reports folder, and the component
name is the subfolder path within this folder.
Microsoft technically documented this.
With the Dtrace code.
Dtrace system.
They documented the LKD built-in routine.
But that built-in routine just directly calls this undocumented routine.
So all of the registry arguments and different settings behind the scenes that control how
this API works are now technically documented as an extension.
My personal favorite trick that Dtrace does is its ability to generate these call stack
traces.
I find this interesting because it works a bit differently than some of your regular
tools, such as, like, Procmon, how they generate their call stack traces.
This is the built-in routine use stack in Dtrace.
Procmon is, for example, going to do something like call the user API, trace walk 64, and
then use, like, the die SDKs to do symbolication.
And then with the help of, like, a kernel driver, it will get the function frame of
the kernel thread.
And it will basically do this call stack trace from user mode is the important part.
The call stack walk is done from a user mode component using user mode APIs.
The Dtrace system doesn't have that luxury.
It is a driver.
It scripts execute within kernel mode.
So it can't call any user mode APIs.
So it's able to somehow initialize a call stack walk from kernel mode that goes down
through and including user mode APIs.
So I had the question of, okay, how is it doing this?
There is a kernel routine called RTL capture stack backtrace.
Which can do.
It can do a kernel level stack walk like this.
But it doesn't traverse the kernel to user mode boundary.
Yet Dtraces does.
And the way that they actually do this is they completely implement unwinding and stack
walking within their driver.
And this essentially operates by finding the current executing function frame.
And then they locate the module that that's within.
And then within that module, they find the image directory entry exception data directory.
That is a series.
Of runtime function entries.
There's one per every function in the binary.
If there's online info for it.
And then within that structure, there is a field unwind data.
Which effectively specifies how do I unwind this current function.
That unwind data is a series of unwind op codes.
Which describe memory effects and different CPU effects that the current function does.
And if you unwind these, if you execute them on like a virtual stack.
You can essentially roll back the stack of this function.
And then you can eventually read the return address off the top of this virtual stack.
So an example of that is this unwind op code push nonvolatile.
This specifies that a register was pushed to the stack.
And if you're executing this on your virtual stack, you can just do RSP plus 8 to undo
this memory effect.
So Dtrace completely implements this unwind op code execution system and everything like
that.
We've done a version of this ourselves in our own implementation.
Dtrace is limited because they executed such a hierarchy well.
So they had to do this entirely custom within their driver.
But quite a cool thing.
Okay.
So we understand the load process.
We understand how to pass these signing requirements.
We understand the interfaces.
And we understand some of the cooler tricks that Dtrace is using to talk to the kernel
and do some of these built in things.
How can we write our own tracing system?
So I'm going to cover the differences between strace and Dtrace.
And then I'm going to go on a tangent about Rust and WebAssembly.
And then I'm going to cover why I rewrote this whole thing in C++ after I did that.
And then how we can do our own symbolically call stack traces.
So strace versus Dtrace.
The biggest difference is that strace executes ‑‑ I don't touch the IRQL level.
Dtrace raises the IRQL to 15 artificially as far as I can tell when it enters this VM.
This could potentially be related to lockup.
Lock ordering or maybe the fact that some of the other probe types use hypervisor features.
I'm just simply not sure.
If anyone knows, I would love to know why.
But my system doesn't touch the IRQL.
So whenever your probe callbacks execute, the IRQL is just whatever it is.
This is usually low level when you're transitioning from user mode to kernel mode.
I'm sure there's cases where it's something else, especially if you're tracing these like
CW driver routines.
This has some very large benefits.
Such as.
Accessing memory is a lot simpler.
If you're accessing page.memory, you can just page it in.
You can also do arbitrary side effects like calling different kernel APIs because we
are at a somewhat regular IRQL.
Plugins within my system are written in moderate C++17 rather than this esoteric D language.
The symbolication, there's also some differences with how I symbolicate things.
Dtrace has this kernel symbol cache.
I don't do anything like that.
I just have a separate tool that runs over the log output.
At the end.
And then my system only covers system call probes.
I don't cover FPT, PID, or ETW probes.
And the main reason for this is because of the signing requirements of the kernel extension
system.
Because it requires a Microsoft code signing certificate, you have to disable driver signature
enforcement.
If you do that, the Windows hypervisor and like all these secure boot features, they
are not compatible with turning off driver signature enforcement.
Which these FPT and PID probes, they use the hypervisor.
So.
I've been researching how those work, if I can ever implement those.
So if Microsoft were to change the signing requirements of the system, implementing these
probe types ourselves would become feasible.
ETW probes are possible, I just didn't do that research.
Okay.
So when I was designing this system, I came at it from an architectural stance.
I essentially wanted to replace this scripting system and keep the rest of the architecture
as similar to the Microsoft one as I could.
But I wasn't exactly sure how to do that.
I followed an iterative process.
I essentially rewrote the system two or three times.
I started with Rust, then I went to C++.
And I'm going to cover this process because I found this quite a learning experience.
The biggest thing was I wanted these user scripts to be user provided from user mode,
shipped off to the driver so that someone didn't have to constantly rebuild a driver
and reboot their system all the time.
I wanted these scripts to be very fast because they're executed for every system call. And,
I wanted it to be a somewhat familiar language, something not esoteric like the D language
was.
Ideally, I wanted it to be the system to be usable for both very experienced kernel
developers as well as people with maybe not so much kernel experience or maybe not even
a ton of development experience at all.
So during this process, I eventually landed on WebAssembly would be a good start.
And I'll cover that in a second.
So WebAssembly is not just for developers.
It's for browsers.
You might think it is.
You might think it's terrible to put it in the Windows kernel.
You might be cringing on the inside.
But it's just a bad enough idea that it almost makes some sense.
The core idea of it is Dtrace has this VM in the kernel already.
The idea of having an isolated runtime environment is something the browser technology has already
solved.
So I could hopefully, the idea was, reuse the WebAssembly language to implement it in
this architecture myself.
And we can compile any language to WebAssembly so that kind of automatically solves the issue
of having an esoteric language.
If you like JavaScript, if you like C, if you like Rust, you could just write it in
whatever and then compile it to WebAssembly.
There was the issue of, okay, this is a browser technology, so a lot of these interpreters
and runtime environments, they expect that you're executing in user mode.
In this case, I ended up finding that the crypto people, thankfully, they did this very
nice thing.
They implemented this interpreter called WASME, which was used by the Parity Ethereum group
to execute their contracts.
And this is a no-stood Rust library that I was able to actually embed within the kernel
within my driver.
And we could essentially load arbitrary user scripts and execute them within the kernel
this way.
I have proof of concept of this.
This does work.
All this code will be available after the talk, and I'll give you a link to that.
There were some issues, though, that eventually made this architecture infeasible.
And I rewrote it into C++.
The biggest was that Rust itself with this library, I was getting extremely deep kernel
stacks, call stacks, so this was not feasible.
And then the library in general was very slow.
It was an interpreter-based system.
I'm of the opinion now that you probably need, like, a WebAssembly JIT for this to be feasible.
And there are no easily embeddable WebAssembly JITs.
Very quickly, I'm running a long time, so I'm going to just skim over this one.
Writing drivers for the Rust NT kernel is very difficult.
There are a couple of open source blogs on how to do this, and I'm very appreciative
for them.
I've essentially just followed those.
But the biggest issue is you have a very esoteric tool chain.
You need a very complicated cargo config.
You need a complicated build file.
The compiler itself has a lot of issues.
So I was passing flags that said always generate unwanted information, which is required for
this trace access memory like ABI.
But...
But the compiler would seemingly just ignore them and sometimes generate it and sometimes
not, which is obviously a huge issue.
Additionally, I turned off things like exception handling and all these different security
features like check stack because they're not easily supported in the kernel mode.
But the compiler was still generating these linker artifacts, and my builds were failing
because it couldn't find the implementations for these, and yeah.
I was able to get around this by just stubbing these out with functions that did nothing,
but I shouldn't have to do this at all.
It should respect my build flags and work.
And because this build chain was so complicated, I was getting a handful of internal compiler
errors and in general this was just a terrible process.
Really the biggest issue, however, was Microsoft doesn't have any Rust projections for any
kernel APIs.
So if you want to do this, every single API, every single type, every single everything
has to be manually converted by hand to Rust.
Which is an issue because the Rust language itself doesn't support a lot of language features
that...
So you have to take a lot of care for the layout, the alignment of all these fields,
and it ends up being tedious and you get it wrong a lot.
And you also don't get the benefits of the borrow checker a lot, which is like one of
the core benefits of Rust.
You end up having to wrap all these API calls in unsafe everywhere anyway.
So this whole process is kind of for naught.
I'm going to skip this.
Essentially, I tried to rewrite this driver in C++, but then embed a WebAssembly interpreter
in C, but I could not find a good C WebAssembly interpreter.
So this system was ‑‑ I essentially deemed this infeasible and I moved on to a different
scripting language.
The final scripting language that I did choose was essentially just shipping what are effectively
normal DLLs to the NT kernel and then manually mapping them in.
This ends up working great.
I'm going to skip this.
It's essentially a totally normal build chain.
Windows supports this concept of having dynamic blobs.
It calls them DLLs, and they tell you what code needs to be relocated where, and you
can reuse the IAT and the EAT for kernel APIs.
The only real thing here is you have to disable the C runtime, you have to disable any security
check mechanisms that rely on user mode functionality.
But once you do this, you can essentially just write a kernel manual mapper that's going
to map these DLLs in the same way you would for a user mode component.
The MSVC compiler has one little quirk where when you disable the CRT like this, if you
want to implement your own, like, mem copy or string routines, it will detect, like,
your version of my mem set or something, and it will replace that with an intrinsic.
Because you've disabled the CRT, though, that intrinsic implementation is not going
to be able to be linked in, so your build is going to fail.
In order to get around that, you have to actually define your own intrinsic.
It's not enough to just, like, stub out your own routines.
Here's an example of what one of those kernel DLLs ends up looking like.
It's just a normal DLL, but with kernel imports.
The way that this works for drivers when you're building these drivers in MSVC is the DDK
has an extension for Visual Studio which is going to inject NTOS kernel.lib into the MSVC
linker path.
And then when it comes across an import like this, the linker is going to consult NTOS
kernel.lib, and it will add your IIT entries like a normal build would.
However, we're building these DLLs, which are a bit weird, so this DDK extension is
not going to be activated for this project type.
So it's not going to inject NTOS kernel.lib into our linker path.
So it's all these kernel APIs are not going to automatically have IIT stubs generated.
We can very simply get around that.
We found this, however, by just manually editing our linker path to include a copy
of NTOS kernel.lib.
This is not hard to do at all.
Once we do do this, we can just specify our imports and exports like we normally would
in C, and then we can generate these DLLs that have kernel imports and also exports
for our plug-ins to handle these different probe types.
Excuse me.
Our DLL manual mapper will then take care.
It's going to walk the output.
We're going to take care of the IIT and resolve these imports by just loading or calling the
get system routine address function to resolve these APIs and then filling in the IIT like
we normally would for any kind of manual mapper.
So what does an S-trace plug-in actually look like once we've gone through this whole process
of generating these custom DLLs, writing this driver in C++ that's going to have this DLL-based
manual mapper built in?
The way that this works—.
is essentially you create an enumeration,
you create the association between your different system calls,
and you give a numeric value to remember that system call type by.
So in this case, I'm going to call set information file the value zero.
You can then set and unset probes.
This is done in what I call the STP initialize
and STP de-initialize functions.
These are functions that I've created.
I'm just following the STP naming scheme
that the Windows kernel is using for these.
Essentially what you do is you just pass the string
to the system call name without the prefix ZW or NT.
So set information file.
You get a Boolean value to specify,
are you setting a trace point for entry or exit?
In this case, I'm setting it just for entry, so true.
And then I'm passing my numeric value
to remember this association by.
So I'm saying set information file is the value zero.
And then similarly, I have a unset callback for the STP de-initialize.
Under the hood, this is calling those KE set system trace point,
I believe it was called.
It might be callback.
And similarly, KE unset system callback.
These are the routines from the KI dynamic trace callouts
or KI dynamic trace context.
These are wrapper routines that I've just created
because it makes it a bit simpler.
Once you do that, you will eventually get a probe type
fired by the kernel.
And then you're going to get a probe type fired by the kernel.
And then your system call routine executes.
So in this case, let's say set information file
was invoked by user mode executable.
That's going to go through KI dynamic,
or KI, whatever the system call handler function is called.
That's going to hit this dynamic trace code.
And it's going to find that you have an entry probe set,
which is then going to fire the STP callback entry.
My driver will then handle that,
and it will dispatch this to this X,
export, which will be within this DLL, and within this DLL, you can handle this event
however you like.
You can do logging.
You can do arbitrary kernel side effects, but like calling any kind of APIs at whatever
RQL level you're at, and you're past this structure called the machine state, which
is going to have all those registers and arguments that you can read out.
You should be able to modify these arguments.
I haven't tested this, so I'm not 100% sure if you're actually able to manipulate the
arguments or not.
I believe it should work based on how I read the code, but that is worth testing.
You cannot manipulate the return value, and you can't cancel the system routine.
So the system routine is always going to execute, and it will always return whatever the normal
return value would be for the execution of that function.
So this is an example of me just logging.
The fact that...
There's about to be a delete of this file.
The file disposition information is going to be set to delete file.
So I'm going to log the fact that some process is about to delete this file.
I'm then going to look up the file path that is about to be deleted, and then I'm going
to copy that file to a backup folder.
And then I'm going to, at the end, generate a call stack trace using my own implementation
of what Dtrace was doing with its complicated call stack trace, which I'll cover on the
next slide of how we're doing that.
But this is just a simple way to record all of the deleted files system-wide for all processes,
and then back them up to a backup folder, and then generate a stack trace to see which
processes were deleting them.
So I mentioned we're doing this call stack trace.
This is our own implementation of something similar to what Dtrace is doing.
But we have the added benefit of we are executing at a low IRQL.
We're not at 15 like Dtrace is.
We're at something...
Probably that's, like, low level.
So we have the added benefit of being able to call some built-ins like RTL walk frame
chain with this flag RTL walk user mode stack, which is going to tell it to actually traverse
that current user mode boundary and continue unwinding the modules through user mode.
This is awesome because it actually is going to execute those unwind op codes the same
way that the Dtrace driver is doing it itself, but we don't have to implement any of that
unwind op code.
We can do this stuff ourselves.
That is documented for the most part, but that is a complicated process, and I didn't
want to have to do that.
So once we call this API with this flag, we get a list of return addresses that include
the user mode return addresses for the current process that we're in the context of.
We can then turn these return addresses back into module and offset pairs by reimplementing,
looping all the kernel modules.
And then also looping all the...
The DLLs that are loaded for the current process context.
We can just subtract the base address of every one of those modules to get offset pairs.
I then created a tool in Rust called PDB Resim.
This is an asynchronous and multithreaded log symbolicator that does not use any Microsoft
Symbol APIs at all.
It is only using other Rust libraries that also don't use Microsoft Symbol APIs.
This is a full implementation of essentially what WinDebug is doing.
When it's loading symbols for a module.
This works by...
We can take these module offset pairs and we can find the PDB file name, GUID, and age
from the debug directory data entry.
We can fetch the PDB by building the symbol download URL like this.
We can then cache those to disk in the same manner that WinDebug is doing.
It's essentially just the PDB GUID plus the age concatenated together and then uppercase.
We can then parse this PDB.
Using a Rust PDB parser library and then we can build a PC to line cache using the
data structure of a range map which is just essentially a binary tree but for key types
it has a minimum virtual address and a maximum virtual address.
Using this we can build a very efficient nearest symbol lookup table and using that lookup
table we can then enumerate all these module plus offset pairs to resolve our log entries
to their symbols.
We can then demangle those symbols and we can fully symbolicate our log output.
From this driver and we have fully reimplemented our own symbolically call stack traces without
using for the most part Microsoft APIs other than this RTL walk frame chain to do that
on Y.code execution.
This PDB resim tool, I'll be open sourcing this plus all the s trace driver and DLL code.
This also supports caching folders or downloading the PDBs of an entire folder and other things
like this.
Because it's asynchronous and multithreaded, I can cache the C colon Windows system 32
directory in about one minute and get all the PDBs and do that extremely fast and this
doesn't use any of the Microsoft APIs.
You could port this to work on something like Linux if you wish.
There's just some hard coded Windows paths within the tool but that could be changed
really easily.
So a quick demo.
Here's an example of me loading a plug-in DLL.
This is a log all system calls DLL.
This is going to register an entry and exit probe for every single system call type supported.
And then executing a DLL via run DLL 32 and I'm calling one of its exports.
This has generated the file s trace.txt on the right.
I've unloaded the plug-in as well.
We can see all the system calls that were executed here.
I'm just going to search for the DLL name that was loaded by run DLL 32.
We can see that it does show up within the log output.
And below all of these system calls plus argument lines, we do see our call stack trace.
However that call stack trace was not symbolicated.
It was just a series of module plus offset pairs.
So I'm going to pass this log file to PDB resim and I'm going to tell it to generate
the output file sims.txt.
Sims PDB resim is going to take all those module names, it's going to locate the PDB
for them.
I've already cached them.
On disk here.
So we're not doing any downloading at the moment.
This is caching the same way window bug does.
It's then going to resolve these to its nearest symbol and I'm demangling those as well.
So in this case, we can see we include things like run DLL 32.
We can see the win main is the root caller of all this eventually.
But we can see our full call stack trace is executed.
This call stack trace is performed from the kernel driver.
So there are no like user mode call stack tracing things.
It's going on here.
This is a fully symbolicated kernel call stack trace.
One more demo.
Here is that file delete record plug-in that I showed earlier in code.
I'm going to load this DLL as well.
This is going to be mapped into the kernel.
We're going to set that file disposition information hook like we showed earlier.
I'm creating a random file here.
I'm going to fill it with some content and then I'm going to delete this.
If you look in the top right, that file as I deleted it was just moved to C colon slash
deleted.
This is not my recycle bin directory.
That delete was captured by this plug-in at the kernel mode level.
The back of that file was created and it was ‑‑ the copy was put in that folder.
This entire process, the S trace log file was still generated in the lower right.
So we can see the callers of this delete.
I'm going to unload this plug‑in.
I'm going to take a look at this trace of these APIs.
We can see the two file deletes right there.
And then ‑‑ I'm going to take a look at this trace of these APIs.
I'm going to also symbolicate this.
We can see the call stack traces, but let's say we're curious which Windows component
is doing this file delete.
Let's wait for this to symbolicate.
Let's open up the symbolicated file.
We can see that this component called Windows.storage is what is doing that delete.
And you can record all file deletes system‑wide this way for any process.
And you can ‑‑ this system, you can target specific processes, too.
By name, by PID, by whatever.
There's a mechanism within these plug‑in DLLs to do that.
This code, PDB resim, the C++ implementation, these two example plug‑in DLLs, the proof
of concept, WebAssembly interpreter in the kernel, all this is open source on the Mandiant
GitHub page, github.com slash Mandiant slash S trace.
You can find everything that I've presented here open source under the MIT license.
If you would like to create plug‑ins ‑‑ if you would like to create plug‑ins, I'll
be very interested in seeing any interesting plug‑ins that you create using this system.
And Microsoft, please consider changing the signing requirements of this kernel extension
interface so that we can implement some of the cooler probe types.
Thank you.