And welcome back, everybody, to DEF CON 28 Safe Mode, Blue Team Village.
We're going to start our next set of speakers. We've got a great topic coming up today.
The title of this talk is Building BlueSpawn, an open-source active defense and EDR software.
Jake Smith and Jack McDowell will be presenting. As always, please direct all questions to the
Tech Talks Track 1 channel, or you can connect to their Discord server as well and send messages to
them directly. I'll have that in there in one second. And on that note, over to you, Jake and
Jack. All right. Thank you. So how's everyone doing today? Just a few quick notes before we
get into the presentation. We definitely want to take a moment to say thank you to all the
volunteers and organizers of the Blue Team Village. Putting this event on in a normal
year is a lot of work, and I can only imagine what they've done to make this happen. So a huge
thank you to them. So today, Jack and I are going to be talking about an open-source project we've
been developing for about a year now. It's called BlueSpawn, and you'll notice the emphasis on
active defense here, so we'll get into more about that later. But first, a little bit of background
about the two of us. We went to the University of Virginia. Both of us got our start in InfoSec doing
really all kinds of cyber competitions, CTFs, and whatnot, and that's been a fantastic way to
kind of get into security. And these competitions we'll talk about more are some of the reasons why
we created BlueSpawn and all. So an important caveat, too, before we get into the presentation.
When we say we this afternoon, we're referring to all six of us on the slide. It turns out that
developing a large open-source project with already like 20,000 lines of C++ is a huge,
huge undertaking. So it's really a team effort that we've been doing to create this.
So, and also before we get into talking about BlueSpawn specifically, we're going to do a brief
overview of some of the available tools out there and kind of where they shine, where they have some
shortfalls, and more. And really, this is just all the tools that we're going to mention do a
fantastic job, which is part of the reason why we want to talk about them as well. So first,
we'll start with your Endpoint Detection and Response, or EDR, tools. That's your CrowdStrikes,
your Carbon Black, your Defender ATP, Sentinel-1, and all. They obviously used to, like 10, 15 years
ago, AVs were pretty simple. They look for bad hashes and whatnot, and now kind of that's, they've
increased, and they've gotten really, really good. So these commercial tools, and they're a bit
expensive in some ways, but they really raise the bar for finding the low-hanging fruit, especially
in some, even some of the more advanced malware. They're obviously not all created equal, but
one of the things we noticed when we started developing BlueSpawn is that
these defensive security solutions are pretty opaque overall. There's obviously a lot of
offensive tools out there, but it's really hard to get your hands on and kind of understand how
these modern AV and EDRs work, which leads us into a bit about some of the free tools that
are out there. There's not a whole lot of great ones, but definitely Sysinternals is probably
one of the best. If you haven't used the Sysmon, if you haven't used much just beyond like Sysmon
to improve your logging, highly recommend taking a look at like Autoruns and Process Explorer and
Procmon. Just, they do a great job of finding malware on the system. They're more geared though
toward the system analysis side, so they don't really, they're not going to say like, hey, this
is malware. They might show like a different color if this is like encoded or something, but they're
not going to point out and like say this is MITRE attack technique 1547, right, for example.
Next is Process Hacker. Process Hacker is another really great tool. There's not much automation or
security focus, but it's really good at just analyzing. It's like Task Manager on steroids,
really, and you can even look at like a specific region in memory. So like if you know that
one process was injected, for example, then you could go and analyze each bit of the memory
yourself. And then on the like purely defensive side, we've definitely found PE-Civ to be one of
the best free and open source tools out there by far. The idea is pretty simple behind the tool.
It's basically look at all the processes running on the system, and then it can find hooks, process
hollowing, process injection, shellcode, all that kind of good stuff. The only downside,
if it's really a downside, is that it's limited to scanning processes. So it's not going to find
like other malware, say, like that's hiding in the registry on a system.
And finally, in this kind of available tool section is like the collections of YARA rules,
Sigma rules, Sysmon configs out there. Even just a few years ago, there just wasn't really a whole
lot out there, especially on the defensive side in way of signatures. But these projects are doing
a really good job of starting to share threat intel a bit more widely and can help you find
threats just on a particular system. The only thing, I guess, the caveat is like with hardening
scripts, for example, a lot of the stuff you'll find out there is very, I guess, more script-like
and not fully fleshed out. So that's one thing to be aware of, like they'll set conflicting values
or miss critical settings and whatnot. But so that, I mean, that's kind of the reasons why we
got into building BlueSpawn, which we'll talk about here more shortly. And one of the key
things we want to emphasize this afternoon is that is the ideas and principles on which we
built this tool. You're probably not going to go into work Monday and deploy BlueSpawn
in production. And I guess you could, but that's maybe not something you want to do quite yet
because the project's not quite that far along. But that said, you can use the information in
today's talk to conduct your own security research and begin kind of improving your own defenses too
and looking at how these kinds of solutions work. So with all things, all great ideas, I guess,
start in Slack and whatnot. So this was, I went into our archives, this was last year, like we
started with this really simple idea, like what if we could like scan one particular thing, like
look at GS registry run keys, like as a way to detect threats and whatnot on a system and like
an open source tool. And we didn't find a whole lot out there. So that kind of led us to creating
BlueSpawn. And as I said, we were students or we are students, so we compete in a lot of these
different cyber defense competitions and whatnot. Honestly, frankly, I tell some people that I just
do go to school for competitions and all. But basically, these cyber defense competitions are,
basically, you get a network and you're asked to secure it against active attackers, who
unfortunately for the students are industry red teams. So you can imagine how that kind of thing
goes. And the red team pretty much always wins. So in these competitions, though, we can't really,
we can't use commercial tools and we can't bring in scripts. So we have to rely on open source tools.
That's kind of another reason why we focus so heavily on the open source aspect. And so we
built BlueSpawn on one really particular focus is that we want to be able to stop threats as
quickly as possible. So I think that you should be able to run this tool and find as many, if not the
majority or more of all the threat, all of the malware that's running on your system as quickly
as possible. And so that is where the idea of active defense comes in.
And then the next is coverage. So when doing these cyber defense competitions, it's time
limited. You're stressed and whatnot. So you don't have the luxury of, say, being able to
look at everything very closely. And you can't pull up a process explorer and look at every
individual process to hunt for process injection. So our idea is that if you can get 60, 70, 80%
of the way there on finding the most common threats, and you know what your tool is going
to be looking for, you can spend more time on those other areas as well. And finally, open
source software. So this is a call to everyone out there. There's a lot of stuff on the offensive
side, but there's just not really as much on the defensive side. So definitely building more
open source blue team software, I think, is really important. And it's a great way for people to learn
about how these kinds of things work and how you might detect, say, one particular technique in the
attack matrix. So as time went on, from that simple idea that we posted a while ago in Slack,
we started moving BlueSpawn towards what you might term as calling EDRs type solution
today. So that's where it's doing continuously monitoring. And we're asking questions like,
what if we could just deploy BlueSpawn across the whole network like you might deploy like Falcon?
This is kind of the ideas that we're heading towards as we spend more and more time developing
this. And then also proactive defense. So it's not enough that you can detect malware and respond
to it, right? We also, one of the core focuses is the idea of being able to prevent these malicious
activities in the first place. So we've talked a lot about active defense so far, but what do we
really mean by it? And how does BlueSpawn approach the idea of finding anomalous or malicious
behavior? And that's kind of what I'll get into in this section. So typical antivirus or EDR
products are meant to kind of run in the background behind everything, only raising alerts and acting
when there's something definitely bad going on. For their use cases, this is a very good thing.
People don't want alerts about everything that may or may not be malicious. But when you've
been compromised, or even when you think you've been compromised, that might not be enough. This
is where our concept of active defense comes into play. Rather than sit in the background,
only rarely making noise, BlueSpawn is meant to be a tool that's actively used. The goal is
to find absolutely everything that could be malicious, not just the things it knows for sure
are malicious. In such an approach, there are drawbacks. In catching all the things, we're sure
to catch some things that aren't actually bad. But for an active defense situation, that's a
trade-off we want to make. So how does BlueSpawn go about approaching this? Originally, this worked
by just performing a scan of the system at a point in time to identify all the threats that
were currently on there. And this hunting involved the typical registry scans, file scans, or process
scans that an AV or EDR product would use. But again, BlueSpawn tried to be more in-depth, taking
more time to look at things, with more of a focus on finding anything that's not normal. And I think
what makes BlueSpawn different is we also factor event logs into our hunts. So correlating what
Windows events have occurred with what is currently on the system to identify any sort of
malicious behavior there. And we're trying to basically use that to bridge the gap between
incident response and what EDR products do. And as we move towards continuous monitoring,
we extended these hunts to be able to run as needed. And I'll talk more about how we did that
shortly. So as I mentioned on the previous slide, BlueSpawn is built pretty heavily around the idea
of what we call hunts. For each MITRE attack technique we support, which is basically a way
attackers do things if you're not familiar, we've created a hunt, which basically is a search for
any sort of evidence of malicious activity using technique. We'll get into more detail about how
specifically these hunts work in a bit, but for now, just know that they're really the starting
point for everything BlueSpawn does. Monitoring, as I mentioned a while back, is really just an
extension of the hunts we've already written. Each hunt creates triggers for when it should be rerun,
and when those triggers occur, it gets run again. Now, MITRE recently released sub-techniques,
which break down each technique attackers may use into subcategories. And BlueSpawn has just
moved to handle that by making each hunt responsible for all of the sub-techniques that fall under it.
So these hunts are really great for finding the starting points in malicious behavior.
We can find registry keys that run files around startup libraries that aren't meant to be
included at all, all sorts of fun stuff. But malware tends to have a number of different
components that it uses, or even if it doesn't, it exists in multiple different forms in multiple
different places. Maybe there's a registry key that points to a file, and then that file is
loaded into a process, and then that process is changing other registry keys, and those keys
point to files. And those files, well, you get the idea. To address this, BlueSpawn searches each
piece of evidence it finds in order to see if it can catch anything related to it. The result from
this is a network of evidence of malicious activity. In the next section, I'll go into
a bit more detail about how this actually gets built. Now that I've given a higher-level overview
of our methodology, I'm going to take a deeper dive into the different things BlueSpawn can do
with a focus on how it does them. And this section is going to get a little bit technical, but I'll
finish it with a more broad overview if you don't catch anything. So before I get into any of the
features, I want to preface with a disclaimer. BlueSpawn is still in alpha development. So far,
we've been mostly focusing on infrastructure, that is, building out the core of the features I'm
getting into. Everything I'm presenting is currently implemented, but it's not entirely
without its bugs. And the biggest example of what I'm talking about is our scanning
subsystem. I'm going to say things like, this is where BlueSpawn scans X, but the actual scans
performed are new and not necessarily heavily tested. At this point, I'd say a lot of the
features described here are better talked about as a framework than a fully fleshed-out product.
And none of this is to say BlueSpawn isn't effective. We've spent a long time building
our hunts and expanding our coverage, and it's proven to be pretty good at its job, as Jake will
showcase in his case studies section. And now that's out of the way, onto the features.
BlueSpawn has five major components, each of which we'll discuss in detail in this section.
There's Mitigate, which applies security settings, Hunt, which runs the hunts I was talking about
earlier, Scan, which determines how bad detections really are and finds associations, Monitor,
which hunts over time, and React, which handles responses to what HuntMode and ScanMode have
identified. So Mitigate mode sits somewhat apart from the rest of BlueSpawn without really
any integrations to the other modes, so I'll just talk about that first to get out of the way.
Mitigations applied by BlueSpawn are all mapped to either a Security Technical Implementation Guide,
or STIG, released by DISA, or the Miner Attack Mitigations Framework. When running Mitigate
mode, BlueSpawn can either audit the current state of the system, which just says this is what it's
currently set up as, or it can go ahead and apply the mitigations it finds aren't properly configured,
with user confirmation, of course. Now, as of now, our mitigation framework is very quite simple,
but we think it has a lot of potential as BlueSpawn continues to grow. We see it eventually
being configurable through some sort of configuration file, so mitigations can be
more uniformly and quickly applied across a network. We also see it mitigating services
and software installed on a system, rather than just the base operating system. And finally, our
biggest envisioned change is for Mitigation mode to serve as really an aid in performing a system
audit. So this would perform some of the same tasks as enumeration tools, such as
Seatbelt or JAWS, with a focus on finding and fixing things actually being vulnerable to being
exploited. Now, before I get into the weeds of how everything works, I'm going to throw some
definitions at you for how I refer to the ideas and concepts BlueSpawn uses. So I'm going to be
using the word detection a lot, and it's just that, detection. It could be something good, it could be
something bad, it's just something BlueSpawn identified as taking a closer look at. It could
be like a file, a registry key, a process, or something else entirely. Next, there's an association,
which is also just what it sounds like. It's a connection between two different detections, and
in BlueSpawn, we represent associations as having a weight, representing how closely they are
associated. Next thing is certainty, and this is the metric we use for determining the likelihood
that a detection is actually bad. There are three components to it. The two of them get combined, so
in practice, there are really only two. The first is what I call intrinsic certainty. This is the
certainty that comes from the detection itself. This would be what we're talking about if a
service was called Mimikatz, or if a file matched a Yara rule from Reterpreter, or something like that.
The next is contextual certainty. This comes from the context surrounding the detection.
In a lot of ways, this is very much like the intrinsic certainty, and in fact, BlueSpawn
doesn't actually differentiate between the two. They're really combined. And what I mean by
context is things like a file in System32 that's not assigned by Microsoft, and that's probably bad
just because they usually are. Or any value listed under an app in a DLL is probably bad just because
they aren't really used legitimately anymore. The last type of certainty I'm going to talk about
is associative certainty, and this is the certainty that comes from the associations of detection.
Say we've got a file and we found it loaded into a process. If the process is behaving maliciously
and the file appears malicious, the file should be treated with a higher certainty than if all
we had was a malicious-looking file. Now that I've got those definitions out of the way, I can move
on to how BlueSpawn actually works. The core of all detection management, and really the core of
BlueSpawn itself, is our detection register. This tracks everything that BlueSpawn's found.
Whenever a hunt or a scan or whatever finds anything that might be bad, it sends it to the
detection register. The register will record the detection and send back a reference to it. Now,
this reference may or may not be the detection that was recorded, and I'll explain why next
slide when I explain how the registration process actually works. The detection register has three
storage mechanisms. So whenever a detection gets registered, it gets queued for a scan and we track
all queued detection in one set and all scanned detections in another. That way we can just make
sure we're not redoing the same work. We also keep a record of all detections regardless of whether
or not they've been scanned. So now for how registration actually works. The first thing
the register actually does when a new detection is being added is it checks if the detection
already exists. If it does, they get merged together. And generally the only thing that
changes when this happens is their contextual certainties and their contexts get merged.
If this merge causes their overall certainty to exceed a certain threshold, it then gets logged
and its reactions get triggered. If the detection being registered was merged with the pre-existing
one, the reference that returned is actually to the pre-existing one. That way it's just keeping
track of the detection that we care about. Otherwise it's to the new detection. After all
this is handled, if the detection register needs to queue scans for the detection, it will. This
constitutes calculating the intrinsic certainty, identifying associated detections, and then if the
certainty is high enough after that, logging it and handling reactions. The associated detections
then get registered and the scans for those get queued and so on. So here I've got a diagram for
how the detection register lays things out. It looks kind of complicated, but I'll try to explain
it so it's maybe not quite as bad as it looks. So the register itself has three main storage
mechanisms. There's the queue of things that haven't been scanned yet, things that have been scanned,
and then everything overall. And you can see that all these storage mechanisms really just store
references to detections that are stored elsewhere. I've also broken down one detection, in this case
ID2, to show what kind of information detection actually holds. So it first has its ID, which is
just an ID, a unique identifier to refer to it. It has a context, which is the information
surrounding how it was made and how it was found and when that happened and all that. So that'd be
what time the detection occurred, what hunt found it, and when the first evidence of the detection
was found. There's also the data, and this stores information about what actually did it find. So in
this case, I've just decided for the purpose of an example, this detection is going to refer to a run
key and its value name. Its name persists and it's pointing to this file called bad.exe.
The next thing I have is the scan information, which stores information about how it was scanned.
That has the certainty, which combines the intrinsic and the contextual certainties,
and then the associative certainty. The last part of this is the associations, which
identifies which detections are associated with this one. In this case, detection ID1,
which I'm using to refer to as bad.exe, is associated with it. There's also a remediator,
and I'll get to that a bit later. So in the methodology section, I mentioned how hunts
represent the starting point for pretty much all the threat hunting that BlueSpawn does.
Each one covers one minor attack technique. Each hunt, we then break down into sub-techniques,
which can be run independently of each other. Within each sub-technique, that gets broken
down then into sub-sections, and one sub-section generally checks just one specific attack surface,
whether it's a folder, a file, an event ID, or a registry key, or something else. The idea is to
keep it as simple and small as possible. This design allows for much more targeted hunts if we
know what we're specifically we're looking for, and I'll get into why that matters when I talk
about how monitoring works. Most of the things we check get a quick scan, which just performs the
most elementary scans. It's a very quick test to determine whether or not something deserves more
detailed scans. If it does, we create a detection for it and register it with the detection register
and kick off the whole scanning thing I just finished talking about. Then the scans I discussed
get run on it and make a determination, find other related things, and so on. Here's an example of
what a hunt looks like. In this case, this is a T1547. The hunt itself really just runs all
sub-techniques underneath it, you can see here. And then here's the sub-technique number four,
which I picked out because it was pretty simple. The first part of it just compares
the winlogon values for the shell and username against known goods. If they don't match,
we create detection with a moderate contextual certainty, because anything that doesn't match,
chances are it's bad, even though the file itself might not be bad. The next sub-section
refers to the winlogon notify values. Any of these that we find get a detection created for it,
just because they aren't used very commonly. And we create that with a weak contextual certainty,
because again, if you find one, chances are it's not great. Now you'll notice during each of these,
I specify cursory when I initialize a sub-section. That comes from... when you run bluespawn,
you specify how closely it should be looking for things. There's cursory, which is a very quick
check. There's intensive, which is a very slow but careful check. And there's normal, which is
somewhere in between. By specifying cursory here, we indicate that bluespawn should run these
sub-sections whenever the intensity is greater or equal to the cursory.
So when we originally had our first working version of bluespawn, it was just a few hunts
we'd written and not too much else. Since then, everything we've done really has been taking what
we had and proving it. When it came time to start monitoring systems in real time, rather than just
performing a single scan, we realized we could reuse the hunts we had already written to do this.
Each hunt we created already defines a number... sorry, each hunt we created defines a number of
events that indicate something pertinent to the hunt has changed. For example, we might make an
event watching a folder for new files or old files being edited. When the event gets triggered,
hunts that care about that folder get rerun. Of course, there's some obvious inefficiencies in
this, which is a big reason why we divided hunts into sub-sections. Now, when hunts create events,
they specify which sub-section should get run for each event. Now, I'll admit monitoring is
unfortunately one of the areas that hasn't received as much attention as others. It's
certainly functional and it's effective at watching for things that hunts can catch,
but so far we don't have that much that acts based off of behavior alone that we observe while
monitoring. In the coming months, this is going to be one of our big focuses. Adding things like
event tracing for Windows or API monitoring should position bluespawn to be able to catch much more.
So, here's an example of how monitoring sets up its... or hunt sets up all of its monitoring
triggers. So, if you noticed my example hunt two slides back, each of these sub-techniques gets a
scope passed into it. Well, under the hood, what's happening is the sub-section macros I have
actually check the scope against that sub-section ID to determine whether or not that sub-section
should be run. So, whenever I create an event here, I specify which sub-sections are pertinent
to it. And yeah, so here we have registry events, file and file events, but we also have event log
events to watch for Windows events being created. So, the last component I'm going to talk about
here is the reaction framework. As the name would imply, this is responsible for taking care of
that BlueSpawn has determined are bad. Since BlueSpawn is designed around finding as much
as possible and is bound to have some false positives, most things here require some sort
of level of user confirmation before taking action. Now, obviously, when I delete a file,
it's no longer an active threat, but there's still a reason for BlueSpawn to track it.
To handle this error detection, each detection... sorry, to handle this error situation,
each detection has a flag where we mark it as stale or not. Whenever detection... sorry,
whenever reaction mitigates the detection, we mark it as stale, so that way other reactions don't go
and try to mitigate it their way. Currently, we have five reactions available, but there is
somewhat of a less defined six, and I'll get into that. BlueSpawn can delete files, quarantine files,
delete registry entries, and that can be values or keys, suspend processes, or carve out memory.
I'll talk about that in the next slide. The less defined reaction I just mentioned is for
special cases. Remember back to the WinLogon hunt where it was comparing some values against known
goods? It wouldn't be very good to just delete these values if they were malicious. They need
to be properly restored. And to handle things like that, when a detection is created, it can
be given a custom remediator. This remediator gets precedence over all other reactions if it's present.
Now, as we continue to develop BlueSpawn, we of course intend to continue adding reactions and
improving the ones we do have. Future reactions might include things like de-registering service
or blocking certain commands. So I promised that I'd get into what I meant by memory carving, so
here we are. The idea behind memory carving is that sometimes libraries or shellcode get shoved
into processes, and these processes still serve a valid purpose and need to remain active. In cases
like these where a restart isn't a good solution, we came up with a way to mitigate the effect of
whatever malicious library or shellcode is living inside a process. It's not perfect, and it can
cause problems, but it's better than nothing. This reaction works in five steps. First, we suspend
the process so that nothing we're doing causes problems with the way the process works.
First, what we do is we enumerate the threads in the process, searching for any thread whose stack
overlaps the memory section in question. And that basically is an indicator that the thread is
acting based off what that malicious memory section wants it to do. If or when we find this,
we kill any thread that does that. Since malware generally doesn't like to share a thread
with something benign, it shouldn't kill anything important to the process. Of course, if you got
things like APC injection going on, there's not too much we can do about that right now.
After that, we scan all memory pointers to the memory segment in question. If or when we find
them, we check the address in question. If it's data, we patch it with zero, so it can't read
anything important there. If it's code, we patch it with return instructions, so if it ever tries
to call there or jump there, it gets immediately bounced right back. Then we go through and patch
the entry point and all functions that are exported with return instructions, so anything trying to
run any function inside the memory gets messed up. This can cause problems if we end up patching any
function that requires the callee to clean up the stack, but without extensive work to figure out
how much of the stack needs to be cleaned up, there's not too much we can do. Once we're done
with all that, we resume the process and let things continue. We have tested this with a couple
different pieces of malware, including CobaltStrike, Beacons, and Returpreter, and it seems to work
without causing too many problems. Unless processes are storing function pointers in an abnormal
manner, it should be very difficult to get anything in this memory segment working again after what we
do occurs. Now, we are working on a few features to improve the stability of this. This includes
things like forcibly walking threads back instead of just killing them, or finding out how much the
stack needs to be cleaned up when patching returns, or properly unloading libraries as seen in Process
Hacker rather than just breaking them. Now, BlueSpawn would have been nearly impossible to build
from the ground up without any libraries or integration. As has been a major theme in our
presentation, the MITRE ATT&CK framework has been invaluable in providing guidance around what sort
of things we should go for. YARA and YARA rules written by the community have provided immense
value in being able to scan files and make determinations. The stakes around which we built
our mitigations make it very clear what we should prioritize and give us a framework to convey our
coverage. Finally, PESib is a great open source tool we use to scan any process for any sort of
fileless execution or any form of execution that's not properly mapped to a file.
In summary, BlueSpawn's abilities can be broken into three categories, prevention, protection,
and response. Under prevention, we have mitigate mode, which integrates STIGs and MITRE's ATT&CK
framework. Under detection, we've got monitor mode, which watches for changes to rerun hunts,
hunt mode, which is built around the MITRE ATT&CK framework, and then checks for these threats.
Scan mode then analyzes what was found in hunt mode to find associations and make determinations.
Finally, under response, we have our reactions, which aim to mitigate or remove a threat,
and then logging to report what was found.
All right, so that was a lot, but we wanted to really give you a deep insight into kind of how
some of these AV and EDR type programs, and obviously BlueSpawn in particular, work.
And so now we're going to move into talking about just like seeing the tool in action, right? So
we're going to walk through a case study with some kind of specific point in time screenshots
of BlueSpawn catching malware, and then also we're going to have a video of kind of the tool
in action. And I mentioned cyber defense competitions have been a really important
learning tool at the beginning, so first I want to emphasize that again. The largest is by CCDC,
and it's held in about 10 regions across the U.S. with qualifiers, regionals, and nationals,
and about 230 universities compete each year actually. So that's one way if you're looking
to volunteer to help in the community, highly recommend looking into a competition like that
or others. So these competitions are basically what's known as an inherit and defend competition.
The student teams work in a group of eight, and they're basically work as a fictional company. So
like they have a whole set of networks they have to defend, and typically that's on-prem. There's
cloud aspects, and that's usually done in like ESXi or AWS. So here you can see an example of a
network map from actually last year. So you have some on-prem workstations, there's like a on-prem
cloud if you will, and like ESXi, and then there's even like an AWS component. And you'll see
the stuff in there ranges from Windows Server 2008, so like really old stuff, to Windows Server
2019, and every flavor of Linux distro really. So that's they kind of emulate a modern
corporate environment the best they can. And there's of course stuff like Exchange, and web apps, and
databases, and all. And so basically there's two goals in these competitions. You have to keep your
services online, and then just like you would in a real company, and then you have to stop the
right team from really just totally destroying everything. And really frankly that ends up in
situations like this. So on the left hand side you see this is from the national competition this
year, and it's where a picture of the red team's Cobalt strike instance, where you can see they
just have pwned dozens of machines against this poor team. And they also do things like defacing
the websites, and RMRFing boxes, or overriding MBRs when they get too much access.
But really what that means is that in all reality is that these are a great testing ground
for defensive tools, right? So the red team spends all year writing malware for these competitions,
and are probably watching right now. So hi red team! But so their tool set is everything from
like Metasploit and Cobalt strike, what's pretty commonly used, to really custom malware that's
just written for the purposes of CCDC. So with that kind of backstop in mind, we're going to look
at some example screenshots of BlueSpawn. So here we just have the system. There's two Cobalt strike
beacons running. One is just an HTTP beacon, and then the other is an SMB beacon living in
explorer.exe. And so this is how BlueSpawn goes about finding it. It's looking at, when Jack
talked about process injection earlier, and the carved memory reaction, you can see BlueSpawn's
finding the malware in both explorer.exe, and then also in the PowerShell that's running.
And then it starts prompting the user like, hey we found this malicious activity, and all they did,
all the user did was just run BlueSpawn. And then it's offering to kill the threads, and once it
kills it, then we reach a situation like this, where those beacons basically just stop calling
back in. So that's an example of how BlueSpawn is responding to the threat. And here's another
classic threat. It's not used obviously as much now, but the sticky keys backdoor is where the
registry key is configured with command prompt as a debugger. So BlueSpawn's able to look for that,
and that's T1546 sub technique 8, if you're following along on the MITRE ATT&CK matrix,
so it knows to look for that, and then it goes and finds it, and it lets you remove
that option. Next, this was probably one of the coolest examples we saw at a cyber defense
competition that BlueSpawn caught earlier this year. Windows has these things called authentication
packages and notification packages, and these exist in like the LSA subsystem, which is used
for authentication and all. So BlueSpawn is able to map and is like, hey we found a registry key,
it understands that in the hunt, that this hunt refers to things on disk. So then it goes and
correlates that with the file associated with it. So we have two detections, one for the file
and one for the registry key that contains the badness. Next, here is a good old run key.
So there's a malicious BBS script that's configured to run in the registry,
and again BlueSpawn is able to correlate the two and be able to connect that to generate detections
and find both pieces of malware. Here's a web shell. This is a great, I like to use this for
showing how the context works. So BlueSpawn uses a combination of regexes and YARA rules,
and it's scanning all the stuff on the system in the web accessible directories,
in say like IIS for PHP files, and it's able to say like, oh this was caught by this YARA rule,
this piece of this file probably contains a web shell. And the next here is a pretty simple
example with like backdooring services. So as hackers like to do this of course to get their
malware to persist, and they can do this in a lot of ways, and so this is just a very simple example.
But BlueSpawn will scan all the services on the system to find which might contain a piece
of malware or what service is suspicious. So we want to get some feedback on RedTeam of course
when we made this tool, and this was actually probably one of the first years they've ever had
to contend with an open source tool being built by one of the teams. No one's really done that
before just because it's a lot of work, and if other teams are using it then maybe you're not
getting quite as much value out of it. But I think it worked out really well, and as you'd expect
our RedTeams spent a lot of time analyzing the tool since it's open source. So the attackers
know exactly what it can detect on, which is it's a bit hard to when you think about it you're like
oh I don't know if I want the RedTeam seeing it, but what we found is that the it really pushed
the attackers to move to new techniques. So they couldn't just rely on the same things over and
over again because they knew we were going to catch that. So it forces them to keep moving and
shifting their tactics, which means they might be less detectable, but it also really increases
the barrier to make it harder for them. And honestly one of the coolest parts is that the
RedTeam unveiled something called RedSpawn this year. So that was a tool that was apparently
supposed to like attack and defeat BlueSpawn. So that feels like a validation to us for sure that
BlueSpawn is effectively killing stuff that they used to rely on.
So from the BlueTeam perspective, what we found in our experiences and other teams and all is
that BlueSpawn works kind of like an EDR product that you'd expect to. So you can identify, it'll
identify and respond to malware on the system. One of the cool things we found is that it's
like a force multiplier, especially in these active defense situations. So imagine if your
tool is deployed out across the multiple systems, right, you don't have to manually look through
all like the auto runs yourself. It'll just flag which ones are most likely to be suspicious.
And that way you can spend more of your efforts hunting where it counts
to find the malware. And so that's really important. So we've prepared a little demonstration
of what BlueSpawn looks like in action and how it can be actually used to hunt and then destroy
malware. So I prepared a Windows Server 2012 R2 VM here and I filled up with a bunch of different
random pieces of malware and their systems. So you can see here that I've got two active
meterpreter sessions on it and I'm going to use BlueSpawn to go hunt and destroy them, among other
things. So first I'll show you BlueSpawn's help menu just so you can see all the options it has.
And that's a pretty simple interface. Because I want to go find bad things, I'm going to run it in
hunt mode. I'm going to specify a normal aggressiveness just because I don't need to go
too in-depth to find anything. When I find stuff, I want to react by deleting files. So I use the
delete file reaction. I'm going to remove values. So remove value reaction and card memory. And then
I want to log to a file so I can view it later. And then the console so I can see it now.
Just press enter and now it runs. That's pretty much all there is to getting it to run.
So here it found a process injection into SPC host PID 632. And I'll open it up in a process hacker.
So you can see here that it actually tells me the memory address where it found the
injection. So I can actually go take a look at that in process hacker. And you'll see here that
it's RWX memory section, which generally is not a great thing. So I'll go ahead and carve that out.
And here I found a couple more. So I'll go ahead and carve those out too.
And here it found some process injection into win-startup.exe. And so I'll go ahead and carve
those out. And now here it found the sticky keys backdoor that Jake was talking about earlier,
the image file execution options debugger. And so I'll open that up in autorun so you can see that
it's also found by autoruns. And it's under the image hijacks tab. Sometimes everything doesn't
show it. You can see it there and there. So we'll go ahead and remove that. And here it found the
run key that was running the process we found some injection in earlier. The window startup
actions, which definitely is not legitimate. So go ahead and remove that run key. Here it found
NetSH helper DLL, which is a sneakier persistence technique. You can see it in regedit. And
BlueSpawn actually protects the files during scans. So this actually won't be able to show
the signatures on it. But I assure you it's not good. So we'll go ahead and remove that value
there. And now it found the PHP web shell. We can go ahead and remove. And here is the malicious
service. So go ahead and remove that. And for some reason autoruns can't find it. But I'll show
you in this command prompt that it absolutely is there. So I'll go ahead and remove that with
BlueSpawn. And now I'll go ahead and remove the NetSH helper DLL file. And here's the
win startup.exe that it didn't like. And here's the malicious service. And that's it. That's all
you have to do. And here's autorun showing you that stuff it found has been cleared out.
So next slide. So here's what happened to the interpreter sessions that I had open.
They got killed when I carved out the memory.
And now on to the future work. BlueSpawn still has a long way to go. A lot more features to
be added. I'm going to talk here about some of the ideas and plans to come up with for future
development on BlueSpawn. So most of our goals right now fall under three categories.
Improving coverage, improving configurability, and improving integrity. So under coverage,
we plan to continue adding new hunts and mitigations. Arguably our biggest goal here
is to expand our data sources. So this would constitute things like API monitoring through
a DLL load into all the processes or a kernel driver. Network traffic monitoring, user auditing,
that sort of thing. We've also placed a large focus on improving our scanning capabilities.
Admittedly, they are somewhat lacking right now, using only the very basic metrics.
One of our goals with BlueSpawn from the start has been to kind of bridge the gap between incident
response and active defense, taking past incidents into account for current threat hunting.
Currently, BlueSpawn has taken some smaller steps towards integration between the two,
but we intend to work on building that out further and making it really a much bigger feature.
As we develop BlueSpawn, we've built most of our features to be very configurable from an API
perspective. But as of now, BlueSpawn doesn't really do a good job of exposing those APIs,
and the command line interface we have barely taps into the surface of that.
We'd really like to expose more of these configurations to allow things like much
more targeted hunts, customizable scanning, and mitigation presets. We'd also like to add a way
for signatures and definitions to be configurable, so that if users have their own signature-based
integrate, they'd be able to. As for integrity, right now, BlueSpawn has very little defenses
against malware or malicious users actually tampering with it. To that end, we intend to
work on hardening BlueSpawn itself, making it much more difficult for a malicious process from
modifying it or causing problems with its execution. And this would most likely be
through a kernel driver. BlueSpawn also assumes it has all the rights that an admin normally has.
But if malware is trying to protect itself, it might restrict access to certain resources that
BlueSpawn was assuming it'd just get with no problem. This also would likely be addressed
to the same kernel driver. Finally, and this is more of a development feature, we intend to set
up detailed Atomic Red Team tests, which are basically tests that test against one specific
malware technique to ensure that each build has proper coverage for everything it should.
Now, everything I've presented so far has been focused on our
Windows client, but our plans go a lot beyond that. Just this past week, actually, we finished
porting a version of BlueSpawn over to Linux, where we are now able to write hunts of mitigations.
The Linux client is a decent bit behind the Windows one in terms of capability and coverage,
but we plan to have an initial release at some point soon. We've also begun developing a server
used to deploy BlueSpawn across the network and then aggregate the logs and manage the client
at scale. This server is still in its early stages, but we plan to ramp up development in
the next few months. Finally, the last major component to BlueSpawn is the cloud. Now,
this is a long way off, and I'll admit we haven't even started thinking about developing it yet.
But the idea is that the clients or servers could send off malware to the cloud to perform
better, more detailed scans, likely doing some form of sandboxing or heuristic analysis. This
could also serve as a threat intelligence repository, sending out IOCs to clients and servers.
So, we've talked a lot. Hopefully, we've given you a really good overview of kind of
at a high level, like why we built BlueSpawn, a bit really into the weeds of how it works,
and how these types of solutions really work in the real world, and then kind of showing you
what it's like in action, right? So, we're going to conclude. First, remember that BlueSpawn is
very much still in the alpha stage. So, it can detect a lot of the most popular techniques,
but there's certainly some things that it's going to miss, and it's a bit rough around some
of the edges. Since all the code and detections are open source, though, we highly recommend
you taking it out, just going to the GitHub page, and you can kind of find and look through,
see exactly how it works. And because it's really important to us to shed light on kind
of how these black box EDR solutions that you might have deployed out in your network might
actually work. And we also want to extend a huge thank you, again, to the community projects that
are listed here. Really, building something like this is not possible without these frameworks
and libraries, so definitely check those two out as well. So, most importantly, if you've
gained anything from this talk, we'd like you to walk away with these three things.
First, using MITRE ATT&CK to be able to detect threats and think about your defenses
is a great springboard. And really, think about it as a springboard. So, like we found when we were
developing BlueSpawn, that MITRE ATT&CK might not have the most in-depth information on a particular
technique, right? But you can use that and go as a rabbit hole. So, if you know attackers are going
to abuse it, you can go look into that technique more to figure out how attackers might use it.
So, that's a good, really good place to start. Second, knowing your coverage. So, we've used this
more times than I can count, right, in our cyber defense competitions and all. And it translates
really good to the real world, right? So, if you start thinking about, like, if you're X-ing
techniques off of the attack matrix, right, you can actually get pretty far because if you think
about it, the attacker isn't going to be able to use those techniques as well against you once
you've got a little X over them. So, that's a great way to think about defense. And then finally,
really, BlueSpawn is kind of like a massive experiment just to figure out
and understand how modern-day defensive software works. So, just like you might tear apart, like,
exploits and malware, we recommend doing the same thing for your defensive security solutions,
right? And thinking about, learning about how they work, and then you can use that
to kind of build and improve your own defenses. So, just a few housekeeping items, I guess, before
we wrap up. Thank you all for listening today. We hope you've had as much fun as we have in creating
this and talking this afternoon. We also would like to thank the conference volunteers, right?
This would not have been possible without all their hard work and setting up the streaming
and just making it all flow. Finally, we've linked the slides on the README. We'll have the
DEF CON release, if you will, go live here in a bit after this talk ends. So, you can download
the tool as we've been talking about it and kind of see a bit more what it's like. You can look at
the slides. We also have a Discord for the project. So, if you have any questions and I think
they're going to post a link to the Discord and all, hop in there and you can go chat with us
afterwards and we'll let you know. We'll be happy to answer any questions you might have. So,
thanks again, guys, for listening. We hope you really enjoyed it. Thank you.
Well, thanks, Jake and Jack. That was a great talk. I look forward to see what
BlueSpawn continues to develop into and appreciate you coming out to DEF CON and presenting to us.
As a reminder, as they've got up here on the slide, the Discord channel takes you over to
their server. It is also up in the General, under the General tab of our Discord channel,
as well as a link in the actual Texthalk channel as well. Other than that, have a great day, guys.