start	end	text
9610	14170	And welcome back, everybody, to DEF CON 28 Safe Mode, Blue Team Village.
14530	19710	We're going to start our next set of speakers. We've got a great topic coming up today.
19990	26070	The title of this talk is Building BlueSpawn, an open-source active defense and EDR software.
26310	31910	Jake Smith and Jack McDowell will be presenting. As always, please direct all questions to the
31910	38970	Tech Talks Track 1 channel, or you can connect to their Discord server as well and send messages to
38970	45390	them directly. I'll have that in there in one second. And on that note, over to you, Jake and
45390	52430	Jack. All right. Thank you. So how's everyone doing today? Just a few quick notes before we
52430	57170	get into the presentation. We definitely want to take a moment to say thank you to all the
57170	62870	volunteers and organizers of the Blue Team Village. Putting this event on in a normal
62870	68050	year is a lot of work, and I can only imagine what they've done to make this happen. So a huge
68050	74890	thank you to them. So today, Jack and I are going to be talking about an open-source project we've
74890	79910	been developing for about a year now. It's called BlueSpawn, and you'll notice the emphasis on
79910	86010	active defense here, so we'll get into more about that later. But first, a little bit of background
86010	92510	about the two of us. We went to the University of Virginia. Both of us got our start in InfoSec doing
92510	98230	really all kinds of cyber competitions, CTFs, and whatnot, and that's been a fantastic way to
98230	105710	kind of get into security. And these competitions we'll talk about more are some of the reasons why
105710	113310	we created BlueSpawn and all. So an important caveat, too, before we get into the presentation.
113310	120050	When we say we this afternoon, we're referring to all six of us on the slide. It turns out that
120050	127670	developing a large open-source project with already like 20,000 lines of C++ is a huge,
127670	134550	huge undertaking. So it's really a team effort that we've been doing to create this.
135810	141590	So, and also before we get into talking about BlueSpawn specifically, we're going to do a brief
141590	146950	overview of some of the available tools out there and kind of where they shine, where they have some
146950	153690	shortfalls, and more. And really, this is just all the tools that we're going to mention do a
153690	159110	fantastic job, which is part of the reason why we want to talk about them as well. So first,
159110	165310	we'll start with your Endpoint Detection and Response, or EDR, tools. That's your CrowdStrikes,
165310	173450	your Carbon Black, your Defender ATP, Sentinel-1, and all. They obviously used to, like 10, 15 years
173450	180390	ago, AVs were pretty simple. They look for bad hashes and whatnot, and now kind of that's, they've
180390	184290	increased, and they've gotten really, really good. So these commercial tools, and they're a bit
184290	190130	expensive in some ways, but they really raise the bar for finding the low-hanging fruit, especially
190130	194910	in some, even some of the more advanced malware. They're obviously not all created equal, but
195670	199130	one of the things we noticed when we started developing BlueSpawn is that
199130	204290	these defensive security solutions are pretty opaque overall. There's obviously a lot of
204290	209930	offensive tools out there, but it's really hard to get your hands on and kind of understand how
209930	217630	these modern AV and EDRs work, which leads us into a bit about some of the free tools that
217630	223970	are out there. There's not a whole lot of great ones, but definitely Sysinternals is probably
223970	229710	one of the best. If you haven't used the Sysmon, if you haven't used much just beyond like Sysmon
229710	235450	to improve your logging, highly recommend taking a look at like Autoruns and Process Explorer and
235450	242230	Procmon. Just, they do a great job of finding malware on the system. They're more geared though
242230	246990	toward the system analysis side, so they don't really, they're not going to say like, hey, this
246990	253750	is malware. They might show like a different color if this is like encoded or something, but they're
253750	259610	not going to point out and like say this is MITRE attack technique 1547, right, for example.
260610	266350	Next is Process Hacker. Process Hacker is another really great tool. There's not much automation or
266350	272310	security focus, but it's really good at just analyzing. It's like Task Manager on steroids,
272310	279530	really, and you can even look at like a specific region in memory. So like if you know that
279530	286050	one process was injected, for example, then you could go and analyze each bit of the memory
286050	292790	yourself. And then on the like purely defensive side, we've definitely found PE-Civ to be one of
292790	297870	the best free and open source tools out there by far. The idea is pretty simple behind the tool.
297870	303870	It's basically look at all the processes running on the system, and then it can find hooks, process
303870	309530	hollowing, process injection, shellcode, all that kind of good stuff. The only downside,
309530	314850	if it's really a downside, is that it's limited to scanning processes. So it's not going to find
314850	319030	like other malware, say, like that's hiding in the registry on a system.
320250	327470	And finally, in this kind of available tool section is like the collections of YARA rules,
327470	332770	Sigma rules, Sysmon configs out there. Even just a few years ago, there just wasn't really a whole
332770	338570	lot out there, especially on the defensive side in way of signatures. But these projects are doing
338710	343710	a really good job of starting to share threat intel a bit more widely and can help you find
343710	349950	threats just on a particular system. The only thing, I guess, the caveat is like with hardening
349950	355570	scripts, for example, a lot of the stuff you'll find out there is very, I guess, more script-like
355570	360250	and not fully fleshed out. So that's one thing to be aware of, like they'll set conflicting values
360250	368550	or miss critical settings and whatnot. But so that, I mean, that's kind of the reasons why we
368550	373890	got into building BlueSpawn, which we'll talk about here more shortly. And one of the key
373890	379330	things we want to emphasize this afternoon is that is the ideas and principles on which we
379330	383590	built this tool. You're probably not going to go into work Monday and deploy BlueSpawn
383590	388730	in production. And I guess you could, but that's maybe not something you want to do quite yet
389370	394350	because the project's not quite that far along. But that said, you can use the information in
394350	400610	today's talk to conduct your own security research and begin kind of improving your own defenses too
400610	408210	and looking at how these kinds of solutions work. So with all things, all great ideas, I guess,
408210	413890	start in Slack and whatnot. So this was, I went into our archives, this was last year, like we
413890	419130	started with this really simple idea, like what if we could like scan one particular thing, like
419130	426790	look at GS registry run keys, like as a way to detect threats and whatnot on a system and like
426790	434470	an open source tool. And we didn't find a whole lot out there. So that kind of led us to creating
434470	440850	BlueSpawn. And as I said, we were students or we are students, so we compete in a lot of these
440850	446410	different cyber defense competitions and whatnot. Honestly, frankly, I tell some people that I just
446410	454850	do go to school for competitions and all. But basically, these cyber defense competitions are,
454850	460190	basically, you get a network and you're asked to secure it against active attackers, who
460190	464450	unfortunately for the students are industry red teams. So you can imagine how that kind of thing
464450	470290	goes. And the red team pretty much always wins. So in these competitions, though, we can't really,
470290	474750	we can't use commercial tools and we can't bring in scripts. So we have to rely on open source tools.
474750	480470	That's kind of another reason why we focus so heavily on the open source aspect. And so we
480470	487710	built BlueSpawn on one really particular focus is that we want to be able to stop threats as
487710	493770	quickly as possible. So I think that you should be able to run this tool and find as many, if not the
493770	499370	majority or more of all the threat, all of the malware that's running on your system as quickly
499370	506990	as possible. And so that is where the idea of active defense comes in.
507570	512530	And then the next is coverage. So when doing these cyber defense competitions, it's time
512530	517870	limited. You're stressed and whatnot. So you don't have the luxury of, say, being able to
517870	523050	look at everything very closely. And you can't pull up a process explorer and look at every
523050	529970	individual process to hunt for process injection. So our idea is that if you can get 60, 70, 80%
529970	536030	of the way there on finding the most common threats, and you know what your tool is going
536030	542030	to be looking for, you can spend more time on those other areas as well. And finally, open
542030	547750	source software. So this is a call to everyone out there. There's a lot of stuff on the offensive
547750	553850	side, but there's just not really as much on the defensive side. So definitely building more
553850	559210	open source blue team software, I think, is really important. And it's a great way for people to learn
559830	565530	about how these kinds of things work and how you might detect, say, one particular technique in the
565530	574210	attack matrix. So as time went on, from that simple idea that we posted a while ago in Slack,
574660	581230	we started moving BlueSpawn towards what you might term as calling EDRs type solution
581230	588610	today. So that's where it's doing continuously monitoring. And we're asking questions like,
588610	594590	what if we could just deploy BlueSpawn across the whole network like you might deploy like Falcon?
594590	599650	This is kind of the ideas that we're heading towards as we spend more and more time developing
599650	607830	this. And then also proactive defense. So it's not enough that you can detect malware and respond
607830	614010	to it, right? We also, one of the core focuses is the idea of being able to prevent these malicious
614010	620590	activities in the first place. So we've talked a lot about active defense so far, but what do we
620590	625930	really mean by it? And how does BlueSpawn approach the idea of finding anomalous or malicious
625930	631810	behavior? And that's kind of what I'll get into in this section. So typical antivirus or EDR
631810	636130	products are meant to kind of run in the background behind everything, only raising alerts and acting
636130	640730	when there's something definitely bad going on. For their use cases, this is a very good thing.
640730	645570	People don't want alerts about everything that may or may not be malicious. But when you've
645570	650470	been compromised, or even when you think you've been compromised, that might not be enough. This
650470	655230	is where our concept of active defense comes into play. Rather than sit in the background,
655230	660370	only rarely making noise, BlueSpawn is meant to be a tool that's actively used. The goal is
660370	665210	to find absolutely everything that could be malicious, not just the things it knows for sure
665210	670350	are malicious. In such an approach, there are drawbacks. In catching all the things, we're sure
670350	675470	to catch some things that aren't actually bad. But for an active defense situation, that's a
675470	681890	trade-off we want to make. So how does BlueSpawn go about approaching this? Originally, this worked
681890	687210	by just performing a scan of the system at a point in time to identify all the threats that
687210	693270	were currently on there. And this hunting involved the typical registry scans, file scans, or process
693270	700430	scans that an AV or EDR product would use. But again, BlueSpawn tried to be more in-depth, taking
700430	705890	more time to look at things, with more of a focus on finding anything that's not normal. And I think
705890	712670	what makes BlueSpawn different is we also factor event logs into our hunts. So correlating what
712670	718270	Windows events have occurred with what is currently on the system to identify any sort of
718270	722370	malicious behavior there. And we're trying to basically use that to bridge the gap between
722370	729010	incident response and what EDR products do. And as we move towards continuous monitoring,
729010	733910	we extended these hunts to be able to run as needed. And I'll talk more about how we did that
733910	741110	shortly. So as I mentioned on the previous slide, BlueSpawn is built pretty heavily around the idea
741110	746410	of what we call hunts. For each MITRE attack technique we support, which is basically a way
746410	752410	attackers do things if you're not familiar, we've created a hunt, which basically is a search for
752410	758330	any sort of evidence of malicious activity using technique. We'll get into more detail about how
758330	762710	specifically these hunts work in a bit, but for now, just know that they're really the starting
762710	767470	point for everything BlueSpawn does. Monitoring, as I mentioned a while back, is really just an
767470	773390	extension of the hunts we've already written. Each hunt creates triggers for when it should be rerun,
773390	778370	and when those triggers occur, it gets run again. Now, MITRE recently released sub-techniques,
778370	783590	which break down each technique attackers may use into subcategories. And BlueSpawn has just
783590	788470	moved to handle that by making each hunt responsible for all of the sub-techniques that fall under it.
790530	795210	So these hunts are really great for finding the starting points in malicious behavior.
795210	799590	We can find registry keys that run files around startup libraries that aren't meant to be
799590	804130	included at all, all sorts of fun stuff. But malware tends to have a number of different
804130	808610	components that it uses, or even if it doesn't, it exists in multiple different forms in multiple
808610	813370	different places. Maybe there's a registry key that points to a file, and then that file is
813370	817470	loaded into a process, and then that process is changing other registry keys, and those keys
817470	823730	point to files. And those files, well, you get the idea. To address this, BlueSpawn searches each
823730	828570	piece of evidence it finds in order to see if it can catch anything related to it. The result from
828570	833190	this is a network of evidence of malicious activity. In the next section, I'll go into
833370	838890	a bit more detail about how this actually gets built. Now that I've given a higher-level overview
838890	843410	of our methodology, I'm going to take a deeper dive into the different things BlueSpawn can do
843410	848330	with a focus on how it does them. And this section is going to get a little bit technical, but I'll
848330	854350	finish it with a more broad overview if you don't catch anything. So before I get into any of the
854350	858970	features, I want to preface with a disclaimer. BlueSpawn is still in alpha development. So far,
858970	863850	we've been mostly focusing on infrastructure, that is, building out the core of the features I'm
863850	867830	getting into. Everything I'm presenting is currently implemented, but it's not entirely
867830	872270	without its bugs. And the biggest example of what I'm talking about is our scanning
872270	877750	subsystem. I'm going to say things like, this is where BlueSpawn scans X, but the actual scans
877750	882530	performed are new and not necessarily heavily tested. At this point, I'd say a lot of the
882530	888350	features described here are better talked about as a framework than a fully fleshed-out product.
888750	892570	And none of this is to say BlueSpawn isn't effective. We've spent a long time building
892570	896930	our hunts and expanding our coverage, and it's proven to be pretty good at its job, as Jake will
896930	903050	showcase in his case studies section. And now that's out of the way, onto the features.
903150	907930	BlueSpawn has five major components, each of which we'll discuss in detail in this section.
907930	912730	There's Mitigate, which applies security settings, Hunt, which runs the hunts I was talking about
912730	919130	earlier, Scan, which determines how bad detections really are and finds associations, Monitor,
919130	924990	which hunts over time, and React, which handles responses to what HuntMode and ScanMode have
924990	931830	identified. So Mitigate mode sits somewhat apart from the rest of BlueSpawn without really
931830	937090	any integrations to the other modes, so I'll just talk about that first to get out of the way.
937550	942350	Mitigations applied by BlueSpawn are all mapped to either a Security Technical Implementation Guide,
942350	947790	or STIG, released by DISA, or the Miner Attack Mitigations Framework. When running Mitigate
947790	952230	mode, BlueSpawn can either audit the current state of the system, which just says this is what it's
952230	957130	currently set up as, or it can go ahead and apply the mitigations it finds aren't properly configured,
957130	963270	with user confirmation, of course. Now, as of now, our mitigation framework is very quite simple,
963270	967410	but we think it has a lot of potential as BlueSpawn continues to grow. We see it eventually
967410	971590	being configurable through some sort of configuration file, so mitigations can be
971590	976530	more uniformly and quickly applied across a network. We also see it mitigating services
976530	981750	and software installed on a system, rather than just the base operating system. And finally, our
981750	988590	biggest envisioned change is for Mitigation mode to serve as really an aid in performing a system
988590	993570	audit. So this would perform some of the same tasks as enumeration tools, such as
993570	998950	Seatbelt or JAWS, with a focus on finding and fixing things actually being vulnerable to being
998950	1004790	exploited. Now, before I get into the weeds of how everything works, I'm going to throw some
1004790	1011070	definitions at you for how I refer to the ideas and concepts BlueSpawn uses. So I'm going to be
1011070	1015870	using the word detection a lot, and it's just that, detection. It could be something good, it could be
1015870	1021050	something bad, it's just something BlueSpawn identified as taking a closer look at. It could
1021050	1026550	be like a file, a registry key, a process, or something else entirely. Next, there's an association,
1026550	1031910	which is also just what it sounds like. It's a connection between two different detections, and
1031910	1037150	in BlueSpawn, we represent associations as having a weight, representing how closely they are
1037150	1042970	associated. Next thing is certainty, and this is the metric we use for determining the likelihood
1042970	1048890	that a detection is actually bad. There are three components to it. The two of them get combined, so
1048890	1054050	in practice, there are really only two. The first is what I call intrinsic certainty. This is the
1054050	1059190	certainty that comes from the detection itself. This would be what we're talking about if a
1059190	1064750	service was called Mimikatz, or if a file matched a Yara rule from Reterpreter, or something like that.
1065380	1071210	The next is contextual certainty. This comes from the context surrounding the detection.
1071210	1075310	In a lot of ways, this is very much like the intrinsic certainty, and in fact, BlueSpawn
1075310	1080070	doesn't actually differentiate between the two. They're really combined. And what I mean by
1080070	1087930	context is things like a file in System32 that's not assigned by Microsoft, and that's probably bad
1087930	1094130	just because they usually are. Or any value listed under an app in a DLL is probably bad just because
1094130	1099310	they aren't really used legitimately anymore. The last type of certainty I'm going to talk about
1099310	1104670	is associative certainty, and this is the certainty that comes from the associations of detection.
1104930	1110250	Say we've got a file and we found it loaded into a process. If the process is behaving maliciously
1110250	1115650	and the file appears malicious, the file should be treated with a higher certainty than if all
1115650	1122050	we had was a malicious-looking file. Now that I've got those definitions out of the way, I can move
1122050	1127430	on to how BlueSpawn actually works. The core of all detection management, and really the core of
1127430	1132910	BlueSpawn itself, is our detection register. This tracks everything that BlueSpawn's found.
1132930	1138410	Whenever a hunt or a scan or whatever finds anything that might be bad, it sends it to the
1138410	1144130	detection register. The register will record the detection and send back a reference to it. Now,
1144130	1149570	this reference may or may not be the detection that was recorded, and I'll explain why next
1149570	1154410	slide when I explain how the registration process actually works. The detection register has three
1154410	1161090	storage mechanisms. So whenever a detection gets registered, it gets queued for a scan and we track
1161090	1165830	all queued detection in one set and all scanned detections in another. That way we can just make
1165830	1170370	sure we're not redoing the same work. We also keep a record of all detections regardless of whether
1170370	1176350	or not they've been scanned. So now for how registration actually works. The first thing
1176350	1182030	the register actually does when a new detection is being added is it checks if the detection
1182030	1186610	already exists. If it does, they get merged together. And generally the only thing that
1186610	1192410	changes when this happens is their contextual certainties and their contexts get merged.
1192410	1197910	If this merge causes their overall certainty to exceed a certain threshold, it then gets logged
1197910	1202290	and its reactions get triggered. If the detection being registered was merged with the pre-existing
1202290	1206590	one, the reference that returned is actually to the pre-existing one. That way it's just keeping
1206590	1212170	track of the detection that we care about. Otherwise it's to the new detection. After all
1212170	1218410	this is handled, if the detection register needs to queue scans for the detection, it will. This
1218410	1223510	constitutes calculating the intrinsic certainty, identifying associated detections, and then if the
1223510	1228370	certainty is high enough after that, logging it and handling reactions. The associated detections
1228370	1236510	then get registered and the scans for those get queued and so on. So here I've got a diagram for
1236510	1241250	how the detection register lays things out. It looks kind of complicated, but I'll try to explain
1241250	1246750	it so it's maybe not quite as bad as it looks. So the register itself has three main storage
1246750	1252110	mechanisms. There's the queue of things that haven't been scanned yet, things that have been scanned,
1252110	1257370	and then everything overall. And you can see that all these storage mechanisms really just store
1257370	1263990	references to detections that are stored elsewhere. I've also broken down one detection, in this case
1263990	1269790	ID2, to show what kind of information detection actually holds. So it first has its ID, which is
1269790	1274810	just an ID, a unique identifier to refer to it. It has a context, which is the information
1274810	1280610	surrounding how it was made and how it was found and when that happened and all that. So that'd be
1280610	1285890	what time the detection occurred, what hunt found it, and when the first evidence of the detection
1285890	1293990	was found. There's also the data, and this stores information about what actually did it find. So in
1293990	1298790	this case, I've just decided for the purpose of an example, this detection is going to refer to a run
1298790	1306130	key and its value name. Its name persists and it's pointing to this file called bad.exe.
1306630	1311370	The next thing I have is the scan information, which stores information about how it was scanned.
1311370	1316350	That has the certainty, which combines the intrinsic and the contextual certainties,
1316350	1320610	and then the associative certainty. The last part of this is the associations, which
1322650	1327130	identifies which detections are associated with this one. In this case, detection ID1,
1327130	1334030	which I'm using to refer to as bad.exe, is associated with it. There's also a remediator,
1334030	1338950	and I'll get to that a bit later. So in the methodology section, I mentioned how hunts
1338950	1342950	represent the starting point for pretty much all the threat hunting that BlueSpawn does.
1342950	1349430	Each one covers one minor attack technique. Each hunt, we then break down into sub-techniques,
1349430	1354290	which can be run independently of each other. Within each sub-technique, that gets broken
1354290	1360650	down then into sub-sections, and one sub-section generally checks just one specific attack surface,
1360650	1365410	whether it's a folder, a file, an event ID, or a registry key, or something else. The idea is to
1365410	1371470	keep it as simple and small as possible. This design allows for much more targeted hunts if we
1371470	1375350	know what we're specifically we're looking for, and I'll get into why that matters when I talk
1375350	1380930	about how monitoring works. Most of the things we check get a quick scan, which just performs the
1380930	1385470	most elementary scans. It's a very quick test to determine whether or not something deserves more
1385470	1390410	detailed scans. If it does, we create a detection for it and register it with the detection register
1390410	1395210	and kick off the whole scanning thing I just finished talking about. Then the scans I discussed
1395210	1401790	get run on it and make a determination, find other related things, and so on. Here's an example of
1401790	1407650	what a hunt looks like. In this case, this is a T1547. The hunt itself really just runs all
1407650	1414670	sub-techniques underneath it, you can see here. And then here's the sub-technique number four,
1414670	1419970	which I picked out because it was pretty simple. The first part of it just compares
1419970	1427390	the winlogon values for the shell and username against known goods. If they don't match,
1427390	1431750	we create detection with a moderate contextual certainty, because anything that doesn't match,
1431750	1438070	chances are it's bad, even though the file itself might not be bad. The next sub-section
1438070	1444610	refers to the winlogon notify values. Any of these that we find get a detection created for it,
1444610	1448970	just because they aren't used very commonly. And we create that with a weak contextual certainty,
1448970	1454190	because again, if you find one, chances are it's not great. Now you'll notice during each of these,
1454190	1460430	I specify cursory when I initialize a sub-section. That comes from... when you run bluespawn,
1460430	1465030	you specify how closely it should be looking for things. There's cursory, which is a very quick
1465030	1469990	check. There's intensive, which is a very slow but careful check. And there's normal, which is
1469990	1475270	somewhere in between. By specifying cursory here, we indicate that bluespawn should run these
1475270	1479770	sub-sections whenever the intensity is greater or equal to the cursory.
1481590	1485870	So when we originally had our first working version of bluespawn, it was just a few hunts
1485870	1491050	we'd written and not too much else. Since then, everything we've done really has been taking what
1491050	1495710	we had and proving it. When it came time to start monitoring systems in real time, rather than just
1495710	1500910	performing a single scan, we realized we could reuse the hunts we had already written to do this.
1500910	1506390	Each hunt we created already defines a number... sorry, each hunt we created defines a number of
1506390	1511210	events that indicate something pertinent to the hunt has changed. For example, we might make an
1511210	1516570	event watching a folder for new files or old files being edited. When the event gets triggered,
1516570	1521270	hunts that care about that folder get rerun. Of course, there's some obvious inefficiencies in
1521270	1526670	this, which is a big reason why we divided hunts into sub-sections. Now, when hunts create events,
1526670	1532170	they specify which sub-section should get run for each event. Now, I'll admit monitoring is
1532170	1536330	unfortunately one of the areas that hasn't received as much attention as others. It's
1536330	1539870	certainly functional and it's effective at watching for things that hunts can catch,
1539870	1545570	but so far we don't have that much that acts based off of behavior alone that we observe while
1545570	1550270	monitoring. In the coming months, this is going to be one of our big focuses. Adding things like
1550830	1556310	event tracing for Windows or API monitoring should position bluespawn to be able to catch much more.
1557790	1564550	So, here's an example of how monitoring sets up its... or hunt sets up all of its monitoring
1564550	1570130	triggers. So, if you noticed my example hunt two slides back, each of these sub-techniques gets a
1570130	1577510	scope passed into it. Well, under the hood, what's happening is the sub-section macros I have
1577510	1582030	actually check the scope against that sub-section ID to determine whether or not that sub-section
1582030	1587670	should be run. So, whenever I create an event here, I specify which sub-sections are pertinent
1587670	1598070	to it. And yeah, so here we have registry events, file and file events, but we also have event log
1598070	1604530	events to watch for Windows events being created. So, the last component I'm going to talk about
1604530	1609210	here is the reaction framework. As the name would imply, this is responsible for taking care of
1609210	1614830	that BlueSpawn has determined are bad. Since BlueSpawn is designed around finding as much
1614830	1621070	as possible and is bound to have some false positives, most things here require some sort
1621070	1626850	of level of user confirmation before taking action. Now, obviously, when I delete a file,
1626850	1631070	it's no longer an active threat, but there's still a reason for BlueSpawn to track it.
1631070	1635730	To handle this error detection, each detection... sorry, to handle this error situation,
1635730	1640970	each detection has a flag where we mark it as stale or not. Whenever detection... sorry,
1640970	1645850	whenever reaction mitigates the detection, we mark it as stale, so that way other reactions don't go
1645850	1651350	and try to mitigate it their way. Currently, we have five reactions available, but there is
1651350	1657770	somewhat of a less defined six, and I'll get into that. BlueSpawn can delete files, quarantine files,
1657770	1662910	delete registry entries, and that can be values or keys, suspend processes, or carve out memory.
1662910	1667530	I'll talk about that in the next slide. The less defined reaction I just mentioned is for
1667530	1672450	special cases. Remember back to the WinLogon hunt where it was comparing some values against known
1672450	1677250	goods? It wouldn't be very good to just delete these values if they were malicious. They need
1677250	1681270	to be properly restored. And to handle things like that, when a detection is created, it can
1681270	1687350	be given a custom remediator. This remediator gets precedence over all other reactions if it's present.
1688050	1692530	Now, as we continue to develop BlueSpawn, we of course intend to continue adding reactions and
1692530	1696750	improving the ones we do have. Future reactions might include things like de-registering service
1696750	1703710	or blocking certain commands. So I promised that I'd get into what I meant by memory carving, so
1703710	1709110	here we are. The idea behind memory carving is that sometimes libraries or shellcode get shoved
1709110	1714950	into processes, and these processes still serve a valid purpose and need to remain active. In cases
1714950	1719790	like these where a restart isn't a good solution, we came up with a way to mitigate the effect of
1719790	1724770	whatever malicious library or shellcode is living inside a process. It's not perfect, and it can
1724770	1731070	cause problems, but it's better than nothing. This reaction works in five steps. First, we suspend
1731070	1735910	the process so that nothing we're doing causes problems with the way the process works.
1736430	1741930	First, what we do is we enumerate the threads in the process, searching for any thread whose stack
1741930	1747550	overlaps the memory section in question. And that basically is an indicator that the thread is
1748030	1753250	acting based off what that malicious memory section wants it to do. If or when we find this,
1753250	1758510	we kill any thread that does that. Since malware generally doesn't like to share a thread
1758510	1762650	with something benign, it shouldn't kill anything important to the process. Of course, if you got
1762650	1767730	things like APC injection going on, there's not too much we can do about that right now.
1768170	1773370	After that, we scan all memory pointers to the memory segment in question. If or when we find
1773370	1777770	them, we check the address in question. If it's data, we patch it with zero, so it can't read
1777770	1781950	anything important there. If it's code, we patch it with return instructions, so if it ever tries
1781950	1788330	to call there or jump there, it gets immediately bounced right back. Then we go through and patch
1788330	1793170	the entry point and all functions that are exported with return instructions, so anything trying to
1793170	1798370	run any function inside the memory gets messed up. This can cause problems if we end up patching any
1798370	1802890	function that requires the callee to clean up the stack, but without extensive work to figure out
1802890	1806890	how much of the stack needs to be cleaned up, there's not too much we can do. Once we're done
1806890	1811330	with all that, we resume the process and let things continue. We have tested this with a couple
1811330	1815510	different pieces of malware, including CobaltStrike, Beacons, and Returpreter, and it seems to work
1815510	1820490	without causing too many problems. Unless processes are storing function pointers in an abnormal
1820490	1825770	manner, it should be very difficult to get anything in this memory segment working again after what we
1825770	1831330	do occurs. Now, we are working on a few features to improve the stability of this. This includes
1831330	1837370	things like forcibly walking threads back instead of just killing them, or finding out how much the
1837370	1841990	stack needs to be cleaned up when patching returns, or properly unloading libraries as seen in Process
1841990	1849000	Hacker rather than just breaking them. Now, BlueSpawn would have been nearly impossible to build
1849000	1853020	from the ground up without any libraries or integration. As has been a major theme in our
1853020	1858960	presentation, the MITRE ATT&CK framework has been invaluable in providing guidance around what sort
1858960	1863180	of things we should go for. YARA and YARA rules written by the community have provided immense
1863180	1867380	value in being able to scan files and make determinations. The stakes around which we built
1867380	1871620	our mitigations make it very clear what we should prioritize and give us a framework to convey our
1871620	1877580	coverage. Finally, PESib is a great open source tool we use to scan any process for any sort of
1877580	1882660	fileless execution or any form of execution that's not properly mapped to a file.
1883600	1889120	In summary, BlueSpawn's abilities can be broken into three categories, prevention, protection,
1889120	1894380	and response. Under prevention, we have mitigate mode, which integrates STIGs and MITRE's ATT&CK
1894380	1900780	framework. Under detection, we've got monitor mode, which watches for changes to rerun hunts,
1900780	1906720	hunt mode, which is built around the MITRE ATT&CK framework, and then checks for these threats.
1907100	1911800	Scan mode then analyzes what was found in hunt mode to find associations and make determinations.
1911800	1916260	Finally, under response, we have our reactions, which aim to mitigate or remove a threat,
1916260	1918120	and then logging to report what was found.
1919540	1925120	All right, so that was a lot, but we wanted to really give you a deep insight into kind of how
1925120	1930640	some of these AV and EDR type programs, and obviously BlueSpawn in particular, work.
1930860	1937200	And so now we're going to move into talking about just like seeing the tool in action, right? So
1937200	1941180	we're going to walk through a case study with some kind of specific point in time screenshots
1941180	1946740	of BlueSpawn catching malware, and then also we're going to have a video of kind of the tool
1946740	1952540	in action. And I mentioned cyber defense competitions have been a really important
1953180	1961940	learning tool at the beginning, so first I want to emphasize that again. The largest is by CCDC,
1962260	1967220	and it's held in about 10 regions across the U.S. with qualifiers, regionals, and nationals,
1967220	1974140	and about 230 universities compete each year actually. So that's one way if you're looking
1974140	1980120	to volunteer to help in the community, highly recommend looking into a competition like that
1980120	1986200	or others. So these competitions are basically what's known as an inherit and defend competition.
1986200	1991100	The student teams work in a group of eight, and they're basically work as a fictional company. So
1991100	1996340	like they have a whole set of networks they have to defend, and typically that's on-prem. There's
1996340	2004880	cloud aspects, and that's usually done in like ESXi or AWS. So here you can see an example of a
2004880	2011660	network map from actually last year. So you have some on-prem workstations, there's like a on-prem
2011660	2017040	cloud if you will, and like ESXi, and then there's even like an AWS component. And you'll see
2017040	2023260	the stuff in there ranges from Windows Server 2008, so like really old stuff, to Windows Server
2023260	2029240	2019, and every flavor of Linux distro really. So that's they kind of emulate a modern
2029740	2034460	corporate environment the best they can. And there's of course stuff like Exchange, and web apps, and
2034460	2038840	databases, and all. And so basically there's two goals in these competitions. You have to keep your
2038840	2044620	services online, and then just like you would in a real company, and then you have to stop the
2044620	2051020	right team from really just totally destroying everything. And really frankly that ends up in
2051020	2058820	situations like this. So on the left hand side you see this is from the national competition this
2058820	2065240	year, and it's where a picture of the red team's Cobalt strike instance, where you can see they
2065240	2071340	just have pwned dozens of machines against this poor team. And they also do things like defacing
2071340	2079260	the websites, and RMRFing boxes, or overriding MBRs when they get too much access.
2079260	2086600	But really what that means is that in all reality is that these are a great testing ground
2087040	2091520	for defensive tools, right? So the red team spends all year writing malware for these competitions,
2092000	2099120	and are probably watching right now. So hi red team! But so their tool set is everything from
2099120	2105280	like Metasploit and Cobalt strike, what's pretty commonly used, to really custom malware that's
2105280	2110560	just written for the purposes of CCDC. So with that kind of backstop in mind, we're going to look
2110560	2118240	at some example screenshots of BlueSpawn. So here we just have the system. There's two Cobalt strike
2118240	2124880	beacons running. One is just an HTTP beacon, and then the other is an SMB beacon living in
2124880	2130980	explorer.exe. And so this is how BlueSpawn goes about finding it. It's looking at, when Jack
2130980	2136460	talked about process injection earlier, and the carved memory reaction, you can see BlueSpawn's
2136460	2143500	finding the malware in both explorer.exe, and then also in the PowerShell that's running.
2143500	2148620	And then it starts prompting the user like, hey we found this malicious activity, and all they did,
2148620	2153920	all the user did was just run BlueSpawn. And then it's offering to kill the threads, and once it
2153920	2159220	kills it, then we reach a situation like this, where those beacons basically just stop calling
2159220	2166140	back in. So that's an example of how BlueSpawn is responding to the threat. And here's another
2166140	2174000	classic threat. It's not used obviously as much now, but the sticky keys backdoor is where the
2174000	2179800	registry key is configured with command prompt as a debugger. So BlueSpawn's able to look for that,
2179800	2186380	and that's T1546 sub technique 8, if you're following along on the MITRE ATT&CK matrix,
2186380	2190300	so it knows to look for that, and then it goes and finds it, and it lets you remove
2190300	2195780	that option. Next, this was probably one of the coolest examples we saw at a cyber defense
2195780	2201540	competition that BlueSpawn caught earlier this year. Windows has these things called authentication
2201540	2207180	packages and notification packages, and these exist in like the LSA subsystem, which is used
2207180	2213940	for authentication and all. So BlueSpawn is able to map and is like, hey we found a registry key,
2213940	2221340	it understands that in the hunt, that this hunt refers to things on disk. So then it goes and
2221340	2226180	correlates that with the file associated with it. So we have two detections, one for the file
2226180	2233920	and one for the registry key that contains the badness. Next, here is a good old run key.
2235020	2239640	So there's a malicious BBS script that's configured to run in the registry,
2239640	2245520	and again BlueSpawn is able to correlate the two and be able to connect that to generate detections
2246300	2252360	and find both pieces of malware. Here's a web shell. This is a great, I like to use this for
2252360	2258700	showing how the context works. So BlueSpawn uses a combination of regexes and YARA rules,
2258700	2263980	and it's scanning all the stuff on the system in the web accessible directories,
2264620	2271120	in say like IIS for PHP files, and it's able to say like, oh this was caught by this YARA rule,
2271120	2279640	this piece of this file probably contains a web shell. And the next here is a pretty simple
2279640	2284960	example with like backdooring services. So as hackers like to do this of course to get their
2284960	2291360	malware to persist, and they can do this in a lot of ways, and so this is just a very simple example.
2291360	2297280	But BlueSpawn will scan all the services on the system to find which might contain a piece
2297280	2304200	of malware or what service is suspicious. So we want to get some feedback on RedTeam of course
2304800	2309400	when we made this tool, and this was actually probably one of the first years they've ever had
2309400	2315420	to contend with an open source tool being built by one of the teams. No one's really done that
2315420	2321040	before just because it's a lot of work, and if other teams are using it then maybe you're not
2321040	2328940	getting quite as much value out of it. But I think it worked out really well, and as you'd expect
2328940	2334140	our RedTeams spent a lot of time analyzing the tool since it's open source. So the attackers
2334140	2340800	know exactly what it can detect on, which is it's a bit hard to when you think about it you're like
2340800	2346000	oh I don't know if I want the RedTeam seeing it, but what we found is that the it really pushed
2346000	2350680	the attackers to move to new techniques. So they couldn't just rely on the same things over and
2350680	2354680	over again because they knew we were going to catch that. So it forces them to keep moving and
2354680	2360780	shifting their tactics, which means they might be less detectable, but it also really increases
2361400	2366900	the barrier to make it harder for them. And honestly one of the coolest parts is that the
2366900	2373040	RedTeam unveiled something called RedSpawn this year. So that was a tool that was apparently
2373040	2379300	supposed to like attack and defeat BlueSpawn. So that feels like a validation to us for sure that
2379880	2384760	BlueSpawn is effectively killing stuff that they used to rely on.
2385500	2391480	So from the BlueTeam perspective, what we found in our experiences and other teams and all is
2391480	2398120	that BlueSpawn works kind of like an EDR product that you'd expect to. So you can identify, it'll
2398120	2403200	identify and respond to malware on the system. One of the cool things we found is that it's
2403200	2407860	like a force multiplier, especially in these active defense situations. So imagine if your
2407860	2413480	tool is deployed out across the multiple systems, right, you don't have to manually look through
2413480	2418980	all like the auto runs yourself. It'll just flag which ones are most likely to be suspicious.
2419060	2422200	And that way you can spend more of your efforts hunting where it counts
2422760	2428860	to find the malware. And so that's really important. So we've prepared a little demonstration
2428860	2432720	of what BlueSpawn looks like in action and how it can be actually used to hunt and then destroy
2432720	2441880	malware. So I prepared a Windows Server 2012 R2 VM here and I filled up with a bunch of different
2441880	2447380	random pieces of malware and their systems. So you can see here that I've got two active
2447380	2452140	meterpreter sessions on it and I'm going to use BlueSpawn to go hunt and destroy them, among other
2452140	2458080	things. So first I'll show you BlueSpawn's help menu just so you can see all the options it has.
2458080	2464240	And that's a pretty simple interface. Because I want to go find bad things, I'm going to run it in
2464240	2469740	hunt mode. I'm going to specify a normal aggressiveness just because I don't need to go
2469740	2475160	too in-depth to find anything. When I find stuff, I want to react by deleting files. So I use the
2475160	2481540	delete file reaction. I'm going to remove values. So remove value reaction and card memory. And then
2481540	2487040	I want to log to a file so I can view it later. And then the console so I can see it now.
2487040	2491660	Just press enter and now it runs. That's pretty much all there is to getting it to run.
2493200	2501560	So here it found a process injection into SPC host PID 632. And I'll open it up in a process hacker.
2503620	2507320	So you can see here that it actually tells me the memory address where it found the
2507320	2512320	injection. So I can actually go take a look at that in process hacker. And you'll see here that
2512320	2519240	it's RWX memory section, which generally is not a great thing. So I'll go ahead and carve that out.
2519400	2522440	And here I found a couple more. So I'll go ahead and carve those out too.
2523800	2529960	And here it found some process injection into win-startup.exe. And so I'll go ahead and carve
2529960	2540980	those out. And now here it found the sticky keys backdoor that Jake was talking about earlier,
2540980	2546080	the image file execution options debugger. And so I'll open that up in autorun so you can see that
2546080	2553880	it's also found by autoruns. And it's under the image hijacks tab. Sometimes everything doesn't
2553880	2566350	show it. You can see it there and there. So we'll go ahead and remove that. And here it found the
2566350	2572310	run key that was running the process we found some injection in earlier. The window startup
2572310	2577190	actions, which definitely is not legitimate. So go ahead and remove that run key. Here it found
2577190	2583090	NetSH helper DLL, which is a sneakier persistence technique. You can see it in regedit. And
2583090	2588870	BlueSpawn actually protects the files during scans. So this actually won't be able to show
2588870	2603680	the signatures on it. But I assure you it's not good. So we'll go ahead and remove that value
2603680	2616660	there. And now it found the PHP web shell. We can go ahead and remove. And here is the malicious
2616660	2622200	service. So go ahead and remove that. And for some reason autoruns can't find it. But I'll show
2622200	2634570	you in this command prompt that it absolutely is there. So I'll go ahead and remove that with
2634570	2642110	BlueSpawn. And now I'll go ahead and remove the NetSH helper DLL file. And here's the
2642110	2648430	win startup.exe that it didn't like. And here's the malicious service. And that's it. That's all
2648430	2657050	you have to do. And here's autorun showing you that stuff it found has been cleared out.
2659250	2664070	So next slide. So here's what happened to the interpreter sessions that I had open.
2664150	2666050	They got killed when I carved out the memory.
2669170	2674370	And now on to the future work. BlueSpawn still has a long way to go. A lot more features to
2674370	2678410	be added. I'm going to talk here about some of the ideas and plans to come up with for future
2678410	2684430	development on BlueSpawn. So most of our goals right now fall under three categories.
2684530	2690690	Improving coverage, improving configurability, and improving integrity. So under coverage,
2690690	2695250	we plan to continue adding new hunts and mitigations. Arguably our biggest goal here
2695250	2700550	is to expand our data sources. So this would constitute things like API monitoring through
2700690	2706050	a DLL load into all the processes or a kernel driver. Network traffic monitoring, user auditing,
2706050	2711210	that sort of thing. We've also placed a large focus on improving our scanning capabilities.
2711210	2715290	Admittedly, they are somewhat lacking right now, using only the very basic metrics.
2715630	2719510	One of our goals with BlueSpawn from the start has been to kind of bridge the gap between incident
2719510	2724670	response and active defense, taking past incidents into account for current threat hunting.
2725090	2729270	Currently, BlueSpawn has taken some smaller steps towards integration between the two,
2729270	2734150	but we intend to work on building that out further and making it really a much bigger feature.
2734890	2739850	As we develop BlueSpawn, we've built most of our features to be very configurable from an API
2739850	2744550	perspective. But as of now, BlueSpawn doesn't really do a good job of exposing those APIs,
2744550	2748250	and the command line interface we have barely taps into the surface of that.
2748730	2752930	We'd really like to expose more of these configurations to allow things like much
2752930	2758290	more targeted hunts, customizable scanning, and mitigation presets. We'd also like to add a way
2758290	2762090	for signatures and definitions to be configurable, so that if users have their own signature-based
2762090	2768150	integrate, they'd be able to. As for integrity, right now, BlueSpawn has very little defenses
2768150	2773350	against malware or malicious users actually tampering with it. To that end, we intend to
2773350	2778210	work on hardening BlueSpawn itself, making it much more difficult for a malicious process from
2778730	2783350	modifying it or causing problems with its execution. And this would most likely be
2783350	2788470	through a kernel driver. BlueSpawn also assumes it has all the rights that an admin normally has.
2788470	2792590	But if malware is trying to protect itself, it might restrict access to certain resources that
2792590	2796890	BlueSpawn was assuming it'd just get with no problem. This also would likely be addressed
2796890	2801810	to the same kernel driver. Finally, and this is more of a development feature, we intend to set
2801810	2807050	up detailed Atomic Red Team tests, which are basically tests that test against one specific
2807630	2811750	malware technique to ensure that each build has proper coverage for everything it should.
2813530	2816570	Now, everything I've presented so far has been focused on our
2816570	2821910	Windows client, but our plans go a lot beyond that. Just this past week, actually, we finished
2821910	2826790	porting a version of BlueSpawn over to Linux, where we are now able to write hunts of mitigations.
2827010	2832550	The Linux client is a decent bit behind the Windows one in terms of capability and coverage,
2832550	2837490	but we plan to have an initial release at some point soon. We've also begun developing a server
2837490	2841890	used to deploy BlueSpawn across the network and then aggregate the logs and manage the client
2841890	2846110	at scale. This server is still in its early stages, but we plan to ramp up development in
2846110	2851050	the next few months. Finally, the last major component to BlueSpawn is the cloud. Now,
2851050	2854930	this is a long way off, and I'll admit we haven't even started thinking about developing it yet.
2854930	2859530	But the idea is that the clients or servers could send off malware to the cloud to perform
2860110	2865630	better, more detailed scans, likely doing some form of sandboxing or heuristic analysis. This
2865630	2871310	could also serve as a threat intelligence repository, sending out IOCs to clients and servers.
2872790	2877350	So, we've talked a lot. Hopefully, we've given you a really good overview of kind of
2877350	2882270	at a high level, like why we built BlueSpawn, a bit really into the weeds of how it works,
2882270	2886810	and how these types of solutions really work in the real world, and then kind of showing you
2886810	2892170	what it's like in action, right? So, we're going to conclude. First, remember that BlueSpawn is
2892170	2898130	very much still in the alpha stage. So, it can detect a lot of the most popular techniques,
2898130	2901630	but there's certainly some things that it's going to miss, and it's a bit rough around some
2901630	2907490	of the edges. Since all the code and detections are open source, though, we highly recommend
2907990	2912890	you taking it out, just going to the GitHub page, and you can kind of find and look through,
2912890	2916950	see exactly how it works. And because it's really important to us to shed light on kind
2916950	2922370	of how these black box EDR solutions that you might have deployed out in your network might
2922370	2928970	actually work. And we also want to extend a huge thank you, again, to the community projects that
2928970	2933730	are listed here. Really, building something like this is not possible without these frameworks
2934310	2941350	and libraries, so definitely check those two out as well. So, most importantly, if you've
2941350	2946170	gained anything from this talk, we'd like you to walk away with these three things.
2946670	2952130	First, using MITRE ATT&CK to be able to detect threats and think about your defenses
2952710	2957390	is a great springboard. And really, think about it as a springboard. So, like we found when we were
2957390	2963390	developing BlueSpawn, that MITRE ATT&CK might not have the most in-depth information on a particular
2963390	2968770	technique, right? But you can use that and go as a rabbit hole. So, if you know attackers are going
2968770	2974130	to abuse it, you can go look into that technique more to figure out how attackers might use it.
2974130	2981510	So, that's a good, really good place to start. Second, knowing your coverage. So, we've used this
2982030	2987230	more times than I can count, right, in our cyber defense competitions and all. And it translates
2987230	2990830	really good to the real world, right? So, if you start thinking about, like, if you're X-ing
2990830	2995570	techniques off of the attack matrix, right, you can actually get pretty far because if you think
2995570	3001430	about it, the attacker isn't going to be able to use those techniques as well against you once
3001430	3008370	you've got a little X over them. So, that's a great way to think about defense. And then finally,
3009770	3013310	really, BlueSpawn is kind of like a massive experiment just to figure out
3013310	3019430	and understand how modern-day defensive software works. So, just like you might tear apart, like,
3019430	3024270	exploits and malware, we recommend doing the same thing for your defensive security solutions,
3024270	3028670	right? And thinking about, learning about how they work, and then you can use that
3028670	3035390	to kind of build and improve your own defenses. So, just a few housekeeping items, I guess, before
3035390	3042570	we wrap up. Thank you all for listening today. We hope you've had as much fun as we have in creating
3042570	3048810	this and talking this afternoon. We also would like to thank the conference volunteers, right?
3048810	3053550	This would not have been possible without all their hard work and setting up the streaming
3054050	3060570	and just making it all flow. Finally, we've linked the slides on the README. We'll have the
3060570	3065430	DEF CON release, if you will, go live here in a bit after this talk ends. So, you can download
3065430	3071850	the tool as we've been talking about it and kind of see a bit more what it's like. You can look at
3071850	3077330	the slides. We also have a Discord for the project. So, if you have any questions and I think
3077330	3082470	they're going to post a link to the Discord and all, hop in there and you can go chat with us
3082470	3088150	afterwards and we'll let you know. We'll be happy to answer any questions you might have. So,
3088150	3092130	thanks again, guys, for listening. We hope you really enjoyed it. Thank you.
3092850	3097430	Well, thanks, Jake and Jack. That was a great talk. I look forward to see what
3097430	3103930	BlueSpawn continues to develop into and appreciate you coming out to DEF CON and presenting to us.
3103930	3108090	As a reminder, as they've got up here on the slide, the Discord channel takes you over to
3108090	3114770	their server. It is also up in the General, under the General tab of our Discord channel,
3114770	3119950	as well as a link in the actual Texthalk channel as well. Other than that, have a great day, guys.