And anyway, I'm not going to waste any time. Eric Capuano, everybody.
Okay, I appreciate you guys. Hey, so thanks a lot for coming to my talk, especially being, you
know, noon on Sunday, final day of con. I know that a lot of people are already on planes
heading home. So the fact that you guys are here with me is very humbling. So thank you. I'll
just go straight into it. So obviously you guys know you're here because we all know that
tabletop scenarios are not exactly the key way to train for incidents, right? But what do you do
about that? If that's all you know, then what other option do you have? So what I'm going to
talk to you guys about is something that I've actually done personally with my incident
response team because I recognized there was a gap that needed to be filled. Tabletop is not
it. And so I built an incident response simulation platform. Now, just so you guys know, you
probably saw in the summary of the talk, this isn't going to be like just, you know, waves and
waves and waves of data and things that you've never heard of before. A lot of you are probably
familiar with virtual environments and things of that nature. But I am going to go a little bit
deeper and go over some of the tips and tricks that I learned in my painstaking process to build
this environment to achieve high fidelity training, right? There's a difference between just
building a bunch of VMs and hacking them. No. In order to really get the most value out of it, you
know, there's some things that you want to do to as closely simulate your enterprise environment
as possible. So quick introduction. Should be next. So a little bit about me. I am the SOC lead
for the Texas Department of Public Safety. We have 12,000 endpoints, over 467 remote locations
across the entire state of Texas. So it's a very large network. We are the state police, the Texas
Rangers, Department of Public Safety, the Department of Public Safety, the Department of Public
Safety, the Department of Emergency Management, driver's license. I mean, we've got a lot of
stuff. So that's a busy job. I'm also a cyber warfare operator for the Texas Air National Guard.
Cyber, cyber. In my spare time that doesn't exist, I'm also a private consultant. I do security,
information security for small and medium businesses in central Texas and surrounding areas.
I'm also a very proud member of the packet hacking village where I run the emerging threats,
honeypots and deception system. So if anybody here got to play in the honeypot challenges, that's
mine. I built that. I built it. I built it. I built it. I built it. I built it. I built it. I built it. I
built it. I built it. I built it. I built it. I built it. That's enough. That's me. With the
help of some really awesome people. I'm a cyber patriot instructor because I believe that anyone in
this field, IT, security, doesn't matter, pay it forward because those of us were once in junior
high in high school didn't have somebody to come and teach us this awesome stuff. We had to
figure it out. Well now you can be the one teaching the next generation of professionals. I'm a
part-time blog author. I don't write nearly as much but security against obscurity is my blog.
You see the link there. And lastly and most importantly I'm just a hobbyist, teacher,
student, librarian, بعah, politician. Thank yous. I'm the post graduate here in Massachusetts. And I've been
just like the rest of you. And as you can see, everything revolves around caffeine. So ask
yourself this. And be honest. Don't tell me, but tell yourself the truth. Is your IR team ready
for a worst case scenario? How would you know? You know, how often do you encounter worst case
scenario? We're not talking about, you know, macros and viruses and e-mails. We're talking
about APT on the network, ex-filling sensitive data. What do you do, right? You know, does your
team have the skill sets that are required to operate in an environment in a situation like
that? More importantly than the skill set, but do they know how to work together as a team when
the shit hits the fan? Or do they all run off into separate rabbit holes and stop talking? Which
is probably more likely. That kind of rolls into the next one. Do they communicate? Do they
communicate? Because I know how my team is. Well, the multiple teams I'm on, when they get to
chasing something, they go quiet. They go silent. Verbally and, you know, if you're in Slack or some
kind of messaging platform, quiet. We're in the middle of a breach. Why are we quiet? We should
be talking. We should be sharing information. I know you're seeing things. I'm seeing things.
He's seeing things. Why aren't we talking about it? So how well are they communicating? And then
lastly, organization, right? And I'm talking, yeah, the boring stuff, the administrative, the
hierarchy of the incident response team. Is there a way to communicate? Is there a way to
communicate? Is there a leader? Does that leader know his role? Do the followers know their
roles? And are they properly executing? Or is everybody just kind of free for all and I hope you
find something. Hope I find something. So those are the things you should be asking yourself.
Because look, even if your daily job is working in a sock, you're like, oh, I'm not new to this. I
respond to incidents all day long. Yeah, you're responding to those routine, mundane incidents
that you can do with your eyes closed, right? But it's a totally different situation when you
find out that there is a way to communicate.
It is a persistent threat on the network. Maybe it's been here for three months and you just found
it. That's not mundane. That's not routine. And neither should your response to such an
incident. So you need to ask yourself, is your team ready for that, oh, shit, the data center's
on fire situation? Obviously, the point that I'm making here is that a tabletop scenario cannot
prepare you or your team for this. Now, it's better than doing nothing. It's better than doing
nothing. I will give you that. To have a sit down and talk to your team and say, okay, guys, here's
what's happened. There's a hacker on the network right now. And he's attacked our SQL database. And
we're pretty sure he's exfilling records right now over DNS. What would you do? We'd block him.
We'd stop him. We'd put in a firewall rule and we'd stop him. Okay. Cool. Awesome. Prove it.
Like, you know, how would you, one, even have detected this to begin with? Two, how would you
identify what is even happening? You know, like, okay, great, we see a huge, you know, influx of
DNS activity. But we don't know what it is. Hackers aren't using clear text. Wall of sheep victims
use clear text. But, you know, how are you going to figure out what's really happening and
respond accurately? Because in a business environment, you can't walk in, you know, the bull in
a china shop mentality and just start throwing firewall rules in. You stop business, you're in
trouble, right? Now you've got two incidents on your hands. So, you know, tabletop is a
great, it's a good start. It can help you identify weaknesses and gaps in your capabilities or
your technologies. But it's not enough to really validate. Can you function with a sense of
urgency when it has hit the fan? Are you validating the procedures that you rely on so closely?
Because that's another thing I hate about tabletops is, okay, what would you do? You're getting
DDoSed. What would you do? Well, we'd follow the procedure. When's the last time that procedure's
ever been put to the test in a real DDoS? No way.
Oh, some CISSP wrote it, hoping it was going to be effective. It's never been tested. So
you're waiting for the day that you actually get DDoSed to find out if your procedures are
going to work. Skills, training, right? I have a sock full of really sharp analysts. I mean, I'm
so lucky. I was really, really lucky to have the team that I have. But let me tell you something.
Even with the experience that all the people on my team have, years and years of experience, the
first time we went through our scenario, the first time we sat in the simulation, the first time we
had a breach, no kidding, this network's being hacked, you'd think they were all beginners.
Because all those years of experience and knowledge and all that stuff, for some reason it almost
goes out the window and all of a sudden you're panicking. Oh, my gosh. Like, stop this attack.
Like, yes, they're exfilling records. Even though it's a simulation, it's funny because it has the
same effect. You put a time constraint on this, be like, you have six hours. Stop the breach. It
really does something to the heart rate, you know? And so my team that's, you know, we've got a lot of
experience, not new to this, responds to incidents all day every day, all of a sudden couldn't
remember that Palo Alto has many different tabs to click on to see traffic logs, threat logs, URL
filter logs. They forgot all of that. And they did not find the exfil. So validation of your
skills and your training, you know, tabletop is not going to cover that. And then lastly, right,
tabletop, since you're not validating or actually doing anything, it's not going to highlight where
your technology might fail you. It's not going to highlight where your personnel might fail you.
Might not have enough coverage. You've got one guy on your incident response team, sure, he's
going to do great at a tabletop. He just has to answer questions. And I'll throw him into the mix
with a real incident and see how many hats he's going to have to wear and how quickly he's going
to quit. It doesn't work. And, you know, one quick point. Think of this, the tabletop, why it's
inadequate. Let's take something like paramedics, for instance, first responders. You know,
imagine if first responders trained to respond to life-threatening situations simply by sitting
down with colleagues and talking about it. You know, hey, you drive up on an accident and there's
someone that's been ejected from a vehicle. They're bleeding profusely. What would you do? I
don't know. I'd put on a bunch of bandages. Okay. Cool. Awesome. That's what we'll do. No, that's
not how it goes. They go and they train. Is anyone here CPR qualified? Awesome. What did you have
to do in your training to become CPR qualified? A lot of fun.
Physical testing. On a dummy, right? Why couldn't you have just watched some videos?
Taken a CBT. Muscle memory. Absolutely. Because there is no CBT, there's no video that's going to give you
that sense of feel. What does it feel like to do chest compressions, right? What does it feel like to
breathe air into an unresponsive person? If you've never felt that, you've never done that, you're not
going to be ready when someone's dying. You're not going to be ready when someone's dying. You're not
going to be ready when someone's dying in front of you. Your heart's going to be racing. You're
going to be panicking. You're going to forget the video you watched. But you know what you won't
forget is that muscle memory. That's going to come in handy. That's going to come in handy. So
it's the same concept. It's the exact same concept. Firefighters don't learn to fight fire by
watching videos on YouTube. Because when they run into a burning building, fear sets in. Panic.
You know? So that training has to be muscle memory. It has to be instinctive. Incident response is
the same way. And no, I'm not saying that incident responders, you know, are paramedics or
firefighters know lots of respect for what those people do. But incident response is very similar
in the nature that if you don't know how to do it in the calm, you're definitely not going to know
how to do it in the chaos. So let's get to the good part. And just so you know, this is not a
very slide-heavy presentation. I didn't want it to be. So I'm going to pull up some diagrams and
I'm going to talk about them for a while. So I hope you all are good with that. So my advice for
how to effectively train for digital forensics initiatives is that you have to train for
incident response. Is attack your own network. Sort of. Obviously we don't want to do this in
production. So we build it. Right? We build an environment. And let me tell you something. I was
able to build a pretty sizable environment on two bare metal ES6I servers. We're not talking
about a cluster. I'm not talking about a full rack in a data center. I'm talking about two one
use servers. I was able to build a very, very high fidelity and elaborate environment. So this
is something you can do on a budget. This is something you can do with yesteryear's hardware that
you've replaced, you know, with your latest refresh and it's just sitting in a closet somewhere. Grab
that stuff. Bring it into an area and start building out an environment. And I'm going to go over
some of the things that you're going to want to think about when you're doing this because, yes,
there's a right and a wrong way to go about it. So I'm going to kind of start from the left and work
over. So my decision was to actually have the ability to connect this range up to the no kidding
net. And the reason that I wanted to do that is because that makes it way easier for me to have
organic traffic generation. Because think about it, right? If you're responding to an incident and,
you know, you're combing through firewall logs and all kinds of good stuff, it would be really easy
to find a bad guy if there was no web traffic. No activity at all. You just sit there and just
wait for the first log to roll by and, oh, there it is. We got it. No. For a high fidelity
environment, you need noise just like in the real world. Who here has ever tailed the logs of their
firewall? Maybe once, right? Never did it again because you're not going to find anything.
Because there's just going to be millions of lines just screaming by. So you need to reproduce that
or else you're not doing good training. So I decided to have, okay, I wanted the ability to connect
this range up to the Internet because when I get to the part later about how we generate traffic,
you'll see that Internet access is kind of required. However, I'll give you a trick here in a minute
just in case you don't have that Internet access. It can still be done. So I'm going to go ahead and
do that. And also I wanted to share a quick resource. So while building this, I had to solve some
problems like how do I generate traffic. I wrote a Python script. It's in my GitHub. If you can just
remember my first initial E and my last name, which is almost impossible to remember, you can find
me on GitHub, web traffic generator. It's just a little simple Python script that you can feed a seed
of URLs to. And it will literally headlessly browse the Internet. And it will go a specific depth. So say
start at these five websites and go to a depth of ten. So it will start at yahoo.com and it's going to
click ten links in. And so it's a really cool way to generate organic web traffic. Because, guys, you
cannot leverage canned TCP dump like playbacks. And let me tell you why. Because what we're using
here, just like in our real environment, are stateful layer seven firewalls, right? Layer seven
firewalls, well, actually, any stateful firewall is not going to play nicely with your canned TCP
dump traffic. You're not going to just be able to replay PCAPs and have these firewalls pass them.
It's not going to work. So unless you've got a breaking point, which I usually don't drop brand
names, sorry, but unless you've got a very expensive system that was built to do that, you're not
going to be able to. So what you need is you need real traffic. You're not going to be able to fake
this because your firewalls are not going to care. So anyway, so traffic generation, and I have
another tip for you guys on that one in a minute that's even easier than my script. The next
component in this mix would be what would be your external and your external router, but this
dotted line right here indicates kind of where this range starts and stops from the perspective of your
C-cert responders. They are not aware of anything to the left of this dotted line. So this router
here serves a very unique purpose. You're using that, and that could be anything. That could be a
Vios router, previously, or Viata. That could also be a PFSense firewall. You don't need any special
gadgetry here, but you do need something that can route because what you're doing there is route
manipulation. So again, right, the effectiveness of this training is going to be how well can we
emulate the things that you're going to see in the real world. Well, it's really hard to emulate C2
traffic to Russia unless you have a VPS in Russia. So what we can do is in this router here that
you're, you're, you're, you're, you're, you're, you're, you're, you're, you're, you're, you're, you're
you're, you're participants aren't even aware of, in this router we're setting up some
destination NATs, right? You can NAT an entire slash 16, if you wanted to, of, you know, a known
Russian IP space. You can just NAT it to a server that you've got sitting up here in your fake
Internet, you know, which is just another V-switch, fake Internet. And so what I can do then is
I can have C2 of any of these systems here, going to a no kidding Russian IP. But it's not even
leaving my range. It's just once it hits this location. You've got a different IP network for
router it's getting destination added to a server that I've got listening up here. Why is that
good to do? Why is that useful? Well, because we want to generate the same or similar types of
artifacts that an incident response team should be looking for. So if you can't simulate traffic
to Russia, well, I mean, you know, that's definitely something that you're, that would add a
lot of value to the training. So, but again, I mean, Russia, that was kind of an obvious
point, but maybe something less obvious, you know. But the point is, if your organization
doesn't do business outside of the country, and you throw something in there, even something
simple, you know, Great Britain, UK, France, whatever, it should still be anomalous. If your
incident response team is looking at it, hey, why are we talking to this country that we don't
even do business with? So anyway, so the other thing that we're leveraging this router here for
is you can hijack DNS. So if for whatever reason you want to stand up, you know, a site, like a
paste bin clone, which actually we did in our scenario, we created ponbin.ru, which our red team
infiltrators use to exfil data. Well, to create that record outside of this environment, because
you don't want your incident response team to log into the firewall and be like, oh, look, there's
a static DNS entry for ponbin.ru. I bet that's going to be used for something. You create it out
here. You hijack DNS out here. Maybe you get a little DNS server up here that's responding to
those queries. I could hijack Google. I could hijack Yahoo. I could hijack anything I want. So
now I can further manipulate what's going on in this environment to make it do what I want it to,
to achieve my training objectives. The next thing, oh, and also I wanted to highlight some of
the resources that I'm using in my environment. So obviously this is also where your attackers
live, right? So you're going to need someone to red team this engagement. So if you actually have a
So your red team guy literally could have one Metasploit box,
one Cali system sitting up here in the fake internet and he's
got a handful of interpreters on it and all the C2 and all the
bad stuff that's going on in here is literally, it could
appear to be going to 100 different IP addresses but it's
all getting added to this one guy up here and one dude is red
teaming this entire engagement which is also a ton of fun. But
you know Apache web server running a bunch of virtual
servers for your special sites and services, you want to fake
Twitter, set up a fake Twitter up here. You want to fake
Facebook, you know you can do all of that up here using very
simple things. Another thing that I'm doing in my fake
internet is, so to really add additional realism to this
environment, email, right? Email is the number one vector. It is,
that's how they get in. Social engineering, someone got an
email that they trusted, they believed in, they popped it
open. So you need the ability to send email into this
environment. So we've got a Python script literally that
is just sending lorem ipsum every minute to every single user
in our fake environment. That way when we send that one
malicious email through, it just blends in. It just blends
right in with all the other noise. So, you know, we're
leveraging some pretty simple concepts here. Let's see. Okay,
so the next piece in this puzzle is your perimeter firewall. Now
here's the thing. I meant to touch on this a little earlier
but, so here's the thing. When you're approaching this and
you're going to build an environment, you're going to
need to figure, there's a component you need to figure out
first. If you're building this on the cheap, obviously you're
going to lean more towards open source solutions, which if you've
been to my previous talk, I'm all about. Open source is
awesome, right? A lot of really powerful commercial tools run on
open source engines, so what's the difference? However, know
that, you know, the effectiveness of the training is
going to be tied to how closely does this mimic your production
environment. So there's a lot of value in trying to reproduce. So
for instance, if you use, you know, FortiGate firewalls, it would
probably be a good idea to try to get FortiGate virtual machines in
here and use those because that is another way to, one, train your
incident response team, to, two, validate how good are my FortiGates
anyway? Have you ever really put them to the test? Do it in
here where the stakes are not high. Well, know that you don't
necessarily need to go out and buy licenses for all this stuff.
Tell your vendor what you're doing. Hey, John, FortiGate, listen, man, I'm building a training environment.
To make my guys better at what they do. Can I get a complimentary
license? I'm going to tell you right now, guys, I've probably, I've
probably been given $200,000, $300,000 worth of free licenses for my
training environment. Vendors don't care. You tell them it's a lab, it's
offline, it's not production, you're just training with it, they will
throw product at you. So don't be afraid to ask, okay? So when you're
approaching this and you're deciding, do I want to use, you know, Palo Alto, FortiGate,
you know, ESAs, whatever, talk to your vendors. You're more than likely going
to be given some leeway there. But if you do go the open source route, I'm going to tell you
right now, you can achieve everything that I'm talking about here with open source solutions.
Short of maybe your Windows workstations that you're going to want to break into over here
eventually. So moving on. So perimeter firewall, you can, you can totally roll like a
PFSense if you want to, or you can throw, like I said, a virtual appliance of, or even a
physical appliance. If you have a little, you know, lab unit sitting around, fork that into
this environment and use that. But what that perimeter firewall is
going to do is just like in your production environment, it's going to have a DMZ probably
with some web applications, some web servers, things like that. Maybe hang a male security
appliance. Whatever you're doing in production, you need to do here. That's, it's really the
extent of it. And hopefully just like in production, you're running IDS IPS here, right? To
simplify, the next, the next piece of it is really just like one giant core router to rule them
all. And, and it could be, it could be, you know, a copy of the same device you've got here or it
could be a different device. It doesn't really matter. But that's where you're going to create all
your different virtual networks to mimic, you know, your different branch offices or divisions or
whatever, whatever subsections you have in your organization. And the reason you want to do this
is because, you know, remember, during our incident response training, we want to create
anomalies, right? We want to create things that are out of the ordinary. Like someone in HR, you
know, hammering a SQL server. Well, in order to mimic that, we need to have an HR subnet, right?
And then the C-cert needs to understand that that is, you know, what that is. On your core
router, it also would be ideal, hopefully, just like in production, to have IDS IPS here so that
you've got visibility on not only your north-south traffic coming into and leaving the
environment, but east-west visibility on HR talking to the server farm, you know, or IT talking
to marketing and sales, right? Now, again, this is cool because when you're architecting this,
this very simple network, it starts to prompt questions in your mind of, well, do we really have
that visibility in our environment? I don't think we do. You know, we've got a giant, you know,
Cisco Nexus, you know, big iron router in our core, and it's not inspecting anything. It's just
routing. So we wouldn't know if HR was beating up a SQL server. So this is cool because you're
going to accidentally start to think about the gaps you've got in your own environment as you're
thinking about how to build this one. But for the fidelity of the training, of course, you're
going to want to have these sensors deployed so that they're inspecting everything because we
want to have that flood of information for our C cert to have to sift through to look for evil. So
server farm, obvious, I mean, I don't really need to go into detail to you guys about what needs
to be in a server farm in a corporate environment, but you just want to have those important items
like a domain controller, exchange slash mail server, right? A file server of some kind,
CIFS, NFS, Samba, whatever. And, you know, in my environment, we rolled a SQL server, you know,
because most organizations have large databases, and so we wanted to have one, too. Branch
offices, self-explanatory, but I'm going to go over some cool tips and tricks here. So back to
that organic traffic generation, right? So first and foremost, you know, what do you put out
here? What do you put in these subnets? Well, ideally, you're going to take a copy of the golden
image that your organization uses. You know, IT has probably some image that they use to image new
systems. Get a copy of that. Use that one. And put it in here. Because now, you're going to have a
copy of that image. Now, during your training, you're going to be validating the security of that
image. How secure is that image? Have you ever really put it to the test? Have you attacked the
crap out of it with Metasploit? Well, you're going to get to in this environment. So when you get
all these workstations set up, Windows boxes, you know, whatever your folks are using, I'm going
to share something really cool with you, save you some time. There's a Chrome extension called
Chaff. And this thing is awesome. And it literally, it, you know, my traffic generation script,
it does that. It does it actually even better. And, of course, it's a little bit more complex.
Of course, I didn't find Chaff until after I'd spent, you know, like seven hours writing mine.
But Chaff is actually really cool because you can install Chrome on each of these workstations. You
can tell Chaff, you can tell Chrome to start at boot, auto log in, start at boot, and Chaff will
start automatically. And Chaff will literally do everything a user will do. Browse the Internet,
download dumb files, go to YouTube, stream some stuff. I've seen Chaff end up in some pretty
risky places of the Internet, which is perfect. Because during your incident response simulation,
you're going to see IPS signatures firing. You're like, I don't know what that is. It's one of the
Chaff users. And that's perfect because that's what real users do. They do dumb stuff and they're
going to go to weird websites and it's going to make your firewall upset and that's going to make
your incident responders have to sort through the noise, right, which is what it takes in order to be
effective at incident response. There's got to be a lot of noise. So Chaff, that's one of my little
secrets there. But also it's important that whatever endpoint agents you're using, if you've got an AV,
IPS, HIDS, whatever, a log forwarder, you want to get that on this image as well, obviously,
because you're going to need data. You're going to need information off these workstations going to
your aggregation, your SIM or whatever. We'll cover that in a second, too. So, you know, get
those endpoints configured. You're going to need a mail client on them because, remember, mail.
Got to have it. You're going to use it in these simulations, the scenarios that we're going to go
over in a minute. And then obviously you want all these guys joined to a domain. So you're going to
have to build out your subnet for your incident responders. And so typically, you know, I hang
that off the same core router, but it is a totally separate subnet. That way, you know, the
ACLs, it's permit, permit, you know, any, any, all that good stuff, no IDS, no IPS, because you
just want your guys to be able to have a straight shot into the environment. Because later on, as
you mature here, right, after you get past the stage of detection, you're going to be able to
find the evil. Can you find the evil? Okay, great. You found it. Now go ahead and, you know,
leverage your Wemmick, your PowerShell, whatever. Reach into that box. Let's get volatile data.
Let's get event logs. Let's get a memory image. Let's, you know, then you can actually start
getting into active response on these systems. And so it's good to have them on a separate subnet
so that you can, you can permit them to do all of these things. So I'm going to go over a couple
more, a couple more solutions that you guys will want to look into for people that are interested.
Building out your environment. And actually, to be honest with you, these are solutions you can
look into putting into your production environment. So if anyone here is familiar with Elastic
Search, right, Elastic Search Logstash has a really cool log shipper called Beats. And there's
two flavors of it, actually. So there's FileBeats, which is just an agent you can put on a
system. And you can point it at any log that you want. It could be a custom log that you wrote for
custom application. Or it could be a Windows event log. It could be syslog. It could be auth. It
could be any log you want. And FileBeats can help you get that log out of your system. And so
you can actually ship those logs to a central repository using TCP with TLS. Like all kinds of
cool stuff. So it's kind of a step up from your simple rsyslog forwarder, right? Which is UDP.
It's plain text. It's, you know, what happens if your syslog server goes down? Well, syslog
keeps forwarding. It doesn't care. It's just going to keep forwarding. Well, Beats is a little
smarter than that. If your syslog server went down, Beats is going to sit there and it's going to
store those logs and wait for it to come back up. So really cool tool. And it integrates really
nicely into Elasticsearch Logstash. Clam AV, which if most of you guys are familiar with
Sourcefire, it's Clam AV under the hood. Who knew? Osec, which is an open source HIDs agent. Very,
very, very, very powerful. This is an awesome tool. Millions of customization options. You put
it on your host and it's looking for rootkits. It's looking for evil. And it's reporting back to
a central server. So I'm just going over some of the things that you can look at using in your
environment that cost nothing.
How about your incident response team, right? Maybe you're in an organization with a ton of money
and you've got a really cool SIM. Maybe you're not. The good news is you're not out of luck.
Greylog. Very, very powerful tool. I mean, it is very close to an open source Splunk. Not to be
confused, Splunk is awesome. Splunk is extremely powerful. But Greylog comes pretty close and the
price is right. $300,000.
Alien vault OSIM. Anyone familiar with OSIM? Extremely powerful SIM. It's got vulnerability
scanning, passive asset detection, all kinds of good stuff. You could stand that up in this
environment for free. The Hive. You saw my talk yesterday. I covered the Hive. Really awesome
incident response collaboration tool. So if you've got a team of analysts that are responding to an
incident, the Hive is an awesome open source platform for them to collaborate on. Check it out.
So I just wanted to share a couple tools that you guys could use if you didn't already have these
systems in place. Okay. So now that you have your environment, now you have to start scripting your
scenarios. And this is one of those things where, you know, it sucks because you might have a team
of ten. But you or someone you designate that's going to have to be the lone wolf that's gonna
have to come up with these scenarios because, obviously, your incident response team can't know
about it. Right? It's got to be a surprise. So I actually have a tourney where, you know, we're going to be
had my Intel analyst draft up these Intel reports. They're totally bogus. I mean, it's like
95% fluff and about 5% actual Intel, just like in real life. Right? So we dreamt up this
scenario. And it's important that you pick one that's applicable to your organization, right?
Like, so you know what kind of threats your organization faces. I'm a law enforcement
agency. We deal with hacktivists. You know, people that don't like law enforcement
agencies. And so we dreamt up a hacktivist scenario where there's this, you know, this group,
you know, like anonymous or something that really wants to embarrass large law enforcement
organizations. And we got all their TTPs and some observables that have been discovered in the
wild and all this stuff. And in all reality, we really are only going to use one or two of the
indicators that we provide to these guys. Because just like in the real world, I mean, these
Intel reports, they're not smoking guns. They're not using a gun. They're not using a gun. They're
not going to give you all the information you need. But this is to test your team. Are they
paying attention? Are they looking at the relevant Intel that's available for a particular
incident? Or are they like, oh, yeah, great, yeah, Intel. Okay, cool. No, I can find bad. I've
done this before. I'll find it. So this is a test on are they ingesting the information that's
available to them to make them more effective at finding a particular threat to the
organization? Yes. And we'll go over in one of the scenarios ways that you can do that. So the
first one is you can really weigh in on, you can test whether or not they're paying attention to
things like this. So now I kind of want to dive into a couple actual scenarios that I've
developed that were really successful in training our team. So first and foremost, though, I want
to preface with this. Your red team injects. So the attacks that you're going to carry out on
your environment, it can't be cowboy. You can't just, you know, wing it. It needs to be
scripted. You need to have a playbook that has repeatable and measurable impacts because
for it to be an effective training solution, for every red team inject, there needs to be a blue
team observable. So as you're going through and you're mapping out your scenario, you literally
need to wear both hats. So, okay, I'm going to try this attack on that system. What did I see on
this system when I did that? Oh, okay. Or what did I see in my sim? What did I see in my logs?
What did I see here and there? So you need to make sure that everything that you're scripting for
your injects is something that your blue team can't even find. If they can't find it, why not? You
know, that's a gap in technology. So another thing, you know, if you go off script during a red
team engagement on, you know, in an environment like this, well, now your blue team, like, you're
not going to know if they found the bad stuff or not because you don't even know what artifacts
you're generating. So that's why it's important to script this out. So that's what we have. We've
got playbooks for different scenarios that we're going to run through. So, you know, for
instance, here's a, you know, and I'll just kind of blaze through these, but one scenario that we
have is we're going to kick off our traffic generation. So we've got a ton of noise and we're
going to get emails flooding into the network every minute. I mean, hundreds of emails. We've
got traffic generation. And then we're going to socially engineer a DBA. We've got a database
administrator, poor Regina Buckley. She's been with the organization for years. And, you know,
not the most security aware, but she does all right. Well, we're going to phish her because
she likes talking about what she does on LinkedIn and Twitter and Facebook. And so we found her.
We're going to spear phish her. And we're going to send her an e-mail. We're going to send her an
email that very urgently tells her to install a Windows update from vendosupdate.com, which,
it's funny, you can, you know, when you're manipulating DNS, you can create any type of DNS
record that you want. So VVIendosupdate.com plays really well into your scenario because a lazy
analyst that's just kind of skimming through logs doesn't see the VV, right? The simple things
that attackers do. They're playing on your lack of attention to detail. So when your red team
injects, you want to do the same thing. Do things that are going to force your analyst to do the
wrong thing. You want your analyst to pay attention to what they're looking at, not to glaze
over it. So she downloads this update, which actually, of course, is a reverse shell that's
going to give our red team attacker a shell on her system. So she's here. We're here. We now
have a shell on her system. Escalate privileges. Get her credentials. We already know she's a
database admin, so we already know what accesses she has. So what we're going to do next is pretty
obvious. We're going to start enumerating the network. Now, remember, this is why it was
important that we have east-west inspection because if we just had a direct link to the SQL
server and we started scanning, our blue team's not going to see that, right? So we have
inspection for east-west, and we're intentionally going to generate some noise. We're going to
scan. We're scanning the server farm for the presence of a SQL server. That's a blue team
observable. You know, port scan, 3306. They should find that. They should see that. Well, we
found it. We found our SQL server. So from Regina Buckley's system, we now have our creds. We're
just going to go ahead and log right into that SQL server, and we're going to start exfilling
records. Now, here's the thing. Anybody here, especially red teamers, there are intentional
dumbing down of red team activities because, again, when you're first getting started with
those guys, your team doesn't know what's going on, okay? So if you go in all stealth, you're
using an RC4 encrypted payload, and you're exfilling over encrypted channels, yes, of course,
your team's not going to find it. So you need to do some things earlier on to enable them, to
build up their confidence, to help them find, to know what to look for. So what we're going to do
is we're going to grab all the SQL records, and we're just going to pull them over to Regina
Buckley's system, even though we could have just pulled them right through our meterpreter shell
and been gone. We're going to pull them to Regina Buckley's system. Through our meterpreter
shell, we're going to drop curl onto her box. And remember, in our fake Internet here, we stood
up a pastebin clone called ponbin.ru. From Regina's box, we are going to, via the API for
ponbin, we're just going to post 5,000 sensitive records right to ponbin. Because remember, I'm a
hacktivist. I'm not in this for the money. I just want to embarrass the hell out of your
organization. So I'm just going to post 5,000 sensitive records right to ponbin.ru. So what
additional observables have we generated? Well, we dropped a binary on the system in a protected
directory. When we exfilled data from the SQL server through the network to Regina Buckley's
box, if you've got a halfway decent IDS, it picked up on the fact that 5,000 PII records just
traversed the network in the clear. That should be noticeable. Then we did it again. We curled in
clear text from Regina's system via an API. We exfilled all 5,000 of those records. Now, did we
do it the same way an attacker would? No, of course not. An attacker would not have used any of
those methods. But remember, to start, your team, you're lucky if your team is going to see any of
that. If your traffic generation is good. If all the noise is there. If your team sees the
scenario I just described. If your team sees the scenario I just described, if your team sees the
scenario I just described, you're better than most. So you've got to practice with the
intentionally obvious scenarios first. Let your team get a feel for what does abnormal stuff look
like. Then you upgrade to the more, you know, sophisticated attacks. Which, if you're not
gathering this already, this is a ton of fun for the red team, too. So, you know, you build this to
train a blue team, but that means that some lucky person gets to be the guy attacking these
systems. And that's a ton of fun, too. So, you know, you build this to train a blue team, but that
is one scenario that we've had. Yes. Absolutely, step by step. Literally copy and paste into the
meta split console. We even pre-built meta split resource files to automate a lot of the things
like... Hey, enumerate SQL database. Well, you want that to be 100 percent repeatable. So I might
have one way of enumerating a network. You might have another. But we can't have those
variances when we have a scoreboard that's expecting a specific answer. How many
systems did the attacker scan while enumerating the network. That's why it needs to be 100%
repeatable. So yes, the playbook is extremely detailed, down to the command that the red team
is going to run for the inject so that we know exactly what the blue team should have seen.
So as we start to migrate to more, I guess, more complex scenarios, excuse me, we're
dehydrated here. So or I wouldn't say complex but different, right? A lot of organizations
have, like I said, web servers sitting out here in the DMZ. Well, how sensitive are the systems
that are in front of those web servers? If you've got F5s, if you've got IDS tipping points,
whatever, are they properly tuned? Are they configured to look for the types of attacks that
your systems are most vulnerable to? Well, try it here. So you get a web server, right,
that's facing the Internet. But it has to be a web server. It has to be a web server that's
a backend connection to the SQL server. Well, maybe that's another scenario where you get an
attacker that's beating up this web server, running SQL injects against it in order to do the
same thing, to exfil data, to exfil data. Would your team know what to look for, know how to
respond to that type of situation? Now this is where things can get a little more complicated,
right? When you want to start really upgrading to more sophisticated scenarios to put your
team to a tougher test, that's where you do start using things like RC4 and other things like
tougher test, that's where you do start using things like RC4 and other things like that. So if you
have more encrypted payloads that are not going to set off IDS IPS, that's where you start using
encrypted channels over, you know, normal web ports, things like that. Things that are not going
to stand out, they're not going to be obvious, you're not going to be scanning the entire
network, making a ton of noise, you're going to be super quiet, super covert, just like a real
attacker would. But then how do you throw your team a bone to see how closely they've been paying
attention, right? Maybe in a previous scenario, the Intel report had a bunch of IP addresses and
none of them were relevant at the time, just like in the real world, right? But two scenarios
later, one of them is. But they haven't even looked at that Intel report in hours. It's in the back
of their mind, they're not even thinking about it. But all along, you've had a really slow and
quiet beacon sitting on a system on the network. And while your team is scrambling, looking for
things that don't exist, because, remember, we're on a stealthier engagement this time, you're
sitting there doing its thing and you've got an attacker doing some really dirty things. You
name it. Your imagination, the sky's the limit. Does your team ever connect the dots to say, wait a
second, the threat Intel we got six hours ago that wasn't relevant then is relevant now. We've
had C2 on this system for all day long. Connecting dots like that, that is analysis. Paying
attention to the details. Not forgetting about previous indicators just because maybe the
weren't relevant at the time. That's another effective way to test your team to pay attention.
This was a really fun one, actually. This one I'm describing. What we did was we wrote ransomware
and we had a persistent beacon on one system. We dropped the ransomware executable onto the box.
We waited until things got really quiet. The incident response team was distracted looking for
rabbits in the rabbit hole. And then we went ahead and we hit the file server and we encrypted
every single file on it. And here's the thing. We had given them Intel of exactly what we were
gonna do. Obviously you saw the Intel document. It didn't look like a really friendly, straight to
the point manual because they're not. But we gave them an Intel document that said exactly what
we were gonna do. They didn't see it coming. What if I walked up to you and said, hey, I'm about
to punch you in the face. You have no right to be surprised if I then punch you in the face. I told you
you I was going to do it. But if you're not listening to me when I tell you, if you're
not paying attention, if you're ignoring the signs, you're ignoring the intel, you're not
diving deeper into the information you have, you are to blame. And so test your team. Are
they using the information that they're given? Are they looking out to gather additional
information? You could go so far to use real indicators. So go to OTX, Open Threat Exchange,
and find some IP addresses that are tied to malicious activity. It won't be hard. And
then use those IP addresses in your environment. Now, make sure that your blue team knows
that, hey, guys, just so you know, the indicators that we're using are real world. So now they
know that they have an option to go and research on OSINT sources. Okay, I think this activity
is malicious. I'm not sure. Let's go check virus total. Let's go check OTX. Let's go
check, you know, Threat Connect, whatever. And now they can actually enrich the intel
because you're using real indicators. So, I mean, you know, put your folks to the test
on the internet.
And then you can go and check on some of the more advanced capabilities that you need
on an incident response team. Because just simply playing IP address whack-a-mole and
blacklisting things all day, that's not effective. That's not solving problems. So
you get these scenarios written and, you know, you've tested them, you've validated
them. Like I said, for every red team inject, they're
going to need to be a blue team observable. How do you measure? How do you validate? And
then at the end of the day, how do you make this fun, right? Because if it's not fun,
no one's going to want to do it. You're not going to want to run it. Guys, this is
a ton of fun. So when I built this, I built it actually just for my own incident response
team. Then B-Sides Austin reached out and said, hey, we want you guys to bring this
platform to B-Sides Austin. I'm like, okay, yeah, sure, I'm not going to say no to that.
We brought it to B-Sides Austin, and we had over 40 teams come and play through our
scenarios. We've had rapid seven guys play through our scenarios. And they loved it. We've got
overwhelming feedback about it because, one, they said, hey, this is awesome because this is
real world. Like this is exactly what it's like, you know, being in a SOC, looking at incidents
versus a CTF, which, don't get me wrong, CTFs are a blast, but, you know, a lot of CTFs, you
know, puzzles, the games, the challenges, the decrypt, you know, all that good stuff, that's fun.
Not directly applicable to what we do on a day-to-day basis. Whereas this challenge was
meant to be exactly that. It's meant to be directly applicable. But we still want it to be
fun, enjoyable. We gamify it a little bit. You pull down an open source CTF scoreboard, like
this one here, this is actually Google's. This is the Google CTF framework. And you gamify it.
You create a Jeopardy style, you know, question and answer board where you break out the
categories and, like, you know, and I hate using this word, but you basically, you know, you
basically create your own kill chain. You say, okay, you know, the first phase of the attack was
the infiltration, you know, then there was the enumeration, then there was the attack, or, well,
actually, attack came before that. But then the exfiltration, right? So you break it out into
those categories and you make them run through the entire scenario. And you give them little
breadcrumbs, right? Because, again, it's all about enabling them. You're not going to get any
value throwing your team into a scenario that you know is over their head and giving them
nothing. So when they're learning how to do it, they're going to learn how to do it. They're
learning how to walk, you show them how to crawl, point them in the right direction, and then
get progressively more difficult as you go on. But, like I said, the CTF scoreboard, something
like that, it's going to gamify, it's going to make it a little more fun, but it also, as someone
leading the training, it's going to give you a true measurement of how well is your team doing,
how far are they getting. If they're doing it as individuals or if they're doing it as a team,
you need to be able to measure the impact that you're making. And that is all I have.
So I will open it up to questions. I know that there's a lot of technical information in here, and
in the essence of time, I have to kind of skim over a lot of things that were actually six months'
worth of development and implementation. So I'm perfectly happy to have a conversation with
anyone that wants to know more about how to build this, how to run this. I'll even share my
playbooks with you so you don't have to reinvent the wheel. Just let me know. But any questions?
Yes?
So are you talking about, like, communication channels to management of the organization, or?
Oh. Okay. Absolutely. Absolutely. So, no, his question is good. It's like, are you including
management in these scenarios? So if it's an incident response, like, of a large scale,
obviously there's more than just the incident response team that's involved at this point, right?
You're dealing with the executive team. Yeah.
You're dealing with executives. You're dealing with legal. You're dealing with HR and so on and so
forth. So, honestly, that's not something that I've baked into our simulations yet, but 100%
should. Absolutely. This is something my team does on a quarterly basis, but we've only been
doing it for three quarters. So we're still growing. We're still evolving. But you're 100%
right that to really test, like, what would we do in this scenario, you're going to be including
your executives. You're going to be up-channeling. So that's kind of part of the organization
piece. You know, this isn't just about, you know, you're going to be up-channeling, you're going to be
testing. This isn't just a skills test, guys, because I don't doubt the skills of anyone in this
room or any of the people you probably work with. This is a test on when it hits the fan, can you
remember the basics? Like, I don't know, talk to someone. Document things, right? Or do we all
just fall into that silo and that rabbit hole of, oh, got to look at the logs. Oh, I think that's
bad. Oh, I think that's bad. Oh, I think that's bad. But I haven't written down anything. I
haven't documented anything. I haven't told, you know, other analysts what I'm looking at, what
I'm doing.
And I haven't up-channeled to the incident response lead. Hey, you know, I think I see
indicators pointing this way. And I might not be on task from what the incident response lead asked
me to do. So, yes, you've got to factor those things in to really test how effective an IR team
is in the midst of it. Yes?
So that's something that we're
still measuring. Obviously the major investment here for me was time, right? To build this, yes, it was a
sizeable investment of time. I'd say there were about three of us working on it. It took us about six
months. We were working on it obviously on the side, right? We have daily operation to maintain.
So, you know, for about six months we had to build this out. We had to learn a lot of hard lessons.
You know, we had to, you know, a lot of troubleshooting. Once it was up and running, it actually became
came very easy. Because now all we're doing is we're changing the scenario. Hey, instead of a
DNSX fill, next time we're going to do this, we're going to do that. So writing the scenarios now is
the upkeep. And that's actually probably the most fun and one of the easier pieces of it. So the
upkeep element hasn't really been an issue. Like I said, the hardware, I'm using recycled power
edges. These things are nothing special. So it was a very small investment in hardware. And like
I said, if you have a good relationship with your vendors, you can go and you can get OVA's or
probably even bare metal like lab units of the appliances that you're using in production and get
those in the environment for no cost at all. You know, get a perpetual lab license. So I mean you
can do this with a relatively small investment. The real kicker is going to be the time to build
the environment. If you don't have the resources on your team, you know, with solid networking
skills, solid sys admin skills and things like that, okay, you might have some resources. You
have a tough time. I'll be honest with you. You're going to have a tough time. But I think I
could say pretty safely that a lot of security folks came from other areas, right? Sys admin,
network engineering, things like that. And so, again, I'm fortunate. I have a pretty, you
know, wide skill set across my team. So I have network engineers. I have sys admins. And so it
enabled us to build this without any outside help at all. But your mileage may vary. Yes?
.
.
Okay. So that would be tough for me because I have 12,000 users. I don't have those types of
resources for this environment, right? I cannot simulate 12,000 people. So no, obviously I'm
definitely scaling down a lot. But I have found that as long as your traffic generation is organic
and it's good. Put in some other things. I think in addition to
the HTTP traffic, we also put some Python scripts that are
doing all kinds of other weird stuff. Pings, ICMP. Remember we
had a file share in the environment, right? So one of
the GPOs that we put in the domain was at boot all those
systems mount the file share. So now you've got a lot of east
west SMB traffic. So you want to make sure that you're
generating as much normal activity as possible. And that
takes a little bit of ingenuity. Because if you can do that,
you'll find that at least we found that it made plenty of
noise. And I think we emulate about 35 or 40 user work
stations. It makes plenty of noise. So it's effective for what
we're doing. Because if you can effectively filter out 40 active
users, it's the same concept to filter out 10,000. It's the
same filtering principles, right? So we found it to be
effective.
What time it is.
Oh, absolutely. Yeah. So from just a good security posture, you
should have sensors that are detecting user behavior, you
know, abnormalities, right? So, you know, Regina Buckley, she's
my DBA I like to pick on. You know, she's the one that's
doing it. So you should absolutely maybe script a
scenario where you're setting the time. Okay, guys, it's 1 a.m.
Regina Buckley is beating up the database. Now, Regina Buckley
is the database administrator. So at first glance, that might
be normal. But not at 1 a.m., right? So absolutely. Factor
these things in. The things that should be unusual, that
should be anomalous, that's what you're testing your team on
detecting, right? Another one that I like to use. So we have
a scenario that we called no worries AV caught it. I love this
one. I love this one because here's what we did. We sent, like,
a spam e-mail campaign to every user in the environment. And it
had a malicious attachment, like a gray ware, you know? And we
made sure that it was going to trigger AV. As soon as it hit
the box, AV was going to squelch and be like, no, you're not
going to run. What is your, in your sock today, or your IT
security folks or whomever is watching these types of systems,
what happens when you see something like that? No worries.
AV caught it, right? Who cares? Moving on. So, I'm going to do a
couple of things here. But let me tell you what the second half
of that scenario is. If you would have taken the sample from
AV that it caught and mitigated, if you would have taken that
sample, it was just visual basic, you know? You know, you
would de-obfuscate it. You would see that it had hard-coded C2
beacons in it, hard-coded URLs for C2 beacons. Those beacons
had been in use for multiple scenarios undetected. So what we
staged was as if an APT had already had a foothold, but they
were trying to increase the number of beacons that were in
use to expand it, maybe to another branch office. But they
got sloppy. They reused a payload that there was already a
signature for, et cetera. These things happen. I'm testing my
team to see how quickly do they disregard something because, oh,
it appeared to have been mitigated. Yes, that was
mitigated. But you failed to look deeper to see if you could
learn something from it that might help you find something
that wasn't mitigated, right? So, I mean, guys, the sky is the
limit. I probably could have just talked to you about the
different types of scenarios that you could do in this
environment. But, I mean, I'll leave it to your imagination. But
these are the things you have to think about. What would our team
do in a really freaky situation like that? Like, would we be
prepared? You won't know until you try it. You won't know until
you test it. You won't know until you've got the muscle
memory that it takes to do these types of things that don't happen
every day. Unless you're RSA. Just kidding. Kidding. Yes.
How do you handle the endpoints? How do I handle opening up?
Users on the endpoints opening up. Okay. That's a great
question. So, now, there are technologies that can do this
that can interact with, like, the DOM and, like, click on
things automatically. I'm not that cool. I don't have anything
like that. So what we do is what we call it's a white cell
task. So it usually falls on the red team guy. Or maybe if
you've got a sys admin that's helping run the range during the
environment, during the exercise, it would be a task item
for them. Hey, we just fished Regina Buckley. Log on to that
system. Open the e-mail. Click the link. Run the executable.
There are ways to automate that. I just I'm not there yet. There
are really expensive technologies and platforms that
can do this. Some of you may be familiar. I'm very familiar. So
my history I didn't really go into a good bit of history on
this. But my time in the air national guard was spent largely
in part going through simulations. Once a year I go up
to Virginia and run through an international cyber warfare
simulation. And the things I've seen there are what prompted me
to build this. Because I got there and I noticed that, hey,
what do you know? We're a bunch of people, very skilled. We
know what we're doing. But we don't do this but once a year.
And it shows. I want to do this more often. We need to get
better. This needs to be more, you know, of a comfortable
feeling for us. And so that's why I built it. But
environments like that, they've got all those cool tools. Larry
and I get, you know, there's all kinds of really cool user
automation to simulate these things. But see, I'm a state
agency. I've got to figure out how to do it on a budget. So
chaff. Chrome extension. Python. Gets it done. Gets it
done. Trust me. And then, you know, when you do have to do
that one thing like open outlook to click an e-mail and open
an attachment, that's a 30 second inject for your white cell
team. Yeah. Oh, but, you know, that reminded me of one point
though. There's one rule that I stick to when I'm writing a
scenario, and I would advise you to do the same. Do not give
in to the temptation to game the system a little bit. Everything
that you do should be organic. You should be using the same
tactics attackers use. Because if you do something out of the
ordinary, you know, and, you know, you might have a technical
limitation that means, oh, I need to side channel. I just
need to, like, load a CD-ROM on the virtual machine of this
workstation so I can put this binary on it undetected. That's
unfair. It's unfair to your instant response team. It's
unfair to your team. You didn't even give them the opportunity.
So you need to up your skills and how are you going to deliver
this attack. Because if you start side channeling and gaming
the system, it's going to drive down the value of the training.
It's going to demotivate your team. Because they're going to
say, really? You did it that way? We were never going to find
that. That's dumb. And they're not going to want to play next
time. So keep it real, you know. Keep your scenarios real.
But Ming's coming to kick me off stage. One more question before
we have a conversation out there. Let's do it. All right.
Thank you, sir. Hey, everyone. Bob Neumann.
Newhart, everybody. Bob Newhart.