Welcome everyone to the Blue Team Village incident response panel. We have an awesome group here for
you today with unique experiences and perspectives on security incidents and incident response.
I'm really excited for today's discussion. But before we get started, take a look at the poll
we posted on the Blue Team Village Twitter earlier today about ransomware and whether or not victims
of ransomware should pay the ransom. We are going to get on that topic in a bit and we would like
your input in the poll. Also, please post questions for the panel in the Blue Team Village Discord
in the Talks Track 1 channel, which we'll be monitoring. My name is Russell Mosley. I go by
SmokeEm on Twitter and I'm a lead with the Blue Team Village. I've been a Blue Teamer for my
entire career, building and managing security operations at small and mid-sized companies.
So I wear a lot of hats. I've done everything from sysadmin. I've been an IT director,
a security ops lead, a compliance auditor, and today I'm a CISO at a rapidly growing government
contractor. All my experiences involve developing and managing small teams. And as a result, I've
had the opportunity to do a lot of different things in the security field and build decent
security programs without a huge budget. I'd like to introduce my co-host, Xavier Ashe. Xavier?
Good afternoon, everyone, or good morning to where you're at. So my name is Xavier Ashe. I go by
Rubix1138. My IR experience is a little varied. I was an IR team lead at bit nine slash carbon
black. And also I was able to start the first IR team at SunTrust, now Truist. So 28-year veteran,
been around for a while. I can remember when my IR plan was you have to scan every floppy by hand.
That's me. Cool. Next I'd like to introduce Lip Moose. Moose is frequently posting to Twitter,
comments on exciting things that she encounters doing IR. And this week it was jump boxes that
look like flaming toilets, I think was one of the highlights. She refers to herself as many things,
including Florida woman, a DFIR violinist, and recently completed a cross-country hacker road
trip with cats and snails on board. Isn't that right, Lip Moose? That's fairly accurate. I did
remote DEF CON. We got to see a few goons. It was nice. So I go by Lip Moose on Twitter, also known
as Moose. My name is also Heather. I'm currently working at CrowdStrike. I'll dox myself today,
which is not a thing I usually do at DEF CON. But I'm very proud of where I work and what I do.
I work with a lot of companies of all sizes. I come from a background
for financial giants, as well as manufacturing giants, and have been doing this long enough to
where I think I've been awake and can engage myself in the 60s if I add all of my cumulative
IR experiences together without sleep. So I'm looking forward to retirement. I don't know when
that will be, but I do love what I do. Outside of work, I herd cats and I play violin. And so
I'd actually like to pivot and introduce somebody that I've worked with a little bit. 1099,
one of my favorite people in the world. Look, I'm sober right now, which is a problem. But,
Brian, I'll go ahead and pass it over to you. Well, hello, all you cool cats and kittens.
My name is Brian Moran, and I go by the incredibly clever Brian J. Moran on Twitter.
I've worked primarily in the Air Force. That's where I started my career, worked for a three-letter
agency, did that for a bit, got out of the military, bounced around a couple of private
sector companies, and eventually, back in 2014, started working completely for myself. So I
started my own company, and I am an IR consultant. That's what I do probably about 60% of the time.
The other 20, 30% is probably primarily focused on training and development plans,
and the other 10% of my time is primarily spent trying to keep the DFIR FIT going, where
it's all about just trying to get a little bit more active,
and hopefully we can all be on this little blue dot a little bit longer.
Cool. Yeah, I certainly was going to mention Brian's Twitter seems to generally be filled
with DIR FIT and comments on rum. You left the rum out. Well, I have some right here.
Oh, funny you should mention that.
I thought you were going to be better than me, Brian.
Let's hear from Virus before we have our toast. Virus, so he says he's only been a blue teamer
for about a year and a half with a background in red team, and he's currently doing what I
call kind of red-blue and innovative network defense for a large financial organization.
If you want to elaborate, welcome, Virus. Yeah, so I work at Stripe currently. I'm the
threat detection team. I guess the word for it is adversary direct engagement and threat hunting.
Background-wise, started in the dark gray end of the pool as a young kid,
like my first DEF CON, I was 17. Fell into penetration testing and then red teaming,
like full-spectrum threat stuff, and passed around the world doing that for a while.
Kind of drifted back towards pen testing, and then, surprise, had a kid, and, you know,
turns out that gallivanting around the world, breaking into buildings in random countries,
and getting arrested is not a good way to be a dad, so had to pivot for a while.
Did a stint doing mergers and acquisitions security at Salesforce and found out I'm not
bureaucrat. Now I'm doing blue team, so, yeah. Cool, that's awesome. Really looking forward to
sharing some of your perspectives that we heard in our planning meeting yesterday.
So before we get going, we wanted to carry out the DEF CON tradition of first-time speaker drink.
Everybody have their glass ready? Awesome. Cheers to IR.
All right, so to kick it off, you know, we're in a unique situation here in 2020, right?
With COVID forcing work from home for so many people. So I wanted to start with a question for
full-time incident responders, Moose and Brian. What types of incidents are you seeing the most
of right now? What are the trends in attack methods, persistence, and anything else you've
observed lately in your work? Yes, I can go ahead and start.
It hasn't changed too, too much. The volume has been a huge uptick. I'm still seeing a ton of
ransomware attacks. A curious thing that I'm seeing more of now is actually, I would say,
more intrusion via remote code execution has had an uptick recently. With a lot of the bots out
there, vulnerability scanning, I would say that those are getting hit before people can catch
just a little bit faster than I used to see. But a lot of the end game for what I'm seeing now
is either encryption and then extortion or exfil and then extortion. And I would
say that the volume has just gone up versus what it was like a year ago.
Yeah, I would 100% agree. The exact same stuff that's happening now has been happening for
the past probably two to three years has been the general trend. But the volume of it has just
immensely increased. And probably part of that is the bad actors have nothing else to do except
stay at home, just like we have nothing else to do. So the idle hands will play, I believe,
is the phrase that's used there. So the one thing that I have seen more recently an uptick of than
I have over the past year or so is actually O365 and not necessarily intrusions. It's users
clicking on links they shouldn't click on, and then the attackers getting access to stuff that
they shouldn't necessarily be able to get. So it's not necessarily a 365 intrusion. It's just
phishing, which is kind of directed more towards with the overall goal of getting some O365 access.
Yeah, that's interesting. I wanted to ask, a lot of companies have focused a lot of their security
on their networks, and perhaps not as much on their endpoints and cloud services.
And now that many of their workers are remote, they don't have the option to do monitoring
on their network, and maybe aren't as well prepared to monitor from their endpoints and
their cloud services. Would you agree that that's the case? And how is that impacting
what you're seeing out there? So I would say it definitely does. But the caveat to that,
and I already kind of doxxed where I work, is I'm getting calls because they're not necessarily
using an EDR to begin with, or maybe there's not as much focus on the endpoint. And I think that
what's neglected in cloud is that there's this perception of security that's not really there.
You're just working on a server in somebody else's data farm. And so you still need to put the same
protections on it, and monitor your VPC flow logs, and look at what VPC you're putting stuff in,
or what segment, rather. And you would monitor it just like you would something on-prem in your
environment. And I'm seeing that being so quick to set up that sometimes those steps are missed,
and then I'll get a call. Yeah, I think as we talked about a little bit last night,
one of the things is the data that you need to actually respond to a breach of some sort,
it's always been there. It's just never really been used or even thought of being looked at,
because before it was all, well, we're just going to block everything at the network layer because
everything is controlled. But now that everything's spidered out, all the logs are still
there. All of that's still there. It just never was bothered to actually be configured properly
in the first place. So that's why we're seeing what we're seeing. So with the more distributed,
I'm hearing more endpoints and more cloud, has that really changed the way that we're doing IR now?
I would say yes and no. So we're still asking for logs that are representative of the same data.
Instead of asking for firewall logs in this case, you're looking for NetFlow,
which might be DPC flow. We're still asking to run scripts to an endpoint, which whether or not
it's in a cloud or on-prem, it's going to be capturing the same artifacts. We're looking at
the same things on Linux that we're looking at on an AWS Linux server that we would look at
in a server in your environment that's physically there. So I would say that while things have
changed in slight ways in how we capture the data, so converting an EBS snapshot versus
a VMPK versus I'm going to run FTK to it, those little minor changes are there. But I think just
getting creative about, I'm still caring about the same things. I'm still caring about how the
thread actor got there. So your original access point, I'm going to look at all logs. I'm going
to look at, has there been any exfil, file creation, file deletion. So the same questions
we've been asking forever, we're still going to keep asking. It's just that creativity around,
we're in a different environment now. Unfortunately, those logs are generally
there for companies that rely on Office 365. A lot of times you just have to have the right licenses
to go look at them in cloud app security and other places. So what happens when an organization
hasn't really invested in logging on their endpoints or anything beyond the defaults
that come with Windows? You have a situation where you have people getting phished or
getting malware, and there's really not much to go on. What do you do in that case?
Generally, you're not going to have the easy access to get a forensic image and conduct
analysis on the device because it's somewhere far away from you, maybe on a slow link.
How do you deal with that?
Well, I think the old way of, well, we would just show up and collect data from somebody's
computer in the event that something happened. Obviously, that is no longer going to be
existing. So you have to get creative with essentially trying to get the same people
together on a call like this and figure out what you actually have access to. Not necessarily what
you have, because more than likely in the event that someone like myself or Moose gets called in,
stuff has already happened that's really, really bad. We're pretty much the last people you want
to see in the event of an incident. So you have to get creative with essentially figuring out
what you have access to and then how far back that goes, if it goes back at all,
and then try to build out from there. Because a lot of times when we show up somewhere,
the first question is, well, okay, do you have NetFlow? Do you have logs? Do you have
X, Y, and Z? And a lot of times, the initial part of an incident isn't even actually responding
to something. It's figuring out what we actually have access to and then trying to get creative
with ways to get access to that data, get it as quickly as possible, and then turn that into
usable information that we can use as quickly as possible. So what are some things...
Go ahead, Moose. Sorry. I was just going to say, I think one of the boons right now is that so
many people have invested in cloud security, so it almost negates the problem. A lot of times,
a threat actor's pivot point of getting into a lot of devices, we can think about, and I almost
want to turn it over to Byers for this one, but if you think like an attacker, you have a better
chance of getting the data you need. And I worked a case recently where, you know, I couldn't run
the scripts I wanted to, but I looked at what they did, and I said, well, let's just replicate
what the red team did, or what the threat actor did in this case, and let's red team it,
and I got the data I needed. So that's... sometimes you just got to look at the problem a little bit.
I think another curveball that I started to see is, you know, for companies that use Wwise terminals,
you really can't take those thin clients home, and so we're seeing an increased use of other
devices like Chromebooks, and so I wanted to see any advice out there if somebody that's
listening is going to be doing any type of instant response involving Chromebooks used as,
you know, thin clients. Is there any advice that you can give to those folks out there?
Looks like we lost Moose for a minute. Well, so I know that there's been some research into
essentially the data that you can collect from Chromebooks. I know Jessica Hyde from Magnet
Forensics has done quite a bit on that, and I believe this past year she's given a few talks
on that. So there is data you can get from Chromebooks. You have to get clever with it,
but one of the things that's interesting is if you use Chromebooks, more than likely you're using
some sort of Google instance, and Google has a ton of data about you. Whether you use Google or not,
Google has a ton of data about you, and because you're able to get all of that data back from
them. And admittedly, there's a lot of stuff to go through. But it's not necessarily... like
Chromebooks are incredibly secure. I love Chromebooks. I recommend Chromebooks to pretty
much everybody who gets click-happy with any kind of links. Just use a Chromebook for a while,
and hopefully that's going to eliminate a big portion of your attack surface. It's not going
to completely eliminate it, but it's going to take a big portion of that away. But essentially,
at the end of the day, you know, we're still trying to look for the oddity outlier information
of data access, you know, data exfil, stuff like that. And with the stuff that you can get from
Google specifically, like all of that's there. It's just a matter of being able to get it,
being able to process it, and being able to digest it.
What are your thoughts, Virus, on doing IR with cloud and distributed endpoints without the
hardened network of tradition?
In some ways, I guess I haven't been around in the Blue Team ecosystem enough to know the
difference. Like, the few roles that I've had that are hard in Blue Team, I've always had the
power to at least have a say in kind of what our telemetry looks like. And I mean, coming from the
red side, I've frankly never really bothered wanting network stuff. It's always, you know,
anything that's a network artifact, I only care about it in the context of what the endpoint did
in terms of interaction. And so it's just usually easier to filter on down and kind of pinhole what
I want. I have definitely, within the past year, I've dealt with some vendor breaches, which are
all public, so I can say that, but I guess I won't say exactly what they are. So, you know, I won't
get fired immediately. And it's kind of interesting. One of the nice things about, I shouldn't say
nice, but one of the interesting things about kind of being tangentially related to a breach
that gets released public is suddenly you have like a legal stick to beat them with to actually
get IOCs back from the breach, as opposed to just reading the news and having a Twitter opinion like
the rest of the world. And from what I've seen, it definitely, like, the game for the vectors
in terms of the game for getting in doesn't seem to have changed. It's usually not the server.
It's usually it's way easier to get on a laptop. And then because no one knows how to network in
the cloud, like everything is just grabbing the AWS token or the Azure token off disk and then
going to town. And then usually from there, it's a, oh, by the way, either there's no logs or there's
a ton of logs, but nobody's ever actually triaged them to find out which logs are useful and which
ones aren't. So there's no logs. And then it's usually going back and being like, oh, well,
after they took this token from your machine, then they directly logged in with this super
weird user agent that no one's ever heard of and stole all your data and you did nothing.
At least that's been the trend from what I've seen. The Chromebook thing, there actually is,
I don't have a lot of specific details that I can share, but I do know that there has been
some publicly released stuff on Google implementing a kind of AuditD type thing
that is going to be very similar to like OS X's security center in terms of accessible data. So
I think that's going to be a game changer with all of the EDR tools that are out there that are
hooking into AuditD as kind of a funnel for data. I think that'll be really cool. And then also
anybody who's looking to do DFIR on Chrome locked devices in general, I would actually
route you to some research from about eight or nine years ago from Mike Osmond and Kyle Osborne.
They did some offensive research on basically how everything in Chrome is a browser and the
browser doesn't know the difference between a file system link and a remote link. And while
Google has obviously tried to address the security concerns that they found with some of that
research, you can actually use a lot of those vectors to gather debug information about the
Chrome browser environment. And that ends up being almost equivalent to system logs on a
Chrome locked device. And you started your comments there with, you know, first getting
on the laptop and then pivoting out from there. What's the most common method for getting on the
laptop? It's the usual. It's either phishing or I've seen some supply chain stuff recently. I
came out relatively recently that I know affected a bunch of shops. I've seen some slightly less
talked about in the media because the Go ecosystem is so blurred in terms of the concept of standard
lib versus some person's stuff on GitHub. But I have seen some bugs there that have caused some
supply chain issues. There's little known fact, I have seen, well, it's a more known fact that OSX
attacks are on the rise, but it's a lesser known fact that Perl is still on the OSX boxes. So,
I have seen kind of some like, you know, oh, I haven't seen that attack since like Red Hat 5,
but, you know, that Perl subsystem is still on OSX and way more people look at Python on OSX
than Perl. So, there's a little more of that. That's great. I'm curious, Brian, your observations,
you know, how they compare to viruses feedback, you know, and what you're seeing.
Yeah, I think it's... so, I don't think the methods have changed. The methods haven't changed. And
honestly, the methods don't really need to change because they still work. So, if they're still
effective, there's no reason to burn some, you know, some zero day exploit or even go the
route of looking for like RCE or anything like that, when if you can get somebody to click on
a link and get the information you need, and that still works, like, that's what the attackers are
going to do. Because, you know, again, at the end of the day, like, the attackers are human just
like us. Like, they will do just as much efforts as needed to get them the goal that they need.
And, you know, you have to be able to account for that. One of the tools I haven't heard yet so far
for telemetry is, you know, cloud-based DNS. So, a couple providers where you can get that from a
centralized source. Are we seeing more people, you know, take that on as we get to a more
distributed environment? Or is that still an afterthought of like, oh, yeah, we can't get good
DNS telemetry because everybody's at home using their home DNS? Has anybody got any experience
with that? I do. I've had some that have it, but not very many. And with the caveat of saying
that if they're calling me, something's gone terribly wrong. Or, or counter, we do do security
assessments in environments that might be something to come back with because they've set up a cloud
environment and they want advice on how to make it better. So, I will say that I have a little bit of
a bias at answering this because even though I don't see it as much, it's usually because they're
calling for help. Yeah, I mean, control is kind of a crapshoot. But in terms of telemetry, like,
man, the combination of a low-level passive DNS service and OS Query seems amazing for DNS to me.
Yeah, so I'm going to say pretty much the exact same thing that Moose did. Like,
I would like to see it more. However, a lot of places don't see it. And the places that call
us, more than likely, there have been several shortcomings that led to calling people like us
in the first place. So, typically, we wouldn't see something like that because we're not the
security architects or anything like that. Like, something horrible has happened,
everything's on fire. I'll call them in and they're hopefully going to calm us down.
So, before we pivot to another topic, if you were called in to give advice to a company today,
and they wanted to spend some additional budget for security with people working from home,
where would you advise them to focus? I mean, I can tell you where we have an
entirely remote workforce right now. And there is some scatter in terms of operating the system
layout. And pretty much 100% of our EDR effort at this point is done using OS Query. That's
pretty public information. I don't really feel like I'm giving away secret sauce saying that.
And at least from a telemetry perspective, even with specific regard to DNS, I can tell you,
it's pretty amazing. Like, we have other issues, but
DNS telemetry is certainly not one of them from my perspective.
I think the one piece of advice I could give is don't ignore any front. I see a lot of people
spending very heavily in one arm rather than maybe ignoring the other. So, they'll focus
just on end point or they'll focus predominantly on network side. And I think that the balance
is still very, very important. And I would say to borrow something that we all as a team talked
about yesterday, too many people are trying to eat the elephant hole. So, don't do that. Don't
keep your network flat either. So, my top advice to people is make a list of priorities in your
environment. What would hurt the worst if that went public? And start segmenting there. Start
segmenting. Know your group permissions. Run some of these Red Team tools. Run Blood Pound,
Sharp Pound in your environment. Get a better picture of what it looks like and then make a
decision. But make an educated decision, not just a flat blanket decision because not all
businesses are the same and not all businesses have the same needs. I think before spending any
money, the first thing that a company truly has to do is take a look at what they already have,
the people they already have, and then try to build a solution from that. Just like Moose said,
a lot of times when I go to places, they don't even have something basic like a network map.
So, then I look for the attackers when essentially they dump IP.txt to some system and then I have
an actual network map that I can go on because the attackers have left their forest, fortunately. So,
that's literally the only insight that we have to what's actually on the network is what the
attackers have done. So, you can't be just spending money while you have all of these
gaping holes that just have never been addressed. You have to do a serious and honest and truthful
evaluation of what you currently have already before you make a decision on what you're actually
going to do to try to improve. And related to that, skills versus tools, right? Should you be
investing in your staff and in their skills, training, sending them to conferences like
DEF CON where they can learn from the community what others are doing and increase their
capabilities instead of buying tools? Thoughts on that, anyone?
I have strong feelings. So, I think it's both. So, you really need to invest in your people but
don't overtax them so much that you're not giving them the tools to work with so that they're
getting alert. You don't want to have them working with the bare nothing and then getting mad
at them when they miss something. So, you know, take the team you have and really build them up
but then also listen to them on their level of what they'd like to use and what they'd like to
have and use that training of sending them to conferences and coming back and gathering that
information to inform your decisions. And I think that it really needs to be that whole cycle of
you're investing in their education and then going to conferences and networking and letting them
come back to you with a solution. And that is going to sometimes be investing in tools for them.
Yeah, I agree 100%. You know, you really want to listen to your team, you know, listen to
their experience, right? You know, get them to get out there and participate in the community
and learn what works and, you know, take their advice on those areas, you know, and figuring
out where to invest in your team. So I wanted to talk a little bit about IR teams and
folks who want to get into IR. So, you know, I think they're two related topics, but a lot of
people have asked if we could give, you know, what's your best advice for people who are interested
in getting into the incident response field? And that's for anyone who wants to go first.
I'll go first. So my personal opinion on how to get into IR or really kind of any info sec,
like specific sort of doctrine at all is just to simply be curious, like ask questions,
try to research stuff on your own, figure things out. Like you may not be the best person at
assembly or firmware architecture or IR or malware reversing or anything like that. But
as you find questions and, you know, find things that you're curious about and want to research
and learn more about, like you'll generally kind of find an area that you want to go down. And
at the end of the day, like you have to kind of decide if you want it to be a typical job
where you work eight to five every day and show up and you get paycheck and that's it.
Or if you want it to be a career and you actually want to be good and want to actually, you know,
be better at what you do and be more effective and be more efficient and network with people
and all of that. And it's not really a lifestyle decision so much as it has to be an internal
passion and desire that you actually want to do something. Anyone else want to comment on that one?
I want to say that you're safe, but I will, I'll chime in and say everything that Brian said
absolutely settled down on that. But just know that there's no wrong path and that to you can
do anything in your spare time that you're passionate about and turn it into IR. The
difference is what I typically see in good IR is you don't stop at the first answer.
So, and you don't stop at the second answer either. It's this stubbornness and that drive
of being stubborn to where, okay, make yourself a home lab, image your own devices, break them. Like
see what you can do and see what you're seeing and use that to train yourself. And it
really is something that it takes a lot of training and that there's not just you're
there one day. So, I'll admit myself, I get imposter syndrome a lot, but that's because
every day I have to Google. There will be something every day that I don't know the
answer to or I haven't seen before and I am researching the hell out of it. I'm saying
sometimes it feels like I'm banging my head against the wall, but that doesn't go away.
And so, if you're okay with that and you have the personality that has that drive,
you're going to do great. And being, like listening to this right now, it shows enough
that you're willing to listen to the five of us ramble for an hour. You're probably
in good company and you should probably just keep pursuing this.
One of the, on that line, you know, talking about getting into IR, talk to the, you know,
more along the lines of the type of work and particularly about the stress. You know,
typically IRs, you're more like a fireman. You're dealing with sometimes some high stress
situations. And so, any comments or suggestions for the listeners around, you know, kind of
dealing with the excitement and the stress that can come along with being part of an IR engagement?
Well, I think one of the things that you have to take into account is, so first of all, like from
an IR perspective, like you have to try your best to be calm. Like you can't overreact like
everybody else, like let everybody else overreact, let everybody else run around and scream and yell
and all of that. Like you have to be the calming voice. However, that doesn't mean internally,
you're not screaming and yelling too, because you absolutely are. You just have to portray yourself
as having that calming presence. And you also have to essentially, not necessarily,
not necessarily have the right answer, but you have to have a answer because the path to recovery
always starts with one step forward. And it doesn't matter if that step is even down the
wrong path initially, like you just have to go forward. Because the takeaway that you can always
have, no matter what, is somebody out there has it much worse than you currently do. And you always
have to remind yourself, like as bad as everything is, like there's some company out there that got
owned and they are in, unbelievably, an even worse situation right now than you are.
I think that everybody in IR is a little bit of an adrenaline junkie,
and that you're kind of lying to yourself if you're not. Brian, didn't you go like
space jumping or something at one of the cons that we've been to recently?
Yeah, I jumped off the stratosphere in Vegas a couple of years ago, twice, because I wanted to.
I was going to take the question and ask each of you individually,
you know, how do you get away from it? How do you deal with the stress?
So Brian goes jumping off the stratosphere in Las Vegas.
How about you, Moose? How do you get away from this? What are your breaks?
Well, it's twofold, right? Like, everything I see on a daily basis,
it's somebody else's worst day. I don't work for their company. And I had to realize this
about myself and every job I've had. So I've not always been IT. I used to be vet tech,
I used to be medical, I was a lifeguard. All of these things have a similar, like,
vibe in that it's really high stress, but it involves helping. So I think, to me, my release
is in seeing that step forward and that progression. And even if it's not happening right
away, being able to reassure the people that I'm working with that it is truly happening,
at the end of this, their environment will be better than we left, than it started at.
And then, if that doesn't do it, I have cats and a violin, and on the worst day,
I'll just shut everything down, everything gets unplugged, and I open up the violin case,
and I walk away. And I think that's really important, is that you don't have to eat,
sleep, and breathe this. You really should walk away sometimes. And that's what's kept me alive.
And how about you, Virus? I mean, it sounds like you certainly have some of the adrenaline junkie
in your experiences, in your past, breaking into buildings, you know? So what do you do for breaks?
Shenanigans. I mean, I guess the first thing is, like, don't get stressed. Like,
that's a big part of it. I mean, I've made a lot of... I've definitely steered a fair
amount of my career decisionary of, like, you know, when I finish an arc, I go,
did I like that? Or was it just stress in the bad kind of way? And then I use those takeaways to just
not do that anymore. You know, kind of quantify how much my time is worth. Because if I spend
the whole time when I'm not at work, stressed out about work, then I'm essentially working for free.
So if I factor that into my paycheck, I'm like, oh, this isn't that worth it. So that's definitely
the first step, is just don't do stuff that stresses you out. You know, know what your zeal
is. But on a practical level, like, I played music for, like, 25, 30 years, semi-professionally. So,
you know, I do that. I kind of zen out. I cook. I dry-age meat. I make my own gin. You know,
I like scotch. I do a lot of drugs. And then I play with my boy. You know, I'll stand up
from my desk when I'm working on a hard problem and just say, hey, let's go for a rock. And he's
great. So that's amazing for him. It's great. He just runs around. But yeah, it's, I mean,
you know, I think a lot of it is just you got to be holistic about your approach. And if you love
the chase, you got to love the chase. And you got to love your chase. You got to love the chase
that you're in and know where it is and just not lie to yourself. You can't, you know, do it. And
I mean, there's definitely an undercurrent if I'm being extremely frank and extremely personal
about it. I do think people that, I think there's a fair amount of people on the blue side,
specifically, who have a very, I don't want to say ethical, but they have a very empathetic
attachment to the world, slightly larger, maybe than most red teamers do, where at some level,
they're kind of in it because they want to, they want to fix something. They want to make something
a little better. And I don't know, I mean, I wouldn't call myself a complete prick, but
I'm definitely on the side of like, yeah, I like money.
Yeah.
Long as I moved the needle a little bit and I got paid, I'm not too worried. Maybe that's a
whole other conversation. If someone knows you from the outside, you're one of the nicest people
I know, so that's a lie. If we could get back on the topic of, you know, of the IR field and IR
teams, we had a question in the BTV discord about, you know, what are your top recommendations,
and maybe some don'ts, you know, for growing your SOC and your IR team? You know,
if you're at a company and, you know, it's growing and, you know, you get budget to
have some full-time folks doing security operations and, you know, skills and tools
ready to do incident response, you know, where do you want to focus? You know, what skills do
the folks need to have? What are some of your preferred tools? And I know it depends a lot,
but just in general, you know, where would you focus? Brian, why don't you go first?
Well, I think it goes back to... it very much does depend, but it also depends on what you
currently have already and essentially what you have the ability... what you want to essentially
try to strive for. Like, if you want to go a very, very, you know, cheap route as much as possible,
you would use something like OS query or use something like Velociraptor and, you know,
try to essentially keep it on the cheaper side. But, you know, at the end of the day,
it's still going to be able to work. Of course, if you have unlimited budget, like, you just line
up boxes, you know, like crazy, and then you have all the tools and everything. But, you know,
to be completely honest, that's not feasible for anybody. Like, companies will not just blindly
give you ridiculous amounts of money to spend on people and tools and resources. So, you have to
try to make whatever you have work with what you can. And that's one of the things that I stress.
It doesn't necessarily matter what the dollar amount is. It's trying to make every dollar
count as much as possible. So, that way, at the end of the day, you end up with a truly better
result. If you have a very high budget, great. Like, you can afford expensive tools. But do you
really need those expensive tools? Do you really need an expensive person to come in maybe for,
like, you know, three, six months or something like that and help train and make the team better
to already use the stuff that they already have? You know, in that case, you could argue very much
that bringing somebody in external for a little bit and helping them not really as a staff log,
more of kind of like an overseeing mentorship almost. Like, that is a much better investment
than just blindly buying, you know, a million dollars worth of random tools. So, it truly does
depend. But, you know, it's like everything else. Like, you have to make a business decision about
it and essentially try to maximize every dollar and get the best value for that as much as possible.
You know, and we've used the word a few times in this panel, telemetry. You know, so,
from my perspective, one of the first things that I would want to do is, you know, is set all that
up, is set up, you know, a central logging system, whether it's SIEM, whether it's, you know,
something much simpler. But, you know, for me, doing incident response, you know, those are the
keys, right? I mean, I've heard it said, you know, logs are everything. And that that would be one
of your first steps would be to make sure that you, you know, you're gathering enough information
so that, you know, when an incident happens and just in general, you know, having logs has so
much value for IT support, for everything that we do in security. My approach to this is really,
it's like a three-step. It's like a three-layer moniker. So, I think the first thing that it's
important to grasp is that a program, we talk about a program, whether it's an IR program or
a security program, large or blue team program, a program is business process. That's what a
program is. It's not blinky boxes. It's not even the talent in your bullpen. It's documented process.
That is the definition of your program. So, the first rule is if you're going to build a program,
build a program. That means encourage your people to, and this kind of leads into the second thing,
which is tribal knowledge is the enemy, right? So, like the first job of everybody on the team
is to take everything in their head and inject it into business process. That is your job. It
doesn't matter if you are, you know, the self-proclaimed bottom rung of the team or if
you're the epic gangster rock star at the top. Your job is to take whatever is in your head that
applies to the program and to build the program. And the more that you encourage people to do that
and the more that you reward them for doing that, it's like a self-incentivizing loop, right? That's
how you balance burnout and all this other stuff because it's not focused on one person or one
tool. And then, like I said, the second thing is, and I feel like this is way more common on red
teams than it is on blue teams, and I kind of don't know why. It is extremely common on the
red side to not use tools that are touted as being the latest and greatest thing just because you
don't know them, right? I mean, as a red teamer, I feel like there's this moniker that is
accepted more and I feel like it's true on both sides. So, I kind of don't understand why it's a
lopsided. We're like, you know, it's the old ninja moniker, right? We're like, not every ninja uses
every weapon, right? You have your weapon and it's got your epic moves in it and you know it like the
back of your hand. It's almost an extension of yourself and that's the point. I don't really
think that's different on blue team. I mean, case in point, the team I'm part of at Stripe that kind
of got built about a year and a half ago, we were all very new to the company and was, you know, granted
we were very fortunate to have leadership with a lot of experience and so they reached out and
you know, there's a lot, there's an extremely high amount of experience and talent on my team that's
a little bit askew for most. I'm not saying that to brag, I'm just saying, you know, a lot of
teams don't have that in their tool bin. But the first thing we did was take what EDR we had
and rip it out. We ripped out almost everything and then built everything off of bare bones telemetry
because those were the systems we could use better. That would, that yielded better to our kung fu.
And then from there, the third thing is build, like tool your kung fu. You know, take your
brain dump, put it into your program and then hypothetically start from scratch and go
what tools do we need to do this, what's written better? Frankly, every deviation from that is a
waste of time and money. So you said that you ripped out your EDR and you built up bare bones telemetry
are the things that are most useful to you. Can you elaborate on what those are? I mean, it's literally
just stuff that's built into the OS. I mean, we use OS query at scale so that because we use the
fleet manager concept in order to kind of achieve orchestration goals. But our detection system
is entirely homegrown. Our response system is entirely homegrown. Our orchestration elements
and the way that we do log tracking is entirely homegrown. Even our pace reporting. I mean,
we're kind of sort of bubbling around the hive right now, but it's honestly, it's all Jupyter
notebooks and Python. It's all just kind of manual tooling. And at the end of the day,
if I take that and I put that in the ROI bin, we spent way less on human time writing some Python
than we would ever spend on third-party utilities. And to be honest, most of the third-party
utilities are trash. I mean, I won't pull any out just because I don't want anybody to
kind of, you know, have to be under the boot because I said something, but like,
man, there's just so much money spent on these tools that just don't even do what they say on
the side of the box. I just, I don't understand why that flies. And then that's just, you know,
I don't know. Say what you will about red team or blue team infra, at least the red team stuff does
what it says. It just doesn't say a lot. Blue team stuff, man, this seems to be a problem.
Cereal box, nutrient banks are just wrong.
So, Rubix, you've built an IR team, right? I'm curious, you know, your thoughts on all this.
Yeah, you know, it was, I always go back to the, you know, kind of build it using the fireman
analogy. So IR for, you know, a, you know, we're in financial services, there's a whole lot of
downtime. And so figuring out how to practice and be ready for the big event and giving them
enough work of a real, you know, I guess, you know, smaller level incidents to be able to stay busy
and relevant, but then also, you know, plenty of time to prepare for, you know, bigger stuff. And
so, you know, lots and lots of practices, you know, tabletops, drills, and, you know,
encouraging our red team to really give us some good scenarios. And so I think that if you are,
you know, not balancing that, you know, if your guys are all, or folks are always spending time,
you know, doing, you know, we talk about being really busy, that they might not be honing their
skills enough. And so, you know, finding that balance between, you know, doing and training
and practicing was very key. I think the one thing I have to add to that, and everybody else
has given really good advice here. And this is a part that I'm really passionate about, because
it's one of the reasons that we lead companies. Don't silo your people and listen to what they
have to say. If you have somebody on your team, it doesn't matter what notch in the chain they are.
If you're calling on them for some kind of instant response, whether it be a SOC person,
somebody who's doing sys admin work, someone who's a network admin, you're trusting them
with that role. Trust them if they say they have a passion in response as well. Let them
play in other fields, because there is not a silo in incident response for our attackers,
so there really shouldn't be a silo for our responders. And one of the reasons that I work
where I work is because there are some days where I'm not incident response. I have the freedom to
do red team stuff. I have the freedom to help build our tools if I want to, or scripts. Some
of the most brilliant scripts I've seen to date were built by my teammates in response to the
Citrix vulnerability, for instance. So, you know, knowing where your strengths are on your team
and building those, and then listening to them and what they say they need. Because, you know,
if you're the one building, that assumes that you're at a higher level position,
and I think that's really key, is to... my cats are wrecking the apartment right now,
so if you hear anything bad, that's what I keep looking over for, is anytime I'm on one of these,
they wreck something. So anyways, but listening to those people that you've hired, and obviously
you trusted them enough in the first place to hire them. They're smart people, and we're smarter
not because we know something, but because we know who to go to when we need certain knowledge.
And I think leaning on your teammates, no matter what level they are in your company,
having that kind of group decision is still important.
Yeah, I think that's really great advice, you know, that more companies should heed,
to try to cross-train their folks and give folks time for training and for research, right,
and have budget for tools and devices, right, and lab time. I think it's really important.
So I wanted to pivot and talk about ransomware. So we had a passionate discussion about this
yesterday, and I'd like to try to rehash some of that for our attendees. And to kick it off,
earlier today, the Blue Team Village Twitter, we posted a poll, and we were looking for people
to give us their opinions. The poll question is, your organization is the victim of a ransomware
attack. Should you pay the ransom? And about 89% of the responders said no, and about 11% said yes.
I think, you know, in summary of our discussion yesterday is, it's not that simple. But,
you know, for the benefit of everyone who wasn't in our private discussion yesterday,
you know, what goes into the decision of paying the ransom or not? And I think
Moose started first yesterday, if you want to go first again today.
Sure. This is the most, like, let me tell you right now, this question made my blood pressure,
like, go up a little bit. Because for me, when somebody says, do I pay the ransom?
On any call I have, or even if I've been in person, I look at them and say,
well, what does your legal team say? And that's still kind of my answer,
is that I don't think you should make that decision for your business in a vacuum. And
I usually recommend that people talk to either their internal counsel or have external counsel
involved in that decision making process. And not necessarily make a decision for themselves.
Because these are, if you look at it this way, the way I like to frame it,
because I'm not going to say yes or no, spoiler alert, I'm not going to say yes or no,
because it really does depend. And that's not my decision as a responder to make.
But I think that the most intelligent things I've seen done was calling on experts who do
this every day. Because if you think about who hit you, you're not their first hit.
You're not the first people, they've probably ransomed, they've hit other companies. These
are professional criminals that are going after you almost all the time, if they're exporting you
for something. So get a professional to represent you. That's kind of my opinion. And I will preface
that with that as well. Okay. Brian, you kind of immediately had a, you know, from the heart
response that you then elaborated on. Could you share that with all of us? So as Brian Moran,
human being, my first response is always no. However, that is simply me as a human being,
because I don't feel that it's right to pay criminals for doing a crime. However,
we'll preface all of that with put on the consultant hat, the business process hat and
all of that. And honestly, the answer is more than likely, it's usually cheaper to pay the ransom
than it is to deal with the cost of everything else involved. Like, it should never, just like
it should never be anybody on the InfoSec team ever. It should never, ever be their call. It
should be legal's call because legal absolutely needs to be involved. Not just from the perspective
of, is this okay with the company? Also from this perspective of, you know, you're potentially
looking at paying somebody in another country, like, are there sanctions against that country?
Like right there, you could violate even more laws just by paying the ransom. So now you're in
double because all your debt is gone and now you're getting sanctions and fines and investigations
levied on you because you are literally, you know, potentially financing something that you shouldn't
be financing. So it's not our call to make at the end of the day. Like, you know, we can respond and
say, here's X, Y, and Z, here's our thought process. But generally from the overall business
perspective, it's usually much cheaper just to pay the ransom and be done with it than to deal
with any of the public scrutiny, deal with any of that. But, you know, we're not lawyers.
I don't think any of us are even going to pretend ever to be a lawyer. Like, even if we say at a
Holiday Inn Express, we are definitely not lawyers by any stretch of the bean. So we can have our
opinion and we can give you thoughtful, evaluated opinions, but like our actual thoughts at the end
of the day truly don't matter because it 100% decides on legal and what the course forward
is from them. Yeah, thank you. You know, I think a lot of folks have that reaction of, no,
you know, I don't want to pay criminals, but there is so much to the decision to be made.
And Virus, you came at this from a different angle than I've ever really thought about it.
And that angle was about ransoms and kidnappings and kind of relating ransomware to that type of
crime. Could you, could you rehash that for everyone? So I basically distill everything
that I tried to kind of get across in our joint meeting as ransomware is a crisis. The breach
that put together is an incident. So if you have a flat out answer on whether or not you should pay
the ransom when it comes to ransomware, you're doing it wrong because that's not how crisis
management works. And I don't understand why as even an industry InfoSec would try to solve
crisis management when every major Western power, every branch of every military and military
contractor that's worth their salt, every political, you know, wing contractor, I mean,
crisis management is a well understood science that has been researched by every tactical
engagement and business risk school in the world forever. And the fact that the thing that's being
ransomed is a piece of electronic capital versus human capital does not change the game. I just
don't understand why people would insist on coming up with a pat answer for that. I think it's stupid.
And while I think that, you know, the response to a ransomware crisis, because it is a crisis,
will likely involve legal, I don't necessarily think that legal is the end all be all shop on
that. It probably is at most shops. But you know what? The world is bigger than the law.
You know, I could tell you all day long about, and all of you could too, about every major
insert fortune 500 company here, where they have a crisis management team that has nothing to do
with computers. And I guarantee you, the first stop they call when the CEO gets kidnapped by,
you know, random activists in South America on a business trip is not the lawyer. They're like
the third. So I don't know why it's different because it's a computer and not them. And then
after you handle the crisis, however you handle it, maybe it's paying, maybe it's not, maybe it's
using shadow currency. I don't know. Right? Then you handle the incident, which is if you have
ransomware, you were breached. Don't handle that. That's your incident. The ransomware, that's not
an incident. That's a crisis. With the shift of a lot of ransomware actors, for example, Maze,
to not only just destroy anymore, but also, you know, steal the data, threaten to, you know,
do more stuff with it than just destroy it. Has that changed your way of dealing with ransomware?
It has in my advice. So I'll go ahead and speak to that in that I have
given a little bit of advice, you know, working with a legal team, of course, if it makes sense
for the company and with their backing. So all of that included. So assume that to from the get-go.
But, you know, a lot of these groups that do extortion, they're saying that they have something
and usually they'll serve up what they have. If there is at all 24 hours
that you can buy your IR team to investigate if that claim is true or not,
negotiate and buy the team time. If you can figure out if what they say was exposed is true,
and if there's any more to that, so if they've gotten more than they say they have,
take that time and lean on your investigations team to see that extra piece and see if you can
see that creation and that within your system. And that's a very hard thing to do in a short
amount of time, especially because it assumes that you have full visibility at every endpoint
on the network of your company. And to have that kind of visibility to begin with
is incredibly difficult for your team, let alone the outside team. So I would say that
the good things that come out can come out of that, not that I'm for negotiating with jurists,
so to speak, that are withholding your data and extorting you for money. But, you know,
sometimes I've seen outside counsel get a lower price so that if you are going to pay,
but time is valuable too, and having that on your side is important.
Yeah, I think one of the things that is hugely important, but also at the same time incredibly
underrated, is the way that ransomware works. Ransomware is a business. Just like any other
business, ransomware at the end of the day is just a business. And I don't recall seeing
personally, and I'm sure it's happened, but I don't recall a case personally where the actors
have claimed to have something and they truly did not, that they were bluffing. Because if you're
running a successful business, you're not going to lie about what you have and everything. Well,
unless you're Enron and stuff like that. But for the most part, you're not going to actually
not tell the truth about certain things. So if they claim that they exfil data and you as a
company have no way to prove that, like number one, you're already in way worse problems than
having ransomware. Like you have so many other problems that you need to deal with. But if they
claim that they have something and they more than likely do, and then you have to evaluate the cost
of all that. Well, what if that data goes public? What if that data goes even more publicly
widespread? What if that data is then sold to other places and everything like that? And you
have to take all of that into account with the entire process. And we often try to look at it
much as a pure like, essentially, is there a correct answer? Or is there not a correct answer?
Is it a yes? Or is it a no? And it's never just a yes or no. There are so many factors that you
have to take into account all of that through it. And we in the information securities field,
we only see a very small portion of all of that. So you have to involve other people,
you have to involve other people in the process. But just like it was said, if you have ransomware,
I guarantee you have at least five other problems besides just how ransomware got there. You have a
lot of problems that need to be addressed. Ransomware is the big public facing thing on
all of that. But there are so many problems that need to be addressed internally. And then you as
the practitioner have to take into account, all right, here's my little take on the ransomware,
here's all that. And then essentially, that decision goes to somebody else. But
ransomware, I've never seen anywhere where like, oh, well, we got infected with ransomware. And
that's the only problem that they have. There are many, many other gaping holes that they absolutely
have. I can guarantee that. And on that note, unless somebody else wanted to say something
else on ransomware virus, were you about to speak up? I was just going to answer Rubik's question.
Yeah, go for it. If I understand the question correctly, it was in a nutshell does introducing
extortion into the ransomware topic change the game? So my answer to that would be no,
at least not for me, because I've already given my two cents on how I think that the orchestration of
the conversation on ransomware is wrong. In terms of have I seen it change the game?
I would say yes. And I would say the way that I think it is changing the game is that
counterintelligence is a thing. Counterterrorism is a thing. Black on black kill squads are a thing.
And they have always been a thing in blue team. They just haven't been very talked about.
Because until words like hacking backs that are being introduced into the info set conversation,
those were mostly gray areas to a lot of professionals. My experience is that the
reintroduction, because I have seen it before, especially dealing with like adult fields,
it's way more common than what I've seen in corporate space. And now I'm seeing it a lot
more in the corporate space. I think those areas are getting a lot less gray. And they're showing
up on the radar of a lot of risk teams, because they're starting to realize that that's how you
fight this. But again, is my prescriptive advice that there should be more of that? No, that
doesn't answer the question. Awesome. Thank you. Any other comments on ransomware?
I will say that the other thing that I've seen is ransomware used to be kind of a singular
program that executes and they've got this programmatic way of doing destruction,
is that now when you hear of ransomware, it's a full-on incident. They have a number of backdoors,
they have a number of accounts, right? You can't just go and contain it and say,
we found ransomware on five machines, we've re-imaged those five machines, we're good.
Now they're going to have spent some time in your environment, stolen accounts,
have other ways of getting back in. And so you really have to do a full incident response,
find out all the things that the attacker got. It's no longer a simple case like we used to deal
with. The only other thing that I would add, this is the hill I die on, because I'm one of our,
say, Linux enthusiasts, where I work, is most people associate ransomware with Windows.
And while that's the case, that doesn't mean you should ignore your Linux environment and
never assume. So make sure that you're building that up and you're looking at it, and don't ever
assume that it's just, you know, if I have access to your environment and I'm a bad guy,
if you give me an easy pivot point, why shouldn't I do good? So kind of have that in the back of
your mind. And don't, like, go tunnel vision only on your Windows systems. And don't make it easier.
Sorry, Virus, I didn't mean to.
I just never heard anyone say that ransomware is a Windows thing. I'm like,
there's been ransomware on mobile phone boot roms since, like, 2001. Who are these people?
I hear it. I do. I hear it. I hear that this usually only hits Windows and look at the
Windows environment and all that. And sometimes that's absolutely true. But you should be scanning
for lateral movement to everything. Don't assume anything. What about Max? I thought Max were immune
from viruses. Mainframe. Move everything to the mainframe. There's no mainframe viruses.
Yeah. So we've already gone over time a little bit, but I think we have about 10 minutes left.
And so I wanted to kind of reward the folks who have stuck around through all of this very,
very serious conversation on incident response and try to end on a lighter note. And so I was
going to ask each of you if you could share one good story about, like, one of the craziest things
that you've seen, whether it was on a response or, you know, in some of your own actions,
Varys. And I will remind you that this is being recorded. So who wants to go first?
Anybody got one queued up? I have one kind of queued up a little bit. So this is actually a
combination of two. So one is the worst thing that I've ever seen in an environment.
The worst thing that I've ever seen in an environment was there's a firewall company,
who is a very, very popular, very, very known firewall company. And for whatever reason,
they somehow implemented all of these firewall rules that were okay. They weren't great. They
were just okay. And then somebody put in a triple anti-firewall rule. Not a double,
an actual triple any. So it was any, any, any. Come to find out from speaking with someone who
will rename nameless, but is on this panel, that company actually allows you to have as many as
five anys. So not only is there a problem with five anys, like even the option for you to be
able to put that many anys in something, like, that doesn't make any sense. Like, what are they
possibly thinking? Like, we're just going to make sure no matter what, that you can actually find
some way through our firewall. So the hybrid of that is it was with the same client. It was kind
of a cool story, because we had a chance to talk about it, I believe, last year in New Orleans,
because the actual indictment came down. So they were a part of a breach that was essentially
targeting the aerospace industry. And the attackers did all kinds of stuff. So we were able to actually
go through essentially, like, their Gen 1 through Gen 5 of their malware, including, like, the last
ditch absolute effort where it was incredibly sophisticated, because it was worth burning
something that was amazing on it. And when I found out, like, that was just incredible. It was amazing.
I actually did a little dance, because like, all right, now we found this. Now we have to find out
how we're actually going to get rid of it. But the way that we're able to track back to exactly
where it was is, again, because attackers are human beings, and they make mistakes. So for the most
part, they bounce their IP through other countries, and everything was just fine.
Except we had a total of 11 instances over the course of, I believe, it was a three-month period
where they forgot, or they just made the connection before it actually made all the hops.
And we had 11 times where we were actually able to pin down pretty much to the exact building
in a certain company that rhymes with Rhina of exactly where the attacker was, exactly what
they were doing, and exactly what they were after. And that was amazing, because we literally
had gigabytes of logs of all this access in, and in, and in, and in. And then we happened to find,
like, oh, well, here's this web shell that they had put in. The only person to access it was this
person. We know that for sure. And 11 times, they just made a mistake, and we were able to pin down
where they were. So it literally takes just one time of messing up, and like, okay, now we can
actually build an investigation. And we turned all that over to law enforcement. And unfortunately,
it took them about five years to actually go through the process, because they had so much
stuff to investigate, primarily for our case. But it was also kind of disheartening to see,
like, flash reports get put out that is essentially the summary of our report that we had written,
and just has an agency logo at the top of it. Like, it's like, that's my stuff. Like, don't do that.
All right. So who can go next?
Oh, man. I guess I should follow, just because, Brian, at the time, I had told you, I saw five
enemies on a firewall, and I was like, what in the hell? And it was like a horror for me, because I
was just like, it was at the top, too. So if you think of how firewalls work, if you've ever
done any kind of architecture on them, it's top-down. And it was five, and it's
any source, any destination, any application, any user. And I don't know which one I left out.
Location and service. There are actually six that can be set, and they set one as
deny. And I didn't even know why they set one as deny, because I was just like,
it basically just works. That's the answer I got, by the way, was it just works.
But my worst story is actually recent. I have a lot of empathy. I see a lot of stuff in
ransomware, but one of the coolest things that I've worked a couple of was not too long ago,
there was a salt stack vulnerability that came out that allowed for, basically, the long story
short is it allowed you to get root to anything that was exposed that was a salt master. So
a salt is used, salt stack is used to manage and administer a Linux environment.
And it's a really cool thing. I actually really like it for administration purposes.
But there was a vulnerability, and if it wasn't patched, you could get root into anything that
was internet exposed. And I just happened to see it in a couple of cases. I thought going into this,
man, if I had root on something automatically that was internet exposed, what could I do?
And I spent a long time looking at this environment, thinking I'm going to find something
cool. We got in and they used it to spam mining bots everywhere using the minion service.
And that was literally it. And so I want to tell that boring horror story because I live
too for the cool, like, when am I going to find an Oday? And I've seen them. Don't get me wrong.
I've seen some really cool stuff from APTs and some really cool tactics. I've seen people
absolutely stomp out servers to where I have to go into unallocated space and get really creative
about carving things out. But this one was one where I was going in going, I'm going to see
like something absolutely crazy because you automatically had root access and you could do
all of these things. And that was not the case. And I've seen so much of that over the course of
doing IR where I'm just like, you know, you have this level and you worked at this level. And I
think that's something to keep in mind just when you're doing these things is, you know, I was
super happy to go back to the client and say, I have great news. All they did is run up your cloud
build. But at the same time, like never assume anything because I was thinking like I'm going
to boil the ocean and find something and I did and I didn't. In 2005, I was on a plane coming back from
a job and a co-worker of mine is sitting next to me on the plane and we're talking about war stories
and he tells me a war story from the week before, which is why he was late to the job that we were
on. And I go, what did you do? And he goes, oh, I was doing IR on an old Windows XP network that
was locked down by this government thing because it was a defense contractor. I'm like, oh, well,
how'd the bug get in? It's like old Java bug, super like 265 day, right? Like old Java on Windows XP.
What was the vector? Facebook. Yeah. Ops were on there, you know, running Facebook and click
something got infected. Oh, that's interesting. Why did it take you a week to clean up? Well,
didn't have a lot of ambient room on the operating system to install tools. Why?
Well, it was a drone control station for a Predator 3.
Yeah, I had a similar story to Moose about, you know, tracking down a, what we realized was a
O-Day for a printer. And, you know, we thought for sure that this was a very advanced threat actor
burning an O-Day for this situation and found out it was just to run a ware server
off of our printers because they have so much storage. And so that, you know, again,
yeah, really excited. All right. Yeah. This is going to be like going to make me famous, right?
No, they're just running ware. So, but they somehow managed to get an O-Day and, you know,
that was what they spent it on. Cool. Well, I certainly don't have any stories as exciting as
those that the rest of you have shared in my somewhat boring career in InfoSec, not getting
to go around and do incident response all over the place. But, you know, I just want to take
the opportunity to thank all of you for accepting our invitations and coming here and sharing your
experience and, you know, helping all the attendees, you know, who are here at the Blue
Team Village to learn from you. I've learned a lot from you. I think this is, you know, this is such
great, valuable information and getting, you know, a variety of people's perspectives, you know, to
help us all, you know, learn new things and different ways to approach problems. It's
awesome. I want to thank you so much for participating. And for the attendees who are
here in the BTV Discord, we're going to move over to the voice talks chat. Anyone who's available,
panelists who are available, pop over there and take some questions like hallway con.
So I guess that's a wrap. Thanks, everyone.
Thank you.
Transcribed by https://otter.ai