start	end	text
8600	14280	Welcome everyone to the Blue Team Village incident response panel. We have an awesome group here for
14280	19940	you today with unique experiences and perspectives on security incidents and incident response.
19940	24640	I'm really excited for today's discussion. But before we get started, take a look at the poll
24640	30540	we posted on the Blue Team Village Twitter earlier today about ransomware and whether or not victims
30540	35460	of ransomware should pay the ransom. We are going to get on that topic in a bit and we would like
35460	41240	your input in the poll. Also, please post questions for the panel in the Blue Team Village Discord
41700	48140	in the Talks Track 1 channel, which we'll be monitoring. My name is Russell Mosley. I go by
48140	52800	SmokeEm on Twitter and I'm a lead with the Blue Team Village. I've been a Blue Teamer for my
52800	57740	entire career, building and managing security operations at small and mid-sized companies.
58000	63700	So I wear a lot of hats. I've done everything from sysadmin. I've been an IT director,
63800	68780	a security ops lead, a compliance auditor, and today I'm a CISO at a rapidly growing government
68780	75200	contractor. All my experiences involve developing and managing small teams. And as a result, I've
75200	79500	had the opportunity to do a lot of different things in the security field and build decent
79500	86540	security programs without a huge budget. I'd like to introduce my co-host, Xavier Ashe. Xavier?
86760	92120	Good afternoon, everyone, or good morning to where you're at. So my name is Xavier Ashe. I go by
92120	101440	Rubix1138. My IR experience is a little varied. I was an IR team lead at bit nine slash carbon
101440	110980	black. And also I was able to start the first IR team at SunTrust, now Truist. So 28-year veteran,
110980	117900	been around for a while. I can remember when my IR plan was you have to scan every floppy by hand.
117900	126440	That's me. Cool. Next I'd like to introduce Lip Moose. Moose is frequently posting to Twitter,
126440	131400	comments on exciting things that she encounters doing IR. And this week it was jump boxes that
131400	138420	look like flaming toilets, I think was one of the highlights. She refers to herself as many things,
138420	145040	including Florida woman, a DFIR violinist, and recently completed a cross-country hacker road
145040	152620	trip with cats and snails on board. Isn't that right, Lip Moose? That's fairly accurate. I did
152620	161500	remote DEF CON. We got to see a few goons. It was nice. So I go by Lip Moose on Twitter, also known
161500	168220	as Moose. My name is also Heather. I'm currently working at CrowdStrike. I'll dox myself today,
168220	174420	which is not a thing I usually do at DEF CON. But I'm very proud of where I work and what I do.
174420	179980	I work with a lot of companies of all sizes. I come from a background
181320	189440	for financial giants, as well as manufacturing giants, and have been doing this long enough to
189440	195200	where I think I've been awake and can engage myself in the 60s if I add all of my cumulative
195200	202740	IR experiences together without sleep. So I'm looking forward to retirement. I don't know when
202740	208680	that will be, but I do love what I do. Outside of work, I herd cats and I play violin. And so
208680	214440	I'd actually like to pivot and introduce somebody that I've worked with a little bit. 1099,
215580	223120	one of my favorite people in the world. Look, I'm sober right now, which is a problem. But,
223120	228860	Brian, I'll go ahead and pass it over to you. Well, hello, all you cool cats and kittens.
228860	234080	My name is Brian Moran, and I go by the incredibly clever Brian J. Moran on Twitter.
235360	239880	I've worked primarily in the Air Force. That's where I started my career, worked for a three-letter
239880	244500	agency, did that for a bit, got out of the military, bounced around a couple of private
244500	250940	sector companies, and eventually, back in 2014, started working completely for myself. So I
250940	257620	started my own company, and I am an IR consultant. That's what I do probably about 60% of the time.
257620	263580	The other 20, 30% is probably primarily focused on training and development plans,
263580	269220	and the other 10% of my time is primarily spent trying to keep the DFIR FIT going, where
269220	272840	it's all about just trying to get a little bit more active,
272840	276180	and hopefully we can all be on this little blue dot a little bit longer.
277940	282200	Cool. Yeah, I certainly was going to mention Brian's Twitter seems to generally be filled
282200	290260	with DIR FIT and comments on rum. You left the rum out. Well, I have some right here.
290580	292540	Oh, funny you should mention that.
294900	296900	I thought you were going to be better than me, Brian.
298800	304220	Let's hear from Virus before we have our toast. Virus, so he says he's only been a blue teamer
304220	309860	for about a year and a half with a background in red team, and he's currently doing what I
309860	315340	call kind of red-blue and innovative network defense for a large financial organization.
315340	322540	If you want to elaborate, welcome, Virus. Yeah, so I work at Stripe currently. I'm the
322540	334360	threat detection team. I guess the word for it is adversary direct engagement and threat hunting.
334360	340900	Background-wise, started in the dark gray end of the pool as a young kid,
340900	348080	like my first DEF CON, I was 17. Fell into penetration testing and then red teaming,
348080	352980	like full-spectrum threat stuff, and passed around the world doing that for a while.
353420	359220	Kind of drifted back towards pen testing, and then, surprise, had a kid, and, you know,
359220	362700	turns out that gallivanting around the world, breaking into buildings in random countries,
362700	367040	and getting arrested is not a good way to be a dad, so had to pivot for a while.
367760	372120	Did a stint doing mergers and acquisitions security at Salesforce and found out I'm not
372120	381360	bureaucrat. Now I'm doing blue team, so, yeah. Cool, that's awesome. Really looking forward to
381360	386080	sharing some of your perspectives that we heard in our planning meeting yesterday.
386800	392680	So before we get going, we wanted to carry out the DEF CON tradition of first-time speaker drink.
395780	400900	Everybody have their glass ready? Awesome. Cheers to IR.
409530	414790	All right, so to kick it off, you know, we're in a unique situation here in 2020, right?
415370	422150	With COVID forcing work from home for so many people. So I wanted to start with a question for
422150	428390	full-time incident responders, Moose and Brian. What types of incidents are you seeing the most
428390	433910	of right now? What are the trends in attack methods, persistence, and anything else you've
433910	440250	observed lately in your work? Yes, I can go ahead and start.
441710	449830	It hasn't changed too, too much. The volume has been a huge uptick. I'm still seeing a ton of
449830	457930	ransomware attacks. A curious thing that I'm seeing more of now is actually, I would say,
457930	464790	more intrusion via remote code execution has had an uptick recently. With a lot of the bots out
464790	469810	there, vulnerability scanning, I would say that those are getting hit before people can catch
469810	478390	just a little bit faster than I used to see. But a lot of the end game for what I'm seeing now
478390	485370	is either encryption and then extortion or exfil and then extortion. And I would
485370	490290	say that the volume has just gone up versus what it was like a year ago.
491430	497450	Yeah, I would 100% agree. The exact same stuff that's happening now has been happening for
497450	503330	the past probably two to three years has been the general trend. But the volume of it has just
503330	509510	immensely increased. And probably part of that is the bad actors have nothing else to do except
509510	515890	stay at home, just like we have nothing else to do. So the idle hands will play, I believe,
515890	523090	is the phrase that's used there. So the one thing that I have seen more recently an uptick of than
523090	530990	I have over the past year or so is actually O365 and not necessarily intrusions. It's users
530990	537850	clicking on links they shouldn't click on, and then the attackers getting access to stuff that
537850	543050	they shouldn't necessarily be able to get. So it's not necessarily a 365 intrusion. It's just
543050	548310	phishing, which is kind of directed more towards with the overall goal of getting some O365 access.
552300	558580	Yeah, that's interesting. I wanted to ask, a lot of companies have focused a lot of their security
558580	565760	on their networks, and perhaps not as much on their endpoints and cloud services.
566280	572980	And now that many of their workers are remote, they don't have the option to do monitoring
574500	580060	on their network, and maybe aren't as well prepared to monitor from their endpoints and
580060	586780	their cloud services. Would you agree that that's the case? And how is that impacting
586780	595280	what you're seeing out there? So I would say it definitely does. But the caveat to that,
595280	600800	and I already kind of doxxed where I work, is I'm getting calls because they're not necessarily
600800	607120	using an EDR to begin with, or maybe there's not as much focus on the endpoint. And I think that
607120	614680	what's neglected in cloud is that there's this perception of security that's not really there.
614680	619520	You're just working on a server in somebody else's data farm. And so you still need to put the same
619520	627220	protections on it, and monitor your VPC flow logs, and look at what VPC you're putting stuff in,
627220	633260	or what segment, rather. And you would monitor it just like you would something on-prem in your
633260	639520	environment. And I'm seeing that being so quick to set up that sometimes those steps are missed,
639520	646220	and then I'll get a call. Yeah, I think as we talked about a little bit last night,
646220	651500	one of the things is the data that you need to actually respond to a breach of some sort,
651500	656920	it's always been there. It's just never really been used or even thought of being looked at,
656920	660320	because before it was all, well, we're just going to block everything at the network layer because
660320	664280	everything is controlled. But now that everything's spidered out, all the logs are still
664280	668520	there. All of that's still there. It just never was bothered to actually be configured properly
668520	676060	in the first place. So that's why we're seeing what we're seeing. So with the more distributed,
676060	684380	I'm hearing more endpoints and more cloud, has that really changed the way that we're doing IR now?
688630	696630	I would say yes and no. So we're still asking for logs that are representative of the same data.
696630	700550	Instead of asking for firewall logs in this case, you're looking for NetFlow,
700550	708130	which might be DPC flow. We're still asking to run scripts to an endpoint, which whether or not
708130	712890	it's in a cloud or on-prem, it's going to be capturing the same artifacts. We're looking at
712890	718270	the same things on Linux that we're looking at on an AWS Linux server that we would look at
718270	725010	in a server in your environment that's physically there. So I would say that while things have
725010	730990	changed in slight ways in how we capture the data, so converting an EBS snapshot versus
732490	739030	a VMPK versus I'm going to run FTK to it, those little minor changes are there. But I think just
739030	743930	getting creative about, I'm still caring about the same things. I'm still caring about how the
743930	749350	thread actor got there. So your original access point, I'm going to look at all logs. I'm going
749350	754790	to look at, has there been any exfil, file creation, file deletion. So the same questions
754790	759850	we've been asking forever, we're still going to keep asking. It's just that creativity around,
759850	764590	we're in a different environment now. Unfortunately, those logs are generally
764590	770450	there for companies that rely on Office 365. A lot of times you just have to have the right licenses
770450	775350	to go look at them in cloud app security and other places. So what happens when an organization
775350	781570	hasn't really invested in logging on their endpoints or anything beyond the defaults
781570	787570	that come with Windows? You have a situation where you have people getting phished or
787570	792790	getting malware, and there's really not much to go on. What do you do in that case?
793210	798570	Generally, you're not going to have the easy access to get a forensic image and conduct
799290	804970	analysis on the device because it's somewhere far away from you, maybe on a slow link.
804970	806430	How do you deal with that?
808650	815090	Well, I think the old way of, well, we would just show up and collect data from somebody's
815090	819770	computer in the event that something happened. Obviously, that is no longer going to be
819770	826850	existing. So you have to get creative with essentially trying to get the same people
826850	831970	together on a call like this and figure out what you actually have access to. Not necessarily what
831970	837430	you have, because more than likely in the event that someone like myself or Moose gets called in,
837430	841810	stuff has already happened that's really, really bad. We're pretty much the last people you want
841810	846770	to see in the event of an incident. So you have to get creative with essentially figuring out
846770	851950	what you have access to and then how far back that goes, if it goes back at all,
851950	856850	and then try to build out from there. Because a lot of times when we show up somewhere,
856850	860970	the first question is, well, okay, do you have NetFlow? Do you have logs? Do you have
860970	865950	X, Y, and Z? And a lot of times, the initial part of an incident isn't even actually responding
865950	869510	to something. It's figuring out what we actually have access to and then trying to get creative
869510	873790	with ways to get access to that data, get it as quickly as possible, and then turn that into
873790	881570	usable information that we can use as quickly as possible. So what are some things...
883730	887890	Go ahead, Moose. Sorry. I was just going to say, I think one of the boons right now is that so
887890	894110	many people have invested in cloud security, so it almost negates the problem. A lot of times,
894210	900390	a threat actor's pivot point of getting into a lot of devices, we can think about, and I almost
900390	904830	want to turn it over to Byers for this one, but if you think like an attacker, you have a better
904830	912490	chance of getting the data you need. And I worked a case recently where, you know, I couldn't run
912490	916690	the scripts I wanted to, but I looked at what they did, and I said, well, let's just replicate
916690	921230	what the red team did, or what the threat actor did in this case, and let's red team it,
921230	927550	and I got the data I needed. So that's... sometimes you just got to look at the problem a little bit.
928830	935010	I think another curveball that I started to see is, you know, for companies that use Wwise terminals,
935010	940690	you really can't take those thin clients home, and so we're seeing an increased use of other
940690	947210	devices like Chromebooks, and so I wanted to see any advice out there if somebody that's
947210	953830	listening is going to be doing any type of instant response involving Chromebooks used as,
954590	960010	you know, thin clients. Is there any advice that you can give to those folks out there?
962630	968590	Looks like we lost Moose for a minute. Well, so I know that there's been some research into
969050	973150	essentially the data that you can collect from Chromebooks. I know Jessica Hyde from Magnet
973150	978110	Forensics has done quite a bit on that, and I believe this past year she's given a few talks
978110	982110	on that. So there is data you can get from Chromebooks. You have to get clever with it,
982110	986450	but one of the things that's interesting is if you use Chromebooks, more than likely you're using
986450	991550	some sort of Google instance, and Google has a ton of data about you. Whether you use Google or not,
991550	996190	Google has a ton of data about you, and because you're able to get all of that data back from
996190	1001830	them. And admittedly, there's a lot of stuff to go through. But it's not necessarily... like
1001830	1006290	Chromebooks are incredibly secure. I love Chromebooks. I recommend Chromebooks to pretty
1006290	1010790	much everybody who gets click-happy with any kind of links. Just use a Chromebook for a while,
1010790	1015090	and hopefully that's going to eliminate a big portion of your attack surface. It's not going
1015090	1019950	to completely eliminate it, but it's going to take a big portion of that away. But essentially,
1019950	1024270	at the end of the day, you know, we're still trying to look for the oddity outlier information
1024270	1029510	of data access, you know, data exfil, stuff like that. And with the stuff that you can get from
1029510	1033750	Google specifically, like all of that's there. It's just a matter of being able to get it,
1033750	1036150	being able to process it, and being able to digest it.
1037250	1043550	What are your thoughts, Virus, on doing IR with cloud and distributed endpoints without the
1044610	1047150	hardened network of tradition?
1049570	1054130	In some ways, I guess I haven't been around in the Blue Team ecosystem enough to know the
1054130	1062410	difference. Like, the few roles that I've had that are hard in Blue Team, I've always had the
1062410	1068110	power to at least have a say in kind of what our telemetry looks like. And I mean, coming from the
1068110	1073830	red side, I've frankly never really bothered wanting network stuff. It's always, you know,
1073830	1078310	anything that's a network artifact, I only care about it in the context of what the endpoint did
1078310	1085150	in terms of interaction. And so it's just usually easier to filter on down and kind of pinhole what
1085150	1093730	I want. I have definitely, within the past year, I've dealt with some vendor breaches, which are
1093730	1098450	all public, so I can say that, but I guess I won't say exactly what they are. So, you know, I won't
1098450	1105670	get fired immediately. And it's kind of interesting. One of the nice things about, I shouldn't say
1105670	1110670	nice, but one of the interesting things about kind of being tangentially related to a breach
1110670	1116390	that gets released public is suddenly you have like a legal stick to beat them with to actually
1116390	1121970	get IOCs back from the breach, as opposed to just reading the news and having a Twitter opinion like
1121970	1130130	the rest of the world. And from what I've seen, it definitely, like, the game for the vectors
1130130	1135430	in terms of the game for getting in doesn't seem to have changed. It's usually not the server.
1135670	1142310	It's usually it's way easier to get on a laptop. And then because no one knows how to network in
1142310	1147410	the cloud, like everything is just grabbing the AWS token or the Azure token off disk and then
1147410	1153030	going to town. And then usually from there, it's a, oh, by the way, either there's no logs or there's
1153150	1157350	a ton of logs, but nobody's ever actually triaged them to find out which logs are useful and which
1157350	1162770	ones aren't. So there's no logs. And then it's usually going back and being like, oh, well,
1162770	1167030	after they took this token from your machine, then they directly logged in with this super
1167030	1170970	weird user agent that no one's ever heard of and stole all your data and you did nothing.
1172490	1178810	At least that's been the trend from what I've seen. The Chromebook thing, there actually is,
1178810	1184190	I don't have a lot of specific details that I can share, but I do know that there has been
1184190	1192970	some publicly released stuff on Google implementing a kind of AuditD type thing
1194290	1199510	that is going to be very similar to like OS X's security center in terms of accessible data. So
1199510	1203030	I think that's going to be a game changer with all of the EDR tools that are out there that are
1203030	1209050	hooking into AuditD as kind of a funnel for data. I think that'll be really cool. And then also
1209050	1213930	anybody who's looking to do DFIR on Chrome locked devices in general, I would actually
1213930	1220830	route you to some research from about eight or nine years ago from Mike Osmond and Kyle Osborne.
1220830	1226990	They did some offensive research on basically how everything in Chrome is a browser and the
1226990	1230770	browser doesn't know the difference between a file system link and a remote link. And while
1230770	1234630	Google has obviously tried to address the security concerns that they found with some of that
1234630	1238950	research, you can actually use a lot of those vectors to gather debug information about the
1238950	1242830	Chrome browser environment. And that ends up being almost equivalent to system logs on a
1242830	1249450	Chrome locked device. And you started your comments there with, you know, first getting
1249450	1254450	on the laptop and then pivoting out from there. What's the most common method for getting on the
1254450	1262890	laptop? It's the usual. It's either phishing or I've seen some supply chain stuff recently. I
1263320	1269650	came out relatively recently that I know affected a bunch of shops. I've seen some slightly less
1269650	1275890	talked about in the media because the Go ecosystem is so blurred in terms of the concept of standard
1275890	1281710	lib versus some person's stuff on GitHub. But I have seen some bugs there that have caused some
1281710	1289990	supply chain issues. There's little known fact, I have seen, well, it's a more known fact that OSX
1289990	1294990	attacks are on the rise, but it's a lesser known fact that Perl is still on the OSX boxes. So,
1294990	1300610	I have seen kind of some like, you know, oh, I haven't seen that attack since like Red Hat 5,
1300610	1305310	but, you know, that Perl subsystem is still on OSX and way more people look at Python on OSX
1305310	1311930	than Perl. So, there's a little more of that. That's great. I'm curious, Brian, your observations,
1311930	1315650	you know, how they compare to viruses feedback, you know, and what you're seeing.
1316990	1324630	Yeah, I think it's... so, I don't think the methods have changed. The methods haven't changed. And
1324630	1329170	honestly, the methods don't really need to change because they still work. So, if they're still
1329170	1335610	effective, there's no reason to burn some, you know, some zero day exploit or even go the
1335610	1339410	route of looking for like RCE or anything like that, when if you can get somebody to click on
1339510	1343090	a link and get the information you need, and that still works, like, that's what the attackers are
1343090	1347570	going to do. Because, you know, again, at the end of the day, like, the attackers are human just
1347570	1352450	like us. Like, they will do just as much efforts as needed to get them the goal that they need.
1352450	1359570	And, you know, you have to be able to account for that. One of the tools I haven't heard yet so far
1359570	1367110	for telemetry is, you know, cloud-based DNS. So, a couple providers where you can get that from a
1367110	1371410	centralized source. Are we seeing more people, you know, take that on as we get to a more
1371410	1379250	distributed environment? Or is that still an afterthought of like, oh, yeah, we can't get good
1379250	1386810	DNS telemetry because everybody's at home using their home DNS? Has anybody got any experience
1386810	1395830	with that? I do. I've had some that have it, but not very many. And with the caveat of saying
1397090	1404110	that if they're calling me, something's gone terribly wrong. Or, or counter, we do do security
1404110	1407790	assessments in environments that might be something to come back with because they've set up a cloud
1407790	1413090	environment and they want advice on how to make it better. So, I will say that I have a little bit of
1413250	1419670	a bias at answering this because even though I don't see it as much, it's usually because they're
1419670	1427750	calling for help. Yeah, I mean, control is kind of a crapshoot. But in terms of telemetry, like,
1427750	1434630	man, the combination of a low-level passive DNS service and OS Query seems amazing for DNS to me.
1440120	1444280	Yeah, so I'm going to say pretty much the exact same thing that Moose did. Like,
1444280	1449020	I would like to see it more. However, a lot of places don't see it. And the places that call
1449020	1454500	us, more than likely, there have been several shortcomings that led to calling people like us
1454500	1459080	in the first place. So, typically, we wouldn't see something like that because we're not the
1460740	1464860	security architects or anything like that. Like, something horrible has happened,
1464860	1468540	everything's on fire. I'll call them in and they're hopefully going to calm us down.
1468720	1473680	So, before we pivot to another topic, if you were called in to give advice to a company today,
1473680	1478620	and they wanted to spend some additional budget for security with people working from home,
1479020	1486740	where would you advise them to focus? I mean, I can tell you where we have an
1486740	1494560	entirely remote workforce right now. And there is some scatter in terms of operating the system
1494560	1500880	layout. And pretty much 100% of our EDR effort at this point is done using OS Query. That's
1500880	1504900	pretty public information. I don't really feel like I'm giving away secret sauce saying that.
1504900	1509460	And at least from a telemetry perspective, even with specific regard to DNS, I can tell you,
1509460	1514280	it's pretty amazing. Like, we have other issues, but
1514820	1518160	DNS telemetry is certainly not one of them from my perspective.
1520340	1526660	I think the one piece of advice I could give is don't ignore any front. I see a lot of people
1526660	1532360	spending very heavily in one arm rather than maybe ignoring the other. So, they'll focus
1532360	1537900	just on end point or they'll focus predominantly on network side. And I think that the balance
1537900	1544480	is still very, very important. And I would say to borrow something that we all as a team talked
1544480	1550720	about yesterday, too many people are trying to eat the elephant hole. So, don't do that. Don't
1550720	1557080	keep your network flat either. So, my top advice to people is make a list of priorities in your
1557080	1563320	environment. What would hurt the worst if that went public? And start segmenting there. Start
1563320	1569180	segmenting. Know your group permissions. Run some of these Red Team tools. Run Blood Pound,
1569180	1573300	Sharp Pound in your environment. Get a better picture of what it looks like and then make a
1573300	1579380	decision. But make an educated decision, not just a flat blanket decision because not all
1579380	1587080	businesses are the same and not all businesses have the same needs. I think before spending any
1587080	1591960	money, the first thing that a company truly has to do is take a look at what they already have,
1591960	1597400	the people they already have, and then try to build a solution from that. Just like Moose said,
1598020	1602380	a lot of times when I go to places, they don't even have something basic like a network map.
1602380	1608300	So, then I look for the attackers when essentially they dump IP.txt to some system and then I have
1608300	1612300	an actual network map that I can go on because the attackers have left their forest, fortunately. So,
1612300	1615940	that's literally the only insight that we have to what's actually on the network is what the
1615940	1621100	attackers have done. So, you can't be just spending money while you have all of these
1621100	1626700	gaping holes that just have never been addressed. You have to do a serious and honest and truthful
1626700	1631320	evaluation of what you currently have already before you make a decision on what you're actually
1631320	1637120	going to do to try to improve. And related to that, skills versus tools, right? Should you be
1637120	1642640	investing in your staff and in their skills, training, sending them to conferences like
1642640	1647820	DEF CON where they can learn from the community what others are doing and increase their
1647820	1651320	capabilities instead of buying tools? Thoughts on that, anyone?
1652440	1663080	I have strong feelings. So, I think it's both. So, you really need to invest in your people but
1663080	1667960	don't overtax them so much that you're not giving them the tools to work with so that they're
1667960	1674820	getting alert. You don't want to have them working with the bare nothing and then getting mad
1674820	1680380	at them when they miss something. So, you know, take the team you have and really build them up
1680380	1686040	but then also listen to them on their level of what they'd like to use and what they'd like to
1686040	1690100	have and use that training of sending them to conferences and coming back and gathering that
1690100	1695940	information to inform your decisions. And I think that it really needs to be that whole cycle of
1695940	1700660	you're investing in their education and then going to conferences and networking and letting them
1700660	1706060	come back to you with a solution. And that is going to sometimes be investing in tools for them.
1707080	1712700	Yeah, I agree 100%. You know, you really want to listen to your team, you know, listen to
1712700	1717980	their experience, right? You know, get them to get out there and participate in the community
1717980	1724040	and learn what works and, you know, take their advice on those areas, you know, and figuring
1724040	1732220	out where to invest in your team. So I wanted to talk a little bit about IR teams and
1732220	1737280	folks who want to get into IR. So, you know, I think they're two related topics, but a lot of
1737280	1741660	people have asked if we could give, you know, what's your best advice for people who are interested
1742200	1750300	in getting into the incident response field? And that's for anyone who wants to go first.
1750300	1759000	I'll go first. So my personal opinion on how to get into IR or really kind of any info sec,
1759000	1764240	like specific sort of doctrine at all is just to simply be curious, like ask questions,
1764240	1769260	try to research stuff on your own, figure things out. Like you may not be the best person at
1769260	1775900	assembly or firmware architecture or IR or malware reversing or anything like that. But
1775900	1780020	as you find questions and, you know, find things that you're curious about and want to research
1780020	1784280	and learn more about, like you'll generally kind of find an area that you want to go down. And
1784280	1789860	at the end of the day, like you have to kind of decide if you want it to be a typical job
1789860	1793520	where you work eight to five every day and show up and you get paycheck and that's it.
1793520	1798440	Or if you want it to be a career and you actually want to be good and want to actually, you know,
1798440	1802200	be better at what you do and be more effective and be more efficient and network with people
1802200	1809620	and all of that. And it's not really a lifestyle decision so much as it has to be an internal
1809620	1820460	passion and desire that you actually want to do something. Anyone else want to comment on that one?
1822080	1828420	I want to say that you're safe, but I will, I'll chime in and say everything that Brian said
1828420	1835780	absolutely settled down on that. But just know that there's no wrong path and that to you can
1835780	1840960	do anything in your spare time that you're passionate about and turn it into IR. The
1840960	1847760	difference is what I typically see in good IR is you don't stop at the first answer.
1847980	1853300	So, and you don't stop at the second answer either. It's this stubbornness and that drive
1853300	1859620	of being stubborn to where, okay, make yourself a home lab, image your own devices, break them. Like
1859620	1866040	see what you can do and see what you're seeing and use that to train yourself. And it
1866040	1871960	really is something that it takes a lot of training and that there's not just you're
1871960	1877960	there one day. So, I'll admit myself, I get imposter syndrome a lot, but that's because
1877960	1882200	every day I have to Google. There will be something every day that I don't know the
1882200	1896020	answer to or I haven't seen before and I am researching the hell out of it. I'm saying
1896020	1900100	sometimes it feels like I'm banging my head against the wall, but that doesn't go away.
1900100	1904120	And so, if you're okay with that and you have the personality that has that drive,
1904120	1909080	you're going to do great. And being, like listening to this right now, it shows enough
1909080	1913420	that you're willing to listen to the five of us ramble for an hour. You're probably
1913420	1917720	in good company and you should probably just keep pursuing this.
1919920	1926980	One of the, on that line, you know, talking about getting into IR, talk to the, you know,
1926980	1935240	more along the lines of the type of work and particularly about the stress. You know,
1936000	1942540	typically IRs, you're more like a fireman. You're dealing with sometimes some high stress
1943340	1950800	situations. And so, any comments or suggestions for the listeners around, you know, kind of
1950800	1958140	dealing with the excitement and the stress that can come along with being part of an IR engagement?
1959180	1965160	Well, I think one of the things that you have to take into account is, so first of all, like from
1965160	1971520	an IR perspective, like you have to try your best to be calm. Like you can't overreact like
1971520	1976440	everybody else, like let everybody else overreact, let everybody else run around and scream and yell
1976440	1980440	and all of that. Like you have to be the calming voice. However, that doesn't mean internally,
1980440	1984820	you're not screaming and yelling too, because you absolutely are. You just have to portray yourself
1984820	1991040	as having that calming presence. And you also have to essentially, not necessarily,
1991040	1996220	not necessarily have the right answer, but you have to have a answer because the path to recovery
1996220	1999800	always starts with one step forward. And it doesn't matter if that step is even down the
1999800	2005540	wrong path initially, like you just have to go forward. Because the takeaway that you can always
2005540	2010660	have, no matter what, is somebody out there has it much worse than you currently do. And you always
2010660	2015540	have to remind yourself, like as bad as everything is, like there's some company out there that got
2015540	2021280	owned and they are in, unbelievably, an even worse situation right now than you are.
2023400	2028020	I think that everybody in IR is a little bit of an adrenaline junkie,
2028020	2032000	and that you're kind of lying to yourself if you're not. Brian, didn't you go like
2032000	2035700	space jumping or something at one of the cons that we've been to recently?
2035920	2042100	Yeah, I jumped off the stratosphere in Vegas a couple of years ago, twice, because I wanted to.
2044540	2050200	I was going to take the question and ask each of you individually,
2050200	2053700	you know, how do you get away from it? How do you deal with the stress?
2054660	2058700	So Brian goes jumping off the stratosphere in Las Vegas.
2059580	2063480	How about you, Moose? How do you get away from this? What are your breaks?
2065760	2071780	Well, it's twofold, right? Like, everything I see on a daily basis,
2071780	2079420	it's somebody else's worst day. I don't work for their company. And I had to realize this
2079420	2084540	about myself and every job I've had. So I've not always been IT. I used to be vet tech,
2084540	2088880	I used to be medical, I was a lifeguard. All of these things have a similar, like,
2088880	2096620	vibe in that it's really high stress, but it involves helping. So I think, to me, my release
2097080	2101760	is in seeing that step forward and that progression. And even if it's not happening right
2101760	2106820	away, being able to reassure the people that I'm working with that it is truly happening,
2106820	2111540	at the end of this, their environment will be better than we left, than it started at.
2111760	2118520	And then, if that doesn't do it, I have cats and a violin, and on the worst day,
2118520	2124220	I'll just shut everything down, everything gets unplugged, and I open up the violin case,
2124220	2129200	and I walk away. And I think that's really important, is that you don't have to eat,
2129200	2135180	sleep, and breathe this. You really should walk away sometimes. And that's what's kept me alive.
2138380	2142540	And how about you, Virus? I mean, it sounds like you certainly have some of the adrenaline junkie
2142540	2150100	in your experiences, in your past, breaking into buildings, you know? So what do you do for breaks?
2151960	2159220	Shenanigans. I mean, I guess the first thing is, like, don't get stressed. Like,
2159220	2163200	that's a big part of it. I mean, I've made a lot of... I've definitely steered a fair
2163200	2169080	amount of my career decisionary of, like, you know, when I finish an arc, I go,
2169080	2175300	did I like that? Or was it just stress in the bad kind of way? And then I use those takeaways to just
2175300	2179780	not do that anymore. You know, kind of quantify how much my time is worth. Because if I spend
2179780	2186420	the whole time when I'm not at work, stressed out about work, then I'm essentially working for free.
2186420	2190240	So if I factor that into my paycheck, I'm like, oh, this isn't that worth it. So that's definitely
2190240	2194060	the first step, is just don't do stuff that stresses you out. You know, know what your zeal
2194060	2200340	is. But on a practical level, like, I played music for, like, 25, 30 years, semi-professionally. So,
2200340	2205660	you know, I do that. I kind of zen out. I cook. I dry-age meat. I make my own gin. You know,
2205660	2212960	I like scotch. I do a lot of drugs. And then I play with my boy. You know, I'll stand up
2212960	2216080	from my desk when I'm working on a hard problem and just say, hey, let's go for a rock. And he's
2216080	2223060	great. So that's amazing for him. It's great. He just runs around. But yeah, it's, I mean,
2223060	2228560	you know, I think a lot of it is just you got to be holistic about your approach. And if you love
2228560	2232280	the chase, you got to love the chase. And you got to love your chase. You got to love the chase
2232280	2237580	that you're in and know where it is and just not lie to yourself. You can't, you know, do it. And
2237580	2241940	I mean, there's definitely an undercurrent if I'm being extremely frank and extremely personal
2241940	2245860	about it. I do think people that, I think there's a fair amount of people on the blue side,
2245860	2252620	specifically, who have a very, I don't want to say ethical, but they have a very empathetic
2252620	2259380	attachment to the world, slightly larger, maybe than most red teamers do, where at some level,
2259380	2262320	they're kind of in it because they want to, they want to fix something. They want to make something
2262500	2267600	a little better. And I don't know, I mean, I wouldn't call myself a complete prick, but
2268460	2271700	I'm definitely on the side of like, yeah, I like money.
2271700	2272580	Yeah.
2273860	2278880	Long as I moved the needle a little bit and I got paid, I'm not too worried. Maybe that's a
2278880	2283880	whole other conversation. If someone knows you from the outside, you're one of the nicest people
2283880	2293160	I know, so that's a lie. If we could get back on the topic of, you know, of the IR field and IR
2293160	2300260	teams, we had a question in the BTV discord about, you know, what are your top recommendations,
2300260	2307880	and maybe some don'ts, you know, for growing your SOC and your IR team? You know,
2307880	2313240	if you're at a company and, you know, it's growing and, you know, you get budget to
2313240	2319600	have some full-time folks doing security operations and, you know, skills and tools
2319600	2324260	ready to do incident response, you know, where do you want to focus? You know, what skills do
2324260	2329560	the folks need to have? What are some of your preferred tools? And I know it depends a lot,
2329560	2333920	but just in general, you know, where would you focus? Brian, why don't you go first?
2334980	2342500	Well, I think it goes back to... it very much does depend, but it also depends on what you
2342500	2347460	currently have already and essentially what you have the ability... what you want to essentially
2347460	2353300	try to strive for. Like, if you want to go a very, very, you know, cheap route as much as possible,
2353300	2357880	you would use something like OS query or use something like Velociraptor and, you know,
2357880	2362140	try to essentially keep it on the cheaper side. But, you know, at the end of the day,
2362140	2366100	it's still going to be able to work. Of course, if you have unlimited budget, like, you just line
2366100	2370060	up boxes, you know, like crazy, and then you have all the tools and everything. But, you know,
2370060	2375560	to be completely honest, that's not feasible for anybody. Like, companies will not just blindly
2375560	2380920	give you ridiculous amounts of money to spend on people and tools and resources. So, you have to
2380920	2385860	try to make whatever you have work with what you can. And that's one of the things that I stress.
2385860	2390780	It doesn't necessarily matter what the dollar amount is. It's trying to make every dollar
2390780	2395380	count as much as possible. So, that way, at the end of the day, you end up with a truly better
2395380	2400240	result. If you have a very high budget, great. Like, you can afford expensive tools. But do you
2400240	2404480	really need those expensive tools? Do you really need an expensive person to come in maybe for,
2404480	2409340	like, you know, three, six months or something like that and help train and make the team better
2409340	2413640	to already use the stuff that they already have? You know, in that case, you could argue very much
2413640	2418660	that bringing somebody in external for a little bit and helping them not really as a staff log,
2418660	2423760	more of kind of like an overseeing mentorship almost. Like, that is a much better investment
2423760	2429460	than just blindly buying, you know, a million dollars worth of random tools. So, it truly does
2429460	2434340	depend. But, you know, it's like everything else. Like, you have to make a business decision about
2434340	2440120	it and essentially try to maximize every dollar and get the best value for that as much as possible.
2443040	2451320	You know, and we've used the word a few times in this panel, telemetry. You know, so,
2451320	2456600	from my perspective, one of the first things that I would want to do is, you know, is set all that
2456600	2462440	up, is set up, you know, a central logging system, whether it's SIEM, whether it's, you know,
2462440	2469840	something much simpler. But, you know, for me, doing incident response, you know, those are the
2469840	2475400	keys, right? I mean, I've heard it said, you know, logs are everything. And that that would be one
2475400	2480820	of your first steps would be to make sure that you, you know, you're gathering enough information
2480820	2486420	so that, you know, when an incident happens and just in general, you know, having logs has so
2486420	2493260	much value for IT support, for everything that we do in security. My approach to this is really,
2493260	2501320	it's like a three-step. It's like a three-layer moniker. So, I think the first thing that it's
2501320	2507660	important to grasp is that a program, we talk about a program, whether it's an IR program or
2507800	2512720	a security program, large or blue team program, a program is business process. That's what a
2512720	2519320	program is. It's not blinky boxes. It's not even the talent in your bullpen. It's documented process.
2519320	2524880	That is the definition of your program. So, the first rule is if you're going to build a program,
2524880	2530580	build a program. That means encourage your people to, and this kind of leads into the second thing,
2530580	2536560	which is tribal knowledge is the enemy, right? So, like the first job of everybody on the team
2536560	2541600	is to take everything in their head and inject it into business process. That is your job. It
2541600	2546460	doesn't matter if you are, you know, the self-proclaimed bottom rung of the team or if
2546460	2551300	you're the epic gangster rock star at the top. Your job is to take whatever is in your head that
2551300	2556220	applies to the program and to build the program. And the more that you encourage people to do that
2556220	2561720	and the more that you reward them for doing that, it's like a self-incentivizing loop, right? That's
2561720	2566980	how you balance burnout and all this other stuff because it's not focused on one person or one
2566980	2572380	tool. And then, like I said, the second thing is, and I feel like this is way more common on red
2572380	2576260	teams than it is on blue teams, and I kind of don't know why. It is extremely common on the
2576260	2581040	red side to not use tools that are touted as being the latest and greatest thing just because you
2581040	2586100	don't know them, right? I mean, as a red teamer, I feel like there's this moniker that is
2586520	2590320	accepted more and I feel like it's true on both sides. So, I kind of don't understand why it's a
2590320	2594740	lopsided. We're like, you know, it's the old ninja moniker, right? We're like, not every ninja uses
2594740	2599120	every weapon, right? You have your weapon and it's got your epic moves in it and you know it like the
2599120	2603000	back of your hand. It's almost an extension of yourself and that's the point. I don't really
2603000	2608040	think that's different on blue team. I mean, case in point, the team I'm part of at Stripe that kind
2608040	2611880	of got built about a year and a half ago, we were all very new to the company and was, you know, granted
2611880	2616260	we were very fortunate to have leadership with a lot of experience and so they reached out and
2616260	2620940	you know, there's a lot, there's an extremely high amount of experience and talent on my team that's
2621080	2626300	a little bit askew for most. I'm not saying that to brag, I'm just saying, you know, a lot of
2626300	2632200	teams don't have that in their tool bin. But the first thing we did was take what EDR we had
2632200	2636480	and rip it out. We ripped out almost everything and then built everything off of bare bones telemetry
2636480	2642040	because those were the systems we could use better. That would, that yielded better to our kung fu.
2642040	2649640	And then from there, the third thing is build, like tool your kung fu. You know, take your
2649640	2654860	brain dump, put it into your program and then hypothetically start from scratch and go
2654860	2661040	what tools do we need to do this, what's written better? Frankly, every deviation from that is a
2661040	2667080	waste of time and money. So you said that you ripped out your EDR and you built up bare bones telemetry
2667080	2672380	are the things that are most useful to you. Can you elaborate on what those are? I mean, it's literally
2672380	2677660	just stuff that's built into the OS. I mean, we use OS query at scale so that because we use the
2677660	2682580	fleet manager concept in order to kind of achieve orchestration goals. But our detection system
2682580	2688960	is entirely homegrown. Our response system is entirely homegrown. Our orchestration elements
2688960	2693320	and the way that we do log tracking is entirely homegrown. Even our pace reporting. I mean,
2693320	2697320	we're kind of sort of bubbling around the hive right now, but it's honestly, it's all Jupyter
2697320	2702960	notebooks and Python. It's all just kind of manual tooling. And at the end of the day,
2702960	2709580	if I take that and I put that in the ROI bin, we spent way less on human time writing some Python
2709580	2715500	than we would ever spend on third-party utilities. And to be honest, most of the third-party
2715500	2719880	utilities are trash. I mean, I won't pull any out just because I don't want anybody to
2720500	2725340	kind of, you know, have to be under the boot because I said something, but like,
2725340	2729740	man, there's just so much money spent on these tools that just don't even do what they say on
2729740	2734480	the side of the box. I just, I don't understand why that flies. And then that's just, you know,
2734480	2738280	I don't know. Say what you will about red team or blue team infra, at least the red team stuff does
2738280	2741840	what it says. It just doesn't say a lot. Blue team stuff, man, this seems to be a problem.
2741840	2744340	Cereal box, nutrient banks are just wrong.
2748420	2755300	So, Rubix, you've built an IR team, right? I'm curious, you know, your thoughts on all this.
2756220	2763240	Yeah, you know, it was, I always go back to the, you know, kind of build it using the fireman
2763240	2771340	analogy. So IR for, you know, a, you know, we're in financial services, there's a whole lot of
2771340	2779700	downtime. And so figuring out how to practice and be ready for the big event and giving them
2779700	2786320	enough work of a real, you know, I guess, you know, smaller level incidents to be able to stay busy
2786320	2792920	and relevant, but then also, you know, plenty of time to prepare for, you know, bigger stuff. And
2792920	2801780	so, you know, lots and lots of practices, you know, tabletops, drills, and, you know,
2801780	2810600	encouraging our red team to really give us some good scenarios. And so I think that if you are,
2811360	2819720	you know, not balancing that, you know, if your guys are all, or folks are always spending time,
2819720	2826920	you know, doing, you know, we talk about being really busy, that they might not be honing their
2826920	2831180	skills enough. And so, you know, finding that balance between, you know, doing and training
2831180	2845280	and practicing was very key. I think the one thing I have to add to that, and everybody else
2845280	2851460	has given really good advice here. And this is a part that I'm really passionate about, because
2851460	2856780	it's one of the reasons that we lead companies. Don't silo your people and listen to what they
2856780	2862720	have to say. If you have somebody on your team, it doesn't matter what notch in the chain they are.
2862800	2868280	If you're calling on them for some kind of instant response, whether it be a SOC person,
2868280	2873580	somebody who's doing sys admin work, someone who's a network admin, you're trusting them
2873580	2881060	with that role. Trust them if they say they have a passion in response as well. Let them
2881060	2887580	play in other fields, because there is not a silo in incident response for our attackers,
2887580	2892820	so there really shouldn't be a silo for our responders. And one of the reasons that I work
2892820	2899320	where I work is because there are some days where I'm not incident response. I have the freedom to
2899320	2905780	do red team stuff. I have the freedom to help build our tools if I want to, or scripts. Some
2905780	2911500	of the most brilliant scripts I've seen to date were built by my teammates in response to the
2911500	2918600	Citrix vulnerability, for instance. So, you know, knowing where your strengths are on your team
2918600	2924100	and building those, and then listening to them and what they say they need. Because, you know,
2924100	2935720	if you're the one building, that assumes that you're at a higher level position,
2935720	2942200	and I think that's really key, is to... my cats are wrecking the apartment right now,
2942200	2946780	so if you hear anything bad, that's what I keep looking over for, is anytime I'm on one of these,
2946780	2952880	they wreck something. So anyways, but listening to those people that you've hired, and obviously
2952880	2958140	you trusted them enough in the first place to hire them. They're smart people, and we're smarter
2958140	2966560	not because we know something, but because we know who to go to when we need certain knowledge.
2966560	2972800	And I think leaning on your teammates, no matter what level they are in your company,
2972800	2978080	having that kind of group decision is still important.
2978720	2985620	Yeah, I think that's really great advice, you know, that more companies should heed,
2985620	2993300	to try to cross-train their folks and give folks time for training and for research, right,
2993300	3001100	and have budget for tools and devices, right, and lab time. I think it's really important.
3002420	3013540	So I wanted to pivot and talk about ransomware. So we had a passionate discussion about this
3013540	3019580	yesterday, and I'd like to try to rehash some of that for our attendees. And to kick it off,
3019580	3027340	earlier today, the Blue Team Village Twitter, we posted a poll, and we were looking for people
3027340	3033800	to give us their opinions. The poll question is, your organization is the victim of a ransomware
3033800	3043280	attack. Should you pay the ransom? And about 89% of the responders said no, and about 11% said yes.
3044080	3050740	I think, you know, in summary of our discussion yesterday is, it's not that simple. But,
3050740	3055060	you know, for the benefit of everyone who wasn't in our private discussion yesterday,
3055440	3061380	you know, what goes into the decision of paying the ransom or not? And I think
3061380	3064820	Moose started first yesterday, if you want to go first again today.
3066220	3073560	Sure. This is the most, like, let me tell you right now, this question made my blood pressure,
3073560	3078780	like, go up a little bit. Because for me, when somebody says, do I pay the ransom?
3079100	3083800	On any call I have, or even if I've been in person, I look at them and say,
3083800	3087520	well, what does your legal team say? And that's still kind of my answer,
3087520	3093340	is that I don't think you should make that decision for your business in a vacuum. And
3093340	3100500	I usually recommend that people talk to either their internal counsel or have external counsel
3100500	3107680	involved in that decision making process. And not necessarily make a decision for themselves.
3108540	3113080	Because these are, if you look at it this way, the way I like to frame it,
3113080	3116460	because I'm not going to say yes or no, spoiler alert, I'm not going to say yes or no,
3116460	3121380	because it really does depend. And that's not my decision as a responder to make.
3121380	3126420	But I think that the most intelligent things I've seen done was calling on experts who do
3126420	3131660	this every day. Because if you think about who hit you, you're not their first hit.
3132000	3137840	You're not the first people, they've probably ransomed, they've hit other companies. These
3137840	3142420	are professional criminals that are going after you almost all the time, if they're exporting you
3142420	3148780	for something. So get a professional to represent you. That's kind of my opinion. And I will preface
3148780	3160640	that with that as well. Okay. Brian, you kind of immediately had a, you know, from the heart
3160640	3167600	response that you then elaborated on. Could you share that with all of us? So as Brian Moran,
3167600	3174900	human being, my first response is always no. However, that is simply me as a human being,
3174900	3182040	because I don't feel that it's right to pay criminals for doing a crime. However,
3182040	3188180	we'll preface all of that with put on the consultant hat, the business process hat and
3188180	3194420	all of that. And honestly, the answer is more than likely, it's usually cheaper to pay the ransom
3194420	3201940	than it is to deal with the cost of everything else involved. Like, it should never, just like
3201940	3208400	it should never be anybody on the InfoSec team ever. It should never, ever be their call. It
3208400	3212700	should be legal's call because legal absolutely needs to be involved. Not just from the perspective
3212700	3217400	of, is this okay with the company? Also from this perspective of, you know, you're potentially
3217400	3222200	looking at paying somebody in another country, like, are there sanctions against that country?
3222200	3228300	Like right there, you could violate even more laws just by paying the ransom. So now you're in
3228300	3233800	double because all your debt is gone and now you're getting sanctions and fines and investigations
3233800	3238180	levied on you because you are literally, you know, potentially financing something that you shouldn't
3238180	3244560	be financing. So it's not our call to make at the end of the day. Like, you know, we can respond and
3244560	3249660	say, here's X, Y, and Z, here's our thought process. But generally from the overall business
3249660	3253960	perspective, it's usually much cheaper just to pay the ransom and be done with it than to deal
3253960	3259120	with any of the public scrutiny, deal with any of that. But, you know, we're not lawyers.
3259120	3263500	I don't think any of us are even going to pretend ever to be a lawyer. Like, even if we say at a
3263500	3268720	Holiday Inn Express, we are definitely not lawyers by any stretch of the bean. So we can have our
3268720	3273240	opinion and we can give you thoughtful, evaluated opinions, but like our actual thoughts at the end
3273240	3278460	of the day truly don't matter because it 100% decides on legal and what the course forward
3278460	3286040	is from them. Yeah, thank you. You know, I think a lot of folks have that reaction of, no,
3286040	3290520	you know, I don't want to pay criminals, but there is so much to the decision to be made.
3290920	3296400	And Virus, you came at this from a different angle than I've ever really thought about it.
3296400	3303760	And that angle was about ransoms and kidnappings and kind of relating ransomware to that type of
3303760	3309460	crime. Could you, could you rehash that for everyone? So I basically distill everything
3310000	3318000	that I tried to kind of get across in our joint meeting as ransomware is a crisis. The breach
3318000	3323580	that put together is an incident. So if you have a flat out answer on whether or not you should pay
3323580	3328180	the ransom when it comes to ransomware, you're doing it wrong because that's not how crisis
3328180	3337880	management works. And I don't understand why as even an industry InfoSec would try to solve
3337880	3345860	crisis management when every major Western power, every branch of every military and military
3345860	3351620	contractor that's worth their salt, every political, you know, wing contractor, I mean,
3351620	3358780	crisis management is a well understood science that has been researched by every tactical
3358780	3366120	engagement and business risk school in the world forever. And the fact that the thing that's being
3366120	3371480	ransomed is a piece of electronic capital versus human capital does not change the game. I just
3371480	3376880	don't understand why people would insist on coming up with a pat answer for that. I think it's stupid.
3376880	3382920	And while I think that, you know, the response to a ransomware crisis, because it is a crisis,
3382920	3389020	will likely involve legal, I don't necessarily think that legal is the end all be all shop on
3389020	3392980	that. It probably is at most shops. But you know what? The world is bigger than the law.
3393780	3399480	You know, I could tell you all day long about, and all of you could too, about every major
3399480	3404580	insert fortune 500 company here, where they have a crisis management team that has nothing to do
3404580	3409080	with computers. And I guarantee you, the first stop they call when the CEO gets kidnapped by,
3409080	3413040	you know, random activists in South America on a business trip is not the lawyer. They're like
3413040	3418040	the third. So I don't know why it's different because it's a computer and not them. And then
3418040	3422700	after you handle the crisis, however you handle it, maybe it's paying, maybe it's not, maybe it's
3422700	3427620	using shadow currency. I don't know. Right? Then you handle the incident, which is if you have
3427620	3432300	ransomware, you were breached. Don't handle that. That's your incident. The ransomware, that's not
3432300	3441460	an incident. That's a crisis. With the shift of a lot of ransomware actors, for example, Maze,
3441460	3448920	to not only just destroy anymore, but also, you know, steal the data, threaten to, you know,
3448920	3456860	do more stuff with it than just destroy it. Has that changed your way of dealing with ransomware?
3460370	3467610	It has in my advice. So I'll go ahead and speak to that in that I have
3469010	3474810	given a little bit of advice, you know, working with a legal team, of course, if it makes sense
3474810	3481110	for the company and with their backing. So all of that included. So assume that to from the get-go.
3481110	3485870	But, you know, a lot of these groups that do extortion, they're saying that they have something
3485870	3492950	and usually they'll serve up what they have. If there is at all 24 hours
3492950	3499610	that you can buy your IR team to investigate if that claim is true or not,
3499610	3507970	negotiate and buy the team time. If you can figure out if what they say was exposed is true,
3507970	3513910	and if there's any more to that, so if they've gotten more than they say they have,
3513910	3521170	take that time and lean on your investigations team to see that extra piece and see if you can
3521170	3526510	see that creation and that within your system. And that's a very hard thing to do in a short
3526510	3533430	amount of time, especially because it assumes that you have full visibility at every endpoint
3533990	3539270	on the network of your company. And to have that kind of visibility to begin with
3540070	3547470	is incredibly difficult for your team, let alone the outside team. So I would say that
3547470	3552470	the good things that come out can come out of that, not that I'm for negotiating with jurists,
3552470	3560050	so to speak, that are withholding your data and extorting you for money. But, you know,
3560050	3567210	sometimes I've seen outside counsel get a lower price so that if you are going to pay,
3567910	3573810	but time is valuable too, and having that on your side is important.
3577410	3585370	Yeah, I think one of the things that is hugely important, but also at the same time incredibly
3585370	3591970	underrated, is the way that ransomware works. Ransomware is a business. Just like any other
3591970	3598690	business, ransomware at the end of the day is just a business. And I don't recall seeing
3598690	3603310	personally, and I'm sure it's happened, but I don't recall a case personally where the actors
3603310	3609450	have claimed to have something and they truly did not, that they were bluffing. Because if you're
3609450	3613330	running a successful business, you're not going to lie about what you have and everything. Well,
3613330	3617670	unless you're Enron and stuff like that. But for the most part, you're not going to actually
3617670	3624430	not tell the truth about certain things. So if they claim that they exfil data and you as a
3624430	3628450	company have no way to prove that, like number one, you're already in way worse problems than
3628450	3634330	having ransomware. Like you have so many other problems that you need to deal with. But if they
3634330	3638470	claim that they have something and they more than likely do, and then you have to evaluate the cost
3638470	3642310	of all that. Well, what if that data goes public? What if that data goes even more publicly
3642310	3646250	widespread? What if that data is then sold to other places and everything like that? And you
3646250	3652330	have to take all of that into account with the entire process. And we often try to look at it
3652330	3658230	much as a pure like, essentially, is there a correct answer? Or is there not a correct answer?
3658230	3663970	Is it a yes? Or is it a no? And it's never just a yes or no. There are so many factors that you
3663970	3669490	have to take into account all of that through it. And we in the information securities field,
3669490	3674850	we only see a very small portion of all of that. So you have to involve other people,
3674850	3681650	you have to involve other people in the process. But just like it was said, if you have ransomware,
3681650	3688390	I guarantee you have at least five other problems besides just how ransomware got there. You have a
3688390	3693290	lot of problems that need to be addressed. Ransomware is the big public facing thing on
3693290	3698850	all of that. But there are so many problems that need to be addressed internally. And then you as
3699490	3703930	the practitioner have to take into account, all right, here's my little take on the ransomware,
3703930	3707770	here's all that. And then essentially, that decision goes to somebody else. But
3709010	3713250	ransomware, I've never seen anywhere where like, oh, well, we got infected with ransomware. And
3713250	3717050	that's the only problem that they have. There are many, many other gaping holes that they absolutely
3717050	3725170	have. I can guarantee that. And on that note, unless somebody else wanted to say something
3725170	3731770	else on ransomware virus, were you about to speak up? I was just going to answer Rubik's question.
3731770	3736910	Yeah, go for it. If I understand the question correctly, it was in a nutshell does introducing
3736910	3742430	extortion into the ransomware topic change the game? So my answer to that would be no,
3742430	3747710	at least not for me, because I've already given my two cents on how I think that the orchestration of
3747710	3754590	the conversation on ransomware is wrong. In terms of have I seen it change the game?
3755290	3760390	I would say yes. And I would say the way that I think it is changing the game is that
3761650	3765930	counterintelligence is a thing. Counterterrorism is a thing. Black on black kill squads are a thing.
3765970	3770570	And they have always been a thing in blue team. They just haven't been very talked about.
3770950	3776150	Because until words like hacking backs that are being introduced into the info set conversation,
3776150	3781090	those were mostly gray areas to a lot of professionals. My experience is that the
3781710	3785250	reintroduction, because I have seen it before, especially dealing with like adult fields,
3785250	3789410	it's way more common than what I've seen in corporate space. And now I'm seeing it a lot
3789410	3796630	more in the corporate space. I think those areas are getting a lot less gray. And they're showing
3796630	3800810	up on the radar of a lot of risk teams, because they're starting to realize that that's how you
3800810	3808390	fight this. But again, is my prescriptive advice that there should be more of that? No, that
3808390	3815730	doesn't answer the question. Awesome. Thank you. Any other comments on ransomware?
3817510	3825570	I will say that the other thing that I've seen is ransomware used to be kind of a singular
3825570	3830850	program that executes and they've got this programmatic way of doing destruction,
3830850	3839210	is that now when you hear of ransomware, it's a full-on incident. They have a number of backdoors,
3839210	3844690	they have a number of accounts, right? You can't just go and contain it and say,
3845510	3850550	we found ransomware on five machines, we've re-imaged those five machines, we're good.
3850710	3856490	Now they're going to have spent some time in your environment, stolen accounts,
3857050	3862990	have other ways of getting back in. And so you really have to do a full incident response,
3862990	3870450	find out all the things that the attacker got. It's no longer a simple case like we used to deal
3870450	3879190	with. The only other thing that I would add, this is the hill I die on, because I'm one of our,
3880250	3887450	say, Linux enthusiasts, where I work, is most people associate ransomware with Windows.
3888010	3892470	And while that's the case, that doesn't mean you should ignore your Linux environment and
3892470	3900530	never assume. So make sure that you're building that up and you're looking at it, and don't ever
3900530	3909650	assume that it's just, you know, if I have access to your environment and I'm a bad guy,
3909650	3916410	if you give me an easy pivot point, why shouldn't I do good? So kind of have that in the back of
3916410	3924890	your mind. And don't, like, go tunnel vision only on your Windows systems. And don't make it easier.
3927440	3929440	Sorry, Virus, I didn't mean to.
3930800	3935280	I just never heard anyone say that ransomware is a Windows thing. I'm like,
3935280	3941980	there's been ransomware on mobile phone boot roms since, like, 2001. Who are these people?
3941980	3950920	I hear it. I do. I hear it. I hear that this usually only hits Windows and look at the
3950920	3957880	Windows environment and all that. And sometimes that's absolutely true. But you should be scanning
3957880	3962640	for lateral movement to everything. Don't assume anything. What about Max? I thought Max were immune
3962640	3972060	from viruses. Mainframe. Move everything to the mainframe. There's no mainframe viruses.
3972180	3977040	Yeah. So we've already gone over time a little bit, but I think we have about 10 minutes left.
3977060	3981340	And so I wanted to kind of reward the folks who have stuck around through all of this very,
3981340	3987820	very serious conversation on incident response and try to end on a lighter note. And so I was
3987820	3993400	going to ask each of you if you could share one good story about, like, one of the craziest things
3993400	3999240	that you've seen, whether it was on a response or, you know, in some of your own actions,
3999240	4007900	Varys. And I will remind you that this is being recorded. So who wants to go first?
4011640	4018940	Anybody got one queued up? I have one kind of queued up a little bit. So this is actually a
4018940	4024440	combination of two. So one is the worst thing that I've ever seen in an environment.
4024540	4030740	The worst thing that I've ever seen in an environment was there's a firewall company,
4030740	4036860	who is a very, very popular, very, very known firewall company. And for whatever reason,
4036860	4042340	they somehow implemented all of these firewall rules that were okay. They weren't great. They
4042340	4047680	were just okay. And then somebody put in a triple anti-firewall rule. Not a double,
4047680	4052940	an actual triple any. So it was any, any, any. Come to find out from speaking with someone who
4052940	4057480	will rename nameless, but is on this panel, that company actually allows you to have as many as
4057480	4064160	five anys. So not only is there a problem with five anys, like even the option for you to be
4064160	4068540	able to put that many anys in something, like, that doesn't make any sense. Like, what are they
4068540	4073400	possibly thinking? Like, we're just going to make sure no matter what, that you can actually find
4073400	4081800	some way through our firewall. So the hybrid of that is it was with the same client. It was kind
4081800	4086820	of a cool story, because we had a chance to talk about it, I believe, last year in New Orleans,
4086820	4093220	because the actual indictment came down. So they were a part of a breach that was essentially
4093220	4098780	targeting the aerospace industry. And the attackers did all kinds of stuff. So we were able to actually
4098780	4103780	go through essentially, like, their Gen 1 through Gen 5 of their malware, including, like, the last
4103780	4108740	ditch absolute effort where it was incredibly sophisticated, because it was worth burning
4108740	4114120	something that was amazing on it. And when I found out, like, that was just incredible. It was amazing.
4114120	4117840	I actually did a little dance, because like, all right, now we found this. Now we have to find out
4117840	4122400	how we're actually going to get rid of it. But the way that we're able to track back to exactly
4122400	4128040	where it was is, again, because attackers are human beings, and they make mistakes. So for the most
4128040	4133280	part, they bounce their IP through other countries, and everything was just fine.
4133280	4139180	Except we had a total of 11 instances over the course of, I believe, it was a three-month period
4139180	4145240	where they forgot, or they just made the connection before it actually made all the hops.
4145240	4149280	And we had 11 times where we were actually able to pin down pretty much to the exact building
4149280	4155340	in a certain company that rhymes with Rhina of exactly where the attacker was, exactly what
4155340	4160960	they were doing, and exactly what they were after. And that was amazing, because we literally
4160960	4166960	had gigabytes of logs of all this access in, and in, and in, and in. And then we happened to find,
4166960	4171620	like, oh, well, here's this web shell that they had put in. The only person to access it was this
4171620	4176780	person. We know that for sure. And 11 times, they just made a mistake, and we were able to pin down
4176780	4182280	where they were. So it literally takes just one time of messing up, and like, okay, now we can
4182280	4187660	actually build an investigation. And we turned all that over to law enforcement. And unfortunately,
4187660	4191540	it took them about five years to actually go through the process, because they had so much
4191540	4196200	stuff to investigate, primarily for our case. But it was also kind of disheartening to see,
4196200	4200680	like, flash reports get put out that is essentially the summary of our report that we had written,
4200680	4206640	and just has an agency logo at the top of it. Like, it's like, that's my stuff. Like, don't do that.
4211780	4214040	All right. So who can go next?
4215520	4222100	Oh, man. I guess I should follow, just because, Brian, at the time, I had told you, I saw five
4222100	4227820	enemies on a firewall, and I was like, what in the hell? And it was like a horror for me, because I
4227820	4232740	was just like, it was at the top, too. So if you think of how firewalls work, if you've ever
4232740	4238000	done any kind of architecture on them, it's top-down. And it was five, and it's
4239300	4246960	any source, any destination, any application, any user. And I don't know which one I left out.
4249060	4253640	Location and service. There are actually six that can be set, and they set one as
4253640	4256820	deny. And I didn't even know why they set one as deny, because I was just like,
4256820	4260740	it basically just works. That's the answer I got, by the way, was it just works.
4261940	4269160	But my worst story is actually recent. I have a lot of empathy. I see a lot of stuff in
4269160	4275040	ransomware, but one of the coolest things that I've worked a couple of was not too long ago,
4275040	4282880	there was a salt stack vulnerability that came out that allowed for, basically, the long story
4282880	4289040	short is it allowed you to get root to anything that was exposed that was a salt master. So
4290600	4296020	a salt is used, salt stack is used to manage and administer a Linux environment.
4296760	4302100	And it's a really cool thing. I actually really like it for administration purposes.
4302620	4306920	But there was a vulnerability, and if it wasn't patched, you could get root into anything that
4306920	4313200	was internet exposed. And I just happened to see it in a couple of cases. I thought going into this,
4313200	4320820	man, if I had root on something automatically that was internet exposed, what could I do?
4320820	4325440	And I spent a long time looking at this environment, thinking I'm going to find something
4325440	4333900	cool. We got in and they used it to spam mining bots everywhere using the minion service.
4333920	4340920	And that was literally it. And so I want to tell that boring horror story because I live
4340920	4345680	too for the cool, like, when am I going to find an Oday? And I've seen them. Don't get me wrong.
4345680	4351580	I've seen some really cool stuff from APTs and some really cool tactics. I've seen people
4351580	4357900	absolutely stomp out servers to where I have to go into unallocated space and get really creative
4357900	4363900	about carving things out. But this one was one where I was going in going, I'm going to see
4363900	4369120	like something absolutely crazy because you automatically had root access and you could do
4369120	4375080	all of these things. And that was not the case. And I've seen so much of that over the course of
4375080	4381980	doing IR where I'm just like, you know, you have this level and you worked at this level. And I
4381980	4387180	think that's something to keep in mind just when you're doing these things is, you know, I was
4387180	4393080	super happy to go back to the client and say, I have great news. All they did is run up your cloud
4393080	4399700	build. But at the same time, like never assume anything because I was thinking like I'm going
4399700	4408260	to boil the ocean and find something and I did and I didn't. In 2005, I was on a plane coming back from
4408400	4416320	a job and a co-worker of mine is sitting next to me on the plane and we're talking about war stories
4416320	4420300	and he tells me a war story from the week before, which is why he was late to the job that we were
4420300	4425520	on. And I go, what did you do? And he goes, oh, I was doing IR on an old Windows XP network that
4425520	4430200	was locked down by this government thing because it was a defense contractor. I'm like, oh, well,
4430200	4438920	how'd the bug get in? It's like old Java bug, super like 265 day, right? Like old Java on Windows XP.
4439040	4444300	What was the vector? Facebook. Yeah. Ops were on there, you know, running Facebook and click
4444300	4448640	something got infected. Oh, that's interesting. Why did it take you a week to clean up? Well,
4448640	4452440	didn't have a lot of ambient room on the operating system to install tools. Why?
4452440	4455300	Well, it was a drone control station for a Predator 3.
4460000	4469300	Yeah, I had a similar story to Moose about, you know, tracking down a, what we realized was a
4469300	4476680	O-Day for a printer. And, you know, we thought for sure that this was a very advanced threat actor
4477540	4484560	burning an O-Day for this situation and found out it was just to run a ware server
4484560	4491700	off of our printers because they have so much storage. And so that, you know, again,
4491700	4497060	yeah, really excited. All right. Yeah. This is going to be like going to make me famous, right?
4497060	4503980	No, they're just running ware. So, but they somehow managed to get an O-Day and, you know,
4503980	4512900	that was what they spent it on. Cool. Well, I certainly don't have any stories as exciting as
4512900	4520280	those that the rest of you have shared in my somewhat boring career in InfoSec, not getting
4520280	4525480	to go around and do incident response all over the place. But, you know, I just want to take
4525480	4531580	the opportunity to thank all of you for accepting our invitations and coming here and sharing your
4531580	4536700	experience and, you know, helping all the attendees, you know, who are here at the Blue
4536700	4542500	Team Village to learn from you. I've learned a lot from you. I think this is, you know, this is such
4542500	4549380	great, valuable information and getting, you know, a variety of people's perspectives, you know, to
4549380	4554740	help us all, you know, learn new things and different ways to approach problems. It's
4554740	4560940	awesome. I want to thank you so much for participating. And for the attendees who are
4560940	4569580	here in the BTV Discord, we're going to move over to the voice talks chat. Anyone who's available,
4569580	4574200	panelists who are available, pop over there and take some questions like hallway con.
4576240	4579400	So I guess that's a wrap. Thanks, everyone.
4581320	4582100	Thank you.
4582100	4582520	Transcribed by https://otter.ai