[00:01.550 --> 00:07.790] Welcome. Today I'm going to tell you a story about things that I learned about incident response [00:07.790 --> 00:14.890] by being a firefighter. Let's get to it. Here's the agenda, what we're going to cover. [00:15.070 --> 00:19.470] I'm going to start with a little introduction about myself, tell you a little bit about my [00:19.470 --> 00:25.250] background in IR, and I'm going to tell you a little bit about myself as a firefighter. [00:25.350 --> 00:30.570] Then we'll delve into what IR is. For those of you who don't know what IR is, [00:30.730 --> 00:34.230] and then we'll talk a little bit about firefighting, because even though you might [00:34.230 --> 00:39.990] know what IR is, you might not be familiar with firefighting beyond, obviously, putting out the [00:39.990 --> 00:46.570] fire. Then we'll do some comparisons between IR and firefighting. Definitely some similarities [00:47.210 --> 00:52.630] and some differences, of course. Then I'll describe some challenges and opportunities, [00:52.630 --> 00:57.490] the things that I learned from being a firefighter that I then brought back to IR. [00:58.210 --> 01:04.710] I'll summarize and then end with some final thoughts. Let's get going. [01:05.590 --> 01:11.720] All righty. So, for those of you who are not familiar with me, my name is Catherine Allman. [01:12.070 --> 01:18.510] I go by Investigator Chih on the internet, in the Twitter-verse, and I've been with the [01:18.510 --> 01:24.550] University of Buffalo for over 20 years. I'm on staff with B-Sides Rochester and a cyber camp [01:24.550 --> 01:29.190] that we hold. I volunteer with some staff. I've spoken at a bunch of conferences. I have some [01:29.190 --> 01:36.170] certifications and a couple degrees. Even those of you who know me from the IT realm may not be [01:36.170 --> 01:41.790] familiar with this information about me as a firefighter. I've been in the fire service since [01:41.790 --> 01:48.410] about 1997, and since then I've become a state fire instructor with New York State. I've done [01:48.410 --> 01:53.790] some credits, college credits, on firefighting. I was a certified fire investigator at both levels [01:53.790 --> 02:00.030] in New York State, and I was also a member of the J-Fire team. So, the fire thing is something I've [02:00.030 --> 02:08.110] been doing a long time. Just in case you're still not convinced, this is me at the scene of a fire, [02:08.110 --> 02:12.770] and this is part of the story that I'm going to tell you. I'm holding what's called a thermal [02:12.770 --> 02:18.490] camera. When you look into a thermal camera and you point it at something, what it's going to show [02:18.490 --> 02:25.030] is a difference in temperature. So, things that are darker are usually cooler in temperature, [02:25.030 --> 02:30.490] and things that are warmer are going to be are going to show up as lighter in color. [02:31.050 --> 02:35.730] We'll see why that matters later. And just in case you weren't sure, [02:35.730 --> 02:39.310] I'm the one with the camera, and the person to my right is our captain. [02:40.390 --> 02:47.070] So, now let's delve into what incident response is. I like this definition of incident response [02:47.070 --> 02:53.070] because it focuses on the technical components that you're going to need to analyze and contain [02:53.070 --> 02:58.730] an incident. We'll see in a bit why that's a little different from incident handling. [02:58.730 --> 03:02.450] So, you're going to be the boots on the floor person. The person who's actually focused [03:02.450 --> 03:07.570] on dealing with this incident directly. You're going to address network events. [03:07.730 --> 03:12.790] You might deal with them in a reactive or a proactive way, depending on your organization. [03:12.790 --> 03:19.630] And an event can be anything that's going to affect the CIA triangle. Your confidentiality, [03:19.630 --> 03:26.710] integrity, and availability of information. The goals of incident response are going to [03:26.710 --> 03:32.330] be to make sure you know when there is, in fact, an incident or, you know, some kind of [03:32.330 --> 03:37.230] thing going on. You're going to stop that attacker. You're going to minimize the damage [03:37.230 --> 03:42.730] that is caused. And ultimately, you want to prevent it from ever happening again [03:42.730 --> 03:49.370] or something similar from happening. These are your basic phases of incident response. [03:49.370 --> 03:55.070] Preparation, identification, containment, eradication, recovery, and lessons learned. [03:55.310 --> 04:02.030] Let's break these down a little bit. Preparation. So, during the preparation phase, [04:02.030 --> 04:08.430] we're going to document, document, document. You want to document both your network itself. [04:08.450 --> 04:13.250] You might want to document the particular machines in question. One of the most important [04:13.250 --> 04:17.670] things you'll want to document is where your important data is, right? Where are the keys [04:17.670 --> 04:23.290] to the kingdom? That's critical. You might do some tool building. Perhaps to help you [04:23.290 --> 04:28.790] respond better to an incident. Or perhaps to help you know when something is happening that [04:28.790 --> 04:35.290] shouldn't be. Then the next step is going to be awareness of the attacks. And ultimately, [04:35.290 --> 04:41.070] this could be something as simple as an automated ticket system that tells you that there's a [04:41.070 --> 04:47.310] problem. Could be from a phone call. Your containment phase, you want to stop whatever [04:47.310 --> 04:53.570] is happening. So, that might be patching a system, blocking some sort of C2 connection, [04:53.570 --> 04:59.730] maybe pulling the power out of the back of a machine. Depends. Then during the eradication [04:59.730 --> 05:04.110] phase, you're ultimately going to remediate. You're going to remove compromised hosts from [05:04.110 --> 05:09.750] your network. Whatever it is you have to do to get them to get the bad guys gone. [05:10.490 --> 05:15.550] During the recovery phase, you need to restore those business functions. Because, of course, [05:15.550 --> 05:21.070] that is the whole point of incident response initially, is to get up and running again. [05:21.070 --> 05:26.130] Then you want to learn from exactly what happened so that the same thing doesn't happen again. [05:27.650 --> 05:32.790] Here we have incident handling. You'll notice it's a little different. It focuses on the other [05:32.790 --> 05:38.190] things that are important in an incident. The logistics, communication, the coordination, [05:38.190 --> 05:43.950] planning, all of these other functions. And ideally, you want these two things to be separate. [05:44.570 --> 05:50.070] Now, I understand in smaller organizations, you might be the only person doing this job. [05:50.070 --> 05:54.870] You might have to do both. But if you're the person whose boots on the ground and you're [05:54.870 --> 06:02.250] working on containing an incident or eradicating the attacker, you don't also want to have to be [06:02.250 --> 06:07.210] the person who's communicating out every five minutes what's going on. So, if you have somebody [06:07.210 --> 06:14.570] else doing that, it's a whole lot easier to focus on what's at hand. Now let's move to firefighting. [06:14.570 --> 06:20.290] What is firefighting? Well, it's the obvious. We're going to put the wet stuff on the red stuff, [06:20.290 --> 06:26.490] as they say. We're going to prevent the spread and extinguish significant unwanted fires. So, [06:26.490 --> 06:31.870] you know, not your backyard barbecue unless it's out of control. And we'll do that for buildings [06:31.870 --> 06:38.930] and vehicles, woodlands, that sort of thing. Our goals, as you might expect, are to protect the [06:38.930 --> 06:45.290] health and safety of ourselves and of the public. We're also going to protect property and the [06:45.290 --> 06:50.690] environment. Just because we're in the process of putting a fire out doesn't necessarily mean [06:50.690 --> 06:56.690] we want to make things worse, right? So, for example, if we have a situation that might [06:56.690 --> 07:01.390] involve hazardous materials, we need to make sure that we're doing our best to prevent that from [07:01.390 --> 07:08.210] making the situation worse. And we want to minimize, excuse me, the disruption of community activities. [07:08.930 --> 07:15.230] Great example of that would be a car fire or a car accident. You have a lot of people who are, [07:15.230 --> 07:19.170] say, trying to get to work in the morning or trying to get home. And if one of those things occurs, [07:19.170 --> 07:24.890] we need to set up traffic support to help you still get from place to place and minimize any [07:24.890 --> 07:31.450] disruption while we're taking care of the particular incident. We, too, have a preparedness cycle. [07:31.450 --> 07:35.830] It looks kind of like this. We'll do some preparing. Then an emergency will happen. [07:35.830 --> 07:41.550] We'll respond to it. We're going to recover from it. We'll mitigate where we can. And we're [07:41.550 --> 07:47.910] back to preparation. Now, in this preparedness cycle in the fire service, we usually start more [07:47.910 --> 07:55.430] on the mitigation side. And let me show you how and why. So, mitigation for the fire service has [07:55.430 --> 08:00.250] to do with preventing future emergencies or minimizing their effects. So, if you've ever [08:00.250 --> 08:06.610] been to any kind of, say, fire department open house where they're doing fire safety demonstrations, [08:06.610 --> 08:11.830] they're handing out smoke detectors, right? That's a mitigation. We want to see if we can [08:11.830 --> 08:17.370] help people understand how to prevent fires and accidents in the first place. Sprinklers in a [08:17.370 --> 08:23.150] building are another great mitigation method. Preparedness. We need to be prepared to handle [08:23.150 --> 08:27.490] that emergency. So, we're going to make sure our gear is maintained, that we're drilling so we know [08:27.490 --> 08:32.050] how to use the equipment on the truck, that we have pre-plans of the building. Perhaps there's [08:32.050 --> 08:38.370] something unique about accessing certain parts of the building or getting people out if necessary. [08:38.490 --> 08:43.210] And then recovering. When we recover from an emergency, we're going to clean up. We're going [08:43.210 --> 08:48.410] to restore things to their functional purpose again. We're going to repair equipment that [08:48.410 --> 08:55.670] might have been damaged. This is pretty standard for us. So, here's the next part of the story. [08:55.670 --> 09:01.550] You saw me looking at the camera, right? This is the same fire. Two more pictures from this fire. [09:01.910 --> 09:09.590] The one on the left has me, if you probably can't tell because my name's not on my coat, but that is [09:09.590 --> 09:16.470] me. And what I'm attempting to do in that picture is change out an air bottle. The way firefighters [09:16.470 --> 09:22.330] breathe inside a building that is on fire is we wear something called self-contained breathing [09:22.330 --> 09:32.090] apparatus. And that's made out of a harness and a bottle and some other pieces that help redirect [09:32.090 --> 09:39.430] airflow. Needless to say, there are several components. My job on the left-hand side is to [09:39.430 --> 09:46.570] change the air bottle. Your average air bottle is about 30 minutes, 45, which can go very quickly if [09:46.570 --> 09:52.730] you're working hard in a scene. And when a firefighter runs out of air or starts to run out of air, [09:52.730 --> 09:58.350] they come out and they ask somebody to change that bottle. So, that was my job at this fire. I was [09:58.350 --> 10:03.730] changing bottles for the people who needed them. If you look in the picture on the right-hand side, [10:03.730 --> 10:08.890] you'll get at least a small idea of what that bottle and harness setup looks like, because you [10:08.890 --> 10:13.810] can kind of see the top of the bottle. What I want you to focus on in that right-hand picture [10:14.630 --> 10:21.530] is the fact that the firefighter's shoulder has a lot of damage. If you look carefully, [10:21.530 --> 10:27.690] you will see there's actually been flame and heat impingement to that shoulder area. [10:28.350 --> 10:36.430] What I didn't realize when I was changing that bottle is that it was not just his coat that had [10:36.430 --> 10:43.790] been damaged, but in fact the harness that holds the bottle and the air hoses that connect to it. [10:44.470 --> 10:49.810] So, I go to change that bottle, and unfortunately when I put the new bottle back in and connect it [10:49.810 --> 10:57.030] all up, I get hit with a blast of air in the face. Fortunately for me, I was wearing my gear correctly, [10:57.030 --> 11:03.990] there was no damage, and ultimately I instructed the firefighter to take the pack and make sure [11:03.990 --> 11:11.250] that it went out of service. Now, why is this important? Nobody got hurt, it wasn't a really [11:11.250 --> 11:18.490] huge deal, nothing bad ultimately happened. We'll get back to that. So, [11:18.490 --> 11:23.750] pay close attention to that picture, think about that story, and remember here, I was just doing [11:23.750 --> 11:30.410] my job. All right. So, one other piece of terminology in the fire system that's really [11:30.410 --> 11:36.290] important to understand is the incident command system. And it's this idea of an organized response [11:36.290 --> 11:43.590] to a problem. You're going to have a coordinated response with potentially multiple agencies, [11:43.590 --> 11:51.030] you need some way to organize that chaos, right? Such that everybody knows what they're doing and [11:51.030 --> 11:56.430] what their job's supposed to be. If you've been in the military before, you're probably familiar [11:56.430 --> 12:01.230] with chain of command. Obviously, if you're in law enforcement, that's something else you're [12:01.230 --> 12:07.310] familiar with. Not dissimilar idea. This is what forms our effective chain of command. [12:08.790 --> 12:12.850] So, here's what the incident command system looks like. And we're not going to spend time looking [12:12.850 --> 12:17.470] at all of this. But you'll see the incident commanders at the top, typically going to be [12:17.470 --> 12:22.750] one of your chief officers. And at the bottom, you're going to have operations and planning, [12:22.750 --> 12:28.090] logistics, finance. And in between, you'll have some other folks who report directly to the [12:28.090 --> 12:33.290] commander. What's important to note here is that the incident commander doesn't have a ton [12:33.290 --> 12:38.750] of people reporting directly to them. And the reason is, it makes this whole situation [12:39.450 --> 12:46.130] way more manageable. So, let's look at the pieces that are most often in smaller incidents. [12:46.810 --> 12:52.450] So, you have your incident commander. They're going to define what the goals and the objectives [12:52.450 --> 12:56.670] are for that particular incident. Again, usually your chief, or it might be a senior officer if [12:56.670 --> 13:04.230] no chief is available. And then you'll have operations. Operations is often the person who [13:04.230 --> 13:10.850] is in charge of making sure that those goals and objectives get met by figuring out what the [13:10.850 --> 13:15.830] strategy and the tactics are going to be. We sometimes have external operations and internal [13:15.830 --> 13:21.190] operations. If it's a large building, we might have operations on multiple floors. So, we might [13:21.190 --> 13:28.650] have operations for the first floor. We have a high-rise facility and sometimes we need [13:28.650 --> 13:35.550] multiple levels of operations. So, that's not uncommon. The last three here, logistics planning [13:35.550 --> 13:42.110] and admin finance, are more common in very, very large incidents. When you need to have lots of [13:42.110 --> 13:47.010] people and extra supplies and equipment brought in. If there's going to be a long-range plan, [13:47.010 --> 13:52.670] something like a hurricane hits and we have to mobilize troops or demobilize them quickly, [13:52.670 --> 13:57.070] that's going to be on the part of planning. And then if we need some kind of licenses or deal [13:57.070 --> 14:02.450] with compliance issues, get some money together, whatever, that's where admin and finance is going [14:02.450 --> 14:08.790] to come in play. All right. So, let's do a comparison of the two and kind of see, [14:08.790 --> 14:16.770] I think you'll find it interesting to see how they do compare. So, methodology-wise, [14:17.010 --> 14:22.190] I think you've seen they're very, very similar, right? The only thing that was really missing [14:22.770 --> 14:28.270] from the firefighting circle were the lessons learned. And I'll tell you, even though it's not [14:28.270 --> 14:33.930] officially part of the methodology, if you are good at what you do as a firefighter, you absolutely [14:33.930 --> 14:39.110] pay attention to lesson learned. So, I would say that these really, in terms of their functionality, [14:39.110 --> 14:44.970] are really very similar. But let's talk about some misconceptions. [14:45.830 --> 14:50.890] In incident response, there's this idea that every event is an incident. Well, that's certainly not [14:50.890 --> 14:56.850] the case. You can have an event where something is, you know, changed, but it's not necessarily [14:56.850 --> 15:02.410] an incident on the part of the particular organization you're with. And not every [15:02.410 --> 15:08.370] incident is handled the same way, right? I mean, certainly a small incident involving maybe [15:09.010 --> 15:12.630] one social security number is going to be different than an incident involving [15:13.290 --> 15:19.290] a hundred social security numbers or a thousand. Every incident is quickly solved. Well, we know [15:19.290 --> 15:24.870] that's not true. We can have an incident that's really fast, maybe gets resolved in 10 minutes, [15:24.870 --> 15:29.930] and it could take one that takes days, weeks, months. It can be a really long time, depending [15:29.930 --> 15:34.090] on how big the incident is. If law enforcement is involved, it can be even longer than that, [15:34.090 --> 15:39.730] depending on what court cases and all the other details play into it. [15:40.310 --> 15:45.590] This is, I think, one of the most important ones. There's this idea that every person on an incident [15:45.590 --> 15:50.630] response team needs to be a rock star. Well, that's not true. I'm not a rock star. I'll tell [15:50.630 --> 15:56.250] you that right now. And as new people come into this field, we shouldn't expect them to be rock [15:56.250 --> 16:03.590] stars either. We can all learn from each other. Another thing that's a misconception in IR is this [16:03.590 --> 16:11.170] idea that we can accurately attribute whatever's happened to a particular entity. It's hard enough [16:11.170 --> 16:17.430] to, you know, figure out attribution at all, but to do it accurately is even harder. Likewise, [16:17.430 --> 16:22.810] in firefighting, we have some misconceptions. How about this idea that firefighters are always paid? [16:23.050 --> 16:27.510] How many of you know that there are tons of firefighters around this country [16:28.250 --> 16:34.270] that aren't paid? I'm one of them. I've been a volunteer firefighter for many years. [16:34.270 --> 16:41.450] I've never, ever earned a dime from firefighting. And yet, I still love it. It's a fantastic hobby [16:41.450 --> 16:46.810] and one that I wouldn't trade for anything. There's also, from television especially, this [16:46.810 --> 16:52.450] idea that firefighters are big tough dudes. Well, I know you can only see my floating head, but guess [16:52.450 --> 16:59.690] what? I'm not a big tough dude. Never was, never will be. We also don't fight fires the same way [16:59.690 --> 17:05.170] every time. Again, yes, we put the wet stuff on the red stuff, as one of my fire instructors [17:05.730 --> 17:13.430] has said, but we don't do it the same way. Maybe we need to do a search and rescue first. Maybe [17:13.430 --> 17:18.010] the building has partially underground. There are all sorts of challenges with the situation [17:18.010 --> 17:25.050] that don't allow us to do the same exact thing every time. Certainly, there's this idea that, [17:25.050 --> 17:29.770] you know, fires get extinguished quickly, especially, again, on television shows, [17:29.770 --> 17:36.630] but it can take days. And in large, really large situations, it can take longer than that. [17:36.770 --> 17:42.570] Just depends. Maybe, maybe all you need is a water can and it's a little bit of burning mulch. [17:42.570 --> 17:48.890] That's extinguished quickly, but not all of them. And definitely not a full-time job, [17:48.890 --> 17:54.070] because that's not my full-time job. My full-time job is incident response. [17:54.130 --> 18:01.030] But we'll see, they're pretty similar things in some ways. First, we'll talk about the differences. [18:02.070 --> 18:07.590] So, in terms of incident response, it rarely involves life safety, except in the universe [18:07.590 --> 18:13.090] of healthcare. If you work in healthcare, you absolutely can be the difference between life and [18:13.090 --> 18:18.630] death in some of the incident response situations you might wind up in. Patient records are critical. [18:19.210 --> 18:22.990] Ideally, there are things in place to prevent that from being, you know, [18:22.990 --> 18:29.590] catastrophic. But in the rest of the world, despite what everyone would have you believe, [18:29.590 --> 18:35.230] life safety isn't usually the issue. That's important to remember. [18:36.090 --> 18:41.310] Certainly, IR teams typically get paid, and as I already mentioned, firefighters, not so much. [18:41.790 --> 18:49.010] We use computers and software for our day-to-day job. And while firefighters also sometimes use [18:49.010 --> 18:55.170] computers in certain capacity, they use mostly water, foam, chemicals, hoses, and show up in [18:55.170 --> 19:00.090] special vehicles, like fire trucks. Most of us doing IR, drive whatever we want, doesn't really [19:00.090 --> 19:04.810] matter. Obviously, if you're in an organization like law enforcement, well, you might have different [19:04.810 --> 19:10.430] experience. And we wear special gear as firefighters. I'm not going to show up at a fire [19:10.430 --> 19:16.790] in just a t-shirt and shorts. I'm certainly going to put on my gear, right? So those are some [19:16.790 --> 19:24.210] differences. Now, here's where they are similar. In both cases, we're going to focus on that [19:24.210 --> 19:30.130] immediate issue first and find that cause later. We need to put that fire out. We need to stop that [19:30.130 --> 19:37.110] attacker. We'll deal with attribution and the cause later. We're also going to use triage to [19:37.110 --> 19:42.070] determine the best course of action. If you're not familiar with triage, it's this idea that [19:42.070 --> 19:47.470] when you have more than one thing to consider, you're going to decide which of those things is [19:47.470 --> 19:54.510] most critical. So what most people are familiar with in terms of triage is a situation where [19:54.510 --> 20:01.430] there's been a motor vehicle accident and you have multiple people who are injured. You might [20:01.430 --> 20:09.130] have somebody who just has a bump on the head. You might have somebody who maybe is beyond saving. [20:09.130 --> 20:14.210] And you might have somebody who's serious, but with the right interventions could easily be [20:14.210 --> 20:20.290] saved. And looking at all those details, you have to decide which one to treat first. [20:20.290 --> 20:25.710] Fire situations are exactly the same thing. You're going to want to take a look at the facts [20:25.710 --> 20:31.970] and make some decisions about what to do first. Ideally, you've pre-planned that, [20:31.970 --> 20:36.390] but we'll come back to that. They're cyclic in nature, both of them. We've seen that already, [20:36.390 --> 20:42.150] right? And they both require some pretty interesting thinking outside the box. [20:42.330 --> 20:46.930] In incident response, sometimes we use tools that we might use for other things [20:47.610 --> 20:55.590] to try to get information for the response. And in firefighting, we do the same kinds of things [20:55.590 --> 21:00.270] in the sense that, for example, I have a friend who's a firefighter in my hometown [21:01.430 --> 21:08.610] who had a patient on a multi-story building who was very large and they didn't have equipment [21:08.610 --> 21:12.970] that would easily facilitate getting him out of the building where he was. [21:12.970 --> 21:20.090] So he went to the local hardware store, got some equipment, and built something to get this [21:20.090 --> 21:25.190] individual out of where he was stuck. Thought outside the box, didn't just use the equipment [21:25.190 --> 21:32.250] he had, worked with other things that he was able to get a hold of. In both cases, we often [21:32.250 --> 21:36.830] bring in outside entities. Firefighters rely heavily on mutual aid, bringing in other fire [21:36.830 --> 21:42.630] departments as necessary or perhaps specialty teams. We do the same thing in incident response. [21:42.630 --> 21:47.990] While I do incident response for the university, we might need another team if something were [21:47.990 --> 21:52.970] significantly serious enough to come in and help us. Maybe if it's a large enough incident, [21:52.970 --> 21:59.010] maybe if it's a certain kind of incident. And sometimes in both cases, there are inside teams. [21:59.030 --> 22:04.450] So again, I'm inside at the university and there are companies that have inside [22:04.450 --> 22:08.870] firefighting teams because they're a manufacturing organization that's large enough. [22:10.010 --> 22:14.750] So now let's talk about the things I've learned from being a firefighter that I think are relevant [22:14.750 --> 22:21.550] to IR specifically. I'll start with this first challenge, tunnel vision. This is an idea where [22:21.550 --> 22:27.670] we focus exclusively on one particular thing instead of seeing that big picture, right? And [22:27.670 --> 22:34.890] we make bad decisions when we do that. Think back to that image, the two images of me changing out [22:34.890 --> 22:42.590] that air bottle for the firefighter and his burn coat. This is an example of tunnel vision. I'm [22:42.590 --> 22:49.550] focused on doing my job. That's all I'm doing. And frankly, I could have gotten really hurt here [22:49.550 --> 22:57.450] because of tunnel vision. I wasn't taking a step back in my head and looking at the bigger picture. [22:57.450 --> 23:03.510] I was just focused on the task at hand. Here's another example from firefighting. [23:03.510 --> 23:10.150] Look carefully at those cones. Pull up to the fire, get off the fire truck, and this is what you see. [23:10.650 --> 23:16.950] What are those cones protecting? At first glance, you see a fire hose. No big deal. So maybe you're [23:16.950 --> 23:22.210] going to use that hose. You're going to help with the fire out. Maybe you're going to bring a piece [23:22.210 --> 23:28.330] of gear to the other person if you look carefully who's standing at the door. If you look more [23:28.330 --> 23:35.550] carefully, you may or may not see there is actually a live power line that those cones [23:36.310 --> 23:42.130] are actually trying to protect you from. And if you go rushing into the scene because you're so [23:42.130 --> 23:48.690] focused on putting the fire out or getting the gear that somebody needs, you could get really hurt. [23:48.930 --> 23:54.850] We have the same kind of thing in IR. We need to consider whether or not a scene is safe. [23:54.850 --> 24:00.130] In many cases, sure, it might be, but it depends. It depends on the situation. [24:00.330 --> 24:07.270] The picture that I'm showing you below is of a server room that has asbestos. If there's an issue [24:07.270 --> 24:13.230] in a server room with asbestos, you don't want to go running in there. Consider if somebody tells [24:13.230 --> 24:19.190] you that you have a malware situation and you get focused on that, but it turns out it's just [24:19.370 --> 24:25.990] a misconfiguration. Or maybe you do blind hardware acquisition, so you go running in and you grab [24:25.990 --> 24:34.410] drives, but that machine you find out later was powered on and was running encryption and now you [24:34.410 --> 24:38.710] don't have the keys anymore because all you've done is grabbed hardware. You haven't thought [24:38.710 --> 24:45.970] about it. Maybe you were told that a situation involved ransomware and you're worried now and [24:45.970 --> 24:50.190] you're focused on how you're going to handle the ransomware, but it turns out it's just a phishing [24:50.190 --> 24:58.990] email. So hand-in-hand with tunnel vision is reactionary behavior. Not only are we focused on [24:58.990 --> 25:06.090] one thing, but we react to it without thinking ahead, right? We're allowing those outside forces [25:06.630 --> 25:12.510] to make that decision instead of relying on the data we have at hand and thinking about that [25:12.510 --> 25:21.390] bigger picture. The fire service, it can lead to really deadly situations. So the photograph you [25:21.390 --> 25:27.510] see here is from a horrible fire that happened not all that long ago in Worcester, Massachusetts. [25:28.210 --> 25:34.380] It was a cold storage fire in an old meat packing plant that was built in 1906. [25:35.290 --> 25:41.670] People hadn't been in that building in a very, very long time. It had been vacant. No pre-plans [25:41.670 --> 25:49.450] had been made. It turns out it had a maze of meat lockers in it. When this fire broke out, there was [25:49.450 --> 25:55.670] supposedly somebody trapped inside. So they sent in two firefighters. The two firefighters went in [25:55.670 --> 26:02.670] and attempted to find the individual they thought was trapped. They got hopelessly lost and radioed [26:02.670 --> 26:11.550] for help. So the chief sent in two more firefighters to help them. Unfortunately, those two firefighters [26:11.550 --> 26:15.830] also got hopelessly lost and could not get out of the building, nor could they find the two [26:15.830 --> 26:23.210] original firefighters that had been sent in. So the fire chief sends in two more firefighters. [26:23.910 --> 26:32.530] Guess what? They too get hopelessly lost. And ultimately, the fire chief says, no more. [26:32.670 --> 26:40.310] I'm not sending any more in. Now, at first he was harshly criticized by his team for this, [26:40.310 --> 26:46.650] because, of course, their own folks were inside that building. But the reality is, [26:46.650 --> 26:51.230] any further reactionary behavior to try to save these individuals would just have led to more [26:51.230 --> 26:58.970] death. These folks died because pre-plans weren't made, and they didn't know what was in there, [26:58.970 --> 27:02.470] and it would have been a reaction to just keep sending firefighters in. [27:03.090 --> 27:11.110] We see the same thing within IR, although usually not as deadly, right? If we're doing forensics, [27:11.110 --> 27:17.390] we need clear, explicit goals. If the reactionary behavior is just figure out the bad stuff, [27:17.390 --> 27:22.690] that's not helpful. You need to know more specifically what it is they want you looking for. [27:23.710 --> 27:28.210] Another example is just pulling a network connection. What if you need more information [27:28.210 --> 27:31.970] about the attacker because of what they're doing on other systems? [27:32.270 --> 27:37.450] Or what happens if you just turn off a machine? We've already talked about the situation where, [27:37.450 --> 27:44.370] you know, you have a key in memory and you turn the machine off and it's gone. Suspending all [27:44.370 --> 27:48.830] accounts is another example where it could be you have one account that's compromised, [27:48.830 --> 27:51.650] now you suspend them all, and that causes more headaches. [27:53.710 --> 28:01.310] Another challenge is freelancing. Now, in our field in IT, a freelancer we often think of as [28:01.310 --> 28:08.610] somebody who's more of, perhaps, somebody who's just not working directly for the company, right? [28:08.610 --> 28:13.630] They're a contractor. But in the fire service, this notion of freelancing is this idea that [28:13.630 --> 28:18.150] somebody is not following chain of command, they go off on their own and kind of do what they want [28:18.150 --> 28:24.170] in an attempt to get the job done, but they're not following what's been laid out for them. [28:24.730 --> 28:29.910] And that can lead to dangerous and reckless situations and bad feelings and all kinds [28:29.910 --> 28:37.110] of problems. In the fire department, we see this typically where we have what's called the [28:37.110 --> 28:41.690] crossing of the streams. And for those of you who don't know, the picture in the bottom right [28:41.690 --> 28:46.010] is Ghostbusters, and they always talk about never cross the streams. [28:46.610 --> 28:50.850] Well, in firefighting, the idea is if I'm streaming water through the front of a building, [28:50.850 --> 28:56.210] and then a team comes to the rear of a building, and that building is a small house, [28:56.210 --> 29:02.330] now I'm fighting the team that's putting the water in, in both directions. And that could [29:02.330 --> 29:08.330] be really dangerous. Another way we see this freelancing in the fire department that can [29:08.330 --> 29:13.790] be really dangerous is something where doors are opened or windows are smashed [29:14.330 --> 29:20.450] without coordinating it with other people. We've seen situations where a window could get smashed [29:20.450 --> 29:23.990] and a firefighter could be standing right on the other side of it because they don't realize [29:23.990 --> 29:29.710] somebody's there. It can also ultimately feed the fire because where there's more oxygen, [29:29.710 --> 29:34.010] that fire is going to go. So if it hasn't been planned out and it's not coordinated, [29:34.010 --> 29:36.930] it can actually make things significantly worse. [29:38.330 --> 29:44.390] In incident response, we see the same kinds of things. A team goes in to do a response, [29:44.390 --> 29:48.450] and you get somebody who thinks they're just going to collect everything. Well, there could [29:48.450 --> 29:54.410] be legal ramifications for that. What happens if they don't collect what's absolutely required? [29:54.430 --> 29:59.770] Maybe they overlook something. Maybe there's a duplication of data collection which wastes time. [29:59.770 --> 30:05.750] Or maybe data gets altered or misrepresented because people are freelancing and not everybody's [30:05.750 --> 30:15.910] not following the same set of rules or the same guidance. Patience. This is a fantastic opportunity. [30:17.010 --> 30:23.290] Ultimately, you want to take that moment to determine what the best course of action is. [30:23.290 --> 30:29.290] As one of my fire instructors would say, slow is fast, right? You want to be deliberate about what [30:29.290 --> 30:37.710] you do because ultimately it will take less time if you take a moment, think things through, [30:37.710 --> 30:46.210] and then proceed. So in terms of firefighting, you may remember the same house here. It's the [30:46.210 --> 30:52.090] same story on the left is what we pulled up to. You'll notice you can see flame in the picture [30:52.090 --> 30:58.150] on the right hand side, the sort of orangey bit that's on the right part of that underneath the [30:58.150 --> 31:04.190] eaves next to the downspout. And you'll see a little bit of a glow toward the front of that [31:04.190 --> 31:10.130] third window on the right. But we don't see flame blowing through the roof. Now what I was seeing in [31:10.130 --> 31:15.490] that thermal camera image was that all of the fire, all of the significant flame and heat [31:16.050 --> 31:23.970] for this particular fire, that was up in that roof area. So what would have been ideal is for [31:23.970 --> 31:33.510] somebody to go in on the inside, break through from the second floor up into that attic area, [31:33.510 --> 31:41.210] and ultimately put the fire out. Unfortunately, we had people who focused solely on where they [31:41.210 --> 31:48.210] saw flame. And as a result, we see what we see on the right hand side where flame [31:48.770 --> 31:55.910] blows through the roof and there's a lot more of it. And it took us a lot longer to get this [31:55.910 --> 32:01.210] fire out. Which isn't to say that what everybody did was wrong, but taking a beat and thinking [32:01.210 --> 32:08.490] this through might have saved a lot of time with this fire. Okay, patience in terms of incident [32:08.490 --> 32:16.170] response. For those of you not familiar with Chegg, Chegg is a company that rents textbooks. [32:16.170 --> 32:23.350] Now what do you think the odds are of an entity that rents textbooks to students in higher ed, [32:23.350 --> 32:30.550] of them using, oh I don't know, same credentials at the university that they use at Chegg or vice [32:30.550 --> 32:39.030] versa? Yeah, pretty darn good. So Chegg had a massive breach that involved millions of accounts [32:39.850 --> 32:46.830] and because they provide services to multiple universities, as you might expect, [32:46.830 --> 32:54.090] many institutions wound up with this exact problem. Tons of compromised accounts. But [32:54.510 --> 33:03.450] is it really the university's problem in the sense that is it their breach? No. [33:03.450 --> 33:10.290] What it is, is password reuse. Chegg had the breach. Yes, the universities and institutions [33:10.290 --> 33:14.830] have to do something about making sure those passwords get reset, right? That's important. [33:15.150 --> 33:19.470] But it's not the same as having those passwords having been breached on the [33:19.470 --> 33:25.990] systems that are actually at the university. That is an incident. That's pretty serious. [33:26.450 --> 33:32.510] So before you panic when you hear about a huge breach, even if it could impact your institution, [33:32.510 --> 33:39.750] think for a moment. Take that beat about exactly what the implications are. What is the real risk [33:39.750 --> 33:46.470] here? And also consider that time is important, but as we saw with the picture of the asbestos [33:46.470 --> 33:52.190] room, life safety is more important. So taking that beat can also be life-saving in incident [33:52.190 --> 33:59.710] response depending on what you're walking into. All right, accountability. It's this idea, [33:59.710 --> 34:04.430] at least in the fire service, of knowing where all your people are. You want to know if they're [34:04.430 --> 34:09.670] in the trucks. You want to know who came with one truck. You want to know who's gone inside a [34:09.670 --> 34:15.290] building. You want to know who's getting rehabbed. But really, in incident response, we do the same [34:15.290 --> 34:20.730] kinds of things. We want to make sure that we prevent hazards and duplication of efforts. We [34:20.730 --> 34:24.170] don't want to step on each other's toes or have any sort of direct interference. [34:26.070 --> 34:33.290] So on the left are accountability tags that we use in firefighting. And for example, we'll use [34:33.290 --> 34:39.210] one of those tags we'll place on the truck so that everybody knows what truck you came in on. [34:39.330 --> 34:47.650] That way, the officer on that truck knows where their people are at all time. They might put [34:47.650 --> 34:51.790] another, if they go inside the building, they'll use a second tag and put it with the accountability [34:51.790 --> 34:56.450] person at the door so that they know where they are inside the building. [34:56.690 --> 35:03.010] For incident response, we can do the same kinds of things. Perhaps wipe card access. If you're [35:03.010 --> 35:10.390] going inside a facility where you work, we'll show you what room you're in. And maybe you need [35:10.390 --> 35:15.410] something more detailed, like an incident access log, depending on the situation. So we can do [35:15.410 --> 35:23.630] accountability with IR, and it's important. Okay, pre-planning. Pre-planning is absolutely [35:23.630 --> 35:28.910] critical. You want to know your environment. You want to know through your own documentation [35:28.910 --> 35:36.330] and planning, third-party documentation, and anything else that you can think of before it [35:36.330 --> 35:43.930] becomes a problem. And in the fire service, we're going to do that with what are called pre-plans. [35:43.930 --> 35:48.910] And in incident response, we're going to do it with things like tabletop and pre-mortem exercises, [35:49.510 --> 35:55.950] so that we have this idea of what we might run into before we run into it, and prevent those [35:55.950 --> 36:05.270] gotchas. So this is the kind of thing we do in firefighting. On the left is a map of a building, [36:05.270 --> 36:11.310] basically. I know it's a little difficult to see, but the idea is that it maps out for us where the [36:11.310 --> 36:18.830] hydrants are, where the electrical shutoff is, and it might tell us if there's any hazardous [36:18.830 --> 36:23.790] materials in the area. The thing on the right might tell us a little more about the building [36:23.790 --> 36:32.250] construction. This is more of a general pre-plan that's going to tell us about other hazards, [36:32.250 --> 36:37.370] other information, but it's the same concept, right? It tells us what we need to know about [36:37.370 --> 36:42.890] the facility we're going into to keep us safe, and those that we're trying to rescue, [36:43.350 --> 36:47.410] and perhaps the general public, depending on the particular situation. [36:47.790 --> 36:54.230] We can do the same thing in incident response. We can do tabletop exercises. If you're not already [36:54.230 --> 37:00.470] familiar with backdoors and breaches, you should be. The Black Hills folks, John Strand and his [37:00.470 --> 37:06.630] crew, have built this amazing card game, which uses a 20-sided die and this set of cards, [37:06.630 --> 37:15.530] and it can help you build an entire event from the ground up, and then you can do the investigation [37:15.530 --> 37:21.410] using those cards. It's really interesting, and I highly recommend it if you haven't already tried [37:21.410 --> 37:30.410] them. But it can walk you through scenarios, everything from something happening in a natural [37:30.410 --> 37:37.050] disaster type emergency to, you know, the intern who's done the wrong kind of thing and made an [37:37.050 --> 37:42.890] accidental mess. Obviously, documentation we've talked about, right? Document your servers, [37:42.890 --> 37:47.150] document your workstations, document where your data is and what your network is like, [37:47.150 --> 37:53.130] and the more information you have, the better off you'll be. And hopefully it goes without saying, [37:53.130 --> 37:57.550] to a certain degree, that this would also include your incident response plan and whatever team [37:57.550 --> 38:06.990] you have. So to summarize, we want you to start thinking like a firefighter. I think that applying [38:06.990 --> 38:12.710] these firefighter ideas to incident response has really made me a better incident responder, [38:12.710 --> 38:19.550] ultimately. You want to avoid tunnel vision. You want to do what's called a 360 of the problem, [38:19.550 --> 38:24.950] which in firefighting might literally mean a 360 around the building, which is what we try to do, [38:24.950 --> 38:29.330] or at least somebody does when we get there, whether it might be the operations person, [38:29.330 --> 38:34.450] it might be the incident commander. But at a minimum, we always pull past the building that's [38:34.450 --> 38:40.090] on fire, or potentially on fire, so that we can see at least three sides of the building. [38:40.430 --> 38:45.910] That gives us that bigger scope. We want to make sure we're acting, we're deliberate, [38:45.910 --> 38:52.930] we don't just react. We don't want to be misled by emotion. Follow whatever plan that you and [38:52.930 --> 38:57.790] your colleagues have come up with. Don't just freelance. It can be, you know, physically [38:57.790 --> 39:04.810] dangerous in certain circumstances, and it can just be plain frustrating in others. [39:04.950 --> 39:10.350] Have patience and take that beat. You never know what you might have missed if you didn't. [39:10.350 --> 39:18.210] Those 30 or 60 seconds could save tons of time. And even in the fire service, [39:18.210 --> 39:23.110] taking those few seconds ultimately could save a life, even if you think going faster [39:23.110 --> 39:31.170] might have done it. Location. Make sure you know where your people are. Document, document, [39:31.170 --> 39:40.630] document your location. Both of your equipment, your people, where your data is, and pre-plans [39:40.630 --> 39:45.310] they rock, right? Less chaos, way easier to respond if you already have a plan. [39:45.310 --> 39:50.430] And consider something like the incident command system, so that you're organizing your response. [39:50.970 --> 39:55.770] The thing about the incident command system that's so cool is that ultimately it is fantastic [39:55.770 --> 40:03.370] for scaling to larger incidents. As I mentioned, you can have only one person in charge of just [40:03.510 --> 40:08.690] a few people. And if you scale that out, it works really well in larger situations. [40:08.790 --> 40:14.350] So bringing in outside entities, this is so much easier because everybody's on the same page. [40:16.050 --> 40:24.650] So, some final thoughts. I know that many of you have never thought about incident response [40:24.650 --> 40:33.330] in this way. And that's okay. That's the idea. Think outside that box. Try some things you've [40:33.330 --> 40:45.290] never tried before. You may find that ultimately it makes your job maybe more challenging in the [40:45.290 --> 40:53.810] And the reality is great things never come from comfort zones. So, do it. Push that envelope. [40:54.670 --> 41:00.490] Hopefully you will find these tactics helpful. I want to thank you for coming to hear me speak. [41:00.490 --> 41:05.590] And I want to thank Wall of Sheep so much for having me. And I hope you have an absolutely [41:05.590 --> 41:09.330] wonderful DEF CON and a wonderful rest of your day.