1 00:00:00,150 --> 00:00:00,983 - All right, hi everyone. 2 00:00:00,983 --> 00:00:02,280 My name's Safa Shahwan Edwards, 3 00:00:02,280 --> 00:00:03,240 and I'm a Deputy Director 4 00:00:03,240 --> 00:00:04,740 with the Cyber Statecraft Initiative 5 00:00:04,740 --> 00:00:05,790 at the Atlantic Council. 6 00:00:05,790 --> 00:00:08,640 We're a foreign policy think tank based in Washington, D.C., 7 00:00:08,640 --> 00:00:09,510 and my program works 8 00:00:09,510 --> 00:00:12,435 at the Nexus of Geopolitics and Cybersecurity. 9 00:00:12,435 --> 00:00:14,550 This morning's discussion is imagining 10 00:00:14,550 --> 00:00:17,190 a cyber policy crisis, where we're gonna examine 11 00:00:17,190 --> 00:00:18,930 how storytelling, simulations, 12 00:00:18,930 --> 00:00:21,840 and war games can help individuals better understand 13 00:00:21,840 --> 00:00:24,450 cybersecurity concepts and challenges. 14 00:00:24,450 --> 00:00:25,980 A few announcements before we begin, 15 00:00:25,980 --> 00:00:28,500 first off, cell phones as a courtesy to our speakers, 16 00:00:28,500 --> 00:00:31,310 please ensure that your cell phones are on silent. 17 00:00:31,310 --> 00:00:33,210 The other part of this is, 18 00:00:33,210 --> 00:00:34,980 we have some microphones scattered around the room. 19 00:00:34,980 --> 00:00:36,940 As you have questions, please feel free to relocate 20 00:00:36,940 --> 00:00:39,390 and direct your questions into the microphones. 21 00:00:39,390 --> 00:00:40,980 As you can hear, the acoustics in this room's 22 00:00:40,980 --> 00:00:43,580 a little funky, so microphones are our friends here. 23 00:00:45,647 --> 00:00:48,450 Last, so this is good, photo policy. 24 00:00:48,450 --> 00:00:49,736 This prohibits anyone from taking pictures 25 00:00:49,736 --> 00:00:52,740 without the explicit permission of anyone in frame. 26 00:00:52,740 --> 00:00:54,363 And please keep your mask on at all times 27 00:00:54,363 --> 00:00:56,460 with the exception of us speaking, eating, 28 00:00:56,460 --> 00:00:57,573 drinking, et cetera. 29 00:00:59,040 --> 00:01:00,390 I think that might be it, 30 00:01:00,390 --> 00:01:01,920 so I think we're gonna kick off here. 31 00:01:01,920 --> 00:01:03,524 So with the discussion today, 32 00:01:03,524 --> 00:01:05,883 what we're hoping to do is examine how do stories, 33 00:01:05,883 --> 00:01:08,940 simulations, crisis simulations, 34 00:01:08,940 --> 00:01:12,210 and war games aid in learning and training. 35 00:01:12,210 --> 00:01:13,260 A little bit about myself, 36 00:01:13,260 --> 00:01:15,030 I've organized simulations for college students. 37 00:01:15,030 --> 00:01:17,010 This is our Cyber 9/12 Strategy Challenge 38 00:01:17,010 --> 00:01:18,973 where we develop these. 39 00:01:18,973 --> 00:01:20,940 (moderator speaking faintly) 40 00:01:20,940 --> 00:01:22,593 Yeah, I'll get closer, thank you. 41 00:01:23,872 --> 00:01:25,470 Should I adjust this? 42 00:01:25,470 --> 00:01:27,630 Okay, thank you. 43 00:01:27,630 --> 00:01:29,850 So I've organized simulations for college students. 44 00:01:29,850 --> 00:01:31,600 This is part of our Cyber 9/12 Strategy Challenge, 45 00:01:31,600 --> 00:01:34,238 where we develop a three part cyber crisis scenario 46 00:01:34,238 --> 00:01:36,966 that examines a technical vulnerability 47 00:01:36,966 --> 00:01:39,469 or an incident that has reaching implications 48 00:01:39,469 --> 00:01:43,662 on national security, public policy, et cetera. 49 00:01:43,662 --> 00:01:46,620 And I am joined today by my colleagues here. 50 00:01:46,620 --> 00:01:47,730 We got Winnona DeSombre. 51 00:01:47,730 --> 00:01:50,490 She is a fellow with the Cyber Statecraft Initiative, 52 00:01:50,490 --> 00:01:52,680 but also with the Belfer Center at Harvard. 53 00:01:52,680 --> 00:01:54,360 We are also joined by Sarah Powazek. 54 00:01:54,360 --> 00:01:57,000 She is a consultant at CrowdStrike. 55 00:01:57,000 --> 00:01:59,010 So before we kick-off with the discussion 56 00:01:59,010 --> 00:01:59,940 and some of the questions, 57 00:01:59,940 --> 00:02:02,010 the Cyber Statecraft team at the Atlantic Council, 58 00:02:02,010 --> 00:02:04,560 we always like to ask speakers some fun questions 59 00:02:04,560 --> 00:02:06,420 to get to know them a little bit better. 60 00:02:06,420 --> 00:02:09,030 So the first one is what is your favorite beverage? 61 00:02:09,030 --> 00:02:11,115 This can be alcoholic or otherwise. 62 00:02:11,115 --> 00:02:14,130 Your favorite book, favorite musical artist, 63 00:02:14,130 --> 00:02:16,830 and if you could have one superpower, what would it be? 64 00:02:16,830 --> 00:02:17,970 Sarah is taking notes, 65 00:02:17,970 --> 00:02:19,264 'cause these are really good questions. 66 00:02:19,264 --> 00:02:20,097 - It's a lot of questions. 67 00:02:20,097 --> 00:02:20,992 (panelists laughing) 68 00:02:20,992 --> 00:02:21,900 - Winnona, you know us better, 69 00:02:21,900 --> 00:02:23,310 so I'll get over to you. 70 00:02:23,310 --> 00:02:24,330 - Yeah, all right. 71 00:02:24,330 --> 00:02:25,163 Hi, everyone. 72 00:02:25,163 --> 00:02:26,100 So I'm Winnona, I'm a fellow. 73 00:02:26,100 --> 00:02:28,080 I also used to work in Cyber Threat Intelligence. 74 00:02:28,080 --> 00:02:30,515 So while I was a fellow doing think tank 75 00:02:30,515 --> 00:02:32,220 and fun cyber war gaming stuff, 76 00:02:32,220 --> 00:02:35,373 I also did in-depth detection response research. 77 00:02:36,580 --> 00:02:39,390 My favorite drink, my husband and I got really 78 00:02:39,390 --> 00:02:41,970 into making elaborate cocktails during the pandemic, 79 00:02:41,970 --> 00:02:44,405 as I'm sure many of us may have also done. 80 00:02:44,405 --> 00:02:45,388 (Winnona laughing) 81 00:02:45,388 --> 00:02:47,640 But we made this cocktail called the five eyes, 82 00:02:47,640 --> 00:02:49,230 which is a muddled Kiwi, 83 00:02:49,230 --> 00:02:51,420 apologies to our New Zealand friends, 84 00:02:51,420 --> 00:02:53,820 Bundaberg ginger beer from Australia, 85 00:02:53,820 --> 00:02:56,640 a shot of scotch, a shot of bourbon, 86 00:02:56,640 --> 00:02:58,320 and a shot of Canadian whiskey. 87 00:02:58,320 --> 00:02:59,700 So highly recommend it. 88 00:02:59,700 --> 00:03:00,553 It's very good. 89 00:03:00,553 --> 00:03:01,980 (Winnona laughing) 90 00:03:01,980 --> 00:03:02,813 Superpower? 91 00:03:02,813 --> 00:03:04,230 I don't know, invisibility? 92 00:03:04,230 --> 00:03:05,670 I feel like any person 93 00:03:05,670 --> 00:03:07,096 in the intel space might wanna say that. 94 00:03:07,096 --> 00:03:09,891 (panelists and audience laughing) 95 00:03:09,891 --> 00:03:13,020 - Okay, I think I got all four of the questions. 96 00:03:13,020 --> 00:03:13,853 So hi, folks. 97 00:03:13,853 --> 00:03:14,686 My name's Sarah Powazek. 98 00:03:14,686 --> 00:03:15,519 I work for CrowdStrike. 99 00:03:15,519 --> 00:03:16,968 I run our tabletop exercises. 100 00:03:16,968 --> 00:03:19,400 We have a bunch of clients, major enterprises, 101 00:03:19,400 --> 00:03:20,820 small businesses. 102 00:03:20,820 --> 00:03:23,040 So I feel like we run the gambit of the sorts 103 00:03:23,040 --> 00:03:25,830 of folks that we walk through these exercises. 104 00:03:25,830 --> 00:03:27,510 My favorite drink, I'm a big tea drinker. 105 00:03:27,510 --> 00:03:28,560 I don't know if I'm the only one. 106 00:03:28,560 --> 00:03:30,936 Any coffee, tea, anybody? 107 00:03:30,936 --> 00:03:31,769 Tea? 108 00:03:31,769 --> 00:03:33,156 A couple, yeah. 109 00:03:33,156 --> 00:03:33,989 (Sarah laughing) 110 00:03:33,989 --> 00:03:35,430 So I'm a big fan of loose leaf tea. 111 00:03:35,430 --> 00:03:37,170 I have a bunch of weird flavors like cupcake 112 00:03:37,170 --> 00:03:39,540 and banana bread, and sort of substitute it 113 00:03:39,540 --> 00:03:41,890 for any sort of dessert that you feel like, so. 114 00:03:43,530 --> 00:03:46,440 Superpower, definitely talking to animals. 115 00:03:46,440 --> 00:03:48,367 I have two really cute cats at home. 116 00:03:48,367 --> 00:03:50,460 Musical artist... 117 00:03:50,460 --> 00:03:52,380 I don't know if you guys have heard of Still Woozy, 118 00:03:52,380 --> 00:03:54,360 sort of like a funky electronic. 119 00:03:54,360 --> 00:03:55,193 Yeah, I didn't think so. 120 00:03:55,193 --> 00:03:56,726 (panelists and audience laughing) 121 00:03:56,726 --> 00:03:58,076 But yeah, happy to be here. 122 00:03:59,070 --> 00:04:01,260 - Awesome, thank you for that. 123 00:04:01,260 --> 00:04:03,187 So to kick off this conversation, 124 00:04:03,187 --> 00:04:05,670 can you tell me about a time that a simulation 125 00:04:05,670 --> 00:04:09,240 or a tabletop that you put together helped either you 126 00:04:09,240 --> 00:04:11,293 or a partner or a client better understand 127 00:04:11,293 --> 00:04:13,953 a cybersecurity challenge or concept? 128 00:04:15,180 --> 00:04:16,133 - I can go first, yeah. 129 00:04:16,133 --> 00:04:19,500 I mean, so I probably can't share too much, 130 00:04:19,500 --> 00:04:21,240 but the way that we do tabletop exercises 131 00:04:21,240 --> 00:04:24,150 is that we actually find a viable attack path 132 00:04:24,150 --> 00:04:24,983 for our client. 133 00:04:24,983 --> 00:04:26,550 And so we'll either do a technical tabletop 134 00:04:26,550 --> 00:04:28,650 or a management or executive tabletop, 135 00:04:28,650 --> 00:04:30,705 but we'll spend a good amount of time talking to someone 136 00:04:30,705 --> 00:04:33,510 on their security team. 137 00:04:33,510 --> 00:04:35,004 You know, what's your MFA coverage? 138 00:04:35,004 --> 00:04:36,807 What does your network look like? 139 00:04:36,807 --> 00:04:37,800 And we actually try 140 00:04:37,800 --> 00:04:42,800 and find a current exploitable vulnerability or attack path. 141 00:04:42,990 --> 00:04:44,250 Like for example, we've been doing a lot 142 00:04:44,250 --> 00:04:45,371 of ESXi ransomware lately. 143 00:04:45,371 --> 00:04:48,990 'Cause that's very popular that that could happen 144 00:04:48,990 --> 00:04:51,750 on their network, so that we can come up with questions 145 00:04:51,750 --> 00:04:54,570 that they would actually need to answer during an attack. 146 00:04:54,570 --> 00:04:56,670 And so that's been pretty successful, I think. 147 00:04:56,670 --> 00:04:58,728 We'll walk a technical team through, 148 00:04:58,728 --> 00:05:00,900 we'll have them look at machine names for example, 149 00:05:00,900 --> 00:05:01,890 and they say, "Oh my God! 150 00:05:01,890 --> 00:05:03,900 We have 20 service accounts. 151 00:05:03,900 --> 00:05:05,280 That's not good." 152 00:05:05,280 --> 00:05:08,730 So that sort of realism in the exercise, 153 00:05:08,730 --> 00:05:10,620 I think, really helps folks take away, 154 00:05:10,620 --> 00:05:13,050 not just what sort of processes they need to improve, 155 00:05:13,050 --> 00:05:15,120 but what sorts of gaps they have in their maturity 156 00:05:15,120 --> 00:05:18,360 that they're gonna need to work on in the future. 157 00:05:18,360 --> 00:05:20,250 - So instead of answering this question first, 158 00:05:20,250 --> 00:05:22,110 I actually wanna make a gauge of the room. 159 00:05:22,110 --> 00:05:27,110 How many people first off know what a tabletop exercise is? 160 00:05:27,120 --> 00:05:27,953 Excellent. 161 00:05:27,953 --> 00:05:30,810 How many people have participated in a tabletop exercise? 162 00:05:30,810 --> 00:05:31,643 Great. 163 00:05:31,643 --> 00:05:32,940 - Great. 164 00:05:32,940 --> 00:05:35,850 - Raise your hand if the tabletop exercise was primarily 165 00:05:35,850 --> 00:05:39,360 geared towards higher level decision makers? 166 00:05:39,360 --> 00:05:40,440 Okay. 167 00:05:40,440 --> 00:05:42,450 And then raise your hand if... 168 00:05:42,450 --> 00:05:43,380 So it's about half the room 169 00:05:43,380 --> 00:05:44,599 for the people on the livestream. 170 00:05:44,599 --> 00:05:45,432 (Winnona laughing) 171 00:05:45,432 --> 00:05:48,150 Raise your hand if the tabletop was designed more 172 00:05:48,150 --> 00:05:49,713 for technical practitioners. 173 00:05:50,580 --> 00:05:52,350 Okay, so the other half. 174 00:05:52,350 --> 00:05:55,740 So at least for me, I've done tabletop exercises 175 00:05:55,740 --> 00:05:58,530 that kind of run that entire gamut, right? 176 00:05:58,530 --> 00:06:00,240 So you have the more technical side 177 00:06:00,240 --> 00:06:01,710 where you're actually going through a simulation, 178 00:06:01,710 --> 00:06:03,210 maybe a red team exercise. 179 00:06:03,210 --> 00:06:06,750 You also have the more policy oriented stuff. 180 00:06:06,750 --> 00:06:10,047 I've done a decent amount of work for the Cyber 9/12, 181 00:06:10,047 --> 00:06:12,450 building that kind of policy, or process, 182 00:06:12,450 --> 00:06:13,980 or people muscle within people 183 00:06:13,980 --> 00:06:15,480 who want to get into cyber policy, 184 00:06:15,480 --> 00:06:18,150 who may not have briefed high level principles 185 00:06:18,150 --> 00:06:19,050 and things like that. 186 00:06:19,050 --> 00:06:21,630 And then there's also the tabletop exercises 187 00:06:21,630 --> 00:06:23,130 that are helpful when doing 188 00:06:23,130 --> 00:06:24,390 more international conversations. 189 00:06:24,390 --> 00:06:26,894 So I've done a track two dialogue 190 00:06:26,894 --> 00:06:28,440 through the Harvard Belfer Center 191 00:06:28,440 --> 00:06:31,770 with a Chinese PLA think tank, which was a wild time. 192 00:06:31,770 --> 00:06:32,850 And so going through 193 00:06:32,850 --> 00:06:34,950 the American delegation crisis simulation 194 00:06:34,950 --> 00:06:36,900 and then looking at the Chinese counterpart 195 00:06:36,900 --> 00:06:39,480 is also an interesting set of ways to determine 196 00:06:39,480 --> 00:06:41,100 how that process is going to work out 197 00:06:41,100 --> 00:06:42,930 when stuff hits the fan. 198 00:06:42,930 --> 00:06:45,360 And so, (laughs) at least for me, 199 00:06:45,360 --> 00:06:47,580 the thing that has been particularly interesting 200 00:06:47,580 --> 00:06:51,240 and I honestly would love to hear more from our audience, 201 00:06:51,240 --> 00:06:53,703 as well as not just having all of us rattling on. 202 00:06:54,720 --> 00:06:57,090 But frankly, going through 203 00:06:57,090 --> 00:06:59,970 and learning about how the other side is thinking 204 00:06:59,970 --> 00:07:02,250 when you're especially doing cross team exercises, 205 00:07:02,250 --> 00:07:03,630 I've found to be the most useful, 206 00:07:03,630 --> 00:07:06,187 as well as exercising the aspects of your job 207 00:07:06,187 --> 00:07:07,680 that may come up in a crisis 208 00:07:07,680 --> 00:07:08,760 that you don't do in your day to day. 209 00:07:08,760 --> 00:07:11,319 So for me, at least from a technical perspective, 210 00:07:11,319 --> 00:07:14,250 Cyber 9/12 way back in the day, like maybe what? 211 00:07:14,250 --> 00:07:15,083 Five, six years ago. 212 00:07:15,083 --> 00:07:15,916 (Winnona laughing) 213 00:07:15,916 --> 00:07:18,750 When I first participated and then also became a judge, 214 00:07:18,750 --> 00:07:21,420 you see a lot of... 215 00:07:21,420 --> 00:07:24,060 There's a wide variety where you see technical students 216 00:07:24,060 --> 00:07:25,380 who are participating in this challenge, 217 00:07:25,380 --> 00:07:27,150 learning how to do bottom line up front, 218 00:07:27,150 --> 00:07:27,983 briefing a principle, 219 00:07:27,983 --> 00:07:30,240 making sure that they get the main points across. 220 00:07:30,240 --> 00:07:31,560 You see West Point Cadets, 221 00:07:31,560 --> 00:07:33,660 Naval Academy Cadets who are used to this sort of briefing, 222 00:07:33,660 --> 00:07:35,220 but may not have the technical expertise, 223 00:07:35,220 --> 00:07:36,600 so doing that sort of in-depth reading 224 00:07:36,600 --> 00:07:38,010 and in terms of the scenario. 225 00:07:38,010 --> 00:07:39,630 And so those cross team, 226 00:07:39,630 --> 00:07:42,780 as well as cross discipline perspectives, I think, 227 00:07:42,780 --> 00:07:46,173 is really what makes the tabletop really, really impactful. 228 00:07:47,124 --> 00:07:48,090 - Thanks for that, Winnona. 229 00:07:48,090 --> 00:07:49,620 I think you raise a really excellent point. 230 00:07:49,620 --> 00:07:51,960 It's like one thing I'd like to highlight is so, 231 00:07:51,960 --> 00:07:56,160 while we created Cyber 9/12 initially to serve more like... 232 00:07:56,160 --> 00:07:58,830 We saw lots of hackathons and capture the fly competitions 233 00:07:58,830 --> 00:08:00,150 and that can add a lot of value 234 00:08:00,150 --> 00:08:03,090 to help students develop their technical understanding. 235 00:08:03,090 --> 00:08:05,400 On the other hand, those sorts of events focus 236 00:08:05,400 --> 00:08:07,830 around the how as opposed to the why. 237 00:08:07,830 --> 00:08:08,850 So there was a real gap there 238 00:08:08,850 --> 00:08:10,920 where we wanted to create an opportunity for students 239 00:08:10,920 --> 00:08:13,320 to focus on the why behind these cyber incidents 240 00:08:13,320 --> 00:08:16,202 and understand why do adversaries engage in certain ways? 241 00:08:16,202 --> 00:08:19,650 What are the priorities of different actors in this space? 242 00:08:19,650 --> 00:08:22,170 I think one thing I'd like to dig in on is like, 243 00:08:22,170 --> 00:08:26,001 how do we identify how we wanna challenge folks? 244 00:08:26,001 --> 00:08:28,440 I know one thing that we've thought about 245 00:08:28,440 --> 00:08:31,320 with Cyber 9/12 in our scenarios is we want students 246 00:08:31,320 --> 00:08:33,750 to dig into attribution. 247 00:08:33,750 --> 00:08:35,187 Ideally, Winnona's cracking up 248 00:08:35,187 --> 00:08:37,521 because attribution takes time. 249 00:08:37,521 --> 00:08:39,390 And she made a really good meme about this, 250 00:08:39,390 --> 00:08:40,950 or I think you retweeted a meme about this, 251 00:08:40,950 --> 00:08:42,699 where one student was trying to explain, 252 00:08:42,699 --> 00:08:44,880 I don't have an answer for you right now 253 00:08:44,880 --> 00:08:46,500 because we're still working through this. 254 00:08:46,500 --> 00:08:47,696 These things take time. 255 00:08:47,696 --> 00:08:50,220 Another part of the scenario that we focused 256 00:08:50,220 --> 00:08:53,130 on has been the vulnerabilities equities process. 257 00:08:53,130 --> 00:08:56,067 Yes, you have a vulnerability, but does the owner 258 00:08:56,067 --> 00:08:59,100 of that proprietary software actually know that it exists? 259 00:08:59,100 --> 00:09:00,810 Are they engaging with the security researcher? 260 00:09:00,810 --> 00:09:02,760 What does that process look like? 261 00:09:02,760 --> 00:09:05,273 Does it have a national security implication? 262 00:09:05,273 --> 00:09:10,001 Or when intelligence operations go awry, 263 00:09:10,001 --> 00:09:12,750 how do we ensure that nation states engage 264 00:09:12,750 --> 00:09:14,898 in certain activities to collect intelligence? 265 00:09:14,898 --> 00:09:16,860 Sometimes this has implications 266 00:09:16,860 --> 00:09:20,516 on broader cybersecurity policy challenges, 267 00:09:20,516 --> 00:09:22,830 but not sure if you guys wanna chime in on any of those; 268 00:09:22,830 --> 00:09:25,830 if you've ever identified a specific thing 269 00:09:25,830 --> 00:09:28,740 that you wanted to challenge a client, or a partner, 270 00:09:28,740 --> 00:09:29,627 or yourself with. 271 00:09:29,627 --> 00:09:31,593 - Yeah, I love this question. 272 00:09:31,593 --> 00:09:34,079 How do we make, is this on still? 273 00:09:34,079 --> 00:09:34,912 Okay, great. 274 00:09:34,912 --> 00:09:38,250 How do we make tabletop exercises the most uncomfortable 275 00:09:38,250 --> 00:09:39,630 for people in the room? 276 00:09:39,630 --> 00:09:40,470 How do we scare them? 277 00:09:40,470 --> 00:09:42,540 I think that I try and do that. 278 00:09:42,540 --> 00:09:45,899 'Cause I think the more realistic an exercise, 279 00:09:45,899 --> 00:09:50,167 the more that we can get an actual threat actor behind it, 280 00:09:50,167 --> 00:09:51,690 the more they get out of it. 281 00:09:51,690 --> 00:09:52,890 So I love ratcheting up the pressure. 282 00:09:52,890 --> 00:09:53,905 I think one way we do that, 283 00:09:53,905 --> 00:09:55,315 I like doing this with ransomware 284 00:09:55,315 --> 00:09:57,360 'cause a lot of clients want to see some sort 285 00:09:57,360 --> 00:10:00,420 of ransomware exercise, see if they're prepared for it. 286 00:10:00,420 --> 00:10:01,890 We try and pick a ransom amount 287 00:10:01,890 --> 00:10:05,550 that's like a decent chunk of their yearly revenue. 288 00:10:05,550 --> 00:10:07,500 We actually try and find the right number 289 00:10:07,500 --> 00:10:11,160 that will scare them enough, that it's a lot of money 290 00:10:11,160 --> 00:10:13,080 and their insurance might not cover it. 291 00:10:13,080 --> 00:10:16,955 But it's just enough that we think that they can afford it. 292 00:10:16,955 --> 00:10:21,810 And we also try and figure out what their crown jewels are. 293 00:10:21,810 --> 00:10:24,360 So what sort of data, if it was stolen, 294 00:10:24,360 --> 00:10:28,350 would really hurt the company or the customers, 295 00:10:28,350 --> 00:10:30,627 or would raise pretty significant legal 296 00:10:30,627 --> 00:10:32,700 and regulatory questions. 297 00:10:32,700 --> 00:10:34,037 And then when we combine all of that 298 00:10:34,037 --> 00:10:35,480 into a sort of powder keg, 299 00:10:35,480 --> 00:10:38,670 we get something that doesn't shut them down, right? 300 00:10:38,670 --> 00:10:42,090 If it's too scary, they will say in this scenario, 301 00:10:42,090 --> 00:10:45,213 the company would fold, and we cannot do anything about it. 302 00:10:45,213 --> 00:10:47,667 So we wanna keep it within their reach, 303 00:10:47,667 --> 00:10:51,570 but we definitely wanna make it stressful enough 304 00:10:51,570 --> 00:10:54,780 that it really tests their processes. 305 00:10:54,780 --> 00:10:57,161 So especially with ransomware, 306 00:10:57,161 --> 00:11:00,000 we like to ask a lot of the hard questions 307 00:11:00,000 --> 00:11:02,190 that people might not realize that they need. 308 00:11:02,190 --> 00:11:03,990 You know, what's your insurance coverage? 309 00:11:03,990 --> 00:11:05,940 How early do you need to bring in your insurer 310 00:11:05,940 --> 00:11:08,070 to make sure that they will actually cover a cyber event? 311 00:11:08,070 --> 00:11:09,690 A lot of folks at the end are like, 312 00:11:09,690 --> 00:11:12,626 and then we would ask insurance for $10 million. 313 00:11:12,626 --> 00:11:13,650 (panelists and audience laughing) 314 00:11:13,650 --> 00:11:16,352 No, yeah, see people are shaking their heads. 315 00:11:16,352 --> 00:11:20,611 But that nitty gritty stuff is not self-explanatory. 316 00:11:20,611 --> 00:11:22,650 And it's really important to make sure 317 00:11:22,650 --> 00:11:24,720 that people have those nitty gritty processes down. 318 00:11:24,720 --> 00:11:26,280 Do you have external legal counsil? 319 00:11:26,280 --> 00:11:29,370 Does your external legal counsil have any cyber expertise? 320 00:11:29,370 --> 00:11:32,250 Would they actually understand what's going on right now 321 00:11:32,250 --> 00:11:33,810 and be able to advise you about 322 00:11:33,810 --> 00:11:35,250 what sort of statement you should put out, 323 00:11:35,250 --> 00:11:37,711 and whether we should use the word breach, 324 00:11:37,711 --> 00:11:42,711 or ransomware attack, or disruption, service outage? 325 00:11:42,891 --> 00:11:45,150 Those sorts of things are really, really important. 326 00:11:45,150 --> 00:11:47,441 So to answer your question, 327 00:11:47,441 --> 00:11:49,560 trying to find the right ransom amount, 328 00:11:49,560 --> 00:11:52,770 poking at specific processes, especially with ransomware, 329 00:11:52,770 --> 00:11:55,020 that might not be a self-explanatory 330 00:11:55,020 --> 00:11:58,170 and making sure that the pressure is high enough 331 00:11:58,170 --> 00:12:00,030 that the exercise is useful to them, 332 00:12:00,030 --> 00:12:03,450 but not so high that it makes them disengage. 333 00:12:03,450 --> 00:12:05,970 - Yeah, and I mean, even on your point, Sarah, 334 00:12:05,970 --> 00:12:08,760 I think that one of the things that's been most beneficial, 335 00:12:08,760 --> 00:12:10,800 at least for me when I run some of these games, 336 00:12:10,800 --> 00:12:15,090 is to be, you want to make sure that they're learning, 337 00:12:15,090 --> 00:12:16,500 but also to be a sounding board, 338 00:12:16,500 --> 00:12:19,117 such that when they're doing the after action, they go, 339 00:12:19,117 --> 00:12:21,240 "Oh crap, we don't actually have a relationship 340 00:12:21,240 --> 00:12:22,073 with our FBI." 341 00:12:22,073 --> 00:12:23,043 (Winnona laughing) 342 00:12:23,043 --> 00:12:24,037 (indistinct) and you're like, 343 00:12:24,037 --> 00:12:27,300 "Well, here's that local office in your area. 344 00:12:27,300 --> 00:12:28,470 Here's their contact." 345 00:12:28,470 --> 00:12:30,600 And making sure that it's not just 346 00:12:30,600 --> 00:12:32,040 about the technical aspect 347 00:12:32,040 --> 00:12:33,420 and running through the internal processes, 348 00:12:33,420 --> 00:12:35,970 but making sure that you have the external as well. 349 00:12:38,040 --> 00:12:40,459 - I'd like to open this up to our guests here. 350 00:12:40,459 --> 00:12:42,420 If you have something you'd like to add, 351 00:12:42,420 --> 00:12:44,310 if you've worked in a war game with a partner 352 00:12:44,310 --> 00:12:48,000 or with your organization, and how you tried to kind of... 353 00:12:48,000 --> 00:12:50,010 I think we've talked about poking holes in processes, 354 00:12:50,010 --> 00:12:51,426 trying to find those gaps. 355 00:12:51,426 --> 00:12:52,770 Please feel free to chime in. 356 00:12:52,770 --> 00:12:53,940 I see some heads nodding, 357 00:12:53,940 --> 00:12:56,185 and I don't wanna call anybody out individually, 358 00:12:56,185 --> 00:12:57,960 but would love to hear your thoughts as well. 359 00:12:57,960 --> 00:12:59,820 Just because you do wanna make, this is a round table. 360 00:12:59,820 --> 00:13:01,143 We want folks to chime in. 361 00:13:03,371 --> 00:13:04,507 Yeah, please. 362 00:13:04,507 --> 00:13:06,360 - Yeah, you guys can pass the mics too. 363 00:13:06,360 --> 00:13:08,431 - Yeah, just be gentle with them. 364 00:13:08,431 --> 00:13:10,680 - [Audience Member] Testing. 365 00:13:10,680 --> 00:13:14,400 In your guys' experience, what is the right way 366 00:13:14,400 --> 00:13:16,710 for organizations to test their engagement 367 00:13:16,710 --> 00:13:17,760 with law enforcement? 368 00:13:19,800 --> 00:13:20,910 - You were just talking about FBI. 369 00:13:20,910 --> 00:13:21,852 You wanna take it? 370 00:13:21,852 --> 00:13:22,685 (panelists laughing) 371 00:13:22,685 --> 00:13:23,850 - I mean, at least for me, 372 00:13:23,850 --> 00:13:26,910 we've never run anything jointly with the FBI. 373 00:13:26,910 --> 00:13:30,953 I think making sure that, first off, 374 00:13:30,953 --> 00:13:33,750 I might not actually be the best person to answer 375 00:13:33,750 --> 00:13:34,583 because yeah. 376 00:13:34,583 --> 00:13:35,903 - Okay, I can talk, yeah. 377 00:13:35,903 --> 00:13:37,710 I mean, I think the number one thing 378 00:13:37,710 --> 00:13:39,480 is knowing your local field agents. 379 00:13:39,480 --> 00:13:41,151 Do you who to call? 380 00:13:41,151 --> 00:13:42,990 A lot of people don't and when you're in the middle 381 00:13:42,990 --> 00:13:44,130 of a stressful situation, 382 00:13:44,130 --> 00:13:46,620 trying to figure out that information is very difficult. 383 00:13:46,620 --> 00:13:47,940 So I think to the extent 384 00:13:47,940 --> 00:13:49,170 that you can establish a relationship 385 00:13:49,170 --> 00:13:50,940 with your local field office, 386 00:13:50,940 --> 00:13:52,620 that you least have one person in the company 387 00:13:52,620 --> 00:13:54,686 that knows one person there that can call, 388 00:13:54,686 --> 00:13:56,455 hopefully multiple. 389 00:13:56,455 --> 00:13:58,620 But I think the other thing that folks don't think about 390 00:13:58,620 --> 00:14:00,450 with law enforcement is like, 391 00:14:00,450 --> 00:14:01,920 when do you want to engage them? 392 00:14:01,920 --> 00:14:03,330 Do you wanna engage them at the beginning, 393 00:14:03,330 --> 00:14:05,283 before you really know what's going on? 394 00:14:05,283 --> 00:14:07,860 That might be a waste of their time. 395 00:14:07,860 --> 00:14:10,830 Do you wanna engage them and tell them that you're thinking 396 00:14:10,830 --> 00:14:12,240 about paying a ransom? 397 00:14:12,240 --> 00:14:13,727 I'm not sure. 398 00:14:13,727 --> 00:14:17,550 I think it's, companies don't really think about... 399 00:14:17,550 --> 00:14:19,177 Some people that we talk to are like, 400 00:14:19,177 --> 00:14:23,640 "Absolutely, as soon as we see any kind of ransomware, 401 00:14:23,640 --> 00:14:25,410 we're talking to the FBI." 402 00:14:25,410 --> 00:14:26,243 And then we ask, 403 00:14:26,243 --> 00:14:28,590 how do you think the FBI will help you? 404 00:14:28,590 --> 00:14:30,180 'Cause I'm a big fan of the FBI, 405 00:14:30,180 --> 00:14:35,180 but their priority sometimes is these long term attacks 406 00:14:35,820 --> 00:14:37,170 against threat actors, right? 407 00:14:37,170 --> 00:14:40,557 They're not necessarily there to be the fire department 408 00:14:40,557 --> 00:14:42,930 for any individual organization. 409 00:14:42,930 --> 00:14:45,360 So I think setting those sorts of expectations 410 00:14:45,360 --> 00:14:47,040 for when you contact law enforcement, 411 00:14:47,040 --> 00:14:49,590 are you expecting them to come and fix things? 412 00:14:49,590 --> 00:14:50,580 Some people are. 413 00:14:50,580 --> 00:14:52,445 And I think it's important to communicate to them, 414 00:14:52,445 --> 00:14:54,690 the engagement with law enforcement is gonna be 415 00:14:54,690 --> 00:14:56,730 about sharing information. 416 00:14:56,730 --> 00:14:58,650 You may not get anything back. 417 00:14:58,650 --> 00:15:01,413 You may not get anything back for a while. 418 00:15:01,413 --> 00:15:03,837 And that's important that they have other folks 419 00:15:03,837 --> 00:15:05,589 that they're prepared to rely on and that they don't think 420 00:15:05,589 --> 00:15:08,790 that law enforcement is gonna come in and fix everything. 421 00:15:08,790 --> 00:15:11,190 'Cause they just don't have the capacity, and I also think 422 00:15:11,190 --> 00:15:12,840 that it's just a slightly different priority 423 00:15:12,840 --> 00:15:14,283 than the organizations. 424 00:15:16,440 --> 00:15:17,747 - Please. 425 00:15:17,747 --> 00:15:19,770 - [Audience Member] Well, my question, 426 00:15:19,770 --> 00:15:22,500 I work in financial services, and I've actually ran 427 00:15:22,500 --> 00:15:25,080 and contributed to some of these attack scenarios 428 00:15:25,080 --> 00:15:28,680 under more of the BCP DR testing. 429 00:15:28,680 --> 00:15:32,853 So I think my two questions would be concentration risk. 430 00:15:32,853 --> 00:15:36,240 We talk about that a lot in financial services. 431 00:15:36,240 --> 00:15:38,850 A lot of time, marketing people give up really, 432 00:15:38,850 --> 00:15:40,470 really good information in OSN. 433 00:15:40,470 --> 00:15:42,510 Oh, we have this strategic partnership 434 00:15:42,510 --> 00:15:43,920 with X, Y, and Z vendor. 435 00:15:43,920 --> 00:15:46,500 Me on the attacking side is like, yeah, it's a good place 436 00:15:46,500 --> 00:15:47,760 to start to go after them, 437 00:15:47,760 --> 00:15:50,910 because that's gonna be my pivot into the organization. 438 00:15:50,910 --> 00:15:55,200 The other aspect is more of what your thoughts 439 00:15:55,200 --> 00:15:58,260 and feelings on bringing that concentration risk 440 00:15:58,260 --> 00:15:59,469 into your scenarios. 441 00:15:59,469 --> 00:16:02,460 The second side of it is threat intelligence. 442 00:16:02,460 --> 00:16:03,360 Who is attacking you? 443 00:16:03,360 --> 00:16:04,980 How are they attacking you? 444 00:16:04,980 --> 00:16:07,740 And how do you play that into your scenario, 445 00:16:07,740 --> 00:16:09,090 so that you know the (indistinct)? 446 00:16:09,090 --> 00:16:13,419 It's basically doing a threat model on your own scenario 447 00:16:13,419 --> 00:16:17,340 and what your thoughts and feelings would be on that. 448 00:16:17,340 --> 00:16:19,068 And then the follow on is, 449 00:16:19,068 --> 00:16:22,320 given what we're talking about here, 450 00:16:22,320 --> 00:16:24,990 why aren't we having a CTF in relation to it 451 00:16:24,990 --> 00:16:26,220 in a future DEF CON? 452 00:16:26,220 --> 00:16:29,310 'Cause I think that's a perfect outplay. 453 00:16:29,310 --> 00:16:31,440 I think it would be wonderful to have that. 454 00:16:31,440 --> 00:16:32,790 So your thoughts, your feelings? 455 00:16:32,790 --> 00:16:35,301 - So I can answer the third and fourth question. 456 00:16:35,301 --> 00:16:38,787 And so the fourth, I'll do it backwards. 457 00:16:38,787 --> 00:16:41,700 We're not having a CTF for DEF CON policy, 458 00:16:41,700 --> 00:16:43,948 because this is the first time we've expanded so rapidly 459 00:16:43,948 --> 00:16:45,480 as DEF CON policy. 460 00:16:45,480 --> 00:16:46,860 So we'll take that feedback, 461 00:16:46,860 --> 00:16:49,750 and we'll see if we can do that next year. 462 00:16:49,750 --> 00:16:52,800 As for the threat intelligence aspect, I mean, 463 00:16:52,800 --> 00:16:54,210 I think it depends on, A, 464 00:16:54,210 --> 00:16:56,392 your relationship with your end customer. 465 00:16:56,392 --> 00:17:01,392 How much are you tailoring a war game or tabletop exercise, 466 00:17:01,680 --> 00:17:04,170 DTX, whatever you wanna call it, to their needs? 467 00:17:04,170 --> 00:17:06,960 Are they specifically asking for a nation state threat 468 00:17:06,960 --> 00:17:08,790 that is going to target them, right? 469 00:17:08,790 --> 00:17:10,800 And then, how good is your relationship 470 00:17:10,800 --> 00:17:13,590 as that war game planner with someone who understands 471 00:17:13,590 --> 00:17:15,150 that type of threat actor? 472 00:17:15,150 --> 00:17:18,510 And finally, why do they want this in the first place? 473 00:17:18,510 --> 00:17:20,568 Is it like a bragging right? 474 00:17:20,568 --> 00:17:23,610 Can we successfully defend against AVT attacks 475 00:17:23,610 --> 00:17:26,485 or is it like a, we wanna be able to map 476 00:17:26,485 --> 00:17:29,310 how well we actually apply our defenses 477 00:17:29,310 --> 00:17:30,690 to the minor attack framework? 478 00:17:30,690 --> 00:17:33,000 And if that's the case, then that might be a little bit 479 00:17:33,000 --> 00:17:35,310 of an easier scenario, right? 480 00:17:35,310 --> 00:17:39,391 But then, once you've figured out which minor attack 481 00:17:39,391 --> 00:17:43,920 killed chain whatever model you wanna go for, 482 00:17:43,920 --> 00:17:46,350 what threat actor actually exists out there 483 00:17:46,350 --> 00:17:47,820 that is going to map very neatly? 484 00:17:47,820 --> 00:17:50,010 And it's never gonna be one to one. 485 00:17:50,010 --> 00:17:52,290 And so setting those expectations with the client, 486 00:17:52,290 --> 00:17:53,750 making sure that they understand 487 00:17:53,750 --> 00:17:55,830 if this is going to be this way, 488 00:17:55,830 --> 00:17:59,433 it might not hit every minor attack boss that they want. 489 00:18:00,630 --> 00:18:02,280 And then also making sure 490 00:18:02,280 --> 00:18:04,341 that they then have the internal processes 491 00:18:04,341 --> 00:18:07,233 to do that type of defense against an APT. 492 00:18:08,340 --> 00:18:09,690 Do they have their own 493 00:18:09,690 --> 00:18:11,520 internal threat intelligence capability? 494 00:18:11,520 --> 00:18:12,450 What is that like? 495 00:18:12,450 --> 00:18:15,630 How does that internal team interact 496 00:18:15,630 --> 00:18:17,310 with the wider detection and response board? 497 00:18:17,310 --> 00:18:20,190 And that is a really good muscle to test in my opinion, 498 00:18:20,190 --> 00:18:23,340 when doing those sorts of APT level attacks. 499 00:18:23,340 --> 00:18:24,472 - [Audience Member] Thank you. 500 00:18:24,472 --> 00:18:26,850 - I echo what Winnona's saying about that. 501 00:18:26,850 --> 00:18:28,866 We try and pick an actual threat actor and say, 502 00:18:28,866 --> 00:18:31,590 these are the sort of behaviors they exhibit. 503 00:18:31,590 --> 00:18:33,090 We wanna make sure that we're picking the right one 504 00:18:33,090 --> 00:18:33,923 for the client. 505 00:18:33,923 --> 00:18:36,600 I don't want small businesses worried about espionage. 506 00:18:36,600 --> 00:18:39,360 That is not what they should be concerned about. 507 00:18:39,360 --> 00:18:41,304 And it's tough 'cause there are all these tools out there 508 00:18:41,304 --> 00:18:43,500 and they all seem really useful, 509 00:18:43,500 --> 00:18:45,240 but they're not all useful for every company. 510 00:18:45,240 --> 00:18:47,490 And so it's interesting that we'll come into someone 511 00:18:47,490 --> 00:18:50,430 in their environment and they have all these fancy tools 512 00:18:50,430 --> 00:18:52,170 about threat intelligence, which is great. 513 00:18:52,170 --> 00:18:55,158 But they don't have MFA coverage. 514 00:18:55,158 --> 00:18:58,110 It's really interesting the way that folks prioritize, 515 00:18:58,110 --> 00:18:59,550 and I'm not blaming them whatsoever. 516 00:18:59,550 --> 00:19:01,860 But we wanna make sure that we're picking a threat actor 517 00:19:01,860 --> 00:19:04,020 that's actually going to attack them, so they can prepare 518 00:19:04,020 --> 00:19:06,168 for something that is actually going to happen; 519 00:19:06,168 --> 00:19:09,900 that we can pick common entry points, 520 00:19:09,900 --> 00:19:11,100 like fishing gets used a lot. 521 00:19:11,100 --> 00:19:13,590 Obviously, it's very boring, but it happens. 522 00:19:13,590 --> 00:19:15,090 Are they ready for a fishing attack? 523 00:19:15,090 --> 00:19:16,710 Do they even know what that's gonna look like? 524 00:19:16,710 --> 00:19:18,610 Would they even be able to catch that? 525 00:19:19,560 --> 00:19:23,370 Another thing that we do is we pick, like I said, 526 00:19:23,370 --> 00:19:24,240 specific threat actors. 527 00:19:24,240 --> 00:19:25,448 So we use Conti, for example. 528 00:19:25,448 --> 00:19:29,123 We use the same ransom note that Conti actually sends out. 529 00:19:29,123 --> 00:19:31,770 It'll have sometimes their name in it and we'll say, 530 00:19:31,770 --> 00:19:32,880 Do you know Conti? 531 00:19:32,880 --> 00:19:34,500 Do you know where they are? 532 00:19:34,500 --> 00:19:37,140 Do you know what sorts of tactics they're using? 533 00:19:37,140 --> 00:19:39,420 Are you able to do any research during this ex- 534 00:19:39,420 --> 00:19:40,950 This is for technical teams obviously, 535 00:19:40,950 --> 00:19:43,792 but is there anything you can do research wise 536 00:19:43,792 --> 00:19:46,620 on this threat actor that you can bring to leadership 537 00:19:46,620 --> 00:19:49,620 and say, this is the reputation that they have 538 00:19:49,620 --> 00:19:51,210 as far as whether or not they leak data 539 00:19:51,210 --> 00:19:52,500 after getting ransom paid; 540 00:19:52,500 --> 00:19:53,793 like that sort of thing. 541 00:19:53,793 --> 00:19:56,070 Like I said, threat intelligence is usually the last thing 542 00:19:56,070 --> 00:19:58,710 that companies do on their maturity. 543 00:19:58,710 --> 00:20:00,720 So usually, they're not prepared for that sort of thing, 544 00:20:00,720 --> 00:20:02,550 but there is a lot that they can do 545 00:20:02,550 --> 00:20:04,590 just with the information of the threat actor 546 00:20:04,590 --> 00:20:05,460 from the ransom node. 547 00:20:05,460 --> 00:20:08,070 So we try and encourage them to do a little digging 548 00:20:08,070 --> 00:20:10,380 and see if they can use that to help them 549 00:20:10,380 --> 00:20:12,180 during the exercise. 550 00:20:12,180 --> 00:20:14,344 - I want to foot stomp on the, 551 00:20:14,344 --> 00:20:15,440 (Sarah laughing) 552 00:20:15,440 --> 00:20:18,270 like this company wants an APT war game, 553 00:20:18,270 --> 00:20:21,270 even though they shouldn't be worried about APTs. 554 00:20:21,270 --> 00:20:24,000 Like threat intelligence, yes, is important, 555 00:20:24,000 --> 00:20:26,708 but only if it is relevant and actionable. 556 00:20:26,708 --> 00:20:28,980 So otherwise, it's just threat news. 557 00:20:28,980 --> 00:20:30,071 Like it's not... 558 00:20:30,071 --> 00:20:30,904 (Winnona laughing) 559 00:20:30,904 --> 00:20:32,550 - And also, I don't think we mentioned, 560 00:20:32,550 --> 00:20:35,070 but the three of us are all DEF CON policy goons 561 00:20:35,070 --> 00:20:36,720 this year as well, so if you have any questions 562 00:20:36,720 --> 00:20:38,640 about DEF CON policy afterwards, come and talk to us. 563 00:20:38,640 --> 00:20:39,660 Like someone had a suggestion 564 00:20:39,660 --> 00:20:42,090 for running a threat scenario next year. 565 00:20:42,090 --> 00:20:44,430 We will actually take that into consideration. 566 00:20:44,430 --> 00:20:46,011 'Cause we're gonna be planning it, so. 567 00:20:46,011 --> 00:20:48,120 - If I could add two pieces really, really quickly, 568 00:20:48,120 --> 00:20:49,410 just on the CTF piece. 569 00:20:49,410 --> 00:20:50,940 We've been talking for a while. 570 00:20:50,940 --> 00:20:52,860 Trey and I have been talking for a while 571 00:20:52,860 --> 00:20:55,427 about trying to find a way to collaborate with CTFs, 572 00:20:55,427 --> 00:20:58,950 and hackathons, and have the Cyber 9/12 storyline. 573 00:20:58,950 --> 00:20:59,970 Because it is a story, 574 00:20:59,970 --> 00:21:02,100 it's a scenario, but there is a storyline, 575 00:21:02,100 --> 00:21:03,630 there are characters, and there are actors, 576 00:21:03,630 --> 00:21:05,241 and there is an arch, et cetera. 577 00:21:05,241 --> 00:21:07,860 How can we collaborate with CTFs and hackathons, 578 00:21:07,860 --> 00:21:09,450 so we can create learning opportunities 579 00:21:09,450 --> 00:21:11,130 that are multidisciplinary, 580 00:21:11,130 --> 00:21:13,646 where you can learn to analyze lots of information 581 00:21:13,646 --> 00:21:15,120 in a short period of time, 582 00:21:15,120 --> 00:21:17,427 learn how to present that to decision makers, 583 00:21:17,427 --> 00:21:19,530 but also learn the technical aspects of it 584 00:21:19,530 --> 00:21:20,919 as well with your team. 585 00:21:20,919 --> 00:21:23,140 So if you have any feedback on that, please let me know. 586 00:21:23,140 --> 00:21:24,423 We'll really keen for that. 587 00:21:24,423 --> 00:21:26,520 The other piece, if I can add tiny bit 588 00:21:26,520 --> 00:21:29,250 of contrast is sometimes, in some of our simulations, 589 00:21:29,250 --> 00:21:32,460 we try to lean away from nation state threat actors, 590 00:21:32,460 --> 00:21:34,257 just because it's easier for students. 591 00:21:34,257 --> 00:21:36,600 There is a policy framework for how to engage. 592 00:21:36,600 --> 00:21:38,220 There are diplomatic channels. 593 00:21:38,220 --> 00:21:39,120 We wanna challenge them. 594 00:21:39,120 --> 00:21:40,440 We want them to be uncomfortable. 595 00:21:40,440 --> 00:21:43,010 So engaging with a non-state actor often pushes them 596 00:21:43,010 --> 00:21:45,600 in that direction of you just can't call for sanctions. 597 00:21:45,600 --> 00:21:47,250 It's not a thing. 598 00:21:47,250 --> 00:21:48,180 So wherever we get... 599 00:21:48,180 --> 00:21:49,980 And students like sanctions. 600 00:21:49,980 --> 00:21:51,660 It's a very American thing. 601 00:21:51,660 --> 00:21:53,730 So we like to make sure that they're not just leaning 602 00:21:53,730 --> 00:21:55,380 towards that first tool that's ready 603 00:21:55,380 --> 00:21:57,240 and getting that knee jerk reaction. 604 00:21:57,240 --> 00:21:59,633 But Chris, I saw you wanna lean in. 605 00:21:59,633 --> 00:22:01,230 - [Chris] Yeah, just a couple things. 606 00:22:01,230 --> 00:22:03,570 One, on Sarah's point about law enforcement, 607 00:22:03,570 --> 00:22:04,920 I agree, they can't solve everything. 608 00:22:04,920 --> 00:22:06,330 And they're not gonna prioritize everything. 609 00:22:06,330 --> 00:22:10,170 But I think the answer in part is for entities 610 00:22:10,170 --> 00:22:11,940 to have an incident response plan in place 611 00:22:11,940 --> 00:22:13,980 that includes who they're gonna call in law enforcement. 612 00:22:13,980 --> 00:22:15,750 So in other words, it's not just person relationships, 613 00:22:15,750 --> 00:22:16,980 which are important. 614 00:22:16,980 --> 00:22:19,610 But also having a plan in place, so you're not scrambling 615 00:22:19,610 --> 00:22:20,910 when something happens. 616 00:22:20,910 --> 00:22:23,506 And there's so many places that don't have that still. 617 00:22:23,506 --> 00:22:25,950 It's amazing, so that that's important. 618 00:22:25,950 --> 00:22:28,350 And law enforcement, and also that relationship before, 619 00:22:28,350 --> 00:22:31,223 I mean join things like InfraGard and things like that. 620 00:22:31,223 --> 00:22:32,760 You know, just have that relationship, 621 00:22:32,760 --> 00:22:34,500 so you're not just coming in cold. 622 00:22:34,500 --> 00:22:38,700 On the idea of tabletops and storytelling, two things. 623 00:22:38,700 --> 00:22:42,180 One, I think kind of an unserved area is 624 00:22:42,180 --> 00:22:44,070 at not the policy policy level, 625 00:22:44,070 --> 00:22:46,440 but at the board level and getting them to understand it. 626 00:22:46,440 --> 00:22:48,085 And they're not gonna understand the technology, 627 00:22:48,085 --> 00:22:50,930 but the side benefit of that is getting the technical people 628 00:22:50,930 --> 00:22:52,920 in a company like the CISO 629 00:22:52,920 --> 00:22:54,570 to actually have some access to the board. 630 00:22:54,570 --> 00:22:56,493 And so that thinking about how you can do that 631 00:22:56,493 --> 00:22:58,410 in a good way, I think, is important. 632 00:22:58,410 --> 00:23:00,389 And then, in all the tabletops I've done 633 00:23:00,389 --> 00:23:02,250 when I was in government and others, 634 00:23:02,250 --> 00:23:04,920 the biggest mistake that I see made 635 00:23:04,920 --> 00:23:07,274 all the time is breaking everything. 636 00:23:07,274 --> 00:23:09,660 So that you can't, it's just hopeless, right? 637 00:23:09,660 --> 00:23:12,720 So I remember all the Cyber Storm exercises. 638 00:23:12,720 --> 00:23:14,430 I remember when the government saying, 639 00:23:14,430 --> 00:23:16,560 if I'm here for next year Cyber Storm exercise, 640 00:23:16,560 --> 00:23:17,393 please kill me. 641 00:23:17,393 --> 00:23:18,685 (audience and panelists laughing) 642 00:23:18,685 --> 00:23:20,483 Because every fricking year, 643 00:23:20,483 --> 00:23:22,980 the whoever designed it wanted to be clever 644 00:23:22,980 --> 00:23:23,813 and so broke everything. 645 00:23:23,813 --> 00:23:26,250 And so you don't learn anything if everything is broken. 646 00:23:26,250 --> 00:23:29,400 So focused exercise, I think, are very important. 647 00:23:29,400 --> 00:23:30,420 - Yeah, I a hundred percent agree. 648 00:23:30,420 --> 00:23:32,940 And I, just to clarify, don't wanna come off saying 649 00:23:32,940 --> 00:23:34,410 that you shouldn't report to law enforcement. 650 00:23:34,410 --> 00:23:35,250 You absolutely should. 651 00:23:35,250 --> 00:23:37,620 I'm, in my personal capacity, very in favor of it. 652 00:23:37,620 --> 00:23:40,357 It helps us, 'cause I do policy stuff too. 653 00:23:40,357 --> 00:23:43,950 It helps us get more data, which helps us get more funding, 654 00:23:43,950 --> 00:23:46,890 which helps us make more nationwide decisions about stuff 655 00:23:46,890 --> 00:23:48,330 like ransomware, ransomware threat actors. 656 00:23:48,330 --> 00:23:50,250 So I do think that contacting law enforcement... 657 00:23:50,250 --> 00:23:51,083 I didn't mean to come off 658 00:23:51,083 --> 00:23:54,285 and say that you shouldn't contact law enforcement, yeah. 659 00:23:54,285 --> 00:23:55,706 - [Audience Member] Counsel wants us to tell people 660 00:23:55,706 --> 00:23:56,906 never call law enforcement. 661 00:23:56,906 --> 00:23:57,739 - Yeah, yeah. 662 00:23:57,739 --> 00:23:58,953 (audience member faintly speaking) 663 00:23:58,953 --> 00:24:01,050 Yeah, and the technical capacity thing, 664 00:24:01,050 --> 00:24:02,160 I'll just say one thing really quick, 665 00:24:02,160 --> 00:24:03,838 'cause I want more questions from people. 666 00:24:03,838 --> 00:24:07,140 I think that tabletops are a really awesome 667 00:24:07,140 --> 00:24:10,500 two to four hour way of getting really good exposure 668 00:24:10,500 --> 00:24:15,500 for your security teams, getting executive support, 669 00:24:15,546 --> 00:24:18,863 teaching management, how valuable the team is, 670 00:24:18,863 --> 00:24:21,780 helping them realize that there is still maturity gaps, 671 00:24:21,780 --> 00:24:23,580 even though the program is going well. 672 00:24:23,580 --> 00:24:24,780 I think that's something that we try 673 00:24:24,780 --> 00:24:26,760 and make sure that we highlight the security team is doing 674 00:24:26,760 --> 00:24:30,120 good things while still saying, but they need more funding, 675 00:24:30,120 --> 00:24:32,370 more resources, more people. 676 00:24:32,370 --> 00:24:34,650 It's actually a very quick way of being able 677 00:24:34,650 --> 00:24:36,660 to get more exposure and more recognition. 678 00:24:36,660 --> 00:24:38,010 And hopefully more buy in. 679 00:24:38,010 --> 00:24:39,900 We often see after tabletop exercises 680 00:24:39,900 --> 00:24:42,600 that executives will approve another chunk of funding 681 00:24:42,600 --> 00:24:44,910 for folks, because they actually get to experience 682 00:24:44,910 --> 00:24:46,370 what would be like if things went wrong 683 00:24:46,370 --> 00:24:49,147 and don't want that to happen, so. 684 00:24:50,735 --> 00:24:52,460 - Excellent, I think we see two people over here. 685 00:24:52,460 --> 00:24:54,120 So you first and then you next. 686 00:24:54,120 --> 00:24:54,953 Perfect. 687 00:24:54,953 --> 00:24:56,940 - [Audience Member] Well on the topic 688 00:24:56,940 --> 00:24:59,451 of notifying law enforcement, 689 00:24:59,451 --> 00:25:02,550 we've got a rapidly changing 690 00:25:02,550 --> 00:25:06,330 and shifting regulatory environment. 691 00:25:06,330 --> 00:25:07,884 So to what extent is it important 692 00:25:07,884 --> 00:25:12,884 to work regulatory compliance into the tabletop? 693 00:25:13,800 --> 00:25:14,990 So you say that, all right, 694 00:25:14,990 --> 00:25:18,878 we're coming up on a point at this hour, 695 00:25:18,878 --> 00:25:23,878 you must inform law enforcement or relevant stock, 696 00:25:25,958 --> 00:25:27,840 that sort of thing. 697 00:25:27,840 --> 00:25:31,141 So that these regulations become forcing mechanisms 698 00:25:31,141 --> 00:25:33,003 on certain communications. 699 00:25:34,890 --> 00:25:35,725 - Yes. 700 00:25:35,725 --> 00:25:37,694 (panelists and audience laughing) 701 00:25:37,694 --> 00:25:41,148 I mean, it's partially two things, right? 702 00:25:41,148 --> 00:25:44,040 The first is that because it's regulation, 703 00:25:44,040 --> 00:25:45,810 you wanna make your war game or your tabletop 704 00:25:45,810 --> 00:25:46,890 as realistic as possible. 705 00:25:46,890 --> 00:25:49,380 And so putting that in is not only that forcing function 706 00:25:49,380 --> 00:25:50,508 that the teams need. 707 00:25:50,508 --> 00:25:52,187 They might not even know. 708 00:25:52,187 --> 00:25:55,200 And so, it's always fun to make that choice 709 00:25:55,200 --> 00:25:56,760 of whether or not you wanna inform them 710 00:25:56,760 --> 00:25:58,380 that this is the parameter of the exercise. 711 00:25:58,380 --> 00:26:02,763 And by the way, in 24 hours or at 23 hours, 712 00:26:02,763 --> 00:26:04,979 you can go in and go, by the way, 713 00:26:04,979 --> 00:26:08,502 have you thought about your SEC obligations? 714 00:26:08,502 --> 00:26:09,660 (Winnona laughing) 715 00:26:09,660 --> 00:26:12,388 And it's a choice that you make when you're designing. 716 00:26:12,388 --> 00:26:14,510 But yeah, short answer, yes. 717 00:26:14,510 --> 00:26:15,343 (Winnona laughing) 718 00:26:15,343 --> 00:26:17,070 - Yeah, there's an even easier way 719 00:26:17,070 --> 00:26:20,760 to do that is what I like doing is who's in charge of that? 720 00:26:20,760 --> 00:26:22,290 Who's your data privacy officer? 721 00:26:22,290 --> 00:26:23,613 Have you talked to them? 722 00:26:24,450 --> 00:26:27,090 A lot of folks, at least recently, 723 00:26:27,090 --> 00:26:29,610 I've asked about a breach disclosure requirements 724 00:26:29,610 --> 00:26:31,710 and someone a few weeks back was like, 725 00:26:31,710 --> 00:26:33,060 well, it was an accident. 726 00:26:33,060 --> 00:26:34,900 So we don't have to report to the breach. 727 00:26:34,900 --> 00:26:35,733 (audience laughing) 728 00:26:35,733 --> 00:26:36,810 And I was like, oh my goodness. 729 00:26:36,810 --> 00:26:40,110 There really is a big range of understanding, 730 00:26:40,110 --> 00:26:41,850 both in management and technical folks 731 00:26:41,850 --> 00:26:43,800 about what the implications of losing data is. 732 00:26:43,800 --> 00:26:44,633 That is the biggest, 733 00:26:44,633 --> 00:26:46,530 I know that there are lots of cyber reporting requirements 734 00:26:46,530 --> 00:26:48,090 that have sort of influx. 735 00:26:48,090 --> 00:26:50,760 But even the state breach reporting requirements have been 736 00:26:50,760 --> 00:26:53,880 there for a while and many, many states have them. 737 00:26:53,880 --> 00:26:56,520 And so I'm asking who's responsible for that, 738 00:26:56,520 --> 00:26:57,540 and are they in the loop? 739 00:26:57,540 --> 00:27:00,000 And most often my answer is no. 740 00:27:00,000 --> 00:27:02,610 And I write lots and lots 741 00:27:02,610 --> 00:27:04,830 of recommendations about you need someone 742 00:27:04,830 --> 00:27:07,110 who understands your reporting requirements. 743 00:27:07,110 --> 00:27:08,645 They need to be internal. 744 00:27:08,645 --> 00:27:09,930 A lot of folks are like, 745 00:27:09,930 --> 00:27:12,990 yeah, well ask for external counsel to do... 746 00:27:12,990 --> 00:27:14,970 And I'm like, do you know how long it's gonna take 747 00:27:14,970 --> 00:27:16,440 for you to do that sort of thing? 748 00:27:16,440 --> 00:27:18,660 Sometimes it takes months, 749 00:27:18,660 --> 00:27:20,850 maybe even years for really large organizations 750 00:27:20,850 --> 00:27:22,120 that have lost a lot of data. 751 00:27:22,120 --> 00:27:24,300 So I'm always checking to make sure 752 00:27:24,300 --> 00:27:26,700 that they have an actual resource with that knowledge. 753 00:27:26,700 --> 00:27:28,560 Because I think maybe it's more useful 754 00:27:28,560 --> 00:27:30,600 than me trying to track every single requirement 755 00:27:30,600 --> 00:27:32,160 and being like you're in the state of New York. 756 00:27:32,160 --> 00:27:33,450 And do you know that you have to report 757 00:27:33,450 --> 00:27:34,950 to these entities specifically in New York? 758 00:27:34,950 --> 00:27:37,050 But I wanna make sure that they know that. 759 00:27:38,640 --> 00:27:39,740 - Gentleman over here. 760 00:27:40,676 --> 00:27:42,090 You can use the microphone too. 761 00:27:42,090 --> 00:27:44,910 - [Audience Member] I'll slide in from the back bench here. 762 00:27:44,910 --> 00:27:45,743 Hi. 763 00:27:45,743 --> 00:27:50,070 So to what extent do you think we need to start modeling 764 00:27:50,070 --> 00:27:54,925 cyber enabled long term cognitive warfare campaigns 765 00:27:54,925 --> 00:27:58,297 that are being directed against our social institutions? 766 00:27:58,297 --> 00:28:00,510 So that's something that, especially for 767 00:28:00,510 --> 00:28:03,235 senior level national security decision makers, 768 00:28:03,235 --> 00:28:08,235 they're very, very focused on the in extremis sicker, 769 00:28:08,940 --> 00:28:10,440 critical infrastructure attack 770 00:28:10,440 --> 00:28:12,900 or financial institution attack. 771 00:28:12,900 --> 00:28:15,145 But I don't think people are calculating 772 00:28:15,145 --> 00:28:18,450 in foreign adversaries using social media 773 00:28:18,450 --> 00:28:20,940 over a long term to create social disruption 774 00:28:20,940 --> 00:28:23,100 and undermining institutions, 775 00:28:23,100 --> 00:28:26,370 which create systemic problems. 776 00:28:26,370 --> 00:28:28,080 Oh, thanks. 777 00:28:28,080 --> 00:28:32,070 - So I love the vagueness of that question 778 00:28:32,070 --> 00:28:35,250 in terms of long term information warfare, 779 00:28:35,250 --> 00:28:36,933 combined with cyber, right? 780 00:28:39,267 --> 00:28:43,710 'Cause I'm headed into cyber law at this point. 781 00:28:43,710 --> 00:28:46,623 And so I'll give you the lawyerly answer, and it depends. 782 00:28:46,623 --> 00:28:47,456 (Winnona laughing) 783 00:28:47,456 --> 00:28:51,150 But it depends on what particular simulation 784 00:28:51,150 --> 00:28:52,410 are you trying to emulate? 785 00:28:52,410 --> 00:28:56,790 Are you trying to emulate a culmination 786 00:28:56,790 --> 00:28:59,910 of a lot of internal dissent that was fermented 787 00:28:59,910 --> 00:29:01,680 by a foreign adversary over social media 788 00:29:01,680 --> 00:29:03,780 in a long term campaign? 789 00:29:03,780 --> 00:29:05,700 Are you looking at a shock and awe 790 00:29:05,700 --> 00:29:08,400 in terms of here's a cyber enabled attack 791 00:29:08,400 --> 00:29:10,560 that also comes with a follow on 792 00:29:10,560 --> 00:29:14,190 of information, dissemination that sows chaos? 793 00:29:14,190 --> 00:29:16,380 And like I, personally, 794 00:29:16,380 --> 00:29:19,200 from a threat intelligence perspective, 795 00:29:19,200 --> 00:29:20,970 I think that the information domain, 796 00:29:20,970 --> 00:29:23,730 as well as a cyber domain are inextricably linked 797 00:29:23,730 --> 00:29:25,200 and adversaries understand that. 798 00:29:25,200 --> 00:29:29,940 That's why you see the IRA alongside other Russian APTs 799 00:29:29,940 --> 00:29:32,040 and it may not be entirely coordinated, 800 00:29:32,040 --> 00:29:34,653 but having similar mission sets. 801 00:29:35,926 --> 00:29:39,630 And so I think that when you're talking about the adversary, 802 00:29:39,630 --> 00:29:42,600 when you're talking about designing a war game 803 00:29:42,600 --> 00:29:44,610 with a nation state level threat in mind, 804 00:29:44,610 --> 00:29:46,080 you have to consider both. 805 00:29:46,080 --> 00:29:50,730 However, that does depend on your primary principle. 806 00:29:50,730 --> 00:29:52,770 If you are designing a war game specifically 807 00:29:52,770 --> 00:29:57,770 for just the technical boots on the ground type IT defender, 808 00:29:58,260 --> 00:29:59,310 that's going to be outta the question. 809 00:29:59,310 --> 00:30:02,070 If you're designing it specifically for DOD policy, 810 00:30:02,070 --> 00:30:05,210 someone a little bit up the chain, then yes. 811 00:30:05,210 --> 00:30:07,320 Because information space, 812 00:30:07,320 --> 00:30:10,650 as well as cyberspace are pretty (indistinct) linked. 813 00:30:10,650 --> 00:30:13,560 The only caveat I would say to that is it depends 814 00:30:13,560 --> 00:30:16,800 on which adversary you are trying to emulate. 815 00:30:16,800 --> 00:30:19,980 Because while Russia and China may understand 816 00:30:19,980 --> 00:30:24,390 that they call it informationization, 817 00:30:24,390 --> 00:30:29,223 intelligentization is the new Chinese translation word. 818 00:30:31,147 --> 00:30:34,320 It's going to be different if you're emulating an actor 819 00:30:34,320 --> 00:30:36,390 from Iran, or an actor from North Korea, 820 00:30:36,390 --> 00:30:38,009 or an actor from Vietnam. 821 00:30:38,009 --> 00:30:38,842 (Winnona laughing) 822 00:30:38,842 --> 00:30:40,860 So I think it depends on the principle, 823 00:30:40,860 --> 00:30:43,260 it depends what area they're particularly interested in, 824 00:30:43,260 --> 00:30:47,250 and it depends on how long term 825 00:30:47,250 --> 00:30:49,200 of a war game you're looking at. 826 00:30:49,200 --> 00:30:51,420 - Yeah, and I defer to Winnona on this one, 827 00:30:51,420 --> 00:30:53,396 'cause she's the expert here on APTs. 828 00:30:53,396 --> 00:30:55,377 I'm not sure that the tabletops 829 00:30:55,377 --> 00:30:58,770 and war games are as well suited to those sort of long term, 830 00:30:58,770 --> 00:31:00,120 more vague scenarios. 831 00:31:00,120 --> 00:31:01,470 'Cause I think it's really important 832 00:31:01,470 --> 00:31:03,960 that they're very targeted, 833 00:31:03,960 --> 00:31:05,750 and they have takeaways at the end of them. 834 00:31:05,750 --> 00:31:09,030 And I think the more vague the scenario gets 835 00:31:09,030 --> 00:31:10,800 and when it gets to be on that scale, 836 00:31:10,800 --> 00:31:13,562 you know, information warfare on a national level, 837 00:31:13,562 --> 00:31:15,750 it's hard to walk away from that, 838 00:31:15,750 --> 00:31:18,097 knowing what to do to fix it. 839 00:31:18,097 --> 00:31:21,270 And I don't want folks to sit in the room for five hours 840 00:31:21,270 --> 00:31:23,580 and say, well, that was interesting, and then go home. 841 00:31:23,580 --> 00:31:25,969 I want them to say, I know what we need to do differently. 842 00:31:25,969 --> 00:31:27,930 And that's why I think they're so useful 843 00:31:27,930 --> 00:31:31,290 for things like ransomware, for even espionage campaigns, 844 00:31:31,290 --> 00:31:32,694 data loss sort of campaigns, 845 00:31:32,694 --> 00:31:34,980 because you can walk away with them and say, 846 00:31:34,980 --> 00:31:36,420 I know what I need to fix. 847 00:31:36,420 --> 00:31:38,370 I know what we need to do better next time. 848 00:31:38,370 --> 00:31:40,350 So not that they can't be done, 849 00:31:40,350 --> 00:31:41,880 but I'm not sure that they're as suited 850 00:31:41,880 --> 00:31:43,413 to those sorts of scenarios. 851 00:31:44,374 --> 00:31:45,780 - If I can- 852 00:31:45,780 --> 00:31:46,689 Do you wanna jump in? 853 00:31:46,689 --> 00:31:48,170 Please. 854 00:31:48,170 --> 00:31:51,073 Okay, I'm just gonna add a boring think tank perspective 855 00:31:51,073 --> 00:31:54,901 just because one, we can make a cyber scenario, 856 00:31:54,901 --> 00:31:59,100 but our colleagues who handle information programming 857 00:31:59,100 --> 00:32:03,780 can really, really dig into how is information manipulated 858 00:32:03,780 --> 00:32:06,321 and used to achieve broader geo-strategic goals? 859 00:32:06,321 --> 00:32:09,030 I think part of our job is making sure 860 00:32:09,030 --> 00:32:10,950 that we are putting our heads together with our colleagues 861 00:32:10,950 --> 00:32:12,570 to make sure that we are developing scenarios 862 00:32:12,570 --> 00:32:14,460 that do examine those stickier issues 863 00:32:14,460 --> 00:32:16,710 that we don't quite understand just yet. 864 00:32:16,710 --> 00:32:18,180 That way we can create more challenging, 865 00:32:18,180 --> 00:32:19,830 high quality scenarios that are beneficial 866 00:32:19,830 --> 00:32:20,850 for our funds and government. 867 00:32:20,850 --> 00:32:22,353 The other part of that is that, one, 868 00:32:22,353 --> 00:32:23,670 that we make a scenario. 869 00:32:23,670 --> 00:32:24,930 Students respond to it; 870 00:32:24,930 --> 00:32:27,180 folks from government and industry or judges. 871 00:32:27,180 --> 00:32:29,220 But after the fact, it's kind of our job 872 00:32:29,220 --> 00:32:30,930 to make sure that we pull that scenario 873 00:32:30,930 --> 00:32:32,880 and the recommendations into a playbook, 874 00:32:32,880 --> 00:32:34,695 and make that available to folks. 875 00:32:34,695 --> 00:32:37,140 Our goal really is to be able to pass that on to folks 876 00:32:37,140 --> 00:32:38,160 in government and industry, 877 00:32:38,160 --> 00:32:40,170 and have them take that and pass it along to somebody 878 00:32:40,170 --> 00:32:42,960 in their organization who can run it internally, 879 00:32:42,960 --> 00:32:44,190 and take some notes on, 880 00:32:44,190 --> 00:32:46,170 this is how it rolled out in a scenario, 881 00:32:46,170 --> 00:32:47,520 which was obviously fictional. 882 00:32:47,520 --> 00:32:49,260 But how would we actually respond to this 883 00:32:49,260 --> 00:32:51,369 if this were to happen tomorrow? 884 00:32:51,369 --> 00:32:53,447 - Safa, I have a question. 885 00:32:53,447 --> 00:32:55,836 You have all, I think everyone raised their hand 886 00:32:55,836 --> 00:32:58,590 when we asked who's been in a war game 887 00:32:58,590 --> 00:32:59,970 or in a tabletop exercise. 888 00:32:59,970 --> 00:33:03,540 Has anyone hated, does anyone dislike them? 889 00:33:03,540 --> 00:33:05,100 Have you been in a bad exercise 890 00:33:05,100 --> 00:33:06,887 and what went wrong with that? 891 00:33:06,887 --> 00:33:08,970 'Cause I think a lot of folks hate them. 892 00:33:08,970 --> 00:33:11,910 Insurance requires them these days for a lot of companies. 893 00:33:11,910 --> 00:33:13,950 I'm curious if anyone has had a bad experience 894 00:33:13,950 --> 00:33:16,050 and what you think contributed to that. 895 00:33:16,050 --> 00:33:16,883 Yeah? 896 00:33:16,883 --> 00:33:18,740 - [Audience Member] It was a budget tabletop exercise. 897 00:33:18,740 --> 00:33:19,949 (panelists and audience laughing) 898 00:33:19,949 --> 00:33:21,019 - What does that mean? 899 00:33:21,019 --> 00:33:23,070 - [Audience Member] It was trying to figure out 900 00:33:23,070 --> 00:33:27,690 how US government would route funding in a given situation 901 00:33:27,690 --> 00:33:29,610 based on international trends. 902 00:33:29,610 --> 00:33:30,810 It was delightful. 903 00:33:30,810 --> 00:33:31,643 - So it was- 904 00:33:31,643 --> 00:33:33,360 - You mean like a tabletop related to budget, 905 00:33:33,360 --> 00:33:35,362 not a very cheaply made tabletop exercise? 906 00:33:35,362 --> 00:33:36,450 (panelists and audience laughing) 907 00:33:36,450 --> 00:33:37,710 - [Audience Member] It was not a poor man's, 908 00:33:37,710 --> 00:33:39,390 poor person's tabletop. 909 00:33:39,390 --> 00:33:40,223 No, no, no. 910 00:33:40,223 --> 00:33:42,450 It was very, a lot of high level government people there. 911 00:33:42,450 --> 00:33:43,885 No, but it was focused on budgeting. 912 00:33:43,885 --> 00:33:44,718 - Okay. 913 00:33:44,718 --> 00:33:45,797 (panelists and audience laughing) 914 00:33:45,797 --> 00:33:46,817 Never heard of that before. 915 00:33:46,817 --> 00:33:49,803 - [Audience Member] I haven't had a tabletop 916 00:33:49,803 --> 00:33:51,889 that I didn't enjoy doing. 917 00:33:51,889 --> 00:33:53,239 But working with clients, 918 00:33:53,239 --> 00:33:54,978 the worst ones that I've had 919 00:33:54,978 --> 00:33:57,441 are where the client was very immature, 920 00:33:57,441 --> 00:33:59,819 maybe had never done a tabletop before. 921 00:33:59,819 --> 00:34:03,291 And they really wanted to get fancy in some way, 922 00:34:03,291 --> 00:34:05,370 and they weren't ready for it. 923 00:34:05,370 --> 00:34:06,360 Because they're still at the point 924 00:34:06,360 --> 00:34:09,960 where they don't know who to call to make a decision, 925 00:34:09,960 --> 00:34:11,619 or to get information, or whatever. 926 00:34:11,619 --> 00:34:14,147 And if they're in a situation like that, 927 00:34:14,147 --> 00:34:16,830 and they're worried about a nation state act, 928 00:34:16,830 --> 00:34:19,847 or doing some sort of supply chain attack or whatever, 929 00:34:19,847 --> 00:34:20,940 well, if that happens, 930 00:34:20,940 --> 00:34:22,290 you're just gonna get run over, right? 931 00:34:22,290 --> 00:34:24,450 Because you're no way prepared for that. 932 00:34:24,450 --> 00:34:25,710 And it would be much better 933 00:34:25,710 --> 00:34:28,260 to just deal with a basic data breach 934 00:34:28,260 --> 00:34:29,641 or a basic ransomware scenario. 935 00:34:29,641 --> 00:34:31,379 And once they've figured that out, 936 00:34:31,379 --> 00:34:34,350 and they actually know their own processes 937 00:34:34,350 --> 00:34:35,730 and kind of iron things out, 938 00:34:35,730 --> 00:34:38,070 then they can start to think bigger 939 00:34:38,070 --> 00:34:40,260 if it's relevant to them. 940 00:34:40,260 --> 00:34:43,128 - Yeah, I'd love to hear both of your opinions on this. 941 00:34:43,128 --> 00:34:44,943 But oh, sorry, go ahead, go ahead. 942 00:34:46,868 --> 00:34:48,510 - Use the mic if you don't mind. 943 00:34:48,510 --> 00:34:50,252 - Yeah, so that the- - Just so- 944 00:34:50,252 --> 00:34:52,258 - [Moderator] (faintly speaking) the stream can hear you. 945 00:34:52,258 --> 00:34:53,091 And also- 946 00:34:53,091 --> 00:34:55,606 - [Audience Member] So for... 947 00:34:55,606 --> 00:34:59,125 I helped run about 10 exercises 948 00:34:59,125 --> 00:35:04,125 and invariably, what would happen is we'd have 20 949 00:35:04,570 --> 00:35:05,730 to 50 people in the room. 950 00:35:05,730 --> 00:35:07,620 These were multi-day events, 951 00:35:07,620 --> 00:35:09,690 and they were told they had to be there. 952 00:35:09,690 --> 00:35:11,610 This was for the DOD. 953 00:35:11,610 --> 00:35:14,550 And invariably, when they show up 954 00:35:14,550 --> 00:35:16,353 on the first day, it's this. 955 00:35:16,353 --> 00:35:19,988 (panelists and audience laughing) 956 00:35:19,988 --> 00:35:22,590 But by the end of the first day, 957 00:35:22,590 --> 00:35:25,020 everyone's leaning forward, elbows on the table, 958 00:35:25,020 --> 00:35:26,640 they're all interacting. 959 00:35:26,640 --> 00:35:28,710 And they were surprised. 960 00:35:28,710 --> 00:35:32,121 This was the first tabletop like this for anyone, 961 00:35:32,121 --> 00:35:34,347 even us in putting them on. 962 00:35:34,347 --> 00:35:35,820 But they were all surprised 963 00:35:35,820 --> 00:35:38,280 at the value they got out of the exercise, 964 00:35:38,280 --> 00:35:41,340 because it wasn't what they were used to 965 00:35:41,340 --> 00:35:43,885 from a security type assessment. 966 00:35:43,885 --> 00:35:45,540 Yeah. 967 00:35:45,540 --> 00:35:47,442 - And I see we got a question over here. 968 00:35:47,442 --> 00:35:50,477 - [Audience Member] Kinda adding on that is the one 969 00:35:50,477 --> 00:35:53,010 that I worked on that I really did not like, 970 00:35:53,010 --> 00:35:56,280 was the element of they knew that they were weak 971 00:35:56,280 --> 00:35:57,690 in a very certain element. 972 00:35:57,690 --> 00:36:01,350 So they tried to put that as outside of the rules 973 00:36:01,350 --> 00:36:02,183 of engagement. 974 00:36:02,183 --> 00:36:03,930 It was like, but this is the reason for it. 975 00:36:03,930 --> 00:36:05,612 And then it was the same element of, 976 00:36:05,612 --> 00:36:07,980 well, I don't want me to look bad 977 00:36:07,980 --> 00:36:09,330 'cause I allowed this to happen. 978 00:36:09,330 --> 00:36:13,260 So it's getting the individuals that are in charge 979 00:36:13,260 --> 00:36:16,680 to accept, hey, allow us to exploit 980 00:36:16,680 --> 00:36:19,560 what we've already told you or what you've already noticed. 981 00:36:19,560 --> 00:36:21,068 Because if you notice it, 982 00:36:21,068 --> 00:36:25,023 then that means it was noticed way before you noticed it. 983 00:36:25,890 --> 00:36:28,800 So it's just that line where also, 984 00:36:28,800 --> 00:36:30,480 you said you don't want to freak them out 985 00:36:30,480 --> 00:36:32,130 to the point where they don't wanna do it. 986 00:36:32,130 --> 00:36:34,980 Because they're like fine, we'll just give up. 987 00:36:34,980 --> 00:36:36,420 - I wanna lean into this a little bit 988 00:36:36,420 --> 00:36:38,370 because this raises a really interesting thing for us, 989 00:36:38,370 --> 00:36:40,410 I think, to dig into is what do you do 990 00:36:40,410 --> 00:36:42,720 with an immature client or an immature player? 991 00:36:42,720 --> 00:36:44,730 And I wanna lean in on the player piece 992 00:36:44,730 --> 00:36:46,980 because like a war game, or a simulation, or a TTX, 993 00:36:46,980 --> 00:36:48,810 we're all players. 994 00:36:48,810 --> 00:36:50,100 I think we've seen this with students 995 00:36:50,100 --> 00:36:54,393 where if they're an immature player, they tend to, 996 00:36:54,393 --> 00:36:57,750 the knee jerk reaction is like, it's Russia or China. 997 00:36:57,750 --> 00:36:59,740 The other thing that we see is sanctions. 998 00:36:59,740 --> 00:37:02,520 The last thing that we see is we want students 999 00:37:02,520 --> 00:37:03,990 to come up with creative solutions, 1000 00:37:03,990 --> 00:37:06,810 whether they be technical or policy centric. 1001 00:37:06,810 --> 00:37:08,370 We want them to be creative. 1002 00:37:08,370 --> 00:37:10,200 We want them to think outside the box. 1003 00:37:10,200 --> 00:37:11,317 This is a tabletop exercise. 1004 00:37:11,317 --> 00:37:14,190 You know, you can lose, but at the same time, 1005 00:37:14,190 --> 00:37:15,390 losing is taking no action. 1006 00:37:15,390 --> 00:37:17,880 So shoot your shot. 1007 00:37:17,880 --> 00:37:20,910 Some students think creativity equals escalation, 1008 00:37:20,910 --> 00:37:23,820 which gets really uncomfortable, really fast. 1009 00:37:23,820 --> 00:37:26,670 You'd be surprised, the military academies tend 1010 00:37:26,670 --> 00:37:30,270 to not be as escalatory as the civilian schools. 1011 00:37:30,270 --> 00:37:31,470 Some students will think that, you know, 1012 00:37:31,470 --> 00:37:34,320 let's do a military operation in this particular place 1013 00:37:34,320 --> 00:37:37,560 because we wanna demonstrate that we care about this thing 1014 00:37:37,560 --> 00:37:38,393 and we're watching. 1015 00:37:38,393 --> 00:37:40,170 But it's like, is that actually, like sir, 1016 00:37:40,170 --> 00:37:41,688 the lights are still off. 1017 00:37:41,688 --> 00:37:44,790 Please, actually respond to the crisis at hand. 1018 00:37:44,790 --> 00:37:46,440 But would love to hear from folks around the room. 1019 00:37:46,440 --> 00:37:47,970 What do you do with an immature player? 1020 00:37:47,970 --> 00:37:51,330 How do you get them to reevaluate their standing 1021 00:37:51,330 --> 00:37:52,163 in the game? 1022 00:37:52,163 --> 00:37:55,520 - [Audience Member] Can you define immature player? 1023 00:37:55,520 --> 00:37:56,803 - Sorry, what was that? 1024 00:37:56,803 --> 00:37:59,183 - [Audience Member] Can you define immature player? 1025 00:37:59,183 --> 00:38:00,510 - I guess an immature player, 1026 00:38:00,510 --> 00:38:02,700 I guess the way I see it is someone who is, one, 1027 00:38:02,700 --> 00:38:05,850 maybe not familiar with engaging in a tabletop, 1028 00:38:05,850 --> 00:38:07,890 but at the same time, maybe not familiar 1029 00:38:07,890 --> 00:38:09,870 or does not have a really strong grasp 1030 00:38:09,870 --> 00:38:10,860 of what are they working with? 1031 00:38:10,860 --> 00:38:12,060 What are the tools in their toolbox 1032 00:38:12,060 --> 00:38:13,361 that is available at their disposal? 1033 00:38:13,361 --> 00:38:15,450 But I see that you leaned into the microphone, 1034 00:38:15,450 --> 00:38:16,413 please jump in. 1035 00:38:17,347 --> 00:38:18,540 - [Audience Member] So to answer your question 1036 00:38:18,540 --> 00:38:20,944 and also a follow up question to that, 1037 00:38:20,944 --> 00:38:24,180 in regards to immature organizations going 1038 00:38:24,180 --> 00:38:27,000 through the exercise, one of the things 1039 00:38:27,000 --> 00:38:30,340 that I found is very helpful is taking a step back 1040 00:38:31,722 --> 00:38:35,520 and preparing, going through the prep exercises 1041 00:38:35,520 --> 00:38:36,690 before the TTX. 1042 00:38:36,690 --> 00:38:41,690 So for example, examining and developing the IR workflow, 1043 00:38:42,300 --> 00:38:44,700 and looking at playbooks, and so on. 1044 00:38:44,700 --> 00:38:49,635 So that there's some sort of guiding standards 1045 00:38:49,635 --> 00:38:52,536 or guidelines and something that they can refer back to 1046 00:38:52,536 --> 00:38:56,475 that's helpful to make them not feel as fearful 1047 00:38:56,475 --> 00:38:58,883 or as intimidated by the exercise. 1048 00:38:58,883 --> 00:39:01,083 But I'd also like to hear your feedback 1049 00:39:01,083 --> 00:39:03,915 on what you think about, you know, 1050 00:39:03,915 --> 00:39:08,707 you have a client who's requesting a war game or a TTX, 1051 00:39:09,990 --> 00:39:11,550 and taking that step back and saying, 1052 00:39:11,550 --> 00:39:14,787 why don't you look at the preparatory exercises 1053 00:39:14,787 --> 00:39:16,230 before you get to that 1054 00:39:16,230 --> 00:39:19,380 in order to increase the effectiveness of a tabletop? 1055 00:39:19,380 --> 00:39:20,930 What are your thoughts on that? 1056 00:39:22,962 --> 00:39:24,150 - Oh, okay. 1057 00:39:24,150 --> 00:39:26,904 I don't know if I have a lot of thoughts on that, actually. 1058 00:39:26,904 --> 00:39:29,190 I don't mind running tabletops with folks 1059 00:39:29,190 --> 00:39:30,263 who've never done it before. 1060 00:39:30,263 --> 00:39:31,297 I think they learn a lot. 1061 00:39:31,297 --> 00:39:32,820 I think in particular, 1062 00:39:32,820 --> 00:39:35,250 immature players like Safa was saying, 1063 00:39:35,250 --> 00:39:38,253 vastly overestimate how good they're going to be at it. 1064 00:39:39,150 --> 00:39:42,360 That's like Sue was saying that people wanna get fancy 1065 00:39:42,360 --> 00:39:44,370 with it 'cause they think they're gonna just do so well 1066 00:39:44,370 --> 00:39:45,907 that it's not gonna be useful to them. 1067 00:39:45,907 --> 00:39:47,340 And even folks who have done it 1068 00:39:47,340 --> 00:39:50,310 a few times still have things to work on. 1069 00:39:50,310 --> 00:39:51,990 And they're mostly boring things, 1070 00:39:51,990 --> 00:39:54,120 like folks will have these incident response plans 1071 00:39:54,120 --> 00:39:55,860 that are 180 pages long. 1072 00:39:55,860 --> 00:39:56,693 They don't open them once. 1073 00:39:56,693 --> 00:39:58,323 They're totally useless to them. 1074 00:39:58,323 --> 00:40:00,000 It's very sad. 1075 00:40:00,000 --> 00:40:02,010 But it's the really boring things 1076 00:40:02,010 --> 00:40:03,600 about not using your processes, 1077 00:40:03,600 --> 00:40:05,070 not having any processes in place, 1078 00:40:05,070 --> 00:40:06,785 not relying on your documentation, 1079 00:40:06,785 --> 00:40:10,101 not having communications, like out of bound communications 1080 00:40:10,101 --> 00:40:12,861 in case your communications are compromised. 1081 00:40:12,861 --> 00:40:15,510 It's sort of the same things over and over again. 1082 00:40:15,510 --> 00:40:18,000 And so mature players, immature players, 1083 00:40:18,000 --> 00:40:21,030 everyone sort of overestimates how good they're gonna be 1084 00:40:21,030 --> 00:40:23,910 at it, but you can run a very simple ransom or exercise. 1085 00:40:23,910 --> 00:40:25,320 And if you ask the right questions, 1086 00:40:25,320 --> 00:40:27,240 you'll still be able to find some gaps 1087 00:40:27,240 --> 00:40:29,190 in the response there. 1088 00:40:29,190 --> 00:40:30,780 I saw two hands up. 1089 00:40:30,780 --> 00:40:32,783 - Well, I don't think we've heard from you before. 1090 00:40:32,783 --> 00:40:34,380 - [Audience Member] So what I usually do 1091 00:40:34,380 --> 00:40:39,380 with my immature players is I start by asking for... 1092 00:40:40,590 --> 00:40:43,133 Well, during the scoping of the entire thing, 1093 00:40:43,133 --> 00:40:46,110 I start going through all their policies and procedures. 1094 00:40:46,110 --> 00:40:47,806 And I'm not asking for them for me. 1095 00:40:47,806 --> 00:40:50,790 I'm just asking to see they have them. 1096 00:40:50,790 --> 00:40:54,360 And with one of my most immature players, 1097 00:40:54,360 --> 00:40:56,820 I actually just told 'em to step back 1098 00:40:56,820 --> 00:40:58,980 from the tabletop completely. 1099 00:40:58,980 --> 00:41:01,170 We switched the entire thing to, 1100 00:41:01,170 --> 00:41:03,072 we're gonna do micro exercises. 1101 00:41:03,072 --> 00:41:07,284 So that way we can go through each of your policies first, 1102 00:41:07,284 --> 00:41:11,220 and then we're gonna go through a larger tabletop exercise. 1103 00:41:11,220 --> 00:41:14,610 Because I found that they had gaps everywhere, 1104 00:41:14,610 --> 00:41:16,710 from communications to governance, 1105 00:41:16,710 --> 00:41:20,664 to even just the technical incident response team. 1106 00:41:20,664 --> 00:41:22,560 Everyone had gaps. 1107 00:41:22,560 --> 00:41:24,908 So kind of like we do with technical teams, 1108 00:41:24,908 --> 00:41:27,796 'cause this was supposed to be a management exercise. 1109 00:41:27,796 --> 00:41:29,980 And we took it down into technical pieces, 1110 00:41:29,980 --> 00:41:33,090 but technical at the different policy levels 1111 00:41:33,090 --> 00:41:34,290 to tear through different policies. 1112 00:41:34,290 --> 00:41:35,580 That's how we did it. 1113 00:41:35,580 --> 00:41:36,540 - That's a really good point. 1114 00:41:36,540 --> 00:41:37,877 I think there are things you can do 1115 00:41:37,877 --> 00:41:39,510 with a really immature client 1116 00:41:39,510 --> 00:41:41,310 to sort hand hold them through it. 1117 00:41:41,310 --> 00:41:42,870 You can also run, we've done this before, 1118 00:41:42,870 --> 00:41:45,419 like tabletop exercises that are more educational. 1119 00:41:45,419 --> 00:41:47,988 You bring them, you say, here's a ransom note. 1120 00:41:47,988 --> 00:41:51,840 What happens now is that you'll need to engage 1121 00:41:51,840 --> 00:41:53,220 your ransomware negotiator. 1122 00:41:53,220 --> 00:41:54,250 'Cause you don't wanna be talking to this... 1123 00:41:54,250 --> 00:41:57,510 We'll sort of teach them as we're going, 1124 00:41:57,510 --> 00:41:58,980 what they need to be thinking about 1125 00:41:58,980 --> 00:42:00,360 versus expecting them to know it. 1126 00:42:00,360 --> 00:42:01,377 Because we know that they don't know it, 1127 00:42:01,377 --> 00:42:03,360 and we don't want an empty, quiet room. 1128 00:42:03,360 --> 00:42:04,830 - I think we have two fingers here 1129 00:42:04,830 --> 00:42:06,510 and then a comment over here. 1130 00:42:06,510 --> 00:42:08,490 - [Audience Member] Yeah, just a quick follow up for that. 1131 00:42:08,490 --> 00:42:11,366 So we're talking about organizations 1132 00:42:11,366 --> 00:42:13,380 at a level of immaturity. 1133 00:42:13,380 --> 00:42:15,660 So when we take a look broadly 1134 00:42:15,660 --> 00:42:17,310 at a capability maturity model, 1135 00:42:17,310 --> 00:42:19,612 okay, if you're at a certain phase, 1136 00:42:19,612 --> 00:42:23,040 what organizational pieces do you need to put in place 1137 00:42:23,040 --> 00:42:26,579 as the building blocks to get you to a more mature state? 1138 00:42:26,579 --> 00:42:29,430 And I don't think we have that on a spectrum yet, 1139 00:42:29,430 --> 00:42:32,518 where we can kind of say, okay, based on this exercise, 1140 00:42:32,518 --> 00:42:36,240 you're at this level of maturity based on X, Y, and Z. 1141 00:42:36,240 --> 00:42:39,750 Before the next exercise, we need you to improve 1142 00:42:39,750 --> 00:42:43,650 in X, Y, and Z to get to that level two or that level three. 1143 00:42:43,650 --> 00:42:48,390 So an organization understands throughout the organization, 1144 00:42:48,390 --> 00:42:51,240 what pieces need to be worked on by which levels 1145 00:42:51,240 --> 00:42:53,828 of the organization to improve their capabilities, 1146 00:42:53,828 --> 00:42:56,400 to defend against these cyber threats. 1147 00:42:56,400 --> 00:43:00,570 So I think it's really kind of vertical going all the way up 1148 00:43:00,570 --> 00:43:04,230 to the board, because they're on the hook for it also. 1149 00:43:04,230 --> 00:43:07,350 But once we get that maturity model in this space, 1150 00:43:07,350 --> 00:43:10,020 I think that's a good guiding measurement. 1151 00:43:10,020 --> 00:43:13,500 So I think that's another kind of more enterprise takeaway. 1152 00:43:13,500 --> 00:43:14,700 How do we measure maturity? 1153 00:43:14,700 --> 00:43:17,280 Because you can't manage it if you can't measure it, 1154 00:43:17,280 --> 00:43:18,113 so that's... 1155 00:43:19,980 --> 00:43:22,620 - And I think, if we can get this gentleman over here 1156 00:43:22,620 --> 00:43:23,880 and then we'll have your comment. 1157 00:43:23,880 --> 00:43:25,263 So please. 1158 00:43:27,302 --> 00:43:30,300 - [Audience Member] To just address a couple things, 1159 00:43:30,300 --> 00:43:31,800 first of all, the one lady was talking 1160 00:43:31,800 --> 00:43:33,690 about setting it up early 1161 00:43:33,690 --> 00:43:37,020 and kind of giving them some pre-advice. 1162 00:43:37,020 --> 00:43:40,470 When it's a first time for a client, 1163 00:43:40,470 --> 00:43:42,690 I found that works extremely well. 1164 00:43:42,690 --> 00:43:44,160 'Cause they don't even know how to play. 1165 00:43:44,160 --> 00:43:47,520 And you can put all the documentation out there, 1166 00:43:47,520 --> 00:43:49,530 but walk 'em through the basics. 1167 00:43:49,530 --> 00:43:54,530 And I also agree, if they do not have any sort of IR plan, 1168 00:43:55,950 --> 00:43:57,480 have some pre-sessions. 1169 00:43:57,480 --> 00:43:59,324 Help them at least create a straw man, 1170 00:43:59,324 --> 00:44:03,180 because what you're gonna do is in many cases, 1171 00:44:03,180 --> 00:44:05,092 you've got leadership in these calls, 1172 00:44:05,092 --> 00:44:06,900 along with the people that are supposed 1173 00:44:06,900 --> 00:44:08,573 to be managing the incident. 1174 00:44:08,573 --> 00:44:12,420 And you really don't want to put them in the position 1175 00:44:12,420 --> 00:44:14,781 of looking extremely bad in front of their leadership. 1176 00:44:14,781 --> 00:44:16,871 Or if you're doing it at the higher levels 1177 00:44:16,871 --> 00:44:20,610 where you've got directors that are being watched over 1178 00:44:20,610 --> 00:44:23,730 by board members or executive officers, 1179 00:44:23,730 --> 00:44:25,170 and you don't wanna put 'em in that level. 1180 00:44:25,170 --> 00:44:27,270 You wanna kind of help them be successful. 1181 00:44:27,270 --> 00:44:30,900 The goal of, for me, the goal of these is to learn, 1182 00:44:30,900 --> 00:44:33,720 identify gaps, takeaways. 1183 00:44:33,720 --> 00:44:35,540 You've all spoke to takeaways. 1184 00:44:35,540 --> 00:44:37,410 If you walk out of one and you're like, 1185 00:44:37,410 --> 00:44:38,940 oh yeah, we did great. 1186 00:44:38,940 --> 00:44:40,530 It wasn't scoped right. 1187 00:44:40,530 --> 00:44:41,363 Bottom line. 1188 00:44:41,363 --> 00:44:42,196 The scenario wasn't right, 1189 00:44:42,196 --> 00:44:43,530 and it wasn't scoped right. 1190 00:44:43,530 --> 00:44:46,530 Because the idea is to find something to work on. 1191 00:44:46,530 --> 00:44:47,850 'Cause none of us are perfect. 1192 00:44:47,850 --> 00:44:50,250 Believe me, we've all been in real incidents, 1193 00:44:50,250 --> 00:44:51,149 and we know that. 1194 00:44:51,149 --> 00:44:55,770 The other thing, I forget who was talking about it. 1195 00:44:55,770 --> 00:44:59,040 We were talking about the level of the incidents 1196 00:44:59,040 --> 00:45:01,590 and who all's involved. 1197 00:45:01,590 --> 00:45:04,740 And so if you're starting with someone 1198 00:45:04,740 --> 00:45:08,310 that's never done this before, you start at the high levels, 1199 00:45:08,310 --> 00:45:09,810 you work through process stuff. 1200 00:45:09,810 --> 00:45:11,340 Don't go deeply technical. 1201 00:45:11,340 --> 00:45:14,010 The scenario should have technical aspect to it, 1202 00:45:14,010 --> 00:45:15,360 and it should be relevant. 1203 00:45:15,360 --> 00:45:17,821 But that shouldn't be the key focus when you don't even know 1204 00:45:17,821 --> 00:45:19,980 who to call, and what to do, 1205 00:45:19,980 --> 00:45:22,053 and when to engage law enforcement. 1206 00:45:23,130 --> 00:45:26,460 My quick comment there is get your legal involved 1207 00:45:26,460 --> 00:45:28,410 before you call law enforcement. 1208 00:45:28,410 --> 00:45:30,683 That has to be a decision made by legal. 1209 00:45:30,683 --> 00:45:33,120 And if it's not in your playbook to do it that way, 1210 00:45:33,120 --> 00:45:34,830 you need to change your playbooks. 1211 00:45:34,830 --> 00:45:36,349 'Cause you need your legal involved. 1212 00:45:36,349 --> 00:45:38,837 Also data privacy, it should be involved 1213 00:45:38,837 --> 00:45:42,413 at step one throughout your entire IR process. 1214 00:45:42,413 --> 00:45:44,565 And they should be involved in your tabletops, 1215 00:45:44,565 --> 00:45:46,560 because they're gonna make decisions, 1216 00:45:46,560 --> 00:45:47,550 and they're gonna give guidance 1217 00:45:47,550 --> 00:45:50,670 that's very important to actions you take. 1218 00:45:50,670 --> 00:45:52,590 Another quick thing, sorry. 1219 00:45:52,590 --> 00:45:55,308 But I've sat here for a long time and not said a thing. 1220 00:45:55,308 --> 00:45:57,554 (panelists and audience laughing) 1221 00:45:57,554 --> 00:46:00,165 So I have done a lot of these 1222 00:46:00,165 --> 00:46:03,030 with a wide variety of clients. 1223 00:46:03,030 --> 00:46:05,130 And what I've found is that as you start 1224 00:46:05,130 --> 00:46:08,460 and then you start having subsequent engagements, 1225 00:46:08,460 --> 00:46:10,710 build a roadmap for that client, 1226 00:46:10,710 --> 00:46:12,969 let them understand at the beginning at the very first one, 1227 00:46:12,969 --> 00:46:15,180 this one's gonna be relatively high level. 1228 00:46:15,180 --> 00:46:16,509 It's not gonna be deeply technical. 1229 00:46:16,509 --> 00:46:18,780 We're gonna get the right policy people involved, 1230 00:46:18,780 --> 00:46:20,426 maybe some leadership involved. 1231 00:46:20,426 --> 00:46:22,660 Then as you progress, 1232 00:46:22,660 --> 00:46:27,090 you can have many table tops that are highly technical, 1233 00:46:27,090 --> 00:46:29,160 that are focused on a specific environment 1234 00:46:29,160 --> 00:46:32,250 or focused solely on your sock and their response. 1235 00:46:32,250 --> 00:46:35,276 And then you can build that out into bigger ones later 1236 00:46:35,276 --> 00:46:37,500 where you can actually split it into 1237 00:46:37,500 --> 00:46:39,960 where you've got leadership in one room on a call; 1238 00:46:39,960 --> 00:46:41,280 if it's physical, they're in a room, 1239 00:46:41,280 --> 00:46:42,307 if not, they're on a call. 1240 00:46:42,307 --> 00:46:43,925 You've got technical on another 1241 00:46:43,925 --> 00:46:46,680 and now, you're managing the way, typically, 1242 00:46:46,680 --> 00:46:48,740 most organizations are gonna manage it in real life 1243 00:46:48,740 --> 00:46:51,643 where you've got multiple different party engagements. 1244 00:46:51,643 --> 00:46:55,406 And they're not always sitting on every room, 1245 00:46:55,406 --> 00:46:57,780 but you don't do that step one. 1246 00:46:57,780 --> 00:46:59,053 That's a roadmap. 1247 00:46:59,053 --> 00:47:00,780 That one gentleman who was talking 1248 00:47:00,780 --> 00:47:03,535 about capability maturity, and I totally agree with that. 1249 00:47:03,535 --> 00:47:06,900 And I don't know that there's a model out there for it, 1250 00:47:06,900 --> 00:47:10,020 but I will tell you the clients that do this year 1251 00:47:10,020 --> 00:47:12,480 after year, after year, twice a year, 1252 00:47:12,480 --> 00:47:13,530 three or four times a year, 1253 00:47:13,530 --> 00:47:15,330 they have mini engagements. 1254 00:47:15,330 --> 00:47:17,779 They're far more mature in their incident response 1255 00:47:17,779 --> 00:47:20,010 than the ones that say we're just gonna do it 1256 00:47:20,010 --> 00:47:23,040 to tick a check box, just to check a box. 1257 00:47:23,040 --> 00:47:24,510 Just anyway, that was a lot. 1258 00:47:24,510 --> 00:47:25,343 I'm sorry. 1259 00:47:25,343 --> 00:47:26,691 - So you should be on this panel. 1260 00:47:26,691 --> 00:47:28,143 - That was great. 1261 00:47:29,073 --> 00:47:31,293 - Down there, you're gonna be next. 1262 00:47:31,293 --> 00:47:34,020 - [Audience Member] What we did was transparency. 1263 00:47:34,020 --> 00:47:36,521 We made sure that the participants 1264 00:47:36,521 --> 00:47:40,530 who were coming had the opportunity to understand 1265 00:47:40,530 --> 00:47:42,480 what was going to happen before they got there. 1266 00:47:42,480 --> 00:47:44,610 And then we had the luxury of time, 1267 00:47:44,610 --> 00:47:46,194 because these were multi-day events. 1268 00:47:46,194 --> 00:47:49,283 So we'd spend the first half day walking them 1269 00:47:49,283 --> 00:47:50,970 through the process. 1270 00:47:50,970 --> 00:47:52,260 This is the process we use. 1271 00:47:52,260 --> 00:47:53,940 This is what we're going to be doing. 1272 00:47:53,940 --> 00:47:57,903 After we found, we said, these are your threat actors. 1273 00:48:00,192 --> 00:48:03,830 And we did research open... 1274 00:48:04,800 --> 00:48:07,250 We did research on the systems we were attacking. 1275 00:48:08,340 --> 00:48:11,880 Basically, the theory was if it's unclassified, 1276 00:48:11,880 --> 00:48:14,100 it's available to the attacker. 1277 00:48:14,100 --> 00:48:17,550 And we would use that to come up with the attack scenarios, 1278 00:48:17,550 --> 00:48:18,660 and we would tell them beforehand, 1279 00:48:18,660 --> 00:48:21,120 these are the attack scenarios we're going to be using. 1280 00:48:21,120 --> 00:48:23,283 And this is how we're going to determine risk 1281 00:48:23,283 --> 00:48:25,853 when we walked them through the entire process, 1282 00:48:25,853 --> 00:48:28,860 all of this before lunch the first day. 1283 00:48:28,860 --> 00:48:31,920 And then when we got back from lunch, 1284 00:48:31,920 --> 00:48:33,770 we'd go ahead and start on scenarios. 1285 00:48:35,220 --> 00:48:36,053 - Thank you. 1286 00:48:36,053 --> 00:48:36,886 And I think we had the gentleman over here, 1287 00:48:36,886 --> 00:48:37,719 if you could use the microphone, 1288 00:48:37,719 --> 00:48:40,710 and then pass it to this man over here or to the right, 1289 00:48:40,710 --> 00:48:41,708 and then to the left. 1290 00:48:41,708 --> 00:48:44,375 (Safa laughing) 1291 00:48:45,806 --> 00:48:48,060 - [Audience Member] Hi. 1292 00:48:48,060 --> 00:48:53,060 I had been involved recently with a multi-agency tabletop, 1293 00:48:53,850 --> 00:48:56,070 and it's a fairly mature organization. 1294 00:48:56,070 --> 00:48:59,100 But it was a well scoped scenario 1295 00:48:59,100 --> 00:49:02,310 because it made all of senior management realize 1296 00:49:02,310 --> 00:49:03,780 that they hadn't actually read 1297 00:49:03,780 --> 00:49:05,910 those incident response plans. 1298 00:49:05,910 --> 00:49:07,890 And they sort of didn't know who to call. 1299 00:49:07,890 --> 00:49:10,620 And it was a little bit embarrassing, 1300 00:49:10,620 --> 00:49:13,590 and they felt a little bit like the Keystone Cops. 1301 00:49:13,590 --> 00:49:14,910 So the point of this 1302 00:49:14,910 --> 00:49:18,960 is that the important tabletop exercise is the next one. 1303 00:49:18,960 --> 00:49:22,350 Because I mean, like I said, it was a mature organization 1304 00:49:22,350 --> 00:49:24,254 and I've seen a whole lot more collaboration 1305 00:49:24,254 --> 00:49:25,750 between these organizations, 1306 00:49:25,750 --> 00:49:27,990 not just to prepare for incidents, 1307 00:49:27,990 --> 00:49:30,678 but they realized the skills that each other had. 1308 00:49:30,678 --> 00:49:33,630 And now they're working together in other ways, 1309 00:49:33,630 --> 00:49:35,940 not just for incident response. 1310 00:49:35,940 --> 00:49:40,110 Second comment, we have used in our technical TTXs, 1311 00:49:40,110 --> 00:49:43,410 we use a Dungeons and Dragons type card game 1312 00:49:43,410 --> 00:49:45,213 called Back Doors and Breaches. 1313 00:49:48,475 --> 00:49:52,020 So there's gotta be a way to put policy into that 1314 00:49:52,020 --> 00:49:54,090 and we get so much engagement from that. 1315 00:49:54,090 --> 00:49:55,110 It's awesome. 1316 00:49:55,110 --> 00:49:57,630 - If I can add one thing on the D&D, 1317 00:49:57,630 --> 00:50:00,330 my team actually really creatively put together 1318 00:50:00,330 --> 00:50:04,410 a D&D style game where folks navigate an incident 1319 00:50:04,410 --> 00:50:06,360 and are trying to pull the information out little by little. 1320 00:50:06,360 --> 00:50:08,400 But it was really, really exciting. 1321 00:50:08,400 --> 00:50:10,934 But if you wouldn't mind passing the microphone 1322 00:50:10,934 --> 00:50:12,017 to your left. 1323 00:50:13,311 --> 00:50:16,715 - How many people here played Dungeons and Dragons? 1324 00:50:16,715 --> 00:50:18,420 Incredible! 1325 00:50:18,420 --> 00:50:19,598 I knew there was an overlap. 1326 00:50:19,598 --> 00:50:20,431 (Winnona laughing) 1327 00:50:20,431 --> 00:50:21,350 - Half the room, yeah. 1328 00:50:22,210 --> 00:50:23,970 - [Audience Member] I just wanted 1329 00:50:23,970 --> 00:50:26,040 to give a kind of a free recommendation. 1330 00:50:26,040 --> 00:50:28,320 'Cause I know we were talking about immature players, 1331 00:50:28,320 --> 00:50:29,883 and I think for a lot of organizations 1332 00:50:29,883 --> 00:50:31,320 that are just kind of getting started, 1333 00:50:31,320 --> 00:50:32,670 don't have a lot of internal resources, 1334 00:50:32,670 --> 00:50:34,230 like an internal data privacy officer 1335 00:50:34,230 --> 00:50:36,575 or experienced legal counsel on inside. 1336 00:50:36,575 --> 00:50:38,366 It's been mentioned before, 1337 00:50:38,366 --> 00:50:40,020 but I just kinda wanna flip stomp of the room 1338 00:50:40,020 --> 00:50:42,750 that you should work with your insurance carrier. 1339 00:50:42,750 --> 00:50:46,950 Your insurance carrier has a board of outside council 1340 00:50:46,950 --> 00:50:47,783 that you should tap. 1341 00:50:47,783 --> 00:50:49,981 And I think everyone's very familiar with vendor assessments 1342 00:50:49,981 --> 00:50:52,080 for their technical vendors. 1343 00:50:52,080 --> 00:50:53,730 I think something that's kind of overlooked 1344 00:50:53,730 --> 00:50:55,980 for new players is vendor assessments 1345 00:50:55,980 --> 00:50:57,300 for your outside council. 1346 00:50:57,300 --> 00:50:58,680 If you really do wanna get prepared 1347 00:50:58,680 --> 00:51:00,930 for dealing with a large scale breach 1348 00:51:00,930 --> 00:51:03,587 and your plan is just to reach out to outside council, 1349 00:51:03,587 --> 00:51:05,880 I've seen so many clients that are like, 1350 00:51:05,880 --> 00:51:07,170 yeah, we'll reach out to outside council, 1351 00:51:07,170 --> 00:51:08,267 and they time it right. 1352 00:51:08,267 --> 00:51:10,590 They give a right threshold for reaching out, 1353 00:51:10,590 --> 00:51:11,730 but they don't know who. 1354 00:51:11,730 --> 00:51:13,050 But on the other hand, they'll know, 1355 00:51:13,050 --> 00:51:15,450 oh, if we need to negotiate, we'll tap Covewear. 1356 00:51:16,320 --> 00:51:17,880 Why do you know that you need to tap Covewear, 1357 00:51:17,880 --> 00:51:20,190 but you don't know who your outside council is? 1358 00:51:20,190 --> 00:51:22,500 It's kind of like your other big player, right? 1359 00:51:22,500 --> 00:51:24,543 Probably bigger for most organizations. 1360 00:51:25,622 --> 00:51:29,738 I also wanted to ask, I think for like on another thread, 1361 00:51:29,738 --> 00:51:33,356 for larger organizations, 1362 00:51:33,356 --> 00:51:36,443 how have people successfully scoped 1363 00:51:36,443 --> 00:51:41,244 for these large scale conglomerates or giant tech companies 1364 00:51:41,244 --> 00:51:44,820 and figured out a way to scope the exercise 1365 00:51:44,820 --> 00:51:48,900 so that it's not Microsoft blows up and disappears, 1366 00:51:48,900 --> 00:51:50,190 but is useful? 1367 00:51:50,190 --> 00:51:51,480 But they're also gigantic, 1368 00:51:51,480 --> 00:51:52,770 so it's difficult to make that scope. 1369 00:51:52,770 --> 00:51:55,020 I'm wondering how people tackle that problem. 1370 00:51:57,720 --> 00:51:58,553 - Please. 1371 00:51:58,553 --> 00:51:59,820 - [Audience Member] It's more kind of use case 1372 00:51:59,820 --> 00:52:02,430 to add to the, in terms of... 1373 00:52:02,430 --> 00:52:04,080 And this ties back to your question 1374 00:52:04,080 --> 00:52:07,020 on can US national security and that sort of thing; 1375 00:52:07,020 --> 00:52:09,840 thinking about the ecosystem in a given country 1376 00:52:09,840 --> 00:52:11,453 where we have the US embassy, 1377 00:52:11,453 --> 00:52:16,260 the USAID mission, all of our NGOs and partners 1378 00:52:16,260 --> 00:52:17,610 that we fund that are collecting data 1379 00:52:17,610 --> 00:52:20,580 and doing work, all kind of under the overall guise 1380 00:52:20,580 --> 00:52:22,979 of US national security and objectives. 1381 00:52:22,979 --> 00:52:27,150 Thinking about how tabletops include 1382 00:52:27,150 --> 00:52:30,210 or don't include all of those diverse actors, 1383 00:52:30,210 --> 00:52:32,400 or if you just proxy them. 1384 00:52:32,400 --> 00:52:34,850 And so that's really interesting to me to see how 1385 00:52:36,337 --> 00:52:38,587 to either run a more complex one or simplify. 1386 00:52:39,486 --> 00:52:44,486 - So this is less from a policy oriented perspective 1387 00:52:44,820 --> 00:52:46,680 and more towards red team engagements, right? 1388 00:52:46,680 --> 00:52:49,500 But just to answer your question directly 1389 00:52:49,500 --> 00:52:50,974 in terms of Microsoft blowing up, 1390 00:52:50,974 --> 00:52:54,360 (laughs) I did front intelligence 1391 00:52:54,360 --> 00:52:56,460 at a fairly large tech firm for a little bit. 1392 00:52:56,460 --> 00:52:59,370 And when we were running, not tabletops, 1393 00:52:59,370 --> 00:53:00,224 but red team engagements, 1394 00:53:00,224 --> 00:53:04,470 usually the red team gets carte blanche. 1395 00:53:04,470 --> 00:53:06,210 But they'll scope it, and they'll consult 1396 00:53:06,210 --> 00:53:08,670 with the organizations that they're planning on targeting. 1397 00:53:08,670 --> 00:53:11,400 And so I think it's truly a conversation 1398 00:53:11,400 --> 00:53:13,800 that you want to have with your client and just say, 1399 00:53:13,800 --> 00:53:16,440 listen, how is everything in scope? 1400 00:53:16,440 --> 00:53:20,373 If that's the case, still choosing a small subset. 1401 00:53:21,750 --> 00:53:23,670 At least from a designing perspective, 1402 00:53:23,670 --> 00:53:26,460 it gives you the creativity to be able to work 1403 00:53:26,460 --> 00:53:27,543 with what you have. 1404 00:53:28,410 --> 00:53:30,879 But it truly depends on, I guess, 1405 00:53:30,879 --> 00:53:33,390 going back to the maturity model of that organization 1406 00:53:33,390 --> 00:53:36,738 and how closely knit those disparate teams are already. 1407 00:53:36,738 --> 00:53:39,353 - On the, oh, go ahead. 1408 00:53:39,353 --> 00:53:40,803 Someone else has a question. 1409 00:53:41,658 --> 00:53:43,320 On the conglomerate thing, 1410 00:53:43,320 --> 00:53:45,090 I think it gets really, really tricky 1411 00:53:45,090 --> 00:53:46,233 to run tabletop exercises 1412 00:53:46,233 --> 00:53:48,208 when you have like an umbrella organization, 1413 00:53:48,208 --> 00:53:49,543 a bunch of small organizations, 1414 00:53:49,543 --> 00:53:51,359 there's a few different ways to do it. 1415 00:53:51,359 --> 00:53:54,833 We've done one where we run management exercises 1416 00:53:54,833 --> 00:53:57,000 for both the umbrella organization 1417 00:53:57,000 --> 00:53:59,490 in each of the subsidiaries 1418 00:53:59,490 --> 00:54:01,350 and ask each of the subsidiaries, 1419 00:54:01,350 --> 00:54:03,480 what are you expecting from your parent company? 1420 00:54:03,480 --> 00:54:05,400 And what do you think you're supposed to be doing? 1421 00:54:05,400 --> 00:54:08,040 And we usually get totally different answers from all them. 1422 00:54:08,040 --> 00:54:09,870 And then we run one with the umbrella organization 1423 00:54:09,870 --> 00:54:12,300 and we're like, well, what sort of information's 1424 00:54:12,300 --> 00:54:14,040 coming down and what do you expect to come back up? 1425 00:54:14,040 --> 00:54:16,950 And then what we can do is sort of map all of those 1426 00:54:16,950 --> 00:54:18,906 and produce an overall report that says 1427 00:54:18,906 --> 00:54:21,322 here's where the communication is gonna break 1428 00:54:21,322 --> 00:54:23,010 during an exercise. 1429 00:54:23,010 --> 00:54:25,950 And here the subsidiaries that are expecting you 1430 00:54:25,950 --> 00:54:27,420 to do all the work and are gonna need your help 1431 00:54:27,420 --> 00:54:28,530 and might need more resources 1432 00:54:28,530 --> 00:54:29,520 than they're currently being allotted. 1433 00:54:29,520 --> 00:54:31,020 So it does get very complicated 1434 00:54:31,020 --> 00:54:33,060 when you're doing maybe 10 exercises 1435 00:54:33,060 --> 00:54:34,350 that are all sort of linked to each other, 1436 00:54:34,350 --> 00:54:35,970 but it can be done. 1437 00:54:35,970 --> 00:54:38,280 - I see two fingers here and I'm gonna ask a question 1438 00:54:38,280 --> 00:54:41,201 that might shift gears a tiny bit, but defer to you all. 1439 00:54:41,201 --> 00:54:43,560 - [Audience Member] Yeah, so the other aspect 1440 00:54:43,560 --> 00:54:46,620 is taking a look at companies that grow by mergers 1441 00:54:46,620 --> 00:54:49,860 and acquisitions, 'cause there's a lot of inherent risk 1442 00:54:49,860 --> 00:54:51,210 with the company you're bringing on 1443 00:54:51,210 --> 00:54:53,938 and their risk profile and what their capabilities are, 1444 00:54:53,938 --> 00:54:55,890 and who they may be connected to. 1445 00:54:55,890 --> 00:54:57,863 So we see this a lot out in industry 1446 00:54:57,863 --> 00:55:00,120 where companies are growing by M&A. 1447 00:55:00,120 --> 00:55:02,400 But from an attacker perspective, 1448 00:55:02,400 --> 00:55:03,570 that's a juicy target. 1449 00:55:03,570 --> 00:55:05,160 For simple fact, it's gonna be a lot 1450 00:55:05,160 --> 00:55:09,060 of disjointed companies together that may 1451 00:55:09,060 --> 00:55:10,950 or may not have their act together. 1452 00:55:10,950 --> 00:55:12,930 In financial services, there's two things 1453 00:55:12,930 --> 00:55:14,520 that we've always concentrated on. 1454 00:55:14,520 --> 00:55:16,241 Anything can effect the GL, 1455 00:55:16,241 --> 00:55:19,135 anything can stop processing card information, 1456 00:55:19,135 --> 00:55:22,738 and anything can stop major financial transactions 1457 00:55:22,738 --> 00:55:25,380 within the financial services organization 1458 00:55:25,380 --> 00:55:28,230 as areas for scoping. 1459 00:55:28,230 --> 00:55:30,780 Because if you can't get those three things right, 1460 00:55:30,780 --> 00:55:33,783 that's making your financial service organization run, 1461 00:55:33,783 --> 00:55:36,542 or the question of how do you make your money, 1462 00:55:36,542 --> 00:55:39,600 those are the areas that we've concentrated on 1463 00:55:39,600 --> 00:55:40,980 from a scope aspect. 1464 00:55:40,980 --> 00:55:44,310 And then we've seen where there's been breakdowns 1465 00:55:44,310 --> 00:55:46,980 in outlying or supporting functions 1466 00:55:46,980 --> 00:55:50,310 or supporting vendor organizations that integrate 1467 00:55:50,310 --> 00:55:51,480 with that supporting function. 1468 00:55:51,480 --> 00:55:54,990 So you start to get into, okay, vendor risk management, 1469 00:55:54,990 --> 00:55:58,394 either as a force multiplier to your organization 1470 00:55:58,394 --> 00:56:01,770 or a detriment in your IR response, 1471 00:56:01,770 --> 00:56:02,670 and things of that nature. 1472 00:56:02,670 --> 00:56:04,860 So that's how we've kind of focused 1473 00:56:04,860 --> 00:56:07,860 in the financial services from my experience. 1474 00:56:07,860 --> 00:56:09,180 Again, things may vary, 1475 00:56:09,180 --> 00:56:12,270 but that's just one area to scope aspect. 1476 00:56:12,270 --> 00:56:13,103 Thanks. 1477 00:56:13,103 --> 00:56:15,209 - I wanna tie that into kind of the conversations, 1478 00:56:15,209 --> 00:56:16,980 like the subsidiary in the M&A 1479 00:56:16,980 --> 00:56:18,510 with kind of the outside counselor, 1480 00:56:18,510 --> 00:56:19,920 outside vendor recommendation. 1481 00:56:19,920 --> 00:56:22,170 I'm dying to see if anyone has ever done this, 1482 00:56:22,170 --> 00:56:26,760 a incident scenario in which the attacker has started 1483 00:56:26,760 --> 00:56:29,105 from the outside law firm and moved inwards. 1484 00:56:29,105 --> 00:56:32,058 'Cause I think that that would be kind of a beautiful tie in 1485 00:56:32,058 --> 00:56:34,320 to like, oh, we'll reach out to outside counsel 1486 00:56:34,320 --> 00:56:36,874 when they're the ones that cause this whole thing, but. 1487 00:56:36,874 --> 00:56:38,105 (Winnona laughing) 1488 00:56:38,105 --> 00:56:38,938 - Anybody? 1489 00:56:40,438 --> 00:56:43,503 - [Audience Member] So yeah, I just had a question. 1490 00:56:44,430 --> 00:56:46,980 So a lot of the conversation has been about how 1491 00:56:46,980 --> 00:56:51,346 if you're scoping a tabletop exercise that you succeed in, 1492 00:56:51,346 --> 00:56:53,594 it's the wrong scope. 1493 00:56:53,594 --> 00:56:56,223 In a lot of self-defense courses 1494 00:56:56,223 --> 00:56:58,736 or that kind of training curriculum, 1495 00:56:58,736 --> 00:57:01,140 one of the concepts is you wanna lead the student 1496 00:57:01,140 --> 00:57:02,310 with the success at the end. 1497 00:57:02,310 --> 00:57:04,050 Because you want them to have the self confidence 1498 00:57:04,050 --> 00:57:04,980 that they're able to do this. 1499 00:57:04,980 --> 00:57:05,813 We've got this. 1500 00:57:05,813 --> 00:57:08,280 We're able to handle these sort of things. 1501 00:57:08,280 --> 00:57:10,380 Is that something that we should be incorporating 1502 00:57:10,380 --> 00:57:12,000 into our tabletops as well, 1503 00:57:12,000 --> 00:57:15,009 that maybe we give them an attaboy at the end 1504 00:57:15,009 --> 00:57:16,800 to give them that self confidence? 1505 00:57:16,800 --> 00:57:20,883 Or are we just really focusing on the negatives? 1506 00:57:23,370 --> 00:57:25,590 - I think delivering feedback is such an important part 1507 00:57:25,590 --> 00:57:27,423 of tabletop exercises. 1508 00:57:27,423 --> 00:57:29,910 I think starting with what was done well 1509 00:57:29,910 --> 00:57:31,350 is always important. 1510 00:57:31,350 --> 00:57:35,070 But what we like to do our practice is ask them 1511 00:57:35,070 --> 00:57:37,680 to say what they think they could do better first, 1512 00:57:37,680 --> 00:57:39,570 because it just sounds a lot better coming from... 1513 00:57:39,570 --> 00:57:42,540 They're like, we really didn't use our IR plan, 1514 00:57:42,540 --> 00:57:45,723 than us saying you did not use your IR plan, bad. 1515 00:57:45,723 --> 00:57:47,981 It makes a big difference, because then, 1516 00:57:47,981 --> 00:57:50,370 they'll all sort talk about it themselves 1517 00:57:50,370 --> 00:57:51,810 and we'll come in at the end and we'll say, 1518 00:57:51,810 --> 00:57:52,909 we totally agree with this. 1519 00:57:52,909 --> 00:57:54,221 We'll add a couple things. 1520 00:57:54,221 --> 00:57:57,030 And then folks walk away feeling like they've taught 1521 00:57:57,030 --> 00:57:59,730 each other something versus us kind of bringing it down 1522 00:57:59,730 --> 00:58:01,650 upon them. 1523 00:58:01,650 --> 00:58:03,765 It's just sort of received a lot better, 1524 00:58:03,765 --> 00:58:06,630 but even with the sort of reports that we're giving out, 1525 00:58:06,630 --> 00:58:08,070 it's always, here's what you did well. 1526 00:58:08,070 --> 00:58:09,060 Here's what you could work on. 1527 00:58:09,060 --> 00:58:12,390 Because if you just tell someone everything they did wrong, 1528 00:58:12,390 --> 00:58:13,800 they just feel like a failure. 1529 00:58:13,800 --> 00:58:14,633 So you're right. 1530 00:58:14,633 --> 00:58:16,837 You don't want them to feel like they owned the exercise, 1531 00:58:16,837 --> 00:58:21,120 but they definitely need to know what they're good at. 1532 00:58:21,120 --> 00:58:23,220 - Or they shut down, and the next time, 1533 00:58:23,220 --> 00:58:25,265 that area is out of scope for the next tabletop. 1534 00:58:25,265 --> 00:58:27,750 (panelists and audience laughing) 1535 00:58:27,750 --> 00:58:30,210 - I can jump in really quickly just 'cause the feedback 1536 00:58:30,210 --> 00:58:32,280 and evaluation part is really interesting, 1537 00:58:32,280 --> 00:58:34,080 especially because we work with students. 1538 00:58:34,080 --> 00:58:36,540 So you wanna be extra sensitive to the fact that, 1539 00:58:36,540 --> 00:58:37,560 hey, you're new here. 1540 00:58:37,560 --> 00:58:38,618 It's okay to make mistakes. 1541 00:58:38,618 --> 00:58:40,196 This is how you improve for next time. 1542 00:58:40,196 --> 00:58:43,290 And we have a rubric that we use for all the briefings 1543 00:58:43,290 --> 00:58:46,410 that they give, evaluating, one, their presentation, 1544 00:58:46,410 --> 00:58:48,565 the policy recommendations they develop, 1545 00:58:48,565 --> 00:58:51,360 how they navigated working with four different people, 1546 00:58:51,360 --> 00:58:52,794 three to four different people on a team. 1547 00:58:52,794 --> 00:58:55,470 I think one thing that's been, I think, 1548 00:58:55,470 --> 00:58:57,090 challenging for us is, one, making sure that 1549 00:58:57,090 --> 00:58:59,340 when they do present that, they give that to a series 1550 00:58:59,340 --> 00:59:01,170 of judges who are from industry and government 1551 00:59:01,170 --> 00:59:02,880 who are practitioners. 1552 00:59:02,880 --> 00:59:05,160 We wanna make sure that that feedback is valuable to them, 1553 00:59:05,160 --> 00:59:07,260 but it's also constructive. 1554 00:59:07,260 --> 00:59:08,490 And that you deliver it in a way 1555 00:59:08,490 --> 00:59:09,660 that focuses on the positives, 1556 00:59:09,660 --> 00:59:11,590 but also identify as areas for growth. 1557 00:59:11,590 --> 00:59:13,500 And the growth part is interesting 1558 00:59:13,500 --> 00:59:15,450 because you wanna make sure that students use that 1559 00:59:15,450 --> 00:59:17,160 for the next year when they do compete again, 1560 00:59:17,160 --> 00:59:19,050 so they can continuously get better. 1561 00:59:19,050 --> 00:59:20,280 But one thing that we've had fun with 1562 00:59:20,280 --> 00:59:21,300 in some events is like, 1563 00:59:21,300 --> 00:59:24,120 we'll let some students redo on day two. 1564 00:59:24,120 --> 00:59:25,980 It's not competitive, so they can't get points for it. 1565 00:59:25,980 --> 00:59:27,743 But you can take the feedback you got on day one, 1566 00:59:27,743 --> 00:59:29,670 redo your briefing on day two, 1567 00:59:29,670 --> 00:59:31,110 and get feedback a second time. 1568 00:59:31,110 --> 00:59:33,736 So how could you redo this again in a safe setting, 1569 00:59:33,736 --> 00:59:36,780 using that feedback that you got from people. 1570 00:59:36,780 --> 00:59:39,164 But I did see some, please. 1571 00:59:39,164 --> 00:59:41,842 - [Audience Member] Parallel with self-defense courses, 1572 00:59:41,842 --> 00:59:44,190 right, and those sorts of things. 1573 00:59:44,190 --> 00:59:45,660 It's basic instruction. 1574 00:59:45,660 --> 00:59:47,595 It's protect your head. 1575 00:59:47,595 --> 00:59:51,870 And how does that compare to a much more complex tabletop 1576 00:59:51,870 --> 00:59:54,150 where the feedback or the solutions 1577 00:59:54,150 --> 00:59:56,257 that you're getting might be 180 pages 1578 00:59:56,257 --> 00:59:58,082 of this is what needs to happen in case of a crisis. 1579 00:59:58,082 --> 01:00:01,756 And so how do you, not water it down, 1580 01:00:01,756 --> 01:00:05,130 but right size it in terms of providing the people 1581 01:00:05,130 --> 01:00:07,362 who need the right steps and information, 1582 01:00:07,362 --> 01:00:10,260 give them the stories and then the tools that they need 1583 01:00:10,260 --> 01:00:11,829 that are actually actionable? 1584 01:00:11,829 --> 01:00:15,030 - Too much feedback is as good as no feedback 1585 01:00:15,030 --> 01:00:17,353 because you reach a point where folks just can't do 1586 01:00:17,353 --> 01:00:18,706 that many things. 1587 01:00:18,706 --> 01:00:22,050 So you wanna pick out the things that you think they need 1588 01:00:22,050 --> 01:00:22,883 to work on the most. 1589 01:00:22,883 --> 01:00:24,802 I usually pick formal processes. 1590 01:00:24,802 --> 01:00:27,420 What can you write down so that you'll have it next time 1591 01:00:27,420 --> 01:00:29,220 so that you don't have to think through it the way 1592 01:00:29,220 --> 01:00:30,600 that you had to do in this exercise? 1593 01:00:30,600 --> 01:00:31,710 Because it's gonna waste time 1594 01:00:31,710 --> 01:00:33,200 and you don't have as much time. 1595 01:00:33,200 --> 01:00:35,700 It's gonna take you 10 times as long to get all the people 1596 01:00:35,700 --> 01:00:37,860 in the room that you think it's gonna take. 1597 01:00:37,860 --> 01:00:39,420 So yeah, I like to keep it 1598 01:00:39,420 --> 01:00:42,393 to maybe less than 15 things they need to fix. 1599 01:00:43,740 --> 01:00:45,300 And at that point, 1600 01:00:45,300 --> 01:00:47,010 if you have more than 15 things to fix, 1601 01:00:47,010 --> 01:00:48,600 you sort of need to fix everything. 1602 01:00:48,600 --> 01:00:50,940 And so you might as well just keep it to the things 1603 01:00:50,940 --> 01:00:52,014 that they need to do first. 1604 01:00:52,014 --> 01:00:54,300 And so the next time they come around the tabletop, 1605 01:00:54,300 --> 01:00:55,500 you can sort drill down deeper 1606 01:00:55,500 --> 01:00:57,510 into the more complex processes 1607 01:00:57,510 --> 01:00:58,571 that they could figure out better, 1608 01:00:58,571 --> 01:01:01,099 but you wanna hit the things that they can do first 1609 01:01:01,099 --> 01:01:02,223 though, that are easy. 1610 01:01:03,060 --> 01:01:04,290 - Also just to add on that. 1611 01:01:04,290 --> 01:01:07,740 I mean, when you're coming back with pieces of feedback 1612 01:01:07,740 --> 01:01:09,120 and trying to narrow that list down, 1613 01:01:09,120 --> 01:01:10,238 especially if it's a larger list, 1614 01:01:10,238 --> 01:01:12,900 at least for me, I focus on feasibility 1615 01:01:12,900 --> 01:01:13,770 within the organization. 1616 01:01:13,770 --> 01:01:16,950 And when I say feasibility, I don't just mean technical. 1617 01:01:16,950 --> 01:01:18,648 I mean political, process wise. 1618 01:01:18,648 --> 01:01:21,658 If you're going to tell someone to fix their IT systems, 1619 01:01:21,658 --> 01:01:25,320 but you're a big company with several mergers 1620 01:01:25,320 --> 01:01:28,350 and acquisitions, and you're like at this part of the org, 1621 01:01:28,350 --> 01:01:29,880 but the people that handle it is this part. 1622 01:01:29,880 --> 01:01:30,780 It makes no sense. 1623 01:01:31,680 --> 01:01:33,330 And then also focusing on, 1624 01:01:33,330 --> 01:01:37,050 when you're looking at your principal audience 1625 01:01:37,050 --> 01:01:39,510 or your principal feedback receiver, 1626 01:01:39,510 --> 01:01:42,420 tailoring that message to more of a high level; 1627 01:01:42,420 --> 01:01:43,410 if it's an executive 1628 01:01:43,410 --> 01:01:45,438 versus here's all of the different lists 1629 01:01:45,438 --> 01:01:47,070 of things you need to do if you're more 1630 01:01:47,070 --> 01:01:49,140 of a technical practitioner. 1631 01:01:49,140 --> 01:01:50,603 - I think I saw a hand raised over here. 1632 01:01:50,603 --> 01:01:53,700 - [Audience Member] So one of the things I'd like 1633 01:01:53,700 --> 01:01:55,014 to say too, is that for anybody 1634 01:01:55,014 --> 01:01:58,750 who's doing tabletop exercises is to make sure that, 1635 01:01:58,750 --> 01:02:00,120 Winn, you said it a couple times, 1636 01:02:00,120 --> 01:02:02,694 make sure that third party council is involved. 1637 01:02:02,694 --> 01:02:06,299 Because what I found in a few of the tabletop exercises 1638 01:02:06,299 --> 01:02:09,863 that I've done is that the company has 1639 01:02:10,860 --> 01:02:14,970 a incident response retainer with one organization. 1640 01:02:14,970 --> 01:02:15,990 And that organization, 1641 01:02:15,990 --> 01:02:17,613 once third party council gets involved 1642 01:02:17,613 --> 01:02:20,130 and cyber insurance gets involved, 1643 01:02:20,130 --> 01:02:23,250 is not approved by the cyber insurer. 1644 01:02:23,250 --> 01:02:25,794 So now it's in a very interesting position, 1645 01:02:25,794 --> 01:02:28,050 and we actually had to do this 1646 01:02:28,050 --> 01:02:30,240 as a live fire tabletop exercise, 1647 01:02:30,240 --> 01:02:33,418 which if anybody's ever done, I don't recommend ever doing. 1648 01:02:33,418 --> 01:02:35,730 (panelists and audience laughing) 1649 01:02:35,730 --> 01:02:38,760 And they had to work it out in real time. 1650 01:02:38,760 --> 01:02:42,300 And there was conversations that had never happened before 1651 01:02:42,300 --> 01:02:46,260 about, okay, well we have this company engaged already. 1652 01:02:46,260 --> 01:02:49,230 They're doing analysis, but now our cyber insurers saying, 1653 01:02:49,230 --> 01:02:51,540 we have to go with this company over here. 1654 01:02:51,540 --> 01:02:53,130 What does that handoff look like? 1655 01:02:53,130 --> 01:02:55,740 And then what does the future plan look like? 1656 01:02:55,740 --> 01:03:00,740 Because obviously, the company has one response retainer 1657 01:03:00,840 --> 01:03:02,310 that they choose. 1658 01:03:02,310 --> 01:03:04,260 They have one of choice and now they have to work 1659 01:03:04,260 --> 01:03:06,540 into the process in the future to get them approved 1660 01:03:06,540 --> 01:03:07,373 and that... 1661 01:03:07,373 --> 01:03:09,870 So highly recommend getting third party council, 1662 01:03:09,870 --> 01:03:11,133 cyber insurance involved. 1663 01:03:13,707 --> 01:03:15,173 - Any other questions from you two? 1664 01:03:15,173 --> 01:03:17,130 I'd love to give you guys a chance 1665 01:03:17,130 --> 01:03:19,650 before I throw a wrench in with some fun questions. 1666 01:03:19,650 --> 01:03:20,610 - Trey's got a question. 1667 01:03:20,610 --> 01:03:21,543 - Yeah, please. 1668 01:03:22,790 --> 01:03:24,370 - [Trey] So I'm a little bit cur- 1669 01:03:24,370 --> 01:03:25,980 We've talked a lot about the stakes 1670 01:03:25,980 --> 01:03:27,270 and mechanics of competitions, 1671 01:03:27,270 --> 01:03:28,868 a lot about adversaries and victims. 1672 01:03:28,868 --> 01:03:30,765 (indistinct) 1673 01:03:30,765 --> 01:03:31,730 - (speaking faintly) Mic, sorry. 1674 01:03:31,730 --> 01:03:32,880 - [Trey] So for you all, 1675 01:03:32,880 --> 01:03:35,160 we've talked a lot about the mechanics of the games 1676 01:03:35,160 --> 01:03:36,720 and the adversaries. 1677 01:03:36,720 --> 01:03:38,010 What have you found works best 1678 01:03:38,010 --> 01:03:41,910 to build the non-player characters, the intermediaries? 1679 01:03:41,910 --> 01:03:43,260 'Cause so often, it's Cyber 9/12. 1680 01:03:43,260 --> 01:03:44,220 One of the things we run into 1681 01:03:44,220 --> 01:03:46,320 is students have trouble interacting 1682 01:03:46,320 --> 01:03:48,120 with the third party company, 1683 01:03:48,120 --> 01:03:50,763 the other insurer that'd be the downstream vendor. 1684 01:03:50,763 --> 01:03:52,380 And so making them credible 1685 01:03:52,380 --> 01:03:54,690 and rich characters is really important 1686 01:03:54,690 --> 01:03:55,560 to the outcome of the scenario. 1687 01:03:55,560 --> 01:03:57,477 I'm just curious for all of you or for folks in the room, 1688 01:03:57,477 --> 01:03:59,070 what have you found works best 1689 01:03:59,070 --> 01:04:00,796 or what are trip falls to avoid 1690 01:04:00,796 --> 01:04:02,973 in building those intermediary characters? 1691 01:04:05,378 --> 01:04:08,669 - I mean, I don't think in corporate tabletop exercises 1692 01:04:08,669 --> 01:04:10,083 versus the Cyber 9/12, 1693 01:04:10,083 --> 01:04:11,907 they really have the same sort of rich characters. 1694 01:04:11,907 --> 01:04:12,740 But one of the things we'll do 1695 01:04:12,740 --> 01:04:15,240 is that if we're doing a combined technical tabletop 1696 01:04:15,240 --> 01:04:18,030 and management tabletop, do the technical one first, 1697 01:04:18,030 --> 01:04:19,137 see what sorts of information 1698 01:04:19,137 --> 01:04:21,720 the technical team would actually bring upwards. 1699 01:04:21,720 --> 01:04:23,370 And then in the management tabletop, 1700 01:04:23,370 --> 01:04:25,260 we will pretend to be the technical team. 1701 01:04:25,260 --> 01:04:28,890 And we'll say the technical team is saying this, this, 1702 01:04:28,890 --> 01:04:30,480 and this, and they're asking this from you. 1703 01:04:30,480 --> 01:04:33,000 So that we're actually sort of creating that bridge 1704 01:04:33,000 --> 01:04:35,445 between the executive management, the security team. 1705 01:04:35,445 --> 01:04:38,100 And if they're saying, well, we'd expect this from them. 1706 01:04:38,100 --> 01:04:40,380 And then we learn that there's gonna be sort of a break 1707 01:04:40,380 --> 01:04:41,213 in that communication. 1708 01:04:41,213 --> 01:04:42,480 We don't have really separate characters, 1709 01:04:42,480 --> 01:04:44,580 but we sort of play the intermediary 1710 01:04:44,580 --> 01:04:47,610 so that we can ask the right questions there. 1711 01:04:47,610 --> 01:04:52,610 - Yeah, I don't think I've built an NPC for my war games 1712 01:04:52,830 --> 01:04:55,623 aside from actual Dungeon and Dragons games 1713 01:04:55,623 --> 01:04:56,883 that is a hobby. 1714 01:04:58,750 --> 01:05:01,350 But I think I'll take that question 1715 01:05:01,350 --> 01:05:04,920 and maybe pivot it into a discussion about technical realism 1716 01:05:04,920 --> 01:05:09,090 in policy war games. 1717 01:05:09,090 --> 01:05:14,040 And there have been times where, especially when, 1718 01:05:14,040 --> 01:05:15,870 I guess going back to Sarah's point, 1719 01:05:15,870 --> 01:05:17,070 one of the ways to do that is to have 1720 01:05:17,070 --> 01:05:19,170 that technical war game first. 1721 01:05:19,170 --> 01:05:23,400 But making sure if you don't have that time, 1722 01:05:23,400 --> 01:05:27,630 and I imagine, we're at DEF CON, talking technical 1723 01:05:27,630 --> 01:05:32,630 to a war gaming panel is not going to be a surprise. 1724 01:05:33,090 --> 01:05:35,970 But making sure that either if you're not that expert, 1725 01:05:35,970 --> 01:05:38,220 making sure that you're checking that all of the war games 1726 01:05:38,220 --> 01:05:39,648 from that perspective are realistic, 1727 01:05:39,648 --> 01:05:42,595 actionable, and current. 1728 01:05:42,595 --> 01:05:45,270 I think making sure that you have someone there 1729 01:05:45,270 --> 01:05:47,243 that will do that checking is important. 1730 01:05:48,360 --> 01:05:49,193 - Rapid fire? 1731 01:05:49,193 --> 01:05:50,280 Answer some questions? 1732 01:05:50,280 --> 01:05:52,500 - Yeah, I saw a wave. 1733 01:05:52,500 --> 01:05:54,455 I don't know if it was a hand raise or a wave. 1734 01:05:54,455 --> 01:05:56,037 - [Audience Member] I just wanted to chime in 1735 01:05:56,037 --> 01:05:59,702 and say that if you get your IR provider to do the war game, 1736 01:05:59,702 --> 01:06:01,590 they work with the third party. 1737 01:06:01,590 --> 01:06:05,160 So it's easier to create those kind of third party NPCs 1738 01:06:05,160 --> 01:06:07,653 and simulate those interactions. 1739 01:06:08,495 --> 01:06:09,423 - Excellent. 1740 01:06:10,417 --> 01:06:12,777 One thing I wanted to ask folks in the room, 1741 01:06:12,777 --> 01:06:15,120 was there ever a time that you 1742 01:06:15,120 --> 01:06:17,370 or somebody that you were working with took an action 1743 01:06:17,370 --> 01:06:19,500 during a war game or a simulation 1744 01:06:19,500 --> 01:06:21,660 that had really bad consequences? 1745 01:06:21,660 --> 01:06:24,261 And how did you navigate that with them? 1746 01:06:24,261 --> 01:06:27,060 Or did you just stand back and watch the chaos happen 1747 01:06:27,060 --> 01:06:27,942 and just kind of smile? 1748 01:06:27,942 --> 01:06:29,462 - [Audience Member] More like that. 1749 01:06:29,462 --> 01:06:31,125 - You asking us or are you asking the room? 1750 01:06:31,125 --> 01:06:33,300 - I'd love for you guys to go first, and then- 1751 01:06:33,300 --> 01:06:35,716 - You're asking for on the record horror stories, Safa? 1752 01:06:35,716 --> 01:06:36,581 (Winnona laughing) 1753 01:06:36,581 --> 01:06:37,531 - I mean, no names. 1754 01:06:40,039 --> 01:06:44,460 - I guess the one that I've had goes back 1755 01:06:44,460 --> 01:06:46,080 to setting expectations. 1756 01:06:46,080 --> 01:06:47,730 And I think that one of the ways 1757 01:06:47,730 --> 01:06:50,790 that we did not set expectations the right way 1758 01:06:50,790 --> 01:06:53,490 when doing a tabletop with this one set 1759 01:06:53,490 --> 01:06:57,090 of groups was just in terms of, these were two organizations 1760 01:06:57,090 --> 01:06:59,520 that had never really done a tabletop together. 1761 01:06:59,520 --> 01:07:02,516 And due to that, due to also cultural differences, 1762 01:07:02,516 --> 01:07:06,300 we came in with the scenario, one team really went in 1763 01:07:06,300 --> 01:07:08,430 with the scenario and was very forthright. 1764 01:07:08,430 --> 01:07:10,020 And the other team shut it down. 1765 01:07:10,020 --> 01:07:12,072 And we're like, okay, what are you going to do? 1766 01:07:12,072 --> 01:07:13,890 Oh, we'll just figure it out when we get there, 1767 01:07:13,890 --> 01:07:16,954 or we will engage with our third parties. 1768 01:07:16,954 --> 01:07:19,502 And then making sure that... 1769 01:07:19,502 --> 01:07:21,573 And when you have one team 1770 01:07:21,573 --> 01:07:22,977 that really actually wants to engage 1771 01:07:22,977 --> 01:07:24,540 and is providing that information, 1772 01:07:24,540 --> 01:07:25,950 you have another team that's just taking it in 1773 01:07:25,950 --> 01:07:26,886 and providing nothing back, 1774 01:07:26,886 --> 01:07:28,860 making sure that both of those teams come in 1775 01:07:28,860 --> 01:07:30,420 with the same type of expectation, I think, 1776 01:07:30,420 --> 01:07:31,980 was something that we really learned the hard way. 1777 01:07:31,980 --> 01:07:32,813 Yeah. 1778 01:07:34,410 --> 01:07:37,320 - I did an exercise with the C-suite 1779 01:07:37,320 --> 01:07:41,309 and when we got to the ransom repayment section, 1780 01:07:41,309 --> 01:07:44,130 unanimously and sort of led by the CEO, 1781 01:07:44,130 --> 01:07:47,730 the C-suite was like, let them leak the data. 1782 01:07:47,730 --> 01:07:50,494 No one cares about our company anyways. 1783 01:07:50,494 --> 01:07:53,850 They won't write news stories about us. 1784 01:07:53,850 --> 01:07:55,560 Let them do their worst. 1785 01:07:55,560 --> 01:07:58,620 And I'm not saying that they should have paid the ransom, 1786 01:07:58,620 --> 01:08:03,620 but to me, it was such a mismatch of understanding of risk, 1787 01:08:04,078 --> 01:08:06,660 which is something that I sort of anticipated 1788 01:08:06,660 --> 01:08:09,210 that the C-suite would have a better handle on. 1789 01:08:09,210 --> 01:08:11,558 So that was a weird exercise. 1790 01:08:11,558 --> 01:08:14,490 (Sarah laughing) 1791 01:08:14,490 --> 01:08:16,767 - [Audience Member] That brings up a good question in, 1792 01:08:16,767 --> 01:08:20,229 and I agree to the question. 1793 01:08:20,229 --> 01:08:24,510 The original question is that sometimes in my view, 1794 01:08:24,510 --> 01:08:26,910 as sometimes having created them, 1795 01:08:26,910 --> 01:08:28,710 having also facilitated them; 1796 01:08:28,710 --> 01:08:31,830 as a facilitator, I'll let things play out a little bit. 1797 01:08:31,830 --> 01:08:33,930 But at some point, you gotta get it back on track. 1798 01:08:33,930 --> 01:08:36,326 When a major decision was made, that was incorrect, 1799 01:08:36,326 --> 01:08:38,160 you have to get it back on track. 1800 01:08:38,160 --> 01:08:39,414 But let it play out, 1801 01:08:39,414 --> 01:08:41,430 and then there are usually some good action items 1802 01:08:41,430 --> 01:08:43,025 that can come out of that. 1803 01:08:43,025 --> 01:08:48,025 But to some of the other points was, 1804 01:08:48,270 --> 01:08:50,134 and I just lost my train of thought, so. 1805 01:08:50,134 --> 01:08:52,132 (audience and panelists laughing) 1806 01:08:52,132 --> 01:08:53,883 - No one noticed that, yeah. 1807 01:08:55,208 --> 01:08:57,750 - [Audience Member] It was around... 1808 01:08:57,750 --> 01:09:00,896 What were you talking about right there at the end? 1809 01:09:00,896 --> 01:09:02,370 - Expectation? 1810 01:09:02,370 --> 01:09:04,717 Or Sarah was... 1811 01:09:04,717 --> 01:09:05,869 - [Audience Member] Oh, yeah. 1812 01:09:05,869 --> 01:09:09,063 The major decisions that are just crazy. 1813 01:09:10,212 --> 01:09:12,960 Again, sometimes that comes down to, 1814 01:09:12,960 --> 01:09:15,390 and I've had that happen. 1815 01:09:15,390 --> 01:09:18,570 If it's an executive making that decision, 1816 01:09:18,570 --> 01:09:20,940 there's only so much you can do. 1817 01:09:20,940 --> 01:09:22,710 And it comes back to then, 1818 01:09:22,710 --> 01:09:27,060 maybe even having education sessions privately 1819 01:09:27,060 --> 01:09:30,150 or making some recommendations, not in public, 1820 01:09:30,150 --> 01:09:33,000 'cause you don't wanna shame somebody at that level. 1821 01:09:33,000 --> 01:09:34,710 But those are very critical things 1822 01:09:34,710 --> 01:09:36,900 that need to be addressed. 1823 01:09:36,900 --> 01:09:38,520 At least in my view as a facilitator, 1824 01:09:38,520 --> 01:09:43,520 I hate to walk away and not address that in some fashion. 1825 01:09:43,770 --> 01:09:45,750 And in some cases, some of that doesn't make it 1826 01:09:45,750 --> 01:09:49,323 into the report, but it makes it into a verbal conversation. 1827 01:09:53,730 --> 01:09:55,950 - Anyone else was looking to chime in 1828 01:09:55,950 --> 01:09:57,933 on bad actions, bad ideas? 1829 01:09:58,830 --> 01:10:00,240 No. 1830 01:10:00,240 --> 01:10:02,850 - I'll follow up on the Cyber 9/12 side, Safa. 1831 01:10:02,850 --> 01:10:06,265 Again, going back to the student angle, 1832 01:10:06,265 --> 01:10:10,866 it's really great to see how many percentage of students 1833 01:10:10,866 --> 01:10:12,870 in the Cyber 9/12 Challenge. 1834 01:10:12,870 --> 01:10:15,240 Because it's usually a shock and awe type cyber attack, 1835 01:10:15,240 --> 01:10:17,790 the lights go off, something's gone wrong, 1836 01:10:17,790 --> 01:10:20,430 gotta brief the NSC in like 12 hours. 1837 01:10:20,430 --> 01:10:22,380 And there's always a team that goes, 1838 01:10:22,380 --> 01:10:25,410 it's trying to send carriers into the Taiwanese Strait 1839 01:10:25,410 --> 01:10:28,563 or into the strait of the South China Sea. 1840 01:10:30,060 --> 01:10:34,140 And being able to walk that back, go okay, 1841 01:10:34,140 --> 01:10:35,880 using the guiding questions of like okay, 1842 01:10:35,880 --> 01:10:39,150 what is the basis of your analysis? 1843 01:10:39,150 --> 01:10:41,940 How confident are you in your assessment? 1844 01:10:41,940 --> 01:10:44,100 Walking them through the intelligence process 1845 01:10:44,100 --> 01:10:45,690 of here's all the things that you need to have 1846 01:10:45,690 --> 01:10:46,950 in order to make that assessment, 1847 01:10:46,950 --> 01:10:48,515 and then making that into a learning process 1848 01:10:48,515 --> 01:10:52,500 rather than that was bad and you should feel bad. 1849 01:10:52,500 --> 01:10:54,060 I think was definitely... 1850 01:10:54,060 --> 01:10:55,200 And it's been great. 1851 01:10:55,200 --> 01:10:57,630 Cyber 9/12 has had fewer recommendations 1852 01:10:57,630 --> 01:11:00,408 about sending the Navy to defend 1853 01:11:00,408 --> 01:11:03,750 or to go into the South China Sea for very little reason 1854 01:11:03,750 --> 01:11:05,373 since 2015, 2016. 1855 01:11:06,218 --> 01:11:07,087 (Winnona laughing) 1856 01:11:07,087 --> 01:11:08,460 (Safa speaking faintly) 1857 01:11:08,460 --> 01:11:10,440 - I mean, I think that's a really good point. 1858 01:11:10,440 --> 01:11:11,940 I think the escalation piece 1859 01:11:11,940 --> 01:11:15,180 of tabletop exercises is really important. 1860 01:11:15,180 --> 01:11:19,410 We'll get folks that the first hint of maybe an incident, 1861 01:11:19,410 --> 01:11:23,400 they're like high severity rating, ring the alarms, 1862 01:11:23,400 --> 01:11:24,233 which is not good. 1863 01:11:24,233 --> 01:11:26,130 And then some folks will say, yeah, I bet it's nothing. 1864 01:11:26,130 --> 01:11:27,113 Like you're in a tabletop exercise, 1865 01:11:27,113 --> 01:11:30,300 so you know it's gonna be something bad. 1866 01:11:30,300 --> 01:11:32,940 I think hitting the sweet spot for escalation 1867 01:11:32,940 --> 01:11:35,430 where you're following with the information 1868 01:11:35,430 --> 01:11:37,320 that you have at the time, 1869 01:11:37,320 --> 01:11:39,480 but you're not doing an all hands on deck 1870 01:11:39,480 --> 01:11:41,370 before you've done any sort of analysis. 1871 01:11:41,370 --> 01:11:43,380 And then, you can do something useful with that information, 1872 01:11:43,380 --> 01:11:45,300 is very, very tricky. 1873 01:11:45,300 --> 01:11:48,120 And I think most folks either over or under escalate 1874 01:11:48,120 --> 01:11:51,570 during that incident and aren't able to hit that sweet spot. 1875 01:11:51,570 --> 01:11:53,248 Is that what you see in Cyber 9/12? 1876 01:11:53,248 --> 01:11:54,081 (Winnona laughing) 1877 01:11:54,081 --> 01:11:56,450 I mean, with the calling in the Navy right away, you know? 1878 01:11:56,450 --> 01:11:59,160 - Yeah, calling the Navy right away, sending a carrier. 1879 01:11:59,160 --> 01:12:02,100 Yeah, it's definitely, you can get either side 1880 01:12:02,100 --> 01:12:03,660 where it's like, it's probably nothing or it's everything. 1881 01:12:03,660 --> 01:12:05,234 And then I think that's a practice thing. 1882 01:12:05,234 --> 01:12:08,790 You'll figure out, based off of what certain teams 1883 01:12:08,790 --> 01:12:10,470 are saying, moving it kind of more into the corporate 1884 01:12:10,470 --> 01:12:12,540 or managerial aspect, depending on what certain teams 1885 01:12:12,540 --> 01:12:14,340 are saying, listening to that 1886 01:12:14,340 --> 01:12:17,040 and understanding what vocabulary means what 1887 01:12:17,040 --> 01:12:19,734 on their this is messed up barometer. 1888 01:12:19,734 --> 01:12:20,567 (Winnona laughing) 1889 01:12:20,567 --> 01:12:21,750 - Yeah, and it also comes with, 1890 01:12:21,750 --> 01:12:23,280 how well do you know your network? 1891 01:12:23,280 --> 01:12:25,650 How well do you know what behavior is suspicious 1892 01:12:25,650 --> 01:12:27,599 versus what's probably some sort of outage? 1893 01:12:27,599 --> 01:12:29,176 That's questions that we ask all. 1894 01:12:29,176 --> 01:12:31,080 Is this behavior normal on this system? 1895 01:12:31,080 --> 01:12:32,805 And most often we get, I don't know, 1896 01:12:32,805 --> 01:12:34,616 We don't have a baseline for that. 1897 01:12:34,616 --> 01:12:35,449 (Winnona laughing) 1898 01:12:35,449 --> 01:12:36,282 Oh no, people. 1899 01:12:36,282 --> 01:12:37,115 I mean, that's okay. 1900 01:12:37,115 --> 01:12:38,040 It's hard. 1901 01:12:38,040 --> 01:12:38,873 There's like a million things that we've talked about today 1902 01:12:38,873 --> 01:12:40,110 that people have to know for these things. 1903 01:12:40,110 --> 01:12:41,130 And they're not intuitive, 1904 01:12:41,130 --> 01:12:43,590 and you have to learn them over time. 1905 01:12:43,590 --> 01:12:46,050 We only know this because we run these all the time. 1906 01:12:46,050 --> 01:12:47,070 But on the other end of it, 1907 01:12:47,070 --> 01:12:48,672 it's not clear that you're supposed to know 1908 01:12:48,672 --> 01:12:50,730 what the baseline is for all these different systems 1909 01:12:50,730 --> 01:12:52,350 so that when you have an alert come in, you can say, 1910 01:12:52,350 --> 01:12:54,420 that's definitely something that we need to worry about 1911 01:12:54,420 --> 01:12:57,660 versus could be an IT incident, we don't really know. 1912 01:12:57,660 --> 01:12:59,190 - I think you raised an interesting point though 1913 01:12:59,190 --> 01:13:00,420 on human decision making. 1914 01:13:00,420 --> 01:13:02,564 And sometimes, someone makes a recommendation or says, 1915 01:13:02,564 --> 01:13:04,110 I wanna take this action. 1916 01:13:04,110 --> 01:13:06,270 And it's helpful to walk that backwards. 1917 01:13:06,270 --> 01:13:08,160 One, what is your intended outcome? 1918 01:13:08,160 --> 01:13:10,230 What are you trying to achieve? 1919 01:13:10,230 --> 01:13:12,292 And then how does this help you get closer to that? 1920 01:13:12,292 --> 01:13:14,130 And sometimes you see the gears churning 1921 01:13:14,130 --> 01:13:15,060 where people are like, 1922 01:13:15,060 --> 01:13:17,640 oh, my recommendation actually doesn't help me get closer 1923 01:13:17,640 --> 01:13:19,500 to that, it's actually counterintuitive. 1924 01:13:19,500 --> 01:13:20,618 But I'm in a stressful situation. 1925 01:13:20,618 --> 01:13:21,916 I have limited information, 1926 01:13:21,916 --> 01:13:24,630 and I had to develop a briefing in 15 minutes. 1927 01:13:24,630 --> 01:13:26,610 Of course I made this recommendation. 1928 01:13:26,610 --> 01:13:28,660 And I think that we've seen that in Cyber 9/12, 1929 01:13:28,660 --> 01:13:31,380 where the judge is saying, hey, tell me more about that. 1930 01:13:31,380 --> 01:13:32,460 Let's unpack that a little bit. 1931 01:13:32,460 --> 01:13:33,750 And then sometimes at the end being like, 1932 01:13:33,750 --> 01:13:34,937 are you sure you wanna do this? 1933 01:13:34,937 --> 01:13:37,710 'Cause there's still time and you can walk this back. 1934 01:13:37,710 --> 01:13:42,000 We haven't taken action yet, but I'd love to get... 1935 01:13:42,000 --> 01:13:44,280 I guess I'm curious about tabletop exercise 1936 01:13:44,280 --> 01:13:47,010 that you guys have participated in or have developed. 1937 01:13:47,010 --> 01:13:48,360 And we do this all the time 1938 01:13:48,360 --> 01:13:49,620 because we're kind of crazy, 1939 01:13:49,620 --> 01:13:51,150 but we love red herrings. 1940 01:13:51,150 --> 01:13:54,570 And we love to, something that kind of pushes the player 1941 01:13:54,570 --> 01:13:57,000 to take an action with this limited information 1942 01:13:57,000 --> 01:13:58,620 that might not be helpful for them, 1943 01:13:58,620 --> 01:14:00,660 so that they get more skeptical 1944 01:14:00,660 --> 01:14:03,810 and try to take the limited variation that they have 1945 01:14:03,810 --> 01:14:04,680 with a grain of salt. 1946 01:14:04,680 --> 01:14:06,180 You're never gonna have all the information 1947 01:14:06,180 --> 01:14:08,850 that you ever need to make a completely accurate decision. 1948 01:14:08,850 --> 01:14:12,241 So how do you operate with limited or imperfect information? 1949 01:14:12,241 --> 01:14:15,347 Have you guys ever seen any red herrings 1950 01:14:15,347 --> 01:14:19,470 in these simulations or in the ones that you've developed 1951 01:14:19,470 --> 01:14:22,503 that people either latch onto or take the wrong way? 1952 01:14:23,790 --> 01:14:25,031 - Open source malware. 1953 01:14:25,031 --> 01:14:27,457 (Winnona laughing) 1954 01:14:27,457 --> 01:14:29,550 Definitely doing those sorts of analyses. 1955 01:14:29,550 --> 01:14:30,960 And that goes back into 1956 01:14:30,960 --> 01:14:35,412 the threat intelligence type capture flag or simulations 1957 01:14:35,412 --> 01:14:37,113 that I've helped design. 1958 01:14:38,250 --> 01:14:39,990 Especially going into like, 1959 01:14:39,990 --> 01:14:42,113 how much do you trust your threat intel feeds? 1960 01:14:42,113 --> 01:14:42,946 (Winnona laughing) 1961 01:14:42,946 --> 01:14:46,062 But that's probably a different panel, but definitely 1962 01:14:46,062 --> 01:14:49,440 when you're talking about the technical accuracy 1963 01:14:49,440 --> 01:14:52,020 and understanding what your assumptions are, 1964 01:14:52,020 --> 01:14:54,633 and not relying just on what a vendor is saying, 1965 01:14:56,072 --> 01:14:58,920 or making sure that you have the right context 1966 01:14:58,920 --> 01:15:00,780 when you're making those decisions. 1967 01:15:00,780 --> 01:15:03,780 - I think that dealing with uncertainty is always a takeaway 1968 01:15:03,780 --> 01:15:05,537 from the tabletop exercises. 1969 01:15:05,537 --> 01:15:08,160 Folks try and sort of weasel out of it where like, 1970 01:15:08,160 --> 01:15:09,900 you have to put out a press statement right now. 1971 01:15:09,900 --> 01:15:10,733 What are you gonna say? 1972 01:15:10,733 --> 01:15:12,440 And they say, well, it will sort of depend on 1973 01:15:12,440 --> 01:15:14,190 what the initial attack path was 1974 01:15:14,190 --> 01:15:15,780 and what the point of entry was. 1975 01:15:15,780 --> 01:15:17,550 And we're like, well, you don't know that. 1976 01:15:17,550 --> 01:15:18,713 You're not gonna know that. 1977 01:15:18,713 --> 01:15:20,280 And so, you might not find out for several days 1978 01:15:20,280 --> 01:15:22,521 and you still need to make a decision. 1979 01:15:22,521 --> 01:15:24,197 That's something that's really difficult to drill in. 1980 01:15:24,197 --> 01:15:27,833 'Cause I think when it's a PowerPoint or something, 1981 01:15:27,833 --> 01:15:29,460 it feels like not real. 1982 01:15:29,460 --> 01:15:31,980 And it feels like because it's vague, you'll just say, 1983 01:15:31,980 --> 01:15:33,897 well, if it was like this, I would do this. 1984 01:15:33,897 --> 01:15:35,490 And if it was like this, I would do this, 1985 01:15:35,490 --> 01:15:36,360 and that's my answer. 1986 01:15:36,360 --> 01:15:38,160 And we have to sort of pin them down and say, 1987 01:15:38,160 --> 01:15:39,780 you don't know the answer 1988 01:15:39,780 --> 01:15:41,768 and you still have to make a decision. 1989 01:15:41,768 --> 01:15:44,550 So to your point, I think no one's really prepared 1990 01:15:44,550 --> 01:15:45,780 to deal with that sort of uncertainty. 1991 01:15:45,780 --> 01:15:47,250 But that's something we try and emulate. 1992 01:15:47,250 --> 01:15:49,110 We, on purpose, don't give them enough information 1993 01:15:49,110 --> 01:15:51,030 to make a decision, 'cause they're not gonna have that 1994 01:15:51,030 --> 01:15:51,873 in real life. 1995 01:15:52,980 --> 01:15:53,994 - Thanks for that. 1996 01:15:53,994 --> 01:15:54,827 I think, and opening this up to the room, 1997 01:15:54,827 --> 01:15:56,010 we have some new folks in the room. 1998 01:15:56,010 --> 01:15:58,600 Please, if you would like to ask a question 1999 01:15:58,600 --> 01:16:00,517 or drop a comment, please make your way 2000 01:16:00,517 --> 01:16:01,890 over to a microphone. 2001 01:16:01,890 --> 01:16:03,319 The acoustics in here is not great. 2002 01:16:03,319 --> 01:16:04,690 Please, sir. 2003 01:16:04,690 --> 01:16:07,230 - [Audience Member] I'd be interested in your thoughts 2004 01:16:07,230 --> 01:16:10,020 in terms of global corporations 2005 01:16:10,020 --> 01:16:12,810 with multi-international regulations. 2006 01:16:12,810 --> 01:16:17,760 When doing a tabletop, trying to unpick the way 2007 01:16:17,760 --> 01:16:20,670 that sometimes regulations conflict each other 2008 01:16:20,670 --> 01:16:22,303 because of a lack of international, 2009 01:16:22,303 --> 01:16:25,200 sort of single coherent approach. 2010 01:16:25,200 --> 01:16:28,590 So I'd just be interested in how people have overcome that 2011 01:16:28,590 --> 01:16:30,840 in their tabletops, in that type of scenario. 2012 01:16:33,030 --> 01:16:34,080 - Can you rephrase that? 2013 01:16:34,080 --> 01:16:36,390 Are you asking about how we make sure that they're aware 2014 01:16:36,390 --> 01:16:37,950 of regulatory requirements? 2015 01:16:37,950 --> 01:16:39,030 - [Audience Member] Well I think, 2016 01:16:39,030 --> 01:16:40,337 if you've got a global corporation 2017 01:16:40,337 --> 01:16:42,780 and they're working in different regions, 2018 01:16:42,780 --> 01:16:46,530 then typically, the regulations can sometimes not be as one. 2019 01:16:46,530 --> 01:16:49,999 And then with the lack of a single international policy 2020 01:16:49,999 --> 01:16:52,170 in terms of cyber security, 2021 01:16:52,170 --> 01:16:55,980 that absolutely complicates the matter even further. 2022 01:16:55,980 --> 01:16:59,273 So if you're trying to do a ransomware, 2023 01:16:59,273 --> 01:17:00,360 well not necessarily a ransomware. 2024 01:17:00,360 --> 01:17:01,890 If you're trying to do a tabletop 2025 01:17:01,890 --> 01:17:05,730 and run that within that complicated scenario, 2026 01:17:05,730 --> 01:17:10,680 any thoughts of how best to run that type of situation? 2027 01:17:10,680 --> 01:17:12,810 - Yeah, this is a bit of a non-answer, 2028 01:17:12,810 --> 01:17:15,150 but we try and scope them to one country, 2029 01:17:15,150 --> 01:17:18,270 because there's usually enough that could go wrong 2030 01:17:18,270 --> 01:17:21,570 even within one country that there is some takeaways there. 2031 01:17:21,570 --> 01:17:22,740 But sometimes we'll do, you know, 2032 01:17:22,740 --> 01:17:24,150 there's an office in New York 2033 01:17:24,150 --> 01:17:25,770 and there's one in Singapore or something, 2034 01:17:25,770 --> 01:17:28,009 and they're both under attack or they both lost data. 2035 01:17:28,009 --> 01:17:30,810 And then we sort of repeat a lot of the questions, right? 2036 01:17:30,810 --> 01:17:31,710 Do you have external counsel? 2037 01:17:31,710 --> 01:17:32,790 But do you have external counsel 2038 01:17:32,790 --> 01:17:35,670 in all the different countries that you're operating in? 2039 01:17:35,670 --> 01:17:37,230 Do you have a data privacy officer 2040 01:17:37,230 --> 01:17:38,063 in all the different countries 2041 01:17:38,063 --> 01:17:39,930 that's tracking those requirements? 2042 01:17:39,930 --> 01:17:42,206 So we don't do it as often, but... 2043 01:17:42,206 --> 01:17:43,039 - [Audience Member] Yeah, no. 2044 01:17:43,039 --> 01:17:44,665 I was just gonna pick up an earlier point 2045 01:17:44,665 --> 01:17:46,410 that we've talked about, law enforcement, 2046 01:17:46,410 --> 01:17:50,889 where most of the narrative was about engaging them. 2047 01:17:50,889 --> 01:17:53,820 And then there is situations where actually, 2048 01:17:53,820 --> 01:17:55,200 they engage you first off. 2049 01:17:55,200 --> 01:17:58,350 Because you weren't aware that something has happened, so. 2050 01:17:58,350 --> 01:18:00,580 And I've had a situation 2051 01:18:01,847 --> 01:18:06,060 where a law enforcement international organization contacted 2052 01:18:06,060 --> 01:18:07,631 a global company and then, 2053 01:18:07,631 --> 01:18:10,143 what were the actions that were taken thereafter? 2054 01:18:11,580 --> 01:18:12,960 - I'd also add, I think it's important 2055 01:18:12,960 --> 01:18:15,570 to have tabletops done by teams in the country 2056 01:18:15,570 --> 01:18:17,220 in which you're operating in. 2057 01:18:17,220 --> 01:18:19,110 If someone really does have, whoops, 2058 01:18:19,110 --> 01:18:20,956 different global offices that they want us to test, 2059 01:18:20,956 --> 01:18:22,980 we would try and get folks who actually live 2060 01:18:22,980 --> 01:18:24,710 in those countries to run those tabletop exercises 2061 01:18:24,710 --> 01:18:27,000 in the native language and who have a better understanding 2062 01:18:27,000 --> 01:18:29,350 of the requirements of that country than we do. 2063 01:18:31,201 --> 01:18:33,150 - If I can chime on that locality piece 2064 01:18:33,150 --> 01:18:33,983 is really interesting. 2065 01:18:33,983 --> 01:18:36,120 'Cause I know we've done that with Cyber 9/12 scenarios 2066 01:18:36,120 --> 01:18:37,969 where if we're operating in another country, 2067 01:18:37,969 --> 01:18:42,030 we have to have a local partner that's either a university 2068 01:18:42,030 --> 01:18:44,130 or a think tank that includes researchers, 2069 01:18:44,130 --> 01:18:44,970 so that we can make sure 2070 01:18:44,970 --> 01:18:46,980 that we're developing a scenario that's, one, 2071 01:18:46,980 --> 01:18:48,090 compelling for the students. 2072 01:18:48,090 --> 01:18:50,970 But also, two, it's helping them build muscle memory 2073 01:18:50,970 --> 01:18:53,550 with specific challenges or gain some familiarity 2074 01:18:53,550 --> 01:18:55,896 with specific legislation that is important to them. 2075 01:18:55,896 --> 01:18:58,200 One, it's good for them to have that understanding, 2076 01:18:58,200 --> 01:18:59,790 but then two, it makes them more marketable 2077 01:18:59,790 --> 01:19:00,960 when they go to apply for a job, 2078 01:19:00,960 --> 01:19:03,617 because they have that understanding. 2079 01:19:03,617 --> 01:19:04,890 - [Audience Member] Yeah, and I also found 2080 01:19:04,890 --> 01:19:06,233 which takes precedence. 2081 01:19:06,233 --> 01:19:08,490 So if you've got multi countries, 2082 01:19:08,490 --> 01:19:11,240 which one are you gonna take first as a prioritization? 2083 01:19:12,280 --> 01:19:13,920 - And I can actually kind of answer that 2084 01:19:13,920 --> 01:19:16,410 because our Geneva scenario is really fascinating 2085 01:19:16,410 --> 01:19:19,200 in that sense where it's in Geneva, Switzerland. 2086 01:19:19,200 --> 01:19:21,150 It includes competitors from all over Europe 2087 01:19:21,150 --> 01:19:24,750 and also in the US and Asia. 2088 01:19:24,750 --> 01:19:26,520 But usually if we do want to focus 2089 01:19:26,520 --> 01:19:27,778 on an international aspect, 2090 01:19:27,778 --> 01:19:29,610 Europe is a pretty good place for that. 2091 01:19:29,610 --> 01:19:31,110 You can look at the EU as a structure. 2092 01:19:31,110 --> 01:19:32,242 You can look at NATO as a structure. 2093 01:19:32,242 --> 01:19:35,036 We do have a track record of pushing students to the EU, 2094 01:19:35,036 --> 01:19:37,350 so they can get familiar with that legislation 2095 01:19:37,350 --> 01:19:39,218 that exists at the supernational level. 2096 01:19:39,218 --> 01:19:41,394 But it also challenges them to look at 2097 01:19:41,394 --> 01:19:43,830 where is the incident actually occurring 2098 01:19:43,830 --> 01:19:45,109 in that one particular country? 2099 01:19:45,109 --> 01:19:47,880 Is it impacting folks in other countries, 2100 01:19:47,880 --> 01:19:49,080 and how do they wanna respond to that? 2101 01:19:49,080 --> 01:19:51,347 Is there a component that might have to bring in NATO? 2102 01:19:51,347 --> 01:19:53,400 It can get a lot stickier and challenging, 2103 01:19:53,400 --> 01:19:55,410 but it's a good exercise, I think, for students, 2104 01:19:55,410 --> 01:19:58,170 especially if they do wanna work on a policy team 2105 01:19:58,170 --> 01:19:59,933 at an organization like a Microsoft or Facebook. 2106 01:19:59,933 --> 01:20:02,730 They're gonna have to be familiar with scaling up 2107 01:20:02,730 --> 01:20:04,590 or down in terms of which legislation 2108 01:20:04,590 --> 01:20:07,593 or government entities they're gonna be interacting with. 2109 01:20:09,547 --> 01:20:10,380 - [Audience Member] Hey, everyone. 2110 01:20:10,380 --> 01:20:11,213 So I'm Gerov from Singapore. 2111 01:20:11,213 --> 01:20:12,900 I don't know if you just randomly mentioned Singapore, 2112 01:20:12,900 --> 01:20:15,340 if it was a specific thing, but I thought I'd jump in. 2113 01:20:15,340 --> 01:20:17,280 So we actually end up having a lot of companies 2114 01:20:17,280 --> 01:20:18,660 who have presence in Singapore 2115 01:20:18,660 --> 01:20:20,250 and obviously presence everywhere else in the world. 2116 01:20:20,250 --> 01:20:23,144 And we have tabletop exercises. 2117 01:20:23,144 --> 01:20:24,870 There are two types of questions. 2118 01:20:24,870 --> 01:20:26,460 One is the policy question. 2119 01:20:26,460 --> 01:20:28,110 The other is a tabletop question. 2120 01:20:28,110 --> 01:20:32,340 The policy question of how do different non-mutually, 2121 01:20:32,340 --> 01:20:34,680 non-integrated international laws work together? 2122 01:20:34,680 --> 01:20:36,180 That's a policy question. 2123 01:20:36,180 --> 01:20:37,197 Sometimes they don't. 2124 01:20:37,197 --> 01:20:39,660 And I don't think the tabletop people can answer 2125 01:20:39,660 --> 01:20:41,340 that question fairly. 2126 01:20:41,340 --> 01:20:44,940 So that is a bigger international policy regulation, 2127 01:20:44,940 --> 01:20:48,269 legislation question, which has no good answer. 2128 01:20:48,269 --> 01:20:51,860 But the tabletop question, which is actually quite useful, 2129 01:20:51,860 --> 01:20:54,346 is sometimes just get people to walk through that. 2130 01:20:54,346 --> 01:20:57,538 So in Singapore we have an incident response disclosure 2131 01:20:57,538 --> 01:20:59,580 or incident disclosure policy, 2132 01:20:59,580 --> 01:21:02,160 which is in two hours or something like that. 2133 01:21:02,160 --> 01:21:04,320 And other countries have 72 hours. 2134 01:21:04,320 --> 01:21:06,570 So we tell them, so you have an incident now 2135 01:21:06,570 --> 01:21:09,660 which is impacting your Singapore systems, 2136 01:21:09,660 --> 01:21:13,470 and your Hong Kong systems, and your British systems. 2137 01:21:13,470 --> 01:21:15,120 What do you do? 2138 01:21:15,120 --> 01:21:17,910 And then they will have to internally figure out 2139 01:21:17,910 --> 01:21:19,410 what is their process? 2140 01:21:19,410 --> 01:21:20,730 Who do they contact first? 2141 01:21:20,730 --> 01:21:23,100 Do they alert the Singapore authorities first 2142 01:21:23,100 --> 01:21:26,340 and then risk having activated the Hong Kong 2143 01:21:26,340 --> 01:21:28,833 and British authorities prematurely? 2144 01:21:29,675 --> 01:21:32,790 That's for them to answer, but at least you force them 2145 01:21:32,790 --> 01:21:34,830 to have that conversation internally. 2146 01:21:34,830 --> 01:21:36,893 And then they realize that the Singapore lawyer team 2147 01:21:36,893 --> 01:21:38,600 doesn't have the contact information 2148 01:21:38,600 --> 01:21:40,309 of the Hong Kong law team, or they didn't know 2149 01:21:40,309 --> 01:21:43,740 that they have a different disclosure policy 2150 01:21:43,740 --> 01:21:44,760 in other countries. 2151 01:21:44,760 --> 01:21:46,590 So sometimes the tabletop exercise is just to get people 2152 01:21:46,590 --> 01:21:48,240 to understand that there are differences 2153 01:21:48,240 --> 01:21:50,220 and that they need to start internally thinking about 2154 01:21:50,220 --> 01:21:52,863 that problem, even if there is no good answer to it. 2155 01:21:55,679 --> 01:21:56,990 - Thanks for that, Gerov. 2156 01:21:56,990 --> 01:22:00,435 I know that we're getting close to closing time. 2157 01:22:00,435 --> 01:22:02,910 Really appreciate you guys joining us today. 2158 01:22:02,910 --> 01:22:05,520 I wanna turn over to Winnona and Sarah, 2159 01:22:05,520 --> 01:22:06,750 if they have any last comments 2160 01:22:06,750 --> 01:22:07,800 or things that they wanted to add 2161 01:22:07,800 --> 01:22:11,160 before we break up the session. 2162 01:22:11,160 --> 01:22:12,720 - Yeah, I'll say one thing. 2163 01:22:12,720 --> 01:22:14,550 I think that with table, 2164 01:22:14,550 --> 01:22:16,800 we talked about giving feedback earlier. 2165 01:22:16,800 --> 01:22:19,230 We can really only give feedback for the couple hours 2166 01:22:19,230 --> 01:22:20,233 that we see folks. 2167 01:22:20,233 --> 01:22:22,440 We really, tabletops are really good 2168 01:22:22,440 --> 01:22:23,945 for understanding people and processes. 2169 01:22:23,945 --> 01:22:26,340 They're not as good for poking holes 2170 01:22:26,340 --> 01:22:28,440 in actual cybersecurity maturity 2171 01:22:28,440 --> 01:22:32,610 and what folks need to be prioritizing for their practice. 2172 01:22:32,610 --> 01:22:33,443 So I'll just say that. 2173 01:22:33,443 --> 01:22:36,150 I think they're very good for particular purposes 2174 01:22:36,150 --> 01:22:39,764 for escalating cybersecurity importance to management, 2175 01:22:39,764 --> 01:22:42,300 but there is a limited amount of information 2176 01:22:42,300 --> 01:22:44,880 that we're able to give feedback on. 2177 01:22:44,880 --> 01:22:47,013 And so I just wanna throw that out there. 2178 01:22:48,090 --> 01:22:50,910 - Yeah, I guess I have a twofer for a last remark. 2179 01:22:50,910 --> 01:22:52,080 I guess the first thing is, 2180 01:22:52,080 --> 01:22:55,920 I know we've talked about more of the technical side 2181 01:22:55,920 --> 01:22:57,720 of the tabletops versus the policy. 2182 01:22:57,720 --> 01:22:59,340 And when those two integrate, 2183 01:22:59,340 --> 01:23:00,540 the one thing that I wanna stress 2184 01:23:00,540 --> 01:23:05,540 is I have seen so many technical tabletops go really well, 2185 01:23:06,480 --> 01:23:08,550 but when they're communicated in a certain way 2186 01:23:08,550 --> 01:23:11,430 to a decision maker, there's a complete disconnect, 2187 01:23:11,430 --> 01:23:12,867 even though they both want the same thing. 2188 01:23:12,867 --> 01:23:15,270 And I think that one of the pieces of feedback 2189 01:23:15,270 --> 01:23:17,730 that I personally try to give, 2190 01:23:17,730 --> 01:23:19,560 especially when I see that disconnect, 2191 01:23:19,560 --> 01:23:22,951 is to talk about technical aspects in terms of risk, 2192 01:23:22,951 --> 01:23:26,430 in terms of monetary value versus what needs to be fixed 2193 01:23:26,430 --> 01:23:29,400 and what they need to best describe the problem 2194 01:23:29,400 --> 01:23:30,233 to their principle; 2195 01:23:30,233 --> 01:23:34,230 and then vice versa, going from high level to low level. 2196 01:23:34,230 --> 01:23:36,491 And then last but not least, if anyone is, 2197 01:23:36,491 --> 01:23:37,950 I know we got that feedback 2198 01:23:37,950 --> 01:23:40,830 of why don't we have a tabletop exercise for DEF CON policy? 2199 01:23:40,830 --> 01:23:43,110 If anyone is interested in leading that, 2200 01:23:43,110 --> 01:23:44,370 we have a one year lead up, 2201 01:23:44,370 --> 01:23:47,820 so (laughs) come talk to us after. 2202 01:23:47,820 --> 01:23:49,710 But yeah, thank you for this opportunity, Safa. 2203 01:23:49,710 --> 01:23:51,840 - I wanna foot stomp that, Winnona. 2204 01:23:51,840 --> 01:23:53,250 If you have any comments or feedback for us, 2205 01:23:53,250 --> 01:23:54,083 please let us know. 2206 01:23:54,083 --> 01:23:55,740 We're really just trying to keep improving 2207 01:23:55,740 --> 01:23:56,610 and growing this program. 2208 01:23:56,610 --> 01:23:59,400 And if I can do one shameless plug, 2209 01:23:59,400 --> 01:24:01,410 if you're interested in cyber simulations 2210 01:24:01,410 --> 01:24:03,990 that are multidisciplinary, take a look at Atlantic Council, 2211 01:24:03,990 --> 01:24:06,669 Cyber 9/12, that's our student competition. 2212 01:24:06,669 --> 01:24:08,190 If you're interested in learning more, 2213 01:24:08,190 --> 01:24:09,780 we do have some playbooks on the website 2214 01:24:09,780 --> 01:24:11,760 that folks can rummage through. 2215 01:24:11,760 --> 01:24:13,577 In addition to that, we also have a, 2216 01:24:13,577 --> 01:24:15,660 I know we didn't get that much into storytelling 2217 01:24:15,660 --> 01:24:17,070 for this session, but I really wanted to lean 2218 01:24:17,070 --> 01:24:18,377 into the bargaining aspects 2219 01:24:18,377 --> 01:24:20,940 since folks were really interested in that. 2220 01:24:20,940 --> 01:24:24,030 We do have a short story series called the Cyber Moonshot, 2221 01:24:24,030 --> 01:24:26,700 which kind of makes fun of the Cyber Moonshot, 2222 01:24:26,700 --> 01:24:29,100 but examines cybersecurity concepts and challenges 2223 01:24:29,100 --> 01:24:31,320 through a comedic story. 2224 01:24:31,320 --> 01:24:32,850 But definitely recommend that folks take a look 2225 01:24:32,850 --> 01:24:34,291 at those two items. 2226 01:24:34,291 --> 01:24:35,178 - Great. 2227 01:24:35,178 --> 01:24:36,011 - Thank you. 2228 01:24:36,011 --> 01:24:36,900 - Thank you so much, Safa. 2229 01:24:36,900 --> 01:24:37,920 - Thanks everyone for joining us. 2230 01:24:37,920 --> 01:24:38,753 - Thank you, guys. 2231 01:24:38,753 --> 01:24:39,960 - And I think the next session 2232 01:24:39,960 --> 01:24:41,460 in this room is gonna be at noon. 2233 01:24:41,460 --> 01:24:43,410 It's gonna be hacking aviation po- 2234 01:24:43,410 --> 01:24:44,243 No, I'm lying. 2235 01:24:44,243 --> 01:24:45,076 Sorry. 2236 01:24:45,076 --> 01:24:45,909 (Winnona laughing) 2237 01:24:45,909 --> 01:24:46,920 It's gonna be addressing the gap between 2238 01:24:46,920 --> 01:24:49,817 and assessing or measuring the HARMer cyber attacks. 2239 01:24:49,817 --> 01:24:50,790 (audience applauding) 2240 01:24:50,790 --> 01:24:51,623 - Thank you. 2241 01:24:51,623 --> 01:24:52,956 Thank you, guys!