all right hi everyone my name is Safa Shawan Edwards and I'm a deputy director with the cyber
statecraft initiative at the Atlantic Council we're a foreign policy think tank based in
Washington DC my program works at the nexus of geopolitics and cyber security this morning's
discussion is imagining a cyber policy crisis where we're going to examine how storytelling
simulations and war games can help individuals better understand cyber security concepts and
challenges a few announcements before we begin first off cell phones as a courtesy to our
speakers please ensure that your cell phones are on silent the other part of this is we have
some microphones scattered around the room as you have questions please feel free to relocate and
and direct your questions into the microphones as you can hear the acoustics in this room is
a little funky so microphones are our friends here last so this is good photo policy it sure
habits anyone from taking pictures without the explicit permission of anyone in frame
and please keep your mask on at all times with the exception of speaking eating drinking etc
um I think that might be it so I think we're going to kick off here so with the discussion
today what we're hoping to do is examine how do stories simulations crisis crisis crisis
simulations and war games uh aid in learning and training a little bit about myself I've
organized simulations for college students this is our cyber 912 strategy challenge where we
develop these yeah I'll get closer thank you um should I adjust this okay thank you um so
I've organized simulations for college students this is part of our cyber 912 strategy challenge
where we develop a three-part cyber crisis scenario that examines a technical vulnerability
or an incident that has reaching implications on national security public policy etc and I am
joined today by my colleagues here we got Winona DeSambre she is a fellow with the cyber statecraft
initiative but also with the belfer center at harvard we are also joined by Sarah Powazek she
is a consultant at CrowdStrike um so before we get started I want to give a shout out to our
students who are joining us today we're going to start off with the discussion and some of the
questions the cyber statecraft team at the atlantic council we always like to ask speakers
some fun questions to get them to know them a little bit better so the first one is what is
your favorite beverage this can be alcoholic or otherwise your favorite book favorite musical
artist and if you could have one superpower what would it be Sarah is taking notes because these
are really good questions a lot of questions oh no no you know us better so I'll get over to you
yeah all right uh hi everyone so I'm Winona I'm a fellow I also used to work in cyber threat and
intelligence so while I was a fellow at uh doing think tank and fun cyber wargaming stuff I also
did in-depth detection response research um when it my favorite drink my husband and I got really
into making elaborate cocktails during the pandemic as I'm sure many of us may have also
done but we made this cocktail called the five eyes which is a muddled kiwi apologies to our
New Zealand friends uh bundaberg ginger beer from Australia a shot of scotch a shot of bourbon and a
shot of Canadian whiskey so highly recommend it it's very good uh superpower I don't know
invisibility I feel like any any person in the intel space might want to say that
okay I I think I got all four of the questions so hi folks my name's Sarah Bois I work for
CrowdStrike I run our tabletop exercises and we have a bunch of clients you know major enterprises
small businesses um so I feel like we run the gambit of the sorts of folks that we we walk
through these exercises um my favorite drink I'm a big drinker I'm a big drinker I'm a big drinker
I don't know if I'm the only one any coffee tea anybody tea a couple yeah um so I am a big fan
of blue sleeve tea I have a bunch of weird flavors like cupcake and banana bread and
um sort of substitute it for any sort of dessert that you feel like so
um superpower definitely talking to animals I have two really cute cats at home um musical artist I
don't know if you guys have heard of still woozy sort of like a funky electronic yeah I didn't think so
happy to be here awesome thank you for that um so to kick off this conversation um can you tell
me about a time that a simulation or or a tabletop that you put together helped either you or a
partner or client better understand a cyber security challenge or concept I can go first
yeah I mean so the I probably can't share too much but the way that we do tabletop exercises
is that um we actually find a viable attack path for a client and so we'll either do a technical
tabletop or a mini tabletop or a mini tabletop or a mini tabletop or a mini tabletop or a mini
management or executive tabletop um but we'll spend a good amount of time talking to someone
on their security team um you know what's your mfa coverage you know what does your network look like
and we actually try and find um a a current exploitable vulnerability or attack path like
for example we've been doing a lot of esxi ransomware lately because that's very popular
that that could happen on their network so that we can come up with questions um that they would
actually need to answer um during an attack and so that's that's been pretty successful I think
we'll walk a technical team through um we'll have them look up machine names for example and I say
oh my god you know we have 20 service accounts that's not good um so I that that sort of realism
in the exercise I think really helps those take away not just what sort of processes they need
to improve but what sorts of gaps they have in their maturity that they're going to need to work
on in the future um so instead of answering this question first I actually want to make a
gauge of the room like how many people first off uh you know know what a tabletop exercise is
excellent how many people have participated in a tabletop exercise
great great um raise your hand if the tabletop exercise was primarily geared towards higher level
decision makers okay and then raise your hand if uh so it's about half the room for the people on
the live stream raise your hand if the tabletop was designed more for technical practitioners
okay so the other half um so at least for me I've done tabletop exercises that kind of run
that entire gamut right um so you have the more technical side where you're actually going
through a simulation maybe a red team exercise you also have the more policy oriented stuff
uh I've done a decent amount of work for the cyber 912 um focusing you know building that
kind of policy or process or people muscle within people who want to get into cyber policy who may
not have you know briefed high level principles and things like that and then there's also the uh
you know tabletop exercises that are helpful when doing more international conversations so I've done
some of the more technical dialogue through the Harvard Belfer Center with a Chinese PLA think
tank which was a wild time and so going through the American delegation crisis simulation and
then looking at the Chinese counterpart is also an interesting set of ways to determine how that
process is going to work out when you know stuff hits the fan and so uh at least for me the the
thing that has been particularly interesting and I honestly would love to hear more from our
our audience as well as you know not just having all of us rattling on um but frankly
just having the opportunity to go through and learning about how the other side is thinking
when you're especially doing cross-team exercises I've found to be the most useful as well as
exercising the aspects of your job that may come up in a crisis that you don't do in your day-to-day
so for me at least from a technical perspective cyber 912 way back in the day like maybe what like
five six years ago when I first uh you know participated and then also became a judge
you see a lot of um there's a wide variety where you see technical students who are participating
in this challenge learning how to do bottom line up front briefing a principal making sure that
they get the main points across you see west point cadets naval academy cadets who are used to this
sort of briefing but may not have the technical expertise so doing that sort of in-depth reading
in terms of the scenario and so those cross team as well as cross-discipline um perspectives I
think is really what makes the tabletop really really impactful thanks for that Winona I think
you raise a really excellent point it's like one thing I'd like to highlight is so while we created
cyber 912 initially to
serve more like we saw lots of hackathons and capture the fly competitions and that can add
a lot of value to help students develop their technical understanding on the other hand those
sorts of events focus around the how as opposed to the why so there was a real gap there where
we wanted to create an opportunity for students to focus on the why behind these cyber incidents and
understand why do adversaries engage in certain ways why do what are the priorities of different
actors in this space I think one thing I'd like to dig in on is like how do we identify
how do we identify how we want to challenge folks I know one thing that we that we've thought about
with cyber 912 in our scenarios is we want students to dig into attribution ideally and
Winona is cracking up because you know attribution takes time and she made a really good meme about
this or I think you like we treated a meme about this where one student was trying to explain like
I don't have an answer for you right now because we're still working through this these things take
time another part of another part of the scenarios that we focused on has been the vulnerabilities
equities process
yes you have a vulnerability but is the does the owner of that of that proprietary software
actually know that it exists are they engaging with a security researcher what does that process
look like does it have a national security implication or you know when does when intel
intelligence operations go all right go awry how do we ensure that you know nation states engage
in certain activities to collect intelligence sometimes this has implications on on broader
like cyber security policy challenges but not sure if you guys want to chime in on any of those
you've ever if you've ever identified a specific thing that you wanted to challenge a client or
a partner or yourself with yeah I love this question how how do we make this on still okay great how do
we make tabletop exercises the most uncomfortable for people in the room how do we scare them I
think that I try and do that because I think the more realistic an exercise and the mo you know the
more that we can get an actual threat actor behind it the more they get out of it so I love ratcheting
I think one way we do that I like doing this with ransomware because you know a lot of clients want
to see some sort of ransomware exercise see if they're prepared for it we try and pick a ransom
amount that's like a decent chunk of their yearly revenue like we actually try and find the right
number that will scare them enough that it's a lot of money and their insurance might not cover it
but it's just enough that we think that they can afford it and we also try and we try and
figure out what their crown jewels are so what what sort of
data if it was stolen would really hurt the company or or the customers or would raise pretty
significant legal and regulatory questions and then when we combine all of that into a sort of
powder keg we get something that that doesn't shut them down right if it's too scary they will
say you know in this scenario the company would fold and we cannot do anything about it so we
want to keep it within their reach but we definitely want to make it stressful enough that
it really tests their processes so I especially with ransomware we like to ask a lot of the the
hard questions that people not might not realize that they need you know what's your insurance
coverage how early do you need to bring in your insurer to make sure that they will actually
cover a cyber event a lot of folks at the end are like and then we would ask insurance for 10
million dollars no yeah see people are shaking their heads but that that nitty-gritty stuff is
not self-explanatory and it's really important to make sure that
people have those nitty-gritty processes down do you have external legal counsel does your
external legal counsel have any cyber expertise like would they actually understand what's going
on right now and be able to advise you about what sort of statement you should put out and
whether you should use the word breach um or ransomware attack or you know disruption uh
service outage um those sorts of things are really really important so to answer your question um
trying to find the right ransom amount poking at specific processes especially with ransomware
that might be a little bit more expensive than the one that we're talking about right now we're
trying to find the right ransom that might not be self-explanatory and making sure that the
pressure is high enough that the exercise is useful to them but not so high that it makes
them disengage yeah and i mean even on your point sarah i think that one of the things that's been
most beneficial at least for me when i run some of these games is to be you know you want to make
sure that they're learning but also to be a sounding board such that when they're doing
the after action they go oh crap we don't actually have a relationship with our fbi
i grab the other branch office and you're like well like here's you know that local office in
your area here's their contact um and making sure that it's not just about the technical
aspects and running through the internal processes but making sure that you have the external
as well i'd like to open this up to our guests here if you if you have something you'd like to
add if you've worked on a war game with a partner or with your organization and how you try to you
know kind of i think we talked about poking holes and processes poking trying to find those gaps
and i've been immediately able to help out with the last few cases and i think being in this
the fbi team please feel free to chime in i see some heads nodding um and i don't want to call
anybody out individually but we'd love to hear your thoughts as well just because you do want to
make this is a round table we want folks to chime in yeah please yeah you guys can pass
mics too yeah just be gentle with them testing um and your kinds of experience like what what is the
I mean at least for me we've never run anything with jointly with the FBI I
think making sure that first off I might not actually be the best person to
answer because yeah yeah I think the number one thing is knowing your local
field agents do you have do you know who to call a lot of people don't and when
you're in the middle of a stressful situation trying to figure out that
information is very difficult so I think to the extent that you can establish a
relationship for a local field office that you at least have one person in the
company that knows one person there that can call hopefully multiple but I think
the other thing that folks don't think about law enforcement is like when do
you want to engage them do you want to engage them at the beginning before you
really know what's going on that might be a waste of their time do you do you
want to engage them and tell them that you're thinking about paying a ransom
I'm not sure I think it's companies don't really think about some people
that we tell them that you're thinking about paying a ransom I'm not sure I
talk to are like absolutely as soon as we see like any kind of ransomware we're
talking to the FBI and then we ask how do you think the FBI will help you
because I'm a big fan of the FBI but their priority sometimes is these
long-term attacks against threat actors right they're not necessarily there to
be the fire department for any individual organization and so I think
setting those sorts of expectations for when you when you contact law
enforcement are you expecting them to come
and fix things some people are and i think it's important to communicate to them the engagement
with law enforcement is going to be about sharing information you may not get anything back you may
not get anything back for a while and that's that's important that they have other folks
that they're prepared to rely on and that they don't think that law enforcement is going to come
in and fix everything because they just don't have the capacity and i also think that it's
just a slightly different priority than the organizations please well my my question i work
in financial services and i've actually ran and contributed to some of these attack scenarios
under more of the bcpdr testing so i think my two questions would be um concentration risk
we talk about that a lot in financial services a lot of time marketing people give up really
really good information in osn oh we have this strategic
partnership with x y and z bender me on the attacking side is like yeah it's a good place
to start to go after them because that's going to be my pivot into the organization the other aspect
is more of what your thoughts and feelings on bringing that concentration risk into your
scenarios the second side of it is threat intelligence who is attacking you how are they
attacking you and how do you play that into your scenario so that you know the av it's basically
doing a threat model on your own
scenario and how what your thoughts and feelings would be on that and then the follow-on is
given what we're talking about here why aren't we having a ctf in relation to it in in a future
defcon because that i think that's a perfect outplay i i think it would be wonderful to have
that so your thoughts your feelings so i can answer the third and fourth question um and so
the fourth quest i'll i'll do it backwards uh we're not having a ctf for defcon policy
because this is the first time we've expanded so rapidly uh as defcon policy so we'll take that
feedback and we'll see if we can do that next year um as for the threat intelligence aspect i mean
i think it depends on a your relationship with your you know end customer right like how much
are you tailoring a war name or tabletop exercise ttx whatever you want to call it to their needs
are they specifically asking for nation state threat that is going to target them right and
then how good is your relationship as that war game relationship as well okay so i want you to
game planner with someone who understands that type of threat actor are and finally why do they
want this in the first place is it like a bragging right like can we successfully defend against apt
attacks or is it like we want to be able to map uh you know how well we actually apply our defenses
to the mitre attack framework and if that's the case then like that might be a little bit of an
easier scenario right um but then once you've figured out you know which minor attack uh you
know kill chain whatever model you wanna you wanna go for what threat actor actually exists
out there that is going to map very neatly and it's never gonna be one-to-one um and so setting
those expectations with the client making sure that they understand you know if this is going
to be this way it might not hit every mitre attack boss that they want um and then also making sure
that they then have the internal processes to do that type of defense against an apt um you know
do they have
their own internal threat intelligence capability what is that like how does that you know internal
team uh interact with the wider detection and response board and and that is a really good
muscle to test in my opinion when doing those sorts of apt level uh attacks thank you i echo
what winona is saying about that um we try and pick an actual threat actor and say you know these
are the sort of behaviors they exhibit um we want to make sure that we're picking the right one for
the client like i don't want small businesses worried about spv that is not what they should
be concerned about
and it's tough because there are all these tools out there and they all seem really useful
but they're not all useful for every company and so it's interesting that we'll come into
to someone in their environment and they have all these fancy tools
about threat intelligence which is great but um they don't have mfa coverage like it's it's
really interesting the way that folks prioritize and i'm not blaming them whatsoever but we want
to make sure that we're picking a threat actor that's actually going to attack them so they
can prepare for something that is actually going to happen um that we can we can pick um common and
entry points like phishing gets used a lot obviously it's very boring but it happens
are they ready for a phishing attack do they even know what that's going to look like would they
even be able to catch that um another thing that that we do is we pick you like i said specific
threat actors so we use conti for example um we use the same ransom note that conti actually
sends out um it'll have sometimes their name in it and we'll say do you know conti um do you know
where they are do you know what sorts of tactics they're using are you able to do any research
this is for technical teams obviously but is there anything you can do um research-wise
on this threat actor that you can bring to leadership and say you know this is
the reputation that they have as far as whether or not they leak data after getting
ransom paid like that sort of thing like i said threat intelligence is usually the last
thing that companies do on their maturity so usually they're not prepared for that sort of
thing but there is a lot that they can do just with the information of the threat actor from
the ransom note so we try and encourage them to do a little digging and see see if they can use the
use that to help them during the exercise.
I want to foot stomp on the, like, this company
wants an APT war game, even though they shouldn't
be worried about APTs.
Like, threat intelligence, yes, is important,
but only if it is relevant and actionable.
Otherwise, it's just threat news.
It's not.
And also, I don't think we mentioned, but the three of us
are all DEF CON policy goons this year as well.
So if you have any questions about DEF CON policy
afterwards, you can talk to us.
Like, someone had a suggestion for running a threat scenario
next year.
And like, we will actually take that into consideration,
because we're going to be planning it, so.
If I could add two pieces really, really quickly,
just on the CTF piece.
We've been talking for a while, Trey and I
have been talking for a while, about trying
to find a way to collaborate with CTFs and hackathons
and have the Cyber 912 storyline.
Because it is a story.
It's a scenario, but there is a storyline.
There are characters, and there are actors,
and there is an arc, et cetera.
How can we collaborate with CTFs
and hackathons so we can create learning opportunities that
are multidisciplinary, where you learn
to analyze lots of information in a short period of time,
learn how to present that to decision makers,
but also learn the technical aspects of it
as well with your team?
So if you have any feedback on that, please let me know.
We're really keen for that.
The other piece, if I can add a tiny bit of contrast,
is sometimes in some of our simulations,
we try to lean away from nation state threat actors,
just because it's easier for students.
There is a policy framework for how to engage.
There are diplomatic channels.
We want to challenge them.
We want them to be uncomfortable.
So engaging with a non-state actor
often pushes them in that direction of,
you just can't call for sanctions.
It's not a thing.
So wherever we get in, students like sanctions.
They like to, it's a very American thing.
So we like to make sure that they're not just
leaning towards that first tool that's ready and getting
that knee-jerk reaction.
But Chris, I saw you .
Just a couple things.
One, on Sarah's point about law enforcement,
I agree they can't solve everything,
and they're not going to prioritize everything.
But I think the answer, in part, is for entities
to have an incident response plan in place
that includes who they're going to call in law enforcement.
So in other words, it's not just personal relationships,
which are important, but also having a plan in place
so you're not scrambling when something happens.
And there's so many places that don't have that still.
It's amazing.
So that's important.
And law enforcement, and also to have a relationship before.
I mean, join things like InfraGard and things
like that, just to have that relationship so you're not
just coming in cold.
On the idea of communication.
Tabletops and storytelling, two things.
One, I think kind of an unserved area
is not the policy, policy level, but at the board level,
and getting them to understand it.
And they're not going to understand the technology.
But the side benefit of that is getting
the technical people in a company like the CISO
to actually have some access to the board.
And so thinking about how you can do that in a good way,
I think, is important.
And then in all the tabletops I've done when I was in government
and others, the biggest mistake that I see made all the time
is breaking everything.
So that you can't, you just, it's hopeless, right?
So I remember all the cyber storm exercises.
I remember when I was in government saying,
if I'm here for next year's cyber storm exercise,
please kill me.
Because every freaking year, whoever designed it
wanted to be clever and so broke everything.
And so you don't learn anything if everything is broken.
So focused exercises, I think, are really important.
Yeah, I 100% agree.
And just to clarify, I don't want to come off saying
that you shouldn't report to law enforcement.
You absolutely should.
I'm, in my personal capacity, very in favor of it.
Like, it helps us, because I do policy stuff, too.
It helps us get more data, which helps us get more funding,
which helps us make more nationwide decisions
about stuff like ransomware, ransomware threat actors.
So I do think that contacting law enforcement,
I didn't mean to come off and say that you shouldn't contact
law enforcement yet.
General counsels and others
that tell people, never call law enforcement.
Yeah, yeah.
Yeah.
And the technical capacity thing,
I'll just say one thing really quick,
because I want more questions from people.
I think that the tabletops are a really awesome two-
to four-hour way of getting really good exposure
for your security teams, getting executive support,
teaching management how valuable the team is,
helping them realize that there are still maturity gaps,
even though the program is going well.
I think that's something that we try and make sure
that we highlight the security team is doing good things
while still saying, but they need more funding,
more resources, more people.
It's actually a very good way to do that,
a very quick way of being able to get more exposure
and more recognition, and hopefully more buy-in.
We often see after tabletop exercises
that executives will approve another chunk of funding
for folks, because they actually get
to experience what would be like if things went wrong
and don't want that to happen, so.
Excellent.
I think we see two people over here.
So you first, and then you next.
Perfect.
Well, on the topic of notifying law enforcement,
we've got a rapidly changing and shifting,
regulatory environment.
So to what extent is it important to work
regulatory compliance into the tabletop so you say that,
all right, we're coming up on a point at this hour.
You must inform law enforcement or relevant stock
regulators, that sort of thing, so that these regulations become
forcing mechanisms on certain communications.
Yes.
I mean, it's partially like two things, right?
Like the first is that because it's regulation,
you want to make your war game or your tabletop
as realistic as possible.
And so putting that in is not only that forcing function
that the teams need, they might not even know.
And so it's always fun to make that choice of whether or not
you want to inform them that this is the parameter
of the exercise, and you know, by the way, in 24 hours.
Or at 23 hours, you can say, OK, this is what we want to do.
You can go in and go, by the way,
have you thought about your SEC obligations?
And you know, it's a choice that you make when you're designing.
But yeah, short answer, yes.
Yeah, there's an even easier way to do that,
is what I like doing, is who's in charge of that?
Who's your data privacy officer?
Have you talked to them?
A lot of folks, at least recently,
I've asked about breach disclosure requirements.
And someone a few weeks back was like, well,
you know it was an accident.
So we don't have to report the breach.
And I was like, oh my goodness, there really
is a big range of understanding, both in management
and technical folks, about what the implications of losing data
is.
That is the biggest.
I know that there are lots of cyber reporting requirements
that have sort of been in flux.
But even the state breach reporting requirements
have been there for a while.
And many, many states have them.
And so I'm asking, who's responsible for that?
And are they in the loop?
And most often, the answer is no.
And I write lots and lots of records.
I write recommendations about, you
need someone who understands your reporting requirements.
They need to be internal.
A lot of folks are like, yeah, well,
ask their external counsel to do.
I'm like, do you know how long it's
going to take for you to do that sort of thing?
Sometimes it takes months, maybe even years,
for really large organizations that have lost a lot of data.
So I'm always checking to make sure
that they have an actual resource with that knowledge.
Because I think maybe it's more useful than me trying
to track every single requirement and being like,
you're in the state of New York.
And do you know that you have to report to these entities,
specifically in New York?
But I want to make sure that they know that.
The gentleman over here.
I'll slide in from the back bench here.
So to what extent do you think we
need to start modeling cyber-enabled long-term cognitive
warfare campaigns that are being directed
against our social institutions?
So that's something that, especially
for senior level national security
decision makers, they're very, very focused
on the in extremis, critical infrastructure attack
or financial institution attack.
But I don't think people are calculating
in foreign adversaries using social media over a long term
to create social disruption and undermining institutions, which
create systemic problems.
Thanks.
So I love the vagueness of that question.
And in terms of long-term information warfare
combined with cyber, right?
Because I'm headed into cyber law at this point.
And so I'll give you the lawyerly answer.
And it depends.
But it depends on what particular simulation
are you trying to emulate?
Are you trying to emulate a culmination
of a lot of internal dissent that
was fomented by a foreign adversary over social media?
Are you looking at a shock and awe in terms of,
here's a cyber-enabled attack that also comes
with a follow-on of information dissemination that sows chaos?
And like, I personally, from a threat intelligence perspective,
I think that the information domain, as well as the cyber
domain, are inextricably linked.
And adversaries understand that.
That's why you see the IRA alongside other Russian APTs.
And it may not be entirely cordial,
and it may not be entirely coordinated,
but having similar mission sets.
And so I think that when you're talking about the adversary,
when you're talking about designing a war game with
a nation-state level threat in mind,
you have to consider both.
However, that does depend on your primary principle.
If you are designing a war game specifically
for just the technical boots-on-the-ground type IT
defender, that's going to be out of the question.
If you're designing it specifically for, you know,
DOD policy, someone a little bit up the chain, then yes.
Because information space, as well as cyberspace,
are pretty inextricably linked.
The only caveat I would say to that is it depends on which
adversary you are trying to emulate.
Because while Russia and China may understand that,
you know, they call it informationization,
intelligentization is the new Chinese translation word.
It's, you know, going to be different if you're emulating
an actor from Iran, or an actor from North Korea,
or an actor from Vietnam.
So I think it depends on the principle,
it depends what area they're particularly interested in,
and it depends on how long-term of a war game you're looking at.
Yeah, and I defer to Winona on this one,
because she's the expert here on APTs.
I'm not sure that the tabletops and war games are as well-suited
to the sort of long-term,
vague scenarios, because I think it's really important that
they're very targeted and they have takeaways at the end of them.
And I think the more vague the scenario gets,
and when it gets to be on that scale,
you know, information warfare on a national level,
it's hard to walk away from that knowing what to do to fix it.
And I don't want folks to sit in a room for five hours and say,
well, that was interesting, and then go home.
I want them to say, I know what we need to do differently.
And that's why I think they're so useful for things like
ransomware, for, you know, even espionage campaigns,
data loss sort of campaigns,
because you can walk away with them and say,
I know what I need to fix.
I know what we need to do better next time.
So I, not that they can't be done,
but I'm not sure that they're as suited to those sorts of scenarios.
Do you want to jump in? Please.
I'm just going to add a boring think tank perspective,
just because, one, I can, we can make a cyber scenario,
but our colleagues who handle information,
information programming can really, really dig into
how is information manipulated and used to achieve broader geostrategic goals.
I think part of our job is, you know, making sure that
we are putting our heads together with our colleagues
to make sure that we are developing scenarios that do examine
those stickier issues that we don't quite understand just yet.
That way we can create more challenging, high-quality scenarios
that are beneficial for our friends in government.
The other part of that is that, one, we make a scenario,
students respond to it, folks from government and industry are judges,
but after the fact, it's kind of our job to make sure that we pull that scenario
and the recommendations into a playbook and make that available to folks.
Our goal really is to be able to pass that on to folks in government and industry
and have them take that and pass it along to somebody in their organization
who can run it internally and take some notes on,
this is how it rolled out in a scenario, which was obviously fictional,
but how would we actually respond to this if this were to happen tomorrow?
Safa, I have a question.
You of all, I think everyone raised their hand.
When we asked who's been in a war game or in a tabletop exercise,
has anyone hated, has anyone disliked them?
Have you been in a bad exercise and what went wrong with that?
Because I think a lot of folks hate them.
Insurance requires them these days for a lot of companies.
I'm curious if anyone has had a bad experience and what you think contributed to that.
It was a budget tabletop exercise.
What does that mean?
It was trying to figure out how U.S. government would route funding
in a given situation based on international trends.
It was delightful.
You mean like a tabletop related to budget,
not like a very cheaply made tabletop exercise?
It was not a poor man's, poor person's tabletop.
No, no, no.
A lot of high-level government people there know,
but it was focused on budgeting.
Okay.
I've never heard of that before.
I haven't had a tabletop that I didn't enjoy doing,
but working with clients, the worst ones that I've had are where,
the client was very immature, maybe had never done a tabletop before,
and they really wanted to get fancy in some way.
And they weren't ready for it because they're still at the point where they don't know
who to call to make a decision or to get information or whatever.
And if they're in a situation like that and they're worried about a nation-state act
or doing some sort of supply chain attack or whatever,
well, if that happens, you're just going to get run over, right?
Because you're in no way prepared for that.
And it would be much better to just deal with a basic data breach
or a basic ransomware scenario.
And once they've figured that out and they actually know their own processes
and kind of iron things out, then they can start to think bigger
if it's relevant to them.
Yeah.
I'd love to hear both of your opinions on this.
Oh, sorry.
Go ahead.
Go ahead.
Use the mic if you don't mind.
Yeah.
Just so .
And also, over here.
So I helped run about 10 exercises.
And invariably, what would happen is we'd have 20 to 50 people in the room.
These were multi-day events.
And they were told they had to be there.
This was for the DoD.
And invariably, when they show up on the first day, it's this.
But by the end of the first day,
everyone's leaning forward, elbows on the table.
They're all interacting.
And they were surprised.
This was the first tabletop like this for anyone, even us, in putting them on.
But they were all surprised at the value they got out of the exercise
because it wasn't what they were used to from a security-type assessment.
Yeah.
And I see we've got a question over here.
Adding on to that is one that I worked on that I really do like.
One of the things I really did not like was the element of they knew that they were weak in a very certain element.
So they tried to put that as like outside of the rules of engagement.
It was like, but this is the reason for it.
And then it was the same element of, well, I don't want me to look bad because I allowed this to happen.
So it's getting the individuals that are in charge to accept, hey, allow us to exploit what we've already told you
or what you've already noticed.
Because if you notice it, then that means it was noticed way before you noticed it.
So it's just that line where, like also you said, you don't want to freak them out to the point where they don't want to do it
because they're like, fine, we'll just give up.
I want to lean into this a little bit because this raises a really interesting thing for us, I think, to dig into is
what do you do with an immature client or an immature player?
And I want to lean in on the player piece because a war game or a simulation or a TTX, we're all players.
I think we've seen this with students where if they're an immature player, they tend to, the knee-jerk reaction is like, it's Russia or China.
The other thing that we see is sanctions.
The last thing that we see is we want students to come up with creative solutions, whether they be technical or policy-centric.
We want them to be creative.
We want them to think outside the box.
This is a tabletop exercise.
You can lose, but at the same time, losing is taking no action.
So shoot your shot.
Some students think creativity equals escalation, which gets really uncomfortable really fast.
You'd be surprised.
The military academies tend to not be as escalatory as the civilian schools.
Some students will think that, you know, let's do a military operation in this particular place because we want to demonstrate that we are,
that we care about this thing and we're watching, but it's like, is that actually, like, sir, the lights are still off.
Please actually respond to the crisis at hand.
But we'd love to hear from folks around the room.
What do you do with an immature player?
How do you get them to kind of reevaluate their standing in the game?
Can you define an immature player?
Sorry, what was that?
Can you define what you mean by an immature player?
I guess an immature player, I guess the way I see it is someone who is, one, maybe not familiar with engaging in a tabletop,
but at the same time, maybe not familiar or does not have a really strong grasp of what are they working with,
what are the tools in their toolbox that is available at their disposal.
But I see that you leaned into the microphone.
Please jump in.
So, to answer your question and also a follow-up question to that in regards to immature organizations going through the exercise,
one of the things that I found is very helpful is taking a step back and preparing,
like going through the prep exercises before the TTX.
So, for example, examining and developing the IR workflow and looking at playbooks and so on,
so that there is some sort of guiding, you know, guiding standards or guidelines
and something that they can refer back to that is helpful to make them not feel as fearful or as intimidated by the exercise.
But I would also like to hear your feedback on what you think about, you know,
you have a client who is requesting, you know, Wargamer or TTX and taking that step back and saying,
why don't you, you know, look at the preparatory exercise?
Why don't you look at the exercises before you get to that in order to increase the effectiveness of a tabletop?
What are your thoughts on that?
Okay.
I don't know if I have a lot of thoughts on that, actually.
I don't mind running tabletops with folks who have never done it before.
I think they learn a lot.
I think, in particular, immature players, like Safa was saying, vastly overestimate how good they're going to be at it.
That's what Sue was saying, that people want to get fancy with it because they think they're going to just do so well
that it's not going to be useful to them.
And even folks who have done it a few times still have things to work on.
And they're mostly boring things.
Like, folks will have these incident response plans that are 180 pages long.
They don't open them once.
They're totally useless to them.
It's very sad.
But it's the really boring things about not using your processes, not having any processes in place,
like not relying on your documentation, not having communications, like out-of-bound communications
in case your communications are compromised.
Like, it's sort of the same thing.
It just happens over and over again.
And so, mature players, immature players, everyone sort of overestimates how good they're going to be at it.
But you can run a very simple ransomware exercise.
And if you ask the right questions, you'll still be able to find some gaps in the response there.
I saw two hands up.
Well, I don't think we've heard from you before.
So, what I usually do with my immature players is I start by asking for—
well, during the scoping of the entire thing,
I start going through all their policies and procedures,
and I'm not asking for them for me.
I'm just asking to see if they have them.
And with one of my most immature players,
I actually just told him to step back from the tabletop completely.
We switched the entire thing to we're going to do micro-exercises
so that way we can go through each of your policies first,
and then we're going to go through a larger tabletop exercise
because I found that they had gaps.
Everywhere.
From communication to governance to even just the technical incident response team,
everyone had gaps.
So, kind of like we do with technical teams,
because this was supposed to be a management exercise,
and we took it down into technical pieces,
but technical at the different policy levels to tear through different policies.
That's how we did it.
That's a really good point.
I think there are things you can do with a really immature client
to sort of hand-hold them through it.
You can also run—
we've done this before,
like tabletop exercises that are more educational.
Like you bring them—you say, here's a ransom note.
What happens now is that you'll need to engage a ransomware negotiator
because you don't want to be talking to them.
So, we'll sort of teach them as we're going
what they need to be thinking about versus expecting them to know it
because we know that they don't know it,
and we don't want an empty, quiet room.
I think we have two fingers here and a comment over here.
Just a quick follow-up for that.
So, we're talking about organizations
at a level of immaturity.
So, when we take a look broadly at a capability maturity model,
okay, if you're at a certain phase,
what organizational pieces do you need to put in place as the building blocks
to get you to a more mature state?
And I don't think we have that on a spectrum yet
where we can kind of say,
okay, based on this exercise,
you're at this level of maturity based on X, Y, and Z.
Before the next exercise,
we need you to improve in X, Y, and Z.
to get to that level two or that level three.
So, an organization understands throughout the organization
what pieces need to be worked on by which levels of the organization
to improve their capabilities to defend against these cyber threats.
So, I think it's really kind of vertical going all the way up to the board
because they're on the hook for it also.
But once we get that maturity model in this space,
I think that's a good guiding measurement.
So, I think that's another kind of more enterprise takeaway.
How do we measure maturity?
Because you can't manage it if you can't measure it.
So, thanks.
And I think if we can get this gentleman over here
and then we'll have your comment.
So, please.
To just address a couple things.
First of all, the one lady was talking about setting it up early
and kind of giving them some pre-advice.
When it's a first time for a client,
I found that works extremely well
because they don't even know how to play.
You can put all the documentation out there,
but walk them through the basics.
And I also agree,
if they do not have any sort of IR plan,
have some pre-sessions.
Help them at least create a straw man
because what you're going to do is you're going to...
In many cases, you've got leadership in these calls
along with the people that are supposed to be managing the incident.
And you really don't...
You don't want to put them in a position
of looking extremely bad in front of their leadership.
Or if you're doing it at the higher levels
where you've got directors that are being watched over
by board members or executive officers
and you don't want to put them in that level.
You want to kind of help them be successful.
The goal of it, for me,
the goal of these is to learn.
Identify gaps.
Takeaways.
You've all spoke to takeaways.
If you walk out of one and you're like,
oh yeah, we did great.
It wasn't scoped right.
Bottom line.
Scenario wasn't right and it wasn't scoped right.
Because the idea is to find something to work on.
Because none of us are perfect.
Believe me.
We've all been in real incidents and we know that.
The other thing...
I forget who was talking about it.
We were talking about the level of the incidents
and who all is involved.
And so if you're starting with someone
that's never done this before,
you start at the high levels.
You work through process stuff.
Don't go deeply technical.
The scenario should have technical aspect to it
and it should be relevant.
But that shouldn't be the key focus.
When you don't even know what you're...
Who to call and what to do
and when to engage law enforcement,
which you...
My quick comment there is
get your legal involved before you call law enforcement.
That has to be a decision made by legal.
And if it's not in your playbook to do it that way,
you need to change your playbooks.
Because you need your legal involved.
Also, data privacy,
it should be involved at step one
throughout your entire IR process.
And they should be involved in your tabletops.
Because they're going to make decisions
and they're going to give guidance.
It's very important to actions you take.
Another quick thing.
Sorry, but I've sat here for a long time
and not said a thing.
I have done a lot of these
with a wide variety of clients.
And what I've found is that as you start
and then you start having...
subsequent engagements,
build a roadmap for that client.
Let them understand at the beginning,
at the very first one,
this one's going to be relatively high level.
It's not going to be deeply technical.
We're going to get the right policy people involved,
maybe some leadership involved.
Then, as you progress,
you can have many tabletops
that are highly technical,
that are focused on a specific environment
or focused solely on your SOC and their response.
And then you can build that out into bigger ones later,
where you can actually split it into where
you've got leadership in one room on a call.
If it's physical, they're in a room.
If not, they're on a call.
You've got technical on another.
And now you're managing it the way
typically most organizations are going to manage it in real life.
Where you've got multiple different party engagements
and they're not always sitting on every room.
But you don't do that step one.
That's a roadmap.
The one gentleman here was talking about capability maturity.
And I totally agree with that.
And I don't know that there's a model out there for it.
But I will tell you,
the clients that do this year after year after year,
twice a year, three or four times a year,
they have many engagements.
They're far more mature in their incident response
than the ones that say,
we're just going to do it to tick a checkbox,
just to check a box.
Anyway, that was a lot. I'm sorry.
Sorry you should be on this panel.
That was great.
What we did was transparency.
We made sure that the participants who were coming
had the opportunity to understand
what was going to happen before they got there.
And then we had the luxury of time
because these were multi-day events.
So we'd spend the first half day
walking them through the process.
This is the process we use.
This is what we're going to be doing.
After we found, we said,
these are your threat actors.
And we did research open...
We did research on the systems we were attacking.
Basically, the theory was,
if it's unclassified, it's available to the attacker.
And we would use that to come up with attack scenarios.
And we would tell them beforehand,
these are the attack scenarios we're going to be using.
And this is how we're going to determine risk.
We walked them through the entire process,
all of them,
all of this before lunch the first day.
And then when we got back from lunch,
we'd go ahead and start on scenarios.
Thank you.
And I think we have the gentleman over here.
If you could use the microphone
and then pass it to this man over here
or to the right and then to the left.
Hi.
I had been involved recently
with a multi-agency tabletop.
And it's a fairly mature organization,
but it was a well-scoped scenario
because it made all of senior management realize
that they hadn't actually read those incident response plans
and they sort of didn't know who to call.
And it was a little bit embarrassing.
And they felt a little bit like the Keystone Cops.
So the point of this is that the important tabletop exercise
is the next one.
Because these...
I mean, like I said, it was a mature organization.
And I've seen a whole lot more collaboration
between these organizations.
Not just to prepare for incidents,
but they realized the skills that each other had.
And now they're working together in other ways,
not just for incident response.
Second comment.
We have used in our technical TTXs,
we use a Dungeons and Dragons type card game
called Backdoors and Breaches.
So if you haven't...
There's got to be a way to put policy into that.
And we get so much engagement from that.
It's awesome.
If I can add one thing on the D&D,
my team actually really creatively put together
a D&D style game
where folks navigate an incident
and are trying to pull the information out little by little.
But it was really, really exciting.
But if you wouldn't mind passing the microphone to your left.
People here play Dungeons and Dragons?
Incredible!
I knew there was an overlap.
Half the room, yeah.
I just wanted to give a kind of like a free recommendation
because I know we were talking about immature players.
And I think for a lot of organizations
that are just kind of getting started,
don't have a lot of internal resources,
like an internal data privacy officer
or experienced legal counsel on the inside.
It's been mentioned before,
but I just kind of want to foot stomp in the room
that you should work with your insurance carrier.
Your insurance carrier has a board of outside counsel
that you should tap.
And I think like everyone's very familiar
with vendor assessments.
For their technical vendors,
I think something that's kind of overlooked for new players
is vendor assessments for your outside counsel.
If you really do want to get prepared
for dealing with a large-scale breach
and your plan is just to reach out to outside counsel,
I've seen so many clients that are like,
yeah, we'll reach out to outside counsel.
And they time it right.
They give a right threshold for reaching out,
but they don't know who.
But on the other hand, they'll know like,
oh, if we need to negotiate, we'll tap Coveware.
Like, why do you know that you need to tap Coveware,
but you don't know who your outside counsel is?
It's kind of like your other big player, right?
Probably bigger for most organizations.
I also wanted to ask, I think for like on another thread,
for larger organizations,
how have people like successfully scoped
for these large-scale conglomerates
or giant tech companies
and figured out a way to scope the exercise
so that it's not Microsoft blows up and disappears,
but is there a way
where it's like useful, but they're also gigantic.
So it's difficult to make that scope.
I'm wondering how people tackle that problem.
Please.
More kind of use case to add to the question
in terms of, and this ties back to your question
on kind of US national security and that sort of thing.
Thinking about the ecosystem in a given country
where we have the US embassy, the USAID mission,
all of our NGOs and partners that we fund
that are collecting data and doing work,
all kind of under the overall guise
of US national security and objectives.
Thinking about how tabletops include
or don't include all of those diverse actors
or if you just kind of proxy them.
And so that's really interesting to me
to see how to either run a more complex one or simplify.
So this is less from a policy-oriented perspective
and more towards like red team engagements, right?
Right.
But just to answer your question directly
in terms of like Microsoft blowing up,
I did threat intelligence at a fairly large tech firm
for a little bit.
And when we were running not tabletops,
but red team engagements,
usually the red team gets carte blanche,
but they'll scope it and they'll consult
with the organizations that they're planning on targeting.
And so I think it's truly a conversation
that you want to have with your client
and just say, listen, how is everything in scope?
If that's the case, still choosing a small subset.
At least from a designing perspective,
it gives you the creativity to be able to work
with what you have.
But it truly depends on, I guess,
going back to the maturity model of that organization
and how closely knit those disparate teams are already.
Oh, go ahead.
Someone else has a question?
On the conglomerate thing,
I think it gets really, really tricky
to run tabletop exercises
when you have an umbrella organization,
a bunch of smaller organizations.
There's a few different ways to do it.
We've done one where we run management exercises
for both the umbrella organization
and each of the subsidiaries
and ask each of the subsidiaries,
what are you expecting from your parent company
and what do you think you're supposed to be doing?
And we usually get totally different answers from all of them.
And then we run one with the umbrella organization
and we're like, well, what sort of information is coming down
and what do you expect to come back up?
And then what we can do is sort of map all of those
and produce an overall report that says,
here's where the communication is going to break during an exercise
and here are the subsidiaries
that are expecting you to do all the work
and are going to need your help
and might need more resources
than they're currently being allotted.
So it does get very complicated
when you're doing maybe 10 exercises
that are all sort of linked to each other,
but it can be done.
I see two fingers here
and I'm going to ask a question
that might shift gears a tiny bit,
but defer to you all.
So the other aspect is
taking a look at companies that grow by mergers and acquisitions
because there's a lot of inherent risk
with the company you're bringing on
and their risk profile
and what their capabilities are
and who they may be connected to.
So we see this a lot out in industry
where companies are growing by M&A.
But from an attacker perspective,
that's a juicy target for a simple fact
is it's going to be a lot of disjoined companies together
that may or may not have their act together.
In financial services,
there's two things that we've always concentrated on.
Anything can affect the GL.
Anything can stop processing card information.
Anything can stop major financial transactions
within the financial services organization
as scope areas for scoping.
Because if you can't get those three things right,
that's making your financial service organization run
or the question of how do you make your money.
Those are the areas that we've concentrated on
from a scope aspect.
And then we've seen, you know,
where there's been breakdowns
in outlying or supporting functions
or supporting vendor organizations
that integrate with that supporting function.
So you start to get into,
okay, vendor risk management
either as a force multiplier to your organization
or a detriment in your IR response
and things of that nature.
So that's how we've kind of focused
in the financial services from my experience.
Again, things may vary,
but that's just one area to scope aspect.
Thanks.
I want to tie that into kind of the conversations,
like the subsidiary in the M&A
with kind of the outside counsel
or outside vendor recommendation.
I'm dying to see if anyone has ever done this,
a incident scenario in which the attacker
has started from the outside law firm
and moved inwards.
Because I think that that would be
kind of a beautiful tie in to like,
oh, we'll reach out to outside counsel
when like they're the ones that cause this whole thing.
Anybody?
So yeah, I just had a question.
So a lot of the conversation has been about
how if you're scoping a tabletop exercise
that you succeed in,
it's the wrong scope.
In a lot of like self-defense courses
or that kind of training curriculum,
one of the concepts is you want to lead the student
with the success at the end
because you want them to have the self-confidence
that they're able to do this.
We've got this.
We're able to handle these sort of things.
Is that something that we should be incorporating
into our tabletops as well?
That maybe we give them like an attaboy at the end
to give them that self-confidence
or are we just really focusing on the negatives?
I think delivering feedback
is such an important part of tabletop exercises.
I think starting with what was done well
is always important.
But what we like to do at our practice
is ask them to,
say what they think they could do better first
because it just sounds a lot better coming from there.
Like, you know, we really didn't use our IR plan
than us saying you did not use your IR plan.
Bad.
It makes a big difference
because then they'll all sort of talk about it themselves
and we'll come in at the end and we'll say,
we totally agree with this.
You know, we'll add a couple of things.
And then folks walk away feeling like
they've taught each other something
versus us like kind of bringing it down upon them.
It's just sort of received a lot better.
But even with the sort of reporting,
reports that we're giving out,
it's always here's what you did well,
here's what you could work on
because if you just tell someone
everything they did wrong,
they just feel like a failure.
So you're right.
You don't want them to feel like they owned the exercise,
but they definitely need to know what they're good at.
Or they shut down and the next time
that area is out of scope for the next tabletop.
I can jump in really quickly
just because the feedback and evaluation part
is really interesting,
especially because we work with students.
So you want to be extra sensitive.
So the fact that like,
hey, you're new here.
It's okay to make mistakes.
This is how you improve for next time.
And like, we have a rubric that we use
for all the briefings that they give evaluating,
you know, one, their presentation,
the policy recommendations they develop,
how they navigated working with four different people,
three to four different people on a team.
I think one thing that's been,
I think, challenging for us is one,
making sure that when they do present that,
they give that to a series of judges
who are from industry and government
who are practitioners.
We want to make sure that that feedback
is valuable to them,
but it's also, it's also constructive
and that you deliver it in a way
that focuses on the positives,
but also identifies areas for growth.
And the growth part is interesting
because you want to make sure
that students use that for the next year
when they do compete again
so they can continuously get better.
But one thing that we've had fun with
in some events is like,
we'll let some students redo on day two.
They don't, it's not competitive,
so they can't get points for it,
but you can take the feedback you got on day one,
redo your briefing on day two
and get feedback a second time.
So how could you redo this
and again, in a safe setting
using that feedback
that you got from people?
But I did see some, please.
Parallel with self-defense courses, right?
And those sorts of things,
it's basic instruction.
It's protect your head.
And like, how does that compare
to a much more complex tabletop
where the feedback or the solutions
that you're getting might be 180 pages of,
this is what needs to happen
in case of a crisis.
And so how do you not water it down,
but like right-size it
in terms of providing the people
who need the right steps and information,
give them the stories
and the tools that they need
that are actually actionable.
Yeah, too much feedback
is as good as no feedback
because you get, you reach a point
where folks just can't do that many things.
So you wanna pick out the things
that you think they need to work on the most.
I usually pick formal processes.
What can you write down
so that you'll have it next time
so that you don't have to think through it
the way that you had to do in this exercise,
because it's gonna waste time
and you don't have as much time.
It's gonna take you 10 times as long
to get all the people in the room
that you think it's gonna take.
So yeah, I like to keep it
to maybe less than 15 things they need to fix.
If you, and at that point,
if you have more than 15 things to fix,
you sort of need to fix everything.
And so you might as well just keep it
to the things that they need to do first.
And so the next time they come on the tabletop,
you can sort of drill down deeper
into the more complex processes
that they could figure out better.
But you wanna hit the things
that they can do first that are easy.
Also, just to add on that,
I mean, one of the things
when you're coming back with pieces of feedback
and trying to narrow that list down,
especially if it's a larger list,
at least for me, I focus on feasibility
within the organization.
I focus on, and when I say feasibility,
I don't just mean technical,
I mean political process-wise.
Like if you're going to tell someone
to fix their IT systems,
but you're a big company
with several mergers and acquisitions
and you're at this part of the org,
but the people that handle it is this part,
it makes no sense.
And then also focusing on
when you're looking at your principal audience
or your principal feedback receiver,
tailoring that message to more of a high level
if it's an executive versus like,
here's all of the different lists
of things you need to do
if you're more of a technical practitioner.
I think I saw a hand raised over here.
So one of the things I'd like to say too
is that for anybody who's doing tabletop exercises
is to make sure that,
and we said it a couple times,
make sure that third-party counsel is involved.
Because what I found in a few of the tabletop
exercise that I've done
is that the company has a incident response retainer
with one organization.
And that organization,
once third-party counsel gets involved
and cyber insurance gets involved,
is not approved by the cyber insurer.
So now it's in a very interesting position.
And we actually had to do this
as a live fire tabletop exercise,
which if anybody's ever done,
I don't recommend ever doing.
And they had to work it out in real time.
And there was conversations
that had never happened before about,
okay, well, we have this company engaged already.
They're doing analysis.
But now our cyber insurer is saying,
we have to go with this company over here.
What does that handoff look like?
And then what does the future plan look like?
Because obviously the company has one responder,
response retainer that they choose.
They have one of choice.
And now they have to work into the process
in the future to get them approved and that.
So highly recommend getting third-party counsel,
cyber insurance involved.
Any other questions from you two?
I'd love to give you guys a chance
before I throw a wrench in with some fun questions.
Trey's got a question.
Yeah, please.
So I'm a little bit curious.
We've talked a lot about the stakes
and mechanics of competitions,
a lot about adversaries and victims.
Mic, sorry.
So for you all,
we've talked a lot about the mechanics of the games
and the adversaries.
What have you found works best to build
the non-player characters, the intermediaries?
Because so often in Cyber Night 12,
one of the things we run into is students have trouble
interacting with the third-party company,
the other insurer, the downstream vendor.
And so making them credible and rich characters
is really important to the outcome of the scenario.
I'm just curious for all of you or for folks in the room,
what have you found works best
or what are tripfalls to avoid
in building those intermediary characters?
I mean, I don't think in corporate tabletop exercises
versus the Cyber Night 12,
they really have the same sort of rich characters.
But one of the things we'll do is that
if we're doing a combined technical tabletop
and management tabletop,
do the technical one first,
see what sorts of information
the technical team would actually bring upwards.
And then in the management tabletop,
we will pretend to be the technical team
and we'll say, you know,
the technical team is saying this, this, and this,
and they're asking this from you.
So that we're actually sort of creating that bridge
between the executive management, the security team.
And if they say, well, we'd expect this from them,
then we learn that there's going to be
sort of a break in that communication.
We don't have like really separate characters,
but we sort of play the intermediary
so that we can ask the right questions there.
Yeah.
I don't think I've built an NPC for my war games
aside from, you know, actual Dungeons & Dragons games.
That is a hobby.
But I think, you know,
I think I'll take that question
and maybe pivot it into a discussion
about technical realism in policy war games.
And there have been times where,
especially when, I guess going back to Sarah's point,
one of the ways to do that
is to have that technical war game first.
But making sure if you don't have that time,
and, you know, I imagine we're at DEF CON
talking technical to a war game agent
or something like that,
that you're playing panel
is not going to be a surprise.
But making sure that either
if you're not that expert,
making sure that you're checking
that all of the war games
from that perspective are realistic,
actionable, and current,
I think making sure that you have someone there
that will do that checking is important.
Yeah.
I have some,
I saw a wave.
I don't know if it was a hand raise or a wave.
I just wanted to chime in and say that,
like, if you get your IR provider
to, you know,
do the war game,
they work with the third party,
so it's easier to kind of create
those kind of third party NPCs
and simulate those interactions.
Excellent.
One thing I wanted to ask folks in the room,
is there ever,
was there ever a time that you or somebody
that you were working with
took an action during a war game
or a simulation
that had really bad consequences
and how did you navigate that with them?
Or did you just stand back
and watch the chaos happen
and just kind of smile?
More likely.
Yeah.
You're asking us,
you're asking the room.
I'm looking for you guys to go first,
and then...
You're asking for
on the record horror stories,
Safa?
I mean, no names.
No names.
I guess the one that I've had
goes back to setting expectations.
And I think that one of the ways
that we did not set expectations
the right way
when doing a tabletop
with this one set of groups
was just in terms of
these were two organizations
that had never really done
a tabletop together.
And due to that,
due to also cultural differences,
we came in with the scenario.
One team really went in
with the scenario
and was very forthright.
And the other team shut it down.
And we're like,
okay, what are you going to do?
Like, oh, we'll just figure it out
when we get there.
Or we will engage
with our third parties.
And then making sure that...
And when you have one team
that really actually wants to engage
and is providing that information,
you have another team
that's just taking it in
and providing nothing back.
Making sure that both of those teams
come in with the same idea
and with the same type of expectation,
I think, was something
that we really learned the hard way.
Yeah.
I did an exercise with the C-suite
and when we got to
the ransom repayment section,
unanimously and sort of led by the CEO,
the C-suite was like,
let them leak the data.
No one cares about our company anyways.
They won't write news stories about us.
Let them do their worst.
And I'm not saying that
they should have paid the ransom,
but to me,
it was such a mismatch
of understanding of risk,
which is something
that I sort of anticipated
that the C-suite
would have a better handle on.
So that was a weird exercise.
That brings up a good question,
and I agree to the question.
The original question
is that sometimes,
in my view,
sometimes having created them,
having also facilitated them,
as a facilitator,
I'll let things play out a little bit.
But at some point,
you've got to get it back on track.
When a major decision was made
that was incorrect,
you have to get it back on track.
But let it play out
and then there are usually
some good action items
that can come out of that.
But to some of the other points was,
and I just lost my train of thought,
so.
It was around,
what were you talking about
right there at the end?
Expectation.
Sarah was.
Oh, yeah.
The major decisions
that are just crazy.
Again, sometimes that comes down to,
and I've had that happen.
If it's an executive
making that decision,
there's only so much you can do.
And it comes back to then
maybe even having education sessions
privately
or making some recommendations,
not in public
because you don't want to shame somebody
at that level.
But those are very critical things
that need to be addressed as,
at least in my view as a facilitator,
I hate to walk away
and not address that in some fashion.
And in some cases,
some of that doesn't make it into the report,
but it makes it into a verbal conversation.
Anyone else was looking to chime in
on bad actions, bad ideas?
No?
I'll follow up on the Cyber 912 side.
Yeah, Safa.
Again, going back to the student angle,
it's really great to see
how many percentage of students
in the Cyber 912 challenge,
because it's usually like
a shock and awe type cyber attack.
The lights go off,
something's gone wrong.
You know, got to brief the NSC
in like 12 hours.
And there's always a team that goes,
it's trying to send carriers
into the Taiwanese street
or into the straight,
yeah,
that's what I was trying to see.
And being able to,
you know, walk that back,
go, okay,
like using the guiding questions of like,
okay, what is the basis of your analysis?
Like how confident are you in your assessment?
Walking them through
like the intelligence process of like,
here's all the things that you need to have
in order to make that assessment
and then making that into a learning process
rather than that was bad
and you should feel bad.
I think was definitely,
and it's been great.
Cyber 912 has had fewer recommendations
about sending the Navy to defend
or like to go into the South China Sea
for very little reason since,
you know, 2015, 2016.
I mean,
it's a really good point.
I think the escalation piece of tabletop exercises
is really important.
We'll get folks that,
you know,
the first,
the first hint of maybe an incident,
they're like highest severity rating,
ring the alarms,
which is not good.
And then some folks will say,
yeah,
I bet it's nothing.
And you're in a tabletop exercise,
so you know it's gonna be something bad.
I think hitting the sweet spot for escalation
where you're following with the information
that you have at the time,
but you're not doing an all hands on deck
before you've done any sort of analysis
and you can do something useful with that information
is very, very tricky.
And I think most folks either over
or under escalate during that incident
and aren't able to hit that sweet spot.
Is that what you see in Cyber 912?
I mean,
with the calling in the Navy right away,
you know,
yeah,
calling the Navy right away,
sending a carrier.
Yeah,
it's definitely like you can get either side
where it's like it's probably nothing
or it's everything.
And then I think that's a practice thing,
right?
Like you'll figure out like based off of,
you know,
what certain teams are saying,
moving it kind of more into the corporate
or managerial aspect,
depending on what certain teams are saying,
like listening to that
and understanding what vocabulary means,
what on their like this is messed up barometer.
Yeah.
And it also comes with like,
how well do you know your network?
How well do you know
what behavior is suspicious
versus what's probably some sort of outage?
That's questions that we ask all like,
is this behavior normal on this system?
And most often we get,
I don't know,
like we don't have a baseline for that.
I mean,
that's okay.
It's hard.
There's like a million things
that we've talked about today
that people have to know for these things
and they're not intuitive
and you have to learn them over time.
I like,
we only know this
because we run these all the time.
But on the other end of it,
it's not clear that you're supposed to know,
you know,
what the baseline is
for all these different systems
so that when you have an alert coming,
you can say,
that's definitely something
that we need to worry about
versus,
you know,
could be an IT incident
we don't really know.
I think you raise
an interesting point though
on like human decision making
and sometimes you,
someone makes a recommendation
or says like,
I want to take this action
and it's helpful to walk that backwards.
Like one,
what is your intended outcome?
What are you trying to,
what are you trying to achieve?
And then like,
how does this help you get closer to that?
And sometimes you see the gears turning
where like people are like,
oh,
my recommendation actually doesn't help me
get closer to that.
It's actually counterintuitive,
but I'm in a stressful situation.
I have limited information
and I had to develop a briefing in 15 minutes.
Of course,
I made this recommendation.
And I think that you see,
we've seen that in Cyber 912
where like the judge is saying like,
hey,
like tell me more about that.
Let's unpack that a little bit.
And then sometimes at the end being like,
are you sure you want to do this?
Because there's still time
when you can,
you know,
you can walk this back.
We haven't taken action yet.
But I'd love to get,
I guess I'm curious about like,
you know,
tabletop exercises
that you guys have participated in
or have developed
and that we do this all the time
because we're kind of crazy,
but we love red herrings
and we'd love to,
you know,
do something that kind of pushes the player
to take an action
with this limited information
that might not be helpful for them
so that they get more skeptical
and try to,
you know,
take the limited information
that they have with a grain of salt.
You're never going to have
all the information that you ever need
to make a completely accurate decision.
So how do you operate
with limited or imperfect information?
Have you guys ever seen any red herrings
in these tabletop simulations
or in the ones that you've developed
that people either latch onto
or take the wrong way?
Well,
open source malware.
Definitely doing those sorts of analyses.
And that goes back into like
the threat intelligence type capture flag
or simulations that I've helped design,
especially going into like
how much do you trust your threat intel feeds?
But that's probably a different panel.
But definitely like
when you're talking about
the technical accuracy
and like understanding
what your assumptions are
and not relying just on,
you know,
what a vendor is saying
or,
you know,
making sure that you have the right context
when you're making those decisions.
I think that dealing with uncertainty
is always a takeaway
from the tabletop exercises.
Folks try and sort of weasel out of it.
We're like,
you know,
you have to put out a press statement right now.
What are you going to say?
And they say,
well,
you know,
it would sort of depend on,
you know,
what the initial attack path was
and like what the point of entry was.
And we're like,
well,
you don't know that.
You're not going to know that.
And you might not find out for several days
and you still need to make a decision.
That's something that's really difficult to drill
because I think when it's a,
you know,
PowerPoint or something,
it feels like not real
and it feels like because it's vague,
you'll just say,
well,
if it was like this,
I would do this.
And if it was like this,
I would do this.
And that's my answer.
And we have to sort of pin them down
and say,
you don't know the answer
and you still have to make a decision.
So to your point,
I think no one's really prepared
to deal with that sort of uncertainty,
but that's something we try
to emulate.
Like we on purpose
don't give them enough information
to make a decision
because they're not going to have that
in real life.
Thanks for that.
I think,
and opening this up to the room,
we have some new folks in the room.
Please,
if you'd like to ask a question
or drop a comment,
please make your way over
to our microphone.
The acoustics in here
is not great.
Please, sir.
I'd be interested in your thoughts
in terms of a global corporation
with multi-international regulations.
When doing a tabletop,
trying to unpick
the way that sometimes regulations
can conflict each other
because of a lack of international
sort of single coherent approach.
So I'd just be interested
in how people have overcome that
in their tabletops
in that type of scenario.
Are you,
can you rephrase that?
Like,
are you asking about
how we make sure
that they're aware
of regulatory requirements or?
Well,
I think if you've got
a global corporation
and they're working
in different regions,
then typically,
the regulations
can sometimes knock them out
as one, right?
And then with the lack
of a single international policy
in terms of cybersecurity,
that absolutely complicates
the matter even further.
So if you're trying
to do a ransomware,
well,
not necessarily a ransomware,
if you're trying
to do a tabletop
and run that
within that
complicated scenario,
any thoughts
of how best
to run
that type
of situation?
Yeah,
this is a bit
of a non-answer,
but we try
and scope them
to one country
because there's usually
enough that could go wrong
even within one country
that there are
some takeaways there.
But sometimes we'll do,
you know,
there's an office,
you know,
in New York
and there's one in Singapore
or something
and they're both under attack
and they both lost data.
And then we sort of repeat
a lot of the questions, right?
Do you have external counsel?
But do you have external counsel
in all the different countries
that you're operating in?
Do you have
a data privacy officer
in all the different countries
that's tracking
those requirements?
So, you know,
we don't do it as often,
but...
I was just going
to pick up an earlier point
about law enforcement
where most of the narrative
was about engaging them.
And then there is situations
where actually they engage you
first off
because you weren't aware
that something has happened.
So I've had a situation
where a law enforcement
international organization
contacted a global company
and then what were the actions
that were taken thereafter?
So...
I'd also add
I think it's important
to have tabletops
done by teams
in the country
that you're operating in.
Like, if someone
really does have,
whoops,
different global offices
that they want us to test,
we would try and get folks
who actually live
in those countries
to run those tabletop exercises
in the native language
and who have a better understanding
of the requirements
of that country than we do.
If I can chime in
on that locality piece.
This piece is really interesting
because I know we've done that
with Cyber 9-12 scenarios
where, like,
if we're operating
in another country,
we have to have
a local partner
that's either a university
or a think tank
that includes researchers
so that we can make sure
that we're developing
a scenario that's, like,
one, compelling
for the students,
but also, two,
it's helping them build,
like, muscle memory
with specific challenges
or, you know,
gain some familiarity
with specific legislation
that is important to them.
One, it's good for them
to have that understanding,
but then, two,
it makes them more marketable
when they go to apply for a job
because they have
that understanding.
I also found
which takes precedence.
So if you've got
multi-countries,
which one are you going
to take first
as a prioritization?
And I can actually
kind of answer that
because our Geneva scenario
is really fascinating
in that sense where
it's in Geneva, Switzerland.
It includes competitors
from all over Europe
and also, like,
in the U.S.
and Asia,
but usually,
if we do want to focus
on an international aspect,
Europe is a pretty good
place for that.
You can look at the EU
as a structure.
You can look at NATO
as a structure.
We do have a track record
of pushing students
to the EU
so they can get familiar
with that legislation
that exists
at the supranational level,
but it also challenges them
to look at which,
where is the incident
actually occurring
in that one
particular country?
Is it impacting folks
in other countries?
And how do they want
to respond to that?
Is there a component
that might have
to bring in NATO?
It can get a lot stickier
and challenging,
but it's a good exercise,
I think, for students,
especially if they do want
to work on a policy team
at an organization
like a Microsoft
or a Facebook.
They're going to have
to be familiar with,
you know, scaling up or down
in terms of, like,
which legislation
or government entities
they're going to be
interacting with.
Hey, everyone.
So I'm Gaurav
from Singapore.
I don't know if you just
randomly mentioned Singapore
if it was a specific thing,
and so we actually end up
having a lot of companies
who have presence
in Singapore
and obviously presence
everywhere else in the world,
and we have tabletop exercises.
There are two types
of questions.
One is the policy question.
The other is
the tabletop question.
The policy question
of how do different
non-integrated
international laws
work together,
that's a policy question.
Sometimes they don't,
and I don't think
the tabletop people
can answer
that question fairly.
So that is a bigger
international policy
regulation,
legislation
framework
question,
which has no good answer.
But the tabletop question,
which is actually
quite useful,
is sometimes just get people
to walk through that.
So in Singapore,
we have an incident
response disclosure
or incident disclosure policy,
which is in like
two hours or something like that.
And other countries
have 72 hours.
So we tell them,
so you have an incident now,
which is impacting
your Singapore systems
and your Hong Kong systems
and your British systems.
What do you do?
And then they
will have to internally
figure out,
you know,
what is their process?
Who do they contact first?
Do they alert
the Singapore authorities
first and then risk
having activated
the Hong Kong
and British authorities
prematurely?
Do they?
That's for them
to answer.
But at least
you force them
to have that conversation
internally.
And then they realize
that the Singapore lawyer
team doesn't have
the contact information
of the Hong Kong
law team,
or they don't have,
they didn't know
that they have a different
disclosure policy
in other countries.
So the top exercise
is just to get people
to understand
that there are differences
and that they need to start
internally thinking
about that problem,
even if there is
no good answer to it.
Thanks for that, Gaurav.
I know that we're getting
close to closing time.
Really appreciate
you guys joining us today.
I want to turn over
to Winona and Sarah
if they have any
last comments
or things that they
wanted to add
before we pick up
the session.
Yeah, I'll say
one thing.
I think that
with TABLE,
we talked about
giving feedback
earlier.
We can really only
give feedback
for the couple hours
that we see, folks.
We really,
TABLEtops are really good
for understanding people
and processes.
They're not as good
for poking holes
in actual cybersecurity
maturity
and what folks
need to be prioritizing
for their practice.
So I'll just say that
I think they're very good
for particular purposes
for, you know,
escalating cybersecurity
importance to management.
But there is like
a limited amount
of information
that we're able
to give feedback on.
And so,
I just want to
throw that out there.
Yeah, I guess
I'll have a twofer
for a last remark.
I guess the first thing
is I know
we've talked about
technical,
more of the technical
side of the TABLEtops
versus the policy.
And when those two
integrate,
the one thing
that I want to stress
is I have seen
so many
technical
TABLEtops
go really well,
but when they're
communicated in a certain
way to a decision maker,
there's a complete
disconnect,
even though they both
want the same thing.
And I think that
one of the pieces
that I personally
try to give,
especially when
I see that disconnect,
is to talk about
technical aspects
in terms of risk,
in terms of monetary value
versus what needs
to be fixed
and what they need
to best describe
the problem
to their principal
and then vice versa,
like going from
high level
to low level.
And then,
last but not least,
if anyone is,
I know we got that feedback
of like,
why don't we have
a TABLEtop exercise
for DEF CON policy?
If anyone is interested
in leading that,
you know,
come talk to us after.
But yeah,
thank you for this opportunity,
Safa.
I want to foot stomp that,
Winona.
If you have any comments
or feedback for us,
please let us know.
We're really just trying
to keep improving
and growing this programming.
If I can do one
shameless plug,
if you're interested
in cyber simulations
that are multidisciplinary,
take a look at
Atlantic Council
Cyber 9 slash 12.
That's our student competition.
If you're interested
in learning more,
we do have some playbooks
on the website
that folks can rummage through.
In addition to that,
we also have a,
I know we didn't get
a lot of storytelling
for this session,
but I really wanted
to lean into the war gaming
aspects since folks
were really interested in that.
We do have a short story series
called the Cyber Moonshot,
which kind of makes fun
of the Cyber Moonshot,
but examines cybersecurity
concepts and challenges
through a comedic story.
But definitely recommend
that folks take a look
at those two items.
Great.
Thank you.
Thank you so much, Safa.
Thanks, everyone,
for joining us.
Thank you, guys.
And I think the next session
in this room
is going to be at noon.
It's going to be
Hacking Aviation...
No, I'm lying.
Sorry.
It's going to be
Addressing the Gap Between
and Assessing or Measuring
the Harm of Cyberattacks.
Thank you.
Thank you, guys.