Hi, I'm Bryson. I'm going to be joined by Nina, who's going to help me transcend ICS and med,
which is our medics med ICS talk. In fact, Nina, you deserve named credit for helping me out on
this perilous task. Thank you. I also wanted to be noted that I got informed of this like,
what, 18 hours ago that I would be on this panel discussion fireside chat with you.
Was it even 18 hours ago? No, I've been awake for way too long. So does this mean that I
am part of the unicorn tribe now? You are now part of the herd. Unicorns are herds. They get herds.
Yes. So hear what we have to say. All right. We're going to stop at the bad jokes.
Welcome to my med ICS talk with now joined by the illustrious Nina. All right. So who am I
besides wearing the unicorn stuff? I founded Scythe and Grimm. I'm the co-founder of my own
village, the ICS village. I feel like there's a pattern here. So also check us out at DEF CON 2
to get even more information about ICS. And of course, all of our talks will be published as
well for more detail. All right. Let's get started. All right. So outline. We're going
to be talking through all things ICS. Why is it relevant to hospitals? What are some of the
pieces around implementation? How is ICS different than traditional information technology?
And then specific issues and threats that are organic specifically to industrial control
systems. A quick little note that I got here from yesterday is IBM with Ponemon published their cost
of a data breach report. Comparing the breach lifecycle, so when a breach happens to when it is
found and expunged, healthcare is 96 days slower on average compared to the leader of finance. So
we already have seen an industry that, yes, you were like building up for it. First interruption,
first question. So then what is the time frame of a financial industry breach lifecycle? And
then how does that how does that 96 days get added to that? Not how, but like what's the
normal time frame for one? Sure, I'm trying to remember what that number was. I want to
say it was either 208 or 280 days. So 200 and some days is the fastest for an industry on average to
find a breach and get rid of it. And healthcare is 96 days slower on average for that. So that
is over three months longer to eradicate a breach from their systems. You said 200 days?
That is the fastest, which is finance. 200 plus 96 is approximately a year. Yes. You said three
months. No, so 96 days is three months. Got it, got it. My math is no good. Okay, yes.
Do you want to co-found the math village? Can we do that next?
Making stats literate for everybody. You know what, that would be great. I would love that part.
So Nina, you have some notes on why this is important.
I love that you added this in here. I'm going to let you talk through it so I can just
bombard you with questions. These were literally your notes. I just didn't take them out.
I know, I get that. Okay, so then I guess I'm speaking to it. So I think as we all know at the
biohacking village and folks listening, so healthcare is a 365 day entity. Downtime for
any medical devices or AMRs would mean dead time for patients. What people don't really consider
in the medical ecosystem is the electricity, the water, gas. How are these things affected and
does it, can it hurt patient care? If you go to the dentist, for instance, because I feel like
that's something that everybody can more or less relate to, that water flow that they have in your
mouth and there's a suction on the other side, that gets treated and then it gets sent out into
the world. Electricity, if a surgical suite was underway and electricity went out, granted there
are generators, but let's think about how often the generators are checked and made sure that they
are still active and can produce any electricity or whatever the needs of the hospital is. Negative
airflow and filtration, super important. We're in COVID right now. We know that it's an airborne disease
and just in generalities, that negative airflow that goes from the hospital removes and reduces
the nosoclonal diseases. Nosoclonal diseases are the diseases that you get
in the hospital while you're in the hospital. There are backups to backups in hospitals usually,
but again, how often are they checked? How often are they changed? Who's checking on these? Where's
I'm a city girl, so one of the problems that happened at NYU during Hurricane Sandy
was that their servers and the generators were in the basement and the unfortunate truth is that
those were then gone when the hospital itself got flooded by all the waters and they had to
move the patients over to nearby hospitals. And just thinking about how patients are transported
from one hospital to another, how are they producing medical notes and medical discharges
so that that patient would get treated accurately by that next physician that took care of them?
Healthcare is a great analog for industrial control systems because unlike traditional IT,
when we think of the confidentiality, integrity, and availability triad,
IT generally goes in that order and it's flipped for industrial control systems because just like
healthcare, we need something where availability is the utmost of importance. If I lose electricity
in the middle of an operation, that can cause patient harm. Modern society rests on critical
infrastructure, electricity, water, finance, transportation. If those went out, we go to
the stone age almost overnight. The longer it's out, the more damage that's done and there are
the statistical models then of understanding what that implication is. The final piece on
industrial control systems is there's two levels of it that's relevant for the hospital setting.
There's of course what we're talking about here, which is the grid, the infrastructure that
nominally is providing all of the different resources to the facility. And then there
are industrial control systems embedded inside the facility to help do particular elements.
Nina talked about air filtration and negative airflow. We will have a slide later on that we
talked about in the podcast a couple of months ago where we're going to go into detail on some
of those systems and how they interconnect. I want to add something to my notes. The wi-fi,
right? Nobody's... we haven't thought about how the internet connectivity, if that went out, would
affect all of this. So again, just if you can't print out a patient record or if you're taking
blood and you can't specifically say that this is this patient's because you have to enter
it into the EMR and then a thing prints out, a label prints out, there's a lot of
flow that continuously in the hospital needs to be contemplated. And I actually don't think
it has been. So this is... we had this conversation on the podcast, it hasn't been published yet,
but this is why Bo and I decided that you would be perfect to give this talk.
Well, thank you for having me on and thank you for helping. So what is ICS? Industrial Control
Systems, sometimes also known as SCADA or Operational Technology. There are wonky
differences between those three, but fundamentally it's a catch-all for a couple of things.
The biggest thing that separates ICS from what we think of a traditional computer is
industrial control systems cause physical effects in the real world. We're not just talking data
that stays on a hard drive. We're talking about something that actually changes something in the
world. If it's a water pump, right? Water pressure, water flow is going to change. If it's airflow,
airflow is going to shift and change. Our environment is actually affected. The reason I
chose this picture is the best joke that I have that I think clearly illustrates both what it is
as well as the problems are any computer that's at least 20 years old is an industrial control system.
They were designed to be available and to have a long life cycle. They were... they're expensive.
Most of them are expensive. Heavy, heavy capital costs where I want to put this, I want to forget
it and I don't have to do anything. The introduction of cybersecurity is still a relatively
new phenomenon as much as cybersecurity is a relatively new phenomenon. But particularly in
industrial control systems, it's a new thing where previously this would be the kind of equipment
that was maintained by a safety engineer who was mostly looking at what do I need to maintain the
equipment so that it works, not how do I protect it because somebody might try to manipulate or
attack it. So I just want to speak to that on the healthcare side. So there's so many legacy
systems and so many legacy devices because same concept, everybody was imagining all these things
to be long lasting. And when you're in the hospital and you have all of these devices,
there's not a lot of funding that... there's not a lot of payment that goes into the hospital.
They're kind of just trying to get to zero and balance themselves out. So having all of these
legacy systems and things like this does create a problem when it comes to us putting security
around it because we literally have to build these things in. We have to build... what's the thing
around the castle? Moats. Perimeter defense. Yes, thank you. You can cut all of my English out.
Right, so we have to build perimeters of defense around all of these legacy devices
so that they work and we can continue making sure that the patient and the systems work.
Yeah, so just being specific since perimeter typically means like the outer shell,
and a lot of folks think of that as controlling the ingress and egress to a network. Since a lot
of these kinds of systems can't be directly patched, which is a slide that's coming up,
typically there is a similar kind of inside, usually we wouldn't use perimeter, but
a way to kind of blanket around, control around the traffic to a device. So this is an example
where like software-defined networking has become very popular in ICS environments because
if I can't patch or improve the configuration of the device itself to make it not vulnerable,
at least I can control the traffic around that and adapt it to those kinds of threats.
So ICS versus IT, workers versus nerds. We talked about the performance of where I need
high availability versus in traditional IT, availability is an issue, but it is not the
primary concern. And then of course, when we're talking about risks here, when we think of IT,
we think about patient data, we think about privacy. On the operational technology side,
we're talking about injury or death. Okay, so some layers of ICS. At the very bottom layer is
the direct control. Like I talked about, ICS is where I am physically affecting the environment.
And so the most common element for that would be like a programmable logic controller.
Everybody at this point knows what a PLC is, even if you've never seen one,
because that's what Stuxnet was. Stuxnet affected the PLCs where they were changing
the speed of the centrifuges that were doing the uranium enrichment. Again,
a device controlling physical effect. And what we would see in a hospital environment would be
a pump, something that is cycling water or air. And then above that is the supervisory level. So
this is critical to understand these differences because a PLC is not a smart device. It doesn't
necessarily know what's the correct operational parameters to function in. When we think about
hacking a PLC, very rarely am I going to try to build some special code that's going to go
onto that PLC to rewrite it. That does happen. More likely what we see a lot of is issuing
commands to the PLC because one, PLCs generally don't accept authentication. Two, the traffic is
unencrypted. And three, the PLC itself doesn't know what's good or what's bad. It just does
what the supervisor above tells it to do. And then it just makes that happen in the world.
And it doesn't always have that feedback loop of understanding if I've gone out of tolerance.
The supervisory level is where you would have like a historian or acquisition. And this is
literally what it sounds like, right? Like I am monitoring multiple devices that are able to do
things and I'm telling them what to do and I'm reading what's happening and then I'm adjusting
that. So examples of where we can see that in a hospital are a doctor's workstation or a PACS,
which I forget what PACS stands for. It's a photo acquisition computer system, I think.
Archiving.
What's that?
Archiving.
Archiving, yes. So patient archiving. So keeping track of all of the different images that are
coming from x-rays, MRIs, and all those things and time them in.
So I'm going to challenge you on this question on this slide a little bit. So as healthcare
hackers, as healthcare security researchers, we tend to talk a lot about the medical devices.
We tend to... that's a lot of where the focus is because that's a lot of where the laws revolve
around. What is not discussed a lot is the 911 system. So where would that lie in the ICS layers?
Right. So I think you're talking more about the fact that we see a lot of embedded device
progress, particularly with the work that's been done with Bow and Company at FDA,
with what a lot of the I am the Calvary has been advocating, what the biohacking village has been
advocating. And then there's this broader question that's even beyond hospitals, which is, okay,
what about the rest of the stuff that's a part of that infrastructure, right?
And I think the challenge and the solutions are the same. We have to design for security.
There is no, I have built this and it is secure. We have to have that actually happen. And it's
harder because with traditional ICS, while there has been a lot of attention that has started to
be drawn to what is, like I said, a new market. I mean, ICS security, I would say is about six to
eight years old. And a lot of the startups are all coming up with different solutions around
those things in particular. But this is not something that the average, unless you're a
manufacturing plant, unless you are a part of critical infrastructure delivery, folks aren't
paying attention or doing anything about this. I'm going to bring you back in then. So why
is EMS not considered critical infrastructure?
That's a bigger question about my pay grade.
I mean, maybe we should discuss it because I bring this up because I think we've had this
conversation before and I think a lot of people know, but my father was a paramedic captain for
the fire department of New York. And I've seen how some of the systems work and completely agree.
It's, it's, there's a lot of things going on. And why, why wouldn't this,
why wouldn't 911 systems, the medics, police, the fire, not be considered critical infrastructure
enough that they be added to that healthcare, like workflow, that it gets secured more,
updated more, thought about more, workload more?
So that's, I think that's a, that's a policy and funding question of the SSAs and how,
because I think there's 16 of them now that define...
Just so that we, we don't have acronyms.
Sure. So I don't remember what SSA stands for, but SSA is the specific authorities that are tied,
putting a particular critical infrastructure sector tied to a specific agency. So for example,
electricity, electricity comes under Department of Energy. Transportation, I believe
TSA has that. I'm not sure. I don't have them all memorized. There's 16 of them.
So I, that's what I was trying to go look up while you were talking, was looking up what
the SSAs were and to see what tied into medical. Cause I don't, I, I don't know if there is some
level of that that's already represented, but that's, that's, so that's like the,
how the United States government breaks and manages it from a funding and accountability
perspective. And then at this lower level, of course, is the actual technical execution,
which is the industrial control system itself.
So I'm afraid I don't know enough to thoroughly answer your question.
You brought me here to challenge you?
Fireside chat, challenge accepted.
You're just getting even for me leaving those notes in, huh?
No, no, kind of.
So building automation systems. So this is everything that you can think of, right? This
is HVAC, fire detection, security and access control, all of those elements that are pretty
much in every building to do it. And we talked about that difference, right? We have the
higher level infrastructure of water, electricity, that's being brought to the building, and
then the elements inside of it that are doing it. Building automation systems, we're looking
at operations and backups. We're looking at efficiencies and savings. And so in the process
of what has been traditionally very proprietary closed systems, these are now going to shifting
to traditional ICS, like programmable logic controllers to do a lot of the things that
used to be proprietary. But now with PLCs, I can work with those on an open ICS development
standard to be able to get those savings and that efficiencies across my complexes.
I'm good. I'm good. I'll raise my hand.
So this is my very fancy explanation of how ICS and IT are two different things.
So we've covered some of this organically through what we've been saying, but fundamentally,
it's important to recognize that the two are very different, and they need to be treated
very differently. As much as there is the debate in IT about patching, as we talked about, that
might not even be an option in ICS. And Nina, as you commented about establishing the perimeter,
the inner perimeters and moats, we have to have different solutions because we can't patch things
to be able to protect them. In fact, ICS can be very brittle. There are a lot of systems on there
where even if you send the wrong packet can, in fact, cause the system to crash, which is,
of course, very dangerous when we consider about what they're doing and the high availability
necessity of it. So they're both different from a deployment and operations perspective. And that,
of course, ties back to they have different security considerations that you can't just
apply IT solutions to ICS solutions to solve them. So this, your explanation and this slide
remind me of the West. Do you remember when the East Coast went dark because somebody hit a line
or something like that? So how does that tie into this? I understand somebody broke a
electricity power thing. See, and that's part of the problem. I don't know how to
articulate this well, so I feel like maybe you can translate for me. Sure. So let's break that
down into multiple things. The first is that was not an attack. Right. That was a misconfiguration.
That was an overload. And then there were, through various circumstances, it rippled out. But I want
to highlight that piece, first of all, because particularly when we're talking about life
and injury, loss of limb, when we're also talking about public trust and understanding into buying
into our problems, fear and uncertainty and doubt is the easiest way to scare people. And then at
some point, they start to shut down and we lose the ability to have that conversation.
Most of the kinds of issues that we've seen are human error. They are failures of the systems,
not that there was somebody who was coming and attacking us or doing something.
And so the question that then follows is, well, why not? Why has critical infrastructure,
if it has these issues, why hasn't it been attacked? And what we have seen is there are
lots of what I call iterative intelligence operations where third party adversaries
are coming and they are learning and they are stepping and they are trying to see what could
happen, learn the environment and map it out. Because the second that they actually,
imagine if that was insert adversarial country that took down the entire East Coast.
The US response would probably be an armed response. It would not be a, we're going to
ping you back and we're going to shut some of your web servers down. No, we're probably going to bomb
you. And so that level of deterrence is a part of the landscape when we're talking about critical
infrastructure and the cybersecurity component. So the other element there is that the grid,
which I put in quotation marks is there is not one monolithic grid. The grid is actually broken
down into numerous subsections. And even that is broken down into, I want to say there are
about 3000 different operating entities that control parts of just the electric grid,
all the way down to like co-ops at a tiny little community level up to some of the large regional
providers. And so they all, that heterogeneity, all that different stuff, as much as it's
interconnected, actually also helps provide resilience for the grid itself, because something
happening here, it's very rare for like what you saw, where it rippled out and took multiple
things down. It's very difficult for an attacker to do that too. Here we go. Here are the SSAs.
Look at that. Right on time. Okay. Well, not right on time. I should have had this earlier.
I know. I was trying to give you credit. Nope. Oh, there you go. Healthcare and public health
is one. Yes. You're on there already. It was. Yes, we knew that. I knew that. So...
Hey, Nina, right on time. Yes. Okay. I don't know what you were going to say about this grid.
No, no. So this is where I wanted to, the conversation that you provoked a couple of slides
ago, I had built a slide for it, but I was so flustered by the hard question that I forgot
that I had a slide for it. This is what it's like having a conversation with me. I know.
Okay. So this is a specific breakout in this case. So I always steal vendor graphics because
vendors, of course, are always promoting their own things. This happens to be Johnson Controls,
but basically it gives us a detailed nominal explanation of all the elements around airborne
infection isolation rooms, which is what you were talking about, right? Particularly in a pandemic,
this is critical. This is where we keep the air where it's supposed to be so that we don't get
it out and infect other people while we're trying to treat them. There's approximately 12, 13
different areas or items to be concerned about that have some technical something that needs
to be secured in some way. And this is on top of all the other things that are in that hospital.
What kind of... do you think the hospitals are taking these things into consideration? Because
again, when we've had conversations with the device manufacturers and other hospitals,
healthcare in general, they're like, yeah, they talk a lot about the medical devices.
This isn't necessarily discussed. So the curiosity is, is this even on their agenda to think about?
So the challenge with hospitals is that they are very much hospital or regionally focused,
right? There is no like, here are how all hospitals do something.
Going to the original slide where I showed the delta of their challenge with responding,
I think that shows both a challenge of priority and capability. So things like this are
so far, are further down the track than worrying about PHI spillage and privacy
and simple IT operations, let alone, this is kind of more like advanced math.
So I see a specific security issues. Like all things, it started with security through
obscurity. Well, we have our thing and you have to get our thing exactly to be able to
do something to it. Over time, that doesn't work anymore. And then we've talked about this
numerous times already. What is a patch? A lot of these things weren't built to be patched
and they were driven by a very expensive capital outlay tied to life cycle management in the 20
to 40 year time range. We've also discussed most of them don't have authentication. Once you're in,
I can tell you what to do. And then my joke here on what does crypto mean? Because in their case,
they don't even know. Encrypted traffic is not something that you can do at all, particularly
if you don't even have authentication to start with. So most of this is transmitted in whatever
protocol it's communicating in. And there are ICS specific communication protocols,
but they're unencrypted. So the surface area specifically, most ICS, the primary attack
vector is through the information technology that connects them. Now, of course, the question is,
why do they connect them? And so the joke that I always throw out here is an air gap is not an air
gap is not an air gap. Everybody says, oh, it's air gapped. And then literally you just keep asking,
keep asking. And then eventually you find, well, it's connected here and it's connected that way.
And we don't call it an air gap because da, da, da. And nothing is ever completely disconnected.
Asterix, there are always examples where there is, I'm not going to go down that, but fundamentally
things do connect. And particularly in a residential setting, they definitely do.
And the reason is there's always one thing that your ICS has to tell your IT no matter what.
How do I bill you? How much did you use? Because we need the money.
And then same sign of problem we see in consumer IoT, there's industrial internet of things,
which are the sensors that help provide the feedback for the system to understand what's
happening in operation. All of these are much more vulnerable and great examples for lateral
movement. What's this on the internet? My joke here being, and so a call out to Chris Kubecka,
who presented on hack the world and galaxy with open system intelligence. In her example,
she showed how Boeing had a number of things that were vulnerable because a hacker can only
hack what they can touch. And if it's internet accessible, then I'm already a good percentage
of the way there. And it's the same kind of problem with ICS is making sure that these
things are not internet accessible. And oftentimes, not oftentimes, but sometimes they are, and that's
a problem. The final vector is physical proximity. Some of these things, particularly when we're
looking at like electricity, oil, natural gas, water, they're going to be elements of that
critical infrastructure as it extends out from production to transmission or delivery,
there's ICS along the way that's out there in public. And maybe it's just surrounded by
a fence. Maybe it's surrounded by a barbed wire fence. But one of the things I learned in the army
is an obstacle is only good if you have overwatch on it. Just because there's a fence does not mean
that somebody can't get to it. So physical proximity to these, particularly when they're
speaking RF, is a real challenge. This is well done. Okay.
So who is doing this? Well, organized crime, North Korea, like I said, organized crime.
So the joke being there that North Korea is fundamentally an organized criminal nation
state in what it's doing with its operations. And so a lot of the kinds of attacks that we've
primarily seen on hospitals, besides the theft of PHI, PHI being worth a lot more than PII,
because I can change my social security number, I can change my name, you can never change your
healthcare data, which is why it goes for a multiple on the dark web. Why it's sold for
a lot more than a traditional PII record. And then of course, we've seen a significant increase
in ransomware. And the IBM report that I referenced earlier, with Ponemon on the cost of
a data breach goes into detail about how much has been paid out to ransomware. And the healthcare
sector in particular, has been particularly challenged on doing this. We addressed the
issues of FUD earlier in the presentation, so we'll skip noting that now. And so, how could
they do this? We talked about lateral movement through IT systems being one of the more prominent
ways that this can go. Otherwise, we're starting to look at targeted attacks. And so, we went
earlier to, well, why haven't we seen these kinds of attacks happen on critical infrastructure
writ large is because of the deterrence component. So, I think mostly what we've seen is, of course,
there's going to be the primary motivator around finance. So, the easiest way for me to do that,
and then potentially the escalation would be a direct denial of service attack. So,
focusing on the services that ICS is delivering specifically to a facility or complex,
and taking those down. I'm reading your numbers. Those are exponential. Yes.
The percentages are high, considering that this is patient care. This is me trusting you
with my information, and what will you do with it? And then it's gone. I'm also just thinking about
how, again, we talked about this a little bit earlier, the hospitals aim at least to zero out,
they're putting $16.5 million out or $640,000. Where is that money coming from? And it's coming
from their insurances, because now cybersecurity insurance in hospitals is almost mandatory
because of the sensitivity of all of the devices and all of the information that's in there. That's
well done you. Yes. Yeah. I mean, the key elements that I would pull out of here on the statistics
are one, from the tax going back to 2016, not including this year, 6.6 million patients have
been affected by this. Of course, the most significant out of that was the one or cry
ransomware that struck the UK national health system several years ago, and that's been
attributed to North Korea. The question of motive on that one was whether that was a test, whether
that was on accident or whether that was on purpose. And then the final part here is the
overall cost of these attacks is $157 million. And mostly that is capturing the overall cost from
having to defend, having to identify, having to do the forensics cleanup, the damages that are
actually done when there is a breach and the loss of that data, because privacy regulations,
around PHI, have become much, much stronger. So there are some teeth to those now.
So I tend to... what is your call to action? There's a lot of information in here. It's very
dense. What do you want the healthcare community to do? What do you want the
ICS community to do? What do you want the security research community to do?
Yeah, that's tough. So, I mean, with the talks that I've been giving around embedded systems and
IoT for several years now, I talk about the need to design for security. I think that there are
some really low-hanging fruit here that we need to do. But that's really easy for me to wave my
researcher wand and say, these things should happen. And it's a lot harder for manufacturers
to have to increase the cost of their equipment to consumers, corporate consumers who are buying
this stuff to actually pay for that. And so that's the chicken and egg there is everybody
knows that these are needed things, but who's going to pay for it? And so typically, whenever
there's that middle ground of, well, who's going to do this? I think the answer is government for
funding those kinds of solutions. But we have a long way to go because it's just like you can't
immediately change equipment when you're producing a car. It's these things have an even longer
lifecycle. And so it's going to take a long time for us to dig out of that hole. And we really
need the patience and the stick-to-itiveness to get there. There are a lot of solutions that are
coming out to the market to solve some of this within, hey, if we can't fix something directly,
at least we can, like I said, we can work around it. We can control the environment around it to
protect it. So as those continue to push out, those will be cheaper solutions to both identity,
anomaly detection, and when I say asset identification, anomaly detection,
and then also providing protection in a more of a real-time way.
That's essentially what's in the innovation hopper for those to continue to progress that way.
So those continuing to roll out would be a good thing. So who would you specifically be looking
for to engage in this conversation? You said government, but again, we've had this conversation
that government is pretty broad. It's dissected into different sectors. Is there a person that
you would say, hi person, I have this information, I would love to have a conference with you, a chat,
so that we can do a deep dive and we understand the problem. Here it is. Let's work on that
solution. So person, place, thing. Sure. So if you can tell me who's the SSA responsible for the
medical and healthcare sector from that earlier slide, I would say that they're a part of this.
What? Not even a little laugh at that one. I started looking. You were looking it up.
So clearly they're a part of it from a sector specific element. I'm with the ICS Village.
We're working with CISA and Department of Energy on additionally pushing out additional programs
for hackers in the community to be able to do independent research and push some of these things.
A lot of it, as we talked about, since IT skillset is different than ICS skillset,
and that's the point of what the ICS Village is doing, is giving folks that are interested
and on road so that it's not just these abstract things like, oh, PLC and historian and SCADA,
DCS, like, what are we talking about? And making those accessible so that infosec,
traditional infosec practitioners can get their feet wet with going this direction.
Okay. Thank you. Thank you for having me.