Welcome everyone to my talk today ICS security operations
active defense concept with effective incident response in industrial control systems.
This is John Kurnas. My name is pronounced as John, which is pretty close in English and John.
And I am working as a OT security consultant and I have GICSP, OSCP, OSWPN certified
hacker certifications. My bachelor is computer engineering and my master is cyber security.
And you can reach me on Twitter and LinkedIn and GitHub as you can see here.
I am of course a bit nervous about the presentation and let's take a deep breath
and have some relaxation then we can continue. Okay, what's today's agenda? So I will discuss
a bit on active defense concept and I will explain briefly threat intelligence,
asset identification and network security monitoring,
incident response for ICS and then taking the advantage of found threats.
So what is active defense concept? It is something like this. Yes, improvising,
adapting and overcoming as we agree. But yeah, it is also related to this picture.
But in general, what I can explain is somewhere in the middle in here in the picture,
it is the process of analysts monitoring for responding to and learning from adversaries
internal to the network. So basically that means it is a combination of analyzing
the adversaries steps and monitoring constantly to the internal network and then when you see
any anomalies responding to that and basically when you see any action, basically create lessons
learned and learn from the adversaries. It is not deployable by itself of course,
it is built on the top of the good practices such as good architecture and then the passive defense.
And it is not definitely a hackback. So you need to stay on your internal network to protect and
you do not need to, you do not have to attack back to the adversaries of course.
So why active defense is needed? As we can see, at least for a couple of years,
traditional methods of protection without constant human interaction such as firewalls,
IPS and antiviruses only provides a certain level of security and adversaries are getting
stronger and stronger every day and the passive defense usually don't stop them, unfortunately.
So let me take a look to the active cyber defense cycle. It consists of four elements,
threat intelligent conception and asset identification and network security monitoring.
And then incident response when it's necessary and using threat and environment manipulation to
get over the adversaries. Basically this diagram is coming from the suns and references down there.
So basically the idea is monitoring your area of responsibility
which is quite necessary to at least have the, let's say, idea of the getting the baselines then
checking the anomalies and that said implementing monitoring will bring you the chance for quick
response when it's necessary when you see something odd and responding to incidents and attacks on
time is of course crucial because especially on the industrial control systems if it affects
the industrial control network it's going to cost you a lot. And constant changes are necessary
since you need to beat the adversaries and kick them out of your industrial control network.
And after that you need to share and you need to consume the lessons learned. At least you
need to share with the community and you need to observe and collect the lessons learned from the
community to develop better defenses on your systems, on your ICS.
So what is threat intelligence? I'm gonna, I'm trying to explain this pretty quick.
Let's start with the intelligence. So intelligence basically means the process of collecting data,
turning into information and producing an assessment that satisfies a previously identified
knowledge gap. So you need to obtain the, from the raw data to important information. So normally
you can do, everybody can say yeah you can you can do this with the tools but unfortunately
tools can only help and your analysts can create intelligence not the tools.
An intelligence life cycle, when we take a look to the intelligence life cycle,
it starts with planning and direction, then collection, then processing and exploitation.
Exploitation here means, doesn't look like the exploitation in the computer science. It is more
like clarifying and filtering the information that you need, that make it useful. And then
analysis and production and dissemination and integration. Basically the intelligence life
cycle consists of these elements. And then if we, if we discuss about the open source intelligence,
it is a type of intelligence which is quite useful for adversaries as well as also for you.
It is low cost and low impact but it gives essential information about you. So most of
the time adversaries are googling on the internet, checking related documents and data about you.
But you can also use this as your own beneficial by checking these documents and
trying to reduce your threat landscape. Normally these sources can be public relation documents
or partnership announcements that you share. Most of the time these documents are not considered
dangerous but they are dangerous because they contain valuable information.
Company information of course. If the company is a public company and if the company shares
monthly or quarterly or, I don't know, annual reports, these reports also have valuable
information about the company. Most of the time job descriptions basically leak the technologies
of course. And advisories and alerts sometimes also might be related to your systems and
give some hints about your systems. And of course your internet connected devices.
These are the most dangerous things. So if you continue with the threat intelligence,
in simple terms again threat intelligence is analyzing information related to adversaries
that have intent, capability and opportunity to harm you.
So that specific type of intelligence gives defenders knowledge of the adversary,
their actions within the defender's environment and the capabilities as well as their tactics,
techniques and procedures. And the threat intelligence is often shared in different
methods, for example indicators of compromise, IOCs, as I mentioned tactics, techniques and
procedures, TTPs and oftentimes complete report about an incident or an attack.
And what do we need to do to identify our threat landscape? Threat landscape is a combination of
information attack space, threat groups, industry and non-technical influences. So basically to
identify your threat landscape you need to ask yourself who is interested to attack your
organization and why, what would be the reason of causing damage and how can they get into your
organization. And after you identify the threat landscape you need to reduce the threat landscape
and most of the times asset and network identification is necessary to succeed.
First of all you need to clarify your external and internal landscapes because you need to focus
on your responsibility areas and as well as you would have some connections from third-party
vendors or any other third-party service providers etc. Then using known information
you need to try to reduce your threat landscape basically by checking these information and
trying to prevent any leaking information like that and performing assessments and measurements
also with the third-party companies or external companies because sometimes you need to have a
second opinion or second eye to check that and after that you need to continue
doing these again and again to reduce your landscape. An external threat intelligence
is the threat intelligence comes from the specialized teams which can be professional
services companies such as Dragos and or it could be from ICS search and other search
basically provides indicator of compromises and TTPs and most of the time it is for large audiences
it might not be relevant for you but still it is important to follow these threat intelligence
reports, but sometimes of course the industry is not relevant for you and you don't need to take
an action. But oftentimes ICS threat intelligence are limited, it's really hard to find, especially
purely ICS threat intelligence, but when you find it, it's really priceless. And ICS search,
multiple ISACs and old ISACs can provide SANS internet storm center, also providing
threat intelligence, threat feeds, and reports from vendors and professional service companies,
as I mentioned, like from Mandiant or from Dragos, for example, or FireEye.
What is internal threat intelligence? Internal threat intelligence comes from your data.
It is usable against the current problems in your environment because the data is
coming from your environment and after you analyze, you basically realize there are some
problems and it's directly relevant to your issues. But it requires personnel in order to
do the analysis and it requires specialized skills such as malware and threat analysis.
And if you summarize threat intelligence, tactical threat intelligence contains TTPs and IOCs,
which are quite useful to add, change and implement your defense, or at least search for
intrusions in your environment. And strategic threat intelligence can be used to look
overall patterns, suspected attribution, trends and teams. It is also useful to identify
where your defense might fail or already failing.
So if you go for asset identification and network security monitoring,
the purpose and importance of the asset identification is it is really hard to defend
if you don't know what you have in your plant, in your facility or in your environment
in general. Because if you don't know what's there, you cannot protect it basically. And
advanced security solutions that you have also not effective if you can't provide the whole map
to defend. So you need to at least provide the fundamental security to have this. And
network security monitoring, threat intelligence and instant response usually works better if you
have the network knowledge, if you have internal network knowledge. And how do we identify the
assets in the ICS network then? We can say first we need to start determining the area of
responsibility. Because as I mentioned, most of the times there are some remote connections that
you need, you shouldn't include. Because yeah, of course, VPN endpoints on your side, that can
count on your responsibility. But still, there are some ways that goes to outbound and not in
your responsibility anymore. And then finding and utilizing the existing information. For example,
if you already have some network diagrams that you have, you need to start with this, then you
can validate non-information. And if it's not sufficient, and if you think you need to, let's
say, create a new one, you can try to collect by doing physical inspection, which is normally a bit
harder and takes much more time. You can do traffic analysis in your ICS network,
passive analysis, that we can also say. We can do configuration file analysis.
And we can do active scanning, which is not recommended, except you're, let's say, you are
not in the production or if you're in a turnaround, you can do that. But otherwise,
it's a bit dangerous. And then collecting the data in all the assets, you need to document it again,
in order to have a nicely done diagram and have the old assets.
And unfortunately, most of the ICS networks are flat networks. You'll notice that once you've done
the asset identification, after identifying the assets, it's often required to have a physical or
logical separation or both. It is quite necessary since it makes more difficult to adversaries to
pivot in your network. Otherwise, if it's a flat network, normally a game would be over within,
let's say, half an hour or one hour. For the active defense concept, it is really crucial
because you need to build on top of the good architecture and passive defense. At least for
me, the aim would be separating the network as explained in purging model. So it could be a good
reference to start with. And then if you want to have more separation, then you can use
micro-segmentation, etc. But it is the fundamental for the ICS networks. And network security
monitoring is a continuous process to collect, detect, analyze indication of threats in order to
respond faster to incidents and attacks in your network. It is threat centering approach instead
of traditional vulnerability centering approach. Network security monitoring requires dedicated
personnel as analysts. And it also requires preparation for infrastructure and also tools
ahead of time. And it brings proactive approach to security and detection of the threats. So
you can take some remediations once you identify the issues in your network.
And network security monitoring provides visibility and helps you to identify the changes
and anomalies. Might not be relevant to only adversaries, but also
some misconfigurations for troubleshooting misconfigurations.
And it helps you to detect intrusion attempts and movements within ICS SCADA networks.
And it can evaluate separation and segregation for levels that you have and zones that you have.
And it can help fine-tuning and validating the settings of passive defense elements. For example,
checking the firewall rules and to see if it's correctly implemented or missing some points,
etc. And it also can help reduce the threat landscape by hardening non-required ports and
services. Since you are seeing this when you are doing the network security monitoring and
you can instantly take remediation. Network security monitoring is more useful in ICS networks
since most of the assets are critical. And most of the times, all the assets or any assets cannot
be immediately patched. Because, yeah, we know ICS networks are quite, let's say, untouchable.
And then you need to have any other layer to have the protection for your endpoints, etc.
And ICS networks have many dependencies and connections, such as enterprise network
connection, vendor connections, business applications, contractor VPNs. And it is
really valuable to monitor these external parties for anomalies because once your vendor or once
your contractor, let's say, compromised, it is just a moment to jump into your network by using
these VPNs. And normally, you don't have any vision on these connections without active defense.
And detection approaches.
Most of the time, what we check is identifying the most used IP addresses and ports used within
the network to create basically a baseline, identifying biggest bandwidth users, identifying
encrypted communication, because most of the times, command and conquer centers are using
encrypted communications. And identifying critical assets and usual traffic is also
important to continue on the baseline to have stability on that. And identifying network
anomalies, of course, and identifying lowest bandwidth and communication, because sometimes
you can see in the network unusual traffic, which could be pretty less than your usual levels.
And of course, you need to identify exfiltration. You need to check that
when you are doing the detection. And asset identification and network security monitoring
takeaways. Asset identification and network security monitoring is the key
for active defense concepts. Network security monitoring is a great approach for ICS because
most of the ICS networks are quite stable. And basically, once you create a baseline, it is
easy to, let's say, monitor. And these two elements support implementation of
better architecture and passive defense. Once you implement that, yeah, you can take a look
back and you can, let's say, improve your basic fundamental security also.
Detection often relies on the sensors in the ICS network. You need to be sure that
you need to detect the baseline changes, anomalies, and IDS rules. Because these can
basically warn you when you are doing the detection. And logs and visibility are really
important. But yeah, you need to have an analyst to, let's say, contribute value on that. Because
by itself, logs and visibility doesn't give you the chance that you can prevent. And analysts
need to follow it constantly and verify it if there is an anomaly. And network security
monitoring will eventually lead to instant response. You need to be ready for instant response.
By saying that instant response for ICS, it is a bit different comparing to
usual IT instant response. Because you cannot simply bring down the systems during the
instant response when you're working on ICS networks. And the focus is a bit different
by saying that maintaining safe and reliable operations is the most important thing.
Acquiring meaningful forensic data within a limited time. Performing a timely analysis.
And containing and eradicating the threats. Basically, these are the four elements that
you need to follow when you're doing the ICS instant response. Before that, preparation.
There are some steps that you need to follow. Preparation, same as traditional IT, but with
some limitations such as testing tools, testing methods, but in a lab. Integrated detection and
identification. Working with network security monitoring team to implement rules and detection
capabilities, which is tailored to your threats in order to identify impacted systems.
And evidence acquisition. Normally, you don't have that much time, like on IT side,
you don't have that much time to do forensic analysis because you cannot stop the operations.
Here, the focus is maintaining operations while acquiring enough evidence to perform later.
You'll do the forensic analysis later. And time critical analysis using fast and well-tested
techniques to quickly determine the overall impact to the operations.
And supported activities. IR teams mostly share information and evidence. All the evidence should
be passed to other teams to begin deep analysis and continue on the active defense cycle.
Containment. Preserving the operations by collaborating with other teams,
operators, and also engineers on the field. Eradication and recovery. You need to neutralize
the threat by, for example, reimagining the system, reinstalling known good software,
implementing patches. And then you need to provide the lessons learned, which is documents,
finding for all, pass information to the network security monitoring team in order to identify and
see if there is any reinfection, etc. And how do you prepare an incident response team?
You need to determine the requirements and dependencies within your facility.
For example, uptime, availability, and specific systems you need to take care. You need to decide
if it's going to be in-house or outsourced. There are some advantages on both sides.
But you need to decide, at least have a couple of people in your facility to do it more faster.
A team size three, four well-trained people on site would be enough to cover, but you also need
to think about the shifts, etc. So in total, it needs to be at least eight to 10 people.
So chain of command is really important because there will be chaos and someone needs to
communicate with the management team, etc. And it should be incident response director.
Then all the evidence and incident handlers need to report to lead responder. And the chain of
command needs to go up and down like this. And to build your ICS incident response team, you need
to find the right personnel. Of course, it's the hardest part because incident response is really,
to focus and develop yourself. And then also you need to take care of your jump kits because
often jump kits, when you need it during the incident, most of the time you are losing some
parts from the jump kit. It's easy to borrow from the jump kit. And then when you go to the incident,
you are missing a couple of hard disks, etc. when you need it over there. So it's important.
So evidence acquisition, you need to take care of the order of volatility.
And you need to decide if it's going to be local or remote acquisition. I would prefer local because
it's less risk and faster. The tools should be tested before on the systems because we are not
talking about regular IT systems. Acquisition should be coordinated with all involved personnel
and you need to discuss beforehand when you are touching any field device with engineers and
operators. You need to gather all the evidence if the time permits. You need to start from registry
and then any memory and then taking the disk images, for example. And you need to take necessary
photos in order to see on the, let's say, devices. If you see any command prompt, you need to
have an evidence because once you turn it off, you will lose it or you can lose it in any second.
And all the data should be analyzed in an approved facility.
What are the sources for the forensic data in ICS networks? Highly volatile data, system memory,
network information, and system processes, and VPN connections, and these logs, and registry
hives, HMIs, mostly Windows computers and you can have system logs, etc. Engineering workstations,
controllers such as PLCs and RTUs, and sometimes virtual resources such as
VMs and cloud environments if it's connected. And you need to do the high quality instant response
in a timely manner. It's really important in ICS when you are doing this.
Timely analysis is important to keep operations safe and reliable.
It should focus on understanding the scope and then the type of incident. And the baseline
information comparison will be really helpful when you do that. And focusing on the new connections
in the bandwidth, new routes, anomalies in the VPN connections,
registry keys, spawn processes, you need to check these first. And I said identification and network
security monitoring help to respond quickly. And utilize good tools to reduce the analysis period.
This is also important. You need to do the practice before going to the real incident.
And how do you use threat intelligence in the instant response? Indicator of compromise
to scope the infected systems. Then after that, you need to identify the network data on host.
And TTPs to identify adversary efforts. And you need to utilize and ensure threats are gone after
you've done the incident response. And instant response takeaways. You need to focus on providing
actionable information about the scope of the threat and its potential impacts while you are
acquiring the evidence without breaking operations and safety. ICS instant response
should be tailor-made. It's a bit different than the IT, as I mentioned. Efforts must align with
the goals and requirements of the operations. This is the first priority. Preparations should
be done ahead of time. You really need to practice before it really happens. And acquired evidence
and lessons learned should be shared with other personnel. So once you get evidence, you need to
share it to move forward the active defense cycle. And let's summarize this. Taking the
advantage of the found threats. Once you complete the instant response, if we can safely interact
and understand with the threat, we can have the best source of defense information from the
adversary's best capability. We can feed the active cyber defense center with the found threat
intelligence to build better defense. It is important to identify and use indicators of
compromises from the threats to help instant response. And understanding the malware's tactics
to identify weaknesses in the current ICS architecture is valuable because after that,
you need to improve your architecture. So these steps are really valuable. And my last presentation
is about lessons learned sessions for long term success. For the success of active cyber defense
cycle, lessons learned should be shared between the teams internally. And also necessary actions
needs to be taken to build better defense. And after that, either during the incident and after
the incident. And after that, once you share it internally, it's also recommended to share
your lessons learned in an appropriate way with the ICS community because if you had that threat,
someone else might get this soon. So it is really important and valuable.
And yeah, I think I'm just on time. Thank you very much for listening to me. If you have any
questions, either you can send me a message on LinkedIn or Twitter or you can ask now. Thank you.
John, this is...
Okay, so everybody look forward to see John and ask questions and all sorts of different things.
On the DEF CON discord server, we're in the ICS village subgroup.
And John, thank you very much. That was a fantastic presentation.
Thank you. Cheers, bye.