start	end	text
2100	6640	Welcome everyone to my talk today ICS security operations
6640	12020	active defense concept with effective incident response in industrial control systems.
14440	20760	This is John Kurnas. My name is pronounced as John, which is pretty close in English and John.
21060	28220	And I am working as a OT security consultant and I have GICSP, OSCP, OSWPN certified
28220	34820	hacker certifications. My bachelor is computer engineering and my master is cyber security.
34820	39300	And you can reach me on Twitter and LinkedIn and GitHub as you can see here.
40180	46160	I am of course a bit nervous about the presentation and let's take a deep breath
46160	54680	and have some relaxation then we can continue. Okay, what's today's agenda? So I will discuss
54860	59580	a bit on active defense concept and I will explain briefly threat intelligence,
59580	62840	asset identification and network security monitoring,
62840	68100	incident response for ICS and then taking the advantage of found threats.
68920	75660	So what is active defense concept? It is something like this. Yes, improvising,
75660	84940	adapting and overcoming as we agree. But yeah, it is also related to this picture.
85840	91840	But in general, what I can explain is somewhere in the middle in here in the picture,
92220	98200	it is the process of analysts monitoring for responding to and learning from adversaries
98200	104800	internal to the network. So basically that means it is a combination of analyzing
104800	112480	the adversaries steps and monitoring constantly to the internal network and then when you see
112480	120960	any anomalies responding to that and basically when you see any action, basically create lessons
120960	128200	learned and learn from the adversaries. It is not deployable by itself of course,
128200	134780	it is built on the top of the good practices such as good architecture and then the passive defense.
135480	141400	And it is not definitely a hackback. So you need to stay on your internal network to protect and
141400	146580	you do not need to, you do not have to attack back to the adversaries of course.
147260	153440	So why active defense is needed? As we can see, at least for a couple of years,
153440	158540	traditional methods of protection without constant human interaction such as firewalls,
158540	165340	IPS and antiviruses only provides a certain level of security and adversaries are getting
165340	170670	stronger and stronger every day and the passive defense usually don't stop them, unfortunately.
172140	179160	So let me take a look to the active cyber defense cycle. It consists of four elements,
179160	184710	threat intelligent conception and asset identification and network security monitoring.
185330	192950	And then incident response when it's necessary and using threat and environment manipulation to
192950	203070	get over the adversaries. Basically this diagram is coming from the suns and references down there.
204070	208670	So basically the idea is monitoring your area of responsibility
209870	217190	which is quite necessary to at least have the, let's say, idea of the getting the baselines then
217190	224850	checking the anomalies and that said implementing monitoring will bring you the chance for quick
224850	230830	response when it's necessary when you see something odd and responding to incidents and attacks on
230830	237650	time is of course crucial because especially on the industrial control systems if it affects
237650	244410	the industrial control network it's going to cost you a lot. And constant changes are necessary
244410	251490	since you need to beat the adversaries and kick them out of your industrial control network.
252210	257990	And after that you need to share and you need to consume the lessons learned. At least you
257990	263570	need to share with the community and you need to observe and collect the lessons learned from the
263570	270430	community to develop better defenses on your systems, on your ICS.
271860	280290	So what is threat intelligence? I'm gonna, I'm trying to explain this pretty quick.
280290	286190	Let's start with the intelligence. So intelligence basically means the process of collecting data,
286190	292850	turning into information and producing an assessment that satisfies a previously identified
293690	306430	knowledge gap. So you need to obtain the, from the raw data to important information. So normally
306850	312150	you can do, everybody can say yeah you can you can do this with the tools but unfortunately
312750	318690	tools can only help and your analysts can create intelligence not the tools.
320070	325150	An intelligence life cycle, when we take a look to the intelligence life cycle,
325150	331950	it starts with planning and direction, then collection, then processing and exploitation.
331950	338430	Exploitation here means, doesn't look like the exploitation in the computer science. It is more
338430	347550	like clarifying and filtering the information that you need, that make it useful. And then
347550	353370	analysis and production and dissemination and integration. Basically the intelligence life
353370	361870	cycle consists of these elements. And then if we, if we discuss about the open source intelligence,
361870	368330	it is a type of intelligence which is quite useful for adversaries as well as also for you.
369170	374210	It is low cost and low impact but it gives essential information about you. So most of
374210	381250	the time adversaries are googling on the internet, checking related documents and data about you.
381250	389290	But you can also use this as your own beneficial by checking these documents and
389290	395510	trying to reduce your threat landscape. Normally these sources can be public relation documents
395510	403650	or partnership announcements that you share. Most of the time these documents are not considered
403650	408850	dangerous but they are dangerous because they contain valuable information.
409770	417150	Company information of course. If the company is a public company and if the company shares
417150	423470	monthly or quarterly or, I don't know, annual reports, these reports also have valuable
423470	430530	information about the company. Most of the time job descriptions basically leak the technologies
430530	438370	of course. And advisories and alerts sometimes also might be related to your systems and
438850	444910	give some hints about your systems. And of course your internet connected devices.
445230	453970	These are the most dangerous things. So if you continue with the threat intelligence,
453970	459710	in simple terms again threat intelligence is analyzing information related to adversaries
459710	464370	that have intent, capability and opportunity to harm you.
464970	470130	So that specific type of intelligence gives defenders knowledge of the adversary,
470130	475750	their actions within the defender's environment and the capabilities as well as their tactics,
475750	482050	techniques and procedures. And the threat intelligence is often shared in different
482860	489750	methods, for example indicators of compromise, IOCs, as I mentioned tactics, techniques and
489750	497110	procedures, TTPs and oftentimes complete report about an incident or an attack.
499390	506970	And what do we need to do to identify our threat landscape? Threat landscape is a combination of
506970	513230	information attack space, threat groups, industry and non-technical influences. So basically to
513230	517690	identify your threat landscape you need to ask yourself who is interested to attack your
517690	525670	organization and why, what would be the reason of causing damage and how can they get into your
525670	533550	organization. And after you identify the threat landscape you need to reduce the threat landscape
534210	540710	and most of the times asset and network identification is necessary to succeed.
540710	547670	First of all you need to clarify your external and internal landscapes because you need to focus
547670	555310	on your responsibility areas and as well as you would have some connections from third-party
555310	564510	vendors or any other third-party service providers etc. Then using known information
565940	574790	you need to try to reduce your threat landscape basically by checking these information and
574790	581770	trying to prevent any leaking information like that and performing assessments and measurements
582490	589870	also with the third-party companies or external companies because sometimes you need to have a
589870	597670	second opinion or second eye to check that and after that you need to continue
597670	604390	doing these again and again to reduce your landscape. An external threat intelligence
604390	608990	is the threat intelligence comes from the specialized teams which can be professional
608990	617690	services companies such as Dragos and or it could be from ICS search and other search
618190	625430	basically provides indicator of compromises and TTPs and most of the time it is for large audiences
625990	632650	it might not be relevant for you but still it is important to follow these threat intelligence
633290	640490	reports, but sometimes of course the industry is not relevant for you and you don't need to take
640490	647830	an action. But oftentimes ICS threat intelligence are limited, it's really hard to find, especially
647830	655730	purely ICS threat intelligence, but when you find it, it's really priceless. And ICS search,
655730	662450	multiple ISACs and old ISACs can provide SANS internet storm center, also providing
663510	669090	threat intelligence, threat feeds, and reports from vendors and professional service companies,
669090	674910	as I mentioned, like from Mandiant or from Dragos, for example, or FireEye.
675710	680570	What is internal threat intelligence? Internal threat intelligence comes from your data.
681050	685570	It is usable against the current problems in your environment because the data is
685570	690830	coming from your environment and after you analyze, you basically realize there are some
690830	697750	problems and it's directly relevant to your issues. But it requires personnel in order to
698290	704490	do the analysis and it requires specialized skills such as malware and threat analysis.
707430	715390	And if you summarize threat intelligence, tactical threat intelligence contains TTPs and IOCs,
715390	721790	which are quite useful to add, change and implement your defense, or at least search for
721790	726670	intrusions in your environment. And strategic threat intelligence can be used to look
726670	733270	overall patterns, suspected attribution, trends and teams. It is also useful to identify
733270	737310	where your defense might fail or already failing.
739070	743630	So if you go for asset identification and network security monitoring,
744830	750750	the purpose and importance of the asset identification is it is really hard to defend
750750	754890	if you don't know what you have in your plant, in your facility or in your environment
754890	761950	in general. Because if you don't know what's there, you cannot protect it basically. And
761950	766990	advanced security solutions that you have also not effective if you can't provide the whole map
766990	776450	to defend. So you need to at least provide the fundamental security to have this. And
776450	781830	network security monitoring, threat intelligence and instant response usually works better if you
781830	789450	have the network knowledge, if you have internal network knowledge. And how do we identify the
789450	795510	assets in the ICS network then? We can say first we need to start determining the area of
795510	800410	responsibility. Because as I mentioned, most of the times there are some remote connections that
800410	808570	you need, you shouldn't include. Because yeah, of course, VPN endpoints on your side, that can
808570	817550	count on your responsibility. But still, there are some ways that goes to outbound and not in
817550	824550	your responsibility anymore. And then finding and utilizing the existing information. For example,
824550	830090	if you already have some network diagrams that you have, you need to start with this, then you
830090	836030	can validate non-information. And if it's not sufficient, and if you think you need to, let's
836030	842110	say, create a new one, you can try to collect by doing physical inspection, which is normally a bit
842110	848570	harder and takes much more time. You can do traffic analysis in your ICS network,
848570	854950	passive analysis, that we can also say. We can do configuration file analysis.
855470	861730	And we can do active scanning, which is not recommended, except you're, let's say, you are
861730	869190	not in the production or if you're in a turnaround, you can do that. But otherwise,
869190	876530	it's a bit dangerous. And then collecting the data in all the assets, you need to document it again,
876530	882050	in order to have a nicely done diagram and have the old assets.
883230	889770	And unfortunately, most of the ICS networks are flat networks. You'll notice that once you've done
889770	898090	the asset identification, after identifying the assets, it's often required to have a physical or
898090	905010	logical separation or both. It is quite necessary since it makes more difficult to adversaries to
905010	910690	pivot in your network. Otherwise, if it's a flat network, normally a game would be over within,
910690	916490	let's say, half an hour or one hour. For the active defense concept, it is really crucial
916490	924410	because you need to build on top of the good architecture and passive defense. At least for
924410	931050	me, the aim would be separating the network as explained in purging model. So it could be a good
931050	938850	reference to start with. And then if you want to have more separation, then you can use
938850	946310	micro-segmentation, etc. But it is the fundamental for the ICS networks. And network security
946310	953770	monitoring is a continuous process to collect, detect, analyze indication of threats in order to
953770	960710	respond faster to incidents and attacks in your network. It is threat centering approach instead
960710	966570	of traditional vulnerability centering approach. Network security monitoring requires dedicated
966570	973870	personnel as analysts. And it also requires preparation for infrastructure and also tools
973870	982890	ahead of time. And it brings proactive approach to security and detection of the threats. So
984010	992310	you can take some remediations once you identify the issues in your network.
994170	999690	And network security monitoring provides visibility and helps you to identify the changes
999690	1006850	and anomalies. Might not be relevant to only adversaries, but also
1008140	1011790	some misconfigurations for troubleshooting misconfigurations.
1011790	1018150	And it helps you to detect intrusion attempts and movements within ICS SCADA networks.
1018250	1024290	And it can evaluate separation and segregation for levels that you have and zones that you have.
1024610	1030930	And it can help fine-tuning and validating the settings of passive defense elements. For example,
1030930	1036790	checking the firewall rules and to see if it's correctly implemented or missing some points,
1036790	1043710	etc. And it also can help reduce the threat landscape by hardening non-required ports and
1043710	1048490	services. Since you are seeing this when you are doing the network security monitoring and
1048490	1057270	you can instantly take remediation. Network security monitoring is more useful in ICS networks
1057270	1066210	since most of the assets are critical. And most of the times, all the assets or any assets cannot
1066210	1074210	be immediately patched. Because, yeah, we know ICS networks are quite, let's say, untouchable.
1074210	1082160	And then you need to have any other layer to have the protection for your endpoints, etc.
1082590	1087790	And ICS networks have many dependencies and connections, such as enterprise network
1087790	1094150	connection, vendor connections, business applications, contractor VPNs. And it is
1094570	1100550	really valuable to monitor these external parties for anomalies because once your vendor or once
1100550	1107250	your contractor, let's say, compromised, it is just a moment to jump into your network by using
1107250	1114870	these VPNs. And normally, you don't have any vision on these connections without active defense.
1115650	1118090	And detection approaches.
1119550	1126490	Most of the time, what we check is identifying the most used IP addresses and ports used within
1126490	1133070	the network to create basically a baseline, identifying biggest bandwidth users, identifying
1133070	1139010	encrypted communication, because most of the times, command and conquer centers are using
1139010	1145290	encrypted communications. And identifying critical assets and usual traffic is also
1145290	1153670	important to continue on the baseline to have stability on that. And identifying network
1153670	1161150	anomalies, of course, and identifying lowest bandwidth and communication, because sometimes
1161150	1168730	you can see in the network unusual traffic, which could be pretty less than your usual levels.
1168730	1175310	And of course, you need to identify exfiltration. You need to check that
1176730	1182070	when you are doing the detection. And asset identification and network security monitoring
1182070	1186710	takeaways. Asset identification and network security monitoring is the key
1187370	1192770	for active defense concepts. Network security monitoring is a great approach for ICS because
1192770	1201670	most of the ICS networks are quite stable. And basically, once you create a baseline, it is
1202370	1210430	easy to, let's say, monitor. And these two elements support implementation of
1210430	1215870	better architecture and passive defense. Once you implement that, yeah, you can take a look
1215870	1222750	back and you can, let's say, improve your basic fundamental security also.
1222770	1228270	Detection often relies on the sensors in the ICS network. You need to be sure that
1228270	1236650	you need to detect the baseline changes, anomalies, and IDS rules. Because these can
1236650	1241470	basically warn you when you are doing the detection. And logs and visibility are really
1241470	1251310	important. But yeah, you need to have an analyst to, let's say, contribute value on that. Because
1252110	1258750	by itself, logs and visibility doesn't give you the chance that you can prevent. And analysts
1258750	1265490	need to follow it constantly and verify it if there is an anomaly. And network security
1265490	1270530	monitoring will eventually lead to instant response. You need to be ready for instant response.
1272170	1278730	By saying that instant response for ICS, it is a bit different comparing to
1279590	1289230	usual IT instant response. Because you cannot simply bring down the systems during the
1289230	1295310	instant response when you're working on ICS networks. And the focus is a bit different
1295310	1300470	by saying that maintaining safe and reliable operations is the most important thing.
1300470	1306850	Acquiring meaningful forensic data within a limited time. Performing a timely analysis.
1306850	1311710	And containing and eradicating the threats. Basically, these are the four elements that
1311710	1320460	you need to follow when you're doing the ICS instant response. Before that, preparation.
1320460	1325640	There are some steps that you need to follow. Preparation, same as traditional IT, but with
1325640	1330980	some limitations such as testing tools, testing methods, but in a lab. Integrated detection and
1330980	1335020	identification. Working with network security monitoring team to implement rules and detection
1335020	1341240	capabilities, which is tailored to your threats in order to identify impacted systems.
1341560	1349440	And evidence acquisition. Normally, you don't have that much time, like on IT side,
1349440	1356180	you don't have that much time to do forensic analysis because you cannot stop the operations.
1356880	1361980	Here, the focus is maintaining operations while acquiring enough evidence to perform later.
1362940	1369400	You'll do the forensic analysis later. And time critical analysis using fast and well-tested
1369980	1376560	techniques to quickly determine the overall impact to the operations.
1377340	1384240	And supported activities. IR teams mostly share information and evidence. All the evidence should
1384240	1390880	be passed to other teams to begin deep analysis and continue on the active defense cycle.
1390880	1395060	Containment. Preserving the operations by collaborating with other teams,
1395060	1401800	operators, and also engineers on the field. Eradication and recovery. You need to neutralize
1401800	1406180	the threat by, for example, reimagining the system, reinstalling known good software,
1406180	1412840	implementing patches. And then you need to provide the lessons learned, which is documents,
1413800	1420060	finding for all, pass information to the network security monitoring team in order to identify and
1420060	1426580	see if there is any reinfection, etc. And how do you prepare an incident response team?
1427480	1431520	You need to determine the requirements and dependencies within your facility.
1431820	1438600	For example, uptime, availability, and specific systems you need to take care. You need to decide
1438600	1443500	if it's going to be in-house or outsourced. There are some advantages on both sides.
1443500	1453220	But you need to decide, at least have a couple of people in your facility to do it more faster.
1453420	1458440	A team size three, four well-trained people on site would be enough to cover, but you also need
1458440	1464120	to think about the shifts, etc. So in total, it needs to be at least eight to 10 people.
1464320	1468640	So chain of command is really important because there will be chaos and someone needs to
1469480	1475880	communicate with the management team, etc. And it should be incident response director.
1476100	1482220	Then all the evidence and incident handlers need to report to lead responder. And the chain of
1482220	1488740	command needs to go up and down like this. And to build your ICS incident response team, you need
1488740	1496060	to find the right personnel. Of course, it's the hardest part because incident response is really,
1496060	1505020	to focus and develop yourself. And then also you need to take care of your jump kits because
1505580	1513180	often jump kits, when you need it during the incident, most of the time you are losing some
1513180	1519520	parts from the jump kit. It's easy to borrow from the jump kit. And then when you go to the incident,
1519520	1525780	you are missing a couple of hard disks, etc. when you need it over there. So it's important.
1526060	1534380	So evidence acquisition, you need to take care of the order of volatility.
1534900	1541580	And you need to decide if it's going to be local or remote acquisition. I would prefer local because
1541580	1551000	it's less risk and faster. The tools should be tested before on the systems because we are not
1551000	1556800	talking about regular IT systems. Acquisition should be coordinated with all involved personnel
1556800	1561880	and you need to discuss beforehand when you are touching any field device with engineers and
1561880	1568900	operators. You need to gather all the evidence if the time permits. You need to start from registry
1568900	1577060	and then any memory and then taking the disk images, for example. And you need to take necessary
1577060	1584340	photos in order to see on the, let's say, devices. If you see any command prompt, you need to
1585020	1593540	have an evidence because once you turn it off, you will lose it or you can lose it in any second.
1593560	1596540	And all the data should be analyzed in an approved facility.
1597660	1603920	What are the sources for the forensic data in ICS networks? Highly volatile data, system memory,
1603920	1610260	network information, and system processes, and VPN connections, and these logs, and registry
1610260	1619520	hives, HMIs, mostly Windows computers and you can have system logs, etc. Engineering workstations,
1619520	1626100	controllers such as PLCs and RTUs, and sometimes virtual resources such as
1626100	1632700	VMs and cloud environments if it's connected. And you need to do the high quality instant response
1634420	1639440	in a timely manner. It's really important in ICS when you are doing this.
1639600	1642920	Timely analysis is important to keep operations safe and reliable.
1643260	1649560	It should focus on understanding the scope and then the type of incident. And the baseline
1649560	1655380	information comparison will be really helpful when you do that. And focusing on the new connections
1655380	1662600	in the bandwidth, new routes, anomalies in the VPN connections,
1662600	1670100	registry keys, spawn processes, you need to check these first. And I said identification and network
1670100	1678160	security monitoring help to respond quickly. And utilize good tools to reduce the analysis period.
1678160	1682700	This is also important. You need to do the practice before going to the real incident.
1683540	1689260	And how do you use threat intelligence in the instant response? Indicator of compromise
1689730	1696140	to scope the infected systems. Then after that, you need to identify the network data on host.
1696220	1704580	And TTPs to identify adversary efforts. And you need to utilize and ensure threats are gone after
1704580	1713360	you've done the incident response. And instant response takeaways. You need to focus on providing
1713360	1718420	actionable information about the scope of the threat and its potential impacts while you are
1718420	1723680	acquiring the evidence without breaking operations and safety. ICS instant response
1723680	1729720	should be tailor-made. It's a bit different than the IT, as I mentioned. Efforts must align with
1729720	1735460	the goals and requirements of the operations. This is the first priority. Preparations should
1735460	1741880	be done ahead of time. You really need to practice before it really happens. And acquired evidence
1741880	1747540	and lessons learned should be shared with other personnel. So once you get evidence, you need to
1747540	1757340	share it to move forward the active defense cycle. And let's summarize this. Taking the
1757340	1763700	advantage of the found threats. Once you complete the instant response, if we can safely interact
1763700	1768700	and understand with the threat, we can have the best source of defense information from the
1768700	1774920	adversary's best capability. We can feed the active cyber defense center with the found threat
1774920	1781140	intelligence to build better defense. It is important to identify and use indicators of
1781140	1787600	compromises from the threats to help instant response. And understanding the malware's tactics
1787600	1792620	to identify weaknesses in the current ICS architecture is valuable because after that,
1792620	1801080	you need to improve your architecture. So these steps are really valuable. And my last presentation
1801080	1808000	is about lessons learned sessions for long term success. For the success of active cyber defense
1808000	1813980	cycle, lessons learned should be shared between the teams internally. And also necessary actions
1814730	1821360	needs to be taken to build better defense. And after that, either during the incident and after
1821360	1827280	the incident. And after that, once you share it internally, it's also recommended to share
1827280	1834720	your lessons learned in an appropriate way with the ICS community because if you had that threat,
1834720	1840400	someone else might get this soon. So it is really important and valuable.
1842080	1847600	And yeah, I think I'm just on time. Thank you very much for listening to me. If you have any
1847600	1855640	questions, either you can send me a message on LinkedIn or Twitter or you can ask now. Thank you.
1856180	1857280	John, this is...
1864960	1872540	Okay, so everybody look forward to see John and ask questions and all sorts of different things.
1872800	1876540	On the DEF CON discord server, we're in the ICS village subgroup.
1877200	1880020	And John, thank you very much. That was a fantastic presentation.
1880020	1882360	Thank you. Cheers, bye.