[00:18.590 --> 00:21.810] Will you guys let me know when to start? [00:27.480 --> 00:32.960] Good afternoon and good evening, everybody. Welcome to my talk. [00:33.060 --> 00:36.800] Thank you very much for being with me or watching this video. [00:36.800 --> 00:41.940] And thank you, ICS Village organization. [01:23.590 --> 01:25.470] Shall I continue? [01:26.130 --> 01:27.530] Go for it. [01:28.430 --> 01:30.090] Oh, okay. [02:10.630 --> 02:14.030] For those of you tuning into the ICS Village talk, [02:14.030 --> 02:18.330] this is Bryson Borth, the lead for the ICS Village. [02:18.330 --> 02:24.110] We're having some technical difficulties with Marina calling in from Europe. [02:24.250 --> 02:28.010] Her connection keeps dropping, so bear with us. [02:28.010 --> 02:29.710] We're trying to work through this. [02:29.710 --> 02:34.350] If we're not able to successfully pull this off during this hour, [02:34.350 --> 02:38.210] we're going to find another time to record a talk with Marina, [02:38.210 --> 02:41.130] and we will still share that with you after the conference. [04:10.670 --> 04:12.530] Hello, and I'm back. [04:12.630 --> 04:18.030] I don't know, I'm glad that I managed to get to internet connections for this talk. [04:18.030 --> 04:20.090] So now I'm on the backup connection. [04:20.090 --> 04:24.630] So I am a true good, I have a good cybersecurity program in here. [04:24.870 --> 04:28.710] So, well, and so I stopped that. [04:28.710 --> 04:32.110] I used to be this offensive vulnerability researcher. [04:32.110 --> 04:36.570] And in recent years, the word offensive is kind of have a very negative meaning. [04:36.570 --> 04:39.550] And that's why I call myself right now a vulnerability researcher. [04:39.550 --> 04:43.230] And I used to do very like crazy and exotic things. [04:43.230 --> 04:46.810] I have six accepted and five presented black hats. [04:46.810 --> 04:50.570] I have two different CCC troopers, SAS, hack in the box. [04:50.570 --> 04:53.270] I even keynoted at ICS JWG. [04:53.270 --> 04:55.090] So I've done offensive stuff. [04:55.090 --> 04:59.830] And I used to create this travels for asset owners and plant operators. [04:59.830 --> 05:03.970] And I believe that my threats are the most important and urgent to defend against. [05:03.970 --> 05:08.350] And I was very opinionated about that. [05:08.430 --> 05:11.570] But now I'm a manager of the cybersecurity program. [05:11.570 --> 05:16.770] And I'm responsible for keeping my customers plant safe and secure. [05:16.930 --> 05:20.870] Yes, I still think my past research was cool and very important. [05:21.930 --> 05:28.970] But no, I don't think that defending against my cool and very important threat is my priority right now. [05:28.970 --> 05:31.450] And this is what this talk is about. [05:33.630 --> 05:35.850] So this is what's my past life. [05:35.850 --> 05:37.910] And this is what you see here on the photo. [05:37.910 --> 05:45.490] It's how I was getting ready for my Black Hat USA 2017 talk. [05:45.490 --> 05:48.230] And this is somewhere 1am in the night. [05:48.230 --> 05:53.350] And I gladly spent time with all of my hardware and servers and troubleshooting. [05:53.350 --> 05:55.850] And it really was very satisfactory. [05:55.850 --> 05:59.370] But my life has changed entirely since then. [05:59.370 --> 06:04.010] And now I became a consumer of problems I used to create. [06:04.850 --> 06:08.870] So my current life is really extraordinarily boring. [06:08.870 --> 06:13.170] All of those things which all of us typically hate. [06:14.510 --> 06:17.710] Agenda is my second nature right now. [06:18.550 --> 06:23.710] I cooperate on terms like standards, compliance, KPI, policies, procedures. [06:23.710 --> 06:26.130] Everything has to be very cost-effective. [06:26.230 --> 06:28.270] And it's all about risks. [06:28.730 --> 06:33.470] So, you know, at the beginning I was kind of a little bit upset. [06:33.470 --> 06:37.090] Like, what did I agree to do for this job? [06:37.090 --> 06:41.210] And as my very good cybersecurity friend, also a researcher, told me, [06:41.210 --> 06:43.870] well, congratulations, dear, you are a manager now. [06:43.870 --> 06:45.850] So, yeah, my life has changed. [06:46.250 --> 06:49.470] And yes, some of you will probably argue, [06:49.470 --> 06:53.730] but Marina, you previously worked as a lead and senior cybersecurity consultant [06:53.730 --> 06:56.670] and an engineer and a principal threat analyst before. [06:56.670 --> 06:57.570] You've done this job. [06:57.570 --> 07:01.650] You've consulted customers about cybersecurity programs. [07:01.650 --> 07:04.350] You did threat assessments. [07:04.350 --> 07:05.570] You should know all of these things. [07:05.570 --> 07:07.730] This shouldn't be something new for you. [07:08.070 --> 07:09.470] And that is a difference. [07:09.470 --> 07:13.830] As a consultant, it's your job to actually point to the problems. [07:13.830 --> 07:15.130] You have to find them. [07:15.130 --> 07:18.070] And as an analyst, you also have to report about problems. [07:18.070 --> 07:20.430] And then you leave the customer with a recommendation, [07:20.430 --> 07:23.430] and you actually do not see the larger picture. [07:23.610 --> 07:26.750] So money is not involved because you're not actually remediating [07:27.210 --> 07:29.690] or building plans to remediate. [07:29.690 --> 07:34.430] You're not really going into the individual circumstances of individual plans, [07:34.430 --> 07:38.770] maybe internal politics or relationships and whatnot. [07:38.910 --> 07:42.610] So this part was not previously present. [07:42.610 --> 07:44.850] And now I'm responsible for the entire picture, [07:44.850 --> 07:47.230] because I'm going to the sales meetings. [07:47.230 --> 07:50.710] I then participate in bids and project proposals. [07:50.710 --> 07:54.580] I am then participating in the building and configuring systems, [07:54.990 --> 07:58.310] like entire solutions for the customers, networks. [07:58.410 --> 08:01.970] And then when the customer is operating the new environment, for example, [08:01.970 --> 08:05.330] or upgraded environment, or even if it is a brownfield, [08:05.330 --> 08:10.130] I also consult on the advanced service or provide advanced services. [08:10.630 --> 08:14.490] So basically, when I go already with the sales meeting into the customer, [08:14.490 --> 08:18.430] I already see in advance how we will be implementing it and building [08:18.430 --> 08:19.750] and what we will be consulting. [08:19.750 --> 08:24.550] And I need to consult customer in advance what they might need in the future, [08:24.550 --> 08:26.410] because I know what they will need. [08:26.410 --> 08:31.490] So basically, it's really a big difference between then and now. [08:31.490 --> 08:37.250] And maybe the best way to explain is that previously I've seen this local optimum. [08:37.250 --> 08:39.390] So I was working on one task, like, for example, [08:39.390 --> 08:45.490] hey, I just do assessment of one plant and then maybe I even propose mitigations [08:46.070 --> 08:49.570] and maybe even in a prioritized fashion, but it's a local optimum. [08:49.570 --> 08:52.390] Now I work, for example, for building cybersecurity programs [08:52.390 --> 08:55.630] for the entire company with multiple plants. [08:55.630 --> 08:58.330] So now I have to optimize the global optimum. [08:58.330 --> 09:03.430] And actually, for me personally, this is the best analogies to describe [09:03.430 --> 09:08.150] what I'm doing right now and overall what I need to achieve right now [09:08.150 --> 09:11.130] in my new role is the best example is linear programming. [09:11.130 --> 09:13.950] I don't know how many of you know about that, but as an engineer, [09:13.950 --> 09:16.950] this is what I used to do a lot for engineering problems. [09:17.390 --> 09:21.270] And I put the link into the Wikipedia. I highly recommend to check it out. [09:21.270 --> 09:25.730] And it's basically every problem you need to solve or optimize, [09:25.730 --> 09:30.030] because this problem, this linear program is frequently used for operating, [09:30.030 --> 09:36.550] for solving operational problems and finding there, optimizing, [09:36.550 --> 09:40.990] for example, profit or something else based on the constraints [09:40.990 --> 09:44.150] which need to be fulfilled. And you then formulate the problem [09:44.150 --> 09:46.990] which you need to optimize and you formulate your constraints [09:46.990 --> 09:52.830] and then specific applications will try to solve you that optimization problem. [09:52.830 --> 09:54.790] And sometimes it takes hours and whatnot. [09:54.790 --> 09:58.530] So this is how I actually think about pretty much every problem [09:58.530 --> 10:04.010] because we always try to find this optimum based on the constraints we have. [10:04.050 --> 10:07.610] And then my new role is just like the optimization equations got longer [10:07.610 --> 10:11.050] and the list of constraints got longer. [10:11.050 --> 10:15.550] So optimization is becoming more and more difficult and challenging and trickier. [10:15.750 --> 10:17.610] But that's the beauty about it. [10:17.970 --> 10:21.730] So, and maybe the situation in which I am right now, [10:21.730 --> 10:24.570] it reminds me a little bit of Carsten Noll's story, [10:25.590 --> 10:29.530] because he used to be a cybersecurity researcher, he is still, [10:29.530 --> 10:31.470] and he was a source of troubles. [10:31.470 --> 10:36.230] And at some point he became a CISO of the richest startup in the world. [10:36.230 --> 10:41.070] It was in India. And basically he became a consumer of own troubles. [10:41.610 --> 10:47.150] And he gives insightful talks on how it is impossible to build [10:47.150 --> 10:53.550] the perfect cybersecurity program or just security for any application or anything, [10:53.550 --> 10:56.130] even with all the money and knowledge in the world. [10:56.130 --> 10:59.830] And it's really fascinating talks which he gives. [10:59.830 --> 11:01.330] Unfortunately, there are no recordings, [11:01.330 --> 11:05.230] but I was lucky enough to see it in person a couple of times. [11:05.330 --> 11:08.570] But it's interesting to see that even if you have all the money [11:08.570 --> 11:11.230] and all the knowledge, you know, finding this, [11:11.230 --> 11:14.650] optimizing that equation with so many constraints is impossible. [11:14.650 --> 11:17.210] You just have to find like basically good enough. [11:18.050 --> 11:21.970] So before I go into jump into the talk, a couple of disclaimers. [11:21.970 --> 11:25.030] This is the least technical talk I've ever gave. [11:25.030 --> 11:31.610] Still too many slides. I'm still working on trying to have fewer slides for my talks. [11:31.750 --> 11:34.830] I think maybe some of you this talk may be boring. [11:34.830 --> 11:36.750] So I'm sorry for that. [11:38.150 --> 11:42.170] And I know that many of the professionals in this field, [11:42.170 --> 11:46.310] especially who has many years of experience after seeing this talk, [11:46.310 --> 11:52.230] they will say, well, I told you so. Yes, I know you did 10 and 7 or 5 years ago. [11:52.230 --> 11:55.850] But I was a different person at the time. I agree with you now. [11:56.850 --> 12:00.970] So this talk is based on my own thoughts and considerations. [12:00.970 --> 12:06.690] So whatever tips and suggestions I provide, no warranty they will work for you. [12:06.690 --> 12:13.530] And honestly, for my current life phase, I work on so many projects at the same time. [12:13.530 --> 12:18.870] I have to switch quickly. I realized I really became comfortable that good enough is actually perfect. [12:18.870 --> 12:21.130] I used to be such a perfectionist. [12:21.410 --> 12:27.010] So basically, like who I was in all good times, you know, that was my entire world. [12:27.010 --> 12:33.810] You know, like I've been looking for the most weird, difficult, exotic ways of exploiting cyber-physical systems. [12:33.810 --> 12:37.870] I was mostly working at the layout of controllers and physical processes. [12:38.030 --> 12:43.810] And the more difficult and challenging problem was, the more I was interested and determined to find the solutions. [12:43.870 --> 12:53.130] And, you know, like I used to have calls with my security researchers friends and bashing about like, [12:53.130 --> 12:58.750] oh my gosh, can you imagine they did not touch that vulnerability, how terrible it is. [12:58.750 --> 13:01.230] You know, I used to be all of that person. [13:01.230 --> 13:13.630] And now when somebody is telling me like my customers telling me, well, do you think we need to publish that remote code execution vulnerability in some device? [13:13.630 --> 13:19.790] And I say like, no, if you have some time and money to spend, please build your demilitarized zone. [13:20.550 --> 13:28.270] So and basically, like if I would try to explain like what suggestions I was previously given, this would be example. [13:28.930 --> 13:36.390] For example, this picture, this cluttered room is basically a good example of how traditional OT environments look like. [13:36.390 --> 13:41.610] Like many of them, you know, it's just like, like this cluttered room, you know, like there is everything everywhere. [13:41.610 --> 13:45.930] There is no really structure. Nobody knows what, where, somehow it works. [13:45.930 --> 13:48.550] And I guess many of my OT colleagues will understand. [13:48.550 --> 13:57.570] And I was saying that like if the context of this cluttered room, I was telling that if we would apply some very advanced cybersecurity solutions. [13:58.370 --> 14:07.890] You know, like for example, an analogy would be to put this beautiful chandelier in this room and it will suddenly this room will become beautiful and organized. [14:07.890 --> 14:19.870] And in reality, it won't, you know, like I was trying to tell like, like if you put this chandelier room or basically it will transform that room into the Omnia club in the Las Vegas. [14:19.870 --> 14:23.870] You know, I'm not nostalgic at this point, but of course it won't happen. [14:23.870 --> 14:31.670] So obviously when I was recommending like skip the basics, apply some advanced cybersecurity controls and you will be fine. [14:31.670 --> 14:37.410] Clearly that was a really wrong recommendation and I am admitting it now. [14:38.390 --> 14:52.970] So, and basically now that I was that person, and now like now when I basically my customer come to me and saying like hey this is my cybersecurity budget, and I need to understand how best I spend it. [14:53.390 --> 15:00.690] You know, like, I have to be pragmatic and cost effective, and I have to change the way how I think. [15:00.690 --> 15:05.030] And this is how I've learned actually to like secure architectures. [15:05.830 --> 15:17.890] So for example, we all know how the traditional ICS network would look like we have a corporate network and skater network and somewhere on the bottom of it we have a physical application which we are trying to protect. [15:20.370 --> 15:36.410] And basically, typically we build such networks in a layered fashion you know about that and it's all determined by the data flows at the bottom we have real time data exchange, and it's become less real time at the top. [15:36.410 --> 15:49.310] And in general, this could be basically divided into two parts like process control and monitoring at the bottom and process management in ideally you should build them in such a way that you could always decouple this upper part. [15:49.490 --> 16:00.290] So you may lose all of your corporate and layer three, like this process management, you should be able to operate, you might not be able to serve your customers like dispatch your chemicals or whatnot. [16:00.290 --> 16:09.990] But you still save and you still can produce and if you're, you know, like vessels are full you may just shut down plant, but in a safe manner. [16:10.370 --> 16:20.490] And like basically if you look at the entire architecture how it's like the very classic Purdue reference architecture look like and this is looks like this. [16:20.490 --> 16:32.830] And actually I previously, I mean of course it always made sense for me from the data processing standpoint but now I started to appreciate it from the organizing and building cybersecurity. [16:33.490 --> 16:46.450] And I will explain why. And this is basically it also helps me to explain customers, how they should prioritize their expenditures so that I could have a meaningful conversation which they also this argumentation which they can follow. [16:46.450 --> 17:01.350] And at the bottom, we have actually this is where our physical process and this is where the hazards leave, and we are trying to prevent hazardous situation and we have sensors and actuators, which are solid units, which are completely trusted. [17:01.350 --> 17:10.390] At the moment there is no security controls which we can apply there and it's considered to be trusted there is so basically there is nothing what we can do from the security standpoint there at the moment. [17:10.390 --> 17:13.870] But this is where we actually this is the layer which we are trying to protect. [17:13.870 --> 17:30.770] And then we have a very gigantic this layer one layer two ways you distributed control system. And here's actually one more disclaimer is that clearly I'm mostly specializing in chemical pharmaceutical oil and gas so basically large continuous processes [17:30.770 --> 17:34.250] so this is a background which I use in my presentation. [17:34.250 --> 17:45.950] So, this is why I'm talking in the context of distributed control system. And while this is already system with the computers and we could actually apply traditional security controls which we apply in the corporate network. [17:45.950 --> 18:01.630] Things are very difficult in here and this is why I put, it's complicated. At the moment, unfortunately, the situation is such that the market is full of really fantastic solutions which we could apply for security and a lot of IT companies also basically [18:01.630 --> 18:04.830] trying to push their solutions into this layer. [18:05.130 --> 18:15.470] But at the moment we still did not find a good way how to basically approve and manage and apply this security solutions in this layers. [18:15.470 --> 18:26.910] And the point is the truth is also that vendors like ABB and all others whatever you think of, like Siemens the most famous Yokogawa Honeywell and so on. [18:27.050 --> 18:42.750] Each vendor has to approve specific solutions to be applied. So the customer is pretty much limited to what they can use. And because at the moment there is such a competition and a variety of products they're still not even sure what is the best and what is most [18:42.750 --> 18:52.270] cost effective, that it's actually very difficult to apply like really implement proper cybersecurity technical cybersecurity controls in this layer. [18:52.590 --> 19:02.570] So where we actually left with is our layer three and layer and demilitarized zone, and this is where you actually can. [19:02.570 --> 19:15.510] This is already full fledged servers, you can patch them you can use a host agent for advanced threat detection you can you can do vulnerability scanning threat hunting whatnot. [19:15.570 --> 19:27.570] It's all yours. So this is where you can actually collect the logs from firewalls monitor data flows and whatnot. So this is actually you can implement the full fledged cybersecurity program in here. [19:29.950 --> 19:35.510] And basically apply patches and reboot service without interrupting productions. [19:35.930 --> 19:45.870] And you would probably argue, but Marina, we still want to have this defense in depth, you know, like we should still do something in layer two and level one. [19:45.870 --> 19:56.070] And here's where I actually find a good argument why we should still concentrate most of our efforts on demilitarized zone and level three. [19:56.070 --> 20:04.490] So when we talk about level one and level two, this is already they are actually part already of the safety protections and I will explain you why. [20:04.550 --> 20:19.910] So this slide you see actually the relationship between security and safety like if you put it like in a temporary temporary fashion. So we have layers of security protections which actually protects us from stress, and behind the every set of human is [20:19.910 --> 20:46.110] always behind me it's something intentional. And if the attacker was able to go through the all the layers of security protections we have already security incident, and then we say, well, this is where we rely on the layers of safety protections, so that the [20:46.110 --> 20:53.150] you can see it on the slide. They are actually already part of the security layers. These are layers of safety [20:57.850 --> 21:08.830] preventions measures. So we basically, it suddenly makes more sense to keep attacker as far away so we basically have to really build a very strong perimeter and make sure [21:08.830 --> 21:27.050] basically minimize chances that even the attacker will even approach our layers of safety protection because it's our last line of defense. But there is one more argument, and it goes into the finances, we all know that, you know, like operations, even a small [21:27.050 --> 21:44.010] hiccup on the network when maybe packets is not arrived, that already may cause process upset and majority of incidents or cyber related incident when which happens in the plans, they're not security they're cyber incident we are not enough memory not not [21:44.010 --> 21:58.530] enough storage network congestion, or whatnot. But all of those, they already introduced interruptions into the normal operations and actually reducing the efficiency of how we run the process. [21:59.230 --> 22:11.130] And this is a really big problem for the operators because they are operating on there. So, yes, the number one priority is safety but their second number two priority is to maintain actually [22:14.010 --> 22:23.030] their operations. And if you look at the, like on the left you have a slide, when we look, look at the about integrity protection layer three we have alarms in there. [22:23.050 --> 22:37.350] And there is even explicitly stated financial alarm, you know like we're losing money. And on the right you see another graph, where we still operating may still operate in the normal operational envelope is still producing our product, but we already not in [22:37.350 --> 22:51.250] profit. We already losing money because we are not producing effectively, the product is still up to specification but we are using too much energy, losing too much useful chemicals maybe in the purge or whatnot. [22:51.250 --> 22:53.010] So they're not. [22:53.170 --> 23:09.510] And this is why you know like we know that pretty much any hiccup on this on the DCS operations may cause this inefficiency in the productions. And this is one second reason why we really want to keep the attacker away. [23:09.510 --> 23:17.370] We want to minimize chances of attacker, even reaching that level we don't want any random person roaming in our industrial control network. [23:17.370 --> 23:22.170] And this is actually to give you an example. [23:22.810 --> 23:38.810] Like how much money we may lose if the operator will not, for example, react on time. So, on the left you see that, for example, when we talk about minor effects something what you know just minor it's not even medium it's not meant major it's minor. [23:38.810 --> 24:08.570] It's when you lose estimated like from 10k and 100k, you know, if you talk about the budget of 100k, I think people cannot even imagine such budgets you know like this will be fantastic pentest of the entire corporate network, and we fight for years to get that, you know, [24:08.570 --> 24:17.930] they basically have to react to emergency alarms first, and this is when they have very little if you can see we have very little reaction time five to 15 minutes. [24:18.410 --> 24:27.010] And if they will not react within that time, like for example within 15 minutes, the loss could be up to from one to 10 million or even more than 10 million. [24:27.010 --> 24:30.490] So imagine just those numbers. So imagine them. [24:30.770 --> 24:40.050] We do not really strong build strong perimeter, and we say well even if the attacker gets to the control layer. [24:40.390 --> 24:42.870] We will still try to detect him in there. [24:42.870 --> 24:56.930] But if the attacker will manage to suppress just one stream was one, this critical alarm. If the operator just basically lost his visibility for a few minutes, he fails to react on time, and the losses can go up to 10 million. [24:56.930 --> 25:14.730] So, as you can see because of this very short reaction time high losses. We really want to keep all the attackers away from our control environments, and this is why I still argue that we have to introduce as many technical controls in the layer three and demilitarized [25:14.730 --> 25:24.690] and try to detect. So make sure that the only conduit into your control network has so basically the attacker has to go through the demilitarized zone and through layer three. [25:24.690 --> 25:35.650] There is no any other way into the control network. So basically the attacker has to go there, and you try to maximize maximum to detect an intrusion in those layers. [25:35.650 --> 25:51.570] So this is why I say this is your top priority. And for example prioritize spending 80% of your expenditures in those layers. And that means also that you first have to actually build your proper reference architecture, you need to form a proper DMZ zone. [25:51.570 --> 26:06.830] You need to properly form your level three, and this is very expensive, it's very difficult, it's complex, but this is should be your number one priority and I can't believe I'm saying that because two years ago I would not believe that I would be saying something like that. [26:06.830 --> 26:21.570] And actually the layer one and two, this is where most your security controls at the moment can take form of policies, procedures, and common sense, prayers, you know, like check karma of your employees and whatnot. [26:22.690 --> 26:34.750] So, of course, you know, like, there is also slight deviation, you know, the standards will tell you how the absolutely ideas, ideal network architecture should look like. [26:34.750 --> 26:45.790] And then, you know, there is a reality because you know it's all about balancing security versus usability and versus cost. Again, we're talking about linear programming optimization. [26:45.790 --> 27:03.550] And for example for our ABB control system and I'm saying this is for ABB, it might not be applicable to every organization. We built for example, slightly, I would not want to say simplified because this is security level four architecture, but it's a little [27:03.550 --> 27:19.150] bit more manageable because, for example, for certain operations like for example for antivirus updates and backup, we do allow routing of the traffic between DMZ and level two, but that is because the way how we implement those services because we still do them [27:19.150 --> 27:34.310] in a very secure very conservative way. And we are very well aware that this is not a risky to do so but it is much more manageable architecture. And that is the point that for every organization that architecture security architecture will look. [27:34.650 --> 27:45.430] I mean it must be individual and might look differently. It's all about basically implementing all of your data flows and access control and identity management in a secure way. [27:45.430 --> 27:56.630] So for example, once you map all of your data flows and file exchange routes, and you make sure that all they are very secure, then you're good. [27:57.050 --> 28:06.010] So, but that is has to be properly analyzed. And again I'm saying that it might not be looking like a traditional Purdue architecture but it still has to be secure. [28:06.010 --> 28:22.630] But once you build a security architecture and unfortunately in this talk I do not even have time to go in all of the zones and conduits and why it is so amazing. But once you have this granular and properly build architecture which is capable of managing, then it is very [28:22.630 --> 28:32.250] easy to actually introduce any other service because, like I will jump back to the slide we have like layer three where you have operations management, this is basically your service layer. [28:32.510 --> 28:44.870] And you actually may share it so if you have a site with several plans, you may share that level three between multiple sites, you will have multiple level two but then one layer three which will be services for all sites. [28:44.870 --> 29:00.070] And again, once you build this properly format, and you're capable of managing your architecture, introducing new services become really plug and play. So this is an example I remove the name of the application but this is one of the ABP application [29:00.070 --> 29:15.470] advanced service, like advanced application for process control. And we already build them and implement them in such a way that it will you just plug and play to your existing infrastructure and integrating a new service and introducing it and configuring [29:15.470 --> 29:17.390] is extremely easy. [29:17.470 --> 29:22.650] And it's just example of ABB but it's could be any application and any vendor. [29:22.650 --> 29:29.450] So, when I work with a customer so this is exactly the processes and explanations which I'm going through. [29:30.390 --> 29:46.830] It's probably sounds for all of you so boring like Marina like ways vulnerability research where CV, but that is my new world right now and I have to break this complex problems into easy arguments so that I could actually implement the fundamentals right [29:46.830 --> 29:49.810] so that I can play to build the advanced security controls. [29:49.810 --> 29:51.990] And on. [29:52.190 --> 29:58.970] So, many customers rightfully argue that it's expensive to implement correctly the entire architecture. [29:58.970 --> 30:12.630] At the beginning. Yes, but in the long term. It's gigantic cost saving factor, and it's, it provides you really very manageable, very high level of security assurance. [30:12.630 --> 30:17.330] So, of course life would be boring if it would be easy. [30:17.410 --> 30:33.350] You know on one hand you know like it's so easy to say, well just put in firewall and then you're done you just like have your DMZ, the reality is not so easy because you know such projects so every customer is constrained by the available budgets and sometimes [30:33.350 --> 30:37.890] by schedule like hey I scheduled my downtime on this time and I need to be very quick. [30:37.890 --> 30:49.630] And you see like forming a DMZ, or even just adding a fire extra firewall. If they have already one firewall and we add in like North, they have already North and we add in South. [30:49.830 --> 31:05.350] It's not that easy it's not only hardware cost but it's also you know like a lot of. I put here on the slide, it's just a snippet an example from our real project where we are building a DMZ it's not complete I hide the details but just give you an idea how many [31:05.350 --> 31:16.810] activities goes into such projects, it all costs money. So eventually it's not just the buying a firewall but executing such project is expensive because a lot of activities goes in there just to configure everything. [31:17.310 --> 31:28.970] And, you know, eventually because also building firewall DMZ meaning you know like you have to proper build your services or servers and split services to make sure that everything is secure and everything. [31:28.970 --> 31:40.630] But then it's suddenly because you still have this amount of budget and it suddenly become this Tetris for adult game. You know you're trying to shuffle this budget like maybe we will remove a little bit in here and do there. [31:40.690 --> 31:57.330] We will remove switch but add a server in there and so on because you're trying to optimize customer value with available budget. So it's not that easy to build it and this is always you know like the most interesting things are like always in the details. [31:57.330 --> 32:15.010] But still, I would argue this point of my life and in the current role that for me the proper DMZ is a higher priority activity than for example, patching a code execution vulnerability in the control at some lower level, you know, like, and anyway, [32:15.010 --> 32:25.630] you know, like, at the moment patching controllers is, I mean depending of course on the CVE but you know like, since most of them do not even have authentication and you can directly talk to them. [32:25.630 --> 32:27.970] Why even bother patching. [32:28.130 --> 32:35.770] But again, probably many of my cyber security researchers don't like what I'm saying right now, and we can talk about that reach out on Twitter. [32:36.430 --> 32:52.330] Well, here's a pro tip, you know, as I said, building security architectures is expensive, and, you know, sometimes plant operator experience really difficulties when making those financial decisions whether or not to invest into extra firewall extra server [32:52.330 --> 33:12.350] and so on. So here's my pro tip, like, how to ease that decision making for the customer, conduct a high level simplified risk assessment of specific architecture in hands or data flow and document identify risk, explain risk and potential consequences [33:12.350 --> 33:13.870] to the operators. [33:15.110 --> 33:30.270] Remind them that ultimately they are responsible for all the risks, they're planned, and they shall eventually address them by accepting or mitigating and if mitigating they considered this point too expensive, they have to be aware that they will have to accept [33:30.270 --> 33:32.950] that risk and this is the potential consequences. [33:32.950 --> 33:44.130] And typically you know what the simple process makes the decision so much more easier. Okay, you know what, I don't want to be responsible for that, put the firewall, we have we will find budget. [33:44.390 --> 33:46.570] So, you're welcome. [33:46.570 --> 34:00.330] And, you know, sometimes, also like this is why I learned to like compliance because sometimes you would say well if they're still doubting shall I shall I not you will say hey the standards is telling you or the best practices are telling you and then it's [34:00.330 --> 34:03.870] actually makes the decision making also easier for the customer. [34:04.690 --> 34:16.550] So, the second part is how I learned to love standards, which I never done before. So, um, in June, last year I actually gave a talk. [34:17.470 --> 34:34.350] About 62 443 how it is applied about to this asset. And I actually was bashing about the standard, a little bit I, I was, of course, a factual but I kind of like express couple of disagreement. [34:34.730 --> 34:39.330] And because I have a lot of very good friends who actually build the standard. [34:39.330 --> 34:56.490] You know like I never published the slides because I was a little bit ashamed, like how could I say some negative things or basically point out to some maybe imperfections in this in the standard and gosh how I'm ashamed of myself because you know like right now. [34:56.650 --> 35:01.730] I absolutely love and adore 62 443 that's everything for me right now. [35:02.110 --> 35:03.770] And here's the point. [35:03.770 --> 35:21.610] So why I started liking it. So, you know, performing technical cybersecurity plans audits is not actually very hard. I have genetic like really extensive experience in auditing plans, and I've done it for multiple organizations for multiple industries [35:21.610 --> 35:39.250] globally so that's something where I'm like skilled and I have no issues with that, where it's becoming difficult and complex is when you actually have to build the entire cybersecurity program or build a program for mitigating and prioritize fashion with timeline [35:39.250 --> 35:44.270] cost, and suddenly it's becoming a very complex expensive and time consuming process. [35:44.270 --> 35:53.530] In addition to that, you know, because if it is a process and you invest money into it. You need actually to justify to find some measured value [35:55.810 --> 36:04.490] to justify the expenditures to the project sponsors sometimes it's government sees or whatnot. You actually need to measure your progress. [36:04.490 --> 36:19.650] And you actually need also to compare, you know, cybersecurity posture and pro mitigation progress of individual plans which are completely different, different maturity, but you need to compare it in some normalized metrics. [36:19.650 --> 36:25.370] And this is where this business risk KPIs all came back into my life, and I suddenly. [36:26.550 --> 36:41.650] It's reminded in all my knowledge from which I got so I have an MBA degree and suddenly all that knowledge came handy to me and I started like basically remembering all of what I've learned in my MBA, and I put it into practice. [36:42.410 --> 36:48.530] And here's another thing. So, the fourth principle of economics are 10 of 10 of them. [36:49.250 --> 37:02.130] The fourth one is my favorite people respond to incentives, it means that we do certain actions and make decisions based on their perceived value to us, it could be also negative and it could be positive value. [37:02.130 --> 37:07.770] And this is a fun story when I had a meeting with a season for very large [37:10.310 --> 37:11.990] industrial organization. [37:13.030 --> 37:29.390] And I will keep telling that him about our success stories where we've done fantastic projects for our customers and then I told him like, hey, we built for a large LNG plan to cyber security program where we matured customer from quadrant five to 1111 it's [37:29.390 --> 37:36.890] kind of recommended for oil and gas industry so basically security level three maturity level three. [37:37.650 --> 37:55.490] And, um, and his immediate questions was, and where we are now, you know like people are competitive, you know like they want to compare themselves, and I was just like, well, let's build your KPIs about those and let's explain you how you can get from some lower [37:55.490 --> 37:57.830] quadrants into the higher quadrant. [37:58.090 --> 38:10.510] And the, what I like about 62443 is that it's provides a lot of logical and meaningful ways to build your KPIs when building cyber security programs. [38:10.690 --> 38:23.010] So for example, even, I must admit I previously was skeptical about the security levels and how they are described. Right now I like them because they really helped me to. [38:23.010 --> 38:35.450] For example, when we talk about security architectures you know like we need to perform risk assessment of every system like fire and gas and emergency shutdown and process control and safety and whatnot. [38:35.590 --> 38:46.390] And then we need to assign security levels based on the criticality risk criticality and also sign those security levels to conduits and then we need to meet those requirements. [38:46.390 --> 39:03.210] And right now, I really like it because it gives me something tangible to work with. And we know that different plans are have different maturity so some of them terribly legacy some of them, a mixture of legacy new some very new. [39:03.210 --> 39:21.510] And the standards is also provide us with additional metrics so that you can for example measure capability security level like well like legacy system will not be very capable, but your target security level is, let's say your capability level is one [39:21.510 --> 39:33.730] and your target is three, then the standards will tell. Well, you can use compensating controls and see whether to achieve the target level and then you measure your progress. [39:34.570 --> 39:48.630] With this achieved security level you probably will go from level one to level two and level three. So all of the security levels in the standard they provide a really way to build very nice KPIs to measure the effectiveness of your program. [39:49.490 --> 40:06.570] And then of course you need to measure also how well you're doing. And that standard has a maturity levels, and this is virtual actually forms those quadrants, it's basically like which security level you were able to achieve and how mature you in maintaining [40:06.570 --> 40:24.130] that security. So security controls, and what I like about this maturity levels so for example, if you already have. So, level one is just like you have a security control and level two if you basically build your documented process about that you already [40:24.130 --> 40:38.590] updated in level two it's so easy to progress in those quadrants and level three is actually if you are practicing so for example, you have antivirus level two you have a documented process how to update an antivirus and level three you actually updating that antivirus [40:38.590 --> 40:55.850] and you already. So basically you can use this quadrants for pretty much every security controls and that will help you to actually monitor and measure the progress, your mitigations progress, which for any season governance will make very fantastic numbers [40:55.850 --> 41:04.110] and of course you can build much more smarter and more interesting KPI but this could be building you a very fantastic foundation. [41:04.110 --> 41:12.270] So, this is also what I can work with in terms of easy to explain to the customer, and it seems like an achievable process. [41:14.290 --> 41:15.170] So, [41:15.170 --> 41:34.470] and this is an interest not this interesting thing that I used to hate, even the word policies and procedure. I used to teach ISO 2701 at the university, and it was the least my favorite topics, I would have an allergy for the word policy and procedure. [41:34.750 --> 41:45.150] And now this is my number one to go. So when I start working as a customer with says like well we've never done anything with the security and now we understand it's an important part. [41:45.150 --> 41:47.170] Government wants it. [41:47.290 --> 41:52.270] We understand that the risks are higher the moment you want to build the cybersecurity program. [41:52.430 --> 42:02.970] And that is the number one what I need from them is like guys, we need to establish policies and procedures. Policies will tell you what I want in terms of security. [42:02.970 --> 42:17.690] Your all security goals need to be described in the policy it's very high level, very in short sentences and procedures will actually tell you how you're going to implement it, because even if you talk about the backup there is thousands of way to implement [42:17.690 --> 42:31.830] the backup. We need to describe procedure will tell you how what is acceptable level way of implementing backups in this specific organization which is based on their network architecture based on the risk assessment on perceived risk [42:31.830 --> 42:35.390] or acceptable risk personal preferences. [42:36.030 --> 42:46.390] And if this is not this documents do not exist, we will always be unsure, but what is a good way of implementing your backup. We need first to formulate the procedures. [42:46.390 --> 43:03.290] And then the beauty part of the procedures is that you number one you have roles and responsibilities, people will be assigned like a responsibility for specific tasks, and they will put their signature, and as soon as people put their signature they feel responsible [43:03.290 --> 43:20.810] things will start happening. If nothing is described nothing is defined, nobody's responsible there is no cyber security program and for me the number one working first step when working with the organization, developing policies and procedures, assigning roles and responsibilities, [43:20.810 --> 43:37.090] is to train those who are responsible and then on top we can suddenly start building a program, because somebody is responsible. And I hate, I remember how much I hate the word Rossi this responsibility assignment metrics, and now I love it. [43:37.690 --> 43:49.410] Well, and then, basically, the last tip is basically it's my. So how do you, because when I'm asked by the customer. [43:49.410 --> 43:58.130] Can you please build me the cyber security program that number one is that they know how it is expensive, and the stakeholders and the CEO they want to know. [43:58.410 --> 44:10.910] Are we doing well, like, was this even budget and just spending the budget was justified and we know that with a security it's difficult to measure because you know like if nothing is happening. [44:11.150 --> 44:15.910] People believe that why do we need that security we are not detecting even anything. [44:15.910 --> 44:35.250] And this is why I measure the effectiveness differently. So, what I do I take the corporate risk metrics, and I tell them that you know what, in order to make sure that you are socially responsible that you're minimizing your potential business risk, you want to [44:35.250 --> 44:52.150] basically in terms of cyber security risk you want to be in quadrant, like in those green area. And if we bring like if we implement security controls and we conduct all of this risk assessment and specific documented way on which we all agree. [44:52.150 --> 45:04.710] And we believe that with implement security controls we minimize risks to acceptable level, then the program was justified and then when the government will knock your door or who not. [45:04.710 --> 45:09.650] They will, you will show hey I'm managing my risk, I am responsible. [45:09.650 --> 45:28.810] And if you talk about the same things with a person, use the word license to operate, because they will understand what it means. For example, when we talk about the safety, you know, if you did not minimize your safety risks to gain this acceptable level so you [45:28.810 --> 45:33.770] never would be allowed to leave red. So it has to be at least amber and green. [45:33.910 --> 45:36.470] You will not get your license to operate. [45:36.470 --> 45:47.290] This is the same with security, tell them the word of license to operate that now your cyber security risks are minimized to the level which is acceptable and they will then understand, oh, then we did a good job. [45:47.450 --> 46:04.970] So you see, the work which I'm doing right now and my current responsibility sounds a little bit so boring high level, but in reality it's so complex and I need really like kind of have to also think hard how to break this complex program problems [46:04.970 --> 46:08.710] into manageable and achievable steps. [46:09.910 --> 46:27.050] So, and now that basically how I wrote in the, like in my abstract but yeah Marina but still people like you exist and there are advanced attacks like do you think your current approach, like basically concentrate a lot of your security controls in the upper [46:27.050 --> 46:31.350] layers is effective against advanced tax and I wanted to give you an example. [46:31.350 --> 46:36.630] So, for many years I was dreaming to do something like killing them. [46:36.630 --> 46:40.670] What a filter in the photo utility and last year. [46:42.130 --> 46:59.850] Yeah, the beginning of last year I was lucky enough that I work with a trust research center in Singapore and their team has nice kindly assisted me in the experiment we are like trying to kill this. [47:00.830 --> 47:05.410] Ultra filtration filter in the water utility. It looks like that. [47:05.630 --> 47:16.350] And of course when we try to cause a physical damage to a specific piece of equipment it's all have to start, how did that things can be damaged like what is harmful to it. [47:16.350 --> 47:33.710] So you have basically have to read the manual and see what are the harmful conditions, and on one hand you can kill it. If you have like too many impurities in water like oil or grease it will basically damage the membrane but I don't know how to implement [47:33.710 --> 47:37.590] it, how will they introduce oil into the closed system remotely. [47:37.590 --> 47:57.030] But the second condition is that is what will damage the membrane is high pressure. So for example in this specific model it's a two bar. So it's basically and based in overall a prolonged operation on the very peak load which is basically close to two bar will kill [47:57.030 --> 48:05.950] that filter. So okay we need to build the pressure in the pipe. So for that we need now to steal the process documentation. [48:06.710 --> 48:23.650] And this is the two things that are two ways how that I can find it and it's again it's all of course individual. So, if the entire process was built and programmed and designed in the company, you actually need to infiltrate the company and steal this [48:23.650 --> 48:42.130] documentation, but very frequently and it's like really very frequently. So, the designs and actually programming of the PLC is done by the subcontractor so you need to actually to find the subcontractor of the company which is of interest to you and basically [48:42.130 --> 48:45.190] in the typically smaller companies not so well protected. [48:45.570 --> 49:02.130] And then you have to steal basically the documentation is in there. And this is why in your cybersecurity policy, it has to be there that you need to require your third party providers to achieve certain security posture in their organization, and we know that for example [49:02.130 --> 49:04.190] that is a requirement in NERC SIP. [49:04.190 --> 49:18.990] So let's assume like I as an attacker I identify the suppliers, I got the documentation I now know that there are two data flows go into this filtering. [49:20.950 --> 49:28.450] Basically, I can do it via the HMI screens or PIND diagram. [49:28.450 --> 49:45.470] And because the second flow is go from backwash I probably need to take a look what's happening in the backwash and how things are working in there. And then I will understand, basically, which PLCs and which control logic I need to enter and which [49:46.230 --> 49:54.490] equipment I need to compromise to actually implement this attack. But then I still, you know, like I don't understand which conditions will trigger. [49:55.210 --> 50:01.770] Basically, because I want to flows to flow simultaneously and I don't know how to do that I need to find the conditions. [50:02.050 --> 50:10.430] And there's actually guided in the state machine in the PLC, and this is something that I already won't find on any diagram, I actually need the PLC code. [50:10.450 --> 50:19.770] So for example, the PLC was programmed in the organization. I need actually already to penetrate the organization and this is where you should really try to prevent me from doing that. [50:19.770 --> 50:24.770] Because I need to steal the PLC code and read it in order to identify those conditions. [50:26.890 --> 50:45.210] And basically, then I need to manipulate the PLC logic or packets on the network, depending really how things are implemented but let's assume that I found the way to actually pull off this attack and this is specific cases, like in the context of this process [50:45.210 --> 50:55.430] I needed basically to implement attacks on two PLCs I need to compromise two different PLC in two different parts of the process so it was stage three and stage four. [50:55.430 --> 50:59.010] So they are basically a lot of things for me to compromise. [50:59.250 --> 51:11.690] And then so basically we'll assume that from the cyber perspective, I was able to implement it, and now the water from two flows, which never should fall simultaneously it flows gushing through the filter. [51:11.690 --> 51:22.970] And I now want to see how much, because I still don't know whether I will be able to reach the pressure, which I need so I'm basically need to measure, am I able to achieve two bars. [51:23.470 --> 51:31.090] And we run those experiments that experiment, and it turned out that the maximum differential pressure we were able to achieve in the filter was just one bar. [51:31.170 --> 51:36.050] So it's not enough for breakage. So as you can see, the attack is not almighty. [51:36.990 --> 51:54.710] But the point the most important point in here that there is no way I can ever figure out this, what the pressure can achieve like for example one bar without actually interacting with the actual physical process, and I might have this fantastic idea [51:54.710 --> 52:08.370] this is what I'm saying that the attack is not almighty. And that's a successful implementation of a damage scenario and it's cyber execution will not necessarily result in successful attack because certain things, I will only be able to measure on the [52:08.370 --> 52:09.830] life process. [52:10.370 --> 52:24.110] And this is what the point which I'm saying that many targeted damage attacks. This is the scenarios where I mostly work in I enjoy working, and it's useful to conduct such research but the point is that such targeted damage attacks require [52:24.110 --> 52:39.470] prolonged access to the process equipment, and this is what I'm saying, limit or eliminate such option for the attacker. If the attacker don't have that option they will not be able to execute it or maybe they may try to guess like well I feel like if I will do [52:39.470 --> 52:43.330] that implement that attack and they will try to create an autonomous malware. [52:43.790 --> 52:49.030] Well, they don't know they will be successful if it's only one bar, you're not damaging anything. [52:49.870 --> 53:07.890] So, on the other hand, like myself and my co researchers we recently came up with a really worrisome and way of compiling completely automated and very targeted payloads for industrial processes. [53:07.890 --> 53:20.450] We actually were accepted with this talk for a black card but we had to withdraw because me and my co researchers we received new job role so we felt it was inappropriate if you will be talking about basically offensive research. [53:20.610 --> 53:22.130] So we have to withdraw. [53:22.810 --> 53:34.730] But it's possible but nevertheless, again, the number of scenarios you will be able to implement with such autonomous payloads is much more limited. [53:34.730 --> 53:52.270] In comparison, so that's why I'm saying, make sure that every conduit, which leads to the control equipment and physical process goes through the military zone and level three and monitor for the attacker that look for him in their limit eliminate his [53:52.270 --> 53:55.990] ways for each having persistent access. [53:56.390 --> 54:01.510] And this is where. So for example, yes, if we talk about defense in depth. [54:01.510 --> 54:16.190] We still cannot of course guarantee that the attacker will not get those persistent access or continuous access to the process so focus on protecting and monitoring high criticality events data flows, when, for example, something in the, especially [54:16.190 --> 54:29.910] the time of the situation which you have alert emergency alarms, and the reaction time five to 15 minutes, maybe have a redundancy so if the attacker managed to spoof or suppress alarm or drop the packet I don't know was this alert. [54:29.910 --> 54:39.090] Make a redundancy for every critical alarm that it goes maybe to communication path is so that the operator will not miss it. [54:39.570 --> 54:45.130] So, I guess, so that was pretty much it from me. [54:45.170 --> 54:57.230] So as you can see, basically the way how I'm building the cyber security program right now is kind of good to actually make life of the advanced attackers like myself. [54:57.230 --> 55:03.470] So it might mean we know that the security no guarantees but it's really make life so much more difficult. [55:04.170 --> 55:18.350] And I don't have really smart afterwards, as I say this was not a very technical talk it was just my confessions, how about how I think differently right now when I'm working in a different role. [55:18.350 --> 55:37.530] And, as I said, building enterprise plant, or plant by cyber security program is not simple but it can be broken into the simplest steps which makes you easier decisions, make your decision making process easier and maybe more transparent and then more [55:37.530 --> 55:45.930] prioritized fashion basically applying this how do you eat an elephant principle and in this talk I shared you how I broke down that for myself. [55:47.590 --> 55:59.870] Focus on establishing the foundation first, even if it is extremely boring, and, you know, like I can't believe like a couple years ago I wouldn't believe that I would say something like that but I'm saying that. [55:59.870 --> 56:15.490] And yes in in process control because you know like if you remember I told it, we don't want even that Tucker be in our control system not even messing up but even be there, it's such a sensitive system, it's just like thinking about like a belly of the cat. [56:15.490 --> 56:28.810] It's something soft, we need to build a very hard shell around it, we just want don't want anybody going be even being there. And the success of our security program is about minimizing chances of anybody reaching those layers. [56:29.570 --> 56:39.690] And yes, of course I'm missing my cyber security vulnerability research. So if I have a couple of minutes and I guess I have, I would take a couple of questions. [56:43.100 --> 56:45.480] Hi, can you hear me. Yes. [56:45.560 --> 56:54.860] Okay, so we don't have any questions yet but I do want to share some feedback that happened because there were multiple times in your, your talk that you were saying this. [56:55.440 --> 57:02.100] For the record, this is not boring at all. It's really good to see the higher level view, long term strategy, etc. [57:02.420 --> 57:07.880] Somebody else, I concur. This is a more realistic topic for asset owners and integrators. [57:08.160 --> 57:18.680] In my opinion, there is a large gap between academia, government research, security researchers, and the reality of real operational technology environments. [57:21.280 --> 57:27.120] Okay, so I'm very glad that people enjoy the talk. I'm really glad that somebody found it useful. [57:28.980 --> 57:37.300] I also informed everybody on the DEF CON discord channel that you would be trying to join there. If you were able to. [57:37.400 --> 57:39.320] So we are not live anymore. [57:39.380 --> 57:40.680] We are still live. [57:40.680 --> 57:41.420] Okay. [57:42.620 --> 57:49.660] All right, so I, I thank everybody for the attention and for being here. [57:49.820 --> 57:59.680] There was like also good feedback on the Twitter so thank you very much and I will try my best to join this, the channel on the DEF CON right now. [57:59.720 --> 58:02.080] Okay, well thank you very much for joining us Marina. [58:02.600 --> 58:03.940] Thank you for having me.