WEBVTT 00:01.110 --> 00:01.870 Okay. 00:01.870 --> 00:03.310 Hello, everyone. 00:03.310 --> 00:04.750 My name is Dori Ardeni. 00:04.750 --> 00:08.470 I'm the Head of Incident Response and Hunting at Artorio. 00:08.470 --> 00:15.030 I've previously served as a SERT Team Leader in the IDF, the Israel Defense Forces. 00:15.190 --> 00:24.430 I've... I managed threat hunting operations and also dealt with targeted cyber threats on big organizations. 00:24.770 --> 00:31.550 I also bring hacking knowledge from my experience as a Red Team Researcher. 00:31.550 --> 00:33.750 So I hope you will enjoy this session. 00:33.750 --> 00:39.190 The mutual effort together with OSIsoft and Mike was really cool. 00:40.230 --> 00:41.350 Thanks, Dori. 00:41.350 --> 00:42.950 And this is Mike Lemley. 00:42.950 --> 00:44.990 I work at OSIsoft. 00:44.990 --> 00:53.030 We were in response to the wonderful research that DOR presented to us on the vulnerability that is covered in our product. 00:53.090 --> 00:57.170 I've been at OSIsoft since 1999. 00:57.210 --> 01:00.590 I focused primarily in software development. 01:00.770 --> 01:13.940 And then about seven years, I started focusing on cybersecurity and now work as part of our central architecture team, and hoping to improve the cybersecurity practices of our development at OSIsoft. 01:15.690 --> 01:17.230 Thanks, Mike. 01:17.690 --> 01:20.490 So let's start with a quick story. 01:20.490 --> 01:23.190 And at the end of it, you'll get my point. 01:23.830 --> 01:30.930 So two weeks ago, a shift leader at a chemical plant received an alert from the data management platform. 01:30.990 --> 01:35.170 Platform like the one of OSIsoft, PyServer. 01:36.590 --> 01:42.110 The alert reported about a boiler that exceeded from its threshold temperature. 01:42.270 --> 01:44.530 The shift leader was really nervous. 01:45.230 --> 01:59.650 And in addition to that, databases of all historical plan data and their backups were deleted, which means they had no access to industry 4.0 analytics, like people like to say. 01:59.870 --> 02:03.730 The plant was shut down due to this abnormal activity. 02:03.730 --> 02:07.710 The concern was physical damage and human loss. 02:09.570 --> 02:16.290 The shutdown caused a loss of significant production time, which means a lot of money. 02:17.790 --> 02:19.990 So why am I telling you this story? 02:19.990 --> 02:22.850 I'm telling you this because it's not a nightmare. 02:22.850 --> 02:25.230 It's not a fairytale story. 02:25.530 --> 02:33.510 When you base your decisions on the data management platform, especially what has been manipulated, you are technically blind. 02:33.510 --> 02:36.970 You are blind from your operational environment. 02:38.610 --> 02:44.250 So we have two core goals that we want to achieve from this session. 02:44.450 --> 02:56.190 The first one is to raise awareness about the implications of industrial-oriented vulnerability, where each vulnerability is a serious threat to our sensitive environment. 02:57.410 --> 03:08.590 And it's not only ICS vulnerability, it could be also IT vulnerabilities that somehow affect directly on our production flow. 03:08.810 --> 03:16.090 The second goal is to showcase how the discovery of one vulnerability can help other vendors. 03:16.310 --> 03:23.670 Mike will talk about the way things proceeded after we identified the vulnerability in their product. 03:25.860 --> 03:30.880 I want to talk a little bit about what the Pi system is from OSIsoft. 03:30.880 --> 03:34.840 We start off with our core customers. 03:34.840 --> 03:42.320 So these six areas of business are where most of our customers are in the industry. 03:42.540 --> 03:52.980 And the company OSIsoft has been around for about 40 years, and we have customers in 140 different countries. 03:52.980 --> 03:59.660 And this world map gives you an indication of where we have support offices to help our customers. 04:01.240 --> 04:07.500 This reference architecture explains how a lot of our customers are in the critical system. 04:08.220 --> 04:16.340 Now, what we want is we want to reduce the amount of people that have access within the critical system. 04:16.400 --> 04:32.220 And the OSIsoft Pi system infrastructure allows them to do that by pulling the data that is needed for analysis and business decisions out of the critical system so that those people don't need to have access there in order to do their job. 04:35.670 --> 04:43.260 This additional data reference architecture also represents how the Pi system is used. 04:43.260 --> 04:56.440 Now, the Pi server inside of the control center is used to extract the data from the control center using hundreds of different protocols based on the different types of equipment there. 04:56.440 --> 05:06.120 Data is then sent over to the corporate access or corporate network zone, where it is then used by those employees in order to do their analysis. 05:06.220 --> 05:11.360 Now, the target for this particular incident is within the vision server. 05:11.720 --> 05:23.360 And that is where the API server exists that is being used and attacked by this particular incident, which is a cross-site security vulnerability. 05:25.560 --> 05:35.300 Okay, so before I dive into the details of the vulnerability, suppose you ask yourself why we even started looking at Pi server. 05:35.740 --> 05:41.540 And the answer is that we had to put some effort into product tasks. 05:41.540 --> 05:51.540 We had to integrate OSI soft platform, the Pi server, into our security orchestration and response platform. 05:51.540 --> 06:00.380 But as a curious researcher, I decided to test the asset framework of the Pi server on XSS. 06:01.000 --> 06:04.660 It was very late at night, so realistic as it sounds. 06:04.660 --> 06:14.200 I fought against the web API parsing mechanism in order to inject JavaScript code into client browsers. 06:14.480 --> 06:16.500 And in the end, I succeeded. 06:16.500 --> 06:18.200 And we are here. 06:20.120 --> 06:24.600 So here is a screenshot from the asset framework. 06:24.600 --> 06:32.820 On the left, we have a simple plant, a petroleum plant with a boiler and a tank. 06:32.820 --> 06:37.720 Each element has attributes. 06:37.720 --> 06:46.100 For example, the boiler has an attribute of temperature and the tank has an attribute of volume. 06:46.400 --> 06:50.660 And element one is the child element only for my tests. 06:50.660 --> 06:56.840 On the right, you can see the description field, the vulnerable field. 06:56.840 --> 07:01.500 We've encoded base64 JavaScript. 07:01.960 --> 07:17.740 In order to bypass the web API parsing mechanism, you have to insert some magic characters at the beginning of it. 07:18.140 --> 07:32.480 Once an attacker managed to do so, he or she can run JavaScript code on client browser for phishing, keylogging, etc. 07:35.560 --> 07:39.540 Okay, so here is a possible scenario for this attack. 07:39.540 --> 07:48.900 First, an attacker is stealing low privileges credentials inside the network. 07:48.960 --> 07:58.580 After that, the attacker is injecting the malicious JavaScript to the vulnerable field that we saw in the previous slide. 07:58.580 --> 08:07.320 Then a victim, for example, an innocent engineer, is accessing the web infected page. 08:08.020 --> 08:25.240 And then a fake login form is prompting for username and password, and also in parallel, changing the notification rule template on behalf of the engineer high privileges credentials. 08:25.740 --> 08:33.160 At the end of it, the attacker receiving the credentials and also causing TemporTool false positive alert. 08:35.900 --> 08:39.420 Okay, so let's see it in action. 08:40.580 --> 08:46.180 On the right is the server of the attacker. 08:46.180 --> 08:50.060 This is on port 8080 for the stolen credentials. 08:50.060 --> 08:58.360 On the left, the victim browser. 08:59.180 --> 09:12.090 Once the victim passes the cursor over the infected field, the descriptive field, the fake login form is prompting for username and password. 09:17.010 --> 09:23.730 The victim is inserting the pyadmin password. 09:26.810 --> 09:42.360 And once the victim submits credentials, the attacker gets the stolen credentials on the right. 09:42.360 --> 09:46.580 Pyadmin and the strong password. 09:47.520 --> 09:50.600 So from here, it's pretty much game over. 09:50.940 --> 10:11.580 And in the same method, in the same technique, I could also add a JavaScript code to change the notification rule template in order to change the rule and to cause false positive alert of high temperature. 10:14.790 --> 10:19.750 Okay, so this screenshot was taken from USAID site. 10:19.750 --> 10:24.410 The vulnerability has got rank of 7.7. 10:24.710 --> 10:31.910 OSIsoft also added us in their all of thanks list. 10:32.090 --> 10:35.150 Here in the bottom, Dory O'Donoghue and Eliad Mualem. 10:36.030 --> 10:40.910 And I suppose you asked yourself how to avoid this kind of attacks. 10:40.910 --> 10:43.370 So the answer is patching. 10:43.370 --> 10:47.890 OSIsoft released a patch for the vulnerability very, very quickly. 10:47.890 --> 11:01.430 But we have to do a lot more, a lot more proactive actions, such as threat hunting operations on our sensitive environment to other firewall rules. 11:01.590 --> 11:15.290 And also to get visibility into our assets and their version in order to perform vulnerability management, etc. 11:15.290 --> 11:18.230 But that's for another talk. 11:20.690 --> 11:27.530 Now I'll talk about the lessons that OSIsoft learned from this incident and how we responded. 11:27.730 --> 11:35.530 Now to start off with, I'm going to talk a little bit about our security development lifecycle process at OSIsoft. 11:35.630 --> 11:39.930 And that'll define how we changed, we responded. 11:39.930 --> 11:45.410 So we at OSIsoft established several different security best practices. 11:45.410 --> 11:56.370 And then what we do is we measure the adoption by our 40, around 40 development teams to apply those security best practices in their development. 11:56.750 --> 11:58.330 Now why do we measure? 11:59.470 --> 12:07.190 I constantly say, I mean, what good are best practices if you don't measure to see how well you are adopting them? 12:07.190 --> 12:11.750 But in addition to that, it really helps teams plan. 12:12.190 --> 12:19.290 When you go through the process of identifying measurements for the best practices, it forces you to identify milestones. 12:19.290 --> 12:26.790 And those milestones are very useful for the teams to plan against and plan their development activities. 12:26.790 --> 12:31.710 Another thing that it does is it forces you to organize and prioritize all these best practices. 12:31.710 --> 12:37.550 And that also helps the teams identify how to best move forward. 12:38.710 --> 12:49.150 Now looking at our SDL process, focused entirely on cross-site scripting allowed us to see where we could improve our process. 12:49.150 --> 12:53.790 So we have education, of defensive programming. 12:53.790 --> 13:05.890 We also have various tools that we use and the vendors which that we use for these various tools are listed on the right side, along with some of the security researchers we engage with. 13:06.170 --> 13:09.430 So as I said, we have various security tools we use. 13:09.430 --> 13:18.930 There's static analysis, software component analysis, which actually helps us find cross-site scripting vulnerabilities in components that we consume, open source products. 13:19.090 --> 13:34.290 We have various security headers that we encourage use of, as well as the external reviews and incident response, which was very key in how we handle this issue that came to us from Otario. 13:34.430 --> 13:42.810 Now in looking at those processes, we defined there were some areas we thought we could improve or add on to. 13:42.850 --> 13:47.050 So the first one is an internal security tech talk. 13:47.050 --> 13:49.930 We do tech talks weekly at OSIsoft. 13:49.930 --> 14:03.310 Every third week they are focused on security and we did a tech talk specifically on this incident so we could share the lessons learned from this with the other of the 40 teams at OSIsoft. 14:03.310 --> 14:13.430 We also looked at how we use the static application security testing tool and identified some process improvements for our use of that tool. 14:13.470 --> 14:21.010 And the last is we decided to get much more aggressive on the adoption of content security policy at OSIsoft. 14:23.310 --> 14:40.910 Now I did say that we do measure our best practices and I just wanted to give you a sense of what some of the measurements are that we use associated with just our cross-site scripting defenses at OSIsoft in the best practice areas I defined. 14:42.950 --> 14:50.830 Now we didn't do any process improvements on our DAST tool and I'm going to talk briefly about that. 14:50.830 --> 15:00.130 Now the DAST tool that we use is from Qualys and for those of you who don't remember what a DAST tool is, it's basically it's run as a server. 15:00.130 --> 15:20.170 Your application is up and running and then the DAST tool will send dynamic tests at your web server, typically OWASP top 10 attacks, and see whether or not your web server has the appropriate implementations in place to protect against it. 15:20.170 --> 15:29.090 Now in this particular cross-site scripting issue, as Dor explained, it was a stored cross-site scripting attack. 15:29.090 --> 15:40.150 But in looking at the tools, it's more of a DOM cross-site scripting vulnerability with insecure use of innerHTML. 15:40.330 --> 15:56.950 Now our DAST tool, Qualys, is very good when the source of the innerHTML parsing, the bad package, is from a document URL, but it doesn't really handle well when the source is from an HTML document. 15:56.950 --> 15:59.510 So we decided not to focus there. 16:02.360 --> 16:13.760 Now our static application security testing tool is from Synopsys and if you look at the bottom right as a reminder for what SAS tools do, they analyze your code. 16:13.760 --> 16:24.820 So they're looking directly at your code, look at call stacks, how functions are reached, and identify insecure call paths or insecure calling sequences. 16:25.120 --> 16:27.640 So we felt that was the best tool. 16:27.640 --> 16:34.520 It should identify where innerHTML is used and it should be able to flag insecure uses of it. 16:34.520 --> 16:36.760 So what went wrong with this? 16:36.760 --> 16:43.280 We identified that there was a tool compatibility problem with the NuGet package that we used. 16:43.280 --> 16:50.080 It's a Razor engine and our tool Synopsys did not work well with it. 16:50.080 --> 16:56.660 It does work very well with the Razor analysis built into ASP.NET. 16:56.820 --> 17:04.980 So consequent to this problem, our scanning for this particular project missed all JavaScript and CSHTML. 17:05.120 --> 17:18.460 So the process improvements we identified is we need to be able to do some central evaluation of the SAS configurations and we're working with Synopsys on how to best move that idea forward. 17:18.460 --> 17:28.820 The other thing we're going to do is for this one particular development team, we're going to move them to using the ASP.NET MVC Razor engine instead. 17:28.900 --> 17:35.640 And then we're also considering increasing the sensitivity to completely disallow the use of innerHTML. 17:35.640 --> 17:42.280 We're not sure exactly how this is going to play out, but innerHTML is definitely not something we want to use. 17:42.280 --> 17:48.120 So we're either going to be heavily scrutinizing the use of innerHTML or completely disallow it. 17:48.120 --> 17:50.020 So we're still working that one out. 17:50.140 --> 17:51.280 Next slide please. 17:51.720 --> 17:55.840 Now I'm going to switch context and talk a little bit about content security policy. 17:55.840 --> 18:00.620 And just to give you a why behind that, I'm focusing on jQuery. 18:00.620 --> 18:07.460 Now over the last couple of years, jQuery has released several different vulnerabilities. 18:08.340 --> 18:12.440 These all on this chart are the cross-site scripting vulnerability. 18:12.440 --> 18:23.860 And as you can see with the overlay that was just added, they all fall in the high range for CVSS scoring, which means you have to take these particular vulnerabilities very seriously. 18:23.860 --> 18:31.160 Cross-site scripting in general does tend to fall into the high vulnerability area. 18:31.160 --> 18:45.960 And dealing with all of these different patch updates from this open source does present challenges to vendors which consume open source and components under the scrutiny of security research. 18:46.280 --> 18:47.140 Next please. 18:48.240 --> 18:56.800 Now content security policy, just to explain how it works, the web server will send down HTML and JavaScript content to your browser. 18:56.800 --> 19:09.660 And now your browser is theoretically susceptible to some sort of JavaScript injection via a cross-site scripting payload which has been architected by the attacker. 19:09.660 --> 19:21.720 Now by adding content security policy from the same web server, you can now inform the browser of the allowed list, where JavaScript is allowed to come from. 19:21.720 --> 19:29.100 So now the browser has the ability to identify that, hey, this cross-site scripting payload is not on my allowed list. 19:29.100 --> 19:31.400 I'm not going to allow it to execute. 19:31.900 --> 19:42.140 And I also added the CDN, Content Delivery Network, so just to let you know that the content security policy's allow list does incorporate more than just the web server that it came from. 19:42.140 --> 19:44.540 So there's there's lots of flexibility there. 19:44.620 --> 19:45.680 Next slide please. 19:46.450 --> 19:56.220 Now we strongly believe that content security policy has reached the point where it is extremely extremely mature. 19:56.360 --> 20:01.100 The major browsers all have implementations for it. 20:01.100 --> 20:11.700 So we think that the industry is in a good place where we can start demanding the use of content security policy in the products that we consume. 20:12.180 --> 20:15.900 Now a question I often get on feedback, i.e. 20:16.220 --> 20:17.520 Internet Explorer. 20:20.780 --> 20:23.700 The Internet Explorer, I just want to talk about that one briefly. 20:23.700 --> 20:29.040 Internet Explorer, I get this question a lot, you know, it doesn't support a content security policy. 20:29.040 --> 20:30.260 Is that a problem? 20:30.280 --> 20:34.400 And this will not stop Internet Explorer from working. 20:34.820 --> 20:39.060 So you don't have to tell customers they can't use Internet Explorer anymore. 20:39.060 --> 20:50.140 But what we are taking the position of is that we want to inform our customers that you should be using modern browsers for their security benefit. 20:50.140 --> 20:56.850 One issue and one protection is content security policy. 20:57.010 --> 21:02.530 Now I talked about having measurements for our best practices. 21:02.530 --> 21:05.590 And this is one of the ways that we do it. 21:05.590 --> 21:12.990 We've identified poor, better, and best practices for our development teams and came up with ways for them to measure their adoption against that. 21:12.990 --> 21:17.730 So the poor practice would be using unsafe inline and unsafe eval. 21:17.730 --> 21:26.130 The next better practice would be if you do have inline JavaScript to provide nonces or hashes for it. 21:26.130 --> 21:50.530 But the best practice is to remove all those inline JavaScript and eval statements and give the browser an endpoint to report any occurrences of defenses or blocks that it had to execute to a central reporting location so that those issues can be investigated and fixed. 21:51.070 --> 22:02.430 And just to give you a general sense of how we do this measurement within OSIsoft, all the measurements we've identified from the best practices have response options. 22:02.430 --> 22:03.730 They're all given scores. 22:03.730 --> 22:07.570 So you get a score, a scale from zero to ten for all of them. 22:07.570 --> 22:14.650 And they can all be combined and you can look at collectively how teams are doing it in response to adoption of these best practices. 22:14.770 --> 22:29.750 But in addition to this, we decided to not only revamp the responses in order to encourage better adoption, but we decided also to take the step of making this required for releases going out by the end of this year. 22:30.090 --> 22:30.610 Excellent. 22:32.470 --> 22:37.070 Now, focusing on the key takeaways, the things that that I covered. 22:37.070 --> 22:38.950 The first one was security tools. 22:38.950 --> 22:40.570 Using security tools is not enough. 22:40.570 --> 22:46.230 You really have to evaluate how you're using them, in our case, for better code coverage. 22:46.930 --> 22:49.110 The next one is content security policy. 22:49.110 --> 22:59.550 We believe the industry is mature enough that content security policy is in a place that this should become an ICS standard practice, we believe. 22:59.550 --> 23:03.030 It gives you a strong second level of defense. 23:03.030 --> 23:17.650 With a lot of open source components out there, most of us consume open source, and when the vulnerabilities are discovered in those open source, the public disclosure happens to the public at the same time it happens to us. 23:17.650 --> 23:21.010 We know about the vulnerability at the same time everyone else does. 23:21.010 --> 23:35.290 So there is limited time that we have to patch and get the protections in place, but by having that second level of defense, content security policy there, it gives us more time to get the patches in place correctly. 23:35.670 --> 23:49.510 And the last thing, engaging with companies like Atario and Door is very, very successful in moving security forward for our products and it's been great experience working with them for us. 23:49.510 --> 23:50.530 Thank you. 23:50.530 --> 23:51.190 Thank you. 23:51.190 --> 23:52.830 Thank you very much, Mike. 23:52.850 --> 23:59.390 The actual effort was really cool and we very enjoyed this research.