[00:01.110 --> 00:07.650]  Okay. Hello, everyone. My name is Dori Ardeni. I'm the Head of Incident Response and Hunting at
[00:07.650 --> 00:15.030]  Artorio. I've previously served as a SERT Team Leader in the IDF, the Israel Defense Forces.
[00:15.190 --> 00:22.170]  I've... I managed threat hunting operations and also dealt with targeted cyber threats
[00:22.170 --> 00:29.670]  on big organizations. I also bring hacking knowledge from my experience as a
[00:29.670 --> 00:35.270]  Red Team Researcher. So I hope you will enjoy this session. The mutual effort together
[00:35.270 --> 00:44.990]  with OSIsoft and Mike was really cool. Thanks, Dori. And this is Mike Lemley. I work at OSIsoft.
[00:44.990 --> 00:51.090]  We were in response to the wonderful research that DOR presented to us on the vulnerability
[00:51.090 --> 00:58.930]  that is covered in our product. I've been at OSIsoft since 1999. I focused primarily
[00:58.930 --> 01:04.350]  in software development. And then about seven years, I started focusing on cybersecurity
[01:05.040 --> 01:08.370]  and now work as part of our central architecture team,
[01:08.370 --> 01:13.940]  and hoping to improve the cybersecurity practices of our development at OSIsoft.
[01:15.690 --> 01:23.190]  Thanks, Mike. So let's start with a quick story. And at the end of it, you'll get my point.
[01:23.830 --> 01:30.250]  So two weeks ago, a shift leader at a chemical plant received an alert from the data management
[01:30.250 --> 01:39.030]  platform. Platform like the one of OSIsoft, PyServer. The alert reported about a boiler
[01:39.030 --> 01:44.530]  that exceeded from its threshold temperature. The shift leader was really nervous.
[01:45.230 --> 01:51.390]  And in addition to that, databases of all historical plan data and their backups were
[01:51.390 --> 01:59.650]  deleted, which means they had no access to industry 4.0 analytics, like people like to say.
[01:59.870 --> 02:03.730]  The plant was shut down due to this abnormal activity.
[02:03.730 --> 02:07.710]  The concern was physical damage and human loss.
[02:09.570 --> 02:16.290]  The shutdown caused a loss of significant production time, which means a lot of money.
[02:17.790 --> 02:23.290]  So why am I telling you this story? I'm telling you this because it's not a nightmare. It's not
[02:23.470 --> 02:30.130]  a fairytale story. When you base your decisions on the data management platform, especially
[02:30.130 --> 02:36.970]  what has been manipulated, you are technically blind. You are blind from your operational environment.
[02:38.610 --> 02:46.270]  So we have two core goals that we want to achieve from this session. The first one is to raise awareness
[02:46.270 --> 02:49.970]  about the implications of industrial-oriented vulnerability,
[02:50.590 --> 02:56.190]  where each vulnerability is a serious threat to our sensitive environment.
[02:57.410 --> 03:05.490]  And it's not only ICS vulnerability, it could be also IT vulnerabilities that somehow affect
[03:05.490 --> 03:13.030]  directly on our production flow. The second goal is to showcase how the discovery of one
[03:13.030 --> 03:18.830]  vulnerability can help other vendors. Mike will talk about the way things proceeded
[03:19.350 --> 03:23.670]  after we identified the vulnerability in their product.
[03:25.860 --> 03:33.080]  I want to talk a little bit about what the Pi system is from OSIsoft. We start off with
[03:33.080 --> 03:39.880]  our core customers. So these six areas of business are where most of our customers
[03:41.220 --> 03:49.860]  are in the industry. And the company OSIsoft has been around for about 40 years, and we have
[03:49.860 --> 03:56.560]  customers in 140 different countries. And this world map gives you an indication of where we
[03:56.560 --> 04:06.040]  have support offices to help our customers. This reference architecture explains how a lot of our
[04:06.040 --> 04:12.980]  customers are in the critical system. Now, what we want is we want to reduce the amount of people
[04:12.980 --> 04:19.300]  that have access within the critical system. And the OSIsoft Pi system infrastructure allows them
[04:19.300 --> 04:25.400]  to do that by pulling the data that is needed for analysis and business decisions out of the
[04:25.400 --> 04:32.220]  critical system so that those people don't need to have access there in order to do their job.
[04:35.670 --> 04:43.260]  This additional data reference architecture also represents how the Pi system is used.
[04:43.260 --> 04:51.240]  Now, the Pi server inside of the control center is used to extract the data from the control center
[04:51.240 --> 04:56.440]  using hundreds of different protocols based on the different types of equipment there.
[04:56.440 --> 05:03.040]  Data is then sent over to the corporate access or corporate network zone, where it is then used by
[05:03.040 --> 05:08.380]  those employees in order to do their analysis. Now, the target for this particular incident
[05:08.380 --> 05:18.180]  is within the vision server. And that is where the API server exists that is being used
[05:18.180 --> 05:23.360]  and attacked by this particular incident, which is a cross-site security vulnerability.
[05:25.560 --> 05:30.340]  Okay, so before I dive into the details of the vulnerability,
[05:31.060 --> 05:35.300]  suppose you ask yourself why we even started looking at Pi server.
[05:35.740 --> 05:43.380]  And the answer is that we had to put some effort into product tasks. We had to integrate
[05:44.020 --> 05:51.540]  OSI soft platform, the Pi server, into our security orchestration and response platform.
[05:51.540 --> 05:56.900]  But as a curious researcher, I decided to test the asset framework of the Pi server
[05:58.320 --> 06:07.640]  on XSS. It was very late at night, so realistic as it sounds. I fought against the web API parsing
[06:07.640 --> 06:16.500]  mechanism in order to inject JavaScript code into client browsers. And in the end, I succeeded.
[06:16.500 --> 06:18.200]  And we are here.
[06:20.120 --> 06:24.600]  So here is a screenshot from the asset framework.
[06:24.600 --> 06:33.380]  On the left, we have a simple plant, a petroleum plant with a boiler and a tank. Each
[06:33.380 --> 06:42.080]  element has attributes. For example, the boiler has an attribute of temperature and the tank
[06:42.080 --> 06:50.660]  has an attribute of volume. And element one is the child element only for my tests.
[06:50.660 --> 06:59.820]  On the right, you can see the description field, the vulnerable field. We've encoded base64
[06:59.820 --> 07:13.020]  JavaScript. In order to bypass the web API parsing mechanism, you have to insert some magic characters
[07:13.020 --> 07:26.680]  at the beginning of it. Once an attacker managed to do so, he or she can run JavaScript code on
[07:26.680 --> 07:39.540]  client browser for phishing, keylogging, etc. Okay, so here is a possible scenario for this attack.
[07:39.540 --> 07:48.900]  First, an attacker is stealing low privileges credentials inside the network.
[07:48.960 --> 07:56.200]  After that, the attacker is injecting the malicious JavaScript to the vulnerable field
[07:56.200 --> 08:03.180]  that we saw in the previous slide. Then a victim, for example, an innocent engineer,
[08:03.180 --> 08:11.920]  is accessing the web infected page. And then a fake login form is prompting for
[08:12.760 --> 08:20.240]  username and password, and also in parallel, changing the notification rule template on behalf
[08:20.940 --> 08:27.580]  of the engineer high privileges credentials. At the end of it, the attacker receiving the
[08:27.580 --> 08:39.420]  credentials and also causing TemporTool false positive alert. Okay, so let's see it in action.
[08:40.580 --> 08:50.060]  On the right is the server of the attacker. This is on port 8080 for the stolen credentials.
[08:50.060 --> 09:04.100]  On the left, the victim browser. Once the victim passes the cursor over the infected field,
[09:04.100 --> 09:12.090]  the descriptive field, the fake login form is prompting for username and password.
[09:17.010 --> 09:32.770]  The victim is inserting the pyadmin password. And once the victim submits
[09:34.680 --> 09:44.880]  credentials, the attacker gets the stolen credentials on the right. Pyadmin and the
[09:44.880 --> 09:54.840]  strong password. So from here, it's pretty much game over. And in the same method,
[09:54.840 --> 10:00.320]  in the same technique, I could also add a JavaScript code to change the notification rule
[10:00.320 --> 10:11.580]  template in order to change the rule and to cause false positive alert of high temperature.
[10:14.790 --> 10:24.410]  Okay, so this screenshot was taken from USAID site. The vulnerability has got rank of 7.7.
[10:24.710 --> 10:34.010]  OSIsoft also added us in their all of thanks list. Here in the bottom, Dory O'Donoghue and Eliad
[10:34.010 --> 10:41.910]  Mualem. And I suppose you asked yourself how to avoid this kind of attacks. So the answer is
[10:41.910 --> 10:47.890]  patching. OSIsoft released a patch for the vulnerability very, very quickly.
[10:47.890 --> 10:55.690]  But we have to do a lot more, a lot more proactive actions, such as threat hunting operations
[10:55.690 --> 11:05.070]  on our sensitive environment to other firewall rules. And also to get visibility into our
[11:06.030 --> 11:16.910]  assets and their version in order to perform vulnerability management, etc. But that's for
[11:16.910 --> 11:25.610]  another talk. Now I'll talk about the lessons that OSIsoft learned from this incident and
[11:25.610 --> 11:32.010]  how we responded. Now to start off with, I'm going to talk a little bit about our security
[11:32.010 --> 11:39.930]  development lifecycle process at OSIsoft. And that'll define how we changed, we responded.
[11:39.930 --> 11:46.790]  So we at OSIsoft established several different security best practices. And then what we do is
[11:46.790 --> 11:53.950]  we measure the adoption by our 40, around 40 development teams to apply those security best
[11:53.950 --> 12:02.430]  practices in their development. Now why do we measure? I constantly say, I mean, what good are
[12:02.430 --> 12:08.530]  best practices if you don't measure to see how well you are adopting them? But in addition to
[12:08.530 --> 12:14.950]  that, it really helps teams plan. When you go through the process of identifying measurements
[12:14.950 --> 12:20.870]  for the best practices, it forces you to identify milestones. And those milestones are very useful
[12:20.870 --> 12:27.570]  for the teams to plan against and plan their development activities. Another thing that it
[12:27.570 --> 12:32.750]  does is it forces you to organize and prioritize all these best practices. And that also helps
[12:33.280 --> 12:43.790]  the teams identify how to best move forward. Now looking at our SDL process, focused entirely on
[12:43.790 --> 12:50.790]  cross-site scripting allowed us to see where we could improve our process. So we have education,
[12:51.910 --> 12:59.290]  of defensive programming. We also have various tools that we use and the vendors which that we
[12:59.290 --> 13:03.830]  use for these various tools are listed on the right side, along with some of the security
[13:03.830 --> 13:10.050]  researchers we engage with. So as I said, we have various security tools we use. There's static
[13:10.050 --> 13:14.750]  analysis, software component analysis, which actually helps us find cross-site scripting
[13:14.750 --> 13:21.170]  vulnerabilities in components that we consume, open source products. We have various security
[13:21.170 --> 13:28.150]  headers that we encourage use of, as well as the external reviews and incident response, which was
[13:28.150 --> 13:36.750]  very key in how we handle this issue that came to us from Otario. Now in looking at those processes,
[13:36.750 --> 13:44.210]  we defined there were some areas we thought we could improve or add on to. So the first one is
[13:44.210 --> 13:51.310]  an internal security tech talk. We do tech talks weekly at OSIsoft. Every third week they are
[13:51.310 --> 13:57.810]  focused on security and we did a tech talk specifically on this incident so we could share
[13:57.810 --> 14:05.190]  the lessons learned from this with the other of the 40 teams at OSIsoft. We also looked at how
[14:05.190 --> 14:11.230]  we use the static application security testing tool and identified some process improvements
[14:11.230 --> 14:16.890]  for our use of that tool. And the last is we decided to get much more aggressive on the
[14:16.890 --> 14:28.250]  adoption of content security policy at OSIsoft. Now I did say that we do measure our best practices
[14:28.250 --> 14:34.790]  and I just wanted to give you a sense of what some of the measurements are that we use associated
[14:34.790 --> 14:40.910]  with just our cross-site scripting defenses at OSIsoft in the best practice areas I defined.
[14:42.950 --> 14:49.630]  Now we didn't do any process improvements on our DAST tool and I'm going to talk briefly
[14:49.630 --> 14:56.390]  about that. Now the DAST tool that we use is from Qualys and for those of you who don't remember
[14:56.390 --> 15:03.390]  what a DAST tool is, it's basically it's run as a server. Your application is up and running
[15:03.390 --> 15:13.290]  and then the DAST tool will send dynamic tests at your web server, typically OWASP top 10 attacks,
[15:13.290 --> 15:18.890]  and see whether or not your web server has the appropriate implementations in place to protect
[15:18.890 --> 15:26.350]  against it. Now in this particular cross-site scripting issue, as Dor explained, it was a
[15:26.350 --> 15:33.210]  stored cross-site scripting attack. But in looking at the tools, it's more of a DOM
[15:33.210 --> 15:42.950]  cross-site scripting vulnerability with insecure use of innerHTML. Now our DAST tool, Qualys,
[15:42.950 --> 15:51.150]  is very good when the source of the innerHTML parsing, the bad package, is from a document URL,
[15:51.150 --> 15:57.590]  but it doesn't really handle well when the source is from an HTML document. So we decided
[15:57.590 --> 16:07.700]  not to focus there. Now our static application security testing tool is from Synopsys and if
[16:07.700 --> 16:13.920]  you look at the bottom right as a reminder for what SAS tools do, they analyze your code. So
[16:13.920 --> 16:18.820]  they're looking directly at your code, look at call stacks, how functions are reached, and identify
[16:18.820 --> 16:27.940]  insecure call paths or insecure calling sequences. So we felt that was the best tool. It should
[16:27.940 --> 16:35.120]  identify where innerHTML is used and it should be able to flag insecure uses of it. So what went
[16:35.120 --> 16:41.980]  wrong with this? We identified that there was a tool compatibility problem with the NuGet package
[16:41.980 --> 16:50.680]  that we used. It's a Razor engine and our tool Synopsys did not work well with it. It does
[16:50.680 --> 16:58.880]  work very well with the Razor analysis built into ASP.NET. So consequent to this problem,
[16:58.880 --> 17:06.040]  our scanning for this particular project missed all JavaScript and CSHTML. So the process
[17:06.040 --> 17:11.880]  improvements we identified is we need to be able to do some central evaluation of the SAS
[17:11.880 --> 17:19.140]  configurations and we're working with Synopsys on how to best move that idea forward. The other
[17:19.140 --> 17:24.200]  thing we're going to do is for this one particular development team, we're going to move them to
[17:24.200 --> 17:31.780]  using the ASP.NET MVC Razor engine instead. And then we're also considering increasing the
[17:31.780 --> 17:37.260]  sensitivity to completely disallow the use of innerHTML. We're not sure exactly how this is
[17:37.260 --> 17:42.760]  going to play out, but innerHTML is definitely not something we want to use. So we're either
[17:42.760 --> 17:48.300]  going to be heavily scrutinizing the use of innerHTML or completely disallow it. So we're
[17:48.300 --> 17:53.940]  still working that one out. Next slide please. Now I'm going to switch context and talk a little
[17:53.940 --> 18:00.620]  bit about content security policy. And just to give you a why behind that, I'm focusing on jQuery.
[18:00.620 --> 18:07.460]  Now over the last couple of years, jQuery has released several different vulnerabilities.
[18:08.340 --> 18:13.260]  These all on this chart are the cross-site scripting vulnerability. And as you can see
[18:13.260 --> 18:19.660]  with the overlay that was just added, they all fall in the high range for CVSS scoring, which
[18:19.660 --> 18:24.240]  means you have to take these particular vulnerabilities very seriously. Cross-site
[18:24.240 --> 18:31.880]  scripting in general does tend to fall into the high vulnerability area. And dealing with
[18:31.880 --> 18:38.380]  all of these different patch updates from this open source does present challenges to vendors
[18:38.380 --> 18:47.140]  which consume open source and components under the scrutiny of security research. Next please.
[18:48.240 --> 18:53.700]  Now content security policy, just to explain how it works, the web server will send down
[18:53.700 --> 19:00.340]  HTML and JavaScript content to your browser. And now your browser is theoretically susceptible to
[19:00.340 --> 19:07.960]  some sort of JavaScript injection via a cross-site scripting payload which has been architected by
[19:07.960 --> 19:15.380]  the attacker. Now by adding content security policy from the same web server, you can now
[19:15.380 --> 19:22.680]  inform the browser of the allowed list, where JavaScript is allowed to come from. So now the
[19:22.680 --> 19:28.340]  browser has the ability to identify that, hey, this cross-site scripting payload is not on my
[19:28.340 --> 19:34.520]  allowed list. I'm not going to allow it to execute. And I also added the CDN, Content Delivery Network,
[19:34.520 --> 19:40.300]  so just to let you know that the content security policy's allow list does incorporate more than
[19:40.300 --> 19:45.680]  just the web server that it came from. So there's there's lots of flexibility there. Next slide please.
[19:46.450 --> 19:54.080]  Now we strongly believe that content security policy has reached the point where it is extremely
[19:54.080 --> 20:02.560]  extremely mature. The major browsers all have implementations for it. So we think that
[20:02.560 --> 20:09.820]  the industry is in a good place where we can start demanding the use of content security policy
[20:09.820 --> 20:17.520]  in the products that we consume. Now a question I often get on feedback, i.e. Internet Explorer.
[20:20.780 --> 20:23.700]  The Internet Explorer, I just want to talk about that one briefly.
[20:23.700 --> 20:29.040]  Internet Explorer, I get this question a lot, you know, it doesn't support a content security policy.
[20:29.040 --> 20:36.160]  Is that a problem? And this will not stop Internet Explorer from working. So you don't have to
[20:36.160 --> 20:41.500]  tell customers they can't use Internet Explorer anymore. But what we are taking the position of
[20:41.500 --> 20:48.960]  is that we want to inform our customers that you should be using modern browsers for their
[20:48.960 --> 20:59.230]  security benefit. One issue and one protection is content security policy. Now I talked about having
[20:59.230 --> 21:05.830]  measurements for our best practices. And this is one of the ways that we do it. We've
[21:05.830 --> 21:10.630]  identified poor, better, and best practices for our development teams and came up with ways for
[21:10.630 --> 21:16.410]  them to measure their adoption against that. So the poor practice would be using unsafe inline and
[21:16.410 --> 21:24.470]  unsafe eval. The next better practice would be if you do have inline JavaScript to provide nonces
[21:24.470 --> 21:32.190]  or hashes for it. But the best practice is to remove all those inline JavaScript and eval
[21:32.190 --> 21:38.050]  statements and give the browser an endpoint to report any
[21:38.830 --> 21:45.090]  occurrences of defenses or blocks that it had to execute to a central reporting location so that
[21:45.090 --> 21:53.430]  those issues can be investigated and fixed. And just to give you a general sense of how we do this
[21:53.430 --> 21:59.570]  measurement within OSIsoft, all the measurements we've identified from the best practices
[22:00.770 --> 22:07.110]  have response options. They're all given scores. So you get a score, a scale from zero to ten for
[22:07.110 --> 22:11.650]  all of them. And they can all be combined and you can look at collectively how teams are doing it
[22:11.650 --> 22:17.630]  in response to adoption of these best practices. But in addition to this, we decided to not only
[22:18.910 --> 22:24.410]  revamp the responses in order to encourage better adoption, but we decided also to take the step
[22:24.410 --> 22:30.610]  of making this required for releases going out by the end of this year. Excellent.
[22:32.470 --> 22:38.550]  Now, focusing on the key takeaways, the things that that I covered. The first one was security
[22:38.550 --> 22:42.770]  tools. Using security tools is not enough. You really have to evaluate how you're using them,
[22:42.770 --> 22:49.770]  in our case, for better code coverage. The next one is content security policy. We believe the
[22:49.770 --> 22:57.030]  industry is mature enough that content security policy is in a place that this should become an
[22:57.030 --> 23:03.830]  ICS standard practice, we believe. It gives you a strong second level of defense. With a lot of
[23:03.830 --> 23:09.910]  open source components out there, most of us consume open source, and when the vulnerabilities
[23:09.910 --> 23:16.750]  are discovered in those open source, the public disclosure happens to the public at the same time
[23:16.750 --> 23:21.930]  it happens to us. We know about the vulnerability at the same time everyone else does. So there is
[23:21.930 --> 23:27.850]  limited time that we have to patch and get the protections in place, but by having that second
[23:27.850 --> 23:34.230]  level of defense, content security policy there, it gives us more time to get the patches in place
[23:34.230 --> 23:42.090]  correctly. And the last thing, engaging with companies like Atario and Door is very, very
[23:42.090 --> 23:48.590]  successful in moving security forward for our products and it's been great experience working
[23:48.590 --> 23:55.730]  with them for us. Thank you. Thank you. Thank you very much, Mike. The actual effort was really cool and we
[23:56.490 --> 23:59.390]  very enjoyed this research.