Okay. Hello, everyone. My name is Dori Ardeni. I'm the Head of Incident Response and Hunting at
Artorio. I've previously served as a SERT Team Leader in the IDF, the Israel Defense Forces.
I've... I managed threat hunting operations and also dealt with targeted cyber threats
on big organizations. I also bring hacking knowledge from my experience as a
Red Team Researcher. So I hope you will enjoy this session. The mutual effort together
with OSIsoft and Mike was really cool. Thanks, Dori. And this is Mike Lemley. I work at OSIsoft.
We were in response to the wonderful research that DOR presented to us on the vulnerability
that is covered in our product. I've been at OSIsoft since 1999. I focused primarily
in software development. And then about seven years, I started focusing on cybersecurity
and now work as part of our central architecture team,
and hoping to improve the cybersecurity practices of our development at OSIsoft.
Thanks, Mike. So let's start with a quick story. And at the end of it, you'll get my point.
So two weeks ago, a shift leader at a chemical plant received an alert from the data management
platform. Platform like the one of OSIsoft, PyServer. The alert reported about a boiler
that exceeded from its threshold temperature. The shift leader was really nervous.
And in addition to that, databases of all historical plan data and their backups were
deleted, which means they had no access to industry 4.0 analytics, like people like to say.
The plant was shut down due to this abnormal activity.
The concern was physical damage and human loss.
The shutdown caused a loss of significant production time, which means a lot of money.
So why am I telling you this story? I'm telling you this because it's not a nightmare. It's not
a fairytale story. When you base your decisions on the data management platform, especially
what has been manipulated, you are technically blind. You are blind from your operational environment.
So we have two core goals that we want to achieve from this session. The first one is to raise awareness
about the implications of industrial-oriented vulnerability,
where each vulnerability is a serious threat to our sensitive environment.
And it's not only ICS vulnerability, it could be also IT vulnerabilities that somehow affect
directly on our production flow. The second goal is to showcase how the discovery of one
vulnerability can help other vendors. Mike will talk about the way things proceeded
after we identified the vulnerability in their product.
I want to talk a little bit about what the Pi system is from OSIsoft. We start off with
our core customers. So these six areas of business are where most of our customers
are in the industry. And the company OSIsoft has been around for about 40 years, and we have
customers in 140 different countries. And this world map gives you an indication of where we
have support offices to help our customers. This reference architecture explains how a lot of our
customers are in the critical system. Now, what we want is we want to reduce the amount of people
that have access within the critical system. And the OSIsoft Pi system infrastructure allows them
to do that by pulling the data that is needed for analysis and business decisions out of the
critical system so that those people don't need to have access there in order to do their job.
This additional data reference architecture also represents how the Pi system is used.
Now, the Pi server inside of the control center is used to extract the data from the control center
using hundreds of different protocols based on the different types of equipment there.
Data is then sent over to the corporate access or corporate network zone, where it is then used by
those employees in order to do their analysis. Now, the target for this particular incident
is within the vision server. And that is where the API server exists that is being used
and attacked by this particular incident, which is a cross-site security vulnerability.
Okay, so before I dive into the details of the vulnerability,
suppose you ask yourself why we even started looking at Pi server.
And the answer is that we had to put some effort into product tasks. We had to integrate
OSI soft platform, the Pi server, into our security orchestration and response platform.
But as a curious researcher, I decided to test the asset framework of the Pi server
on XSS. It was very late at night, so realistic as it sounds. I fought against the web API parsing
mechanism in order to inject JavaScript code into client browsers. And in the end, I succeeded.
And we are here.
So here is a screenshot from the asset framework.
On the left, we have a simple plant, a petroleum plant with a boiler and a tank. Each
element has attributes. For example, the boiler has an attribute of temperature and the tank
has an attribute of volume. And element one is the child element only for my tests.
On the right, you can see the description field, the vulnerable field. We've encoded base64
JavaScript. In order to bypass the web API parsing mechanism, you have to insert some magic characters
at the beginning of it. Once an attacker managed to do so, he or she can run JavaScript code on
client browser for phishing, keylogging, etc. Okay, so here is a possible scenario for this attack.
First, an attacker is stealing low privileges credentials inside the network.
After that, the attacker is injecting the malicious JavaScript to the vulnerable field
that we saw in the previous slide. Then a victim, for example, an innocent engineer,
is accessing the web infected page. And then a fake login form is prompting for
username and password, and also in parallel, changing the notification rule template on behalf
of the engineer high privileges credentials. At the end of it, the attacker receiving the
credentials and also causing TemporTool false positive alert. Okay, so let's see it in action.
On the right is the server of the attacker. This is on port 8080 for the stolen credentials.
On the left, the victim browser. Once the victim passes the cursor over the infected field,
the descriptive field, the fake login form is prompting for username and password.
The victim is inserting the pyadmin password. And once the victim submits
credentials, the attacker gets the stolen credentials on the right. Pyadmin and the
strong password. So from here, it's pretty much game over. And in the same method,
in the same technique, I could also add a JavaScript code to change the notification rule
template in order to change the rule and to cause false positive alert of high temperature.
Okay, so this screenshot was taken from USAID site. The vulnerability has got rank of 7.7.
OSIsoft also added us in their all of thanks list. Here in the bottom, Dory O'Donoghue and Eliad
Mualem. And I suppose you asked yourself how to avoid this kind of attacks. So the answer is
patching. OSIsoft released a patch for the vulnerability very, very quickly.
But we have to do a lot more, a lot more proactive actions, such as threat hunting operations
on our sensitive environment to other firewall rules. And also to get visibility into our
assets and their version in order to perform vulnerability management, etc. But that's for
another talk. Now I'll talk about the lessons that OSIsoft learned from this incident and
how we responded. Now to start off with, I'm going to talk a little bit about our security
development lifecycle process at OSIsoft. And that'll define how we changed, we responded.
So we at OSIsoft established several different security best practices. And then what we do is
we measure the adoption by our 40, around 40 development teams to apply those security best
practices in their development. Now why do we measure? I constantly say, I mean, what good are
best practices if you don't measure to see how well you are adopting them? But in addition to
that, it really helps teams plan. When you go through the process of identifying measurements
for the best practices, it forces you to identify milestones. And those milestones are very useful
for the teams to plan against and plan their development activities. Another thing that it
does is it forces you to organize and prioritize all these best practices. And that also helps
the teams identify how to best move forward. Now looking at our SDL process, focused entirely on
cross-site scripting allowed us to see where we could improve our process. So we have education,
of defensive programming. We also have various tools that we use and the vendors which that we
use for these various tools are listed on the right side, along with some of the security
researchers we engage with. So as I said, we have various security tools we use. There's static
analysis, software component analysis, which actually helps us find cross-site scripting
vulnerabilities in components that we consume, open source products. We have various security
headers that we encourage use of, as well as the external reviews and incident response, which was
very key in how we handle this issue that came to us from Otario. Now in looking at those processes,
we defined there were some areas we thought we could improve or add on to. So the first one is
an internal security tech talk. We do tech talks weekly at OSIsoft. Every third week they are
focused on security and we did a tech talk specifically on this incident so we could share
the lessons learned from this with the other of the 40 teams at OSIsoft. We also looked at how
we use the static application security testing tool and identified some process improvements
for our use of that tool. And the last is we decided to get much more aggressive on the
adoption of content security policy at OSIsoft. Now I did say that we do measure our best practices
and I just wanted to give you a sense of what some of the measurements are that we use associated
with just our cross-site scripting defenses at OSIsoft in the best practice areas I defined.
Now we didn't do any process improvements on our DAST tool and I'm going to talk briefly
about that. Now the DAST tool that we use is from Qualys and for those of you who don't remember
what a DAST tool is, it's basically it's run as a server. Your application is up and running
and then the DAST tool will send dynamic tests at your web server, typically OWASP top 10 attacks,
and see whether or not your web server has the appropriate implementations in place to protect
against it. Now in this particular cross-site scripting issue, as Dor explained, it was a
stored cross-site scripting attack. But in looking at the tools, it's more of a DOM
cross-site scripting vulnerability with insecure use of innerHTML. Now our DAST tool, Qualys,
is very good when the source of the innerHTML parsing, the bad package, is from a document URL,
but it doesn't really handle well when the source is from an HTML document. So we decided
not to focus there. Now our static application security testing tool is from Synopsys and if
you look at the bottom right as a reminder for what SAS tools do, they analyze your code. So
they're looking directly at your code, look at call stacks, how functions are reached, and identify
insecure call paths or insecure calling sequences. So we felt that was the best tool. It should
identify where innerHTML is used and it should be able to flag insecure uses of it. So what went
wrong with this? We identified that there was a tool compatibility problem with the NuGet package
that we used. It's a Razor engine and our tool Synopsys did not work well with it. It does
work very well with the Razor analysis built into ASP.NET. So consequent to this problem,
our scanning for this particular project missed all JavaScript and CSHTML. So the process
improvements we identified is we need to be able to do some central evaluation of the SAS
configurations and we're working with Synopsys on how to best move that idea forward. The other
thing we're going to do is for this one particular development team, we're going to move them to
using the ASP.NET MVC Razor engine instead. And then we're also considering increasing the
sensitivity to completely disallow the use of innerHTML. We're not sure exactly how this is
going to play out, but innerHTML is definitely not something we want to use. So we're either
going to be heavily scrutinizing the use of innerHTML or completely disallow it. So we're
still working that one out. Next slide please. Now I'm going to switch context and talk a little
bit about content security policy. And just to give you a why behind that, I'm focusing on jQuery.
Now over the last couple of years, jQuery has released several different vulnerabilities.
These all on this chart are the cross-site scripting vulnerability. And as you can see
with the overlay that was just added, they all fall in the high range for CVSS scoring, which
means you have to take these particular vulnerabilities very seriously. Cross-site
scripting in general does tend to fall into the high vulnerability area. And dealing with
all of these different patch updates from this open source does present challenges to vendors
which consume open source and components under the scrutiny of security research. Next please.
Now content security policy, just to explain how it works, the web server will send down
HTML and JavaScript content to your browser. And now your browser is theoretically susceptible to
some sort of JavaScript injection via a cross-site scripting payload which has been architected by
the attacker. Now by adding content security policy from the same web server, you can now
inform the browser of the allowed list, where JavaScript is allowed to come from. So now the
browser has the ability to identify that, hey, this cross-site scripting payload is not on my
allowed list. I'm not going to allow it to execute. And I also added the CDN, Content Delivery Network,
so just to let you know that the content security policy's allow list does incorporate more than
just the web server that it came from. So there's there's lots of flexibility there. Next slide please.
Now we strongly believe that content security policy has reached the point where it is extremely
extremely mature. The major browsers all have implementations for it. So we think that
the industry is in a good place where we can start demanding the use of content security policy
in the products that we consume. Now a question I often get on feedback, i.e. Internet Explorer.
The Internet Explorer, I just want to talk about that one briefly.
Internet Explorer, I get this question a lot, you know, it doesn't support a content security policy.
Is that a problem? And this will not stop Internet Explorer from working. So you don't have to
tell customers they can't use Internet Explorer anymore. But what we are taking the position of
is that we want to inform our customers that you should be using modern browsers for their
security benefit. One issue and one protection is content security policy. Now I talked about having
measurements for our best practices. And this is one of the ways that we do it. We've
identified poor, better, and best practices for our development teams and came up with ways for
them to measure their adoption against that. So the poor practice would be using unsafe inline and
unsafe eval. The next better practice would be if you do have inline JavaScript to provide nonces
or hashes for it. But the best practice is to remove all those inline JavaScript and eval
statements and give the browser an endpoint to report any
occurrences of defenses or blocks that it had to execute to a central reporting location so that
those issues can be investigated and fixed. And just to give you a general sense of how we do this
measurement within OSIsoft, all the measurements we've identified from the best practices
have response options. They're all given scores. So you get a score, a scale from zero to ten for
all of them. And they can all be combined and you can look at collectively how teams are doing it
in response to adoption of these best practices. But in addition to this, we decided to not only
revamp the responses in order to encourage better adoption, but we decided also to take the step
of making this required for releases going out by the end of this year. Excellent.
Now, focusing on the key takeaways, the things that that I covered. The first one was security
tools. Using security tools is not enough. You really have to evaluate how you're using them,
in our case, for better code coverage. The next one is content security policy. We believe the
industry is mature enough that content security policy is in a place that this should become an
ICS standard practice, we believe. It gives you a strong second level of defense. With a lot of
open source components out there, most of us consume open source, and when the vulnerabilities
are discovered in those open source, the public disclosure happens to the public at the same time
it happens to us. We know about the vulnerability at the same time everyone else does. So there is
limited time that we have to patch and get the protections in place, but by having that second
level of defense, content security policy there, it gives us more time to get the patches in place
correctly. And the last thing, engaging with companies like Atario and Door is very, very
successful in moving security forward for our products and it's been great experience working
with them for us. Thank you. Thank you. Thank you very much, Mike. The actual effort was really cool and we
very enjoyed this research.