start	end	text
1110	7650	Okay. Hello, everyone. My name is Dori Ardeni. I'm the Head of Incident Response and Hunting at
7650	15030	Artorio. I've previously served as a SERT Team Leader in the IDF, the Israel Defense Forces.
15190	22170	I've... I managed threat hunting operations and also dealt with targeted cyber threats
22170	29670	on big organizations. I also bring hacking knowledge from my experience as a
29670	35270	Red Team Researcher. So I hope you will enjoy this session. The mutual effort together
35270	44990	with OSIsoft and Mike was really cool. Thanks, Dori. And this is Mike Lemley. I work at OSIsoft.
44990	51090	We were in response to the wonderful research that DOR presented to us on the vulnerability
51090	58930	that is covered in our product. I've been at OSIsoft since 1999. I focused primarily
58930	64350	in software development. And then about seven years, I started focusing on cybersecurity
65040	68370	and now work as part of our central architecture team,
68370	73940	and hoping to improve the cybersecurity practices of our development at OSIsoft.
75690	83190	Thanks, Mike. So let's start with a quick story. And at the end of it, you'll get my point.
83830	90250	So two weeks ago, a shift leader at a chemical plant received an alert from the data management
90250	99030	platform. Platform like the one of OSIsoft, PyServer. The alert reported about a boiler
99030	104530	that exceeded from its threshold temperature. The shift leader was really nervous.
105230	111390	And in addition to that, databases of all historical plan data and their backups were
111390	119650	deleted, which means they had no access to industry 4.0 analytics, like people like to say.
119870	123730	The plant was shut down due to this abnormal activity.
123730	127710	The concern was physical damage and human loss.
129570	136290	The shutdown caused a loss of significant production time, which means a lot of money.
137790	143290	So why am I telling you this story? I'm telling you this because it's not a nightmare. It's not
143470	150130	a fairytale story. When you base your decisions on the data management platform, especially
150130	156970	what has been manipulated, you are technically blind. You are blind from your operational environment.
158610	166270	So we have two core goals that we want to achieve from this session. The first one is to raise awareness
166270	169970	about the implications of industrial-oriented vulnerability,
170590	176190	where each vulnerability is a serious threat to our sensitive environment.
177410	185490	And it's not only ICS vulnerability, it could be also IT vulnerabilities that somehow affect
185490	193030	directly on our production flow. The second goal is to showcase how the discovery of one
193030	198830	vulnerability can help other vendors. Mike will talk about the way things proceeded
199350	203670	after we identified the vulnerability in their product.
205860	213080	I want to talk a little bit about what the Pi system is from OSIsoft. We start off with
213080	219880	our core customers. So these six areas of business are where most of our customers
221220	229860	are in the industry. And the company OSIsoft has been around for about 40 years, and we have
229860	236560	customers in 140 different countries. And this world map gives you an indication of where we
236560	246040	have support offices to help our customers. This reference architecture explains how a lot of our
246040	252980	customers are in the critical system. Now, what we want is we want to reduce the amount of people
252980	259300	that have access within the critical system. And the OSIsoft Pi system infrastructure allows them
259300	265400	to do that by pulling the data that is needed for analysis and business decisions out of the
265400	272220	critical system so that those people don't need to have access there in order to do their job.
275670	283260	This additional data reference architecture also represents how the Pi system is used.
283260	291240	Now, the Pi server inside of the control center is used to extract the data from the control center
291240	296440	using hundreds of different protocols based on the different types of equipment there.
296440	303040	Data is then sent over to the corporate access or corporate network zone, where it is then used by
303040	308380	those employees in order to do their analysis. Now, the target for this particular incident
308380	318180	is within the vision server. And that is where the API server exists that is being used
318180	323360	and attacked by this particular incident, which is a cross-site security vulnerability.
325560	330340	Okay, so before I dive into the details of the vulnerability,
331060	335300	suppose you ask yourself why we even started looking at Pi server.
335740	343380	And the answer is that we had to put some effort into product tasks. We had to integrate
344020	351540	OSI soft platform, the Pi server, into our security orchestration and response platform.
351540	356900	But as a curious researcher, I decided to test the asset framework of the Pi server
358320	367640	on XSS. It was very late at night, so realistic as it sounds. I fought against the web API parsing
367640	376500	mechanism in order to inject JavaScript code into client browsers. And in the end, I succeeded.
376500	378200	And we are here.
380120	384600	So here is a screenshot from the asset framework.
384600	393380	On the left, we have a simple plant, a petroleum plant with a boiler and a tank. Each
393380	402080	element has attributes. For example, the boiler has an attribute of temperature and the tank
402080	410660	has an attribute of volume. And element one is the child element only for my tests.
410660	419820	On the right, you can see the description field, the vulnerable field. We've encoded base64
419820	433020	JavaScript. In order to bypass the web API parsing mechanism, you have to insert some magic characters
433020	446680	at the beginning of it. Once an attacker managed to do so, he or she can run JavaScript code on
446680	459540	client browser for phishing, keylogging, etc. Okay, so here is a possible scenario for this attack.
459540	468900	First, an attacker is stealing low privileges credentials inside the network.
468960	476200	After that, the attacker is injecting the malicious JavaScript to the vulnerable field
476200	483180	that we saw in the previous slide. Then a victim, for example, an innocent engineer,
483180	491920	is accessing the web infected page. And then a fake login form is prompting for
492760	500240	username and password, and also in parallel, changing the notification rule template on behalf
500940	507580	of the engineer high privileges credentials. At the end of it, the attacker receiving the
507580	519420	credentials and also causing TemporTool false positive alert. Okay, so let's see it in action.
520580	530060	On the right is the server of the attacker. This is on port 8080 for the stolen credentials.
530060	544100	On the left, the victim browser. Once the victim passes the cursor over the infected field,
544100	552090	the descriptive field, the fake login form is prompting for username and password.
557010	572770	The victim is inserting the pyadmin password. And once the victim submits
574680	584880	credentials, the attacker gets the stolen credentials on the right. Pyadmin and the
584880	594840	strong password. So from here, it's pretty much game over. And in the same method,
594840	600320	in the same technique, I could also add a JavaScript code to change the notification rule
600320	611580	template in order to change the rule and to cause false positive alert of high temperature.
614790	624410	Okay, so this screenshot was taken from USAID site. The vulnerability has got rank of 7.7.
624710	634010	OSIsoft also added us in their all of thanks list. Here in the bottom, Dory O'Donoghue and Eliad
634010	641910	Mualem. And I suppose you asked yourself how to avoid this kind of attacks. So the answer is
641910	647890	patching. OSIsoft released a patch for the vulnerability very, very quickly.
647890	655690	But we have to do a lot more, a lot more proactive actions, such as threat hunting operations
655690	665070	on our sensitive environment to other firewall rules. And also to get visibility into our
666030	676910	assets and their version in order to perform vulnerability management, etc. But that's for
676910	685610	another talk. Now I'll talk about the lessons that OSIsoft learned from this incident and
685610	692010	how we responded. Now to start off with, I'm going to talk a little bit about our security
692010	699930	development lifecycle process at OSIsoft. And that'll define how we changed, we responded.
699930	706790	So we at OSIsoft established several different security best practices. And then what we do is
706790	713950	we measure the adoption by our 40, around 40 development teams to apply those security best
713950	722430	practices in their development. Now why do we measure? I constantly say, I mean, what good are
722430	728530	best practices if you don't measure to see how well you are adopting them? But in addition to
728530	734950	that, it really helps teams plan. When you go through the process of identifying measurements
734950	740870	for the best practices, it forces you to identify milestones. And those milestones are very useful
740870	747570	for the teams to plan against and plan their development activities. Another thing that it
747570	752750	does is it forces you to organize and prioritize all these best practices. And that also helps
753280	763790	the teams identify how to best move forward. Now looking at our SDL process, focused entirely on
763790	770790	cross-site scripting allowed us to see where we could improve our process. So we have education,
771910	779290	of defensive programming. We also have various tools that we use and the vendors which that we
779290	783830	use for these various tools are listed on the right side, along with some of the security
783830	790050	researchers we engage with. So as I said, we have various security tools we use. There's static
790050	794750	analysis, software component analysis, which actually helps us find cross-site scripting
794750	801170	vulnerabilities in components that we consume, open source products. We have various security
801170	808150	headers that we encourage use of, as well as the external reviews and incident response, which was
808150	816750	very key in how we handle this issue that came to us from Otario. Now in looking at those processes,
816750	824210	we defined there were some areas we thought we could improve or add on to. So the first one is
824210	831310	an internal security tech talk. We do tech talks weekly at OSIsoft. Every third week they are
831310	837810	focused on security and we did a tech talk specifically on this incident so we could share
837810	845190	the lessons learned from this with the other of the 40 teams at OSIsoft. We also looked at how
845190	851230	we use the static application security testing tool and identified some process improvements
851230	856890	for our use of that tool. And the last is we decided to get much more aggressive on the
856890	868250	adoption of content security policy at OSIsoft. Now I did say that we do measure our best practices
868250	874790	and I just wanted to give you a sense of what some of the measurements are that we use associated
874790	880910	with just our cross-site scripting defenses at OSIsoft in the best practice areas I defined.
882950	889630	Now we didn't do any process improvements on our DAST tool and I'm going to talk briefly
889630	896390	about that. Now the DAST tool that we use is from Qualys and for those of you who don't remember
896390	903390	what a DAST tool is, it's basically it's run as a server. Your application is up and running
903390	913290	and then the DAST tool will send dynamic tests at your web server, typically OWASP top 10 attacks,
913290	918890	and see whether or not your web server has the appropriate implementations in place to protect
918890	926350	against it. Now in this particular cross-site scripting issue, as Dor explained, it was a
926350	933210	stored cross-site scripting attack. But in looking at the tools, it's more of a DOM
933210	942950	cross-site scripting vulnerability with insecure use of innerHTML. Now our DAST tool, Qualys,
942950	951150	is very good when the source of the innerHTML parsing, the bad package, is from a document URL,
951150	957590	but it doesn't really handle well when the source is from an HTML document. So we decided
957590	967700	not to focus there. Now our static application security testing tool is from Synopsys and if
967700	973920	you look at the bottom right as a reminder for what SAS tools do, they analyze your code. So
973920	978820	they're looking directly at your code, look at call stacks, how functions are reached, and identify
978820	987940	insecure call paths or insecure calling sequences. So we felt that was the best tool. It should
987940	995120	identify where innerHTML is used and it should be able to flag insecure uses of it. So what went
995120	1001980	wrong with this? We identified that there was a tool compatibility problem with the NuGet package
1001980	1010680	that we used. It's a Razor engine and our tool Synopsys did not work well with it. It does
1010680	1018880	work very well with the Razor analysis built into ASP.NET. So consequent to this problem,
1018880	1026040	our scanning for this particular project missed all JavaScript and CSHTML. So the process
1026040	1031880	improvements we identified is we need to be able to do some central evaluation of the SAS
1031880	1039140	configurations and we're working with Synopsys on how to best move that idea forward. The other
1039140	1044200	thing we're going to do is for this one particular development team, we're going to move them to
1044200	1051780	using the ASP.NET MVC Razor engine instead. And then we're also considering increasing the
1051780	1057260	sensitivity to completely disallow the use of innerHTML. We're not sure exactly how this is
1057260	1062760	going to play out, but innerHTML is definitely not something we want to use. So we're either
1062760	1068300	going to be heavily scrutinizing the use of innerHTML or completely disallow it. So we're
1068300	1073940	still working that one out. Next slide please. Now I'm going to switch context and talk a little
1073940	1080620	bit about content security policy. And just to give you a why behind that, I'm focusing on jQuery.
1080620	1087460	Now over the last couple of years, jQuery has released several different vulnerabilities.
1088340	1093260	These all on this chart are the cross-site scripting vulnerability. And as you can see
1093260	1099660	with the overlay that was just added, they all fall in the high range for CVSS scoring, which
1099660	1104240	means you have to take these particular vulnerabilities very seriously. Cross-site
1104240	1111880	scripting in general does tend to fall into the high vulnerability area. And dealing with
1111880	1118380	all of these different patch updates from this open source does present challenges to vendors
1118380	1127140	which consume open source and components under the scrutiny of security research. Next please.
1128240	1133700	Now content security policy, just to explain how it works, the web server will send down
1133700	1140340	HTML and JavaScript content to your browser. And now your browser is theoretically susceptible to
1140340	1147960	some sort of JavaScript injection via a cross-site scripting payload which has been architected by
1147960	1155380	the attacker. Now by adding content security policy from the same web server, you can now
1155380	1162680	inform the browser of the allowed list, where JavaScript is allowed to come from. So now the
1162680	1168340	browser has the ability to identify that, hey, this cross-site scripting payload is not on my
1168340	1174520	allowed list. I'm not going to allow it to execute. And I also added the CDN, Content Delivery Network,
1174520	1180300	so just to let you know that the content security policy's allow list does incorporate more than
1180300	1185680	just the web server that it came from. So there's there's lots of flexibility there. Next slide please.
1186450	1194080	Now we strongly believe that content security policy has reached the point where it is extremely
1194080	1202560	extremely mature. The major browsers all have implementations for it. So we think that
1202560	1209820	the industry is in a good place where we can start demanding the use of content security policy
1209820	1217520	in the products that we consume. Now a question I often get on feedback, i.e. Internet Explorer.
1220780	1223700	The Internet Explorer, I just want to talk about that one briefly.
1223700	1229040	Internet Explorer, I get this question a lot, you know, it doesn't support a content security policy.
1229040	1236160	Is that a problem? And this will not stop Internet Explorer from working. So you don't have to
1236160	1241500	tell customers they can't use Internet Explorer anymore. But what we are taking the position of
1241500	1248960	is that we want to inform our customers that you should be using modern browsers for their
1248960	1259230	security benefit. One issue and one protection is content security policy. Now I talked about having
1259230	1265830	measurements for our best practices. And this is one of the ways that we do it. We've
1265830	1270630	identified poor, better, and best practices for our development teams and came up with ways for
1270630	1276410	them to measure their adoption against that. So the poor practice would be using unsafe inline and
1276410	1284470	unsafe eval. The next better practice would be if you do have inline JavaScript to provide nonces
1284470	1292190	or hashes for it. But the best practice is to remove all those inline JavaScript and eval
1292190	1298050	statements and give the browser an endpoint to report any
1298830	1305090	occurrences of defenses or blocks that it had to execute to a central reporting location so that
1305090	1313430	those issues can be investigated and fixed. And just to give you a general sense of how we do this
1313430	1319570	measurement within OSIsoft, all the measurements we've identified from the best practices
1320770	1327110	have response options. They're all given scores. So you get a score, a scale from zero to ten for
1327110	1331650	all of them. And they can all be combined and you can look at collectively how teams are doing it
1331650	1337630	in response to adoption of these best practices. But in addition to this, we decided to not only
1338910	1344410	revamp the responses in order to encourage better adoption, but we decided also to take the step
1344410	1350610	of making this required for releases going out by the end of this year. Excellent.
1352470	1358550	Now, focusing on the key takeaways, the things that that I covered. The first one was security
1358550	1362770	tools. Using security tools is not enough. You really have to evaluate how you're using them,
1362770	1369770	in our case, for better code coverage. The next one is content security policy. We believe the
1369770	1377030	industry is mature enough that content security policy is in a place that this should become an
1377030	1383830	ICS standard practice, we believe. It gives you a strong second level of defense. With a lot of
1383830	1389910	open source components out there, most of us consume open source, and when the vulnerabilities
1389910	1396750	are discovered in those open source, the public disclosure happens to the public at the same time
1396750	1401930	it happens to us. We know about the vulnerability at the same time everyone else does. So there is
1401930	1407850	limited time that we have to patch and get the protections in place, but by having that second
1407850	1414230	level of defense, content security policy there, it gives us more time to get the patches in place
1414230	1422090	correctly. And the last thing, engaging with companies like Atario and Door is very, very
1422090	1428590	successful in moving security forward for our products and it's been great experience working
1428590	1435730	with them for us. Thank you. Thank you. Thank you very much, Mike. The actual effort was really cool and we
1436490	1439390	very enjoyed this research.