1
00:00:01,110 --> 00:00:01,870
Okay.

2
00:00:01,870 --> 00:00:03,310
Hello, everyone.

3
00:00:03,310 --> 00:00:04,750
My name is Dori Ardeni.

4
00:00:04,750 --> 00:00:08,470
I'm the Head of Incident Response and Hunting at Artorio.

5
00:00:08,470 --> 00:00:15,030
I've previously served as a SERT Team Leader in the IDF, the Israel Defense Forces.

6
00:00:15,190 --> 00:00:24,430
I've... I managed threat hunting operations and also dealt with targeted cyber threats on big organizations.

7
00:00:24,770 --> 00:00:31,550
I also bring hacking knowledge from my experience as a Red Team Researcher.

8
00:00:31,550 --> 00:00:33,750
So I hope you will enjoy this session.

9
00:00:33,750 --> 00:00:39,190
The mutual effort together with OSIsoft and Mike was really cool.

10
00:00:40,230 --> 00:00:41,350
Thanks, Dori.

11
00:00:41,350 --> 00:00:42,950
And this is Mike Lemley.

12
00:00:42,950 --> 00:00:44,990
I work at OSIsoft.

13
00:00:44,990 --> 00:00:53,030
We were in response to the wonderful research that DOR presented to us on the vulnerability that is covered in our product.

14
00:00:53,090 --> 00:00:57,170
I've been at OSIsoft since 1999.

15
00:00:57,210 --> 00:01:00,590
I focused primarily in software development.

16
00:01:00,770 --> 00:01:13,940
And then about seven years, I started focusing on cybersecurity and now work as part of our central architecture team, and hoping to improve the cybersecurity practices of our development at OSIsoft.

17
00:01:15,690 --> 00:01:17,230
Thanks, Mike.

18
00:01:17,690 --> 00:01:20,490
So let's start with a quick story.

19
00:01:20,490 --> 00:01:23,190
And at the end of it, you'll get my point.

20
00:01:23,830 --> 00:01:30,930
So two weeks ago, a shift leader at a chemical plant received an alert from the data management platform.

21
00:01:30,990 --> 00:01:35,170
Platform like the one of OSIsoft, PyServer.

22
00:01:36,590 --> 00:01:42,110
The alert reported about a boiler that exceeded from its threshold temperature.

23
00:01:42,270 --> 00:01:44,530
The shift leader was really nervous.

24
00:01:45,230 --> 00:01:59,650
And in addition to that, databases of all historical plan data and their backups were deleted, which means they had no access to industry 4.0 analytics, like people like to say.

25
00:01:59,870 --> 00:02:03,730
The plant was shut down due to this abnormal activity.

26
00:02:03,730 --> 00:02:07,710
The concern was physical damage and human loss.

27
00:02:09,570 --> 00:02:16,290
The shutdown caused a loss of significant production time, which means a lot of money.

28
00:02:17,790 --> 00:02:19,990
So why am I telling you this story?

29
00:02:19,990 --> 00:02:22,850
I'm telling you this because it's not a nightmare.

30
00:02:22,850 --> 00:02:25,230
It's not a fairytale story.

31
00:02:25,530 --> 00:02:33,510
When you base your decisions on the data management platform, especially what has been manipulated, you are technically blind.

32
00:02:33,510 --> 00:02:36,970
You are blind from your operational environment.

33
00:02:38,610 --> 00:02:44,250
So we have two core goals that we want to achieve from this session.

34
00:02:44,450 --> 00:02:56,190
The first one is to raise awareness about the implications of industrial-oriented vulnerability, where each vulnerability is a serious threat to our sensitive environment.

35
00:02:57,410 --> 00:03:08,590
And it's not only ICS vulnerability, it could be also IT vulnerabilities that somehow affect directly on our production flow.

36
00:03:08,810 --> 00:03:16,090
The second goal is to showcase how the discovery of one vulnerability can help other vendors.

37
00:03:16,310 --> 00:03:23,670
Mike will talk about the way things proceeded after we identified the vulnerability in their product.

38
00:03:25,860 --> 00:03:30,880
I want to talk a little bit about what the Pi system is from OSIsoft.

39
00:03:30,880 --> 00:03:34,840
We start off with our core customers.

40
00:03:34,840 --> 00:03:42,320
So these six areas of business are where most of our customers are in the industry.

41
00:03:42,540 --> 00:03:52,980
And the company OSIsoft has been around for about 40 years, and we have customers in 140 different countries.

42
00:03:52,980 --> 00:03:59,660
And this world map gives you an indication of where we have support offices to help our customers.

43
00:04:01,240 --> 00:04:07,500
This reference architecture explains how a lot of our customers are in the critical system.

44
00:04:08,220 --> 00:04:16,340
Now, what we want is we want to reduce the amount of people that have access within the critical system.

45
00:04:16,400 --> 00:04:32,220
And the OSIsoft Pi system infrastructure allows them to do that by pulling the data that is needed for analysis and business decisions out of the critical system so that those people don't need to have access there in order to do their job.

46
00:04:35,670 --> 00:04:43,260
This additional data reference architecture also represents how the Pi system is used.

47
00:04:43,260 --> 00:04:56,440
Now, the Pi server inside of the control center is used to extract the data from the control center using hundreds of different protocols based on the different types of equipment there.

48
00:04:56,440 --> 00:05:06,120
Data is then sent over to the corporate access or corporate network zone, where it is then used by those employees in order to do their analysis.

49
00:05:06,220 --> 00:05:11,360
Now, the target for this particular incident is within the vision server.

50
00:05:11,720 --> 00:05:23,360
And that is where the API server exists that is being used and attacked by this particular incident, which is a cross-site security vulnerability.

51
00:05:25,560 --> 00:05:35,300
Okay, so before I dive into the details of the vulnerability, suppose you ask yourself why we even started looking at Pi server.

52
00:05:35,740 --> 00:05:41,540
And the answer is that we had to put some effort into product tasks.

53
00:05:41,540 --> 00:05:51,540
We had to integrate OSI soft platform, the Pi server, into our security orchestration and response platform.

54
00:05:51,540 --> 00:06:00,380
But as a curious researcher, I decided to test the asset framework of the Pi server on XSS.

55
00:06:01,000 --> 00:06:04,660
It was very late at night, so realistic as it sounds.

56
00:06:04,660 --> 00:06:14,200
I fought against the web API parsing mechanism in order to inject JavaScript code into client browsers.

57
00:06:14,480 --> 00:06:16,500
And in the end, I succeeded.

58
00:06:16,500 --> 00:06:18,200
And we are here.

59
00:06:20,120 --> 00:06:24,600
So here is a screenshot from the asset framework.

60
00:06:24,600 --> 00:06:32,820
On the left, we have a simple plant, a petroleum plant with a boiler and a tank.

61
00:06:32,820 --> 00:06:37,720
Each element has attributes.

62
00:06:37,720 --> 00:06:46,100
For example, the boiler has an attribute of temperature and the tank has an attribute of volume.

63
00:06:46,400 --> 00:06:50,660
And element one is the child element only for my tests.

64
00:06:50,660 --> 00:06:56,840
On the right, you can see the description field, the vulnerable field.

65
00:06:56,840 --> 00:07:01,500
We've encoded base64 JavaScript.

66
00:07:01,960 --> 00:07:17,740
In order to bypass the web API parsing mechanism, you have to insert some magic characters at the beginning of it.

67
00:07:18,140 --> 00:07:32,480
Once an attacker managed to do so, he or she can run JavaScript code on client browser for phishing, keylogging, etc.

68
00:07:35,560 --> 00:07:39,540
Okay, so here is a possible scenario for this attack.

69
00:07:39,540 --> 00:07:48,900
First, an attacker is stealing low privileges credentials inside the network.

70
00:07:48,960 --> 00:07:58,580
After that, the attacker is injecting the malicious JavaScript to the vulnerable field that we saw in the previous slide.

71
00:07:58,580 --> 00:08:07,320
Then a victim, for example, an innocent engineer, is accessing the web infected page.

72
00:08:08,020 --> 00:08:25,240
And then a fake login form is prompting for username and password, and also in parallel, changing the notification rule template on behalf of the engineer high privileges credentials.

73
00:08:25,740 --> 00:08:33,160
At the end of it, the attacker receiving the credentials and also causing TemporTool false positive alert.

74
00:08:35,900 --> 00:08:39,420
Okay, so let's see it in action.

75
00:08:40,580 --> 00:08:46,180
On the right is the server of the attacker.

76
00:08:46,180 --> 00:08:50,060
This is on port 8080 for the stolen credentials.

77
00:08:50,060 --> 00:08:58,360
On the left, the victim browser.

78
00:08:59,180 --> 00:09:12,090
Once the victim passes the cursor over the infected field, the descriptive field, the fake login form is prompting for username and password.

79
00:09:17,010 --> 00:09:23,730
The victim is inserting the pyadmin password.

80
00:09:26,810 --> 00:09:42,360
And once the victim submits credentials, the attacker gets the stolen credentials on the right.

81
00:09:42,360 --> 00:09:46,580
Pyadmin and the strong password.

82
00:09:47,520 --> 00:09:50,600
So from here, it's pretty much game over.

83
00:09:50,940 --> 00:10:11,580
And in the same method, in the same technique, I could also add a JavaScript code to change the notification rule template in order to change the rule and to cause false positive alert of high temperature.

84
00:10:14,790 --> 00:10:19,750
Okay, so this screenshot was taken from USAID site.

85
00:10:19,750 --> 00:10:24,410
The vulnerability has got rank of 7.7.

86
00:10:24,710 --> 00:10:31,910
OSIsoft also added us in their all of thanks list.

87
00:10:32,090 --> 00:10:35,150
Here in the bottom, Dory O'Donoghue and Eliad Mualem.

88
00:10:36,030 --> 00:10:40,910
And I suppose you asked yourself how to avoid this kind of attacks.

89
00:10:40,910 --> 00:10:43,370
So the answer is patching.

90
00:10:43,370 --> 00:10:47,890
OSIsoft released a patch for the vulnerability very, very quickly.

91
00:10:47,890 --> 00:11:01,430
But we have to do a lot more, a lot more proactive actions, such as threat hunting operations on our sensitive environment to other firewall rules.

92
00:11:01,590 --> 00:11:15,290
And also to get visibility into our assets and their version in order to perform vulnerability management, etc.

93
00:11:15,290 --> 00:11:18,230
But that's for another talk.

94
00:11:20,690 --> 00:11:27,530
Now I'll talk about the lessons that OSIsoft learned from this incident and how we responded.

95
00:11:27,730 --> 00:11:35,530
Now to start off with, I'm going to talk a little bit about our security development lifecycle process at OSIsoft.

96
00:11:35,630 --> 00:11:39,930
And that'll define how we changed, we responded.

97
00:11:39,930 --> 00:11:45,410
So we at OSIsoft established several different security best practices.

98
00:11:45,410 --> 00:11:56,370
And then what we do is we measure the adoption by our 40, around 40 development teams to apply those security best practices in their development.

99
00:11:56,750 --> 00:11:58,330
Now why do we measure?

100
00:11:59,470 --> 00:12:07,190
I constantly say, I mean, what good are best practices if you don't measure to see how well you are adopting them?

101
00:12:07,190 --> 00:12:11,750
But in addition to that, it really helps teams plan.

102
00:12:12,190 --> 00:12:19,290
When you go through the process of identifying measurements for the best practices, it forces you to identify milestones.

103
00:12:19,290 --> 00:12:26,790
And those milestones are very useful for the teams to plan against and plan their development activities.

104
00:12:26,790 --> 00:12:31,710
Another thing that it does is it forces you to organize and prioritize all these best practices.

105
00:12:31,710 --> 00:12:37,550
And that also helps the teams identify how to best move forward.

106
00:12:38,710 --> 00:12:49,150
Now looking at our SDL process, focused entirely on cross-site scripting allowed us to see where we could improve our process.

107
00:12:49,150 --> 00:12:53,790
So we have education, of defensive programming.

108
00:12:53,790 --> 00:13:05,890
We also have various tools that we use and the vendors which that we use for these various tools are listed on the right side, along with some of the security researchers we engage with.

109
00:13:06,170 --> 00:13:09,430
So as I said, we have various security tools we use.

110
00:13:09,430 --> 00:13:18,930
There's static analysis, software component analysis, which actually helps us find cross-site scripting vulnerabilities in components that we consume, open source products.

111
00:13:19,090 --> 00:13:34,290
We have various security headers that we encourage use of, as well as the external reviews and incident response, which was very key in how we handle this issue that came to us from Otario.

112
00:13:34,430 --> 00:13:42,810
Now in looking at those processes, we defined there were some areas we thought we could improve or add on to.

113
00:13:42,850 --> 00:13:47,050
So the first one is an internal security tech talk.

114
00:13:47,050 --> 00:13:49,930
We do tech talks weekly at OSIsoft.

115
00:13:49,930 --> 00:14:03,310
Every third week they are focused on security and we did a tech talk specifically on this incident so we could share the lessons learned from this with the other of the 40 teams at OSIsoft.

116
00:14:03,310 --> 00:14:13,430
We also looked at how we use the static application security testing tool and identified some process improvements for our use of that tool.

117
00:14:13,470 --> 00:14:21,010
And the last is we decided to get much more aggressive on the adoption of content security policy at OSIsoft.

118
00:14:23,310 --> 00:14:40,910
Now I did say that we do measure our best practices and I just wanted to give you a sense of what some of the measurements are that we use associated with just our cross-site scripting defenses at OSIsoft in the best practice areas I defined.

119
00:14:42,950 --> 00:14:50,830
Now we didn't do any process improvements on our DAST tool and I'm going to talk briefly about that.

120
00:14:50,830 --> 00:15:00,130
Now the DAST tool that we use is from Qualys and for those of you who don't remember what a DAST tool is, it's basically it's run as a server.

121
00:15:00,130 --> 00:15:20,170
Your application is up and running and then the DAST tool will send dynamic tests at your web server, typically OWASP top 10 attacks, and see whether or not your web server has the appropriate implementations in place to protect against it.

122
00:15:20,170 --> 00:15:29,090
Now in this particular cross-site scripting issue, as Dor explained, it was a stored cross-site scripting attack.

123
00:15:29,090 --> 00:15:40,150
But in looking at the tools, it's more of a DOM cross-site scripting vulnerability with insecure use of innerHTML.

124
00:15:40,330 --> 00:15:56,950
Now our DAST tool, Qualys, is very good when the source of the innerHTML parsing, the bad package, is from a document URL, but it doesn't really handle well when the source is from an HTML document.

125
00:15:56,950 --> 00:15:59,510
So we decided not to focus there.

126
00:16:02,360 --> 00:16:13,760
Now our static application security testing tool is from Synopsys and if you look at the bottom right as a reminder for what SAS tools do, they analyze your code.

127
00:16:13,760 --> 00:16:24,820
So they're looking directly at your code, look at call stacks, how functions are reached, and identify insecure call paths or insecure calling sequences.

128
00:16:25,120 --> 00:16:27,640
So we felt that was the best tool.

129
00:16:27,640 --> 00:16:34,520
It should identify where innerHTML is used and it should be able to flag insecure uses of it.

130
00:16:34,520 --> 00:16:36,760
So what went wrong with this?

131
00:16:36,760 --> 00:16:43,280
We identified that there was a tool compatibility problem with the NuGet package that we used.

132
00:16:43,280 --> 00:16:50,080
It's a Razor engine and our tool Synopsys did not work well with it.

133
00:16:50,080 --> 00:16:56,660
It does work very well with the Razor analysis built into ASP.NET.

134
00:16:56,820 --> 00:17:04,980
So consequent to this problem, our scanning for this particular project missed all JavaScript and CSHTML.

135
00:17:05,120 --> 00:17:18,460
So the process improvements we identified is we need to be able to do some central evaluation of the SAS configurations and we're working with Synopsys on how to best move that idea forward.

136
00:17:18,460 --> 00:17:28,820
The other thing we're going to do is for this one particular development team, we're going to move them to using the ASP.NET MVC Razor engine instead.

137
00:17:28,900 --> 00:17:35,640
And then we're also considering increasing the sensitivity to completely disallow the use of innerHTML.

138
00:17:35,640 --> 00:17:42,280
We're not sure exactly how this is going to play out, but innerHTML is definitely not something we want to use.

139
00:17:42,280 --> 00:17:48,120
So we're either going to be heavily scrutinizing the use of innerHTML or completely disallow it.

140
00:17:48,120 --> 00:17:50,020
So we're still working that one out.

141
00:17:50,140 --> 00:17:51,280
Next slide please.

142
00:17:51,720 --> 00:17:55,840
Now I'm going to switch context and talk a little bit about content security policy.

143
00:17:55,840 --> 00:18:00,620
And just to give you a why behind that, I'm focusing on jQuery.

144
00:18:00,620 --> 00:18:07,460
Now over the last couple of years, jQuery has released several different vulnerabilities.

145
00:18:08,340 --> 00:18:12,440
These all on this chart are the cross-site scripting vulnerability.

146
00:18:12,440 --> 00:18:23,860
And as you can see with the overlay that was just added, they all fall in the high range for CVSS scoring, which means you have to take these particular vulnerabilities very seriously.

147
00:18:23,860 --> 00:18:31,160
Cross-site scripting in general does tend to fall into the high vulnerability area.

148
00:18:31,160 --> 00:18:45,960
And dealing with all of these different patch updates from this open source does present challenges to vendors which consume open source and components under the scrutiny of security research.

149
00:18:46,280 --> 00:18:47,140
Next please.

150
00:18:48,240 --> 00:18:56,800
Now content security policy, just to explain how it works, the web server will send down HTML and JavaScript content to your browser.

151
00:18:56,800 --> 00:19:09,660
And now your browser is theoretically susceptible to some sort of JavaScript injection via a cross-site scripting payload which has been architected by the attacker.

152
00:19:09,660 --> 00:19:21,720
Now by adding content security policy from the same web server, you can now inform the browser of the allowed list, where JavaScript is allowed to come from.

153
00:19:21,720 --> 00:19:29,100
So now the browser has the ability to identify that, hey, this cross-site scripting payload is not on my allowed list.

154
00:19:29,100 --> 00:19:31,400
I'm not going to allow it to execute.

155
00:19:31,900 --> 00:19:42,140
And I also added the CDN, Content Delivery Network, so just to let you know that the content security policy's allow list does incorporate more than just the web server that it came from.

156
00:19:42,140 --> 00:19:44,540
So there's there's lots of flexibility there.

157
00:19:44,620 --> 00:19:45,680
Next slide please.

158
00:19:46,450 --> 00:19:56,220
Now we strongly believe that content security policy has reached the point where it is extremely extremely mature.

159
00:19:56,360 --> 00:20:01,100
The major browsers all have implementations for it.

160
00:20:01,100 --> 00:20:11,700
So we think that the industry is in a good place where we can start demanding the use of content security policy in the products that we consume.

161
00:20:12,180 --> 00:20:15,900
Now a question I often get on feedback, i.e.

162
00:20:16,220 --> 00:20:17,520
Internet Explorer.

163
00:20:20,780 --> 00:20:23,700
The Internet Explorer, I just want to talk about that one briefly.

164
00:20:23,700 --> 00:20:29,040
Internet Explorer, I get this question a lot, you know, it doesn't support a content security policy.

165
00:20:29,040 --> 00:20:30,260
Is that a problem?

166
00:20:30,280 --> 00:20:34,400
And this will not stop Internet Explorer from working.

167
00:20:34,820 --> 00:20:39,060
So you don't have to tell customers they can't use Internet Explorer anymore.

168
00:20:39,060 --> 00:20:50,140
But what we are taking the position of is that we want to inform our customers that you should be using modern browsers for their security benefit.

169
00:20:50,140 --> 00:20:56,850
One issue and one protection is content security policy.

170
00:20:57,010 --> 00:21:02,530
Now I talked about having measurements for our best practices.

171
00:21:02,530 --> 00:21:05,590
And this is one of the ways that we do it.

172
00:21:05,590 --> 00:21:12,990
We've identified poor, better, and best practices for our development teams and came up with ways for them to measure their adoption against that.

173
00:21:12,990 --> 00:21:17,730
So the poor practice would be using unsafe inline and unsafe eval.

174
00:21:17,730 --> 00:21:26,130
The next better practice would be if you do have inline JavaScript to provide nonces or hashes for it.

175
00:21:26,130 --> 00:21:50,530
But the best practice is to remove all those inline JavaScript and eval statements and give the browser an endpoint to report any occurrences of defenses or blocks that it had to execute to a central reporting location so that those issues can be investigated and fixed.

176
00:21:51,070 --> 00:22:02,430
And just to give you a general sense of how we do this measurement within OSIsoft, all the measurements we've identified from the best practices have response options.

177
00:22:02,430 --> 00:22:03,730
They're all given scores.

178
00:22:03,730 --> 00:22:07,570
So you get a score, a scale from zero to ten for all of them.

179
00:22:07,570 --> 00:22:14,650
And they can all be combined and you can look at collectively how teams are doing it in response to adoption of these best practices.

180
00:22:14,770 --> 00:22:29,750
But in addition to this, we decided to not only revamp the responses in order to encourage better adoption, but we decided also to take the step of making this required for releases going out by the end of this year.

181
00:22:30,090 --> 00:22:30,610
Excellent.

182
00:22:32,470 --> 00:22:37,070
Now, focusing on the key takeaways, the things that that I covered.

183
00:22:37,070 --> 00:22:38,950
The first one was security tools.

184
00:22:38,950 --> 00:22:40,570
Using security tools is not enough.

185
00:22:40,570 --> 00:22:46,230
You really have to evaluate how you're using them, in our case, for better code coverage.

186
00:22:46,930 --> 00:22:49,110
The next one is content security policy.

187
00:22:49,110 --> 00:22:59,550
We believe the industry is mature enough that content security policy is in a place that this should become an ICS standard practice, we believe.

188
00:22:59,550 --> 00:23:03,030
It gives you a strong second level of defense.

189
00:23:03,030 --> 00:23:17,650
With a lot of open source components out there, most of us consume open source, and when the vulnerabilities are discovered in those open source, the public disclosure happens to the public at the same time it happens to us.

190
00:23:17,650 --> 00:23:21,010
We know about the vulnerability at the same time everyone else does.

191
00:23:21,010 --> 00:23:35,290
So there is limited time that we have to patch and get the protections in place, but by having that second level of defense, content security policy there, it gives us more time to get the patches in place correctly.

192
00:23:35,670 --> 00:23:49,510
And the last thing, engaging with companies like Atario and Door is very, very successful in moving security forward for our products and it's been great experience working with them for us.

193
00:23:49,510 --> 00:23:50,530
Thank you.

194
00:23:50,530 --> 00:23:51,190
Thank you.

195
00:23:51,190 --> 00:23:52,830
Thank you very much, Mike.

196
00:23:52,850 --> 00:23:59,390
The actual effort was really cool and we very enjoyed this research.