1
00:00:01,020 --> 00:00:07,090
Hello and welcome to 10,000 Dependencies Under the Sea, Exploring and Securing Open Source Dependencies.

2
00:00:07,380 --> 00:00:08,830
My name is Greg Horton.

3
00:00:08,830 --> 00:00:11,210
I'm a Product Security Engineer at Slack.

4
00:00:11,210 --> 00:00:17,050
And I'm Ryan Slama, an Associate Software Engineer on the Product Security Foundations team.

5
00:00:17,690 --> 00:00:21,490
At Slack, AppSec is product security.

6
00:00:21,490 --> 00:00:26,010
Our product organization is split into Classics and Foundations.

7
00:00:26,010 --> 00:00:36,890
Classics were the original team, and they focus on most of the traditional AppSec responsibilities, like reviewing new features, penetration testing, and running a healthy bug bounty program.

8
00:00:37,270 --> 00:00:45,250
I'm on the Foundations team, which focuses on reducing risk through automated tooling and creating secure by default libraries and patterns.

9
00:00:45,410 --> 00:00:52,070
Both of these teams work together to ensure the security of Slack, the product, using a multifaceted approach.

10
00:00:53,330 --> 00:00:56,070
For some context, what is Slack?

11
00:00:56,070 --> 00:00:58,890
Slack is a channel-based messaging platform.

12
00:00:58,890 --> 00:01:10,470
And with Slack, people can work together more effectively, connect all their tools and services, and find the information they need to do their best work, all within a secure, enterprise-grade environment.

13
00:01:11,070 --> 00:01:14,750
Let's begin by talking a bit about the Slack stack.

14
00:01:15,270 --> 00:01:17,690
Users expect Slack to work everywhere.

15
00:01:17,690 --> 00:01:24,670
To that end, we use and are a partial maintainer of the Electron project for a consistent experience across devices.

16
00:01:25,010 --> 00:01:30,350
In practice, this means our front-end web code also runs in all of our desktop clients.

17
00:01:30,590 --> 00:01:37,190
Our back-end is primarily in Hacklang, Facebook's fork of PHP, with strong typing and other enhancements.

18
00:01:37,350 --> 00:01:41,410
We also have some services written in Go, like our caching layer.

19
00:01:41,470 --> 00:01:47,610
Finally, our mobile apps are primarily in Swift and Kotlin, but we won't be talking about them much today.

20
00:01:49,070 --> 00:01:55,030
Today, we'll be talking about an OWASP top 10 issue, using components with known vulnerabilities.

21
00:01:55,030 --> 00:02:00,470
Specifically, we'll be focusing on vulnerability management for third-party dependencies.

22
00:02:01,850 --> 00:02:05,410
Our story today begins with an intern project.

23
00:02:05,470 --> 00:02:12,650
Matt Dwanzig and I, who are now full-time engineers at Slack, were interning on the Product Security Foundations team last summer.

24
00:02:12,810 --> 00:02:18,610
When we got there, we were given an open-ended project of understanding and limiting our dependency risk.

25
00:02:18,610 --> 00:02:30,930
Today, we will walk you through our journey building a tool, and Greg will walk you through how we implemented the tool and built a process around it to actually limit the risk at Slack.

26
00:02:32,170 --> 00:02:36,370
Modern codebases often require tons of third-party code.

27
00:02:36,490 --> 00:02:42,950
At Slack, we have one main repository that contains our entire front-end and most of our back-end.

28
00:02:43,250 --> 00:02:48,430
Our main repository currently requires over 6,500 packages.

29
00:02:48,450 --> 00:02:50,610
A year ago, we had half that.

30
00:02:50,810 --> 00:02:55,150
This trend would be concerning if we didn't have systems in place to limit risk.

31
00:02:55,270 --> 00:03:00,470
All of our first-party code has to be reviewed, but what about random stuff you find on GitHub?

32
00:03:00,470 --> 00:03:01,710
That's exempt.

33
00:03:01,930 --> 00:03:05,230
Clearly, some process must be in place to manage risk.

34
00:03:07,050 --> 00:03:14,510
It's important to note that we have a mature process for adding packages, and the count still doubled in a year.

35
00:03:14,890 --> 00:03:19,350
For PRs that add packages, developers have to explain why it's needed.

36
00:03:20,070 --> 00:03:26,790
Packages must be actively maintained and save meaningful engineering effort over just building the functionality ourselves.

37
00:03:27,230 --> 00:03:34,610
Additionally, all packages are assigned a team or a directly responsible individual to update them and maintain them.

38
00:03:34,610 --> 00:03:38,210
This is especially important for security updates and fixes.

39
00:03:38,870 --> 00:03:41,890
So how did our package count double?

40
00:03:42,550 --> 00:03:44,830
In a word, NPM.

41
00:03:44,830 --> 00:03:49,990
We only directly require 350 packages in our package JSON.

42
00:03:50,010 --> 00:03:58,610
However, when you resolve the nested dependencies, our dependency tree expands to almost 6,500 unique versions of packages.

43
00:03:59,970 --> 00:04:03,690
Running this much third-party code can be a risk.

44
00:04:03,690 --> 00:04:10,670
The downside of using common software is that we often find out about vulnerabilities at the same time as everyone else.

45
00:04:10,670 --> 00:04:13,810
Let's take a look at some examples of related issues.

46
00:04:14,890 --> 00:04:17,990
One of the most notable was Equifax.

47
00:04:17,990 --> 00:04:24,690
Equifax got hacked because they were running an old version of Apache struts that had known vulnerabilities.

48
00:04:24,850 --> 00:04:29,910
This hack leaked 143 million social security numbers.

49
00:04:30,050 --> 00:04:37,830
Preventing this could have been as simple as upgrading their version of Apache struts to one that did not have publicly known vulnerabilities.

50
00:04:40,150 --> 00:04:45,330
Vulnerabilities are also discovered in old versions of popular NPM packages.

51
00:04:45,690 --> 00:04:49,230
For example, both of these advisories were issued this year.

52
00:04:49,410 --> 00:04:54,150
Next.js and AngularJS are respected, well-maintained packages.

53
00:04:54,150 --> 00:05:08,810
But running those in your production codebases without adequately maintaining them is a recipe for disaster because new vulnerabilities can be discovered at any time no matter how many years you've been running the same code.

54
00:05:09,770 --> 00:05:14,690
NPM also had issues with malicious code in popular packages.

55
00:05:14,690 --> 00:05:24,610
For example, EventStream, which was a popular NPM library, required a dependency that had been compromised.

56
00:05:24,610 --> 00:05:29,370
And that compromised dependency included a targeted Bitcoin wallet stealer.

57
00:05:29,370 --> 00:05:33,850
And that malicious code was downloaded over 8 million times.

58
00:05:35,570 --> 00:05:41,690
ESLint Scope is an even more popular NPM package used for linting JavaScript code.

59
00:05:41,690 --> 00:05:49,050
And a compromised version was uploaded that exfiltrated NPM RC files, which contained package publishing credentials.

60
00:05:49,050 --> 00:05:56,430
Those credentials could allow an attacker to push a new version of a package as a semver patch with malicious code.

61
00:05:56,430 --> 00:06:03,830
And because of the way NPM's semver works, all installations of that package would automatically upgrade to the malicious code.

62
00:06:05,830 --> 00:06:09,990
And here's for some figures that might be slightly biased.

63
00:06:09,990 --> 00:06:15,470
Snyk found an 88% increase in library vulnerabilities over the past two years.

64
00:06:15,470 --> 00:06:20,950
And 78% of vulnerabilities that they found are in indirect dependencies.

65
00:06:21,050 --> 00:06:25,210
Just take a look at our dependency tree from earlier to understand why.

66
00:06:25,210 --> 00:06:34,790
When we only require 345, but end up with almost 6,500 packages, there's a lot more code beneath the surface than on top of it.

67
00:06:35,750 --> 00:06:38,350
And we're not the only ones who have this problem.

68
00:06:38,490 --> 00:06:41,490
Our customers know about this problem and they ask us about it.

69
00:06:41,490 --> 00:06:47,550
We need a good program in place so we can show them that we take care of their data and we limit risk.

70
00:06:48,430 --> 00:06:50,650
What do we do about this problem now?

71
00:06:51,410 --> 00:06:54,970
When we started, we defined three goals for a good solution.

72
00:06:55,070 --> 00:07:03,130
The first is we want to detect vulnerabilities as soon as they're publicly available because we want to be on top of the fixes as quickly as possible.

73
00:07:03,710 --> 00:07:08,090
Second, we want to track any vulnerabilities or weaknesses in our code base.

74
00:07:08,090 --> 00:07:14,770
We want to be able to have insight into where we have risk and so we can better fix it.

75
00:07:14,950 --> 00:07:17,850
Finally, we want to alert when we have a problem.

76
00:07:17,870 --> 00:07:23,270
No developer should ever have to think, oh, I should scan my repo before I deployed a production.

77
00:07:23,670 --> 00:07:28,670
We want to alert when we have vulnerabilities so this just happens automatically.

78
00:07:29,490 --> 00:07:33,130
The next question we asked was, can we use an off-the-shelf tool?

79
00:07:33,150 --> 00:07:37,070
But our requirements ended up making that surprisingly difficult.

80
00:07:37,710 --> 00:07:45,810
First of all, we use Hacklang, which is not a popular language outside Facebook, so it was difficult to find vendors that supported it.

81
00:07:45,870 --> 00:08:01,090
Second, we use GitHub Enterprise rather than GitHub Cloud, which makes it harder for Dependabot because Dependabot still isn't out for GitHub Enterprise and it was even further from being out a year ago when we began this journey.

82
00:08:02,170 --> 00:08:06,610
Next, we needed a tool that will scan the entire dependency trees.

83
00:08:06,930 --> 00:08:11,170
Knowing about vulnerabilities and packages we require directly isn't enough.

84
00:08:11,170 --> 00:08:19,450
As we saw earlier, there are so many more packages beneath the surface and we need to make sure that those are a stable foundation that we can build our app on.

85
00:08:19,690 --> 00:08:24,850
And finally, at Slack, we're heavy users of Slack as a product.

86
00:08:24,850 --> 00:08:28,830
All of our alerts and things like that just go in Slack.

87
00:08:28,830 --> 00:08:35,910
So we need a tool that has support for routing alerts for different code bases or packages to different teams.

88
00:08:36,370 --> 00:08:41,350
And finally, you might be saying, oh, we're a vendor and we have a tool that does all that.

89
00:08:41,390 --> 00:08:49,030
But we're still a relatively small company and we need something that's not going to cost us millions of dollars over the next few years.

90
00:08:50,750 --> 00:08:52,690
So we built Ossify.

91
00:08:53,830 --> 00:08:58,710
Our solutions to the three points mentioned earlier were first, the in-detection.

92
00:08:58,710 --> 00:09:06,610
We run daily scans of our code bases to both figure out what packages we're requiring.

93
00:09:06,610 --> 00:09:17,430
And also, we upload those packages to the Sonatype open source vulnerability index and see if there are any new vulnerabilities that have been reported for our packages.

94
00:09:17,770 --> 00:09:26,350
Second, we built a dashboard to track the status of repositories, as well as tracking remediation efforts for individual findings.

95
00:09:26,350 --> 00:09:32,330
And finally, we built robust Slack alerting, which will be covered more later.

96
00:09:35,010 --> 00:09:37,830
Ossify supports three package ecosystems.

97
00:09:37,830 --> 00:09:44,330
The first, of course, is Hackline, because that's our backend and that's where some of the most dangerous possibilities are.

98
00:09:44,790 --> 00:09:57,510
We added custom metadata to track composer packages upstream because we had to manually fork and vendor some of these packages to add strong types and other Hackline features.

99
00:09:57,910 --> 00:10:04,790
Second, we added NPM, because NPM is the biggest offender for vast amounts of packages.

100
00:10:04,790 --> 00:10:09,530
And we found, by far, the most findings coming from NPM packages.

101
00:10:09,890 --> 00:10:16,910
And finally, we added Go support because a number of our high value services, like our caching layer, are written in Go.

102
00:10:18,530 --> 00:10:21,490
Here is what the Ossify dashboard looks like.

103
00:10:21,490 --> 00:10:28,530
This is an example repository created for this demo just to show you what it looks like when you scan a repository.

104
00:10:28,730 --> 00:10:34,650
So we support scanning multiple branches, which could be used potentially for future CI integration.

105
00:10:35,050 --> 00:10:37,390
Here we have an example finding page.

106
00:10:37,390 --> 00:10:43,630
This is for NodeForge, which was being pulled in in our sample repo's package JSON.

107
00:10:43,930 --> 00:10:47,790
NodeForge has a weakness, but we don't actually require NodeForge.

108
00:10:47,790 --> 00:10:52,010
Instead, we require a Google Auth library all the way on the left.

109
00:10:52,010 --> 00:11:02,390
We built out a dependency graph tool because we want our developers to understand where the vulnerability is coming from and what packages need upgraded or removed to fix the issue.

110
00:11:04,290 --> 00:11:07,070
Here's a more complicated dependency graph.

111
00:11:07,470 --> 00:11:10,210
This is from a dev dependency.

112
00:11:10,490 --> 00:11:13,510
The weakness was on the right in isURL.

113
00:11:13,510 --> 00:11:23,950
However, if you trace back all the way to the left, it was actually some Gulp plugins we were using as part of a build process that were pulling in the weak version of isURL.

114
00:11:24,810 --> 00:11:28,810
And finally, like any good security tool, it comes with dark mode.

115
00:11:28,810 --> 00:11:34,550
Now I'm going to hand things off to Greg to cover everything that happened after the original development of the tool.

116
00:11:38,070 --> 00:11:40,930
Thanks, Ryan, for that great overview of the tool.

117
00:11:41,110 --> 00:11:48,690
So now that we have this fully featured tool that did everything that we wanted, it was time to integrate it into our wider processes here at Slack.

118
00:11:49,170 --> 00:11:50,970
So this would be easy, right?

119
00:11:50,970 --> 00:12:02,390
You know, we have an app that scans our repositories, looks at our third-party integrations, sees if there's any vulnerabilities, and then lets us know about them.

120
00:12:02,890 --> 00:12:11,990
So our first workflow for this tool was that Ossify would scan daily and then post to a Slack channel any currently known vulnerabilities.

121
00:12:12,650 --> 00:12:23,270
And using the power of Slack, we could make Slack notifications that were actionable, meaning that we could give the information that was needed for quick remediation.

122
00:12:23,270 --> 00:12:29,550
They were non-obtrusive, so we could set those notifications to snooze or ignore if they weren't relevant to us.

123
00:12:29,710 --> 00:12:39,590
And then make them configurable so we could notify individual users and specific channels about these dependencies.

124
00:12:40,250 --> 00:12:42,970
Here's an example of what those looked like.

125
00:12:43,010 --> 00:12:54,410
So as you can see, this dependency detection of the app, it scanned a repository, it found some vulnerabilities in some third-party libraries, and it would tell us about them.

126
00:12:54,410 --> 00:13:00,090
And as you can see, also, you could set them to ignore, you could snooze them, etc.

127
00:13:01,010 --> 00:13:05,170
But the problem with that is that this channel was way too noisy.

128
00:13:05,290 --> 00:13:15,350
Every single day, it was posting all the vulnerabilities that were found, and they haven't been remediated yet, so it would repeat a lot of the ones that we'd seen the day before.

129
00:13:15,350 --> 00:13:20,810
Maybe we didn't want to ignore them yet because they were still relevant or we were still trying to find somebody to fix them.

130
00:13:21,010 --> 00:13:28,590
But it was a very long list, and so we couldn't really work to work them down in this format.

131
00:13:30,030 --> 00:13:37,750
And also, blindly throwing them into a channel made it so it was everybody's problem, which in practice made it nobody's problem.

132
00:13:37,750 --> 00:13:48,490
Nobody took initiative to see which vulnerabilities were actually a threat to us, and nobody was taking responsibility to fix them.

133
00:13:48,490 --> 00:13:54,890
So it was pretty ineffective at solving our main problem at first, so we had to go back to the drawing board and figure that out.

134
00:13:56,170 --> 00:14:06,230
So we had to admit that Ossify, you know, was an effective tool, but it was not only reporting vulnerabilities, but it was also acting as an internal ticketing system.

135
00:14:06,810 --> 00:14:14,010
For something we wanted to maintain long term, we wanted Ossify to do one thing well, find vulnerabilities and report them.

136
00:14:14,630 --> 00:14:20,410
And to do this, we had some prior art that we could work from to figure out the best way to do that at Slack.

137
00:14:21,210 --> 00:14:23,170
And that answer was Jira.

138
00:14:24,330 --> 00:14:31,550
We have a Jira system, we have security tickets that go into our Jira that can be actioned upon by developers.

139
00:14:31,910 --> 00:14:35,650
And so our solution was to put it in Jira.

140
00:14:35,650 --> 00:14:46,170
At Slack, we have a method already for triaging security -based tickets that come from multiple sources, say if we have bug bounty reports or internal findings .

141
00:14:46,450 --> 00:14:56,930
Our SLAs, or service -level agreements, are 180 days for low tickets, 90 days for medium, 30 days for highs, and then seven days for critical findings.

142
00:14:57,690 --> 00:15:05,530
So pushing, upgrading these vulnerabilities to a ticket in Jira puts these already into the processes that our developers know.

143
00:15:05,530 --> 00:15:08,910
It doesn't add in any more friction .

144
00:15:09,170 --> 00:15:14,410
And it puts them into an established workflow that also ties to company -wide objectives.

145
00:15:14,750 --> 00:15:17,870
So upgrading these libraries just can't be another thing that developers ignore.

146
00:15:17,870 --> 00:15:20,130
They have to do it to meet their OKRs.

147
00:15:21,170 --> 00:15:27,210
So now that we had a new flow, we had a new way of dealing with and finding these vulnerable libraries.

148
00:15:28,970 --> 00:15:34,690
So our new flow is that Ossify would find the vulnerability and then throw it right into Jira.

149
00:15:34,690 --> 00:15:35,650
It would file a ticket.

150
00:15:35,650 --> 00:15:40,890
If multiple vulnerabilities were caused by the same package, we would roll those up into the same ticket.

151
00:15:40,890 --> 00:15:51,390
Say you have a couple of highs and a few mediums , but that all would be remediated by upgrading a single package to a certain version, that would be one ticket.

152
00:15:51,390 --> 00:15:54,590
So we weren't making multiple tickets for multiple vulnerabilities per se.

153
00:15:54,590 --> 00:15:57,710
Just if you upgrade this library, these are all soft.

154
00:15:59,630 --> 00:16:02,530
Now we can talk about our triage process a little bit.

155
00:16:02,570 --> 00:16:09,950
So the first step in this would be that the ticket would be filed and the product security engineer on call would triage it.

156
00:16:09,950 --> 00:16:15,410
So our triage process is just enough so that we can find how serious the vulnerability is in our systems.

157
00:16:15,590 --> 00:16:22,710
We would read the proof of concept from the database and then check our source code to see if we are in fact using those vulnerable functions.

158
00:16:22,710 --> 00:16:25,790
Because if we weren't, then it wouldn't be a critical vulnerability for us .

159
00:16:25,790 --> 00:16:30,910
It would be a low, or even maybe a non-fix.

160
00:16:32,030 --> 00:16:37,030
If we are using that, we might try to reproduce the attack.

161
00:16:37,030 --> 00:16:44,150
Or if it's used in multiple places, we're just going to determine that it's enough to determine the severity for fixing.

162
00:16:44,290 --> 00:16:48,230
If it's used in hundreds of places, we can't test every single one.

163
00:16:48,850 --> 00:16:54,810
So we would then recommend it be a medium or high depending on what we thought.

164
00:16:55,070 --> 00:17:02,830
And we're also going to review the fix that's in the upgraded version, just to see if there's anything more we need to consider when implementing the update.

165
00:17:03,330 --> 00:17:10,110
All of this is just to either confirm the severity based on the CVSS score, or adjust it as needed.

166
00:17:12,570 --> 00:17:20,710
From there, you know, we would find the responsible team.

167
00:17:20,870 --> 00:17:28,230
So after determining whether or not we get exploited, we need to find the responsible team to fix that.

168
00:17:28,230 --> 00:17:36,470
That team is then going to be responsible for mediation in the SLA time that we've already determined based on the severity.

169
00:17:37,350 --> 00:17:45,810
Lastly, once the package is updated, Ossify is going to go in and automatically see that that version is no longer used and will remove it from our list.

170
00:17:45,810 --> 00:17:48,750
And therefore, the vulnerability is remediated.

171
00:17:51,540 --> 00:17:58,740
Now that we have a working process, and that fits into what we're already doing at Slack, there's still a lot of future work to be done .

172
00:17:58,980 --> 00:18:11,180
Right now, we're working on getting a clean state for vulnerable packages , and working down our backlog to determine if the packages are currently being... that are currently being flagged are vulnerable and get them fixed.

173
00:18:12,400 --> 00:18:17,740
Next, this process, as you can tell, involves a manual work still.

174
00:18:18,080 --> 00:18:27,000
There's a lot of manual work for the product security engineer to find out who owns the vulnerable dependency and also seeing if it's exploitable.

175
00:18:27,120 --> 00:18:32,960
We would like to figure out a way to automatically figure that out instead of having that be a manual process.

176
00:18:32,960 --> 00:18:39,460
That's just going to be looking into how we can track when new libraries are added .

177
00:18:39,920 --> 00:18:45,260
And thirdly, we'd love to integrate this process into our CI pipeline .

178
00:18:45,640 --> 00:18:52,280
So packages that are vulnerable aren't uploaded to our code base when they're committed.

179
00:18:52,280 --> 00:19:01,760
So on commit, it would run the scanner, look for any open vulnerabilities, and then either block that or let it through if there aren't any.

180
00:19:03,360 --> 00:19:04,800
Thank you so much.

181
00:19:05,060 --> 00:19:11,300
Special thanks to Matt, who did ossify development for us, and Nikki and Oliver for a previous version of this talk .

182
00:19:11,300 --> 00:19:14,640
If you have any questions , we'll be in the Discord chat to answer them .

183
00:19:14,640 --> 00:19:15,180
Thank you.