WEBVTT 00:01.020 --> 00:07.090 Hello and welcome to 10,000 Dependencies Under the Sea, Exploring and Securing Open Source Dependencies. 00:07.380 --> 00:08.830 My name is Greg Horton. 00:08.830 --> 00:11.210 I'm a Product Security Engineer at Slack. 00:11.210 --> 00:17.050 And I'm Ryan Slama, an Associate Software Engineer on the Product Security Foundations team. 00:17.690 --> 00:21.490 At Slack, AppSec is product security. 00:21.490 --> 00:26.010 Our product organization is split into Classics and Foundations. 00:26.010 --> 00:36.890 Classics were the original team, and they focus on most of the traditional AppSec responsibilities, like reviewing new features, penetration testing, and running a healthy bug bounty program. 00:37.270 --> 00:45.250 I'm on the Foundations team, which focuses on reducing risk through automated tooling and creating secure by default libraries and patterns. 00:45.410 --> 00:52.070 Both of these teams work together to ensure the security of Slack, the product, using a multifaceted approach. 00:53.330 --> 00:56.070 For some context, what is Slack? 00:56.070 --> 00:58.890 Slack is a channel-based messaging platform. 00:58.890 --> 01:10.470 And with Slack, people can work together more effectively, connect all their tools and services, and find the information they need to do their best work, all within a secure, enterprise-grade environment. 01:11.070 --> 01:14.750 Let's begin by talking a bit about the Slack stack. 01:15.270 --> 01:17.690 Users expect Slack to work everywhere. 01:17.690 --> 01:24.670 To that end, we use and are a partial maintainer of the Electron project for a consistent experience across devices. 01:25.010 --> 01:30.350 In practice, this means our front-end web code also runs in all of our desktop clients. 01:30.590 --> 01:37.190 Our back-end is primarily in Hacklang, Facebook's fork of PHP, with strong typing and other enhancements. 01:37.350 --> 01:41.410 We also have some services written in Go, like our caching layer. 01:41.470 --> 01:47.610 Finally, our mobile apps are primarily in Swift and Kotlin, but we won't be talking about them much today. 01:49.070 --> 01:55.030 Today, we'll be talking about an OWASP top 10 issue, using components with known vulnerabilities. 01:55.030 --> 02:00.470 Specifically, we'll be focusing on vulnerability management for third-party dependencies. 02:01.850 --> 02:05.410 Our story today begins with an intern project. 02:05.470 --> 02:12.650 Matt Dwanzig and I, who are now full-time engineers at Slack, were interning on the Product Security Foundations team last summer. 02:12.810 --> 02:18.610 When we got there, we were given an open-ended project of understanding and limiting our dependency risk. 02:18.610 --> 02:30.930 Today, we will walk you through our journey building a tool, and Greg will walk you through how we implemented the tool and built a process around it to actually limit the risk at Slack. 02:32.170 --> 02:36.370 Modern codebases often require tons of third-party code. 02:36.490 --> 02:42.950 At Slack, we have one main repository that contains our entire front-end and most of our back-end. 02:43.250 --> 02:48.430 Our main repository currently requires over 6,500 packages. 02:48.450 --> 02:50.610 A year ago, we had half that. 02:50.810 --> 02:55.150 This trend would be concerning if we didn't have systems in place to limit risk. 02:55.270 --> 03:00.470 All of our first-party code has to be reviewed, but what about random stuff you find on GitHub? 03:00.470 --> 03:01.710 That's exempt. 03:01.930 --> 03:05.230 Clearly, some process must be in place to manage risk. 03:07.050 --> 03:14.510 It's important to note that we have a mature process for adding packages, and the count still doubled in a year. 03:14.890 --> 03:19.350 For PRs that add packages, developers have to explain why it's needed. 03:20.070 --> 03:26.790 Packages must be actively maintained and save meaningful engineering effort over just building the functionality ourselves. 03:27.230 --> 03:34.610 Additionally, all packages are assigned a team or a directly responsible individual to update them and maintain them. 03:34.610 --> 03:38.210 This is especially important for security updates and fixes. 03:38.870 --> 03:41.890 So how did our package count double? 03:42.550 --> 03:44.830 In a word, NPM. 03:44.830 --> 03:49.990 We only directly require 350 packages in our package JSON. 03:50.010 --> 03:58.610 However, when you resolve the nested dependencies, our dependency tree expands to almost 6,500 unique versions of packages. 03:59.970 --> 04:03.690 Running this much third-party code can be a risk. 04:03.690 --> 04:10.670 The downside of using common software is that we often find out about vulnerabilities at the same time as everyone else. 04:10.670 --> 04:13.810 Let's take a look at some examples of related issues. 04:14.890 --> 04:17.990 One of the most notable was Equifax. 04:17.990 --> 04:24.690 Equifax got hacked because they were running an old version of Apache struts that had known vulnerabilities. 04:24.850 --> 04:29.910 This hack leaked 143 million social security numbers. 04:30.050 --> 04:37.830 Preventing this could have been as simple as upgrading their version of Apache struts to one that did not have publicly known vulnerabilities. 04:40.150 --> 04:45.330 Vulnerabilities are also discovered in old versions of popular NPM packages. 04:45.690 --> 04:49.230 For example, both of these advisories were issued this year. 04:49.410 --> 04:54.150 Next.js and AngularJS are respected, well-maintained packages. 04:54.150 --> 05:08.810 But running those in your production codebases without adequately maintaining them is a recipe for disaster because new vulnerabilities can be discovered at any time no matter how many years you've been running the same code. 05:09.770 --> 05:14.690 NPM also had issues with malicious code in popular packages. 05:14.690 --> 05:24.610 For example, EventStream, which was a popular NPM library, required a dependency that had been compromised. 05:24.610 --> 05:29.370 And that compromised dependency included a targeted Bitcoin wallet stealer. 05:29.370 --> 05:33.850 And that malicious code was downloaded over 8 million times. 05:35.570 --> 05:41.690 ESLint Scope is an even more popular NPM package used for linting JavaScript code. 05:41.690 --> 05:49.050 And a compromised version was uploaded that exfiltrated NPM RC files, which contained package publishing credentials. 05:49.050 --> 05:56.430 Those credentials could allow an attacker to push a new version of a package as a semver patch with malicious code. 05:56.430 --> 06:03.830 And because of the way NPM's semver works, all installations of that package would automatically upgrade to the malicious code. 06:05.830 --> 06:09.990 And here's for some figures that might be slightly biased. 06:09.990 --> 06:15.470 Snyk found an 88% increase in library vulnerabilities over the past two years. 06:15.470 --> 06:20.950 And 78% of vulnerabilities that they found are in indirect dependencies. 06:21.050 --> 06:25.210 Just take a look at our dependency tree from earlier to understand why. 06:25.210 --> 06:34.790 When we only require 345, but end up with almost 6,500 packages, there's a lot more code beneath the surface than on top of it. 06:35.750 --> 06:38.350 And we're not the only ones who have this problem. 06:38.490 --> 06:41.490 Our customers know about this problem and they ask us about it. 06:41.490 --> 06:47.550 We need a good program in place so we can show them that we take care of their data and we limit risk. 06:48.430 --> 06:50.650 What do we do about this problem now? 06:51.410 --> 06:54.970 When we started, we defined three goals for a good solution. 06:55.070 --> 07:03.130 The first is we want to detect vulnerabilities as soon as they're publicly available because we want to be on top of the fixes as quickly as possible. 07:03.710 --> 07:08.090 Second, we want to track any vulnerabilities or weaknesses in our code base. 07:08.090 --> 07:14.770 We want to be able to have insight into where we have risk and so we can better fix it. 07:14.950 --> 07:17.850 Finally, we want to alert when we have a problem. 07:17.870 --> 07:23.270 No developer should ever have to think, oh, I should scan my repo before I deployed a production. 07:23.670 --> 07:28.670 We want to alert when we have vulnerabilities so this just happens automatically. 07:29.490 --> 07:33.130 The next question we asked was, can we use an off-the-shelf tool? 07:33.150 --> 07:37.070 But our requirements ended up making that surprisingly difficult. 07:37.710 --> 07:45.810 First of all, we use Hacklang, which is not a popular language outside Facebook, so it was difficult to find vendors that supported it. 07:45.870 --> 08:01.090 Second, we use GitHub Enterprise rather than GitHub Cloud, which makes it harder for Dependabot because Dependabot still isn't out for GitHub Enterprise and it was even further from being out a year ago when we began this journey. 08:02.170 --> 08:06.610 Next, we needed a tool that will scan the entire dependency trees. 08:06.930 --> 08:11.170 Knowing about vulnerabilities and packages we require directly isn't enough. 08:11.170 --> 08:19.450 As we saw earlier, there are so many more packages beneath the surface and we need to make sure that those are a stable foundation that we can build our app on. 08:19.690 --> 08:24.850 And finally, at Slack, we're heavy users of Slack as a product. 08:24.850 --> 08:28.830 All of our alerts and things like that just go in Slack. 08:28.830 --> 08:35.910 So we need a tool that has support for routing alerts for different code bases or packages to different teams. 08:36.370 --> 08:41.350 And finally, you might be saying, oh, we're a vendor and we have a tool that does all that. 08:41.390 --> 08:49.030 But we're still a relatively small company and we need something that's not going to cost us millions of dollars over the next few years. 08:50.750 --> 08:52.690 So we built Ossify. 08:53.830 --> 08:58.710 Our solutions to the three points mentioned earlier were first, the in-detection. 08:58.710 --> 09:06.610 We run daily scans of our code bases to both figure out what packages we're requiring. 09:06.610 --> 09:17.430 And also, we upload those packages to the Sonatype open source vulnerability index and see if there are any new vulnerabilities that have been reported for our packages. 09:17.770 --> 09:26.350 Second, we built a dashboard to track the status of repositories, as well as tracking remediation efforts for individual findings. 09:26.350 --> 09:32.330 And finally, we built robust Slack alerting, which will be covered more later. 09:35.010 --> 09:37.830 Ossify supports three package ecosystems. 09:37.830 --> 09:44.330 The first, of course, is Hackline, because that's our backend and that's where some of the most dangerous possibilities are. 09:44.790 --> 09:57.510 We added custom metadata to track composer packages upstream because we had to manually fork and vendor some of these packages to add strong types and other Hackline features. 09:57.910 --> 10:04.790 Second, we added NPM, because NPM is the biggest offender for vast amounts of packages. 10:04.790 --> 10:09.530 And we found, by far, the most findings coming from NPM packages. 10:09.890 --> 10:16.910 And finally, we added Go support because a number of our high value services, like our caching layer, are written in Go. 10:18.530 --> 10:21.490 Here is what the Ossify dashboard looks like. 10:21.490 --> 10:28.530 This is an example repository created for this demo just to show you what it looks like when you scan a repository. 10:28.730 --> 10:34.650 So we support scanning multiple branches, which could be used potentially for future CI integration. 10:35.050 --> 10:37.390 Here we have an example finding page. 10:37.390 --> 10:43.630 This is for NodeForge, which was being pulled in in our sample repo's package JSON. 10:43.930 --> 10:47.790 NodeForge has a weakness, but we don't actually require NodeForge. 10:47.790 --> 10:52.010 Instead, we require a Google Auth library all the way on the left. 10:52.010 --> 11:02.390 We built out a dependency graph tool because we want our developers to understand where the vulnerability is coming from and what packages need upgraded or removed to fix the issue. 11:04.290 --> 11:07.070 Here's a more complicated dependency graph. 11:07.470 --> 11:10.210 This is from a dev dependency. 11:10.490 --> 11:13.510 The weakness was on the right in isURL. 11:13.510 --> 11:23.950 However, if you trace back all the way to the left, it was actually some Gulp plugins we were using as part of a build process that were pulling in the weak version of isURL. 11:24.810 --> 11:28.810 And finally, like any good security tool, it comes with dark mode. 11:28.810 --> 11:34.550 Now I'm going to hand things off to Greg to cover everything that happened after the original development of the tool. 11:38.070 --> 11:40.930 Thanks, Ryan, for that great overview of the tool. 11:41.110 --> 11:48.690 So now that we have this fully featured tool that did everything that we wanted, it was time to integrate it into our wider processes here at Slack. 11:49.170 --> 11:50.970 So this would be easy, right? 11:50.970 --> 12:02.390 You know, we have an app that scans our repositories, looks at our third-party integrations, sees if there's any vulnerabilities, and then lets us know about them. 12:02.890 --> 12:11.990 So our first workflow for this tool was that Ossify would scan daily and then post to a Slack channel any currently known vulnerabilities. 12:12.650 --> 12:23.270 And using the power of Slack, we could make Slack notifications that were actionable, meaning that we could give the information that was needed for quick remediation. 12:23.270 --> 12:29.550 They were non-obtrusive, so we could set those notifications to snooze or ignore if they weren't relevant to us. 12:29.710 --> 12:39.590 And then make them configurable so we could notify individual users and specific channels about these dependencies. 12:40.250 --> 12:42.970 Here's an example of what those looked like. 12:43.010 --> 12:54.410 So as you can see, this dependency detection of the app, it scanned a repository, it found some vulnerabilities in some third-party libraries, and it would tell us about them. 12:54.410 --> 13:00.090 And as you can see, also, you could set them to ignore, you could snooze them, etc. 13:01.010 --> 13:05.170 But the problem with that is that this channel was way too noisy. 13:05.290 --> 13:15.350 Every single day, it was posting all the vulnerabilities that were found, and they haven't been remediated yet, so it would repeat a lot of the ones that we'd seen the day before. 13:15.350 --> 13:20.810 Maybe we didn't want to ignore them yet because they were still relevant or we were still trying to find somebody to fix them. 13:21.010 --> 13:28.590 But it was a very long list, and so we couldn't really work to work them down in this format. 13:30.030 --> 13:37.750 And also, blindly throwing them into a channel made it so it was everybody's problem, which in practice made it nobody's problem. 13:37.750 --> 13:48.490 Nobody took initiative to see which vulnerabilities were actually a threat to us, and nobody was taking responsibility to fix them. 13:48.490 --> 13:54.890 So it was pretty ineffective at solving our main problem at first, so we had to go back to the drawing board and figure that out. 13:56.170 --> 14:06.230 So we had to admit that Ossify, you know, was an effective tool, but it was not only reporting vulnerabilities, but it was also acting as an internal ticketing system. 14:06.810 --> 14:14.010 For something we wanted to maintain long term, we wanted Ossify to do one thing well, find vulnerabilities and report them. 14:14.630 --> 14:20.410 And to do this, we had some prior art that we could work from to figure out the best way to do that at Slack. 14:21.210 --> 14:23.170 And that answer was Jira. 14:24.330 --> 14:31.550 We have a Jira system, we have security tickets that go into our Jira that can be actioned upon by developers. 14:31.910 --> 14:35.650 And so our solution was to put it in Jira. 14:35.650 --> 14:46.170 At Slack, we have a method already for triaging security -based tickets that come from multiple sources, say if we have bug bounty reports or internal findings . 14:46.450 --> 14:56.930 Our SLAs, or service -level agreements, are 180 days for low tickets, 90 days for medium, 30 days for highs, and then seven days for critical findings. 14:57.690 --> 15:05.530 So pushing, upgrading these vulnerabilities to a ticket in Jira puts these already into the processes that our developers know. 15:05.530 --> 15:08.910 It doesn't add in any more friction . 15:09.170 --> 15:14.410 And it puts them into an established workflow that also ties to company -wide objectives. 15:14.750 --> 15:17.870 So upgrading these libraries just can't be another thing that developers ignore. 15:17.870 --> 15:20.130 They have to do it to meet their OKRs. 15:21.170 --> 15:27.210 So now that we had a new flow, we had a new way of dealing with and finding these vulnerable libraries. 15:28.970 --> 15:34.690 So our new flow is that Ossify would find the vulnerability and then throw it right into Jira. 15:34.690 --> 15:35.650 It would file a ticket. 15:35.650 --> 15:40.890 If multiple vulnerabilities were caused by the same package, we would roll those up into the same ticket. 15:40.890 --> 15:51.390 Say you have a couple of highs and a few mediums , but that all would be remediated by upgrading a single package to a certain version, that would be one ticket. 15:51.390 --> 15:54.590 So we weren't making multiple tickets for multiple vulnerabilities per se. 15:54.590 --> 15:57.710 Just if you upgrade this library, these are all soft. 15:59.630 --> 16:02.530 Now we can talk about our triage process a little bit. 16:02.570 --> 16:09.950 So the first step in this would be that the ticket would be filed and the product security engineer on call would triage it. 16:09.950 --> 16:15.410 So our triage process is just enough so that we can find how serious the vulnerability is in our systems. 16:15.590 --> 16:22.710 We would read the proof of concept from the database and then check our source code to see if we are in fact using those vulnerable functions. 16:22.710 --> 16:25.790 Because if we weren't, then it wouldn't be a critical vulnerability for us . 16:25.790 --> 16:30.910 It would be a low, or even maybe a non-fix. 16:32.030 --> 16:37.030 If we are using that, we might try to reproduce the attack. 16:37.030 --> 16:44.150 Or if it's used in multiple places, we're just going to determine that it's enough to determine the severity for fixing. 16:44.290 --> 16:48.230 If it's used in hundreds of places, we can't test every single one. 16:48.850 --> 16:54.810 So we would then recommend it be a medium or high depending on what we thought. 16:55.070 --> 17:02.830 And we're also going to review the fix that's in the upgraded version, just to see if there's anything more we need to consider when implementing the update. 17:03.330 --> 17:10.110 All of this is just to either confirm the severity based on the CVSS score, or adjust it as needed. 17:12.570 --> 17:20.710 From there, you know, we would find the responsible team. 17:20.870 --> 17:28.230 So after determining whether or not we get exploited, we need to find the responsible team to fix that. 17:28.230 --> 17:36.470 That team is then going to be responsible for mediation in the SLA time that we've already determined based on the severity. 17:37.350 --> 17:45.810 Lastly, once the package is updated, Ossify is going to go in and automatically see that that version is no longer used and will remove it from our list. 17:45.810 --> 17:48.750 And therefore, the vulnerability is remediated. 17:51.540 --> 17:58.740 Now that we have a working process, and that fits into what we're already doing at Slack, there's still a lot of future work to be done . 17:58.980 --> 18:11.180 Right now, we're working on getting a clean state for vulnerable packages , and working down our backlog to determine if the packages are currently being... that are currently being flagged are vulnerable and get them fixed. 18:12.400 --> 18:17.740 Next, this process, as you can tell, involves a manual work still. 18:18.080 --> 18:27.000 There's a lot of manual work for the product security engineer to find out who owns the vulnerable dependency and also seeing if it's exploitable. 18:27.120 --> 18:32.960 We would like to figure out a way to automatically figure that out instead of having that be a manual process. 18:32.960 --> 18:39.460 That's just going to be looking into how we can track when new libraries are added . 18:39.920 --> 18:45.260 And thirdly, we'd love to integrate this process into our CI pipeline . 18:45.640 --> 18:52.280 So packages that are vulnerable aren't uploaded to our code base when they're committed. 18:52.280 --> 19:01.760 So on commit, it would run the scanner, look for any open vulnerabilities, and then either block that or let it through if there aren't any. 19:03.360 --> 19:04.800 Thank you so much. 19:05.060 --> 19:11.300 Special thanks to Matt, who did ossify development for us, and Nikki and Oliver for a previous version of this talk . 19:11.300 --> 19:14.640 If you have any questions , we'll be in the Discord chat to answer them . 19:14.640 --> 19:15.180 Thank you.