[00:01.020 --> 00:04.030] Hello and welcome to 10,000 Dependencies Under the Sea, [00:04.030 --> 00:07.090] Exploring and Securing Open Source Dependencies. [00:07.380 --> 00:08.830] My name is Greg Horton. [00:08.830 --> 00:11.210] I'm a Product Security Engineer at Slack. [00:11.210 --> 00:13.990] And I'm Ryan Slama, an Associate Software Engineer [00:13.990 --> 00:17.050] on the Product Security Foundations team. [00:17.690 --> 00:21.490] At Slack, AppSec is product security. [00:21.490 --> 00:23.710] Our product organization is split [00:23.710 --> 00:26.010] into Classics and Foundations. [00:26.010 --> 00:27.650] Classics were the original team, [00:27.650 --> 00:29.690] and they focus on most of the traditional [00:29.690 --> 00:32.970] AppSec responsibilities, like reviewing new features, [00:32.970 --> 00:34.630] penetration testing, and running [00:34.830 --> 00:36.890] a healthy bug bounty program. [00:37.270 --> 00:38.910] I'm on the Foundations team, [00:38.910 --> 00:41.890] which focuses on reducing risk through automated tooling [00:41.890 --> 00:45.250] and creating secure by default libraries and patterns. [00:45.410 --> 00:47.010] Both of these teams work together [00:47.010 --> 00:49.790] to ensure the security of Slack, the product, [00:49.790 --> 00:52.070] using a multifaceted approach. [00:53.330 --> 00:56.070] For some context, what is Slack? [00:56.070 --> 00:58.890] Slack is a channel-based messaging platform. [00:58.890 --> 01:02.390] And with Slack, people can work together more effectively, [01:02.390 --> 01:04.350] connect all their tools and services, [01:04.350 --> 01:07.270] and find the information they need to do their best work, [01:07.270 --> 01:10.470] all within a secure, enterprise-grade environment. [01:11.070 --> 01:14.750] Let's begin by talking a bit about the Slack stack. [01:15.270 --> 01:17.690] Users expect Slack to work everywhere. [01:17.690 --> 01:20.250] To that end, we use and are a partial maintainer [01:20.250 --> 01:21.450] of the Electron project [01:21.450 --> 01:24.670] for a consistent experience across devices. [01:25.010 --> 01:27.210] In practice, this means our front-end web code [01:27.210 --> 01:30.350] also runs in all of our desktop clients. [01:30.590 --> 01:33.210] Our back-end is primarily in Hacklang, [01:33.210 --> 01:34.850] Facebook's fork of PHP, [01:34.850 --> 01:37.190] with strong typing and other enhancements. [01:37.350 --> 01:39.950] We also have some services written in Go, [01:39.950 --> 01:41.410] like our caching layer. [01:41.470 --> 01:45.010] Finally, our mobile apps are primarily in Swift and Kotlin, [01:45.010 --> 01:47.610] but we won't be talking about them much today. [01:49.070 --> 01:52.710] Today, we'll be talking about an OWASP top 10 issue, [01:52.710 --> 01:55.030] using components with known vulnerabilities. [01:55.030 --> 01:58.430] Specifically, we'll be focusing on vulnerability management [01:58.430 --> 02:00.470] for third-party dependencies. [02:01.850 --> 02:05.410] Our story today begins with an intern project. [02:05.470 --> 02:06.630] Matt Dwanzig and I, [02:06.630 --> 02:08.750] who are now full-time engineers at Slack, [02:08.750 --> 02:11.350] were interning on the Product Security Foundations team [02:11.350 --> 02:12.650] last summer. [02:12.810 --> 02:15.450] When we got there, we were given an open-ended project [02:15.450 --> 02:18.610] of understanding and limiting our dependency risk. [02:18.610 --> 02:21.150] Today, we will walk you through our journey [02:21.150 --> 02:22.570] building a tool, [02:22.570 --> 02:26.530] and Greg will walk you through how we implemented the tool [02:26.530 --> 02:28.010] and built a process around it [02:28.010 --> 02:30.930] to actually limit the risk at Slack. [02:32.170 --> 02:36.370] Modern codebases often require tons of third-party code. [02:36.490 --> 02:38.950] At Slack, we have one main repository [02:38.950 --> 02:42.950] that contains our entire front-end and most of our back-end. [02:43.250 --> 02:48.430] Our main repository currently requires over 6,500 packages. [02:48.450 --> 02:50.610] A year ago, we had half that. [02:50.810 --> 02:52.110] This trend would be concerning [02:52.110 --> 02:55.150] if we didn't have systems in place to limit risk. [02:55.270 --> 02:58.010] All of our first-party code has to be reviewed, [02:58.010 --> 03:00.470] but what about random stuff you find on GitHub? [03:00.470 --> 03:01.710] That's exempt. [03:01.930 --> 03:05.230] Clearly, some process must be in place to manage risk. [03:07.050 --> 03:10.130] It's important to note that we have a mature process [03:10.130 --> 03:14.510] for adding packages, and the count still doubled in a year. [03:14.890 --> 03:16.830] For PRs that add packages, [03:16.830 --> 03:19.350] developers have to explain why it's needed. [03:20.070 --> 03:22.170] Packages must be actively maintained [03:22.170 --> 03:24.170] and save meaningful engineering effort [03:24.170 --> 03:26.790] over just building the functionality ourselves. [03:27.230 --> 03:29.770] Additionally, all packages are assigned a team [03:29.770 --> 03:31.750] or a directly responsible individual [03:31.750 --> 03:34.610] to update them and maintain them. [03:34.610 --> 03:38.210] This is especially important for security updates and fixes. [03:38.870 --> 03:41.890] So how did our package count double? [03:42.550 --> 03:44.830] In a word, NPM. [03:44.830 --> 03:49.990] We only directly require 350 packages in our package JSON. [03:50.010 --> 03:53.110] However, when you resolve the nested dependencies, [03:53.110 --> 03:54.830] our dependency tree expands [03:54.830 --> 03:58.610] to almost 6,500 unique versions of packages. [03:59.970 --> 04:03.690] Running this much third-party code can be a risk. [04:03.690 --> 04:05.490] The downside of using common software [04:05.490 --> 04:08.030] is that we often find out about vulnerabilities [04:08.030 --> 04:10.670] at the same time as everyone else. [04:10.670 --> 04:13.810] Let's take a look at some examples of related issues. [04:14.890 --> 04:17.990] One of the most notable was Equifax. [04:17.990 --> 04:19.290] Equifax got hacked [04:19.290 --> 04:22.650] because they were running an old version of Apache struts [04:22.650 --> 04:24.690] that had known vulnerabilities. [04:24.850 --> 04:29.910] This hack leaked 143 million social security numbers. [04:30.050 --> 04:31.890] Preventing this could have been as simple [04:31.890 --> 04:34.110] as upgrading their version of Apache struts [04:34.110 --> 04:37.830] to one that did not have publicly known vulnerabilities. [04:40.150 --> 04:42.150] Vulnerabilities are also discovered [04:42.150 --> 04:45.330] in old versions of popular NPM packages. [04:45.690 --> 04:49.230] For example, both of these advisories were issued this year. [04:49.410 --> 04:54.150] Next.js and AngularJS are respected, well-maintained packages. [04:54.150 --> 04:57.530] But running those in your production codebases [04:57.530 --> 05:02.210] without adequately maintaining them is a recipe for disaster [05:02.210 --> 05:05.310] because new vulnerabilities can be discovered at any time [05:05.310 --> 05:08.810] no matter how many years you've been running the same code. [05:09.770 --> 05:14.690] NPM also had issues with malicious code in popular packages. [05:14.690 --> 05:21.130] For example, EventStream, which was a popular NPM library, [05:21.130 --> 05:24.610] required a dependency that had been compromised. [05:24.610 --> 05:26.670] And that compromised dependency included [05:27.030 --> 05:29.370] a targeted Bitcoin wallet stealer. [05:29.370 --> 05:33.850] And that malicious code was downloaded over 8 million times. [05:35.570 --> 05:39.010] ESLint Scope is an even more popular NPM package [05:39.010 --> 05:41.690] used for linting JavaScript code. [05:41.690 --> 05:43.730] And a compromised version was uploaded [05:43.730 --> 05:46.450] that exfiltrated NPM RC files, [05:46.450 --> 05:49.050] which contained package publishing credentials. [05:49.050 --> 05:51.130] Those credentials could allow an attacker [05:51.130 --> 05:53.610] to push a new version of a package [05:53.610 --> 05:56.430] as a semver patch with malicious code. [05:56.430 --> 06:00.130] And because of the way NPM's semver works, [06:00.130 --> 06:01.810] all installations of that package [06:01.810 --> 06:03.830] would automatically upgrade to the malicious code. [06:05.830 --> 06:09.990] And here's for some figures that might be slightly biased. [06:09.990 --> 06:13.610] Snyk found an 88% increase in library vulnerabilities [06:13.610 --> 06:15.470] over the past two years. [06:15.470 --> 06:18.530] And 78% of vulnerabilities that they found [06:18.530 --> 06:20.950] are in indirect dependencies. [06:21.050 --> 06:23.170] Just take a look at our dependency tree [06:23.170 --> 06:25.210] from earlier to understand why. [06:25.210 --> 06:27.530] When we only require 345, [06:27.530 --> 06:30.830] but end up with almost 6,500 packages, [06:30.830 --> 06:33.150] there's a lot more code beneath the surface [06:33.150 --> 06:34.790] than on top of it. [06:35.750 --> 06:38.350] And we're not the only ones who have this problem. [06:38.490 --> 06:39.990] Our customers know about this problem [06:39.990 --> 06:41.490] and they ask us about it. [06:41.490 --> 06:43.210] We need a good program in place [06:43.210 --> 06:45.850] so we can show them that we take care of their data [06:45.850 --> 06:47.550] and we limit risk. [06:48.430 --> 06:50.650] What do we do about this problem now? [06:51.410 --> 06:53.670] When we started, we defined three goals [06:53.670 --> 06:54.970] for a good solution. [06:55.070 --> 06:57.370] The first is we want to detect vulnerabilities [06:57.370 --> 06:59.250] as soon as they're publicly available [06:59.250 --> 07:01.550] because we want to be on top of the fixes [07:01.550 --> 07:03.130] as quickly as possible. [07:03.710 --> 07:05.970] Second, we want to track any vulnerabilities [07:05.970 --> 07:08.090] or weaknesses in our code base. [07:08.090 --> 07:10.290] We want to be able to have insight [07:10.290 --> 07:12.410] into where we have risk [07:12.410 --> 07:14.770] and so we can better fix it. [07:14.950 --> 07:17.850] Finally, we want to alert when we have a problem. [07:17.870 --> 07:19.770] No developer should ever have to think, [07:19.770 --> 07:21.270] oh, I should scan my repo [07:21.270 --> 07:23.270] before I deployed a production. [07:23.670 --> 07:25.930] We want to alert when we have vulnerabilities [07:25.930 --> 07:28.670] so this just happens automatically. [07:29.490 --> 07:31.470] The next question we asked was, [07:31.470 --> 07:33.130] can we use an off-the-shelf tool? [07:33.150 --> 07:35.430] But our requirements ended up making [07:35.430 --> 07:37.070] that surprisingly difficult. [07:37.710 --> 07:39.890] First of all, we use Hacklang, [07:39.890 --> 07:42.750] which is not a popular language outside Facebook, [07:42.750 --> 07:44.390] so it was difficult to find vendors [07:44.390 --> 07:45.810] that supported it. [07:45.870 --> 07:47.830] Second, we use GitHub Enterprise [07:47.830 --> 07:49.670] rather than GitHub Cloud, [07:49.670 --> 07:52.210] which makes it harder for Dependabot [07:52.210 --> 07:54.850] because Dependabot still isn't out [07:54.850 --> 07:55.670] for GitHub Enterprise [07:55.670 --> 07:58.470] and it was even further from being out [07:58.790 --> 08:01.090] a year ago when we began this journey. [08:02.170 --> 08:04.510] Next, we needed a tool that will scan [08:04.510 --> 08:06.610] the entire dependency trees. [08:06.930 --> 08:08.170] Knowing about vulnerabilities [08:08.170 --> 08:10.010] and packages we require directly [08:10.010 --> 08:11.170] isn't enough. [08:11.170 --> 08:12.370] As we saw earlier, [08:12.370 --> 08:13.650] there are so many more packages [08:13.650 --> 08:14.530] beneath the surface [08:14.530 --> 08:15.630] and we need to make sure [08:15.630 --> 08:17.510] that those are a stable foundation [08:17.510 --> 08:19.450] that we can build our app on. [08:19.690 --> 08:21.430] And finally, at Slack, [08:21.430 --> 08:23.250] we're heavy users of Slack [08:23.250 --> 08:24.850] as a product. [08:24.850 --> 08:27.290] All of our alerts and things like that [08:27.290 --> 08:28.830] just go in Slack. [08:28.830 --> 08:31.190] So we need a tool that has support [08:31.190 --> 08:32.690] for routing alerts [08:32.690 --> 08:33.790] for different code bases [08:33.790 --> 08:35.910] or packages to different teams. [08:36.370 --> 08:38.370] And finally, you might be saying, [08:38.370 --> 08:39.350] oh, we're a vendor [08:39.350 --> 08:41.350] and we have a tool that does all that. [08:41.390 --> 08:43.910] But we're still a relatively small company [08:43.910 --> 08:45.030] and we need something [08:45.030 --> 08:46.330] that's not going to cost us [08:46.330 --> 08:47.230] millions of dollars [08:47.230 --> 08:49.030] over the next few years. [08:50.750 --> 08:52.690] So we built Ossify. [08:53.830 --> 08:55.830] Our solutions to the three points [08:55.830 --> 08:57.490] mentioned earlier were first, [08:57.490 --> 08:58.710] the in-detection. [08:58.710 --> 09:01.430] We run daily scans of our code bases [09:01.430 --> 09:04.410] to both figure out [09:04.410 --> 09:06.610] what packages we're requiring. [09:06.610 --> 09:08.870] And also, we upload those packages [09:08.870 --> 09:10.950] to the Sonatype open source [09:10.950 --> 09:12.090] vulnerability index [09:12.090 --> 09:13.870] and see if there are [09:13.870 --> 09:15.150] any new vulnerabilities [09:15.150 --> 09:16.190] that have been reported [09:16.190 --> 09:17.430] for our packages. [09:17.770 --> 09:19.270] Second, we built a dashboard [09:19.270 --> 09:21.850] to track the status of repositories, [09:21.850 --> 09:24.270] as well as tracking remediation efforts [09:24.270 --> 09:26.350] for individual findings. [09:26.350 --> 09:27.970] And finally, we built [09:27.970 --> 09:29.790] robust Slack alerting, [09:29.790 --> 09:32.330] which will be covered more later. [09:35.010 --> 09:37.830] Ossify supports three package ecosystems. [09:37.830 --> 09:39.630] The first, of course, is Hackline, [09:39.630 --> 09:40.870] because that's our backend [09:40.870 --> 09:42.210] and that's where some of the most [09:42.210 --> 09:44.330] dangerous possibilities are. [09:44.790 --> 09:46.910] We added custom metadata [09:46.910 --> 09:50.030] to track composer packages upstream [09:50.030 --> 09:52.170] because we had to manually fork [09:52.170 --> 09:53.850] and vendor some of these packages [09:53.850 --> 09:55.250] to add strong types [09:55.250 --> 09:57.510] and other Hackline features. [09:57.910 --> 09:59.950] Second, we added NPM, [09:59.950 --> 10:02.170] because NPM is the biggest offender [10:02.170 --> 10:04.790] for vast amounts of packages. [10:04.790 --> 10:06.270] And we found, by far, [10:06.270 --> 10:07.250] the most findings [10:07.250 --> 10:09.530] coming from NPM packages. [10:09.890 --> 10:12.130] And finally, we added Go support [10:12.130 --> 10:13.050] because a number [10:13.050 --> 10:14.910] of our high value services, [10:14.910 --> 10:15.930] like our caching layer, [10:15.930 --> 10:16.910] are written in Go. [10:18.530 --> 10:20.590] Here is what the Ossify dashboard [10:20.590 --> 10:21.490] looks like. [10:21.490 --> 10:23.090] This is an example repository [10:23.090 --> 10:24.530] created for this demo [10:24.530 --> 10:25.590] just to show you [10:25.590 --> 10:26.690] what it looks like [10:26.690 --> 10:28.530] when you scan a repository. [10:28.730 --> 10:29.890] So we support scanning [10:29.890 --> 10:31.130] multiple branches, [10:31.130 --> 10:32.690] which could be used potentially [10:32.690 --> 10:34.650] for future CI integration. [10:35.050 --> 10:36.050] Here we have an example [10:36.050 --> 10:37.390] finding page. [10:37.390 --> 10:39.090] This is for NodeForge, [10:39.090 --> 10:40.890] which was being pulled in [10:40.890 --> 10:43.630] in our sample repo's package JSON. [10:43.930 --> 10:45.350] NodeForge has a weakness, [10:45.350 --> 10:46.350] but we don't actually [10:46.350 --> 10:47.790] require NodeForge. [10:47.790 --> 10:49.710] Instead, we require a Google Auth [10:49.710 --> 10:52.010] library all the way on the left. [10:52.010 --> 10:53.450] We built out a dependency [10:53.450 --> 10:55.130] graph tool because we want [10:55.130 --> 10:56.490] our developers to understand [10:56.490 --> 10:57.470] where the vulnerability [10:57.470 --> 10:59.210] is coming from and what packages [10:59.210 --> 11:00.890] need upgraded or removed [11:00.890 --> 11:02.390] to fix the issue. [11:04.290 --> 11:05.610] Here's a more complicated [11:05.610 --> 11:07.070] dependency graph. [11:07.470 --> 11:10.210] This is from a dev dependency. [11:10.490 --> 11:12.210] The weakness was on the right [11:12.210 --> 11:13.510] in isURL. [11:13.510 --> 11:15.150] However, if you trace back [11:15.150 --> 11:16.550] all the way to the left, [11:16.550 --> 11:18.030] it was actually some Gulp plugins [11:18.030 --> 11:19.430] we were using as part of a build [11:19.430 --> 11:21.530] process that were pulling in [11:21.530 --> 11:23.950] the weak version of isURL. [11:24.810 --> 11:26.690] And finally, like any good [11:26.690 --> 11:27.890] security tool, it comes with [11:27.890 --> 11:28.810] dark mode. [11:28.810 --> 11:29.910] Now I'm going to hand things [11:29.910 --> 11:31.370] off to Greg to cover everything [11:31.370 --> 11:33.010] that happened after the original [11:33.010 --> 11:34.550] development of the tool. [11:38.070 --> 11:39.710] Thanks, Ryan, for that great [11:39.710 --> 11:40.930] overview of the tool. [11:41.110 --> 11:42.110] So now that we have this [11:42.110 --> 11:43.350] fully featured tool that [11:43.350 --> 11:45.070] did everything that we wanted, [11:45.070 --> 11:46.170] it was time to integrate it [11:46.170 --> 11:47.570] into our wider processes [11:47.570 --> 11:48.690] here at Slack. [11:49.170 --> 11:50.970] So this would be easy, right? [11:50.970 --> 11:53.790] You know, we have an app that [11:53.790 --> 11:55.830] scans our repositories, [11:55.830 --> 11:56.750] looks at our third-party [11:57.550 --> 11:59.330] integrations, sees if there's [11:59.330 --> 12:00.990] any vulnerabilities, and then [12:00.990 --> 12:02.390] lets us know about them. [12:02.890 --> 12:04.410] So our first workflow for this [12:04.410 --> 12:05.890] tool was that Ossify would [12:05.890 --> 12:09.390] scan daily and then post to a [12:09.390 --> 12:10.850] Slack channel any currently [12:10.850 --> 12:12.930] known vulnerabilities. [12:12.930 --> 12:15.210] And using the power of Slack, [12:15.210 --> 12:16.910] we could make Slack notifications [12:16.910 --> 12:19.410] that were actionable, [12:19.410 --> 12:20.970] meaning that we could give the [12:20.970 --> 12:21.750] information that was needed [12:21.750 --> 12:23.650] for quick remediation. [12:23.650 --> 12:25.050] They were non-obtrusive, so we [12:25.050 --> 12:26.190] could set those notifications [12:26.190 --> 12:28.030] to snooze or ignore if they [12:28.030 --> 12:29.550] weren't relevant to us. [12:29.710 --> 12:32.370] And then make them configurable [12:32.370 --> 12:34.350] so we could notify individual [12:34.350 --> 12:36.550] users and specific channels [12:37.290 --> 12:39.590] about these dependencies. [12:40.250 --> 12:41.690] Here's an example of what [12:41.690 --> 12:42.970] those looked like. [12:43.010 --> 12:44.190] So as you can see, this [12:44.190 --> 12:45.550] dependency detection of the [12:45.550 --> 12:48.670] app, it scanned a repository, [12:48.670 --> 12:49.810] it found some vulnerabilities [12:49.810 --> 12:52.130] in some third-party libraries, [12:52.810 --> 12:54.410] and it would tell us about them. [12:54.410 --> 12:56.190] And as you can see, also, [12:56.190 --> 12:57.890] you could set them to ignore, [12:57.890 --> 13:00.090] you could snooze them, etc. [13:01.010 --> 13:02.150] But the problem with that is [13:02.150 --> 13:04.170] that this channel was way [13:04.170 --> 13:05.170] too noisy. [13:05.290 --> 13:06.550] Every single day, it was [13:06.550 --> 13:07.970] posting all the vulnerabilities [13:07.970 --> 13:10.630] that were found, and they [13:10.630 --> 13:12.090] haven't been remediated yet, [13:12.090 --> 13:13.630] so it would repeat a lot of [13:13.630 --> 13:14.550] the ones that we'd seen the [13:14.550 --> 13:15.350] day before. [13:15.350 --> 13:16.470] Maybe we didn't want to ignore [13:16.470 --> 13:17.890] them yet because they were [13:17.890 --> 13:18.970] still relevant or we were [13:18.970 --> 13:19.770] still trying to find somebody [13:19.770 --> 13:20.810] to fix them. [13:21.010 --> 13:22.670] But it was a very long list, [13:22.670 --> 13:23.830] and so we couldn't really [13:24.610 --> 13:26.970] work to work them down [13:27.470 --> 13:28.590] in this format. [13:30.030 --> 13:31.730] And also, blindly throwing [13:31.730 --> 13:33.670] them into a channel made it [13:33.670 --> 13:35.310] so it was everybody's problem, [13:35.310 --> 13:36.670] which in practice made it [13:36.670 --> 13:37.750] nobody's problem. [13:37.750 --> 13:39.150] Nobody took initiative to [13:39.150 --> 13:41.250] see which vulnerabilities [13:41.250 --> 13:44.670] were actually a threat to us, [13:45.290 --> 13:47.050] and nobody was taking [13:47.050 --> 13:48.490] responsibility to fix them. [13:48.490 --> 13:49.610] So it was pretty ineffective [13:49.610 --> 13:50.890] at solving our main problem [13:50.890 --> 13:51.890] at first, so we had to go [13:51.890 --> 13:52.770] back to the drawing board [13:53.310 --> 13:54.890] and figure that out. [13:56.170 --> 13:57.250] So we had to admit that [13:57.250 --> 13:59.490] Ossify, you know, was an [13:59.490 --> 14:01.590] effective tool, but it was [14:01.590 --> 14:02.330] not only reporting [14:02.330 --> 14:03.690] vulnerabilities, but it was [14:03.690 --> 14:04.990] also acting as an internal [14:04.990 --> 14:06.230] ticketing system. [14:06.810 --> 14:07.690] For something we wanted to [14:07.690 --> 14:09.810] maintain long term, we [14:09.810 --> 14:11.030] wanted Ossify to do one [14:11.030 --> 14:12.070] thing well, find [14:12.070 --> 14:13.330] vulnerabilities and report [14:13.330 --> 14:16.190] them. And to do this, we [14:16.190 --> 14:17.190] had some prior art that we [14:17.190 --> 14:18.750] could work from to figure [14:18.750 --> 14:19.590] out the best way to do that [14:19.590 --> 14:20.410] at Slack. [14:21.210 --> 14:23.170] And that answer was Jira. [14:24.330 --> 14:26.330] We have a Jira system, we [14:26.330 --> 14:27.810] have security tickets that [14:27.810 --> 14:29.090] go into our Jira that can [14:29.090 --> 14:30.430] be actioned upon by [14:30.430 --> 14:33.030] developers. And so our [14:33.030 --> 14:34.730] solution was to put it in [14:34.730 --> 14:37.110] Jira. At Slack, we have a [14:37.110 --> 14:39.610] method already for triaging [14:39.610 --> 14:41.090] security-based tickets that [14:41.090 --> 14:42.610] come from multiple sources, [14:42.610 --> 14:44.190] say if we have bug bounty [14:44.190 --> 14:45.570] reports or internal [14:45.570 --> 14:47.750] findings. Our SLAs, or [14:47.750 --> 14:49.050] service-level agreements, [14:49.050 --> 14:50.770] are 180 days for low [14:50.770 --> 14:52.270] tickets, 90 days for [14:52.270 --> 14:53.870] medium, 30 days for [14:53.870 --> 14:55.230] highs, and then seven days [14:55.230 --> 14:56.930] for critical findings. [14:57.690 --> 14:59.030] So pushing, upgrading [14:59.030 --> 15:00.350] these vulnerabilities to a [15:00.350 --> 15:02.430] ticket in Jira puts these [15:02.430 --> 15:04.210] already into the processes [15:04.210 --> 15:05.850] that our developers know. [15:05.850 --> 15:08.350] It doesn't add in any more [15:08.350 --> 15:10.330] friction. And it puts them [15:10.330 --> 15:11.410] into an established [15:11.410 --> 15:13.230] workflow that also ties to [15:13.230 --> 15:14.910] company-wide objectives. [15:14.910 --> 15:15.770] So upgrading these [15:15.770 --> 15:16.530] libraries just can't be [15:16.530 --> 15:17.290] another thing that [15:17.290 --> 15:18.270] developers ignore. They [15:18.270 --> 15:18.970] have to do it to meet [15:18.970 --> 15:20.130] their OKRs. [15:21.170 --> 15:22.350] So now that we had a new [15:22.350 --> 15:24.410] flow, we had a new way of [15:24.410 --> 15:25.670] dealing with and finding [15:25.670 --> 15:27.210] these vulnerable libraries. [15:28.970 --> 15:30.970] So our new flow is that [15:30.970 --> 15:32.110] Ossify would find the [15:32.110 --> 15:33.510] vulnerability and then [15:33.510 --> 15:34.690] throw it right into Jira. [15:34.690 --> 15:35.650] It would file a ticket. [15:35.650 --> 15:37.130] If multiple vulnerabilities [15:37.130 --> 15:38.130] were caused by the same [15:38.130 --> 15:39.570] package, we would roll [15:39.570 --> 15:40.390] those up into the same [15:40.390 --> 15:42.490] ticket. Say you have a [15:42.490 --> 15:44.070] couple of highs and a few [15:44.070 --> 15:45.470] mediums, but that all would [15:45.470 --> 15:48.110] be remediated by upgrading [15:48.110 --> 15:48.230] a single package to a [15:48.230 --> 15:50.650] certain version, that would [15:50.650 --> 15:51.550] be one ticket. So we [15:51.550 --> 15:52.570] weren't making multiple [15:52.570 --> 15:53.350] tickets for multiple [15:53.350 --> 15:54.590] vulnerabilities per se. [15:54.590 --> 15:56.030] Just if you upgrade this [15:56.030 --> 15:56.950] library, these are all [15:56.950 --> 16:00.570] soft. Now we can talk [16:00.570 --> 16:01.850] about our triage process a [16:01.850 --> 16:03.350] little bit. So the first [16:03.350 --> 16:05.050] step in this would be that [16:05.050 --> 16:06.150] the ticket would be filed [16:06.150 --> 16:07.370] and the product security [16:07.370 --> 16:09.490] engineer on call would [16:09.490 --> 16:11.330] triage it. So our triage [16:11.330 --> 16:12.650] process is just enough so [16:12.650 --> 16:13.690] that we can find how [16:13.690 --> 16:14.450] serious the vulnerability [16:14.450 --> 16:16.170] is in our systems. [16:16.170 --> 16:17.130] We would read the proof [16:17.130 --> 16:17.530] of concept from the [16:17.530 --> 16:19.430] database and then check [16:19.430 --> 16:20.410] our source code to see if [16:20.410 --> 16:21.610] we are in fact using those [16:21.610 --> 16:22.710] vulnerable functions. [16:22.710 --> 16:23.810] Because if we weren't, [16:23.810 --> 16:24.790] then it wouldn't be a [16:24.790 --> 16:25.710] critical vulnerability for [16:25.710 --> 16:27.510] us. It would be a low, [16:27.510 --> 16:32.470] or even maybe a non-fix. [16:32.470 --> 16:35.090] If we are using that, we [16:35.090 --> 16:36.450] might try to reproduce [16:36.450 --> 16:37.850] the attack. Or if it's [16:37.850 --> 16:39.170] used in multiple places, [16:39.170 --> 16:40.210] we're just going to [16:40.210 --> 16:41.550] determine that it's [16:41.550 --> 16:43.410] enough to determine the [16:43.410 --> 16:44.790] severity for fixing. If [16:44.790 --> 16:46.170] it's used in hundreds of [16:46.170 --> 16:46.970] places, we can't test [16:46.970 --> 16:49.070] every single one. So we [16:49.070 --> 16:50.690] would then recommend it [16:50.690 --> 16:51.990] be a medium or high [16:51.990 --> 16:54.130] depending on what we [16:54.130 --> 16:55.730] thought. And we're also [16:55.730 --> 16:56.770] going to review the fix [16:56.770 --> 16:58.670] that's in the upgraded [16:58.670 --> 17:00.090] version, just to see if [17:00.090 --> 17:00.910] there's anything more we [17:00.910 --> 17:01.650] need to consider when [17:01.650 --> 17:02.830] implementing the update. [17:03.330 --> 17:04.310] All of this is just to [17:04.310 --> 17:05.830] either confirm the [17:05.830 --> 17:07.130] severity based on the [17:07.130 --> 17:09.230] CVSS score, or adjust it [17:09.230 --> 17:10.110] as needed. [17:12.570 --> 17:14.930] From there, you know, we [17:16.970 --> 17:18.390] would find the [17:19.370 --> 17:21.070] responsible team. So [17:21.070 --> 17:22.870] after determining whether [17:22.870 --> 17:23.510] or not we get [17:23.510 --> 17:25.190] exploited, we need to [17:25.190 --> 17:26.270] find the responsible team [17:27.070 --> 17:28.810] to fix that. That team is [17:28.810 --> 17:29.290] then going to be [17:29.290 --> 17:30.430] responsible for mediation [17:31.650 --> 17:33.510] in the SLA time that [17:33.510 --> 17:34.790] we've already determined [17:34.790 --> 17:36.470] based on the severity. [17:37.350 --> 17:38.490] Lastly, once the package [17:38.490 --> 17:39.950] is updated, Ossify is [17:39.950 --> 17:41.170] going to go in and [17:41.170 --> 17:42.250] automatically see that [17:42.250 --> 17:43.070] that version is no longer [17:43.070 --> 17:44.710] used and will remove it [17:44.710 --> 17:45.890] from our list. And [17:45.890 --> 17:46.970] therefore, the [17:46.970 --> 17:47.770] vulnerability is [17:47.770 --> 17:52.220] remediated. Now that we [17:52.220 --> 17:53.600] have a working process, [17:53.600 --> 17:55.100] and that fits into what [17:55.100 --> 17:55.920] we're already doing at [17:55.920 --> 17:57.280] Slack, there's still a [17:57.280 --> 17:58.260] lot of future work to be [17:58.260 --> 18:00.660] done. Right now, we're [18:00.660 --> 18:01.480] working on getting a [18:01.480 --> 18:03.120] clean state for vulnerable [18:03.120 --> 18:05.220] packages, and working [18:05.220 --> 18:06.400] down our backlog to [18:06.400 --> 18:07.280] determine if the packages [18:07.280 --> 18:08.460] are currently being... [18:08.460 --> 18:09.420] that are currently being [18:09.420 --> 18:10.240] flagged are vulnerable [18:10.240 --> 18:13.420] and get them fixed. [18:13.420 --> 18:14.620] Next, this process, as [18:14.620 --> 18:15.720] you can tell, involves a [18:16.160 --> 18:18.520] manual work still. There's [18:18.620 --> 18:19.320] a lot of manual work for [18:19.320 --> 18:20.020] the product security [18:20.020 --> 18:21.300] engineer to find out who [18:21.300 --> 18:23.420] owns the vulnerable [18:23.420 --> 18:24.760] dependency and also [18:25.300 --> 18:27.000] seeing if it's exploitable. [18:27.120 --> 18:27.980] We would like to figure [18:27.980 --> 18:29.160] out a way to automatically [18:29.160 --> 18:31.260] figure that out instead [18:31.260 --> 18:32.200] of having that be a [18:32.200 --> 18:33.240] manual process. That's [18:33.240 --> 18:34.820] just going to be looking [18:34.820 --> 18:36.820] into how we can track [18:37.860 --> 18:39.020] when new libraries are [18:39.020 --> 18:41.080] added. And thirdly, [18:41.080 --> 18:42.500] we'd love to integrate [18:42.500 --> 18:44.820] this process into our [18:44.820 --> 18:45.700] CI pipeline. So [18:45.700 --> 18:46.440] packages that are [18:46.440 --> 18:48.700] vulnerable aren't uploaded [18:48.700 --> 18:51.460] to our code base when [18:51.460 --> 18:52.520] they're committed. So [18:52.520 --> 18:54.940] on commit, it would run [18:54.940 --> 18:56.060] the scanner, look for any [18:56.060 --> 18:58.300] open vulnerabilities, and [18:58.300 --> 19:00.080] then either block that or [19:00.080 --> 19:00.860] let it through if there [19:00.860 --> 19:04.100] aren't any. Thank you so [19:04.100 --> 19:05.960] much. Special thanks to [19:05.960 --> 19:07.640] Matt, who did ossify [19:07.640 --> 19:08.820] development for us, and [19:08.820 --> 19:09.980] Nikki and Oliver for a [19:09.980 --> 19:10.880] previous version of this [19:10.880 --> 19:12.200] talk. If you have any [19:12.200 --> 19:12.780] questions, we'll be in [19:12.780 --> 19:14.180] the Discord chat to answer [19:14.180 --> 19:15.180] them. Thank you.