start	end	text
1020	4030	Hello and welcome to 10,000 Dependencies Under the Sea,
4030	7090	Exploring and Securing Open Source Dependencies.
7380	8830	My name is Greg Horton.
8830	11210	I'm a Product Security Engineer at Slack.
11210	13990	And I'm Ryan Slama, an Associate Software Engineer
13990	17050	on the Product Security Foundations team.
17690	21490	At Slack, AppSec is product security.
21490	23710	Our product organization is split
23710	26010	into Classics and Foundations.
26010	27650	Classics were the original team,
27650	29690	and they focus on most of the traditional
29690	32970	AppSec responsibilities, like reviewing new features,
32970	34630	penetration testing, and running
34830	36890	a healthy bug bounty program.
37270	38910	I'm on the Foundations team,
38910	41890	which focuses on reducing risk through automated tooling
41890	45250	and creating secure by default libraries and patterns.
45410	47010	Both of these teams work together
47010	49790	to ensure the security of Slack, the product,
49790	52070	using a multifaceted approach.
53330	56070	For some context, what is Slack?
56070	58890	Slack is a channel-based messaging platform.
58890	62390	And with Slack, people can work together more effectively,
62390	64350	connect all their tools and services,
64350	67270	and find the information they need to do their best work,
67270	70470	all within a secure, enterprise-grade environment.
71070	74750	Let's begin by talking a bit about the Slack stack.
75270	77690	Users expect Slack to work everywhere.
77690	80250	To that end, we use and are a partial maintainer
80250	81450	of the Electron project
81450	84670	for a consistent experience across devices.
85010	87210	In practice, this means our front-end web code
87210	90350	also runs in all of our desktop clients.
90590	93210	Our back-end is primarily in Hacklang,
93210	94850	Facebook's fork of PHP,
94850	97190	with strong typing and other enhancements.
97350	99950	We also have some services written in Go,
99950	101410	like our caching layer.
101470	105010	Finally, our mobile apps are primarily in Swift and Kotlin,
105010	107610	but we won't be talking about them much today.
109070	112710	Today, we'll be talking about an OWASP top 10 issue,
112710	115030	using components with known vulnerabilities.
115030	118430	Specifically, we'll be focusing on vulnerability management
118430	120470	for third-party dependencies.
121850	125410	Our story today begins with an intern project.
125470	126630	Matt Dwanzig and I,
126630	128750	who are now full-time engineers at Slack,
128750	131350	were interning on the Product Security Foundations team
131350	132650	last summer.
132810	135450	When we got there, we were given an open-ended project
135450	138610	of understanding and limiting our dependency risk.
138610	141150	Today, we will walk you through our journey
141150	142570	building a tool,
142570	146530	and Greg will walk you through how we implemented the tool
146530	148010	and built a process around it
148010	150930	to actually limit the risk at Slack.
152170	156370	Modern codebases often require tons of third-party code.
156490	158950	At Slack, we have one main repository
158950	162950	that contains our entire front-end and most of our back-end.
163250	168430	Our main repository currently requires over 6,500 packages.
168450	170610	A year ago, we had half that.
170810	172110	This trend would be concerning
172110	175150	if we didn't have systems in place to limit risk.
175270	178010	All of our first-party code has to be reviewed,
178010	180470	but what about random stuff you find on GitHub?
180470	181710	That's exempt.
181930	185230	Clearly, some process must be in place to manage risk.
187050	190130	It's important to note that we have a mature process
190130	194510	for adding packages, and the count still doubled in a year.
194890	196830	For PRs that add packages,
196830	199350	developers have to explain why it's needed.
200070	202170	Packages must be actively maintained
202170	204170	and save meaningful engineering effort
204170	206790	over just building the functionality ourselves.
207230	209770	Additionally, all packages are assigned a team
209770	211750	or a directly responsible individual
211750	214610	to update them and maintain them.
214610	218210	This is especially important for security updates and fixes.
218870	221890	So how did our package count double?
222550	224830	In a word, NPM.
224830	229990	We only directly require 350 packages in our package JSON.
230010	233110	However, when you resolve the nested dependencies,
233110	234830	our dependency tree expands
234830	238610	to almost 6,500 unique versions of packages.
239970	243690	Running this much third-party code can be a risk.
243690	245490	The downside of using common software
245490	248030	is that we often find out about vulnerabilities
248030	250670	at the same time as everyone else.
250670	253810	Let's take a look at some examples of related issues.
254890	257990	One of the most notable was Equifax.
257990	259290	Equifax got hacked
259290	262650	because they were running an old version of Apache struts
262650	264690	that had known vulnerabilities.
264850	269910	This hack leaked 143 million social security numbers.
270050	271890	Preventing this could have been as simple
271890	274110	as upgrading their version of Apache struts
274110	277830	to one that did not have publicly known vulnerabilities.
280150	282150	Vulnerabilities are also discovered
282150	285330	in old versions of popular NPM packages.
285690	289230	For example, both of these advisories were issued this year.
289410	294150	Next.js and AngularJS are respected, well-maintained packages.
294150	297530	But running those in your production codebases
297530	302210	without adequately maintaining them is a recipe for disaster
302210	305310	because new vulnerabilities can be discovered at any time
305310	308810	no matter how many years you've been running the same code.
309770	314690	NPM also had issues with malicious code in popular packages.
314690	321130	For example, EventStream, which was a popular NPM library,
321130	324610	required a dependency that had been compromised.
324610	326670	And that compromised dependency included
327030	329370	a targeted Bitcoin wallet stealer.
329370	333850	And that malicious code was downloaded over 8 million times.
335570	339010	ESLint Scope is an even more popular NPM package
339010	341690	used for linting JavaScript code.
341690	343730	And a compromised version was uploaded
343730	346450	that exfiltrated NPM RC files,
346450	349050	which contained package publishing credentials.
349050	351130	Those credentials could allow an attacker
351130	353610	to push a new version of a package
353610	356430	as a semver patch with malicious code.
356430	360130	And because of the way NPM's semver works,
360130	361810	all installations of that package
361810	363830	would automatically upgrade to the malicious code.
365830	369990	And here's for some figures that might be slightly biased.
369990	373610	Snyk found an 88% increase in library vulnerabilities
373610	375470	over the past two years.
375470	378530	And 78% of vulnerabilities that they found
378530	380950	are in indirect dependencies.
381050	383170	Just take a look at our dependency tree
383170	385210	from earlier to understand why.
385210	387530	When we only require 345,
387530	390830	but end up with almost 6,500 packages,
390830	393150	there's a lot more code beneath the surface
393150	394790	than on top of it.
395750	398350	And we're not the only ones who have this problem.
398490	399990	Our customers know about this problem
399990	401490	and they ask us about it.
401490	403210	We need a good program in place
403210	405850	so we can show them that we take care of their data
405850	407550	and we limit risk.
408430	410650	What do we do about this problem now?
411410	413670	When we started, we defined three goals
413670	414970	for a good solution.
415070	417370	The first is we want to detect vulnerabilities
417370	419250	as soon as they're publicly available
419250	421550	because we want to be on top of the fixes
421550	423130	as quickly as possible.
423710	425970	Second, we want to track any vulnerabilities
425970	428090	or weaknesses in our code base.
428090	430290	We want to be able to have insight
430290	432410	into where we have risk
432410	434770	and so we can better fix it.
434950	437850	Finally, we want to alert when we have a problem.
437870	439770	No developer should ever have to think,
439770	441270	oh, I should scan my repo
441270	443270	before I deployed a production.
443670	445930	We want to alert when we have vulnerabilities
445930	448670	so this just happens automatically.
449490	451470	The next question we asked was,
451470	453130	can we use an off-the-shelf tool?
453150	455430	But our requirements ended up making
455430	457070	that surprisingly difficult.
457710	459890	First of all, we use Hacklang,
459890	462750	which is not a popular language outside Facebook,
462750	464390	so it was difficult to find vendors
464390	465810	that supported it.
465870	467830	Second, we use GitHub Enterprise
467830	469670	rather than GitHub Cloud,
469670	472210	which makes it harder for Dependabot
472210	474850	because Dependabot still isn't out
474850	475670	for GitHub Enterprise
475670	478470	and it was even further from being out
478790	481090	a year ago when we began this journey.
482170	484510	Next, we needed a tool that will scan
484510	486610	the entire dependency trees.
486930	488170	Knowing about vulnerabilities
488170	490010	and packages we require directly
490010	491170	isn't enough.
491170	492370	As we saw earlier,
492370	493650	there are so many more packages
493650	494530	beneath the surface
494530	495630	and we need to make sure
495630	497510	that those are a stable foundation
497510	499450	that we can build our app on.
499690	501430	And finally, at Slack,
501430	503250	we're heavy users of Slack
503250	504850	as a product.
504850	507290	All of our alerts and things like that
507290	508830	just go in Slack.
508830	511190	So we need a tool that has support
511190	512690	for routing alerts
512690	513790	for different code bases
513790	515910	or packages to different teams.
516370	518370	And finally, you might be saying,
518370	519350	oh, we're a vendor
519350	521350	and we have a tool that does all that.
521390	523910	But we're still a relatively small company
523910	525030	and we need something
525030	526330	that's not going to cost us
526330	527230	millions of dollars
527230	529030	over the next few years.
530750	532690	So we built Ossify.
533830	535830	Our solutions to the three points
535830	537490	mentioned earlier were first,
537490	538710	the in-detection.
538710	541430	We run daily scans of our code bases
541430	544410	to both figure out
544410	546610	what packages we're requiring.
546610	548870	And also, we upload those packages
548870	550950	to the Sonatype open source
550950	552090	vulnerability index
552090	553870	and see if there are
553870	555150	any new vulnerabilities
555150	556190	that have been reported
556190	557430	for our packages.
557770	559270	Second, we built a dashboard
559270	561850	to track the status of repositories,
561850	564270	as well as tracking remediation efforts
564270	566350	for individual findings.
566350	567970	And finally, we built
567970	569790	robust Slack alerting,
569790	572330	which will be covered more later.
575010	577830	Ossify supports three package ecosystems.
577830	579630	The first, of course, is Hackline,
579630	580870	because that's our backend
580870	582210	and that's where some of the most
582210	584330	dangerous possibilities are.
584790	586910	We added custom metadata
586910	590030	to track composer packages upstream
590030	592170	because we had to manually fork
592170	593850	and vendor some of these packages
593850	595250	to add strong types
595250	597510	and other Hackline features.
597910	599950	Second, we added NPM,
599950	602170	because NPM is the biggest offender
602170	604790	for vast amounts of packages.
604790	606270	And we found, by far,
606270	607250	the most findings
607250	609530	coming from NPM packages.
609890	612130	And finally, we added Go support
612130	613050	because a number
613050	614910	of our high value services,
614910	615930	like our caching layer,
615930	616910	are written in Go.
618530	620590	Here is what the Ossify dashboard
620590	621490	looks like.
621490	623090	This is an example repository
623090	624530	created for this demo
624530	625590	just to show you
625590	626690	what it looks like
626690	628530	when you scan a repository.
628730	629890	So we support scanning
629890	631130	multiple branches,
631130	632690	which could be used potentially
632690	634650	for future CI integration.
635050	636050	Here we have an example
636050	637390	finding page.
637390	639090	This is for NodeForge,
639090	640890	which was being pulled in
640890	643630	in our sample repo's package JSON.
643930	645350	NodeForge has a weakness,
645350	646350	but we don't actually
646350	647790	require NodeForge.
647790	649710	Instead, we require a Google Auth
649710	652010	library all the way on the left.
652010	653450	We built out a dependency
653450	655130	graph tool because we want
655130	656490	our developers to understand
656490	657470	where the vulnerability
657470	659210	is coming from and what packages
659210	660890	need upgraded or removed
660890	662390	to fix the issue.
664290	665610	Here's a more complicated
665610	667070	dependency graph.
667470	670210	This is from a dev dependency.
670490	672210	The weakness was on the right
672210	673510	in isURL.
673510	675150	However, if you trace back
675150	676550	all the way to the left,
676550	678030	it was actually some Gulp plugins
678030	679430	we were using as part of a build
679430	681530	process that were pulling in
681530	683950	the weak version of isURL.
684810	686690	And finally, like any good
686690	687890	security tool, it comes with
687890	688810	dark mode.
688810	689910	Now I'm going to hand things
689910	691370	off to Greg to cover everything
691370	693010	that happened after the original
693010	694550	development of the tool.
698070	699710	Thanks, Ryan, for that great
699710	700930	overview of the tool.
701110	702110	So now that we have this
702110	703350	fully featured tool that
703350	705070	did everything that we wanted,
705070	706170	it was time to integrate it
706170	707570	into our wider processes
707570	708690	here at Slack.
709170	710970	So this would be easy, right?
710970	713790	You know, we have an app that
713790	715830	scans our repositories,
715830	716750	looks at our third-party
717550	719330	integrations, sees if there's
719330	720990	any vulnerabilities, and then
720990	722390	lets us know about them.
722890	724410	So our first workflow for this
724410	725890	tool was that Ossify would
725890	729390	scan daily and then post to a
729390	730850	Slack channel any currently
730850	732930	known vulnerabilities.
732930	735210	And using the power of Slack,
735210	736910	we could make Slack notifications
736910	739410	that were actionable,
739410	740970	meaning that we could give the
740970	741750	information that was needed
741750	743650	for quick remediation.
743650	745050	They were non-obtrusive, so we
745050	746190	could set those notifications
746190	748030	to snooze or ignore if they
748030	749550	weren't relevant to us.
749710	752370	And then make them configurable
752370	754350	so we could notify individual
754350	756550	users and specific channels
757290	759590	about these dependencies.
760250	761690	Here's an example of what
761690	762970	those looked like.
763010	764190	So as you can see, this
764190	765550	dependency detection of the
765550	768670	app, it scanned a repository,
768670	769810	it found some vulnerabilities
769810	772130	in some third-party libraries,
772810	774410	and it would tell us about them.
774410	776190	And as you can see, also,
776190	777890	you could set them to ignore,
777890	780090	you could snooze them, etc.
781010	782150	But the problem with that is
782150	784170	that this channel was way
784170	785170	too noisy.
785290	786550	Every single day, it was
786550	787970	posting all the vulnerabilities
787970	790630	that were found, and they
790630	792090	haven't been remediated yet,
792090	793630	so it would repeat a lot of
793630	794550	the ones that we'd seen the
794550	795350	day before.
795350	796470	Maybe we didn't want to ignore
796470	797890	them yet because they were
797890	798970	still relevant or we were
798970	799770	still trying to find somebody
799770	800810	to fix them.
801010	802670	But it was a very long list,
802670	803830	and so we couldn't really
804610	806970	work to work them down
807470	808590	in this format.
810030	811730	And also, blindly throwing
811730	813670	them into a channel made it
813670	815310	so it was everybody's problem,
815310	816670	which in practice made it
816670	817750	nobody's problem.
817750	819150	Nobody took initiative to
819150	821250	see which vulnerabilities
821250	824670	were actually a threat to us,
825290	827050	and nobody was taking
827050	828490	responsibility to fix them.
828490	829610	So it was pretty ineffective
829610	830890	at solving our main problem
830890	831890	at first, so we had to go
831890	832770	back to the drawing board
833310	834890	and figure that out.
836170	837250	So we had to admit that
837250	839490	Ossify, you know, was an
839490	841590	effective tool, but it was
841590	842330	not only reporting
842330	843690	vulnerabilities, but it was
843690	844990	also acting as an internal
844990	846230	ticketing system.
846810	847690	For something we wanted to
847690	849810	maintain long term, we
849810	851030	wanted Ossify to do one
851030	852070	thing well, find
852070	853330	vulnerabilities and report
853330	856190	them. And to do this, we
856190	857190	had some prior art that we
857190	858750	could work from to figure
858750	859590	out the best way to do that
859590	860410	at Slack.
861210	863170	And that answer was Jira.
864330	866330	We have a Jira system, we
866330	867810	have security tickets that
867810	869090	go into our Jira that can
869090	870430	be actioned upon by
870430	873030	developers. And so our
873030	874730	solution was to put it in
874730	877110	Jira. At Slack, we have a
877110	879610	method already for triaging
879610	881090	security-based tickets that
881090	882610	come from multiple sources,
882610	884190	say if we have bug bounty
884190	885570	reports or internal
885570	887750	findings. Our SLAs, or
887750	889050	service-level agreements,
889050	890770	are 180 days for low
890770	892270	tickets, 90 days for
892270	893870	medium, 30 days for
893870	895230	highs, and then seven days
895230	896930	for critical findings.
897690	899030	So pushing, upgrading
899030	900350	these vulnerabilities to a
900350	902430	ticket in Jira puts these
902430	904210	already into the processes
904210	905850	that our developers know.
905850	908350	It doesn't add in any more
908350	910330	friction. And it puts them
910330	911410	into an established
911410	913230	workflow that also ties to
913230	914910	company-wide objectives.
914910	915770	So upgrading these
915770	916530	libraries just can't be
916530	917290	another thing that
917290	918270	developers ignore. They
918270	918970	have to do it to meet
918970	920130	their OKRs.
921170	922350	So now that we had a new
922350	924410	flow, we had a new way of
924410	925670	dealing with and finding
925670	927210	these vulnerable libraries.
928970	930970	So our new flow is that
930970	932110	Ossify would find the
932110	933510	vulnerability and then
933510	934690	throw it right into Jira.
934690	935650	It would file a ticket.
935650	937130	If multiple vulnerabilities
937130	938130	were caused by the same
938130	939570	package, we would roll
939570	940390	those up into the same
940390	942490	ticket. Say you have a
942490	944070	couple of highs and a few
944070	945470	mediums, but that all would
945470	948110	be remediated by upgrading
948110	948230	a single package to a
948230	950650	certain version, that would
950650	951550	be one ticket. So we
951550	952570	weren't making multiple
952570	953350	tickets for multiple
953350	954590	vulnerabilities per se.
954590	956030	Just if you upgrade this
956030	956950	library, these are all
956950	960570	soft. Now we can talk
960570	961850	about our triage process a
961850	963350	little bit. So the first
963350	965050	step in this would be that
965050	966150	the ticket would be filed
966150	967370	and the product security
967370	969490	engineer on call would
969490	971330	triage it. So our triage
971330	972650	process is just enough so
972650	973690	that we can find how
973690	974450	serious the vulnerability
974450	976170	is in our systems.
976170	977130	We would read the proof
977130	977530	of concept from the
977530	979430	database and then check
979430	980410	our source code to see if
980410	981610	we are in fact using those
981610	982710	vulnerable functions.
982710	983810	Because if we weren't,
983810	984790	then it wouldn't be a
984790	985710	critical vulnerability for
985710	987510	us. It would be a low,
987510	992470	or even maybe a non-fix.
992470	995090	If we are using that, we
995090	996450	might try to reproduce
996450	997850	the attack. Or if it's
997850	999170	used in multiple places,
999170	1000210	we're just going to
1000210	1001550	determine that it's
1001550	1003410	enough to determine the
1003410	1004790	severity for fixing. If
1004790	1006170	it's used in hundreds of
1006170	1006970	places, we can't test
1006970	1009070	every single one. So we
1009070	1010690	would then recommend it
1010690	1011990	be a medium or high
1011990	1014130	depending on what we
1014130	1015730	thought. And we're also
1015730	1016770	going to review the fix
1016770	1018670	that's in the upgraded
1018670	1020090	version, just to see if
1020090	1020910	there's anything more we
1020910	1021650	need to consider when
1021650	1022830	implementing the update.
1023330	1024310	All of this is just to
1024310	1025830	either confirm the
1025830	1027130	severity based on the
1027130	1029230	CVSS score, or adjust it
1029230	1030110	as needed.
1032570	1034930	From there, you know, we
1036970	1038390	would find the
1039370	1041070	responsible team. So
1041070	1042870	after determining whether
1042870	1043510	or not we get
1043510	1045190	exploited, we need to
1045190	1046270	find the responsible team
1047070	1048810	to fix that. That team is
1048810	1049290	then going to be
1049290	1050430	responsible for mediation
1051650	1053510	in the SLA time that
1053510	1054790	we've already determined
1054790	1056470	based on the severity.
1057350	1058490	Lastly, once the package
1058490	1059950	is updated, Ossify is
1059950	1061170	going to go in and
1061170	1062250	automatically see that
1062250	1063070	that version is no longer
1063070	1064710	used and will remove it
1064710	1065890	from our list. And
1065890	1066970	therefore, the
1066970	1067770	vulnerability is
1067770	1072220	remediated. Now that we
1072220	1073600	have a working process,
1073600	1075100	and that fits into what
1075100	1075920	we're already doing at
1075920	1077280	Slack, there's still a
1077280	1078260	lot of future work to be
1078260	1080660	done. Right now, we're
1080660	1081480	working on getting a
1081480	1083120	clean state for vulnerable
1083120	1085220	packages, and working
1085220	1086400	down our backlog to
1086400	1087280	determine if the packages
1087280	1088460	are currently being...
1088460	1089420	that are currently being
1089420	1090240	flagged are vulnerable
1090240	1093420	and get them fixed.
1093420	1094620	Next, this process, as
1094620	1095720	you can tell, involves a
1096160	1098520	manual work still. There's
1098620	1099320	a lot of manual work for
1099320	1100020	the product security
1100020	1101300	engineer to find out who
1101300	1103420	owns the vulnerable
1103420	1104760	dependency and also
1105300	1107000	seeing if it's exploitable.
1107120	1107980	We would like to figure
1107980	1109160	out a way to automatically
1109160	1111260	figure that out instead
1111260	1112200	of having that be a
1112200	1113240	manual process. That's
1113240	1114820	just going to be looking
1114820	1116820	into how we can track
1117860	1119020	when new libraries are
1119020	1121080	added. And thirdly,
1121080	1122500	we'd love to integrate
1122500	1124820	this process into our
1124820	1125700	CI pipeline. So
1125700	1126440	packages that are
1126440	1128700	vulnerable aren't uploaded
1128700	1131460	to our code base when
1131460	1132520	they're committed. So
1132520	1134940	on commit, it would run
1134940	1136060	the scanner, look for any
1136060	1138300	open vulnerabilities, and
1138300	1140080	then either block that or
1140080	1140860	let it through if there
1140860	1144100	aren't any. Thank you so
1144100	1145960	much. Special thanks to
1145960	1147640	Matt, who did ossify
1147640	1148820	development for us, and
1148820	1149980	Nikki and Oliver for a
1149980	1150880	previous version of this
1150880	1152200	talk. If you have any
1152200	1152780	questions, we'll be in
1152780	1154180	the Discord chat to answer
1154180	1155180	them. Thank you.