Well, without further ado, let's welcome Jonathan, who's a first-time speaker. Jonathan.
Thank you, everybody. So the talk that I'm going to give here in front of you all is Scaling
the Security Researcher to Eliminate Open Source Security Vulnerabilities Once and For All.
So, yeah, let's get started. Thank you all for coming, and thank you for being here this early.
I know that I had a hard time waking up. I can imagine some of you did as well. Okay. So my name
is Jonathan Lightchew. I'm a software engineer and software security researcher. I'm the first
ever Dan Kaminski Fellow at Human Security. I'm a GitHub star, a GitHub security ambassador, and
you can find me on jlightchew on Twitter. You can also find me at the same handle on GitHub.
Thank you.
Yes.
Oh.
Everybody can see your second slide.
Oh, interesting. One moment, please.
Got a logo.
Got a logo?
Yeah.
Yeah.
Yeah.
Yeah.
Yeah.
Just coming out. The same thing you're seeing here is the same thing they're seeing up there.
Yeah.
Sorry. And that dude walked out. I tried to tell him when he was up here. Okay. It's extended.
Maybe just because we changed it mid-stream. Maybe if you try reloading it, it'll work.
Okay.
It's the one there.
No.
Try it again. Do this.
Here. Hold on. Hold on. No, just go back.
Yeah.
Can you take us back to screen?
SDI.
No, no, no. I got you.
Awesome.
All right.
Let's take this from the top. All right. So, yes. Welcome to Scaling the Security Researcher to Eliminate Open Source Security Vulnerabilities Once and for All.
My name is Jonathan Lightchew. I'm not going to run through all this again. But, anyways, yeah, you can find me on Twitter, NJLightchew. My DMs are open.
Also, my co-presenter was not able to make it to DEF CON, but his name is Patrick Way, and he's from Modern. And you can also reach out to him on Twitter.
So, a little bit of a story.
A little bit of a disclaimer. I will be...
First off, I'm sponsored by GitHub. I have to say that legally. And also, the talk...
This will discuss a SaaS offering that is sold to customers or to companies. But all the technologies and tools that are discussed here are available for free for open source.
And you can use them as security researchers to find these tools without paying anything. So...
And fix vulnerabilities as well.
So, this talk and the work that I've been doing is sponsored by GitHub.
And sponsored by the new Dan Kaminski Fellowship.
For those who don't know, Dan was a security researcher who was known for a famous vulnerability in DNS back in 2008.
I, sadly, never got the opportunity to meet Dan.
He was known as a very, very kind person and a very, you know, people in this community.
He was a cornerstone of this community.
And the Dan Kaminski Fellowship was created to celebrate Dan's legacy and memory.
And the Dan Kaminski Fellowship was created to celebrate Dan's legacy and memory.
It's funding open source work that makes the world a better and more secure place.
HUMAN is currently accepting the 2020 applications for the next Dan Kaminski Fellow.
So, if you're looking to...
If you have a project that helps improve the security of the Internet and you're looking for a yearlong way to finance that as a project that you do full time, consider applying to the Dan Kaminski Fellowship.
So, my research and this work started with a simple vulnerability.
This.
this. This was in my company's build.gradle file. I was curious how this ended up there.
It was the use of HTTP to resolve dependencies. And the reason this ended up there was because
I had copied and pasted it from an open source project. And the reason that using HTTP instead
of HTTPS to resolve your dependencies in a build tool like Gradle or Maven is because
it's important because you can have your build tool or the jars that you're downloading compromised
in flight as they're getting downloaded with an attacker in the middle attack. And so this
vulnerability wasn't just in Gradle builds. I found that it was also this is an example of it
existing in Maven builds. This is where the jar that you're downloading gets downloaded and used
in the compiler and test dependencies, which means it's actually executed on the developer's
machine. And this is where it appears in the upload of your artifact as you're releasing the
artifact, which usually includes credentials. And this is where it appears in the upload of
credentials as the upload. So of course you're leaking the credentials to upload artifacts as
part of your build process over plain text. And this vulnerability was everywhere. It impacted the
organizations like Spring, Red Hat, Apache, Kotlin, JetBrains, Gradle, Jenkins, Groovy,
Elastic, Eclipse. And on top of that, it impacted Oracle, the NSA, LinkedIn, and Stripe. All of
the open source projects from all of these different organizations were impacted by this
vulnerability.
So I reached out to Maven Sonotype. Maven Sonotype is the equivalent of PIP for the Python
ecosystem and NPM for the Java ecosystem. And they said that when they looked at their traffic in
2019, 25 percent of their traffic was still using HTTP to download their dependencies. How do we
fix this? So on January 15, 2020, I pushed forward an initiative that decommissioned the
support for HTTP in favor of HTTPS only across the major artifact servers in the Java ecosystem.
and reached back out to Maven Sonotype in January 2020 and they said even after the blog posts,
the discussion of this vulnerability, there were still about 20 percent of their traffic that
they were seeing using HTTP instead of HTTPS. So you can imagine what might have happened on
January 15, 2020. Broken software everywhere. There's a lot of Stack Overflow posts saying what
happened to my build? But we stopped the bleeding. However, what about the other repositories?
These are only the most commonly used repositories in the Java ecosystem. Maven Central, J
Center, Spring, and the Gradle Plugin Portal. Other companies host artifacts and you see them
in the builds of companies that are using JavaScript. You see them in the builds of
companies or builds across the ecosystem. Unlike NPM or PIP, you can declare multiple
repositories to resolve your dependencies from. And so you'll see that there's some builds that
rely upon five, six, seven, eight repositories to resolve their dependencies across the Java
ecosystem. So how do we fix the rest? And I said, well, let's go to the root of the problem and
let's just generate pull requests. Let's fix it this way. So how did I do this? The first thing
that I needed to do was I needed to identify the projects that were vulnerable. You can know the
vulnerability exists, but like how do you get a list of projects to go fix it in? And so I used
CodeQL. This is the query. I wrote this query to find this vulnerability in Maven POM files. It's
very simple. That's it, right? It's very short. And the reason that I use CodeQL is CodeQL scans
hundreds of thousands of open source projects on every commit. And you can write it as it's
building this, as you're building the software, as it's building, you can write the project on a
They extract the ASTs and they build a database of the code that you can write queries against.
And GitHub also has a bug bounty program called the GitHub security lab bug bounty program.
There's actually two bug bounty programs under it.
And for this very simple query that I wrote, GitHub bountied me $2,300 for this.
You know, not much code, little bit of documentation.
And so using the list of vulnerable projects that I was able to retrieve from running this query across the corpus of projects that CodeQL indexes,
I wrote a pull request generator.
It was a Python-based wrapper over hub, which is GitHub's hub CLI.
It had one nasty regular expression and a lot of logic for bouncing off of GitHub's rate limiter.
And this is the logic.
There's an underlying engine, but at a high level.
This is the code.
This is what allowed us to generate the commits and the pull requests.
And this is the regular expression that we used.
And you might ask, why use a regular expression?
We have XML parsers.
And the reason that we use regular expressions is because if you're trying,
when you parse XML into an XML parser and then you make a modification and then you dump it back out,
all XML parsers have a standard output.
And so you'll end up generating massive work.
You'll end up getting a lot of white space diffs across all of the files you're changing.
And the maintainers will receive that as a pull request.
And they'll be like, I'm not accepting this.
Sure, you fix a security vulnerability.
But, like, it doesn't look like the code that I have.
It doesn't match the editor format that I have.
So, unfortunately, you have to use a regular expression to fix the issue.
But the problem with regular expressions is that if you use regular expressions, now you have two problems.
Yeah.
But it worked.
This is my GitHub feed for after I did this.
You know, just tons of pull requests, all of the same thing.
And this is an example of the diff.
You know, you can see the replacement of HTTP with HTTPS.
And for this campaign, I generated 1,596 pull requests.
And as of today, or about a week ago, we have about a 40% merge rate for this.
And for engaging in this campaign and doing this,
in addition to the original pull request,
GitHub additionally bountied me an additional $4,000 for this work.
So, thank you.
So, I got hooked on bulk pull request generation.
I got hooked on this idea that, like, we could actually fix these vulnerabilities at scale across open source.
This is my GitHub contribution graph for 2020.
You can see I actually engaged in two campaigns this year.
And, you know, you can see the actual impact of, like, you know,
generating it as what it has on your contribution graph.
So, I have a problem.
I have ADHD.
And that's not my problem.
Yeah, so ADHD is not my problem.
The problem is that I love reading vulnerability disclosures.
I go through them, the GitHub security advisories feed.
And I can look at advisories and I can read the vulnerability and I can be like,
okay, I wonder where else this exists.
And so I'll run a code code query.
Or I'll do a GitHub code search search.
And the problem is I find too many vulnerabilities.
There's more vulnerabilities than I can reasonably report as a security researcher.
This is an example of a code code query for zip slip.
I can scroll through pages of these results across open source.
In fact, GitHub gave me a list of 900 projects that were potentially vulnerable to zip slip.
So, if I'm finding too many vulnerabilities, how do I fix this?
How do I, like, I can report one by one.
I have to pick out which projects are worth reporting to.
Or I need automation.
So, now I want to discuss with you a tool that has become something that I've really fell in love with
because it's allowed me to do this.
Automated accurate transformations at a massive scale.
And so I want to introduce you the tool Open Rewrite.
And Open Rewrite, it started out at Netflix.
It was written by Jonathan Schneider.
And it was developed because Netflix, every single team at Netflix,
was given and enabled by the organization to write code in whatever format, style they wanted.
As long as they owned the code, they were allowed to do whatever they wanted to develop software.
And so if you need to update, for example, Google Guava,
which traditionally has a breaking API changes every time they release a new version,
how do you get your entire organization up to date and using the latest version of a dependency
the latest version of a dependency
that has a breaking API change every time you update?
And so the answer was you need to not just update the version dependency,
but you also need to update the code to fix, to update.
And so Open Rewrite was created.
The traditional problem is that if you compile code,
you'll see that the compiler produces what's called an abstract syntax tree.
It's a representation of the code in a tree format.
But the problem is, you'll notice,
that if you grab this tree,
if you wanted to dump it back out into text,
you lose the formatting, the white space, the tabs,
because the compiler doesn't care about that information, right?
And so Open Rewrite, instead of,
it builds, it uses the underlying AST that the Java compiler uses,
but it also captures all the other stuff you need,
the white space, the tabs, the spaces,
into the tree, and preserves it,
so that you can take this tree,
and it'll transfer back and forth to the source code.
So the white space and spaces are preserved.
And so the generated, and you can also generate new code,
because as the compiler, or as the,
as Open Rewrite is capturing the AST,
it's also capturing the tabs, the spaces, the format,
do they use, you know, does the developer use braces on new lines?
So it can capture all of that,
and as you're generating new code,
also generate it in a way that looks like the developer's code around you.
Or the developing code around you.
And on top of that, it's also type attributed.
For example, this bit of code, if you just saw this,
how do you know if this is log4j, slf4j, logback?
They all have exactly the same API,
so you need type attribution across the tree
to also be able to make intelligent decisions
about how you're going to fix these vulnerabilities.
And Open Rewrite's AST is,
it's syntactically and semantically aware.
You can see that,
just the syntax alone,
when you have type attribution,
you get a much more rich graph
that lets you make much more intelligent decisions.
Actually the image on the right is missing about 6,000 lines
because it would just be a mess.
You wouldn't even be able to see it.
And then on top of that,
if you're trying to replace code,
you need to be able to take,
you need to be able to take new Java code
and put it into the AST
to fix these vulnerabilities.
And so you need a way to generate new ASTs
in an easy way.
As an example,
for ZipSlip,
which I'm going to talk to you about a little bit later,
we need to inject this bit of code
into the surrounding code.
And so how do we do that
in a way that's easy
and you don't have to write a ton of code
to generate a bunch of graph nodes
to inject into the tree?
And so Open Rewrite comes with a templating engine
that lets you just put a string in your code
and the templating engine
will generate new code
that matches the surrounding style
of the developer's code
that you're working with.
And then if you want to inject that into the code,
it has a coordinate system that says,
place this chunk of code
after a certain location in the code.
And that's all that's required
to fix this,
you know, to add this little bit of code
to fix ZipSlip.
And so it lets us take,
you know, transform code like this
and add that fix
where it's required.
So what's possible now
what other vulnerabilities can we fix
with this unlock that Open Rewrite provides us?
I'm going to talk to you about three different vulnerabilities
that I tackled with Open Rewrite
to fix widespread common security vulnerabilities
across open source.
The first one is called
temporary directory hijacking.
The second one is partial path traversal.
And the third one is called ZipSlip.
Let's start with temporary directory hijacking.
The system temp,
so the basis of temporary directory hijacking
is that the system temp directory
on Unix-like systems
is shared between all users.
So this is temporary directory hijacking.
And the reason that you'll find this code
in lots of open source
and lots of closed source projects
is that prior to Java 1.7
there did not exist a public API
in the Java standard library
to create a temporary directory.
So what people did is they created a temporary file
that uses a CSPRNG
to generate a temporary file name.
So it's guaranteed to be uniquely random
and non-colliding with any of the other files.
And they called delete
and they called mkdir.
And the reason this vulnerability is so prevalent
is because if you looked up on Stack Overflow
how do you create a temporary directory,
you'd unintentionally get a vulnerability.
And so why is this vulnerable?
This is vulnerable because
there's a race condition here.
You can have another local user on this
system waiting for the deletion call
and then mkdir
when you call mkdir
you'll
lost my place
when you call mkdir
there's a race condition between the Java process
and the attacker to create that directory first.
Because if the Java process fails
or if the Java
if the Java process is beaten
instead of throwing an exception
it returns false.
So what's the fix?
Well, this is a simple fix, but it's imperfect.
It does fix temporary directory hijacking
but the problem is that the mkdir
will still use the default uname permissions
when it's creating the directory
and so the directory will be exposed to all
will still be visible to all local users.
And so instead of temporary directory hijacking
you have temporary directory information disclosure.
So what's the true fix?
This API was introduced in Java 1.7.
Java 1.7 is very old.
It's, I think, end of life even now.
So this API has been around for a long time.
You can safely use it.
And it lets you create a temporary directory
that also sets the POSIX permissions
on the file correctly to be secure
and not visible to other local users.
And so I got a bunch of CVs
for this vulnerability.
I have a lot of history of getting vulnerabilities
but for this one I actually reported it to a bunch of projects.
But again, I had this problem
where even though I reported it to all these projects
I still had
there were still more vulnerable projects
than I could reasonably deal with.
And so I said, alright, let's generate some pull requests.
So for this fix
I generated 64 pull requests
to fix this across open source.
And there are probably more possible on top of this
but for now this is as far as I've gotten.
And I plan to do more of this in the future.
But still,
even consider this.
Reporting to 64 projects
each time, like each one being reported
takes a lot of time.
Right?
So multiply, you know, my time times 64 projects
it's a lot of time spent
that this has saved.
So second vulnerability
actually I wanted to, so the example
this is an example of the diff generated
and you can see in this change
where, you know, we've deleted the delete
and the make dir
and we've put in place
this fixed code.
And here's an even more complicated fix
you can see that we've deleted the if blocks
that are no longer required
and put in place that single line of code
that's now required.
So the second vulnerability
that I want to introduce you all to
is called partial path traversal.
So partial path traversal
let's assume that you have two users
on a local system.
User Sam
and user Samantha.
And you want to isolate your code
to only allow an attacker to access
user Sam.
Partial path traversal allows an attacker
to access a sibling directory
with the same prefix.
So for example you have user Sam
and user Samantha.
This vulnerability exists because
Sam is a prefix of user Samantha.
And this is the vulnerable code.
It's vulnerable because
when you take a file
and you call
get canonical path on it
it returns a string
and that string, you'll notice
has the, it also
it normalizes the path
so it removes the dot dots
you know, any path traversal payloads
but it also drops the trailing slash.
And so when this code ends up
being used in a guard
you can see that when you take user Sam
and then you have an attacker supply
user Samantha
when get canonical path gets called
and the path gets normalized
that starts with
check passes
and so the exception doesn't get thrown.
And so what's the fix for this?
Well the fix
looking back at the vulnerable code
is to
the first fix you can do is
you can just add the slash back in.
However the better fix for this
is to use
the Java path API
which is instead of using
get canonical path use get canonical file
and then call to path on it
which uses the Java path object
and when you call starts with on it
it does path comparisons instead of
string comparisons which is going to be safe.
So how do we find this vulnerability?
Well
first we're looking for a string
starts with call
but then we need to look for before and
inside of the arguments
a call to get canonical path.
And then on top of that
we also want to look for cases
when we're trying to fix this vulnerability
we want to look for cases where
the file separator is not present
because if it is present you don't want to fix
at the core you don't want to fix a vulnerability
that's not there.
But it can't be that easy.
What if
a developer because developers are going to write code
in a lot of different ways
what happens if a developer extracts that code
into a variable
or what if they pull
that argument into a variable
or they have the fix in a variable
we need something new to make it possible
to fix this.
So we need data flow analysis.
Data flow analysis allows us
to track the
assignment of values
through the program and see what the
final value will be at run time.
So we can see that dir canonical
is being assigned from that
get canonical path call.
And additionally you can do that
for other things and it lets you do this
for intermediate steps that may be
even more complicated than this.
So data flow allows us to uncover
hard to find vulnerabilities and prevents
false positives.
And this is the API for data flow for open rewrite
something that I developed based upon
CodeQL's data flow analysis API
so if you learn CodeQL
or you learn open rewrite you can translate
your knowledge very easily back and forth
between these two paradigms
which makes fixing these vulnerabilities
and identifying these vulnerabilities
significantly easier.
So by putting it all together
we're able to replace this
vulnerable call with
the safe version.
I want to give you a case study
on this vulnerability.
An example from the AWS
Java SDK.
It has a CV number.
I found this vulnerability in the Amazon SDK.
It was for the
transfer manager which is used to download
the contents of AWS buckets.
And they had a check in there to prevent
path traversal if the AWS
bucket key was potentially
a path traversal payload.
And so you can see this
logic was vulnerable to
partial path traversal because
this leaves root logic was used in this guard
to check if the
AWS S3
key was outside
of a path. And so we got a vulnerability
for this. The Amazon security team was
very pleasant to work with. However with any
good story there was a little bit of vulnerability disclosure drama.
And this story
I had to throw it in there. It was too funny not to share
with you all.
This is my email conversation with the
Amazon web services security team.
We'd like to award you a bug bounty for this.
However you need to sign an NDA.
And I said I don't normally agree to NDAs.
Can I read it first
before potentially agreeing? And they said
we're unable to share bug bounty program
NDAs since it and other
contract documents are considered sensitive by the
legal team.
.
Yeah.
So
yeah.
Again, too funny not to share with you all.
I have asked
the Amazon web services team instead of
bountying me to double the bounty amount and
just donate it to the electronic frontier
foundation. They have yet to get back to me
on that particular front. So
we'll see what happens.
Yeah, right? Thank you.
So the third
vulnerability I want to share with you all is
ZipSlip. So ZipSlip is an old vulnerability.
It's been around since Frack Magazine
many, many years ago.
And the
SNCC team actually did a bunch of research about
ZipSlip a couple years ago and
found a bunch of projects that were unzipping
logic that were vulnerable to this.
ZipSlip fundamentally is
not a partial path traversal vulnerability
but it's a path traversal vulnerability
while unpacking
zip files. However, partial path
traversal can exist
in ZipSlip. And some of the cases that I
found that the SNCC team reported to
when I looked at the code again several
years later, I found that that code, even
though it had been patched against path traversal,
it still was vulnerable to partial path traversal
which we discussed earlier.
And so this is the vulnerable code for ZipSlip
in Java. And the reason
it's vulnerable is because you have
the name of a zip
entry flowing to the
creation of a file, which that file,
that get name can be a path traversal
payload and that flows
to the creation of a file output
stream. And so
if that's a
path traversal payload, then you
can overwrite the contents of
an executable file and then you can get code
execution. So ZipSlip, although
not directly, it can lead to code
execution if you overwrite the right files.
And ZipSlip is complicated
and the reason that it's complicated
is that in order to fix it,
you need to add this chunk of logic
to the code. However,
the problem with ZipSlip in this case
is that
this is a valid fix to this vulnerability
and you'll find this in open source code.
But this is also a valid
fix. So how, again,
we don't want to fix non-vulnerable code.
So how do you identify
is this vulnerable or is this not vulnerable?
And so we need something new
that we haven't talked about yet. It's called
control flow analysis.
Control flow analysis lets us
determine the difference between
the vulnerable code on the left
and the non-vulnerable code on the right.
So control flow analysis,
what you do
is you walk over the AST
and you build a tree
or graph. It's a graph
of the connected
nodes where you have basic blocks
which are the set of contiguous operations that are
going to occur in order
followed by any
of the conditional nodes that exist in the logic.
And so looking at partial
path traversal,
sorry, looking at zip slip,
you can see that this
is an example of the
control flow analysis graph built for this
non-vulnerable code.
And so we're able to traverse this graph
in our logic
and determine that, oh hey look,
you know, there is a guard in place
that when it's false
throws an exception.
So we're not reaching the vulnerable code
where we're creating that new file output stream.
And so
when we put all this together,
we're able to not fix
code that's not vulnerable, and we are able to fix
code that is vulnerable, which is what we want to do.
And this is an example of the
diff generated to fix this vulnerability.
And here's another one where
not only have we fixed the vulnerability, we've also cleaned up
the code a little bit around it.
And so I want to talk to you next
about actually doing pull request generation.
Because if you've got security vulnerabilities,
everybody gets a pull request.
So
there's a problem with pull request generation.
And one of the problems is, how fast
can we generate pull requests?
So as a part of
generating pull requests, there's three
major types of operation you have to make.
File IO, Git operations, and
GitHub API calls.
The first one is pretty free.
The second one, Git operations against GitHub
are completely free as well.
GitHub does not rate limit it, and GitHub has a rate limit
which is very annoying
on their API calls.
So the first one you need to do, check up and download the code
repository. Then you need to branch,
apply the diff, and commit the change.
Then you need to fork the repository,
and rename the repository on GitHub.
And the reason this is important is because when you're
forking hundreds
or thousands of repositories,
you're going to end up with name conflicts.
And so if you don't rename them,
you're going to end up saying, GitHub's going to come back
and say, there's already a fork with that name,
so you have to rename it. And unfortunately,
GitHub does not offer an API to do
both of those steps in one
API call. And then you
push the changes, and then you create
a pull request on GitHub.
And so you'll notice that there's
three API calls here that are rate limited.
And GitHub asks that
you wait at least one second per user,
per request.
So that means that for every pull request you're generating,
you're going to have to wait at least three seconds.
And on top of that,
there's additional secondary and tertiary
rate limits that they document
with mixed
success.
And that you have to be aware of
and just deal with.
So,
if anybody from GitHub is in the audience,
it would make my life a lot easier
as a security researcher if you just backed off a
little bit on your rate limiting.
So, we've made it this far.
We've detected the vulnerability,
the style's been detected,
the code's been fixed,
and a diff's been generated, and the rate limit,
although annoyingly,
has been bypassed.
How do we fix this for all the
repositories across GitHub?
And I want to introduce to you
Modern.
Modern is a free for open source
projects SaaS offering
that currently indexes
over 7,000 repositories.
And it lets you run open rewrite
transformations at scale.
And it lets you generate and update pull requests.
Modern
has over 800
open rewrite recipes, including
complete framework migrations.
And the reason that framework migrations are important is
take, for example,
Spring.
If you want to use the latest version of Spring,
unfortunately, the latest
version of Spring requires
the latest version of JUnit,
which is a testing framework.
There's JUnit 4 and JUnit 5.
And there's a completely different API between the two of them.
And so, if you want to be in the latest version
of Spring, which, as security practitioners,
we've probably all heard about why keeping your
version of Spring up to date is important.
You also need to keep your
test framework up to date.
And if you want to migrate all of your code,
you have to migrate your entire code from
JUnit 4 to JUnit 5 to be on the latest version of Spring.
And so, even your
indirect test dependencies can be
a security risk if you can't update
them, because it'll prevent you from updating the
critical components like Spring.
So, open rewrite has transformations that'll let you
upgrade those things very, very
easily with just a click of a button.
And on top of that, it also supports
pull request generation.
And this is an example of, if you
found the vulnerability in open source,
you can generate pull requests at scale
using this technology.
And so, this is a quick video of
when I wanted to fix
temp directory hijacking.
What you're able to do, you can commit the
chain, it lets you commit with forks,
you can set the commit, the branch name,
the commit title,
it lets you create the pull request with
GPG key signing.
Yes, I know I'm pasting my GPG key into a SaaS
service, but it lets you generate your
commands with GPG keys and then generates the pull request.
And this is the end result, right?
Actually generating pull requests.
But,
I said open,
modern indexes 7,000
repositories. There are more than 7,000
open source repositories in the world.
How do we find the other projects?
And so, looping back to the beginning,
CodeQL. So, CodeQL
lets it, there's over
100,000 open source projects that they index
and over 35,000 open source
Java projects that are indexed.
And so, when you write a CodeQL query
to find these vulnerabilities,
you can get a list of all the vulnerable open source projects
and then if you
contribute that list to the
modern Jenkins ingest
repository to
the CSV, they will
ingest
those repositories and let you run
recipes against them.
So finally, we've finally
gotten there, let's go generate some
pull requests.
And so,
here's the results of all the different
campaigns of open source pull request generation
that I've been involved in.
The,
one of them actually was not
directly mine, the our host name
array overflow was a project
done by GitHub
using my bot, my Python based bot.
But yeah, for
this past year, I've generated 64 pull
requests for temporary directory hijacking, partial
pass reversals 32, and zip slip
100 pull requests.
And for
me, I've generated
over 590
new pull requests in this year alone
to fix various different vulnerabilities, new and
old projects. And personally,
I've generated over 5200
pull requests across my history of being
a software security researcher.
Thank you.
One
unfortunate project, um,
of the three projects that I engaged in
as part of the Dan Kaminski fellowship,
was the unfortunate recipient of all
three pull requests.
Um, this is actually
a project that's owned by
Perforce.
They didn't respond immediately when I opened the pull request,
so I took to Twitter and said, hey, by the way,
I dropped ODE on you, sorry.
And they are now aware of it.
Um,
and so this
is the, this is, you know,
my contribution graph.
Um, you know, actually have an impact in open
source.
So, I've talked about the technology,
but now I want to talk to you about some of the
best practices for this technique.
First off, messaging.
You're dealing with real people,
you're dealing with real maintainers,
um, there's this great saying,
all software problems
are people problems in disguise.
This is very true for this case.
You're trying to fix these vulnerabilities
with technology,
but dealing with a real person
on the other end of this, which is receiving
this pull request, and so you need to be kind,
compassionate, you need to understand that
you're potentially dropping
an ODE on them,
and maintainers are used to bug reports,
like that's been normalized,
but when you're
challenging them with a security vulnerability,
um,
sometimes ego gets
involved a little bit, and that's not a bad thing,
but people are not normalized to, hey,
you've got a bug in your code, but not only is it a bug,
you've potentially put your users at risk because
of it. So you need to be conscious of that
in your messaging, and detail the vulnerability,
and be compassionate when you're communicating with a maintainer.
And so,
some shorter lessons. First one,
sign off on all your commits.
Um, this is what a
git commit sign off looks like,
and you might ask, why?
Um, and it was,
there was a bunch of lawsuits in open source,
TLDR
lawyers.
So, um, if you don't want your
pull requests rejected by evil dragon
bureaucrats, uh, consider
just, just gpg, or just, uh, just commit
sign, and you'll be good.
Um, be a good committison, gpg
sign your commits, um,
this is what commit signing looks like, and, uh,
you won't end up getting impersonated
like Linus Torvald has multiple times
on GitHub. Um,
uh,
ccom, so ccom
is a standard for, uh,
the commit message, um, I'm not gonna
go into details on it, but, and it's not
my talk to give, but, um, it fundamentally
it's a format for commit messages
that fix security vulnerabilities, and if you follow
this, it just helps us fall into a
more normalized, uh, way of
communicating about vulnerabilities in the commit feed.
There are risks
to using your GitHub account.
Is anybody here familiar with GitHub's
angry unicorn?
This is my GitHub profile for most of 2020.
I broke
my GitHub account by doing this.
Um, so, be warned.
However, I
do recommend using your personal GitHub
account, um, because when maintainers
have questions, concerns, um,
rants, sometimes, not often,
um, you'd want to be able to
respond with kindness and compassion and
understanding, and it'll make them
understand this is coming from a person
who cares about trying to fix vulnerabilities
and trying to improve their software,
but just can't do it in a way
that's private or the problem
is too big to do this at, you know,
report it to them one by one by
one. And then,
coordinate with GitHub.
Um, before attempting, reach out to the
GitHub security lab. Let them know you're gonna do
this because you don't want to get banned from GitHub.
Um, keep them in the loop.
They want to know. They're probably, they're gonna be,
they want to enable you. They want to support you. That's what
I've done. Um, I have a really good relationship with
them. Um, so just keep them in the loop.
And then, consider the implications.
I received this issue shortly
after engaging in this campaign,
one of my recent campaigns.
And I use the term coordinated disclosure.
Um, I, a lot of people
still use responsible disclosure. It's an outdated
term. Um, but coordinated
disclosure, regardless of the term
you use, the answer is no.
This is not responsible disclosure.
This is not coordinated disclosure.
It's full disclosure of a vulnerability.
If GitHub offered a way to create pull requests
that were private, I would use it.
100%. I've asked
them for it. Um,
uh, so you need to
be considerate of, you know,
that you're dropping an Oday on a maintainer,
and you're gonna add stress to their life
that they didn't have before.
And now they've gotta deal with what you've given them.
So just keep that in mind.
But I argue that
at the scale at which the vulnerabilities
we have, this
is the great, is it, it's the
risks and the downsides outweigh
this potential
negative, you know, the negative consequences of
dropping Oday because of the amount of good
we can have by fixing these vulnerabilities at scale.
So
in conclusion,
as security researchers,
I believe we have
an obligation to society.
We are the ones that know
that these vulnerabilities exist.
We've seen them in pen, we've seen them in
pen test reports, we've written about them,
we've seen them in code reviews,
they come up again and again, but we are the ones that
know that these vulnerabilities exist,
and we know how to fix them.
Unfortunately, a lot of
developers out there,
surprise, surprise, don't watch DEF CON or Black Hat.
Talks.
So,
we need to be able to, we need to help
the community where we can
and contribute back.
There's a problem, GitHub
estimates that for every 500 developers
you've won security researcher.
The odds are stacked against us.
But I believe
that with this technique
and with this, with deploying this,
we can fix vulnerabilities
at scale.
And I believe this is the best way to fix
vulnerabilities at scale across open source.
And imagine deploying this technology
for, for example,
to eliminate SQL injection across open source.
Or if, when open
reports C and C++.
We could eliminate entire classes
of memory corruption bugs.
Simply by generating tens,
hundreds of pull requests across open source.
Right? Just gone.
We've dealt with that vulnerability.
Let's move on to something more interesting.
So I want to leave you
with one final quote.
It's from Dan Kaminski.
It's on his Twitter profile today.
We can fix it.
We have the technology.
Okay, we need to create the technology.
Alright, the policy guys are mucking with the technology.
Relax.
We're on it.
Thank you.
A couple of little sound bites.
Learn CodeQL.
It's seriously, it's an incredibly
powerful language.
Contribute to Open Rewrite
and CodeQL.
And you can deploy your security
fixes at scale.
And then if you want to chat more with me,
my Twitter DMs are open.
Join the GitHub Security Lab Slack channel
and the Open Rewrite Slack channel.
And if you want to discuss the wider problem,
it's not on the slide, but if you want to discuss
open source security vulnerabilities in general,
there is the Linux Foundation's
Open Source Security Foundation
where they're having weekly meetings
discussing the critical vulnerabilities
and issues impacting open source and securing
those things.
So if you're looking for a place to get involved
and is not directly involved with this,
but you want to get involved with open source security,
that's a great channel to get involved in
and learn more about this topic.
And I want to leave a final thanks to Human,
my sponsor,
and the Dan Kaminski Fellowship sponsor
who enabled me to do this.
Madern, who has been an awesome companion
and team to work with.
Lydia Giuliano for being the Black Hat speaker coach
to make this talk what it was.
And Sean, my intern,
who enabled,
he did some of the graphics
and without control flow analysis
would not have come together.
So thank you, yes.
Thank you all for coming.