All right, so welcome everybody and thank you for taking the time to see our talk.
We are happy to be here at the DEF CON Red Team Village and we are very excited to share with you
some of the findings that we've been doing since we did a talk in the previous Red Team Village
Summit where we try to showcase and expose basically how dangerous is the leakage of
credentials on the internet. And today me and Jose Hernandez, my colleague, are going to be presenting
How'd My Keys Bing Pound API Edition. So let's get going.
So most of you know who we are. I'm a Principal Security Research Engineer at Splunk.
I worked with Jose at Prolexic, which is now Akamai. I worked at Akamai for a while. Then I went to
Caspita and came back at one point to Splunk. I co-founded Hack Miami and Pacific Hacker Meetups
and Conferences and I grow my own CDFs. Some of you may have Bladen, which is the Command and Control
and of course CDF. And Jose, can you tell us a little bit about yourself? Yeah, I'm also a Principal
Security Researcher at Splunk. I'm an old longtime friend of Brock from our Prolexic days, which got
purchased by Akamai. I co-founded a company called Zenitch, which is now Oracle's Web Application
Firewall and DDoS services. And returned to Splunk to do research, security research. And this is one
of the things that we've been working on that I'm super excited about. Awesome, so let's move on.
So just before we get into basically what we're going to show you today, it is important
that we recap a little bit on how we got to this point. And in order to recap to
to a point where we can understand how bad and how we got to this very bad situation, we need
to understand or take a look at what is DevOps. So as we have approached it before and explained
it, DevOps is a set of practices that basically is within software development and team operations.
It has become very popular. It's not really new, but it has become very popular and it has been
widely adopted as most companies are starting to somehow get a foot in cloud platforms.
Some of them have radically moved most of their operations to the cloud. And when we're talking
about developing software and producing software, building, coding it, planning it, testing it,
releasing it, the first thing that comes to your mind is DevOps. And DevOps is a
set of practices usually guided by some software development principles. And the most popular
currently is Agile. And some of you might be familiar with Agile if you have worked in a
software development company. I work for now for three software development companies and
all of them were using Agile. And this is how I was exposed to the, not only the infrastructure,
but the risks that are associated with this. So next slide, please. So one of the things that
I noticed throughout the years, and me and Jose have been researching on, is that when you use
this set of principles and divide it in things that are called tool chains that go in the DevOps
process, there's a number of products and a number of tools that are constantly used, reused, shared
and repurposed. One of the characteristics of the, to say in a certain way, the platform that is
for development, software development is at times his ephemeral character, meaning for example,
that you can create containers, destroy them and then simply recreate them again. So in order for
us to give this a little more structure into what we can look at when we're trying to gauge the
disposal and risks associated with DevOps, we had to look at DevOps tool chains. And DevOps tool
chains basically are a combination of tools that aid in the delivery, development, management of
software and applications for the entire cycle. As you saw previously, there is an actual cycle
where you code, you build, you plan, release, you test it and then you go back again. And if you are
for example, in a specific methodologies, for example, Agile, usually there are sprints and
sprints are sets of times that you have to produce a number of features or bug fixes. And all of this
is based in a flow that goes through what they call the tool chains. So in the tool chains,
you can see there are things that are used for planning, such as Git or Jira. There are things
that are used for coding that involve coding, code repositories. There are things used for
testing that implicate things such as Selenium or Bagram or Docker containers, for example.
There are things that are tools that are made for software build, such as Ansible or Terraform
or Chaff. And then of course, there are things that are within the tool chains for what they call
deployment, which imply things such as Kubernetes. In Docker, for example, most of these things
like Kubernetes at times are throughout the entire process. And depending on your
cloud provider, you may be very familiar with some orchestration automation languages such as
Ansible or Terraform. And then finally, part of this tool chain is the monitoring part. And then
we have things usually the main two monitoring tools are either based on EOK, which is Elastic
Search, and of course, Blunk. Next, please. So once we have seen the picture of the two chains
that are associated with the software development, plus the cycle throughout the flow of planning it,
coding it, building it, testing it, releasing it, and then coming back,
there's one thing that we're focusing today, and that is credentials. Credentials are part of the
entire process. Credentials are needed for many reasons. And here's a little bit of some of the
highlights of what happens in this process with credentials. So developers, for example,
usually have high privilege credentials. Why? Because they had to be able to test things that
would run with low privileges. They are supposed to develop things that may be
kernel libraries, sockets that need to be created with services or connections.
These environments at times are usually ephemeral, like I said, and as a result of that,
many times they're dismissed and poorly monitored. We're seeing cases of developers
downloading anything from Docker Hub or who knows what have you, a container repository on the
internet and simply putting them in their DevOps tool chains without any checking, without any
scanning, and now you may have implanted containers, vulnerable libraries, vulnerable
operating systems that will eventually affect and be published into your production environment.
Obviously, this is related as well. There's a disconnection between development and security
operations. I've been part of this problem many times and I have interacted with developers.
Developers usually don't like when you tell them that their code has bugs or vulnerabilities.
Most of the times there's not a straight link in between what they're developing, so I'll give you
an example. You have a development department and they download, for example, all these containers
or libraries or code. Many times there are no tools to verify this. Many times there are not
inventory of what is it that they're downloading, what is it that they're using, what libraries
they're putting into the software packages. So this by itself is a risk. And as well,
this is also very popular and this is part of the nature of the DevOps process.
There's a spread use of open source tools and code. So basically, many times I notice in this
environment that this code is trusted by default, meaning they just go, oh, I know this developer.
Oh, I know this group. I'm just going to download this and use it in my application. And again,
this goes back to the disconnection, but there is some sort of an honor code between the open
source community where do no harm is always the driver of software development. But it does not
mean, and we have seen this in supply chain attacks, that malicious actors, nation states,
criminals in general, may target these repositories, these open source communities,
and embed by stuff in it. Also, embedded credentials usually end up in public repositories
and that's pretty much what we're going to show you today. How even big companies, how individuals
that... unfortunately, there is no mechanisms. All in all, we don't blame them for what you're
about to see. We don't think that they're purposely leaking these credentials. However,
it is important that we point this out in order to bring awareness that there has to be
mechanisms created in order to avoid the rampant, because there's no other name for it, this rampant
leakage of credentials. And we will show today how bad it can get. So when we have also
dev development departments and DevOps processes, where most of these developers have
high privilege credentials or permissions, there is obviously a higher risk of insider,
because it takes less of an effort to cause harm, to embed malicious stuff in it, or to even destroy
it. Like we have seen in some cases before where a person that was part of a development department
has come back or a system has come back and do harm to an employer. And another couple of points
is that due to the CSD nature, the continuous delivery, basically that cycle that goes around
building, testing, developing, and coming back to planning and on software, these things
get published immediately. That's one of the nature of the DevOps process, is the DevOps
process has shortened the time where you plan, co-build, test, release, basically becomes something
almost immediate that goes into production. And this by itself represents a risk, because when you,
like I told somebody, when you have vulnerabilities in environments that are
driven by CICD and things like, for example, implanted container, and you have very large
environments, the risk and the opportunity of exploitation increases in orders of magnitude.
So finally, and just to put this, to make this even worse, unfortunately, the cloud
environments have made this risk even higher. Why? Because basically you are connected to the
internet, you publish right away, you stage right away, and if there are attackers that are
knowledgeable and are able to pretty much footprint your process, they can do a lot of harm.
Next slide, please. So here's a little bit for you to have a reference of how
the cloud providers manage credentials. We're going to focus mostly on cloud-related environments.
So here's, for example, AWS, they have their own AIM credential service, which usually
have things such as passwords, access keys, key pairs, or SSH keys. And also they do have a number
of temporary security credentials that can be created on the go, and that sometimes have a
feature, whereas you can give a specific user access to, temporary access to a resource that
otherwise that user does not have access for. So next up. Most of the providers have had
sort of similar systems for managing credentials. However, I had to give kudos to Microsoft because
they are trying actually to tackle this problem. And as you can see here, they do have, this is an
example of how they manage credentials. They use the Azure, there's a framework within Azure
Active Directory where it tries to avoid the embedding of creds, and they use a different
mechanism. We're not going to focus on this feature of Azure in this talk, but it's important for you
to consider it and look at it because they definitely seem to be aware of the issue.
Next slide please. And then here's, we wanted to give you a little sample of basically what's
happening with the three main cloud providers, which is AWS, GCP, and Azure. In the case of
GCP or Google Cloud, they are, most of the stuff is based on OAuth, which is a
protocol that's used for identity federation and single sign-on. And for the most part,
because of the constant interaction and services that are present in cloud environments,
obviously these providers have to come up with a way to allow the interaction of either devices,
users, and services, dividing and trying to establish boundaries between these entities.
And as we will see soon, this is very challenging. Next slide. So here's a general,
that are usually used and as such, potentially exposed when you have developers that are
publishing or storing code in public repositories. So things as email and password, username and
password. Remember, there's a difference between your local, for example, Active Directory or LDAP
and the cloud identity access management. Sometimes this brings up a lot of confusion.
And depending on the integration that you have with your cloud environment, this may or may not
play in your favor. Meaning, if you're not very integrated, losing, for example, the username and
password from a specific cloud service will not allow the attacker to access your internal
environment. Also, multiple factor authentication is something that's been coming and at times it
can be bypassed by certain frameworks. We've done some work before with Evil Jinx, which basically
is able to capture the second, the authentication or the TLTP, whatever interface is presented to
the user and bypass MFA. Access keys, we talked about it. Key pairs, we talked about it.
Specific account identifiers and at times they use x.509 certificates. Next up.
So, what are the primary source of leak credentials? Well, as you will see soon,
GitHub is probably the most popular code repository on the internet. GitHub is now used for many other
things, storing files, even hosting web pages, which is kind of cool. So, GitHub is like the
reference when it comes to the leading internet code repository, not only publicly, but many
companies use it. And then we also have GitLab and we also have Amazon S3
buckets, storages, because the reason why I put this here is because you can definitely search
for Amazon S3 buckets that are open or have writing or read privileges. And there's not
only data stored in it, but tons of code with possibly embedded keys are usually found in this
environment. Next up. All right. So, as I was explaining with GitHub and GitLab and even S3
buckets, they're not just the only source of leak credentials. As you can see right now,
I basically Google, Google dork, aka, which is usually how Amazon permanent keys start.
And I was able to find the snippet. Fortunately, the person that posted this sanitizes keys,
but that doesn't mean this happens all the time. And with this, I just wanted to show you an
example that it's not just code repositories. It can be anything. I actually had a friend that
lost his username and password for his Gmail and it turned into an absolute nightmare.
The attackers actually reset everything he had and it took him like a week and even him being
part of the community to get a hold of Google in order to reset this. So, please be very careful
with these things. And that was a... my friend was working for a very large company, but there
are other examples that we're going to show you where the attacker may not be so obvious,
yet cause even more damage. So, let's go on the next one.
So, before we can continue on this presentation, it is important for you to understand
that when we look at the context and nomenclature of attacks on the internet, on the cloud, or inside
the perimeter, we always look in a MITRE cloud attack matrix in this case. Like I said before,
we were going to focus on cloud-related type of environments. So, in this case, we're talking
about basically unsecured credentials. Unsecured credentials, for example, if you leave a credential
in an Amazon S3 bucket or it's embedded in some code in GitHub, can lead to things such as what
is called valid accounts. And valid accounts can be used for initial access, persistence,
lateral movement, and privilege escalation. And I'm going to give you an example of it as we move
on in this presentation. So, just keep this in mind. So, let's move on.
One of the things that we're looking at here is a technique which is T1078.004,
which is valid cloud accounts. So, you're obtaining cloud accounts that basically,
in one of the scenarios that we're going to propose, we're able to find access keys that
then allow you to not only access the provider, but move laterally and escalate privileges.
These attack vectors are real. And many times, they get dismissed because the company,
and being honest to you, they do not have an awareness of the reach of the cloud within their
perimeters. As we move on in the cloud adoption, there is many hybrid environments. And in these
hybrid environments, parts of your cloud infrastructure would allow access to your
perimeter, either because the developers do it or because you are in IT operations,
and there are some servers that have some access to S3 buckets, for example, or you have a WAN,
or you have a cloud VPN. So, these are scenarios that are important to consider that are real.
There is the line between the perimeter and the internet gets blurred or even disappear
with the adoption of cloud technologies. And here's an example, something that you should read
on. I know this is an evolving framework, and we may be giving you a number that will change
tomorrow or even the definition of it. But it's important to understand that there is some work
associated with these attack vectors and these vulnerabilities. And this is what we're trying to
showcase today. Next, please.
So here, as I set the stage, I wanted to give you a little bit of an example and something that
we are actually working right now, and we will be presenting more research in the future, which is
lateral movement and escalation of privilege by simply obtaining keys, right? You can obtain
keys. Jose and I did a presentation, which was called Red Teaming DevOps in the Mayan Summit of
Red Team Village, where we explained ways of either phishing or finding these credentials,
which we're going to show part of it today. But for example, if you were able to get...
sure if you remember how I was able to go Google for a permanent keys,
AWS, which is started A-K-I-A, you can do things as you can, depending on the actual
user, you can even create new trust role policies, you can add yourself to role trust policies. So
for example, let's say there was a trust policy that allows you to access certain buckets where
there is sensitive data, you will be able to basically access that data by simply starting
from the compromise of these keys, or you can create temporary keys by either STS assume role
or get session token, it will depend on the policies that are in place and the privilege
of the users. However, this is not... the boundaries are not that strict. And because
this is an evolving and new technology, it's not that difficult, or far fetched to say that if
you're able to get a permanent keys that have been leaked on the internet, you might be able to
basically go around in its specific environments. So here's an example how you can have used tokens
specifically in AWS by either obtaining permanent keys or compromising a session
where they're already using temporary keys, which usually start with ASIA. And then from there,
you can have used temporary tokens using things such as assume role. Assume role is a
cross account feature given by AWS to, for example, provide temporary access to a user to a
resource that he may not have access to, or things like get session token, which are permanent,
temporary, sorry, temporary tokens that can be used for specific features. So please be on the
lookout for these things because they are... this has not really exploited as publicly available,
but it's definitely something that is happening right now. To next, please. And with that,
I'm going to pass it to Jose, and we're going to see an awesome demo.
Hey, so give me two seconds here. I'm going to go ahead and share my terminal as well as the slides.
Let me know when... Rock, can you confirm you can see both the terminal and the slides?
Perfect. Okay. Sweet. So I want to do a demo live today, if everyone's okay with that. I don't
usually do this, but I'm so confident that... again, I'm so confident about how comfortable
this tool runs and finds leaks. I want to do it live. And what I'm going to show in the demo today
are basically three basic steps of how you would use GitLab, the tool that we built to find new
credentials in GitHub. The first step is going to be installing it and deploying it. The second
step is going to be searching for leaked AWS credentials. And then the third step, we're going
to just really quickly dig into that data that is generated from one of our searches for hunts.
So the tool, I already have the tab here prepared. The tool is on GitHub. Unsurprisingly,
literally, we're searching GitHub for leaks, and we're also using GitHub to host our code
on their DBS1 GitHub. So I'm just going to go ahead and just clone the project really quickly
here. And that's going to bring our project down. I'm just going to get back to our slides.
So just a few notes. Again, going back to just... while this is cloning down, to Brad's point,
this leaking credential is totally normal. I've actually made these mistakes before. I've been
in incident responses where colleagues have made these mistakes. It happens, and there's
very few mechanisms today out of the box to protect it, but there are good mitigation mechanisms
out there. Again, to Brad's point, it hasn't exploded yet, so it's not actively
being mitigated for either. So here we go. We've cloned down GitWell Hunt,
and the first step to install it is... I'm going to go prepare a virtual environment,
and I've used VirtualM for this. So we're creating a virtual environment in Python 3,
and this is just in order for us to install all of our dependencies separated from our system Python.
And so the next step here is I'm going to activate my virtual environment and then
install my required dependencies for the project. The project is pretty lightweight. There's not a
lot of dependencies. And now I have a... yeah, the tool should execute just perfectly fine.
And the tool is pretty straightforward. There's a config file that you really don't have to do
much with it besides configuring the actual GitHub API token. So now that we've configured our token
in there, we should be ready to go. And what I'm going to do here in this case is I'm going to
use one of the example searches that the tool already has out of the box. By the way, the tool,
actually, before I jump into running it really quickly, let me walk you through how the tool
kind of functions. The first function of it is you pass it a search parameter, and this is the
equivalent of a GitHub advanced search. And we have a few examples in here, like how to find
GCP JWT tokens, AWS API secrets, Azure JWT tokens, so on and so forth. And then once,
what it does is it's going to go ahead and search GitHub for files that match these patterns. And
then it checks, it reads every single file that returns in the results, and it checks whether
that file has a valid credential from a set of regexes. And by the way, huge kudos and credits
to Trufflehug, which is the project we're actually borrowing a lot of these regexes from.
But it's basically a two-step function tool, right? First, it searches for
potentially leaked files, and then it verifies that inside those files,
you actually have credentials in them. In this case, again, let's try to run an example for
pulling back AWS secrets, right? And so once we run it, the first thing we're going to get back
is the total results that we found in GitHub for files that match this. And then you're going to,
the tool's going to go ahead again and process every single result, right? And it's printing
out here the actual URLs, checking whether there's actual credentials in them. And here we have our
first kind of hit from Bill Rosey, where he actually committed an AWS secret. And if we
actually go into this URI, we should be able to see his AWA access key. I'm just going to open it
here in my browser, and yep, there you go. That's literally his secret key, and it's out of the EU
central region. So as I explained earlier, with this key, we literally have the exact same
permissions that Bill Rosey had in managing AWS. And so the tool, again, the tool runs,
and it's going to take a few minutes because there's 200 something results, right? We're about
a leap number five, 95. And it collects all this data and saves it into a JSON file, essentially.
Now, while the tool is running, I want to go ahead and show you a bit of... so we grabbed the data
out of this JSON file. I'll show you at the end of the execution what it looks like. But we grabbed
all that data. We've been collecting these leaks for about seven days now. And I pulled some quick
reports on what we've collected so far in the past seven days. And for example, for top leaks
or top leaks by technology, we have by far a whole lot of AWS API keys out there that we've collected
and second to that is GCP actually service accounts. I couldn't say I wasn't... this did
not surprise me. I don't know, Rod, if you got surprised when you saw this data set, but this
was kind of somewhat expected. AWS by far is the secret that gets leaked the most.
Somewhat surprising, there's still a lot of private keys for ISA out there, which I was not
expecting. Another thing that, again, just another curiosity is that AWS secrets, if you break this
down by the last seven days, it doesn't vary a whole lot. It seems that by far, again, AWS seems
to be the normal thing that gets leaked the most, second to GCP. Although what they call
YouTube OAuth tokens or the tool calls YouTube OAuth tokens are actually generic OAuth tokens as
well for Google. It just considers it like a specific YouTube one, but it's not. And so you
can see that actually varies every so often across days. And so for the most part, people are either
leaking their Google suite tokens, which is really what the YouTube OAuth token are,
or the GCP tokens, which is pretty bad because, again, now you have full access to whatever they
can do with Google on their account. We did a breakdown as well by top companies. And mind you,
where this data comes from is that if the user, because we're searching GitHub, and GitHub gives
users abilities to input things like what blog they have, what their Twitter handle is, and the
profile, and what company they work for, we pull that information back when we have a match,
since that's open data. And this is just a very, very ugly picture of what the most credentials
by company that we collected were. X by Orange is by far the company that leaked the most,
or at least that was labeled to leak the most, but we got some pretty big ones out here like
Nordstrom, VMware, and again, these are the top. You see here the other flag that means that
there's a bunch of credentials that were leaked, but not necessarily multiple of them. These are
multiple credentials for some of these companies. Microsoft was in there. Yeah, pretty bad, at least
to say. Yeah, so this is very revealing because many times, and it's like what Jose was just
saying, there just doesn't seem to be awareness of how bad this is. And we're trying to show you
a picture of we collected, what, a week or so, and look all the stuff we have. And we actually,
we basically did this for analytic purposes and to show awareness. We could have
delved even deeper into this data, and who knows what we would have been able to get. And if we are
doing it, bad guys are doing it too. So that's, that's, this is something that's very remarkable
that you see some big names in there. I understand there is a number of mitigation issues, like you
can say, well, you can get my key, but if your IP is not in my security group, you will not be
able to load again. True, but we don't know that. And the fact that you prove that, or the, those
keys are being leaked, it opens the possibility. It's almost like an open port. That's how I see
it. It's almost like an open port of a vulnerable application.
And one note I wanted to say on this data also, and it was a really good point about this. This,
again, because this is what individuals have put in their company profile. We, there's some garbage
in here too, like this LinkedIn profile or ABC, which we necessarily haven't cleaned out of our
data yet. But this is straight up, again, the users that we collected or we've seen leaks for,
what companies have put in their profile. And again, we also kind of want to break this down
by region. My favorite region so far that users have listed is in your heart. This is pretty sweet.
But unsurprising, and actually, Rod, you made this point to me, unsurprising.
There is big cities that have a lot of development footprint, right? Like a lot of developers and a
lot of huge tech footprint are showing up here, right? New York, New York, Santa Monica, San
Francisco, Sweden, Russia, Pittsburgh, right? So again, the cities that typically have huge
companies at the development are a lot of developers. And there, again, the data,
we haven't really cleaned up the data, hence why you have in your heart in here. But I just want
to give you a really rough view of like, what if we were just trying to aggregate this really,
really quickly, and we did it for seven days, what it looks like.
Right. And if you were targeting a company, right, which is where you and I were talking yesterday,
we can hypothesize that by the number of leaks, and if we reduce that to the regions, because we
know where most of the developers are working, even though that's changing a little bit because
of the work from home, that sort of cuts off a little bit of the work that an adversary
needs to do in order to try to infiltrate one of these big companies that by omission or
by or willingly may have developers that are posting keys that are revealing too much.
That's true. By the way, our tool finished here. I just want to give
the viewers here a quick preview of what the results file gets written as. So if I just,
since this is a valid JSON, you can see here the data we're collecting, right? So again,
we're dumping every result that's getting matched into an array of JSONs here, or actually
various JSON objects, and we collect a URL where we found the leak that checked that it actually
matched the different matches, right? So in the matches, I purposely went out of my way to not
necessarily store the actual secret, but merely just the keys if possible. The owner, so who owns
that repository that leaked that credential, owner URL, the type, again, if it's a company,
it's going to get listed as a company. The name, email, again, if they listed the company in their
GitHub profile, it's in their blog, the location that they got listed in, this is where this data
is coming from over here, Twitter handles, so on and so forth, right? And again, some of these
fields tend to be null if the users don't allow their profile in GitHub to show this data. That's
the only way we can actually read it. But yeah, again, pretty telling dataset. And with that,
I kind of want to talk about one example that stood out to the CEO of a telecom. It was pretty
wild that, again, because the CTO had listed on his GitHub profile, his Twitter handle,
as well as his personal blog site, we were able to find his LinkedIn and essentially...
Pretty much everything. We found pretty much everything about this gentleman. And
obviously, we have sanitized everything, so you can't find him yourself. But just like him,
there's many, many cases of people that we found. Right, it's just an example.
We've contacted them, again, to clean things up. Our intent is to make sure these things stay
clean up. Again, we want to bring awareness, as this is a big issue. And again, I've made these
mistakes in the past, but this is how bad things can get, essentially. Right?
You can go for a simple... It's amazing that you can go for a simple key to pretty much everything,
from a leak key to opening the doors of your company, revealing your personal life,
personal things. So, please be careful with these things and try to implement some measures.