WEBVTT 00:01.050 --> 00:09.110 All right, so welcome everybody and thank you for taking the time to see our talk. 00:09.110 --> 00:39.630 We are happy to be here at the DEF CON Red Team Village and we are very excited to share with you some of the findings that we've been doing since we did a talk in the previous Red Team Village Summit where we try to showcase and expose basically how dangerous is the leakage of credentials on the internet. 00:39.630 --> 00:47.750 And today me and Jose Hernandez, my colleague, are going to be presenting How'd My Keys Bing Pound API Edition. 00:47.750 --> 00:50.170 So let's get going. 00:52.250 --> 00:54.990 So most of you know who we are. 00:54.990 --> 00:58.710 I'm a Principal Security Research Engineer at Splunk. 00:58.710 --> 01:03.030 I worked with Jose at Prolexic, which is now Akamai. 01:03.030 --> 01:04.430 I worked at Akamai for a while. 01:04.430 --> 01:09.970 Then I went to Caspita and came back at one point to Splunk. 01:09.970 --> 01:16.610 I co-founded Hack Miami and Pacific Hacker Meetups and Conferences and I grow my own CDFs. 01:16.610 --> 01:20.650 Some of you may have Bladen, which is the Command and Control and of course CDF. 01:20.930 --> 01:23.190 And Jose, can you tell us a little bit about yourself? 01:23.670 --> 01:27.910 Yeah, I'm also a Principal Security Researcher at Splunk. 01:27.910 --> 01:35.870 I'm an old longtime friend of Brock from our Prolexic days, which got purchased by Akamai. 01:36.010 --> 01:42.950 I co-founded a company called Zenitch, which is now Oracle's Web Application Firewall and DDoS services. 01:42.990 --> 01:46.350 And returned to Splunk to do research, security research. 01:46.350 --> 01:49.370 And this is one of the things that we've been working on that I'm super excited about. 01:50.690 --> 01:52.890 Awesome, so let's move on. 01:52.890 --> 02:04.690 So just before we get into basically what we're going to show you today, it is important that we recap a little bit on how we got to this point. 02:04.910 --> 02:20.770 And in order to recap to to a point where we can understand how bad and how we got to this very bad situation, we need to understand or take a look at what is DevOps. 02:20.770 --> 02:32.270 So as we have approached it before and explained it, DevOps is a set of practices that basically is within software development and team operations. 02:32.270 --> 02:34.590 It has become very popular. 02:34.650 --> 02:50.610 It's not really new, but it has become very popular and it has been widely adopted as most companies are starting to somehow get a foot in cloud platforms. 02:50.610 --> 02:55.790 Some of them have radically moved most of their operations to the cloud. 02:55.850 --> 03:10.150 And when we're talking about developing software and producing software, building, coding it, planning it, testing it, releasing it, the first thing that comes to your mind is DevOps. 03:10.150 --> 03:18.910 And DevOps is a set of practices usually guided by some software development principles. 03:19.450 --> 03:23.750 And the most popular currently is Agile. 03:23.750 --> 03:29.350 And some of you might be familiar with Agile if you have worked in a software development company. 03:29.350 --> 03:35.850 I work for now for three software development companies and all of them were using Agile. 03:36.330 --> 03:45.470 And this is how I was exposed to the, not only the infrastructure, but the risks that are associated with this. 03:45.470 --> 03:47.250 So next slide, please. 03:47.710 --> 04:14.250 So one of the things that I noticed throughout the years, and me and Jose have been researching on, is that when you use this set of principles and divide it in things that are called tool chains that go in the DevOps process, there's a number of products and a number of tools that are constantly used, 04:14.250 --> 04:18.850 reused, shared and repurposed. 04:19.310 --> 04:41.890 One of the characteristics of the, to say in a certain way, the platform that is for development, software development is at times his ephemeral character, meaning for example, that you can create containers, destroy them and then simply recreate them again. 04:41.890 --> 04:56.770 So in order for us to give this a little more structure into what we can look at when we're trying to gauge the disposal and risks associated with DevOps, we had to look at DevOps tool chains. 04:56.770 --> 05:05.750 And DevOps tool chains basically are a combination of tools that aid in the delivery, development, management of software and applications for the entire cycle. 05:05.750 --> 05:15.810 As you saw previously, there is an actual cycle where you code, you build, you plan, release, you test it and then you go back again. 05:16.170 --> 05:29.490 And if you are for example, in a specific methodologies, for example, Agile, usually there are sprints and sprints are sets of times that you have to produce a number of features or bug fixes. 05:29.850 --> 05:37.230 And all of this is based in a flow that goes through what they call the tool chains. 05:37.230 --> 05:42.550 So in the tool chains, you can see there are things that are used for planning, such as Git or Jira. 05:42.570 --> 05:50.210 There are things that are used for coding that involve coding, code repositories. 05:50.330 --> 06:03.010 There are things used for testing that implicate things such as Selenium or Bagram or Docker containers, for example. 06:03.010 --> 06:14.010 There are things that are tools that are made for software build, such as Ansible or Terraform or Chaff. 06:14.010 --> 06:22.530 And then of course, there are things that are within the tool chains for what they call deployment, which imply things such as Kubernetes. 06:22.610 --> 06:32.410 In Docker, for example, most of these things like Kubernetes at times are throughout the entire process. 06:32.850 --> 06:49.190 And depending on your cloud provider, you may be very familiar with some orchestration automation languages such as Ansible or Terraform. 06:49.190 --> 06:54.430 And then finally, part of this tool chain is the monitoring part. 06:54.430 --> 07:05.470 And then we have things usually the main two monitoring tools are either based on EOK, which is Elastic Search, and of course, Blunk. 07:06.170 --> 07:07.730 Next, please. 07:09.370 --> 07:30.910 So once we have seen the picture of the two chains that are associated with the software development, plus the cycle throughout the flow of planning it, coding it, building it, testing it, releasing it, and then coming back, there's one thing that we're focusing today, 07:30.910 --> 07:32.610 and that is credentials. 07:32.610 --> 07:36.070 Credentials are part of the entire process. 07:36.070 --> 07:39.610 Credentials are needed for many reasons. 07:39.650 --> 07:49.810 And here's a little bit of some of the highlights of what happens in this process with credentials. 07:49.810 --> 07:54.690 So developers, for example, usually have high privilege credentials. 07:54.690 --> 07:55.110 Why? 07:55.110 --> 08:02.050 Because they had to be able to test things that would run with low privileges. 08:02.090 --> 08:14.530 They are supposed to develop things that may be kernel libraries, sockets that need to be created with services or connections. 08:14.530 --> 08:28.530 These environments at times are usually ephemeral, like I said, and as a result of that, many times they're dismissed and poorly monitored. 08:29.310 --> 08:50.070 We're seeing cases of developers downloading anything from Docker Hub or who knows what have you, a container repository on the internet and simply putting them in their DevOps tool chains without any checking, without any scanning, and now you may have implanted containers, 08:50.070 --> 08:58.750 vulnerable libraries, vulnerable operating systems that will eventually affect and be published into your production environment. 09:00.070 --> 09:02.310 Obviously, this is related as well. 09:02.310 --> 09:07.650 There's a disconnection between development and security operations. 09:08.450 --> 09:14.590 I've been part of this problem many times and I have interacted with developers. 09:14.590 --> 09:19.990 Developers usually don't like when you tell them that their code has bugs or vulnerabilities. 09:21.270 --> 09:27.130 Most of the times there's not a straight link in between what they're developing, so I'll give you an example. 09:27.130 --> 09:34.530 You have a development department and they download, for example, all these containers or libraries or code. 09:35.130 --> 09:37.870 Many times there are no tools to verify this. 09:37.870 --> 09:49.510 Many times there are not inventory of what is it that they're downloading, what is it that they're using, what libraries they're putting into the software packages. 09:49.510 --> 09:53.050 So this by itself is a risk. 09:53.630 --> 10:01.130 And as well, this is also very popular and this is part of the nature of the DevOps process. 10:01.130 --> 10:05.770 There's a spread use of open source tools and code. 10:06.270 --> 10:16.050 So basically, many times I notice in this environment that this code is trusted by default, meaning they just go, oh, I know this developer. 10:16.050 --> 10:16.990 Oh, I know this group. 10:16.990 --> 10:20.750 I'm just going to download this and use it in my application. 10:21.110 --> 10:35.450 And again, this goes back to the disconnection, but there is some sort of an honor code between the open source community where do no harm is always the driver of software development. 10:35.450 --> 10:51.590 But it does not mean, and we have seen this in supply chain attacks, that malicious actors, nation states, criminals in general, may target these repositories, these open source communities, and embed by stuff in it. 10:51.690 --> 11:00.550 Also, embedded credentials usually end up in public repositories and that's pretty much what we're going to show you today. 11:00.550 --> 11:12.450 How even big companies, how individuals that... unfortunately, there is no mechanisms. 11:12.710 --> 11:17.250 All in all, we don't blame them for what you're about to see. 11:17.250 --> 11:21.790 We don't think that they're purposely leaking these credentials. 11:21.790 --> 11:39.110 However, it is important that we point this out in order to bring awareness that there has to be mechanisms created in order to avoid the rampant, because there's no other name for it, this rampant leakage of credentials. 11:39.110 --> 11:41.850 And we will show today how bad it can get. 11:41.870 --> 12:09.630 So when we have also dev development departments and DevOps processes, where most of these developers have high privilege credentials or permissions, there is obviously a higher risk of insider, because it takes less of an effort to cause harm, to embed malicious stuff in it, 12:09.630 --> 12:11.150 or to even destroy it. 12:11.150 --> 12:22.970 Like we have seen in some cases before where a person that was part of a development department has come back or a system has come back and do harm to an employer. 12:23.350 --> 12:48.990 And another couple of points is that due to the CSD nature, the continuous delivery, basically that cycle that goes around building, testing, developing, and coming back to planning and on software, these things get published immediately. 12:48.990 --> 13:06.450 That's one of the nature of the DevOps process, is the DevOps process has shortened the time where you plan, co-build, test, release, basically becomes something almost immediate that goes into production. 13:06.450 --> 13:31.650 And this by itself represents a risk, because when you, like I told somebody, when you have vulnerabilities in environments that are driven by CICD and things like, for example, implanted container, and you have very large environments, the risk and the opportunity of exploitation increases in orders of magnitude. 13:33.290 --> 13:42.330 So finally, and just to put this, to make this even worse, unfortunately, the cloud environments have made this risk even higher. 13:42.330 --> 13:42.670 Why? 13:42.670 --> 14:01.090 Because basically you are connected to the internet, you publish right away, you stage right away, and if there are attackers that are knowledgeable and are able to pretty much footprint your process, they can do a lot of harm. 14:01.090 --> 14:02.390 Next slide, please. 14:03.870 --> 14:11.710 So here's a little bit for you to have a reference of how the cloud providers manage credentials. 14:11.710 --> 14:16.330 We're going to focus mostly on cloud-related environments. 14:16.330 --> 14:31.830 So here's, for example, AWS, they have their own AIM credential service, which usually have things such as passwords, access keys, key pairs, or SSH keys. 14:32.030 --> 14:53.930 And also they do have a number of temporary security credentials that can be created on the go, and that sometimes have a feature , whereas you can give a specific user access to, temporary access to a resource that otherwise that user does not have access for. 14:53.930 --> 14:55.250 So next up. 14:56.170 --> 15:03.890 Most of the providers have had sort of similar systems for managing credentials. 15:03.890 --> 15:11.750 However, I had to give kudos to Microsoft because they are trying actually to tackle this problem. 15:11.750 --> 15:18.890 And as you can see here, they do have, this is an example of how they manage credentials. 15:18.890 --> 15:36.050 They use the Azure, there's a framework within Azure Active Directory where it tries to avoid the embedding of creds, and they use a different mechanism. 15:36.050 --> 15:49.810 We're not going to focus on this feature of Azure in this talk, but it's important for you to consider it and look at it because they definitely seem to be aware of the issue. 15:49.810 --> 15:50.970 Next slide please. 15:52.770 --> 16:06.430 And then here's, we wanted to give you a little sample of basically what's happening with the three main cloud providers, which is AWS, GCP, and Azure. 16:06.430 --> 16:19.430 In the case of GCP or Google Cloud, they are, most of the stuff is based on OAuth, which is a protocol that's used for identity federation and single sign-on. 16:20.230 --> 16:45.590 And for the most part, because of the constant interaction and services that are present in cloud environments, obviously these providers have to come up with a way to allow the interaction of either devices, users, and services, dividing and trying to establish boundaries between these entities. 16:45.590 --> 16:50.210 And as we will see soon, this is very challenging. 16:50.410 --> 16:51.750 Next slide. 16:52.210 --> 17:16.990 So here's a general, that are usually used and as such, potentially exposed when you have developers that are publishing or storing code in public repositories. 17:16.990 --> 17:21.590 So things as email and password, username and password. 17:21.730 --> 17:33.130 Remember, there's a difference between your local, for example, Active Directory or LDAP and the cloud identity access management. 17:33.410 --> 17:37.890 Sometimes this brings up a lot of confusion. 17:40.290 --> 17:47.790 And depending on the integration that you have with your cloud environment, this may or may not play in your favor. 17:47.790 --> 18:01.690 Meaning, if you're not very integrated, losing, for example, the username and password from a specific cloud service will not allow the attacker to access your internal environment. 18:01.690 --> 18:14.730 Also, multiple factor authentication is something that's been coming and at times it can be bypassed by certain frameworks. 18:14.730 --> 18:30.790 We've done some work before with Evil Jinx, which basically is able to capture the second, the authentication or the TLTP, whatever interface is presented to the user and bypass MFA. 18:30.790 --> 18:32.630 Access keys, we talked about it. 18:32.630 --> 18:34.730 Key pairs, we talked about it. 18:34.750 --> 18:41.810 Specific account identifiers and at times they use x.509 certificates. 18:42.470 --> 18:43.770 Next up. 18:45.150 --> 18:51.010 So, what are the primary source of leak credentials? 18:51.010 --> 19:00.530 Well, as you will see soon, GitHub is probably the most popular code repository on the internet. 19:00.530 --> 19:09.350 GitHub is now used for many other things, storing files, even hosting web pages, which is kind of cool. 19:09.930 --> 19:21.990 So, GitHub is like the reference when it comes to the leading internet code repository, not only publicly, but many companies use it. 19:21.990 --> 19:26.870 And then we also have GitLab and we also have Amazon S3 19:30.550 --> 19:44.290 buckets, storages, because the reason why I put this here is because you can definitely search for Amazon S3 buckets that are open or have writing or read privileges. 19:44.330 --> 19:55.290 And there's not only data stored in it, but tons of code with possibly embedded keys are usually found in this environment. 19:55.290 --> 19:56.850 Next up. 19:56.850 --> 19:57.210 All right. 19:57.210 --> 20:10.890 So, as I was explaining with GitHub and GitLab and even S3 buckets, they're not just the only source of leak credentials. 20:10.890 --> 20:21.470 As you can see right now, I basically Google, Google dork, aka, which is usually how Amazon permanent keys start. 20:21.470 --> 20:23.570 And I was able to find the snippet. 20:23.570 --> 20:29.870 Fortunately, the person that posted this sanitizes keys, but that doesn't mean this happens all the time. 20:30.710 --> 20:35.790 And with this, I just wanted to show you an example that it's not just code repositories. 20:35.790 --> 20:36.950 It can be anything. 20:36.950 --> 20:47.110 I actually had a friend that lost his username and password for his Gmail and it turned into an absolute nightmare. 20:47.150 --> 20:57.270 The attackers actually reset everything he had and it took him like a week and even him being part of the community to get a hold of Google in order to reset this. 20:57.270 --> 21:00.070 So, please be very careful with these things. 21:00.210 --> 21:14.570 And that was a... my friend was working for a very large company, but there are other examples that we're going to show you where the attacker may not be so obvious, yet cause even more damage. 21:14.570 --> 21:16.330 So, let's go on the next one. 21:17.990 --> 21:43.270 So, before we can continue on this presentation, it is important for you to understand that when we look at the context and nomenclature of attacks on the internet, on the cloud, or inside the perimeter, we always look in a MITRE cloud attack matrix in this case. 21:43.510 --> 21:48.890 Like I said before, we were going to focus on cloud-related type of environments. 21:48.890 --> 21:52.930 So, in this case, we're talking about basically unsecured credentials. 21:52.930 --> 22:04.150 Unsecured credentials, for example, if you leave a credential in an Amazon S3 bucket or it's embedded in some code in GitHub, can lead to things such as what is called valid accounts. 22:04.150 --> 22:11.150 And valid accounts can be used for initial access, persistence, lateral movement, and privilege escalation. 22:11.150 --> 22:18.890 And I'm going to give you an example of it as we move on in this presentation. 22:18.890 --> 22:20.470 So, just keep this in mind. 22:20.470 --> 22:22.170 So, let's move on. 22:24.110 --> 22:39.090 One of the things that we're looking at here is a technique which is T1078.004, which is valid cloud accounts. 22:39.090 --> 22:57.870 So, you're obtaining cloud accounts that basically, in one of the scenarios that we're going to propose, we're able to find access keys that then allow you to not only access the provider, but move laterally and escalate privileges. 22:59.710 --> 23:02.850 These attack vectors are real. 23:03.010 --> 23:16.610 And many times, they get dismissed because the company, and being honest to you, they do not have an awareness of the reach of the cloud within their perimeters. 23:16.610 --> 23:21.930 As we move on in the cloud adoption, there is many hybrid environments. 23:21.930 --> 23:39.250 And in these hybrid environments, parts of your cloud infrastructure would allow access to your perimeter, either because the developers do it or because you are in IT operations, and there are some servers that have some access to S3 buckets, for example, 23:39.250 --> 23:42.710 or you have a WAN, or you have a cloud VPN. 23:42.710 --> 23:46.770 So, these are scenarios that are important to consider that are real. 23:46.770 --> 23:56.250 There is the line between the perimeter and the internet gets blurred or even disappear with the adoption of cloud technologies. 23:56.250 --> 24:00.150 And here's an example, something that you should read on. 24:00.150 --> 24:09.530 I know this is an evolving framework, and we may be giving you a number that will change tomorrow or even the definition of it. 24:09.530 --> 24:15.650 But it's important to understand that there is some work associated with these attack vectors and these vulnerabilities. 24:15.870 --> 24:20.090 And this is what we're trying to showcase today. 24:20.270 --> 24:21.610 Next, please. 24:22.470 --> 24:46.090 So here, as I set the stage, I wanted to give you a little bit of an example and something that we are actually working right now, and we will be presenting more research in the future, which is lateral movement and escalation of privilege by simply obtaining keys, 24:46.090 --> 24:46.310 right? 24:46.310 --> 24:48.230 You can obtain keys. 24:48.890 --> 25:06.110 Jose and I did a presentation, which was called Red Teaming DevOps in the Mayan Summit of Red Team Village, where we explained ways of either phishing or finding these credentials, which we're going to show part of it today. 25:06.110 --> 25:08.180 But for example, if you were able to get... 25:09.450 --> 25:30.990 sure if you remember how I was able to go Google for a permanent keys, AWS, which is started A-K-I-A, you can do things as you can, depending on the actual user, you can even create new trust role policies, you can add yourself to role trust policies. 25:30.990 --> 25:55.090 So for example, let's say there was a trust policy that allows you to access certain buckets where there is sensitive data, you will be able to basically access that data by simply starting from the compromise of these keys, or you can create temporary keys by either STS assume role or get session token, 25:55.090 --> 26:01.250 it will depend on the policies that are in place and the privilege of the users. 26:01.250 --> 26:08.370 However, this is not... the boundaries are not that strict. 26:08.790 --> 26:25.870 And because this is an evolving and new technology, it's not that difficult, or far fetched to say that if you're able to get a permanent keys that have been leaked on the internet, you might be able to basically go around in its specific environments. 26:25.870 --> 26:42.570 So here's an example how you can have used tokens specifically in AWS by either obtaining permanent keys or compromising a session where they're already using temporary keys, which usually start with ASIA. 26:42.830 --> 26:48.130 And then from there, you can have used temporary tokens using things such as assume role. 26:48.130 --> 27:08.150 Assume role is a cross account feature given by AWS to, for example, provide temporary access to a user to a resource that he may not have access to, or things like get session token, which are permanent, temporary, sorry, temporary tokens that can be used for specific features. 27:08.150 --> 27:23.450 So please be on the lookout for these things because they are... this has not really exploited as publicly available, but it's definitely something that is happening right now. 27:23.670 --> 27:24.990 To next, please. 27:25.670 --> 27:30.090 And with that, I'm going to pass it to Jose, and we're going to see an awesome demo. 27:31.610 --> 27:34.030 Hey, so give me two seconds here. 27:34.030 --> 27:39.510 I'm going to go ahead and share my terminal as well as the slides. 27:42.550 --> 27:47.990 Let me know when... Rock, can you confirm you can see both the terminal and the slides? 27:49.850 --> 27:50.590 Perfect. 27:50.590 --> 27:51.070 Okay. 27:51.070 --> 27:51.670 Sweet. 27:51.670 --> 27:56.090 So I want to do a demo live today, if everyone's okay with that. 27:56.090 --> 28:03.690 I don't usually do this, but I'm so confident that... again, I'm so confident about how comfortable this tool runs and finds leaks. 28:03.690 --> 28:04.910 I want to do it live. 28:04.910 --> 28:15.330 And what I'm going to show in the demo today are basically three basic steps of how you would use GitLab, the tool that we built to find new credentials in GitHub. 28:16.270 --> 28:18.690 The first step is going to be installing it and deploying it. 28:18.690 --> 28:22.410 The second step is going to be searching for leaked AWS credentials. 28:22.490 --> 28:29.090 And then the third step, we're going to just really quickly dig into that data that is generated from one of our searches for hunts. 28:29.370 --> 28:33.790 So the tool, I already have the tab here prepared. 28:33.790 --> 28:35.670 The tool is on GitHub. 28:36.530 --> 28:45.470 Unsurprisingly, literally, we're searching GitHub for leaks, and we're also using GitHub to host our code on their DBS1 GitHub. 28:45.470 --> 28:49.210 So I'm just going to go ahead and just clone the project really quickly here. 28:49.530 --> 28:54.370 And that's going to bring our project down. 28:54.790 --> 28:57.510 I'm just going to get back to our slides. 28:58.750 --> 29:00.590 So just a few notes. 29:00.590 --> 29:09.550 Again, going back to just... while this is cloning down, to Brad's point, this leaking credential is totally normal. 29:09.630 --> 29:12.010 I've actually made these mistakes before. 29:12.010 --> 29:16.450 I've been in incident responses where colleagues have made these mistakes. 29:16.590 --> 29:25.790 It happens, and there's very few mechanisms today out of the box to protect it, but there are good mitigation mechanisms out there. 29:25.790 --> 29:31.450 Again, to Brad's point, it hasn't exploded yet, so it's not actively being mitigated for either. 29:32.710 --> 29:33.610 So here we go. 29:33.610 --> 29:43.370 We've cloned down GitWell Hunt, and the first step to install it is... I'm going to go prepare a virtual environment, and I've used VirtualM for this. 29:44.530 --> 29:51.250 So we're creating a virtual environment in Python 3, and this is just in order for us to install all of our dependencies separated from our system Python. 29:52.170 --> 29:59.790 And so the next step here is I'm going to activate my virtual environment and then install my required dependencies for the project. 30:00.010 --> 30:01.630 The project is pretty lightweight. 30:01.630 --> 30:06.090 There's not a lot of dependencies. 30:06.090 --> 30:12.390 And now I have a... yeah, the tool should execute just perfectly fine. 30:12.390 --> 30:13.910 And the tool is pretty straightforward. 30:14.290 --> 30:25.470 There's a config file that you really don't have to do much with it besides configuring the actual GitHub API token. 30:26.190 --> 30:30.330 So now that we've configured our token in there, we should be ready to go. 30:30.330 --> 30:38.110 And what I'm going to do here in this case is I'm going to use one of the example searches that the tool already has out of the box. 30:39.370 --> 30:46.290 By the way, the tool, actually, before I jump into running it really quickly, let me walk you through how the tool kind of functions. 30:46.890 --> 30:54.450 The first function of it is you pass it a search parameter, and this is the equivalent of a GitHub advanced search. 30:55.050 --> 31:05.150 And we have a few examples in here, like how to find GCP JWT tokens, AWS API secrets, Azure JWT tokens, so on and so forth. 31:05.290 --> 31:12.530 And then once, what it does is it's going to go ahead and search GitHub for files that match these patterns. 31:12.610 --> 31:23.030 And then it checks, it reads every single file that returns in the results, and it checks whether that file has a valid credential from a set of regexes. 31:23.090 --> 31:31.490 And by the way, huge kudos and credits to Trufflehug, which is the project we're actually borrowing a lot of these regexes from. 31:31.830 --> 31:34.710 But it's basically a two-step function tool, right? 31:34.710 --> 31:41.170 First, it searches for potentially leaked files, and then it verifies that inside those files, you actually have credentials in them. 31:41.810 --> 31:48.550 In this case, again, let's try to run an example for pulling back AWS secrets, right? 31:50.830 --> 32:00.310 And so once we run it, the first thing we're going to get back is the total results that we found in GitHub for files that match this. 32:00.470 --> 32:04.730 And then you're going to, the tool's going to go ahead again and process every single result, right? 32:04.730 --> 32:09.670 And it's printing out here the actual URLs, checking whether there's actual credentials in them. 32:09.990 --> 32:18.990 And here we have our first kind of hit from Bill Rosey, where he actually committed an AWS secret. 32:18.990 --> 32:25.970 And if we actually go into this URI, we should be able to see his AWA access key. 32:26.170 --> 32:29.110 I'm just going to open it here in my browser, and yep, there you go. 32:29.110 --> 32:34.370 That's literally his secret key, and it's out of the EU central region. 32:34.610 --> 32:44.530 So as I explained earlier, with this key, we literally have the exact same permissions that Bill Rosey had in managing AWS. 32:45.710 --> 32:52.270 And so the tool, again, the tool runs, and it's going to take a few minutes because there's 200 something results, right? 32:52.270 --> 32:54.610 We're about a leap number five, 95. 32:54.610 --> 32:59.590 And it collects all this data and saves it into a JSON file, essentially. 32:59.830 --> 33:08.670 Now, while the tool is running, I want to go ahead and show you a bit of... so we grabbed the data out of this JSON file. 33:08.670 --> 33:11.090 I'll show you at the end of the execution what it looks like. 33:11.090 --> 33:12.510 But we grabbed all that data. 33:12.510 --> 33:16.050 We've been collecting these leaks for about seven days now. 33:16.050 --> 33:23.530 And I pulled some quick reports on what we've collected so far in the past seven days. 33:23.650 --> 33:37.530 And for example, for top leaks or top leaks by technology, we have by far a whole lot of AWS API keys out there that we've collected and second to that is GCP actually service accounts. 33:38.530 --> 33:41.930 I couldn't say I wasn't... this did not surprise me. 33:41.930 --> 33:46.010 I don't know, Rod, if you got surprised when you saw this data set, but this was kind of somewhat expected. 33:46.010 --> 33:49.230 AWS by far is the secret that gets leaked the most. 33:50.330 --> 33:55.830 Somewhat surprising, there's still a lot of private keys for ISA out there, which I was not expecting. 33:57.250 --> 34:09.590 Another thing that, again, just another curiosity is that AWS secrets, if you break this down by the last seven days, it doesn't vary a whole lot. 34:09.750 --> 34:16.870 It seems that by far, again, AWS seems to be the normal thing that gets leaked the most, second to GCP. 34:17.070 --> 34:27.910 Although what they call YouTube OAuth tokens or the tool calls YouTube OAuth tokens are actually generic OAuth tokens as well for Google. 34:27.910 --> 34:31.250 It just considers it like a specific YouTube one, but it's not. 34:31.250 --> 34:35.690 And so you can see that actually varies every so often across days. 34:36.290 --> 34:53.010 And so for the most part, people are either leaking their Google suite tokens, which is really what the YouTube OAuth token are, or the GCP tokens, which is pretty bad because, again, now you have full access to whatever they can do with Google on their account. 34:53.610 --> 34:57.370 We did a breakdown as well by top companies. 34:57.570 --> 35:17.010 And mind you, where this data comes from is that if the user, because we're searching GitHub, and GitHub gives users abilities to input things like what blog they have, what their Twitter handle is, and the profile, and what company they work for, we pull that information back when we have a match, 35:17.010 --> 35:18.950 since that's open data. 35:19.010 --> 35:26.770 And this is just a very, very ugly picture of what the most credentials by company that we collected were. 35:27.030 --> 35:39.010 X by Orange is by far the company that leaked the most, or at least that was labeled to leak the most, but we got some pretty big ones out here like Nordstrom, VMware, and again, these are the top. 35:39.910 --> 35:47.630 You see here the other flag that means that there's a bunch of credentials that were leaked, but not necessarily multiple of them. 35:47.630 --> 35:50.990 These are multiple credentials for some of these companies. 35:51.710 --> 35:53.230 Microsoft was in there. 35:53.990 --> 35:56.490 Yeah, pretty bad, at least to say. 35:56.510 --> 36:09.170 Yeah, so this is very revealing because many times, and it's like what Jose was just saying, there just doesn't seem to be awareness of how bad this is. 36:09.170 --> 36:16.710 And we're trying to show you a picture of we collected, what, a week or so, and look all the stuff we have. 36:16.710 --> 36:26.070 And we actually, we basically did this for analytic purposes and to show awareness. 36:26.070 --> 36:31.630 We could have delved even deeper into this data, and who knows what we would have been able to get. 36:31.630 --> 36:34.590 And if we are doing it, bad guys are doing it too. 36:34.590 --> 36:39.370 So that's, that's, this is something that's very remarkable that you see some big names in there. 36:39.450 --> 36:49.450 I understand there is a number of mitigation issues, like you can say, well, you can get my key, but if your IP is not in my security group, you will not be able to load again. 36:49.650 --> 36:51.970 True, but we don't know that. 36:51.970 --> 37:03.170 And the fact that you prove that, or the, those keys are being leaked, it opens the possibility. 37:03.170 --> 37:04.510 It's almost like an open port. 37:04.510 --> 37:05.390 That's how I see it. 37:05.390 --> 37:08.910 It's almost like an open port of a vulnerable application. 37:10.190 --> 37:14.890 And one note I wanted to say on this data also, and it was a really good point about this. 37:14.890 --> 37:19.730 This, again, because this is what individuals have put in their company profile. 37:20.530 --> 37:29.450 We, there's some garbage in here too, like this LinkedIn profile or ABC, which we necessarily haven't cleaned out of our data yet. 37:29.670 --> 37:36.450 But this is straight up, again, the users that we collected or we've seen leaks for, what companies have put in their profile. 37:37.590 --> 37:41.070 And again, we also kind of want to break this down by region. 37:41.070 --> 37:45.930 My favorite region so far that users have listed is in your heart. 37:45.930 --> 37:47.570 This is pretty sweet. 37:50.570 --> 37:54.470 But unsurprising, and actually, Rod, you made this point to me, unsurprising. 37:54.670 --> 38:00.670 There is big cities that have a lot of development footprint, right? 38:00.670 --> 38:04.650 Like a lot of developers and a lot of huge tech footprint are showing up here, right? 38:04.650 --> 38:13.090 New York, New York, Santa Monica, San Francisco , Sweden, Russia, Pittsburgh, right? 38:13.090 --> 38:18.830 So again, the cities that typically have huge companies at the development are a lot of developers. 38:18.890 --> 38:24.610 And there, again, the data, we haven't really cleaned up the data, hence why you have in your heart in here. 38:25.050 --> 38:34.310 But I just want to give you a really rough view of like, what if we were just trying to aggregate this really, really quickly, and we did it for seven days, what it looks like. 38:34.310 --> 38:34.710 Right. 38:35.130 --> 38:56.770 And if you were targeting a company, right, which is where you and I were talking yesterday, we can hypothesize that by the number of leaks, and if we reduce that to the regions, because we know where most of the developers are working, even though that's changing a little bit because of the work from home, 38:56.770 --> 39:19.030 that sort of cuts off a little bit of the work that an adversary needs to do in order to try to infiltrate one of these big companies that by omission or by or willingly may have developers that are posting keys that are revealing too much. 39:20.330 --> 39:22.130 That's true. 39:22.130 --> 39:24.330 By the way, our tool finished here. 39:24.330 --> 39:33.910 I just want to give the viewers here a quick preview of what the results file gets written as. 39:33.910 --> 39:41.350 So if I just, since this is a valid JSON, you can see here the data we're collecting, right? 39:41.350 --> 39:57.650 So again, we're dumping every result that's getting matched into an array of JSONs here, or actually various JSON objects, and we collect a URL where we found the leak that checked that it actually matched the different matches, right? 39:57.650 --> 40:06.590 So in the matches, I purposely went out of my way to not necessarily store the actual secret, but merely just the keys if possible. 40:07.130 --> 40:17.990 The owner, so who owns that repository that leaked that credential, owner URL, the type, again, if it's a company, it's going to get listed as a company. 40:18.910 --> 40:30.510 The name, email, again, if they listed the company in their GitHub profile, it's in their blog, the location that they got listed in, this is where this data is coming from over here, Twitter handles, so on and so forth, right? 40:30.510 --> 40:38.650 And again, some of these fields tend to be null if the users don't allow their profile in GitHub to show this data. 40:40.310 --> 40:42.450 That's the only way we can actually read it. 40:43.270 --> 40:47.190 But yeah, again, pretty telling dataset. 40:47.990 --> 40:52.670 And with that, I kind 41:01.140 --> 41:06.300 of want to talk about one example that stood out to the CEO of a telecom. 41:07.300 --> 41:23.020 It was pretty wild that, again, because the CTO had listed on his GitHub profile, his Twitter handle, as well as his personal blog site, we were able to find his LinkedIn and essentially... 41:23.800 --> 41:25.520 Pretty much everything. 41:25.520 --> 41:28.880 We found pretty much everything about this gentleman. 41:28.880 --> 41:36.480 And obviously, we have sanitized everything, so you can't find him yourself. 41:36.520 --> 41:42.380 But just like him, there's many, many cases of people that we found. 41:42.620 --> 41:44.580 Right, it's just an example. 41:45.080 --> 41:46.940 We've contacted them, again, to clean things up. 41:46.940 --> 41:49.080 Our intent is to make sure these things stay clean up. 41:49.340 --> 41:52.960 Again, we want to bring awareness, as this is a big issue. 41:53.520 --> 42:00.380 And again, I've made these mistakes in the past, but this is how bad things can get, essentially. 42:00.380 --> 42:00.900 Right? 42:02.160 --> 42:18.900 You can go for a simple... It's amazing that you can go for a simple key to pretty much everything, from a leak key to opening the doors of your company, revealing your personal life, personal things. 42:18.900 --> 42:23.760 So, please be careful with these things and try to implement some measures.