1
00:00:01,050 --> 00:00:09,110
All right, so welcome everybody and thank you for taking the time to see our talk.

2
00:00:09,110 --> 00:00:39,630
We are happy to be here at the DEF CON Red Team Village and we are very excited to share with you some of the findings that we've been doing since we did a talk in the previous Red Team Village Summit where we try to showcase and expose basically how dangerous is the leakage of credentials on the internet.

3
00:00:39,630 --> 00:00:47,750
And today me and Jose Hernandez, my colleague, are going to be presenting How'd My Keys Bing Pound API Edition.

4
00:00:47,750 --> 00:00:50,170
So let's get going.

5
00:00:52,250 --> 00:00:54,990
So most of you know who we are.

6
00:00:54,990 --> 00:00:58,710
I'm a Principal Security Research Engineer at Splunk.

7
00:00:58,710 --> 00:01:03,030
I worked with Jose at Prolexic, which is now Akamai.

8
00:01:03,030 --> 00:01:04,430
I worked at Akamai for a while.

9
00:01:04,430 --> 00:01:09,970
Then I went to Caspita and came back at one point to Splunk.

10
00:01:09,970 --> 00:01:16,610
I co-founded Hack Miami and Pacific Hacker Meetups and Conferences and I grow my own CDFs.

11
00:01:16,610 --> 00:01:20,650
Some of you may have Bladen, which is the Command and Control and of course CDF.

12
00:01:20,930 --> 00:01:23,190
And Jose, can you tell us a little bit about yourself?

13
00:01:23,670 --> 00:01:27,910
Yeah, I'm also a Principal Security Researcher at Splunk.

14
00:01:27,910 --> 00:01:35,870
I'm an old longtime friend of Brock from our Prolexic days, which got purchased by Akamai.

15
00:01:36,010 --> 00:01:42,950
I co-founded a company called Zenitch, which is now Oracle's Web Application Firewall and DDoS services.

16
00:01:42,990 --> 00:01:46,350
And returned to Splunk to do research, security research.

17
00:01:46,350 --> 00:01:49,370
And this is one of the things that we've been working on that I'm super excited about.

18
00:01:50,690 --> 00:01:52,890
Awesome, so let's move on.

19
00:01:52,890 --> 00:02:04,690
So just before we get into basically what we're going to show you today, it is important that we recap a little bit on how we got to this point.

20
00:02:04,910 --> 00:02:20,770
And in order to recap to to a point where we can understand how bad and how we got to this very bad situation, we need to understand or take a look at what is DevOps.

21
00:02:20,770 --> 00:02:32,270
So as we have approached it before and explained it, DevOps is a set of practices that basically is within software development and team operations.

22
00:02:32,270 --> 00:02:34,590
It has become very popular.

23
00:02:34,650 --> 00:02:50,610
It's not really new, but it has become very popular and it has been widely adopted as most companies are starting to somehow get a foot in cloud platforms.

24
00:02:50,610 --> 00:02:55,790
Some of them have radically moved most of their operations to the cloud.

25
00:02:55,850 --> 00:03:10,150
And when we're talking about developing software and producing software, building, coding it, planning it, testing it, releasing it, the first thing that comes to your mind is DevOps.

26
00:03:10,150 --> 00:03:18,910
And DevOps is a set of practices usually guided by some software development principles.

27
00:03:19,450 --> 00:03:23,750
And the most popular currently is Agile.

28
00:03:23,750 --> 00:03:29,350
And some of you might be familiar with Agile if you have worked in a software development company.

29
00:03:29,350 --> 00:03:35,850
I work for now for three software development companies and all of them were using Agile.

30
00:03:36,330 --> 00:03:45,470
And this is how I was exposed to the, not only the infrastructure, but the risks that are associated with this.

31
00:03:45,470 --> 00:03:47,250
So next slide, please.

32
00:03:47,710 --> 00:04:14,250
So one of the things that I noticed throughout the years, and me and Jose have been researching on, is that when you use this set of principles and divide it in things that are called tool chains that go in the DevOps process, there's a number of products and a number of tools that are constantly used,

33
00:04:14,250 --> 00:04:18,850
reused, shared and repurposed.

34
00:04:19,310 --> 00:04:41,890
One of the characteristics of the, to say in a certain way, the platform that is for development, software development is at times his ephemeral character, meaning for example, that you can create containers, destroy them and then simply recreate them again.

35
00:04:41,890 --> 00:04:56,770
So in order for us to give this a little more structure into what we can look at when we're trying to gauge the disposal and risks associated with DevOps, we had to look at DevOps tool chains.

36
00:04:56,770 --> 00:05:05,750
And DevOps tool chains basically are a combination of tools that aid in the delivery, development, management of software and applications for the entire cycle.

37
00:05:05,750 --> 00:05:15,810
As you saw previously, there is an actual cycle where you code, you build, you plan, release, you test it and then you go back again.

38
00:05:16,170 --> 00:05:29,490
And if you are for example, in a specific methodologies, for example, Agile, usually there are sprints and sprints are sets of times that you have to produce a number of features or bug fixes.

39
00:05:29,850 --> 00:05:37,230
And all of this is based in a flow that goes through what they call the tool chains.

40
00:05:37,230 --> 00:05:42,550
So in the tool chains, you can see there are things that are used for planning, such as Git or Jira.

41
00:05:42,570 --> 00:05:50,210
There are things that are used for coding that involve coding, code repositories.

42
00:05:50,330 --> 00:06:03,010
There are things used for testing that implicate things such as Selenium or Bagram or Docker containers, for example.

43
00:06:03,010 --> 00:06:14,010
There are things that are tools that are made for software build, such as Ansible or Terraform or Chaff.

44
00:06:14,010 --> 00:06:22,530
And then of course, there are things that are within the tool chains for what they call deployment, which imply things such as Kubernetes.

45
00:06:22,610 --> 00:06:32,410
In Docker, for example, most of these things like Kubernetes at times are throughout the entire process.

46
00:06:32,850 --> 00:06:49,190
And depending on your cloud provider, you may be very familiar with some orchestration automation languages such as Ansible or Terraform.

47
00:06:49,190 --> 00:06:54,430
And then finally, part of this tool chain is the monitoring part.

48
00:06:54,430 --> 00:07:05,470
And then we have things usually the main two monitoring tools are either based on EOK, which is Elastic Search, and of course, Blunk.

49
00:07:06,170 --> 00:07:07,730
Next, please.

50
00:07:09,370 --> 00:07:30,910
So once we have seen the picture of the two chains that are associated with the software development, plus the cycle throughout the flow of planning it, coding it, building it, testing it, releasing it, and then coming back, there's one thing that we're focusing today,

51
00:07:30,910 --> 00:07:32,610
and that is credentials.

52
00:07:32,610 --> 00:07:36,070
Credentials are part of the entire process.

53
00:07:36,070 --> 00:07:39,610
Credentials are needed for many reasons.

54
00:07:39,650 --> 00:07:49,810
And here's a little bit of some of the highlights of what happens in this process with credentials.

55
00:07:49,810 --> 00:07:54,690
So developers, for example, usually have high privilege credentials.

56
00:07:54,690 --> 00:07:55,110
Why?

57
00:07:55,110 --> 00:08:02,050
Because they had to be able to test things that would run with low privileges.

58
00:08:02,090 --> 00:08:14,530
They are supposed to develop things that may be kernel libraries, sockets that need to be created with services or connections.

59
00:08:14,530 --> 00:08:28,530
These environments at times are usually ephemeral, like I said, and as a result of that, many times they're dismissed and poorly monitored.

60
00:08:29,310 --> 00:08:50,070
We're seeing cases of developers downloading anything from Docker Hub or who knows what have you, a container repository on the internet and simply putting them in their DevOps tool chains without any checking, without any scanning, and now you may have implanted containers,

61
00:08:50,070 --> 00:08:58,750
vulnerable libraries, vulnerable operating systems that will eventually affect and be published into your production environment.

62
00:09:00,070 --> 00:09:02,310
Obviously, this is related as well.

63
00:09:02,310 --> 00:09:07,650
There's a disconnection between development and security operations.

64
00:09:08,450 --> 00:09:14,590
I've been part of this problem many times and I have interacted with developers.

65
00:09:14,590 --> 00:09:19,990
Developers usually don't like when you tell them that their code has bugs or vulnerabilities.

66
00:09:21,270 --> 00:09:27,130
Most of the times there's not a straight link in between what they're developing, so I'll give you an example.

67
00:09:27,130 --> 00:09:34,530
You have a development department and they download, for example, all these containers or libraries or code.

68
00:09:35,130 --> 00:09:37,870
Many times there are no tools to verify this.

69
00:09:37,870 --> 00:09:49,510
Many times there are not inventory of what is it that they're downloading, what is it that they're using, what libraries they're putting into the software packages.

70
00:09:49,510 --> 00:09:53,050
So this by itself is a risk.

71
00:09:53,630 --> 00:10:01,130
And as well, this is also very popular and this is part of the nature of the DevOps process.

72
00:10:01,130 --> 00:10:05,770
There's a spread use of open source tools and code.

73
00:10:06,270 --> 00:10:16,050
So basically, many times I notice in this environment that this code is trusted by default, meaning they just go, oh, I know this developer.

74
00:10:16,050 --> 00:10:16,990
Oh, I know this group.

75
00:10:16,990 --> 00:10:20,750
I'm just going to download this and use it in my application.

76
00:10:21,110 --> 00:10:35,450
And again, this goes back to the disconnection, but there is some sort of an honor code between the open source community where do no harm is always the driver of software development.

77
00:10:35,450 --> 00:10:51,590
But it does not mean, and we have seen this in supply chain attacks, that malicious actors, nation states, criminals in general, may target these repositories, these open source communities, and embed by stuff in it.

78
00:10:51,690 --> 00:11:00,550
Also, embedded credentials usually end up in public repositories and that's pretty much what we're going to show you today.

79
00:11:00,550 --> 00:11:12,450
How even big companies, how individuals that... unfortunately, there is no mechanisms.

80
00:11:12,710 --> 00:11:17,250
All in all, we don't blame them for what you're about to see.

81
00:11:17,250 --> 00:11:21,790
We don't think that they're purposely leaking these credentials.

82
00:11:21,790 --> 00:11:39,110
However, it is important that we point this out in order to bring awareness that there has to be mechanisms created in order to avoid the rampant, because there's no other name for it, this rampant leakage of credentials.

83
00:11:39,110 --> 00:11:41,850
And we will show today how bad it can get.

84
00:11:41,870 --> 00:12:09,630
So when we have also dev development departments and DevOps processes, where most of these developers have high privilege credentials or permissions, there is obviously a higher risk of insider, because it takes less of an effort to cause harm, to embed malicious stuff in it,

85
00:12:09,630 --> 00:12:11,150
or to even destroy it.

86
00:12:11,150 --> 00:12:22,970
Like we have seen in some cases before where a person that was part of a development department has come back or a system has come back and do harm to an employer.

87
00:12:23,350 --> 00:12:48,990
And another couple of points is that due to the CSD nature, the continuous delivery, basically that cycle that goes around building, testing, developing, and coming back to planning and on software, these things get published immediately.

88
00:12:48,990 --> 00:13:06,450
That's one of the nature of the DevOps process, is the DevOps process has shortened the time where you plan, co-build, test, release, basically becomes something almost immediate that goes into production.

89
00:13:06,450 --> 00:13:31,650
And this by itself represents a risk, because when you, like I told somebody, when you have vulnerabilities in environments that are driven by CICD and things like, for example, implanted container, and you have very large environments, the risk and the opportunity of exploitation increases in orders of magnitude.

90
00:13:33,290 --> 00:13:42,330
So finally, and just to put this, to make this even worse, unfortunately, the cloud environments have made this risk even higher.

91
00:13:42,330 --> 00:13:42,670
Why?

92
00:13:42,670 --> 00:14:01,090
Because basically you are connected to the internet, you publish right away, you stage right away, and if there are attackers that are knowledgeable and are able to pretty much footprint your process, they can do a lot of harm.

93
00:14:01,090 --> 00:14:02,390
Next slide, please.

94
00:14:03,870 --> 00:14:11,710
So here's a little bit for you to have a reference of how the cloud providers manage credentials.

95
00:14:11,710 --> 00:14:16,330
We're going to focus mostly on cloud-related environments.

96
00:14:16,330 --> 00:14:31,830
So here's, for example, AWS, they have their own AIM credential service, which usually have things such as passwords, access keys, key pairs, or SSH keys.

97
00:14:32,030 --> 00:14:53,930
And also they do have a number of temporary security credentials that can be created on the go, and that sometimes have a feature , whereas you can give a specific user access to, temporary access to a resource that otherwise that user does not have access for.

98
00:14:53,930 --> 00:14:55,250
So next up.

99
00:14:56,170 --> 00:15:03,890
Most of the providers have had sort of similar systems for managing credentials.

100
00:15:03,890 --> 00:15:11,750
However, I had to give kudos to Microsoft because they are trying actually to tackle this problem.

101
00:15:11,750 --> 00:15:18,890
And as you can see here, they do have, this is an example of how they manage credentials.

102
00:15:18,890 --> 00:15:36,050
They use the Azure, there's a framework within Azure Active Directory where it tries to avoid the embedding of creds, and they use a different mechanism.

103
00:15:36,050 --> 00:15:49,810
We're not going to focus on this feature of Azure in this talk, but it's important for you to consider it and look at it because they definitely seem to be aware of the issue.

104
00:15:49,810 --> 00:15:50,970
Next slide please.

105
00:15:52,770 --> 00:16:06,430
And then here's, we wanted to give you a little sample of basically what's happening with the three main cloud providers, which is AWS, GCP, and Azure.

106
00:16:06,430 --> 00:16:19,430
In the case of GCP or Google Cloud, they are, most of the stuff is based on OAuth, which is a protocol that's used for identity federation and single sign-on.

107
00:16:20,230 --> 00:16:45,590
And for the most part, because of the constant interaction and services that are present in cloud environments, obviously these providers have to come up with a way to allow the interaction of either devices, users, and services, dividing and trying to establish boundaries between these entities.

108
00:16:45,590 --> 00:16:50,210
And as we will see soon, this is very challenging.

109
00:16:50,410 --> 00:16:51,750
Next slide.

110
00:16:52,210 --> 00:17:16,990
So here's a general, that are usually used and as such, potentially exposed when you have developers that are publishing or storing code in public repositories.

111
00:17:16,990 --> 00:17:21,590
So things as email and password, username and password.

112
00:17:21,730 --> 00:17:33,130
Remember, there's a difference between your local, for example, Active Directory or LDAP and the cloud identity access management.

113
00:17:33,410 --> 00:17:37,890
Sometimes this brings up a lot of confusion.

114
00:17:40,290 --> 00:17:47,790
And depending on the integration that you have with your cloud environment, this may or may not play in your favor.

115
00:17:47,790 --> 00:18:01,690
Meaning, if you're not very integrated, losing, for example, the username and password from a specific cloud service will not allow the attacker to access your internal environment.

116
00:18:01,690 --> 00:18:14,730
Also, multiple factor authentication is something that's been coming and at times it can be bypassed by certain frameworks.

117
00:18:14,730 --> 00:18:30,790
We've done some work before with Evil Jinx, which basically is able to capture the second, the authentication or the TLTP, whatever interface is presented to the user and bypass MFA.

118
00:18:30,790 --> 00:18:32,630
Access keys, we talked about it.

119
00:18:32,630 --> 00:18:34,730
Key pairs, we talked about it.

120
00:18:34,750 --> 00:18:41,810
Specific account identifiers and at times they use x.509 certificates.

121
00:18:42,470 --> 00:18:43,770
Next up.

122
00:18:45,150 --> 00:18:51,010
So, what are the primary source of leak credentials?

123
00:18:51,010 --> 00:19:00,530
Well, as you will see soon, GitHub is probably the most popular code repository on the internet.

124
00:19:00,530 --> 00:19:09,350
GitHub is now used for many other things, storing files, even hosting web pages, which is kind of cool.

125
00:19:09,930 --> 00:19:21,990
So, GitHub is like the reference when it comes to the leading internet code repository, not only publicly, but many companies use it.

126
00:19:21,990 --> 00:19:26,870
And then we also have GitLab and we also have Amazon S3

127
00:19:30,550 --> 00:19:44,290
buckets, storages, because the reason why I put this here is because you can definitely search for Amazon S3 buckets that are open or have writing or read privileges.

128
00:19:44,330 --> 00:19:55,290
And there's not only data stored in it, but tons of code with possibly embedded keys are usually found in this environment.

129
00:19:55,290 --> 00:19:56,850
Next up.

130
00:19:56,850 --> 00:19:57,210
All right.

131
00:19:57,210 --> 00:20:10,890
So, as I was explaining with GitHub and GitLab and even S3 buckets, they're not just the only source of leak credentials.

132
00:20:10,890 --> 00:20:21,470
As you can see right now, I basically Google, Google dork, aka, which is usually how Amazon permanent keys start.

133
00:20:21,470 --> 00:20:23,570
And I was able to find the snippet.

134
00:20:23,570 --> 00:20:29,870
Fortunately, the person that posted this sanitizes keys, but that doesn't mean this happens all the time.

135
00:20:30,710 --> 00:20:35,790
And with this, I just wanted to show you an example that it's not just code repositories.

136
00:20:35,790 --> 00:20:36,950
It can be anything.

137
00:20:36,950 --> 00:20:47,110
I actually had a friend that lost his username and password for his Gmail and it turned into an absolute nightmare.

138
00:20:47,150 --> 00:20:57,270
The attackers actually reset everything he had and it took him like a week and even him being part of the community to get a hold of Google in order to reset this.

139
00:20:57,270 --> 00:21:00,070
So, please be very careful with these things.

140
00:21:00,210 --> 00:21:14,570
And that was a... my friend was working for a very large company, but there are other examples that we're going to show you where the attacker may not be so obvious, yet cause even more damage.

141
00:21:14,570 --> 00:21:16,330
So, let's go on the next one.

142
00:21:17,990 --> 00:21:43,270
So, before we can continue on this presentation, it is important for you to understand that when we look at the context and nomenclature of attacks on the internet, on the cloud, or inside the perimeter, we always look in a MITRE cloud attack matrix in this case.

143
00:21:43,510 --> 00:21:48,890
Like I said before, we were going to focus on cloud-related type of environments.

144
00:21:48,890 --> 00:21:52,930
So, in this case, we're talking about basically unsecured credentials.

145
00:21:52,930 --> 00:22:04,150
Unsecured credentials, for example, if you leave a credential in an Amazon S3 bucket or it's embedded in some code in GitHub, can lead to things such as what is called valid accounts.

146
00:22:04,150 --> 00:22:11,150
And valid accounts can be used for initial access, persistence, lateral movement, and privilege escalation.

147
00:22:11,150 --> 00:22:18,890
And I'm going to give you an example of it as we move on in this presentation.

148
00:22:18,890 --> 00:22:20,470
So, just keep this in mind.

149
00:22:20,470 --> 00:22:22,170
So, let's move on.

150
00:22:24,110 --> 00:22:39,090
One of the things that we're looking at here is a technique which is T1078.004, which is valid cloud accounts.

151
00:22:39,090 --> 00:22:57,870
So, you're obtaining cloud accounts that basically, in one of the scenarios that we're going to propose, we're able to find access keys that then allow you to not only access the provider, but move laterally and escalate privileges.

152
00:22:59,710 --> 00:23:02,850
These attack vectors are real.

153
00:23:03,010 --> 00:23:16,610
And many times, they get dismissed because the company, and being honest to you, they do not have an awareness of the reach of the cloud within their perimeters.

154
00:23:16,610 --> 00:23:21,930
As we move on in the cloud adoption, there is many hybrid environments.

155
00:23:21,930 --> 00:23:39,250
And in these hybrid environments, parts of your cloud infrastructure would allow access to your perimeter, either because the developers do it or because you are in IT operations, and there are some servers that have some access to S3 buckets, for example,

156
00:23:39,250 --> 00:23:42,710
or you have a WAN, or you have a cloud VPN.

157
00:23:42,710 --> 00:23:46,770
So, these are scenarios that are important to consider that are real.

158
00:23:46,770 --> 00:23:56,250
There is the line between the perimeter and the internet gets blurred or even disappear with the adoption of cloud technologies.

159
00:23:56,250 --> 00:24:00,150
And here's an example, something that you should read on.

160
00:24:00,150 --> 00:24:09,530
I know this is an evolving framework, and we may be giving you a number that will change tomorrow or even the definition of it.

161
00:24:09,530 --> 00:24:15,650
But it's important to understand that there is some work associated with these attack vectors and these vulnerabilities.

162
00:24:15,870 --> 00:24:20,090
And this is what we're trying to showcase today.

163
00:24:20,270 --> 00:24:21,610
Next, please.

164
00:24:22,470 --> 00:24:46,090
So here, as I set the stage, I wanted to give you a little bit of an example and something that we are actually working right now, and we will be presenting more research in the future, which is lateral movement and escalation of privilege by simply obtaining keys,

165
00:24:46,090 --> 00:24:46,310
right?

166
00:24:46,310 --> 00:24:48,230
You can obtain keys.

167
00:24:48,890 --> 00:25:06,110
Jose and I did a presentation, which was called Red Teaming DevOps in the Mayan Summit of Red Team Village, where we explained ways of either phishing or finding these credentials, which we're going to show part of it today.

168
00:25:06,110 --> 00:25:08,180
But for example, if you were able to get...

169
00:25:09,450 --> 00:25:30,990
sure if you remember how I was able to go Google for a permanent keys, AWS, which is started A-K-I-A, you can do things as you can, depending on the actual user, you can even create new trust role policies, you can add yourself to role trust policies.

170
00:25:30,990 --> 00:25:55,090
So for example, let's say there was a trust policy that allows you to access certain buckets where there is sensitive data, you will be able to basically access that data by simply starting from the compromise of these keys, or you can create temporary keys by either STS assume role or get session token,

171
00:25:55,090 --> 00:26:01,250
it will depend on the policies that are in place and the privilege of the users.

172
00:26:01,250 --> 00:26:08,370
However, this is not... the boundaries are not that strict.

173
00:26:08,790 --> 00:26:25,870
And because this is an evolving and new technology, it's not that difficult, or far fetched to say that if you're able to get a permanent keys that have been leaked on the internet, you might be able to basically go around in its specific environments.

174
00:26:25,870 --> 00:26:42,570
So here's an example how you can have used tokens specifically in AWS by either obtaining permanent keys or compromising a session where they're already using temporary keys, which usually start with ASIA.

175
00:26:42,830 --> 00:26:48,130
And then from there, you can have used temporary tokens using things such as assume role.

176
00:26:48,130 --> 00:27:08,150
Assume role is a cross account feature given by AWS to, for example, provide temporary access to a user to a resource that he may not have access to, or things like get session token, which are permanent, temporary, sorry, temporary tokens that can be used for specific features.

177
00:27:08,150 --> 00:27:23,450
So please be on the lookout for these things because they are... this has not really exploited as publicly available, but it's definitely something that is happening right now.

178
00:27:23,670 --> 00:27:24,990
To next, please.

179
00:27:25,670 --> 00:27:30,090
And with that, I'm going to pass it to Jose, and we're going to see an awesome demo.

180
00:27:31,610 --> 00:27:34,030
Hey, so give me two seconds here.

181
00:27:34,030 --> 00:27:39,510
I'm going to go ahead and share my terminal as well as the slides.

182
00:27:42,550 --> 00:27:47,990
Let me know when... Rock, can you confirm you can see both the terminal and the slides?

183
00:27:49,850 --> 00:27:50,590
Perfect.

184
00:27:50,590 --> 00:27:51,070
Okay.

185
00:27:51,070 --> 00:27:51,670
Sweet.

186
00:27:51,670 --> 00:27:56,090
So I want to do a demo live today, if everyone's okay with that.

187
00:27:56,090 --> 00:28:03,690
I don't usually do this, but I'm so confident that... again, I'm so confident about how comfortable this tool runs and finds leaks.

188
00:28:03,690 --> 00:28:04,910
I want to do it live.

189
00:28:04,910 --> 00:28:15,330
And what I'm going to show in the demo today are basically three basic steps of how you would use GitLab, the tool that we built to find new credentials in GitHub.

190
00:28:16,270 --> 00:28:18,690
The first step is going to be installing it and deploying it.

191
00:28:18,690 --> 00:28:22,410
The second step is going to be searching for leaked AWS credentials.

192
00:28:22,490 --> 00:28:29,090
And then the third step, we're going to just really quickly dig into that data that is generated from one of our searches for hunts.

193
00:28:29,370 --> 00:28:33,790
So the tool, I already have the tab here prepared.

194
00:28:33,790 --> 00:28:35,670
The tool is on GitHub.

195
00:28:36,530 --> 00:28:45,470
Unsurprisingly, literally, we're searching GitHub for leaks, and we're also using GitHub to host our code on their DBS1 GitHub.

196
00:28:45,470 --> 00:28:49,210
So I'm just going to go ahead and just clone the project really quickly here.

197
00:28:49,530 --> 00:28:54,370
And that's going to bring our project down.

198
00:28:54,790 --> 00:28:57,510
I'm just going to get back to our slides.

199
00:28:58,750 --> 00:29:00,590
So just a few notes.

200
00:29:00,590 --> 00:29:09,550
Again, going back to just... while this is cloning down, to Brad's point, this leaking credential is totally normal.

201
00:29:09,630 --> 00:29:12,010
I've actually made these mistakes before.

202
00:29:12,010 --> 00:29:16,450
I've been in incident responses where colleagues have made these mistakes.

203
00:29:16,590 --> 00:29:25,790
It happens, and there's very few mechanisms today out of the box to protect it, but there are good mitigation mechanisms out there.

204
00:29:25,790 --> 00:29:31,450
Again, to Brad's point, it hasn't exploded yet, so it's not actively being mitigated for either.

205
00:29:32,710 --> 00:29:33,610
So here we go.

206
00:29:33,610 --> 00:29:43,370
We've cloned down GitWell Hunt, and the first step to install it is... I'm going to go prepare a virtual environment, and I've used VirtualM for this.

207
00:29:44,530 --> 00:29:51,250
So we're creating a virtual environment in Python 3, and this is just in order for us to install all of our dependencies separated from our system Python.

208
00:29:52,170 --> 00:29:59,790
And so the next step here is I'm going to activate my virtual environment and then install my required dependencies for the project.

209
00:30:00,010 --> 00:30:01,630
The project is pretty lightweight.

210
00:30:01,630 --> 00:30:06,090
There's not a lot of dependencies.

211
00:30:06,090 --> 00:30:12,390
And now I have a... yeah, the tool should execute just perfectly fine.

212
00:30:12,390 --> 00:30:13,910
And the tool is pretty straightforward.

213
00:30:14,290 --> 00:30:25,470
There's a config file that you really don't have to do much with it besides configuring the actual GitHub API token.

214
00:30:26,190 --> 00:30:30,330
So now that we've configured our token in there, we should be ready to go.

215
00:30:30,330 --> 00:30:38,110
And what I'm going to do here in this case is I'm going to use one of the example searches that the tool already has out of the box.

216
00:30:39,370 --> 00:30:46,290
By the way, the tool, actually, before I jump into running it really quickly, let me walk you through how the tool kind of functions.

217
00:30:46,890 --> 00:30:54,450
The first function of it is you pass it a search parameter, and this is the equivalent of a GitHub advanced search.

218
00:30:55,050 --> 00:31:05,150
And we have a few examples in here, like how to find GCP JWT tokens, AWS API secrets, Azure JWT tokens, so on and so forth.

219
00:31:05,290 --> 00:31:12,530
And then once, what it does is it's going to go ahead and search GitHub for files that match these patterns.

220
00:31:12,610 --> 00:31:23,030
And then it checks, it reads every single file that returns in the results, and it checks whether that file has a valid credential from a set of regexes.

221
00:31:23,090 --> 00:31:31,490
And by the way, huge kudos and credits to Trufflehug, which is the project we're actually borrowing a lot of these regexes from.

222
00:31:31,830 --> 00:31:34,710
But it's basically a two-step function tool, right?

223
00:31:34,710 --> 00:31:41,170
First, it searches for potentially leaked files, and then it verifies that inside those files, you actually have credentials in them.

224
00:31:41,810 --> 00:31:48,550
In this case, again, let's try to run an example for pulling back AWS secrets, right?

225
00:31:50,830 --> 00:32:00,310
And so once we run it, the first thing we're going to get back is the total results that we found in GitHub for files that match this.

226
00:32:00,470 --> 00:32:04,730
And then you're going to, the tool's going to go ahead again and process every single result, right?

227
00:32:04,730 --> 00:32:09,670
And it's printing out here the actual URLs, checking whether there's actual credentials in them.

228
00:32:09,990 --> 00:32:18,990
And here we have our first kind of hit from Bill Rosey, where he actually committed an AWS secret.

229
00:32:18,990 --> 00:32:25,970
And if we actually go into this URI, we should be able to see his AWA access key.

230
00:32:26,170 --> 00:32:29,110
I'm just going to open it here in my browser, and yep, there you go.

231
00:32:29,110 --> 00:32:34,370
That's literally his secret key, and it's out of the EU central region.

232
00:32:34,610 --> 00:32:44,530
So as I explained earlier, with this key, we literally have the exact same permissions that Bill Rosey had in managing AWS.

233
00:32:45,710 --> 00:32:52,270
And so the tool, again, the tool runs, and it's going to take a few minutes because there's 200 something results, right?

234
00:32:52,270 --> 00:32:54,610
We're about a leap number five, 95.

235
00:32:54,610 --> 00:32:59,590
And it collects all this data and saves it into a JSON file, essentially.

236
00:32:59,830 --> 00:33:08,670
Now, while the tool is running, I want to go ahead and show you a bit of... so we grabbed the data out of this JSON file.

237
00:33:08,670 --> 00:33:11,090
I'll show you at the end of the execution what it looks like.

238
00:33:11,090 --> 00:33:12,510
But we grabbed all that data.

239
00:33:12,510 --> 00:33:16,050
We've been collecting these leaks for about seven days now.

240
00:33:16,050 --> 00:33:23,530
And I pulled some quick reports on what we've collected so far in the past seven days.

241
00:33:23,650 --> 00:33:37,530
And for example, for top leaks or top leaks by technology, we have by far a whole lot of AWS API keys out there that we've collected and second to that is GCP actually service accounts.

242
00:33:38,530 --> 00:33:41,930
I couldn't say I wasn't... this did not surprise me.

243
00:33:41,930 --> 00:33:46,010
I don't know, Rod, if you got surprised when you saw this data set, but this was kind of somewhat expected.

244
00:33:46,010 --> 00:33:49,230
AWS by far is the secret that gets leaked the most.

245
00:33:50,330 --> 00:33:55,830
Somewhat surprising, there's still a lot of private keys for ISA out there, which I was not expecting.

246
00:33:57,250 --> 00:34:09,590
Another thing that, again, just another curiosity is that AWS secrets, if you break this down by the last seven days, it doesn't vary a whole lot.

247
00:34:09,750 --> 00:34:16,870
It seems that by far, again, AWS seems to be the normal thing that gets leaked the most, second to GCP.

248
00:34:17,070 --> 00:34:27,910
Although what they call YouTube OAuth tokens or the tool calls YouTube OAuth tokens are actually generic OAuth tokens as well for Google.

249
00:34:27,910 --> 00:34:31,250
It just considers it like a specific YouTube one, but it's not.

250
00:34:31,250 --> 00:34:35,690
And so you can see that actually varies every so often across days.

251
00:34:36,290 --> 00:34:53,010
And so for the most part, people are either leaking their Google suite tokens, which is really what the YouTube OAuth token are, or the GCP tokens, which is pretty bad because, again, now you have full access to whatever they can do with Google on their account.

252
00:34:53,610 --> 00:34:57,370
We did a breakdown as well by top companies.

253
00:34:57,570 --> 00:35:17,010
And mind you, where this data comes from is that if the user, because we're searching GitHub, and GitHub gives users abilities to input things like what blog they have, what their Twitter handle is, and the profile, and what company they work for, we pull that information back when we have a match,

254
00:35:17,010 --> 00:35:18,950
since that's open data.

255
00:35:19,010 --> 00:35:26,770
And this is just a very, very ugly picture of what the most credentials by company that we collected were.

256
00:35:27,030 --> 00:35:39,010
X by Orange is by far the company that leaked the most, or at least that was labeled to leak the most, but we got some pretty big ones out here like Nordstrom, VMware, and again, these are the top.

257
00:35:39,910 --> 00:35:47,630
You see here the other flag that means that there's a bunch of credentials that were leaked, but not necessarily multiple of them.

258
00:35:47,630 --> 00:35:50,990
These are multiple credentials for some of these companies.

259
00:35:51,710 --> 00:35:53,230
Microsoft was in there.

260
00:35:53,990 --> 00:35:56,490
Yeah, pretty bad, at least to say.

261
00:35:56,510 --> 00:36:09,170
Yeah, so this is very revealing because many times, and it's like what Jose was just saying, there just doesn't seem to be awareness of how bad this is.

262
00:36:09,170 --> 00:36:16,710
And we're trying to show you a picture of we collected, what, a week or so, and look all the stuff we have.

263
00:36:16,710 --> 00:36:26,070
And we actually, we basically did this for analytic purposes and to show awareness.

264
00:36:26,070 --> 00:36:31,630
We could have delved even deeper into this data, and who knows what we would have been able to get.

265
00:36:31,630 --> 00:36:34,590
And if we are doing it, bad guys are doing it too.

266
00:36:34,590 --> 00:36:39,370
So that's, that's, this is something that's very remarkable that you see some big names in there.

267
00:36:39,450 --> 00:36:49,450
I understand there is a number of mitigation issues, like you can say, well, you can get my key, but if your IP is not in my security group, you will not be able to load again.

268
00:36:49,650 --> 00:36:51,970
True, but we don't know that.

269
00:36:51,970 --> 00:37:03,170
And the fact that you prove that, or the, those keys are being leaked, it opens the possibility.

270
00:37:03,170 --> 00:37:04,510
It's almost like an open port.

271
00:37:04,510 --> 00:37:05,390
That's how I see it.

272
00:37:05,390 --> 00:37:08,910
It's almost like an open port of a vulnerable application.

273
00:37:10,190 --> 00:37:14,890
And one note I wanted to say on this data also, and it was a really good point about this.

274
00:37:14,890 --> 00:37:19,730
This, again, because this is what individuals have put in their company profile.

275
00:37:20,530 --> 00:37:29,450
We, there's some garbage in here too, like this LinkedIn profile or ABC, which we necessarily haven't cleaned out of our data yet.

276
00:37:29,670 --> 00:37:36,450
But this is straight up, again, the users that we collected or we've seen leaks for, what companies have put in their profile.

277
00:37:37,590 --> 00:37:41,070
And again, we also kind of want to break this down by region.

278
00:37:41,070 --> 00:37:45,930
My favorite region so far that users have listed is in your heart.

279
00:37:45,930 --> 00:37:47,570
This is pretty sweet.

280
00:37:50,570 --> 00:37:54,470
But unsurprising, and actually, Rod, you made this point to me, unsurprising.

281
00:37:54,670 --> 00:38:00,670
There is big cities that have a lot of development footprint, right?

282
00:38:00,670 --> 00:38:04,650
Like a lot of developers and a lot of huge tech footprint are showing up here, right?

283
00:38:04,650 --> 00:38:13,090
New York, New York, Santa Monica, San Francisco , Sweden, Russia, Pittsburgh, right?

284
00:38:13,090 --> 00:38:18,830
So again, the cities that typically have huge companies at the development are a lot of developers.

285
00:38:18,890 --> 00:38:24,610
And there, again, the data, we haven't really cleaned up the data, hence why you have in your heart in here.

286
00:38:25,050 --> 00:38:34,310
But I just want to give you a really rough view of like, what if we were just trying to aggregate this really, really quickly, and we did it for seven days, what it looks like.

287
00:38:34,310 --> 00:38:34,710
Right.

288
00:38:35,130 --> 00:38:56,770
And if you were targeting a company, right, which is where you and I were talking yesterday, we can hypothesize that by the number of leaks, and if we reduce that to the regions, because we know where most of the developers are working, even though that's changing a little bit because of the work from home,

289
00:38:56,770 --> 00:39:19,030
that sort of cuts off a little bit of the work that an adversary needs to do in order to try to infiltrate one of these big companies that by omission or by or willingly may have developers that are posting keys that are revealing too much.

290
00:39:20,330 --> 00:39:22,130
That's true.

291
00:39:22,130 --> 00:39:24,330
By the way, our tool finished here.

292
00:39:24,330 --> 00:39:33,910
I just want to give the viewers here a quick preview of what the results file gets written as.

293
00:39:33,910 --> 00:39:41,350
So if I just, since this is a valid JSON, you can see here the data we're collecting, right?

294
00:39:41,350 --> 00:39:57,650
So again, we're dumping every result that's getting matched into an array of JSONs here, or actually various JSON objects, and we collect a URL where we found the leak that checked that it actually matched the different matches, right?

295
00:39:57,650 --> 00:40:06,590
So in the matches, I purposely went out of my way to not necessarily store the actual secret, but merely just the keys if possible.

296
00:40:07,130 --> 00:40:17,990
The owner, so who owns that repository that leaked that credential, owner URL, the type, again, if it's a company, it's going to get listed as a company.

297
00:40:18,910 --> 00:40:30,510
The name, email, again, if they listed the company in their GitHub profile, it's in their blog, the location that they got listed in, this is where this data is coming from over here, Twitter handles, so on and so forth, right?

298
00:40:30,510 --> 00:40:38,650
And again, some of these fields tend to be null if the users don't allow their profile in GitHub to show this data.

299
00:40:40,310 --> 00:40:42,450
That's the only way we can actually read it.

300
00:40:43,270 --> 00:40:47,190
But yeah, again, pretty telling dataset.

301
00:40:47,990 --> 00:40:52,670
And with that, I kind

302
00:41:01,140 --> 00:41:06,300
of want to talk about one example that stood out to the CEO of a telecom.

303
00:41:07,300 --> 00:41:23,020
It was pretty wild that, again, because the CTO had listed on his GitHub profile, his Twitter handle, as well as his personal blog site, we were able to find his LinkedIn and essentially...

304
00:41:23,800 --> 00:41:25,520
Pretty much everything.

305
00:41:25,520 --> 00:41:28,880
We found pretty much everything about this gentleman.

306
00:41:28,880 --> 00:41:36,480
And obviously, we have sanitized everything, so you can't find him yourself.

307
00:41:36,520 --> 00:41:42,380
But just like him, there's many, many cases of people that we found.

308
00:41:42,620 --> 00:41:44,580
Right, it's just an example.

309
00:41:45,080 --> 00:41:46,940
We've contacted them, again, to clean things up.

310
00:41:46,940 --> 00:41:49,080
Our intent is to make sure these things stay clean up.

311
00:41:49,340 --> 00:41:52,960
Again, we want to bring awareness, as this is a big issue.

312
00:41:53,520 --> 00:42:00,380
And again, I've made these mistakes in the past, but this is how bad things can get, essentially.

313
00:42:00,380 --> 00:42:00,900
Right?

314
00:42:02,160 --> 00:42:18,900
You can go for a simple... It's amazing that you can go for a simple key to pretty much everything, from a leak key to opening the doors of your company, revealing your personal life, personal things.

315
00:42:18,900 --> 00:42:23,760
So, please be careful with these things and try to implement some measures.