00:00:00.417-->00:00:05.250 >> Uh welcome to exploiting continuous integration and automated build systems. I'll 00:00:05.250-->00:00:10.250 also be introducing uh a framework I wrote today called Cider. Uh I'm SpaceBOx. Also 00:00:12.625-->00:00:17.625 Tyler. I'm a security engineer at a software company called LeanKit. We're uh agile and lean 00:00:19.833-->00:00:25.417 project management software company based out of Nashville. Uh I do application and network 00:00:25.417-->00:00:29.917 um security. Both offensive and defensive for them. Although this talk is in no way 00:00:29.917-->00:00:34.917 reflective of um them as a company. Um I like to break into systems uh I think like many of 00:00:36.917-->00:00:43.542 us that's how we got started. Uh I also like building systems, both defensive and offensive and 00:00:43.542-->00:00:50.417 weaponized. I like learning but I don't need to explain to this whole group of people. Um I do 00:00:50.417-->00:00:56.250 security consulting and bug bounty stuff. I'm also part of the uh Nashville eh- DC six one 00:00:56.250-->00:01:01.250 five group when my children and schedule allow. We're gonna go over a crash course of 00:01:04.042-->00:01:08.667 continuous integration and some of the high level concepts uh just to get everyone familiar 00:01:08.667-->00:01:15.000 and on board with the systems. Then we'll do um I'll list some of the tools and talk a little 00:01:15.000-->00:01:20.458 bit of the tooling involved in these processes. We'll go over the build chain um and some 00:01:20.458-->00:01:26.292 configuration problems that are inherent in those in these build systems. Um I'll give a real 00:01:26.292-->00:01:30.708 world example of an exploit uh just to get everyone familiar with what exploiting these might 00:01:30.708-->00:01:36.458 look like. Um and then I'll go over some bad practices I found during my research with the 00:01:36.458-->00:01:42.792 deployment systems at the end of these build chains. Um then I'll do an example of another attack 00:01:42.792-->00:01:47.792 and then I'll introduce CIDER. So we need to start off with some definitions here. Um 00:01:50.083-->00:01:55.208 continuous integration according to my definition would be um systems and processes in place 00:01:55.208-->00:02:01.000 to allow and enable quick and iterative releases of code into production servers although it 00:02:01.000-->00:02:03.833 doesn't necessarily need to be production servers, it just needs to be a working 00:02:03.833-->00:02:07.750 environment of some kind. I'm sure this definition has some sort of oversight that I'm 00:02:07.750-->00:02:13.208 missing, but for the purpose of this talk, that's gonna be the definition. Um people use these 00:02:13.208-->00:02:18.208 systems to enable um quick iterative releases um many times per day or week is often the end 00:02:20.333-->00:02:26.917 goal. Um into environments. So its dee- duh deployment chain from um development all the way 00:02:26.917-->00:02:31.917 to production. Um these systems tend to be repository centric which makes sense when you have 00:02:35.250-->00:02:41.292 um developers who are developing code and that code is the genesis for this entire process 00:02:41.292-->00:02:47.625 so it it would make sense th- the repository is very s- sort of um centerpiece of this whole 00:02:47.625-->00:02:54.417 thing. The repositories are in uh chain with the automated build systems, so they often 00:02:54.417-->00:02:59.708 kick off these processes um and so they're a point of exploit which we're gonna be exploring. 00:02:59.708-->00:03:04.708 Um and another thing to note and not necessarily for my examples, but for in in for in general um 00:03:09.750-->00:03:15.875 these systems are being ada- adapted to not only application code or server side code but um 00:03:15.875-->00:03:22.125 the more we move towards more of a cloud based uh infrastructure and automated infrastructure, 00:03:22.125-->00:03:28.417 these systems are ca- are and can be used for more um infrastructure components to 00:03:28.417-->00:03:33.958 allow for continuous integration of the not only the application code, but the systems on which 00:03:33.958-->00:03:39.875 the application code live. We have to talk about Microservices. And its a buzz 00:03:39.875-->00:03:45.417 word and I needed probably to add the TM here for trademark, but they're so hot right now. 00:03:45.417-->00:03:51.250 But I just needed to uh, they're basically abstraction at its finest. Like the that's all it 00:03:51.250-->00:03:55.750 really is, its not a new concept at all, its the same old abstraction we've encountered in 00:03:55.750-->00:04:00.750 software development for the last eighty years, um but as it relates to um application code, 00:04:02.750-->00:04:07.750 its the concept of breaking down large applications into small uh small applications that do one 00:04:10.625-->00:04:17.083 or two things really well. And this allows for uh autonomous um development in that you give a 00:04:17.083-->00:04:22.833 development team potentially the goal of saying hey, we we have this service, we need it to do 00:04:22.833-->00:04:28.875 these two things. We need it to take x input and provide y output. And that team can 00:04:28.875-->00:04:33.125 develop this thing and it can function on its own in respective of the other services 00:04:33.125-->00:04:39.000 its built to handle failure and its built to handle errors in a way that don't uh impact other 00:04:39.000-->00:04:45.917 things. If, if its done that that's the theory. Uh. So there's some good security 00:04:45.917-->00:04:52.083 implications here right? Uh frequent release cycles are fabulous. Um faster code 00:04:52.083-->00:04:57.417 deployments equal quick remediation in theory. I think it was Zane Lackey formed Etsy 00:04:57.417-->00:05:03.917 who said that fast iterations were their greater security feature. I would kind of second 00:05:03.917-->00:05:07.458 that say its really good security feature. If you think about, you have a c cert 00:05:07.458-->00:05:14.000 vulnerability or something and you drop it on the developers to fix and the developers fix it 00:05:14.000-->00:05:19.667 the time from the point of fix to the time its deployed is significantly reduced uh which 00:05:19.667-->00:05:25.417 is really awesome especially when you can reduce these to like twenty minutes. Um and uh 00:05:25.417-->00:05:30.417 the systems enable uh no outage deployments a lot lot of times. Um they reduce single points of 00:05:32.500-->00:05:35.542 failure to some degree. So imagine you have a web application and it has like a 00:05:35.542-->00:05:41.792 left hand toold bar and the left hand tool bar is run by a single service and so that if that 00:05:41.792-->00:05:48.125 service goes down uh the entire applications not rendered unusable, just that component of 00:05:48.125-->00:05:54.667 it is. Likewise, um compromise of a certain portion of these applications is not is not 00:05:54.667-->00:06:00.542 necessarily indicative of pwnage of the entire system as a whole. So you might compromise 00:06:00.542-->00:06:05.958 something at some sort of server side injection and take run code and somehow take control of this 00:06:05.958-->00:06:10.500 container in which the microservice runs, but the uh it does not necessarily mean that 00:06:10.500-->00:06:16.625 you have control of the entire application stack, it means control over that portion of the 00:06:16.625-->00:06:22.042 containerized application stack. But there are some inherently bad things too. Um eh- just sort 00:06:22.042-->00:06:28.250 of human error sort of things. When you have um systems that enable quick deployments uh it 00:06:28.250-->00:06:34.000 maybe puts undue pressure on those who are fixing the code to push fixes out quickly when 00:06:34.000-->00:06:40.000 perhaps more review is needed. Um I'd also argue that the automated build systems like the 00:06:40.000-->00:06:46.542 CI systems and the CI pipelines are um checked less for security than the code in which they're 00:06:46.542-->00:06:50.292 deploying. So they kind of sit in this like middle sandwich layer where you have like 00:06:50.292-->00:06:54.875 infrastructure components that are being tested maybe through like a network pen test. You 00:06:54.875-->00:06:59.542 have application code which is being handled through like an application pen test and you got 00:06:59.542-->00:07:04.708 this quasi containerized environment that's sitting on its own IP space in its own 00:07:04.708-->00:07:09.958 container on top of that, er, on top of the infrastructure but below the code and its really 00:07:09.958-->00:07:14.958 not being uh uh tested um and so that's what we're going to be exploiting. We need to talk 00:07:21.625-->00:07:27.625 about tools. [laughter] So we'll start with the build systems because its the next step 00:07:27.625-->00:07:32.625 iteratively after the code repositories and the build systems um basically listen and 00:07:32.625-->00:07:37.000 take code and they build it conditionally. These conditions are um environment so it 00:07:37.000-->00:07:42.000 provisions an environment and also usually runs tests. So you provide it some sort of test 00:07:42.000-->00:07:47.333 script, its supposed to run this test script against the code base, if it succeeds it gives 00:07:47.333-->00:07:54.292 you a thumbs up maybe back in github. If it fails, it lets you know. Um but these are mostly 00:07:54.292-->00:07:58.625 most of the vendors are quasi containerized types of enviornments right. So you have, 00:07:58.625-->00:08:03.875 you give it the code, it takes the code, it spins up a docker container or some sort of quasi 00:08:03.875-->00:08:09.667 virtual environment, does the build and then tears it all down. Um the uh the systems uh 00:08:09.667-->00:08:14.667 or its popular to have cloud and on frame based systems. So for example Travis uh drone, well 00:08:17.375-->00:08:24.208 and circle CI have uh cloud based solutions where you you basically give it permission oh 00:08:24.208-->00:08:29.208 to look into your github um and it takes the code and build it out there. On but if you want to 00:08:31.375-->00:08:36.250 a lot of these systems have enterprise versions, um in fact some of them are exclusively 00:08:36.250-->00:08:41.875 enterprise and you pull these build frameworks down behind your own firewall um and do the 00:08:41.875-->00:08:48.083 builds there. Um so its, th- there's varying levels of threat depending on how these are 00:08:48.083-->00:08:52.542 hosted. Um but that's something to be aware of and something that we'll surely take advantage 00:08:52.542-->00:08:57.542 of. But there's more. Um the deployment systems would be the next iteration after this. But 00:09:03.458-->00:09:09.833 the deployment systems very well might be one in the same as the build system. So I I feel like 00:09:09.833-->00:09:13.542 I'm drawing a distinction here where there might not necessarily need to be one. Um 00:09:13.542-->00:09:18.375 Jenkins would be a vendor here that is a very popular one but it also will do the build 00:09:18.375-->00:09:23.792 portion as well and some on the other page would do the build and also handle the deployant. 00:09:23.792-->00:09:28.667 So I'm not saying that these are hardened one way or the other, although some do take, some are 00:09:28.667-->00:09:32.958 just a build service and some are just a deployment service. Um but some vendors here uh 00:09:32.958-->00:09:37.958 would be like Jenkins, Octopus Deploy, Kubernetes, Rancher, Mesosphere. Um the uh, a lot of 00:09:40.917-->00:09:44.917 these are headed in a containerized direction as well so once they take your code, 00:09:44.917-->00:09:49.500 maybe they even take an image and they deploy it to its respective infrastructure like a 00:09:49.500-->00:09:55.792 a cluster of servers in which they are in control of. They would then spinup containers and 00:09:55.792-->00:10:02.125 basically use those hosts as a uh resource pool to then pr- provide containerized services. 00:10:02.125-->00:10:07.292 And this in a microservice architecture is where the services would go. So basic 00:10:07.292-->00:10:12.292 chain of deployment might be developer man uh or woman pushes up to github and that code can 00:10:14.667-->00:10:19.625 trigger uh a web hook type of even and let's say your build service I use Travis in this 00:10:19.625-->00:10:26.625 example. Has already established a trust relationship to the uh eh- repository and it sees that 00:10:26.625-->00:10:32.000 a pole request or commit or something, some event has happened and knows to pull that 00:10:32.000-->00:10:37.208 code down and run its build. From there, it might let github know hey we've completed 00:10:37.208-->00:10:42.667 successfully or we've failed. Or it some of these bundle an image up and it will actually create 00:10:42.667-->00:10:47.875 the image of the code you just built and it might ship it to uh a code or an image repository 00:10:47.875-->00:10:52.750 like dockerhub and then from there the deployment service would either take the code from 00:10:52.750-->00:10:57.583 the source or it would take the code, take the image from the image repository and then from 00:10:57.583-->00:11:01.875 there it would deploy it out to like this cluster of servers where it's gonna run some 00:11:01.875-->00:11:07.458 microservices and to you know whoever might be consuming them. These are not necessarily web 00:11:07.458-->00:11:14.417 facing these are just services. Services to do whatever you want them to do. Well when we look at 00:11:14.417-->00:11:18.250 these, um especially when we start attacking the build servers. And that's kind of 00:11:18.250-->00:11:23.333 where I'm gonna focus, um the configurations of for these are the largest exposure and there 00:11:23.333-->00:11:28.542 are some trends that are being adopted and uh kind of the development world that maybe 00:11:28.542-->00:11:34.500 aren't the safest. Um with the software development life cycle its extremely common um to be 00:11:34.500-->00:11:39.500 building codes before building the code before merging it. In fact, its uh being it its being 00:11:41.708-->00:11:47.292 adopted to um build the code before even reviewing the code. So that you can, you don't waste 00:11:47.292-->00:11:51.667 your time reviewing the code right? Why would you want to waste your time reviewing the 00:11:51.667-->00:11:56.750 code if you know its not even gonna build? So a lot of these systems are triggering off of 00:11:56.750-->00:12:01.750 pole requests and commits to github repositories. Um the pole request is is the largest uh 00:12:04.167-->00:12:09.167 probably threat vector here. Um but you are essentially allowing manipulation of a repo and that 00:12:11.208-->00:12:15.333 repo holds downstream instructions for that build process. Um most of the 00:12:15.333-->00:12:20.958 configurations for a lot of these systems is held in the root of the repository. And so 00:12:20.958-->00:12:27.167 you have scenarios where you can make a pole request to uh, a repo that you don't have 00:12:27.167-->00:12:32.542 permission over and in doing so you're triggering a build but not only that, the PR could 00:12:32.542-->00:12:37.875 actually contain changes to the build process, which allows for command injection and things 00:12:37.875-->00:12:43.250 like this. So that's what we're going to be exploiting. Um this is really nothing vendor 00:12:43.250-->00:12:49.208 specific, I'm not really dropping any oh deys on any vendors or any anything like 00:12:49.208-->00:12:54.708 that. This is vulnerability stacking. So attack some of these you might have three 00:12:54.708-->00:13:01.250 components misconfigured here that um like eh- in an example um you taking advantage of the 00:13:01.250-->00:13:06.250 fact A that you're doing builds off of pole requests uh th- and B that you are um you can you've 00:13:08.875-->00:13:14.042 allowed the sudo required flag on the container so now you can run your builds, build commands 00:13:14.042-->00:13:19.042 as root and maybe a third thing that you can uh uh shove an image up to uh image repository 00:13:22.042-->00:13:27.458 and so those three things stuck together allow you to poison an image repository for a user for 00:13:27.458-->00:13:33.250 a company. Um each of these services might be functioning perfectly exactly as its 00:13:33.250-->00:13:36.792 intended. The inter- it's the interaction between these services and the trust 00:13:36.792-->00:13:41.417 relationships that we're really going to be exploiting. Um because th- there's just a lot 00:13:41.417-->00:13:45.333 of oversight that can happen. These things are like legos and they're kind of up to the 00:13:45.333-->00:13:49.333 automation engineers or sys admins or whatever to like figure out how to fit them 00:13:49.333-->00:13:54.750 together. When you provide a web hook o- or a series of events that are being admitted, its 00:13:54.750-->00:13:58.000 really up to the consumer of those events to know what to do with them, not the service 00:13:58.000-->00:14:03.417 that's providing them. So that's what we're gonna take advantage of. Uh i would say that the most 00:14:03.417-->00:14:08.792 volatile attack surface here uh is the scenario where you have a- maybe a company that has a 00:14:08.792-->00:14:14.458 public facing repository that they want contribution to for uh open source eh we want an open 00:14:14.458-->00:14:20.750 source project, we want to give back to the world, but we also don't want to validate any of 00:14:20.750-->00:14:25.750 our uh code before we do builds because you know we're we- it takes a lot of time, right? 00:14:27.875-->00:14:33.417 Well, the thing is if if they're hosting one of the internal base services and have them on a 00:14:33.417-->00:14:38.125 public repository, they're basically opening it up to allow you to run code on these se- 00:14:38.125-->00:14:42.208 build services that are internal and they're allowing that to happen from public repositories 00:14:42.208-->00:14:46.333 which is basically the world. So anyone can make a pole request, they don't have to accept the 00:14:46.333-->00:14:50.417 pole request, they don't even have to commit it to a branch or anything like that, you just cut 00:14:50.417-->00:14:56.417 a branch and make a PR and run code on their servers. So specifically when we're 00:14:56.417-->00:15:01.417 attacking the build services um, there are three main ways and I- a lot of the configuration files 00:15:03.708-->00:15:09.500 for these build services follow very very similar pattern. I mean, its almost uh its, they're 00:15:09.500-->00:15:14.667 very small differences, some of them are super set of like a docker compose file, um but most 00:15:14.667-->00:15:19.458 of them are some sort of yaml base thing and they follow kind of some patterns. So the ones 00:15:19.458-->00:15:23.958 that I really take advantage of are these pre post commands. These are commands that are raw 00:15:23.958-->00:15:29.375 Linux commands most of the time, that you're allowed to run to provision the environment in 00:15:29.375-->00:15:35.917 which you're gonna build your test. Um so in many cases they actually allow you to specify a 00:15:35.917-->00:15:40.958 shell so if you wanna use bash instead of the regular shell or something else, it will allow 00:15:40.958-->00:15:46.458 you to specify the shell on the image that you want to run this code in. Uh and not only that 00:15:46.458-->00:15:52.542 but some of these systems allow you to, if you need to run as root, just tell me. Just add the 00:15:52.542-->00:15:58.250 uh sudo required field to the to the repo and we'll give you a root container instead. Uh cause 00:15:58.250-->00:16:03.750 that's easier. Um the other way which I think this would be really fun to explore and I 00:16:03.750-->00:16:07.667 haven't explored it at all yet. But I think it would be really awesome. A lot of these specify 00:16:07.667-->00:16:12.250 an image in which to build. Like a docker image and some of them allow you to blindly take an 00:16:12.250-->00:16:17.417 image from uh docker hub and pull that down and do you builds on that, but other ones actually 00:16:17.417-->00:16:22.542 uh will allow you to provide a docker file and it will do a build of the image you want to 00:16:22.542-->00:16:29.500 run on. So rather than worrying about injecting uh arbitrary commands on Linux you know 00:16:29.500-->00:16:33.375 command line to uh compromise these machines, why don't you just pull down an already 00:16:33.375-->00:16:39.292 poisoned uh image and run that. It's already weaponized. You don't have to do any work. Um 00:16:39.292-->00:16:43.000 and the third way would be test builds. Some of these systems are getting privy to the fact 00:16:43.000-->00:16:47.708 that you can uh change the configuration files although most of them really haven't done 00:16:47.708-->00:16:53.542 much about it. Um but like the new version of Drone I think has the capability to say hey, if 00:16:53.542-->00:16:58.125 the config cha- file is changed don't don't do a build. Well that's fine except most of these 00:16:58.125-->00:17:03.500 build files actually specify a test file to run. So who cares if I can't run uh change the 00:17:03.500-->00:17:09.167 configuration file right? I can just change the test script that runs and do all my malicious 00:17:09.167-->00:17:14.000 stuff through the test. So I'll give you a real world example um, this was a colleague of mine 00:17:14.000-->00:17:18.792 came to me. I really didn't have quite a full understanding of these systems yet, I knew what 00:17:18.792-->00:17:23.833 they did, I knew how they worked, but I hadn't really poked around in them. But this 00:17:23.833-->00:17:28.833 is a drone um configuration. This is the real one. And uh he came to me and say I think you 00:17:31.667-->00:17:36.000 know if someone malicious had access to one of our repos, they could actually end up running 00:17:36.000-->00:17:40.917 code on the build server. And I was like ok. He he was worried about them changing environment 00:17:40.917-->00:17:44.208 variables. That seemed to be his concern he was cha- worried about environment variables 00:17:44.208-->00:17:49.583 being changed. But as you can see here uh it specifies the image on which to build. Some 00:17:49.583-->00:17:54.958 environment variables to provision, um some commands to run and that's where the that's 00:17:54.958-->00:18:00.250 where it gets awesome. Um and and then later you have the published published to the 00:18:00.250-->00:18:05.500 docker repository um and then some plugin information for alerting and suck. Uh so I 00:18:05.500-->00:18:10.208 thought alright, so let's do some command injection and just do like uh like an echo 00:18:10.208-->00:18:15.208 statement right? Just see if it says uh hello, in the output of the build service. So I did and 00:18:18.333-->00:18:24.667 it worked. And I I couldn't believe it. Um but it wasn't that of a you know it was just 00:18:24.667-->00:18:29.000 an echo system and it seemed according to my reading that this is exactly how these 00:18:29.000-->00:18:32.208 systems were intended to function. It didn't seem like that big of a deal to me. Um but 00:18:32.208-->00:18:37.875 I thought, if there's no limit to these commands right, if they're not limiting the set of 00:18:37.875-->00:18:42.292 commands that I'm able to run on these containers. I could do something like that. Right? Is 00:18:42.292-->00:18:48.333 one command. And then something like that is another command and then if I had some sort of like 00:18:48.333-->00:18:53.167 listening server, right? So with something like this I was actually able to exploit the 00:18:53.167-->00:18:58.167 same repo they were talking about and get root on the build container. Which is awesome, but 00:19:01.500-->00:19:06.417 I really I did not fully understand this. Um I had a whole lot of questions, so like, 00:19:06.417-->00:19:12.042 who's aware of this, right? Surely I'm not the only person to think this might be a 00:19:12.042-->00:19:16.833 concern. And I'm doing a reading of a continuous integration build server providers, its 00:19:16.833-->00:19:23.292 like, um most of them feel that this is the intended functionality. Yet all there's a 00:19:23.292-->00:19:27.083 whole bunch of exposure through public facing repos to these build systems. Even if they're 00:19:27.083-->00:19:31.667 cloud based I feel like there's implications here. But I wasn't fully aware of what the 00:19:31.667-->00:19:35.917 implications might be. Like what are all the different scenarios of public facing versus private 00:19:35.917-->00:19:41.667 versus cloud based versus on prim. Um and what is the real exposure here, like what's the 00:19:41.667-->00:19:47.292 real attack surface? How many people are actually doing builds on their servers. Um through 00:19:47.292-->00:19:53.250 public facing repositories. Or just repositories in general. So it turns out I was not the first 00:19:53.250-->00:20:00.000 to uh eh- go down this avenue. Uh Jonathan Claudius I think he's now Mozilla, but he came up 00:20:00.000-->00:20:04.625 with a tool a number of like three or four years ago called Rotten Apple. And it was really 00:20:04.625-->00:20:10.458 cool and I drew a lot from it. It um is an exploit or its a framework for exploiting CI, I 00:20:10.458-->00:20:14.667 think he focused on Jenkins through Ruby code builds. Now I showed you there's three ways 00:20:14.667-->00:20:19.833 that you might exploit through the configuration file, his focused on um uh malicious uh 00:20:19.833-->00:20:24.833 test runs so the the te- the code tests the third way. Um and he had an audit framework for 00:20:28.917-->00:20:34.792 kind of telling you wh- how your Ruby Jenkins build might succeed or fail. He also had an attack 00:20:34.792-->00:20:41.375 mode on this thing where you could actually like do call backs. Um uh you know dump SSH 00:20:41.375-->00:20:46.292 keys and things like this on the potentially vulnerable servers. Um there was a few other 00:20:46.292-->00:20:52.542 whitepapers out here on it, but for being like three years of time in there, uh I was kind of 00:20:52.542-->00:20:57.833 surprised that like more hackers hadn't hopped on this. Um but I though I'll kee- I'll poke 00:20:57.833-->00:21:03.167 around. Seems interesting to me. Either I don't understand this well enough to uh uh do anything 00:21:03.167-->00:21:08.458 successful with it or uh its just something that isn't being looked at. Um so I started 00:21:08.458-->00:21:13.583 poking around in bug bounty uh in trying some of this on some repos which as it runs out is 00:21:13.583-->00:21:19.083 out of scope for these tests. Um but that was learned lessons learned later. Um the so I was 00:21:19.083-->00:21:24.333 doing a test against I think it was square and I did this sam- same drone type attack except 00:21:24.333-->00:21:29.292 against there Travis servers, but it was a cloud hosted Travis servers and I let them know, I 00:21:29.292-->00:21:34.500 said hey, I got, got on your Travis server I think I could mess up your build process or 00:21:34.500-->00:21:38.833 whatever. And they uh came back and said, basically like this is not a concern to us. Like this 00:21:38.833-->00:21:43.833 doe- this affects us in no way. So I'm thinking like, I feel like it does. I- [laughter] You 00:21:47.875-->00:21:52.875 know eh, ih- the to their point, it wasn't running code on their own servers but it was um, uh it 00:21:55.917-->00:22:00.875 was at least running code in their deployment chain and it was queuing up builds in their 00:22:00.875-->00:22:05.500 build process so, I felt like at this point I still wasn't quite sure of the full implications of 00:22:05.500-->00:22:10.500 these. So I started to try to figure out who cares about this. I care. But it seemed like this 00:22:13.625-->00:22:17.583 vendor in particular didn't care. So some of the implications for cloud based 00:22:17.583-->00:22:21.708 services at the very least if I'm building code and I'm doing it through like your cloud based 00:22:21.708-->00:22:26.583 version of like Travis for example. Yeah, I'm not doing anything on your infrastructure. 00:22:26.583-->00:22:31.708 I am leveraging your repositories to do things on a server that's running code. Um I 00:22:31.708-->00:22:37.708 could clog up your build chain at the very least. Um cause these things queue up right? So 00:22:37.708-->00:22:41.417 if I open like six hundred pole requests, I'm going to open six hundred queue builds that are 00:22:41.417-->00:22:47.167 just gonna sit there until they go through so. Quasi denial service maybe? Um but this is 00:22:47.167-->00:22:53.875 free computing time. Uh these are free computing resources. So uh for example drones computing 00:22:53.875-->00:22:58.875 resources I think max out at fifty minutes per build, they max out at, they close after ten 00:23:01.000-->00:23:05.208 minutes of node standard out, but I wrote some loops in my exploits to kind of like push 00:23:05.208-->00:23:10.250 that to you know dump a standard outline every minute so it it reaches the full fifty minutes 00:23:10.250-->00:23:16.000 of potential. Drone is like sixty minutes by default. Um but it's free computing power, so 00:23:16.000-->00:23:22.583 like why hasn't anyone built a botnet in like dos the crap out of something? Um I'm not 00:23:22.583-->00:23:26.500 encouraging this. I said yes, please. But that has a question mark. I'm really not condoning 00:23:26.500-->00:23:30.458 this. But thinking about it. Open up like a whole bunch of pole requests to a whole bunch 00:23:30.458-->00:23:34.667 of exposed um services that are all triggering builds through one service and you're gonna 00:23:34.667-->00:23:41.083 have this massive CI service, like swarming on some target. Right? Um I'm kind of surprised 00:23:41.083-->00:23:47.042 that it hasn't happened yet. Um or you could, yeah this was s joke. Right. I don't know how 00:23:47.042-->00:23:52.042 you do block chain on a temporary build container. But you could. Um but for the 00:23:54.167-->00:23:59.167 on-prem this is like where the real threat might might exist. Um so if you take over one of 00:24:02.458-->00:24:07.083 these built systems even though its on a container um you could take over a network. Because 00:24:07.083-->00:24:11.208 these containers share IP space, even though they're on, might be on their own mesh network of 00:24:11.208-->00:24:16.208 IPs, they still have often times ports mapped to the hosts. Um there was an example on 00:24:20.125-->00:24:25.125 Jonathan's uh product er um his uh code where he actually was able to um submit a pole request 00:24:30.417-->00:24:34.500 that would uh come make a commit back to master because that trust relationship between the 00:24:34.500-->00:24:38.583 build service and the repository wasn't in place, so even though you didn't have any permissions 00:24:38.583-->00:24:43.792 on the repository, uh it had a sufficient SSH keys on the build service to talk back to the 00:24:43.792-->00:24:48.000 repository and make a commit to master so you could actually change master code without even 00:24:48.000-->00:24:54.458 having permission to access the repo. Um or like the other ones. And that's that's plausible too 00:24:54.458-->00:25:00.500 uh in cloud based services, but you could also have the ability to alter downstream deployment 00:25:00.500-->00:25:06.958 environments. Um moving on from the build services down to the deployment services or the 00:25:06.958-->00:25:11.667 hosted services rather, um there's some microservice sort of practices that are being 00:25:11.667-->00:25:14.875 adopted that aren't the healthiest. One is the environment variables that are 00:25:14.875-->00:25:20.792 being used to store um now this is again once you've done the build and you've deployed to 00:25:20.792-->00:25:24.458 production, environment variables are being used to store credentials to access 00:25:24.458-->00:25:28.667 other services. This is one step better than having hard coded credentials in like a config 00:25:28.667-->00:25:33.125 file, but its its when you compromise one of these services, like I mentioned 00:25:33.125-->00:25:37.625 earlier. Yeah, you you haven't compromised the entire system, but dump some environment 00:25:37.625-->00:25:43.958 variables and you'll probably be able to pivot to some of the other systems. Um additionally 00:25:43.958-->00:25:50.417 these um the only recently has so sort of the docker namespace stuff improved to a point where 00:25:50.417-->00:25:55.417 its not really really hard or a pain to run uh a container not as root. Um and so that's what 00:25:57.917-->00:26:02.875 you find a lot of these services are running this root. Now you might be thinking well, huh, who 00:26:02.875-->00:26:07.667 cares its a container right? It's a temporary container at that. Um yeah but like I said 00:26:07.667-->00:26:10.417 earlier you have ports that are mapped to the network, so even if you're on like a one 00:26:10.417-->00:26:14.250 ninety-two one six eight, I don't know, some sort of private IP mesh that the containers are 00:26:14.250-->00:26:20.458 sitting on, chances are if you scan like a ten zero zero zero, like a slash eight network and 00:26:20.458-->00:26:25.708 like scan that, you're going to be able to access stuff on the host level. Um this the true 00:26:25.708-->00:26:31.583 most of the time. Um and also just because you might have a reduced footprint image like an 00:26:31.583-->00:26:35.792 alpine Linux image which has a really footprint and that's really cool and it has its 00:26:35.792-->00:26:41.083 place, if you compromise that Linux image and you have access to the internet on that image, 00:26:41.083-->00:26:45.708 it doesn't really matter that it has a reduced footprint. I can augment that image to be 00:26:45.708-->00:26:50.917 whatever I want it to be. So I'll give you another example. So I hadn't learned my lesson at 00:26:50.917-->00:26:56.917 this point and I was still poking around um the and in this case I was uh, I was doing the 00:26:56.917-->00:27:02.000 same thing I did to the other vendor but I was doing it to Facebook and I was doing it 00:27:02.000-->00:27:07.208 through uh their Travis server as well. I was just trying to explore and find things. But 00:27:07.208-->00:27:13.042 this is an example where I actually ended up compromising the um uh the CI vendor 00:27:13.042-->00:27:18.750 themselves. Um so I was doing some basic reconnaissance work here. So just some basic 00:27:18.750-->00:27:22.750 information to see what was on the container uh I do this sometimes just to see if I'm 00:27:22.750-->00:27:27.250 running on-prem or if I'm running the cloud if I'm too lazy to go to the Travis UI and 00:27:27.250-->00:27:32.250 check. Um but uh I noticed a strange set of environment variables that were like GC they 00:27:36.792-->00:27:42.583 lent themself to like google and google cloud and so I started doing some research. I hadn't 00:27:42.583-->00:27:47.500 had much experience with uh google cloud and I um uh I had more experience with Azure and 00:27:47.500-->00:27:52.500 AWS and so I started to figure out that these environments variables lent themselves to the 00:27:54.792-->00:27:59.667 GCloud CLI which is a command line utility to uh control infrastructure, right? So if I'm 00:27:59.667-->00:28:04.333 a company that's using google cloud to host all my servers on, uh this is this is the utility 00:28:04.333-->00:28:09.625 to actually uh provision those environments. This is this should be something that the 00:28:09.625-->00:28:15.917 user of the CI server should have no part in using even having access to has no value 00:28:15.917-->00:28:22.750 that I can see anyways to um the user uh this something that uh Travis would want control over 00:28:22.750-->00:28:27.292 in order to provision their service to everyone. So I thought, in particular I found 00:28:27.292-->00:28:32.125 this command set in the first two uh were of th- most interest to me because they actually are 00:28:32.125-->00:28:37.125 what are used to provision resources. Um and eh- you get project information. So I 00:28:39.750-->00:28:45.833 thought alright, I'll give it a shot right, I will do a basic command here in my pole request 00:28:45.833-->00:28:52.625 just to see if I can look at the project. And it worked. I I couldn't believe it. I dumped 00:28:52.625-->00:28:57.417 project info for that current build which included SSH keys and some other things about that 00:28:57.417-->00:29:02.542 service. So I was like ah man, this is this is pretty cool. But this again is kind of like read 00:29:02.542-->00:29:07.500 only information I don't know how big of a deal this is. Like the real trial right is going to 00:29:07.500-->00:29:11.333 be if I can use this utility through a pole request to provision resources on their 00:29:11.333-->00:29:16.333 behalf. So I tried. And I couldn't believe it, but I tried and I did the sudo gcloud 00:29:19.833-->00:29:24.625 compute networks create test network mode auto all this does is create a network called test 00:29:24.625-->00:29:30.917 networks and it ran and it didn't error out. And I was super happy. And then the second 00:29:30.917-->00:29:37.542 command down there uh lists them just so I wanted to verify that this indeed had happened as it 00:29:37.542-->00:29:42.750 happened its called test network and not only that, but it provisioned these test networks 00:29:42.750-->00:29:47.542 in um multiple regions. [laughter] Yeah, so that was pretty exciting. I was really 00:29:47.542-->00:29:53.500 really tempted to uh do this again and open up servers, but I know that like this IP space 00:29:53.500-->00:29:58.542 stuff is usually virtual until its used and servers tend to charge money so I didn't have to 00:29:58.542-->00:30:01.458 want to come them and say hey I did this thing I found this vulnerability and I spun up 00:30:01.458-->00:30:08.375 servers and you'll get a bill for it. [laughter] But this was, this was, this was it. Um really 00:30:08.375-->00:30:14.750 this could be reduced to uh two lines. One is the sudo required which says hey I need a 00:30:14.750-->00:30:19.375 container that allows me to run as root and then I just needed one line. So with a two line PR, 00:30:19.375-->00:30:24.375 to a um repository that I didn't have access to I was able to uh deploy a g redundant network 00:30:32.583-->00:30:37.583 that spanned three continents. [applause] Um so now we'll talk about CIDER and I there's some 00:30:41.625-->00:30:46.708 video problems here, so I was gonna try to do a live demo for you, but I hopefully have a 00:30:46.708-->00:30:51.917 recorded demo if it works, so we'll do that instead. Um but I really wanted to show that to 00:30:51.917-->00:30:58.250 you. So maybe I'll show you afterwards in the hallway. Um but CIDER is the Continuous 00:30:58.250-->00:31:03.000 Integration and Deployment ExploiteR. And there's not a lot of words that start with CI, so 00:31:03.000-->00:31:08.917 I was really scraping for acronyms here. Um its a framework for exploiting and 00:31:08.917-->00:31:13.917 attacking the CI build chain. Mine primarily leverages um the github as its attack service to 00:31:15.917-->00:31:21.667 get access the build services. So I'm really exploiting the build services here, um although 00:31:21.667-->00:31:26.125 there are so much potential to do I I'm hoping this kind of catches on and people want to 00:31:26.125-->00:31:31.250 use it and exploit maybe further downstream services, like deployment services, image 00:31:31.250-->00:31:35.875 repositories or find other vendors or avenues of exploitation, but this is what 00:31:35.875-->00:31:40.875 mind does. Um it takes out sort of the mess out of um forking and PR-ing. Um and callbacking. 00:31:43.167-->00:31:49.083 I have it handling callback shells. Um so you can basically exploit a whole bunch and just 00:31:49.083-->00:31:55.625 have your nice little sessions running and hop into the shells whenever you want to. Um in a- I 00:31:55.625-->00:32:00.167 have a vendor specific exploit in which you just load and run. So its kind of point and shoot, 00:32:00.167-->00:32:06.333 you give it a list of target repos um it figures out which ones the exploit is currently a- 00:32:06.333-->00:32:12.500 able to work against um and then it it just runs them. I did it for fun. And I did it because 00:32:12.500-->00:32:19.167 I'm also lazy and man when you have to uh when you have to fork a target repository, pull that 00:32:19.167-->00:32:24.083 fork down, make the change commit that back to your attackers repository, request a 00:32:24.083-->00:32:29.458 PR over and back to the target repository, have that build and then have your little test fail 00:32:29.458-->00:32:33.875 and then you have to do it all over again. It gets like, it gets very repetitive and very 00:32:33.875-->00:32:40.167 very old. So part of it was my own selfish laziness. But I also wanted to really bring 00:32:40.167-->00:32:45.292 awareness, I feel like this isn't being looked at very much at all and in fact these systems 00:32:45.292-->00:32:51.250 often times have in their user agreements uh uh things in place that will um, like for example 00:32:51.250-->00:32:57.750 the the exploit I just showed you. I got banned from github for that because um yeah, as it 00:32:57.750-->00:33:02.000 turns out you're not su- ha, right. It turns out you're not uh its against the user 00:33:02.000-->00:33:06.333 agreement to test against repos that you don't own. But I was really inspired by uh Jonathan 00:33:06.333-->00:33:13.042 Claudius' um uh Rotten Apple product and I sort of wanted to expand upon that and facilitate 00:33:13.042-->00:33:17.875 for the research. So I'm not gonna go over all the commands but here is it's basically a CLI 00:33:17.875-->00:33:21.667 kind of like a social engineering toolkit on Metasploit things like that. Um 00:33:21.667-->00:33:28.667 and I'll be posting the code after this talk. Um basic help menus. Uh add targets and list 00:33:28.667-->00:33:34.750 your targets. Uh you could load your repository or load your eh exploit and actually you list 00:33:34.750-->00:33:39.750 exploits as well, so list the ones, find the ones you want and load them up. You can dump some 00:33:41.833-->00:33:48.208 basic information about them and hit run. Um its written in node version eight. Um its uh you'll 00:33:48.208-->00:33:55.083 also need a github account and an Ngrok account um these are free. So create a free account, 00:33:55.083-->00:34:01.375 get your keys, um because in in there's a feature in CIDER where you can do login and provide 00:34:01.375-->00:34:07.167 your credentials so it stores them in an encrypted manner um that way it can use these keys 00:34:07.167-->00:34:13.833 in part of the attack. Um it can handle bulk list of repos. Uh I'm going to add a feature to do 00:34:13.833-->00:34:20.333 some cleanup right now. It just reads your commits and poll request and wide opens. Um so it 00:34:20.333-->00:34:25.500 is a little messy at this point. Eh. From an attackers perspective if you're trying to 00:34:25.500-->00:34:30.375 be stealthy about this. Um, which I'm not condoning by the way. Uh but in I'm using Ngrok 00:34:30.375-->00:34:35.708 too uh because port forwarding sucks. Um sometimes I'm just to lazy to open a port and have the 00:34:35.708-->00:34:41.000 callback shell talk to my machine. I also am sometimes too lazy to hop on a pen test box 00:34:41.000-->00:34:45.750 and load it up with the tools that I want. Like if you have a pen test box that already has a 00:34:45.750-->00:34:52.417 public facing endpoint um so Ngrok works basically like this. Its a service and it opens up a 00:34:52.417-->00:34:58.000 random uh subnet on their endpoint on a random port and it provides you that and it tunnels 00:34:58.000-->00:35:02.625 that traffic back to your machine that's running Ngrok and then you can forward, its 00:35:02.625-->00:35:08.625 basically a proxy that tunnels out public web. That uh then you can forward it onto your machine 00:35:08.625-->00:35:12.708 so like when I'm spinning up a shell, I open up a net cat instance in Ngrok pointing to 00:35:12.708-->00:35:18.042 the net cat instance and then I have all my callbacks hit the Ngrok endpoint and call back in. 00:35:18.042-->00:35:23.792 Its just so I can like sit here and know on uh a network that I don't have uh access to and get 00:35:23.792-->00:35:28.792 callbacks. Uh like I said earlier, it's against the github user agreement to test against a 00:35:31.667-->00:35:36.667 repository even if you have permission from the web or from the owner of the repo. This 00:35:40.000-->00:35:45.000 means that you must be the owner owner of the repo to test the repo. So in testing be sure to 00:35:50.292-->00:35:55.292 ask them to make you an owner. [laughter] Alright, so I'll do a video demo time. Yes. So I am 00:36:00.042-->00:36:04.917 listing stuff here. I'm listing the exploits, so I listed targets, now I'm listing the 00:36:04.917-->00:36:11.583 exploits. Um I'm going to load an exploit. In this case I'm loading the Travis net cat uh 00:36:11.583-->00:36:18.000 reverse shell. Which all this does is take every possible target and creates a shell for 00:36:18.000-->00:36:23.000 it and uh poisons the necessary repositories and uh creates a callback to the um to the shells 00:36:26.833-->00:36:32.625 that I have open. Uh so what this is doing in this example has already been forked um these 00:36:32.625-->00:36:38.958 uh s- s- I'd already forked them so it saw that. I had already cloned them locally so it saw 00:36:38.958-->00:36:44.958 that as well. It determined which of these of the four that I was trying to attack, the four 00:36:44.958-->00:36:50.042 in the list only two of them were Travis repositories uh its only, it figured that out and 00:36:50.042-->00:36:55.667 only opening ports for those two. It started the Ngrok service. Grabbed the data from 00:36:55.667-->00:37:00.917 the Ngrok service and shoved it into the poisoned um Travis yaml file. And then it shoved the 00:37:00.917-->00:37:05.917 commit back up to my hackers repository and requested a PR. So now what's happening is the 00:37:07.958-->00:37:13.625 PR happened is triggering a build um and in theory there's a toolbar here where I could 00:37:13.625-->00:37:20.292 fast-forward, but you know. But I have it doing a callback, so you'll see. It will say, the 00:37:20.292-->00:37:25.500 eagle has landed. Uh when I get a callback from that shell. I'm probably gonna take that feature 00:37:25.500-->00:37:30.167 out. I did it for demo purposes, but it is kind of feels good when that pops up, so there it 00:37:30.167-->00:37:35.167 pops up, the eagle has landed um so now I drop into my sessions menu after I celebrate by text. 00:37:37.833-->00:37:42.833 Um and with the sessions menu it will actually um drop me into a new sort of area with its own 00:37:47.042-->00:37:52.500 help menu where I can list the currently available sessions and I just select the sessions. The 00:37:52.500-->00:37:56.708 sessions are named by target repository type, so I'm attacking a group called space 00:37:56.708-->00:37:58.708 testers on. And I have shell on that Travis server. [applause] And then I can back out of 00:37:58.708-->00:38:00.708 there, I hit back, its the only Linux command that will not work if there's a Linux command 00:38:00.708-->00:38:02.708 called back. It will hop out of the session, back to the session root menu and then if I hit back 00:38:02.708-->00:38:04.708 again uh out to the uh other menu so, that is CIDER in a nutshell. There's other exploits 00:38:04.708-->00:38:09.708 I've written for this uh and I've wrote it modularly to help urge contribution. Um but the uh 00:38:32.750-->00:38:37.958 there's some limitations here again the build queues, they really build up, they stack up. 00:38:37.958-->00:38:43.542 Um so if you're trying to get code on multiple repos to execute and attack or do 00:38:43.542-->00:38:48.625 something at the exact same time, you're probably going to have variances in in the times 00:38:48.625-->00:38:53.208 in which your code executes. Um it generates a lot of noise in github. I don't know who's 00:38:53.208-->00:38:57.417 monitoring your github logs for malicious stuff, but if they are, they're gonna, they're 00:38:57.417-->00:39:01.583 gonna see it. The good thing is if you get control of their build server, it doesn't matter 00:39:01.583-->00:39:06.708 if they ban you from github, um you already have access over their ac- access over their 00:39:06.708-->00:39:09.583 build server. So it doesn't really matter anymore. Um there's timeout issues. So I've 00:39:09.583-->00:39:14.417 tried to maximize the timeouts here so for things like the shell and you get fifty minutes 00:39:14.417-->00:39:18.917 of that shell. After fifty minutes the build is gonna fail. I've thought about doing some 00:39:18.917-->00:39:24.292 things like uh having it spawn a new build process or something like that to kind of, kind of 00:39:24.292-->00:39:29.583 persist access, but I I haven't really gone down that route yet. Um and then uh keep in mind it 00:39:29.583-->00:39:35.708 this is leveraging the github API at this point, so they have throttling in place I think at 00:39:35.708-->00:39:42.208 five thousand requests per hour um so if you try and do more than that those APer I requests 00:39:42.208-->00:39:47.250 against target repos then you're gonna hit a wall there, so. Um this is just the beginning, I 00:39:47.250-->00:39:51.167 would like to add more CI framework. I fell like Jenkins is a popular one that I just 00:39:51.167-->00:39:55.417 haven't explored yet. And I think that needs to be added. I'd like to start tackling more 00:39:55.417-->00:40:01.167 deployment services as well um downstream deployment services. I really haven't done much of 00:40:01.167-->00:40:06.500 that. And I'd also like to start exploring other entry points. Uh right I'm using only github, but 00:40:06.500-->00:40:11.542 I could do git lab or bit bucket or something like that that might kick of builds. I also um 00:40:11.542-->00:40:16.750 there's a new movement called chatOps, it just makes me kind of nauseous, but its just 00:40:16.750-->00:40:21.792 basically kicking off production builds and deployments from slack. So I feel like that's 00:40:21.792-->00:40:26.792 just ripe. Like that's just, oh man. Uh I thanks to the Leankit DevOpps team. Uh Eval for 00:40:31.000-->00:40:35.458 sending me down a rabbit hole. Jonathan for his work and my wife for being cool through 00:40:35.458-->00:40:41.792 staying up til four AM for the last month uh working on this software and this talk. Um I can 00:40:41.792-->00:40:47.792 be found on github where I'll be posting this code in the next hour or so and twitter at 00:40:47.792-->00:40:52.792 spacebox zero two X's and my blog is untamed theory. Thank you. [applause]