WEBVTT 00:02.880 --> 00:14.260 Hi everyone, welcome to Shopping for Vulnerabilities, how cloud service provider marketplaces can help white and black hat vulnerability research. 00:14.260 --> 00:21.900 This is a presentation that's being recorded after DEF CON. 00:21.900 --> 00:29.220 You know, I hope you all enjoyed Vegas, the ones that could attend, that you are all safe and were able to dodge the COVID. 00:29.900 --> 00:37.740 And if you're just watching this and didn't go to the event, hope I see you on a future edition of the event. 00:37.740 --> 00:39.760 My name is Alex Sierra. 00:40.340 --> 00:49.760 I am only one of three people that was involved in creating this content. 00:49.780 --> 00:57.580 I am the co-founder and CTO at Tenchi Security, but I've been working in information security for over 25 years now. 00:57.580 --> 01:00.960 You know, the copious gray hair. 01:01.840 --> 01:17.620 And previously I was co-founder and CTO at Cypher and then at Needle, two startups you might have heard of, and was for a brief moment, global head of detection response products and head of product management for those products at Verizon. 01:17.640 --> 01:21.480 I am certified AWS certified security specialty as well. 01:21.480 --> 01:25.440 You can find me on Twitter and you have my email address there. 01:25.440 --> 01:32.480 And I was mostly focused on the AWS part of this presentation, as well as setting up the whole thing. 01:32.860 --> 01:47.520 And I have Gleison Tomas from our security research team to thank for the Azure part of the research and for Marcelo Lima, as far as the Google Cloud part of the research as well, right? 01:47.520 --> 01:59.420 So Tenchi is a Brazilian security company that's focused on third-party security and cloud security, which is why this subject matter that we're going to be talking about is near and dear to our hearts. 01:59.660 --> 02:20.280 So hopefully by the end of this meeting, of this meeting, of this presentation, you will have learned a little bit more about those marketplaces' work, and in particular, how can they be used by both well-meaning and malicious players, unfortunately, to advance finding vulnerabilities in security products. 02:20.280 --> 02:22.280 And why is this important? 02:22.280 --> 02:23.140 Why is this relevant? 02:23.140 --> 02:39.180 The inspiration for this talk came maybe a year and a year and a half ago, when there was like a couple of weeks where tons of very, very high CVSS score vulnerabilities were found in internet-facing networking or security products, right? 02:39.420 --> 02:44.240 And so something triggered in my brain saying, why now? 02:44.240 --> 02:57.280 Why are we suddenly finding so many vulnerabilities in so many of those internet-facing products, and especially like SSL VPNs, and firewalls, and web application firewalls, and load balancers? 02:57.280 --> 02:59.260 So that was really, really weird, right? 02:59.280 --> 03:02.400 And we all know those are juicy targets, right? 03:02.400 --> 03:04.820 Those are interesting things. 03:05.280 --> 03:16.220 So the Conti leaks came out a while back, where someone was able to get the internal communications of a big ransomware crew. 03:16.560 --> 03:26.800 And one of the things that, again, caught my attention was how they explicitly talked about how they would love to get their hands on security products, right? 03:26.800 --> 03:38.260 They wanted to get their hands, in particular, on SonicWall firewalls to be able to exploit and test the exploitation of a vulnerability. 03:38.260 --> 03:49.380 And also, they wanted to get their hands on some endpoint security products, so they could check if those products were efficient in blocking their payloads and things like that. 03:49.600 --> 03:51.940 And so why are black hats interested in that? 03:51.940 --> 04:13.520 So if you think about access brokers, those players in the malicious actor marketplaces that gain the initial foothold into companies and then sell them, of course, any internet-facing product that allows them to get the initial access into a company and then maybe even pivot internally means money to them. 04:13.520 --> 04:24.300 So any internet-facing networking or security appliance that they can get their hands on to find vulnerabilities or to learn how to exploit previously known vulnerabilities is a big win. 04:24.380 --> 04:28.640 Malware developers want to bypass detection and prevention on security products. 04:28.720 --> 04:32.800 And so if you think about it, perimeter security appliances are like a double whammy. 04:32.800 --> 04:34.220 They check both boxes. 04:34.220 --> 04:37.240 So they're particularly interesting to black hats. 04:37.240 --> 04:42.100 And white hats also want the same things, but for different reasons, right? 04:42.100 --> 04:51.080 So maybe you are thinking about purchasing one of those products at the company you work for and you want to evaluate how secure they are independently. 04:51.080 --> 04:53.240 So we want to test their security. 04:53.560 --> 04:56.640 Maybe you want to profit from bug bounties. 04:56.640 --> 05:05.380 A lot of those large vendors, networking equipment and security vendors, offer cash prizes if you find vulnerabilities in their products. 05:05.720 --> 05:15.200 But most importantly of all, right, you want to find a vulnerability and brand it and create a logo and a webpage for it and become rich and famous, right? 05:15.620 --> 05:17.220 Well, anyway. 05:18.120 --> 05:31.800 And so one of the things that changed recently, for some definition of recently, is that a lot of those products used to be sold just as hardware appliances. 05:32.000 --> 05:39.400 And when you sell something as hardware, right, you have some additional level of control over who gets it. 05:39.400 --> 05:50.200 Because for example, you know, some countries are under sanctions and they have a hard time buying products, you know, from Western technology companies. 05:50.200 --> 06:00.540 So it's an extra step for them to get hardware that contains running software and even licenses for some of those products. 06:00.540 --> 06:06.460 It's not like the nice people at Hacking Village that can just go on eBay because they are in the U.S. 06:06.460 --> 06:13.460 and buy, you know, discontinued or previously owned election equipment. 06:13.620 --> 06:22.320 If you were in Russia, if you're in Iran or, you know, North Korea, it's not as easy to do that to buy a major appliance. 06:22.360 --> 06:26.200 There are ways around it, of course, and you see that happening with the Conti League. 06:26.200 --> 06:32.900 I think they end up being able to acquire some of those things through a proxy company in France or something. 06:33.040 --> 06:34.540 But it's an added step. 06:34.540 --> 06:36.020 It's an added difficulty, right? 06:36.020 --> 06:43.500 And even if you do have the hardware, who knows if some of that hardware has temporary protection that might delete the software. 06:43.840 --> 06:50.220 If you're going to need a valid license to run that, that's still going to be hard to get. 06:50.320 --> 07:03.340 But at the very least, if you get your hands on hardware, you could maybe detach the hard disk where the software is installed, and then you mount that as a secondary disk on another machine you control, and then you get to play around with it. 07:03.340 --> 07:12.080 You get to see the low-level content of that and try to, you know, reverse engineer binaries, just understand how that product works. 07:12.080 --> 07:14.200 There's a benefit to that. 07:14.440 --> 07:20.000 And relying on hardware means you don't have a ton of control over which versions you get, necessarily. 07:20.240 --> 07:22.320 So it's a mess. 07:22.820 --> 07:34.980 But luckily for both white and black hat attackers, a lot of those products, like most of them, are now available for purchase very conveniently through cloud service provider marketplaces. 07:34.980 --> 07:57.140 If you go to AWS Marketplace, Google Cloud Marketplace, and Azure Marketplace, which are the three major cloud providers and the ones we covered in this research, you just see a huge number of products, right, that you can very easily get and run like in minutes in a virtual machine on one of your environments. 07:57.180 --> 07:59.440 And you can play around with them very easily. 07:59.440 --> 08:17.140 So just immediately, even if you don't do anything too weird with it, you immediately are able to get the latest version of those products to run and do black box testing against it, right, paying typically by the hour or by the minute. 08:17.140 --> 08:27.840 You don't even need to do like the down payment, the investment of buying an appliance, which can be very expensive in some of those cases, to start testing it. 08:27.840 --> 08:32.700 And you can stop it when you're not working on it and not be charged anymore, and then start it up again. 08:33.280 --> 08:38.360 And in some cases, even run different versions in parallel, right, to do testing. 08:38.360 --> 08:49.980 So right off the bat, the same kind of benefit the automation of Infrastructure as a Service gives to IT in general, it's now paying off for attackers as well. 08:49.980 --> 08:52.160 It becomes a very convenient way to do testing. 08:52.160 --> 08:58.200 And we can see no shortage of products that are available on those marketplaces. 08:58.200 --> 09:00.640 But how far can you go with it, right? 09:00.640 --> 09:02.680 What can you do, right? 09:03.680 --> 09:16.360 So let's talk about how AWS Marketplace works, and let's try to do the same sort of things with AWS Marketplace products that we want to do with hardware appliances, right, that I just mentioned. 09:17.620 --> 09:29.580 So looking at how AWS Marketplace works, and we're particularly focused, by the way, on this research on virtual machine type products, like Infrastructure as a Service products. 09:29.580 --> 09:37.920 We're not looking at SaaS because you can buy SaaS licenses from marketplaces, you can buy machine learning models, you can buy datasets. 09:37.920 --> 09:39.580 We're not looking at any of that. 09:39.580 --> 09:49.960 We're focusing on, again, the cloud equivalent of buying a used hardware on eBay and then doing reverse engineering, right? 09:49.960 --> 09:54.020 This is what we're trying to do here as a service, like on the cloud. 09:54.380 --> 10:03.760 So in AWS, they have a few things that are very specific to when you run an EC2 instance that came from the marketplace. 10:03.760 --> 10:07.140 One of those things is the product codes, right? 10:07.160 --> 10:11.460 So AWS Marketplace is not a very friendly product. 10:11.460 --> 10:14.820 It doesn't have APIs for the buyers. 10:14.820 --> 10:17.100 It does have APIs for the sellers. 10:17.100 --> 10:33.040 If you're a vendor that's selling a VM-based product on the marketplace, you have APIs to know who bought your product, when, what licenses did they purchase, to automate the creation of new offers or publishing a new version of your VM, things like that. 10:33.040 --> 10:37.700 But as a seller , those APIs, they do exist if you go on the console. 10:37.700 --> 10:45.760 And when you see there's a tab for marketplace , you can see there are hidden APIs and documented APIs at discovery.marketplace.region. 10:46.200 --> 10:47.520 amazonaws .com. 10:47.520 --> 10:49.940 But they're not on any of the SDKs. 10:49.940 --> 10:51.580 They're not anywhere, right? 10:51.580 --> 10:56.480 They're really driven to being used by the browser, which is very atypical for AWS. 10:56.480 --> 11:00.320 And I really found that experience rather icky. 11:00.780 --> 11:10.120 But if you look a little bit, you're going to notice that every EC2 instance you create from the marketplace gets something called the product code. 11:10.180 --> 11:10.340 Right? 11:10.340 --> 11:25.600 It's an ID, much like an instance ID, that marks that instance as having come from the marketplace and uniquely identifies which product version it's associated with, right? 11:25.600 --> 11:30.940 So, one of the places it appears that surprised me is on the EC2 instance metadata service. 11:30.940 --> 11:41.760 If you query what's called an instance identity document on the EC2 instance metadata service, you're going to see a field called marketplace product codes. 11:41.760 --> 11:43.220 That's actually an array. 11:43.220 --> 11:46.620 I've never seen it be populated with more than one value. 11:46.620 --> 11:47.580 But there it is. 11:47.580 --> 11:52.580 It's a product code for which product is associated with that. 11:52.580 --> 11:56.860 It was used... which product image was used to create that instance, right? 11:56.860 --> 12:17.300 Based on the seller documentation for the marketplace on AWS, it seems like this exists so that any software that the seller puts into their image is able to query at runtime to see if the instance it's running from actually has a valid product code. 12:17.300 --> 12:29.900 It's kind of an anti-cheat measure so that software can check if it was copied off of a VM that came from the marketplace and copied somewhere else to be run, right? 12:29.900 --> 12:37.960 So, kind of, again , trying to help you as a seller enforce if your software was properly licensed for the marketplace , right? 12:38.860 --> 12:42.080 And it's shown to buyers at least in four places. 12:42.080 --> 12:51.980 So, in the marketplace wizard, when you're creating an instance, you see the... it's been confusingly called offer ID instead of product code. 12:52.000 --> 13:03.640 But unless you had like a bespoke offer where someone offered you a non -standard discount, the offer ID is the same as the product code, right? 13:03.900 --> 13:13.920 When you get a notification email, you know, telling you that you've accepted the offer, again, the offer ID will be populated with the product code. 13:13.920 --> 13:21.300 You can see it on the identity document on the EC2 instance metadata service, which I just described. 13:21.300 --> 13:28.320 But also, if you call EC2 describe instances, that same field marketplace product codes will also be present. 13:28.320 --> 13:40.300 So, you don't have to be inside the instance to check, to realize it was created from the marketplace and to have some indication of which product it was created from, right? 13:41.700 --> 13:46.900 So, that immediately gives us some useful uses, right? 13:46.900 --> 13:54.740 So, first of all, if you do have an instance product code and you visit this URL here on a browser, unfortunately, it's human-centric. 13:54.740 --> 13:58.760 Again, there are no buyer-specific APIs for this. 13:58.760 --> 14:01.640 You can find out what product it is. 14:01.640 --> 14:03.520 Oh, it's a firewall, it's a load balancer, etc. 14:03.520 --> 14:12.060 So, immediately, that becomes useful, say, if you're a defender, if you're working on a security operation center doing incident response. 14:12.060 --> 14:15.240 And one of the things that you ask first is, what is this instance? 14:15.240 --> 14:16.900 Someone's attacking this instance. 14:16.920 --> 14:17.800 What is this? 14:17.800 --> 14:19.100 Does it have any private data ? 14:19.100 --> 14:20.860 What is this instance's role? 14:20.860 --> 14:33.440 If you suddenly realize, oh yeah, this is a Palo Alto firewall , you immediately learn a lot about what that instance is, what it does, what its role is, what its criticality is in your company, right? 14:34.240 --> 14:37.180 And so, here are a few commands that you can use. 14:37.180 --> 14:55.240 If you have the right to call EC2 describe instances on a particular AWS account, you can use filters on the describe instances command to filter out, in this case, companies that have come from the marketplace, regardless of which product code they belong to. 14:55.240 --> 15:03.740 And you can see here, searching for a particular product code, individual product code, right? 15:03.920 --> 15:10.060 So, I just want to see, you know, things created from this particular version of this product on the marketplace. 15:10.060 --> 15:12.200 So, it can be very specific, right? 15:12.240 --> 15:16.360 So, this could be used if you're a defender as an enrichment to SIEM events. 15:16.360 --> 15:19.820 This can be used to populate asset management details. 15:19.820 --> 15:21.640 So, there's a ton you can do. 15:21.640 --> 15:24.720 And for attackers as well, like this can be used for recon. 15:24.720 --> 15:28.760 If you're able to break into an instance, you don't know what it does. 15:29.020 --> 15:40.220 You can run commands, or if you have an SSRF, you can run, get the identity document from the EC2 metadata service. 15:40.220 --> 15:50.080 And again, if it has a product code, without even touching the file system, without doing anything that could be noisy, you immediately know what that machine is, right? 15:50.080 --> 15:55.840 Because you know the role of that machine, which product it was used to create it, right? 15:55.900 --> 16:07.920 So, that is very subtle, very unlikely to trigger any alarms anywhere, and gives you a ton of information about the machine you just landed on, right? 16:07.920 --> 16:10.620 So, very helpful for attackers as well. 16:12.060 --> 16:28.080 So, one of the things we did was you can also list AMIs, because in order for a product to be on the marketplace, the image, the AMI, or the Amazon machine image that is used to create new instances from it, has to be made public. 16:28.080 --> 16:33.160 It has to be listed on the describe images command, right? 16:33.160 --> 16:39.780 So, again, we have here a command that says, give me the AMIs associated with a particular product code. 16:40.420 --> 16:44.440 So, what we did first is the obvious thing. 16:44.440 --> 16:53.580 Let's try to create a new instance directly from that public AMI, without subscribing to it first on the marketplace. 16:53.580 --> 16:54.840 And it fails. 16:54.920 --> 16:56.040 And it says, no, no, no, no, no. 16:56.040 --> 16:57.320 You have to opt in first. 16:57.320 --> 16:58.580 You have to accept the offer. 16:58.580 --> 17:02.920 You need to have a valid license for this product in order to use that AMI. 17:02.920 --> 17:09.460 So, the AMI is marked the same way that, I don't know, binaries in Windows have the mark of the internet. 17:09.460 --> 17:15.340 If you downloaded it from the web and it's, you know, restricts what you can do immediately with it. 17:15.340 --> 17:20.880 So, the AMIs and the snapshots associated with that AMI have the mark of the marketplace. 17:21.000 --> 17:27.080 And you can't immediately start an instance from them unless you accepted the license first. 17:28.060 --> 17:40.100 So, after you accepted the offer for the marketplace, you can then start as many instances as you want based on those images. 17:40.500 --> 17:42.060 And they will be charged. 17:42.060 --> 17:44.120 They will work as expected, right? 17:44.120 --> 17:53.160 And you can see, again, after you run them, as I mentioned before, the describe instances command, the output will show you the product code ID and the product code type, right? 17:53.160 --> 17:54.920 Saying this came from the marketplace. 17:54.920 --> 17:58.760 This is the product ID of the product associated. 18:00.820 --> 18:05.600 So, now, this immediately allows an attacker again to do black box testing. 18:05.820 --> 18:07.560 So, let's try the next thing. 18:07.560 --> 18:18.280 So, let's try the equivalent of bought the hardware, unplugged the hard drive for the boot disk, and then I start rummaging through it and seeing, you know, looking at the software that's running on this appliance. 18:18.280 --> 18:28.860 Because as we know, most appliances will not just give you like SSH and a shell as root to mess around with the binaries and the file system that the vendor created, right? 18:28.860 --> 18:34.620 So, can we bypass that protection in getting to the contents of that boot disk? 18:34.740 --> 18:37.600 So, the first thing I tried is the obvious thing. 18:37.600 --> 18:54.280 Again, create a marketplace instance, stop it, detach the boot disk, create a new Linux instance, and attach that disk as a secondary volume, right, on that new instance whose boot I control and where it can SSH as root. 18:54.320 --> 18:55.860 And it fails. 18:55.860 --> 19:15.000 It seems, again, that the volumes and the snapshots that came somewhere from the marketplace have a mark of the marketplace and the boot volumes from instances or AMIs that came from the marketplace can only ever be used as boot volumes. 19:15.000 --> 19:20.000 They are not allowed to be accessed as secondary volumes. 19:20.280 --> 19:24.000 So, no, no luck there. 19:24.260 --> 19:25.960 So, let's try something else. 19:25.960 --> 19:28.880 I'm going to create a snapshot from that volume. 19:28.880 --> 19:43.360 I'm going to create a completely new volume from that snapshot and then try to attach it as a secondary volume on EBS volume on a new instance and no, again, it's not working. 19:43.360 --> 19:51.020 AWS saw that coming and that mark of the marketplace persists even though I went through a snapshot, right? 19:51.020 --> 19:53.000 So, no dice. 19:53.680 --> 19:57.520 So, maybe the associations through the AMI. 19:57.700 --> 20:00.400 So, I stop the marketplace instance. 20:00.400 --> 20:02.440 I create a completely new AMI. 20:02.440 --> 20:06.060 That's not the AMI that those volumes came originally from. 20:06.060 --> 20:08.680 Create a new instance from that AMI. 20:08.680 --> 20:09.920 It still works. 20:09.920 --> 20:11.740 And then stop it. 20:11.740 --> 20:15.340 Detach the boot volume and try to add it as a secondary disk. 20:15.560 --> 20:17.260 No, no, no, no, no, no. 20:17.440 --> 20:24.300 AWS's team did a great job there and also covered that avenue. 20:24.420 --> 20:26.500 That also does not work. 20:28.420 --> 20:46.560 Looked at a mailing list and found some people from the Packer tool, which is used to create AMIs and container images, et cetera, saying they found this, they bumped into this problem and they suggested a workaround where you create an AMI from the marketplace instance, 20:46.560 --> 20:52.260 and then you make that new AMI public, or at least share it with a different unrelated account. 20:52.260 --> 20:54.960 And then you create an instance on that account. 20:54.960 --> 21:00.820 And apparently crossing that AWS account barrier made the mark of the marketplace go away. 21:01.040 --> 21:02.980 And it's not working anymore. 21:02.980 --> 21:05.080 This was from a couple of years ago. 21:05.080 --> 21:10.200 Someone in AWS clearly saw that post and closed that loophole as well. 21:11.820 --> 21:21.140 So I tried everything I could think of, and maybe some of you listening to this talk are smarter than me and can think of different things and ways to bypass this. 21:21.140 --> 21:23.440 Actually, a great topic for research. 21:23.440 --> 21:24.880 I tried two other things. 21:24.880 --> 21:31.000 I tried copying a snapshot to S3 and try to get access to direct raw content that way. 21:31.000 --> 21:32.660 Also didn't work. 21:32.660 --> 21:42.680 And AWS also launched recently something called direct access APIs, where you can get, again, low-level access to the bytes inside of an EBS volume. 21:42.680 --> 21:51.220 And this is meant to be used by traditional backup solutions to be able to directly access the content of EBS volumes. 21:51.260 --> 21:53.340 And that didn't work either. 21:53.340 --> 22:06.600 AWS did a great job of closing every possible avenue I can think of to try to have raw access to the content of that marketplace instance boot disk. 22:06.820 --> 22:08.340 So I don't know. 22:08.340 --> 22:10.480 Great job, AWS, I guess. 22:11.120 --> 22:13.800 And you had to think, by the way, about this. 22:13.800 --> 22:24.640 If you are counting on having access to all your instances for direct access to do backups, including virtual appliances you bought from the marketplace, that's not going to work. 22:24.740 --> 22:31.540 You will need to make sure as a defender or as an infrastructure manager on the cloud that you realize this is a limitation. 22:32.080 --> 22:45.180 And you will have to rely on vendor-specific backup strategies, like whatever backup functionality the maker of that virtual appliance provides. 22:45.320 --> 22:51.800 And you will not be able to use AWS backup or direct access to do backups of that. 22:51.800 --> 23:11.500 So keep that in mind on your backup strategy to make sure you can still recover those virtual machine instances, those marketplace-originated instances in your environment in the case of failures or ransomware or whatever. 23:13.180 --> 23:24.580 And just to make sure we don't give AWS a free pass and a complete win on this, there was one problem I found not related to accessing the boot volume, which is the main goal. 23:24.580 --> 23:33.800 But it seems like AWS is very fast and loose with who can publish stuff on the marketplace. 23:33.860 --> 23:40.580 So, for example, you're not necessarily obligated to charge for products on the marketplace. 23:40.580 --> 23:47.400 You can create a bring-your-own license or even a free product, for example, for open-source stuff on the marketplaces. 23:47.520 --> 23:59.100 And they do no KYC, no know-your-customer at all, if you are trying to publish free products or even if you're publishing paid products outside of EMEA. 23:59.100 --> 24:06.340 Their documentation is pretty clear, actually, that they only require know-your-customer because they're forced to by European legislations. 24:06.680 --> 24:13.920 I haven't tried, by the way, publishing a paid product, but I did try publishing a free product. 24:13.920 --> 24:20.040 So, when you create your seller profile, you can choose whatever logo and company URL you choose. 24:20.040 --> 24:24.560 I chose things that were completely unrelated to my company. 24:24.560 --> 24:44.840 So, I could very well try to create a seller account for trying to mimic a large organization, do a lookalike domain, grab someone's logo, and try to pretend to be a large technology vendor and try to list products there and ensnare unsuspecting victims. 24:44.840 --> 24:52.300 The same way people do with software packages on npm or pypi and typosquatting on domains. 24:52.300 --> 24:54.880 Seems like this is fully automated. 24:54.880 --> 24:59.900 As soon as I added a logo in a URL, it was immediately available on the marketplace. 24:59.900 --> 25:03.360 So, there seems to be no human review whatsoever of this. 25:03.480 --> 25:10.920 So, I think it's an interesting place here for AWS to address and improve their security. 25:11.220 --> 25:20.980 The other thing that I found interesting is AWS purports to do regular scanning of images to check for vulnerabilities and even provides a functionality for sellers to check their images. 25:20.980 --> 25:25.220 I haven't tested it, but it seems like an interesting feature. 25:25.220 --> 25:31.580 And they have standards, such as no known vulnerabilities, no hard-coded passwords, no remote access by seller. 25:31.880 --> 25:39.940 But it's not clear how regularly they scan images, only at upload, or how that works. 25:40.720 --> 25:49.100 But at the very least, they should be checking that any URLs you publish, you are the owner of that domain. 25:49.100 --> 25:51.180 They should prove that they own that domain. 25:51.180 --> 26:00.060 They might want to do a human review to make sure that the logos being published are based on trademarks that actually belong to that seller. 26:00.060 --> 26:07.320 They're not trying to purport to be, I don't know, Cisco or Palo Alto or Checkpoint or a well-known company to try to fool people, right? 26:07.320 --> 26:19.020 So, I think this is a bit dangerous because people will trust that if this is on the AWS marketplace that it went through some vetting, where all the indications I found is that it does not. 26:19.020 --> 26:32.760 As soon as you hit publish, it immediately goes out to the listing on the marketplace, unless you're trying to sell a paid product, and you say you want to target EMEA customers, right? 26:32.760 --> 26:40.780 So, that's the only scenario where some know your customer will be done and documentation will be requested, as per the documentation and my testing. 26:42.320 --> 26:49.760 If you go to the Azure marketplace, it's actually a breath of fresh air as far as the user experience. 26:49.820 --> 27:00.760 And that's very on-brand for Azure, actually, in my experience, where they have very well-documented APIs for buyers to interact with their marketplace. 27:00.760 --> 27:01.200 Who knew? 27:01.200 --> 27:03.360 Who knew that was possible, right? 27:03.360 --> 27:06.180 And they have very good documentation about this. 27:06.180 --> 27:18.960 So, by the way, learning all those things about AWS meant trying them out and reading the seller documentation, I could find very little documentation on the buyer side that explained the internals of how things work. 27:18.960 --> 27:23.700 Not so much for Azure, especially because their API is very well-documented. 27:23.700 --> 27:33.880 So, if you look at how Azure marketplace products work, they have essentially four attributes. 27:33.880 --> 27:45.780 So, the publisher, which is like the seller, the name of the organization, the offer is kind of the product, the SKU would be like a major release version of the product or presentation of the product, and then the version. 27:45.820 --> 27:56.940 And if you combine the four things separated by colons, it's like a unique identifier of a particular virtual machine that you can download and install on your environment. 27:56.940 --> 28:10.660 But you can also replace the version number with latest, like you can do with containers, typically, and get the latest possible version, as long as you know the publisher, offer, and SKU, right? 28:10.660 --> 28:15.100 And you can click here to see the documentation that Azure publishes on that. 28:16.120 --> 28:34.040 So, one of the things we found is that differently from AWS, you can detach the boot disk from a marketplace image and attach it as a secondary disk on another virtual machine you created and rummage around. 28:35.300 --> 28:37.780 Gleason was able to find a way to do this. 28:37.780 --> 28:54.860 So, the way you start is you find the URL for a particular marketplace product, and on that URL, you're going to realize that the publisher and the offer are already contained on the URL, right after slash apps. 28:55.040 --> 28:56.900 So, in this case, Tidal Media Inc. 28:56.900 --> 29:03.420 is the publisher, and Tinyproxy, Easy, Ubuntu, I just picked the first product that appeared on my listing, right? 29:03.420 --> 29:05.360 I don't even know what this is. 29:05.520 --> 29:07.020 This is the offer, right? 29:07.020 --> 29:21.920 If you know the publisher and the offer, you can use this Azure CLI command on the Azure shell, the cloud shell, to figure out the SKU name, or the SKU names, if they have multiple ones. 29:21.920 --> 29:23.540 This case just has one. 29:24.000 --> 29:30.160 And then you can obtain the URN of all of the images, right? 29:30.160 --> 29:44.700 You can, basically, if you already have these three, you have the publisher, offer, and SKU, you could just append colon latest and get the latest version, or you can list what versions exist, and you can choose which one you're going to explore, right? 29:44.700 --> 29:46.240 So, this is very convenient. 29:46.240 --> 29:53.040 Keep old versions around, apparently, so you can try and find vulnerabilities in multiple versions of a product. 29:53.040 --> 29:54.880 How cool is that, right? 29:55.660 --> 30:06.060 So, the next thing you would do, if you were following their web wizard, would be to accept the license terms, before you run a VM based on that product. 30:06.200 --> 30:09.660 This seems to be optional, as far as Gleason was concerned. 30:09.660 --> 30:17.300 Even if he skipped this step, he was still able to create virtual machines, based on the marketplace product. 30:17.300 --> 30:28.900 So, again, I'm not a lawyer, but it seems like if you never accepted the license terms of the product, maybe some restrictions or reverse engineering don't apply. 30:28.900 --> 30:30.680 Again, consult a lawyer. 30:30.680 --> 30:36.160 I am not a lawyer, but this certainly doesn't look good, as far as Azure. 30:36.200 --> 30:46.280 But again, you know, giving company A access to company B's data seems to be very on-brand for Azure recently, given how many cross-tenant vulnerabilities have been found. 30:46.340 --> 30:48.860 So, again, very on-brand for Azure, right? 30:48.860 --> 30:56.580 Allowing people to use the intellectual property of a seller on their marketplace without accepting their license terms? 30:56.580 --> 30:57.500 I don't know. 30:57.500 --> 31:05.860 In any case, after you do that, what you can do is create a disk based on that image, right? 31:05.860 --> 31:30.240 So, instead of creating a VM based on the marketplace entry, you can just create a disk and providing, as the image reference, that full unique identifier with publisher, offer, SKU, and version, and then just create a new VM with your preferred operating system and attach that recently created disk as a secondary disk on that VM. 31:30.240 --> 31:43.340 And then, boom, you can boot your VM, you can mount that disk, and you can play around with the raw contents and, you know, find all sorts of interesting things inside the products you want to do vulnerability research for. 31:43.340 --> 31:44.780 How cool is that? 31:46.140 --> 31:51.160 On Google Cloud, we were also able to do the same thing, right? 31:51.160 --> 31:59.640 Again, Marcello found this procedure here that allows you to get any product there that's a virtual machine based and do the same thing. 31:59.640 --> 32:16.900 You go to the cloud console, the Google Cloud console, go to the marketplace section, you pick up a vendor, which use whatever filters you want, deploy the application, so follow the regular steps to create a virtual machine, a compute instance, from that product, 32:16.900 --> 32:34.880 then stop it, create a snapshot from that instance that you just stopped, create a new disk from that snapshot, and then create a new instance and attach that disk as a secondary disk to that VM, and then just log into your newly created VM, mount that disk, 32:34.880 --> 32:38.400 and explore it away to your heart's content, right? 32:38.700 --> 32:47.320 He gives a little bit more feedback on, you know, using fdisk to figure out which partitions there are and mount them independently, things like that, right? 32:48.100 --> 32:51.480 So you can play around with it. 32:51.480 --> 33:06.420 You can also just do like a DD, get the raw contents of that disk and download it, like save it to, you know, S3 or cloud storage or blob storage, and then download it to your machine so you don't even have to continue the analysis by running it on the cloud, 33:06.420 --> 33:09.820 maybe run a local VM, or do something else, right? 33:09.820 --> 33:19.560 At that point, if you have raw access to the bytes contained in that boot volume, you know, you can do whatever you want and, you know, play around with it. 33:21.040 --> 33:34.880 So this hopefully provides some context into how marketplaces can be a big boom to both white and black hat hackers. 33:35.580 --> 33:44.380 I think of the three marketplaces we analyzed, AWS seems to offer the best protection for sellers, intellectual property. 33:44.700 --> 33:52.720 And actually, it's even safer than selling hardware is my current conclusion based on what I've seen on the AWS marketplace. 33:52.720 --> 34:01.980 If you are a product vendor, you're less likely to have your binaries reversed engineered by selling it through AWS marketplace than through hardware. 34:01.980 --> 34:06.220 Because if someone does get a hold of your hardware, they basically can do whatever they want. 34:06.220 --> 34:14.960 Whereas if you're running on AWS, AWS is enforcing a ton of stuff that would have no parallel in the physical world. 34:14.960 --> 34:18.080 So that's pretty interesting. 34:18.120 --> 34:29.160 And as an owner, like an IT manager or blue team person, beware of the restrictions that that imposes on what you can do with your instances, right? 34:29.160 --> 34:43.160 Again, backing them up seems to be the biggest sticking point where you have to use vendor-specific strategies to back them up, because most of the standard things could have limitations, right? 34:44.060 --> 34:47.660 Azure and GCP marketplaces, though, are fair game, right? 34:47.660 --> 34:49.880 And they're the second and third largest marketplaces. 34:49.880 --> 34:56.060 AWS is the largest, same as with their market share, as we've seen on the beginning slide. 34:56.200 --> 35:07.000 But you can use Azure and GCP marketplaces to do a lot of research and find vulnerabilities and get some money on bug bounties, right? 35:07.040 --> 35:09.460 So a few things to keep in mind. 35:09.460 --> 35:13.680 There are low or no cost options, so either bring your own license or pay as you go. 35:13.680 --> 35:22.880 And you only need to run the actual instance from the marketplace for a short amount of time before you take an image and do other things, and you can play around with the content. 35:22.880 --> 35:25.900 So you wouldn't need to run it for more than an hour, perhaps. 35:27.080 --> 35:34.620 You don't have any sanctions or logistics-related difficulties that are involved in obtaining hardware. 35:35.100 --> 35:36.940 Just virtualize it. 35:36.960 --> 35:38.640 Very low barrier to entry. 35:38.640 --> 35:49.260 If you're an attacker, if you're someone malicious, you can just create a cloud account with a fake or stolen credit card and start plugging away with things there. 35:49.260 --> 35:50.940 Don't even have to pay for it. 35:51.300 --> 35:54.540 And actually, this is very amenable to automation, right? 35:54.700 --> 36:14.120 I'm sorry to release this out into the world, but DevSecResearchOps is now a thing, apparently, because you can automate creating instances, creating those images, rummaging through the content, or even trying out as a black box your exploits to figure out how many versions are affected by a particular thing, 36:14.120 --> 36:16.880 just because the cloud is automatable, right? 36:16.880 --> 36:23.400 So attackers of both colored hats can now enjoy this. 36:23.560 --> 36:27.080 And I want to emphasize again, I am not a lawyer. 36:27.080 --> 36:33.000 We never try to actually access any of the content of those boot disks to do our research. 36:33.000 --> 36:37.100 We just confirm that we could see the partition table and that's it. 36:37.900 --> 36:44.220 We don't know how legal or illegal this will be in different places in the world. 36:45.320 --> 36:51.660 And so make sure whatever you're doing, you don't break any laws, you don't get arrested. 36:52.440 --> 36:54.080 And, you know, please don't. 36:54.080 --> 36:55.980 I don't want that on my conscience. 36:55.980 --> 37:09.860 Having said that, this could be a very interesting strategy if you know what you're doing to getting some money on black bounties or to help move the security of those products forward by finding and reporting vulnerabilities. 37:11.860 --> 37:13.160 And that's it. 37:13.160 --> 37:14.860 That's the content of my talk. 37:14.860 --> 37:17.880 I hope you guys found this enjoyable. 37:18.220 --> 37:21.200 Let me know you have my contact information here. 37:21.200 --> 37:33.020 If you liked it, if you have any feedback, if you were able to find ways around the AWS protections that I wasn't able to, I'd love to hear more about the subject because I think it's pretty interesting. 37:33.020 --> 37:52.940 I think it's an underappreciated, this new level of cheap, easy automated access to things that were previously very hard to test security-wise could be one of the reasons why we're seeing so many new vulnerabilities in this kind of enterprise product recently. 37:52.940 --> 37:54.480 Hope you enjoyed it. 37:54.480 --> 38:09.280 My deepest thank you, heartfelt thanks to Jayesh and the entire Cloud Village team for putting this up together again, for keeping up with my and tolerating my nonsense. 38:09.340 --> 38:13.320 I'm very lucky and privileged to be able to present here. 38:13.460 --> 38:14.660 Hope you enjoyed this. 38:14.660 --> 38:15.380 Bye-bye.