Hi everyone, welcome to Shopping for Vulnerabilities, how cloud service provider marketplaces can help white and black hat vulnerability research.
This is a presentation that's being recorded after DEF CON.
You know, I hope you all enjoyed Vegas, the ones that could attend, that you are all safe and were able to dodge the COVID.
And if you're just watching this and didn't go to the event, hope I see you on a future edition of the event.
My name is Alex Sierra.
I am only one of three people that was involved in creating this content.
I am the co-founder and CTO at Tenchi Security, but I've been working in information security for over 25 years now.
You know, the copious gray hair.
And previously I was co-founder and CTO at Cypher and then at Needle, two startups you might have heard of, and was for a brief moment, global head of detection response products and head of product management for those products at Verizon.
I am certified AWS certified security specialty as well.
You can find me on Twitter and you have my email address there.
And I was mostly focused on the AWS part of this presentation, as well as setting up the whole thing.
And I have Gleison Tomas from our security research team to thank for the Azure part of the research and for Marcelo Lima, as far as the Google Cloud part of the research as well, right?
So Tenchi is a Brazilian security company that's focused on third-party security and cloud security, which is why this subject matter that we're going to be talking about is near and dear to our hearts.
So hopefully by the end of this meeting, of this meeting, of this presentation, you will have learned a little bit more about those marketplaces' work, and in particular, how can they be used by both well-meaning and malicious players, unfortunately, to advance finding vulnerabilities in security products.
And why is this important?
Why is this relevant?
The inspiration for this talk came maybe a year and a year and a half ago, when there was like a couple of weeks where tons of very, very high CVSS score vulnerabilities were found in internet-facing networking or security products, right?
And so something triggered in my brain saying, why now?
Why are we suddenly finding so many vulnerabilities in so many of those internet-facing products, and especially like SSL VPNs, and firewalls, and web application firewalls, and load balancers?
So that was really, really weird, right?
And we all know those are juicy targets, right?
Those are interesting things.
So the Conti leaks came out a while back, where someone was able to get the internal communications of a big ransomware crew.
And one of the things that, again, caught my attention was how they explicitly talked about how they would love to get their hands on security products, right?
They wanted to get their hands, in particular, on SonicWall firewalls to be able to exploit and test the exploitation of a vulnerability.
And also, they wanted to get their hands on some endpoint security products, so they could check if those products were efficient in blocking their payloads and things like that.
And so why are black hats interested in that?
So if you think about access brokers, those players in the malicious actor marketplaces that gain the initial foothold into companies and then sell them, of course, any internet-facing product that allows them to get the initial access into a company and then maybe even pivot internally means money to them.
So any internet-facing networking or security appliance that they can get their hands on to find vulnerabilities or to learn how to exploit previously known vulnerabilities is a big win.
Malware developers want to bypass detection and prevention on security products.
And so if you think about it, perimeter security appliances are like a double whammy.
They check both boxes.
So they're particularly interesting to black hats.
And white hats also want the same things, but for different reasons, right?
So maybe you are thinking about purchasing one of those products at the company you work for and you want to evaluate how secure they are independently.
So we want to test their security.
Maybe you want to profit from bug bounties.
A lot of those large vendors, networking equipment and security vendors, offer cash prizes if you find vulnerabilities in their products.
But most importantly of all, right, you want to find a vulnerability and brand it and create a logo and a webpage for it and become rich and famous, right?
Well, anyway.
And so one of the things that changed recently, for some definition of recently, is that a lot of those products used to be sold just as hardware appliances.
And when you sell something as hardware, right, you have some additional level of control over who gets it.
Because for example, you know, some countries are under sanctions and they have a hard time buying products, you know, from Western technology companies.
So it's an extra step for them to get hardware that contains running software and even licenses for some of those products.
It's not like the nice people at Hacking Village that can just go on eBay because they are in the U.S.
and buy, you know, discontinued or previously owned election equipment.
If you were in Russia, if you're in Iran or, you know, North Korea, it's not as easy to do that to buy a major appliance.
There are ways around it, of course, and you see that happening with the Conti League.
I think they end up being able to acquire some of those things through a proxy company in France or something.
But it's an added step.
It's an added difficulty, right?
And even if you do have the hardware, who knows if some of that hardware has temporary protection that might delete the software.
If you're going to need a valid license to run that, that's still going to be hard to get.
But at the very least, if you get your hands on hardware, you could maybe detach the hard disk where the software is installed, and then you mount that as a secondary disk on another machine you control, and then you get to play around with it.
You get to see the low-level content of that and try to, you know, reverse engineer binaries, just understand how that product works.
There's a benefit to that.
And relying on hardware means you don't have a ton of control over which versions you get, necessarily.
So it's a mess.
But luckily for both white and black hat attackers, a lot of those products, like most of them, are now available for purchase very conveniently through cloud service provider marketplaces.
If you go to AWS Marketplace, Google Cloud Marketplace, and Azure Marketplace, which are the three major cloud providers and the ones we covered in this research, you just see a huge number of products, right, that you can very easily get and run like in minutes in a virtual machine on one of your environments.
And you can play around with them very easily.
So just immediately, even if you don't do anything too weird with it, you immediately are able to get the latest version of those products to run and do black box testing against it, right, paying typically by the hour or by the minute.
You don't even need to do like the down payment, the investment of buying an appliance, which can be very expensive in some of those cases, to start testing it.
And you can stop it when you're not working on it and not be charged anymore, and then start it up again.
And in some cases, even run different versions in parallel, right, to do testing.
So right off the bat, the same kind of benefit the automation of Infrastructure as a Service gives to IT in general, it's now paying off for attackers as well.
It becomes a very convenient way to do testing.
And we can see no shortage of products that are available on those marketplaces.
But how far can you go with it, right?
What can you do, right?
So let's talk about how AWS Marketplace works, and let's try to do the same sort of things with AWS Marketplace products that we want to do with hardware appliances, right, that I just mentioned.
So looking at how AWS Marketplace works, and we're particularly focused, by the way, on this research on virtual machine type products, like Infrastructure as a Service products.
We're not looking at SaaS because you can buy SaaS licenses from marketplaces, you can buy machine learning models, you can buy datasets.
We're not looking at any of that.
We're focusing on, again, the cloud equivalent of buying a used hardware on eBay and then doing reverse engineering, right?
This is what we're trying to do here as a service, like on the cloud.
So in AWS, they have a few things that are very specific to when you run an EC2 instance that came from the marketplace.
One of those things is the product codes, right?
So AWS Marketplace is not a very friendly product.
It doesn't have APIs for the buyers.
It does have APIs for the sellers.
If you're a vendor that's selling a VM-based product on the marketplace, you have APIs to know who bought your product, when, what licenses did they purchase, to automate the creation of new offers or publishing a new version of your VM, things like that.
But as a seller , those APIs, they do exist if you go on the console.
And when you see there's a tab for marketplace , you can see there are hidden APIs and documented APIs at discovery.marketplace.region.
amazonaws .com.
But they're not on any of the SDKs.
They're not anywhere, right?
They're really driven to being used by the browser, which is very atypical for AWS.
And I really found that experience rather icky.
But if you look a little bit, you're going to notice that every EC2 instance you create from the marketplace gets something called the product code.
Right?
It's an ID, much like an instance ID, that marks that instance as having come from the marketplace and uniquely identifies which product version it's associated with, right?
So, one of the places it appears that surprised me is on the EC2 instance metadata service.
If you query what's called an instance identity document on the EC2 instance metadata service, you're going to see a field called marketplace product codes.
That's actually an array.
I've never seen it be populated with more than one value.
But there it is.
It's a product code for which product is associated with that.
It was used... which product image was used to create that instance, right?
Based on the seller documentation for the marketplace on AWS, it seems like this exists so that any software that the seller puts into their image is able to query at runtime to see if the instance it's running from actually has a valid product code.
It's kind of an anti-cheat measure so that software can check if it was copied off of a VM that came from the marketplace and copied somewhere else to be run, right?
So, kind of, again , trying to help you as a seller enforce if your software was properly licensed for the marketplace , right?
And it's shown to buyers at least in four places.
So, in the marketplace wizard, when you're creating an instance, you see the... it's been confusingly called offer ID instead of product code.
But unless you had like a bespoke offer where someone offered you a non -standard discount, the offer ID is the same as the product code, right?
When you get a notification email, you know, telling you that you've accepted the offer, again, the offer ID will be populated with the product code.
You can see it on the identity document on the EC2 instance metadata service, which I just described.
But also, if you call EC2 describe instances, that same field marketplace product codes will also be present.
So, you don't have to be inside the instance to check, to realize it was created from the marketplace and to have some indication of which product it was created from, right?
So, that immediately gives us some useful uses, right?
So, first of all, if you do have an instance product code and you visit this URL here on a browser, unfortunately, it's human-centric.
Again, there are no buyer-specific APIs for this.
You can find out what product it is.
Oh, it's a firewall, it's a load balancer, etc.
So, immediately, that becomes useful, say, if you're a defender, if you're working on a security operation center doing incident response.
And one of the things that you ask first is, what is this instance?
Someone's attacking this instance.
What is this?
Does it have any private data ?
What is this instance's role?
If you suddenly realize, oh yeah, this is a Palo Alto firewall , you immediately learn a lot about what that instance is, what it does, what its role is, what its criticality is in your company, right?
And so, here are a few commands that you can use.
If you have the right to call EC2 describe instances on a particular AWS account, you can use filters on the describe instances command to filter out, in this case, companies that have come from the marketplace, regardless of which product code they belong to.
And you can see here, searching for a particular product code, individual product code, right?
So, I just want to see, you know, things created from this particular version of this product on the marketplace.
So, it can be very specific, right?
So, this could be used if you're a defender as an enrichment to SIEM events.
This can be used to populate asset management details.
So, there's a ton you can do.
And for attackers as well, like this can be used for recon.
If you're able to break into an instance, you don't know what it does.
You can run commands, or if you have an SSRF, you can run, get the identity document from the EC2 metadata service.
And again, if it has a product code, without even touching the file system, without doing anything that could be noisy, you immediately know what that machine is, right?
Because you know the role of that machine, which product it was used to create it, right?
So, that is very subtle, very unlikely to trigger any alarms anywhere, and gives you a ton of information about the machine you just landed on, right?
So, very helpful for attackers as well.
So, one of the things we did was you can also list AMIs, because in order for a product to be on the marketplace, the image, the AMI, or the Amazon machine image that is used to create new instances from it, has to be made public.
It has to be listed on the describe images command, right?
So, again, we have here a command that says, give me the AMIs associated with a particular product code.
So, what we did first is the obvious thing.
Let's try to create a new instance directly from that public AMI, without subscribing to it first on the marketplace.
And it fails.
And it says, no, no, no, no, no.
You have to opt in first.
You have to accept the offer.
You need to have a valid license for this product in order to use that AMI.
So, the AMI is marked the same way that, I don't know, binaries in Windows have the mark of the internet.
If you downloaded it from the web and it's, you know, restricts what you can do immediately with it.
So, the AMIs and the snapshots associated with that AMI have the mark of the marketplace.
And you can't immediately start an instance from them unless you accepted the license first.
So, after you accepted the offer for the marketplace, you can then start as many instances as you want based on those images.
And they will be charged.
They will work as expected, right?
And you can see, again, after you run them, as I mentioned before, the describe instances command, the output will show you the product code ID and the product code type, right?
Saying this came from the marketplace.
This is the product ID of the product associated.
So, now, this immediately allows an attacker again to do black box testing.
So, let's try the next thing.
So, let's try the equivalent of bought the hardware, unplugged the hard drive for the boot disk, and then I start rummaging through it and seeing, you know, looking at the software that's running on this appliance.
Because as we know, most appliances will not just give you like SSH and a shell as root to mess around with the binaries and the file system that the vendor created, right?
So, can we bypass that protection in getting to the contents of that boot disk?
So, the first thing I tried is the obvious thing.
Again, create a marketplace instance, stop it, detach the boot disk, create a new Linux instance, and attach that disk as a secondary volume, right, on that new instance whose boot I control and where it can SSH as root.
And it fails.
It seems, again, that the volumes and the snapshots that came somewhere from the marketplace have a mark of the marketplace and the boot volumes from instances or AMIs that came from the marketplace can only ever be used as boot volumes.
They are not allowed to be accessed as secondary volumes.
So, no, no luck there.
So, let's try something else.
I'm going to create a snapshot from that volume.
I'm going to create a completely new volume from that snapshot and then try to attach it as a secondary volume on EBS volume on a new instance and no, again, it's not working.
AWS saw that coming and that mark of the marketplace persists even though I went through a snapshot, right?
So, no dice.
So, maybe the associations through the AMI.
So, I stop the marketplace instance.
I create a completely new AMI.
That's not the AMI that those volumes came originally from.
Create a new instance from that AMI.
It still works.
And then stop it.
Detach the boot volume and try to add it as a secondary disk.
No, no, no, no, no, no.
AWS's team did a great job there and also covered that avenue.
That also does not work.
Looked at a mailing list and found some people from the Packer tool, which is used to create AMIs and container images, et cetera, saying they found this, they bumped into this problem and they suggested a workaround where you create an AMI from the marketplace instance,
and then you make that new AMI public, or at least share it with a different unrelated account.
And then you create an instance on that account.
And apparently crossing that AWS account barrier made the mark of the marketplace go away.
And it's not working anymore.
This was from a couple of years ago.
Someone in AWS clearly saw that post and closed that loophole as well.
So I tried everything I could think of, and maybe some of you listening to this talk are smarter than me and can think of different things and ways to bypass this.
Actually, a great topic for research.
I tried two other things.
I tried copying a snapshot to S3 and try to get access to direct raw content that way.
Also didn't work.
And AWS also launched recently something called direct access APIs, where you can get, again, low-level access to the bytes inside of an EBS volume.
And this is meant to be used by traditional backup solutions to be able to directly access the content of EBS volumes.
And that didn't work either.
AWS did a great job of closing every possible avenue I can think of to try to have raw access to the content of that marketplace instance boot disk.
So I don't know.
Great job, AWS, I guess.
And you had to think, by the way, about this.
If you are counting on having access to all your instances for direct access to do backups, including virtual appliances you bought from the marketplace, that's not going to work.
You will need to make sure as a defender or as an infrastructure manager on the cloud that you realize this is a limitation.
And you will have to rely on vendor-specific backup strategies, like whatever backup functionality the maker of that virtual appliance provides.
And you will not be able to use AWS backup or direct access to do backups of that.
So keep that in mind on your backup strategy to make sure you can still recover those virtual machine instances, those marketplace-originated instances in your environment in the case of failures or ransomware or whatever.
And just to make sure we don't give AWS a free pass and a complete win on this, there was one problem I found not related to accessing the boot volume, which is the main goal.
But it seems like AWS is very fast and loose with who can publish stuff on the marketplace.
So, for example, you're not necessarily obligated to charge for products on the marketplace.
You can create a bring-your-own license or even a free product, for example, for open-source stuff on the marketplaces.
And they do no KYC, no know-your-customer at all, if you are trying to publish free products or even if you're publishing paid products outside of EMEA.
Their documentation is pretty clear, actually, that they only require know-your-customer because they're forced to by European legislations.
I haven't tried, by the way, publishing a paid product, but I did try publishing a free product.
So, when you create your seller profile, you can choose whatever logo and company URL you choose.
I chose things that were completely unrelated to my company.
So, I could very well try to create a seller account for trying to mimic a large organization, do a lookalike domain, grab someone's logo, and try to pretend to be a large technology vendor and try to list products there and ensnare unsuspecting victims.
The same way people do with software packages on npm or pypi and typosquatting on domains.
Seems like this is fully automated.
As soon as I added a logo in a URL, it was immediately available on the marketplace.
So, there seems to be no human review whatsoever of this.
So, I think it's an interesting place here for AWS to address and improve their security.
The other thing that I found interesting is AWS purports to do regular scanning of images to check for vulnerabilities and even provides a functionality for sellers to check their images.
I haven't tested it, but it seems like an interesting feature.
And they have standards, such as no known vulnerabilities, no hard-coded passwords, no remote access by seller.
But it's not clear how regularly they scan images, only at upload, or how that works.
But at the very least, they should be checking that any URLs you publish, you are the owner of that domain.
They should prove that they own that domain.
They might want to do a human review to make sure that the logos being published are based on trademarks that actually belong to that seller.
They're not trying to purport to be, I don't know, Cisco or Palo Alto or Checkpoint or a well-known company to try to fool people, right?
So, I think this is a bit dangerous because people will trust that if this is on the AWS marketplace that it went through some vetting, where all the indications I found is that it does not.
As soon as you hit publish, it immediately goes out to the listing on the marketplace, unless you're trying to sell a paid product, and you say you want to target EMEA customers, right?
So, that's the only scenario where some know your customer will be done and documentation will be requested, as per the documentation and my testing.
If you go to the Azure marketplace, it's actually a breath of fresh air as far as the user experience.
And that's very on-brand for Azure, actually, in my experience, where they have very well-documented APIs for buyers to interact with their marketplace.
Who knew?
Who knew that was possible, right?
And they have very good documentation about this.
So, by the way, learning all those things about AWS meant trying them out and reading the seller documentation, I could find very little documentation on the buyer side that explained the internals of how things work.
Not so much for Azure, especially because their API is very well-documented.
So, if you look at how Azure marketplace products work, they have essentially four attributes.
So, the publisher, which is like the seller, the name of the organization, the offer is kind of the product, the SKU would be like a major release version of the product or presentation of the product, and then the version.
And if you combine the four things separated by colons, it's like a unique identifier of a particular virtual machine that you can download and install on your environment.
But you can also replace the version number with latest, like you can do with containers, typically, and get the latest possible version, as long as you know the publisher, offer, and SKU, right?
And you can click here to see the documentation that Azure publishes on that.
So, one of the things we found is that differently from AWS, you can detach the boot disk from a marketplace image and attach it as a secondary disk on another virtual machine you created and rummage around.
Gleason was able to find a way to do this.
So, the way you start is you find the URL for a particular marketplace product, and on that URL, you're going to realize that the publisher and the offer are already contained on the URL, right after slash apps.
So, in this case, Tidal Media Inc.
is the publisher, and Tinyproxy, Easy, Ubuntu, I just picked the first product that appeared on my listing, right?
I don't even know what this is.
This is the offer, right?
If you know the publisher and the offer, you can use this Azure CLI command on the Azure shell, the cloud shell, to figure out the SKU name, or the SKU names, if they have multiple ones.
This case just has one.
And then you can obtain the URN of all of the images, right?
You can, basically, if you already have these three, you have the publisher, offer, and SKU, you could just append colon latest and get the latest version, or you can list what versions exist, and you can choose which one you're going to explore, right?
So, this is very convenient.
Keep old versions around, apparently, so you can try and find vulnerabilities in multiple versions of a product.
How cool is that, right?
So, the next thing you would do, if you were following their web wizard, would be to accept the license terms, before you run a VM based on that product.
This seems to be optional, as far as Gleason was concerned.
Even if he skipped this step, he was still able to create virtual machines, based on the marketplace product.
So, again, I'm not a lawyer, but it seems like if you never accepted the license terms of the product, maybe some restrictions or reverse engineering don't apply.
Again, consult a lawyer.
I am not a lawyer, but this certainly doesn't look good, as far as Azure.
But again, you know, giving company A access to company B's data seems to be very on-brand for Azure recently, given how many cross-tenant vulnerabilities have been found.
So, again, very on-brand for Azure, right?
Allowing people to use the intellectual property of a seller on their marketplace without accepting their license terms?
I don't know.
In any case, after you do that, what you can do is create a disk based on that image, right?
So, instead of creating a VM based on the marketplace entry, you can just create a disk and providing, as the image reference, that full unique identifier with publisher, offer, SKU, and version, and then just create a new VM with your preferred operating system and attach that recently created disk as a secondary disk on that VM.
And then, boom, you can boot your VM, you can mount that disk, and you can play around with the raw contents and, you know, find all sorts of interesting things inside the products you want to do vulnerability research for.
How cool is that?
On Google Cloud, we were also able to do the same thing, right?
Again, Marcello found this procedure here that allows you to get any product there that's a virtual machine based and do the same thing.
You go to the cloud console, the Google Cloud console, go to the marketplace section, you pick up a vendor, which use whatever filters you want, deploy the application, so follow the regular steps to create a virtual machine, a compute instance, from that product,
then stop it, create a snapshot from that instance that you just stopped, create a new disk from that snapshot, and then create a new instance and attach that disk as a secondary disk to that VM, and then just log into your newly created VM, mount that disk,
and explore it away to your heart's content, right?
He gives a little bit more feedback on, you know, using fdisk to figure out which partitions there are and mount them independently, things like that, right?
So you can play around with it.
You can also just do like a DD, get the raw contents of that disk and download it, like save it to, you know, S3 or cloud storage or blob storage, and then download it to your machine so you don't even have to continue the analysis by running it on the cloud,
maybe run a local VM, or do something else, right?
At that point, if you have raw access to the bytes contained in that boot volume, you know, you can do whatever you want and, you know, play around with it.
So this hopefully provides some context into how marketplaces can be a big boom to both white and black hat hackers.
I think of the three marketplaces we analyzed, AWS seems to offer the best protection for sellers, intellectual property.
And actually, it's even safer than selling hardware is my current conclusion based on what I've seen on the AWS marketplace.
If you are a product vendor, you're less likely to have your binaries reversed engineered by selling it through AWS marketplace than through hardware.
Because if someone does get a hold of your hardware, they basically can do whatever they want.
Whereas if you're running on AWS, AWS is enforcing a ton of stuff that would have no parallel in the physical world.
So that's pretty interesting.
And as an owner, like an IT manager or blue team person, beware of the restrictions that that imposes on what you can do with your instances, right?
Again, backing them up seems to be the biggest sticking point where you have to use vendor-specific strategies to back them up, because most of the standard things could have limitations, right?
Azure and GCP marketplaces, though, are fair game, right?
And they're the second and third largest marketplaces.
AWS is the largest, same as with their market share, as we've seen on the beginning slide.
But you can use Azure and GCP marketplaces to do a lot of research and find vulnerabilities and get some money on bug bounties, right?
So a few things to keep in mind.
There are low or no cost options, so either bring your own license or pay as you go.
And you only need to run the actual instance from the marketplace for a short amount of time before you take an image and do other things, and you can play around with the content.
So you wouldn't need to run it for more than an hour, perhaps.
You don't have any sanctions or logistics-related difficulties that are involved in obtaining hardware.
Just virtualize it.
Very low barrier to entry.
If you're an attacker, if you're someone malicious, you can just create a cloud account with a fake or stolen credit card and start plugging away with things there.
Don't even have to pay for it.
And actually, this is very amenable to automation, right?
I'm sorry to release this out into the world, but DevSecResearchOps is now a thing, apparently, because you can automate creating instances, creating those images, rummaging through the content, or even trying out as a black box your exploits to figure out how many versions are affected by a particular thing,
just because the cloud is automatable, right?
So attackers of both colored hats can now enjoy this.
And I want to emphasize again, I am not a lawyer.
We never try to actually access any of the content of those boot disks to do our research.
We just confirm that we could see the partition table and that's it.
We don't know how legal or illegal this will be in different places in the world.
And so make sure whatever you're doing, you don't break any laws, you don't get arrested.
And, you know, please don't.
I don't want that on my conscience.
Having said that, this could be a very interesting strategy if you know what you're doing to getting some money on black bounties or to help move the security of those products forward by finding and reporting vulnerabilities.
And that's it.
That's the content of my talk.
I hope you guys found this enjoyable.
Let me know you have my contact information here.
If you liked it, if you have any feedback, if you were able to find ways around the AWS protections that I wasn't able to, I'd love to hear more about the subject because I think it's pretty interesting.
I think it's an underappreciated, this new level of cheap, easy automated access to things that were previously very hard to test security-wise could be one of the reasons why we're seeing so many new vulnerabilities in this kind of enterprise product recently.
Hope you enjoyed it.
My deepest thank you, heartfelt thanks to Jayesh and the entire Cloud Village team for putting this up together again, for keeping up with my and tolerating my nonsense.
I'm very lucky and privileged to be able to present here.
Hope you enjoyed this.
Bye-bye.