we're about to get we're about to get started i'd like to welcome uh justin and samuel uh welcome
to defcon first time speakers uh they will be given a speech on crossing the chasm uh about
an exploit that they uh performed and um as a reminder this is a public service announcement
watch your schedules online and use hacker tracker to be able to see the latest information and what
uh what's going on there's a there's a lot that goes on at the con so once again please give a
warm welcome uh to our speakers welcome guys hello hello um so i'm sam uh so this is crossing
the chasm a web app pen testing story so this talk will get a little bit technical but it's worth it
in the end i promise so who am i um i am sam aka irby sam i'm a hacker and i also have two defcon
black badges from a few years back and most importantly i got a puppy about six months ago
and so there'll be more
puppies pictures later i promise
nice and uh i'm justin gardner aka rhino raider i'm a full-time bug bounty hunter and here are
some additional facts about me um you can find me on twitter at my handle rhino raider and my dms
are open if you guys have any questions about the presentation or have any similar interests
so when we're dealing with a product called chasm we had to do something punny right so i figured
i'd do a little bit of a presentation for you guys here representing what it's like to be a bug
bounty hunter on a pretty hardened target so here here's sam and i am in the in the middle of the
desert and uh on the other side of this chasm we can we can see a mirage of rce in the distance
but we have to go for it we have no choice and unfortunately this uh this mirage is uh you know
separated from us by a technology called chasm
know represented here by the deep chasm this is a vdi software enterprise for
essentially managing what your users what applications your user can run and in what
context they do that so we threw a bunch of exploits at it nothing was working nothing
was sticking but in the end we did find the bridge over this chasm and this is that bridge
and we'll explain that a little bit more but all you need to know for now is we just hooked this up
and hopped right on over the chasm and we we made our way to the rce so i'm going to give it back to
sam to give you a little bit more context on what the heck we're talking about right now and hopefully
it'll make a little bit more sense in a moment thanks so for some important background before
we get into hacking um so chasm workspaces is you know streaming containerized apps and desktops and
users so it's really intended for enter the enterprise world where you want to administer
your end users running
ubuntu or chrome or windows inside of docker containers and it will deploy them for you
and then allow end users to connect to them and then you can you know monitor them
you know do various things enterprising things with them and it's built on top of docker and
vnc and so it'd give you an idea of just how dockerized this world is um it's turtles all
the way down it's docker containers all the way down and i apologize for putting so much
information on a slide but i'll give a visualization in a minute of what all this looks like
and so at the bottom of this list you can actually see the ubuntu desktop ubuntu docker container
that i deployed and at the top of this list you can actually see the nginx container that i deployed
or sorry the nginx uh chasm web chasm proxy and so the chasm proxy is actually listening remotely
and then every other docker container is listening on local ports and if you're not familiar with
docker you can kind of think of it as a virtual machine but instead of running on bare metal it
so it's really just kind of one layer up and then it has a bunch of access to underlying
operating system depending on what permissions you give it and so diving in a little bit into
a little more detail into that chasm proxy when you actually spin up a new container a
new nginx configuration actually gets pushed to the chasm proxy and so this containers.d folder
will be updated with a new file and so when you push a new container this container this folder
gets restarted and then the end user can connect to the newly formed docker container
and throughout this talk we'll try and share at the bottom the commands we ran as well as
references if you want to repeat this or learn more about this and so at a high level this is
what if you deploy chasm workspaces onto a single server this is what it will actually look like
your end users will connect through as well as you will connect through chasm proxy
and they'll be able to connect to their end user
containers so every blue square here actually represents a docker container
and the bottom row are the kind of command and control containers that are used to to manage
the rest of the system and they're all built on top of python it's either going to be tornado or
cherry py and so these are the purpose of all the containers the most important one being chasm proxy
with routes all your traffic and then chasm api which is actually the api you're interacting with
when you're
using the application chasm agent is also important because that actually has
the ability to and permission to deploy new containers so it's a privilege
docker container in the docker parlance with multiple servers you actually end up with
a series of secondary or when you want to deploy this into a real enterprise environment
when you have multiple users you want to deploy it across multiple machines and so in order to
accomplish this it actually will on the primary server you'll it'll keep chasm api and then chasm
proxy will proxy through to the secondary servers to run Chasm agent which will then deploy
your containers or your end user containers. And this allows you to scale infinitely. And so
very high level hand wavy way, this is roughly what credentials look like in the system. Your
end user session tokens will get passed to Chasm API as a cookie. It will also get passed
through to their containers. And then Chasm API will interact with the rest of the system using
secret keys which are all internal to the Chasm system. So, you know, at this point, you know,
we were given user credentials to a bug bounty program and we were asked to look at this without
source code access. We didn't find any vulnerabilities. This was actually really secure
software. And this talk wouldn't be here today if this wasn't secure software. It would be a very
boring talk.
So, at this point, we decided, you know, we were kind of given up looking at this without source
code and we wanted to start testing this with source code. And so before we did that, we
actually stopped at their issue tracker and just made sure that nobody else reported any security
issues first. Not really finding anything, you know, we moved on. This, so we went to their
website to see if we could download their software and we came across this page. And this page is
like the most exciting thing you can see as a bug bounty hunter.
I'll add some arrows to show why. And a few more arrows. This button means that you don't
have to call sales. You don't have to put your credit card down for some hopes of finding some
bug in the future. You can just download the software and see how it works. And as you can
see in the community edition, it has all features in enterprise which is exactly what you want to see.
So, at this point, we went through and installed this onto a server. And when we looked at the
installation process, we saw it was actually downloading from Docker Hub and installing the
Docker Container system.
That way. We then played the game of find the API that you're calling. And in this case,
we picked one called API get user attributes and we looked through all of the servers that
were running and we were able to find it in the Kazim API binary. So, we copied the binary out
of that Docker Container. This is the proper way to reverse it. The actual way that we went about
it was running strings on the binary and then Googling a whole bunch of stuff. But this is the
actual way that you're...
This is the actual way that you're
supposed to identify that it's running as a Python binary. You can then identify the version,
drop into a Python environment, extract the Python files, compiled Python files,
and then decompile them using uncompile six. The steps are listed here in case you want to
repeat this yourself. We then dove into client underscore API dot py and we were able to find
our get user attributes function. And this actually gave... Well, this concluded our game of
finding the endpoint and now we can actually do the real security work of looking at all the
endpoints for security vulnerabilities. And so, at this point, I'd actually given up almost
entirely and then Justin came along and said, hey, when you create a new Kazim workspace,
there's a host header that gets passed through as one of the inputs. And so, we were a little bit
curious to see how that worked, so like a good bug bounty hunter, we spun up Burp Collaborator
and we got callbacks. And...
And so we immediately thought that this was a crit, and game over.
We found the coolest vulnerability ever.
And unfortunately, we were only getting our own session token back.
So it's not really a crit.
It's more like a really limited blind SSRF from a security standpoint.
It's not super interesting.
And so we obviously wanted to maximize payout as bug bounty hunters.
So we decided to dive deeper.
So now it's hacking time.
All righty, so let's look a little bit more into this.
We've seen so far how we were able to reverse engineer the software,
get access to the code.
And now we're going to start finding some vulnerabilities and
figuring out why we're getting that callback.
So as we mentioned at the beginning, the configuration for
Nginx for this software is a little bit strange.
Whenever a container is spun up,
there is a new,
a new configuration file injected into this containers.d directory.
And the callbacks that we were seeing from this was other parts of the system
hitting these internal routes that they created to communicate with our new API.
We can see that here in the proxy pass definition for
both of the locations defined here.
So we had something interesting here.
After playing around with it a little bit more, we realized that it would actually,
Let's see.
leak the session token of anybody who visited the page where the thumbnail of
this container was displayed.
So if an admin came.
And they, they viewed that thumbnail, then their session would get leaked.
So, that, that made us feel a little bit better, right?
Cuz we, we have a, a, you know, a vulnerability that we,
we've definitely got secured.
But like Sam said, and like you know, my,
my lovely desert graphic in the beginning showed, we saw RCE in the distance and
we wanted to fly.
follow it. So we continued. So looking back at this configuration right here, we thought about
what other things we could do. And, of course, when you're injecting into a proxy pass, most
people would think of SSRF. We already had blind SSRF. Let's see if we can get it to be full read
SSRF. So the way the proxy pass works is when we passed this host header that we were able to
inject from the command that we, the HTTP request we did before, we just changed the host header,
that got injected right into the proxy pass here. And if we could hit these routes, then we could
hit any arbitrary host from the perspective of the server, right? And in today's, you know,
modern cloud environments, this normally results in at least something a little fun, if not full
RCE or takeover of the AWS account that this is hosted on. So we thought this would be a good
target.
to go for. But we ran into some issues here. Yeah. So the goal would be full read SSRF. We ran into
some issues here, hitting these NGINX configurations. The first one, as you can probably see, is the
first entry here has the internal directive. The internal directive in NGINX only allows this route
to be routed by internal redirects or other internal requests. So there was no way that we could
possibly hit that endpoint. The second one, we've also done a lot of work on, you know,
we also thought looked great, but that GUID is not leaked to the user in any way, shape, or form. It just gets
pulled straight out of the database and used by other pieces of the API. And we couldn't leak it to
ourselves because it's being overwritten by mod rewrite there. So that resulted in a big sad. So we
had to think a little bit more about what we were going to do. So next we noticed that there are some
variables that we can access within NGINX.
configuration file. The remote address is one of them and actually NGINX provides a really nice
alphabetized list of variables that we could go through. So we went through, I don't know if you can see it in the
screenshot there, but all 222 of those variables. And there's nothing interesting in there.
At all. Not even one. So we went through all of that and couldn't find anything fun to do with it.
But then we started thinking, okay, variables, what about
environmental variables. Could we leak environmental variables from the host machine? So we started,
like any good hackers, we Googled how to get access to environmental variables in Nginx
configuration, and this lovely stack overflow article popped up. And it mentioned running
arbitrary Lua and Perl in the context of an Nginx configuration file. So don't mind if we do.
So we decided to quickly pivot and go down that path, and, yeah, no, it's not going to be that
easy for us, unfortunately. There was no Perl or, I mean, there's no Lua or JS modules loaded,
and even though they did tease us a little bit with defining a Perl's module path in the Nginx
command, that path did not exist.
So there was no Perl modules being loaded, and we couldn't use the Perl modules.
So we're back to the drawing board, and we're trying to come up, and we're like, okay, maybe we can
modify this Nginx configuration file to do something funky.
So we start, this is the vulnerable request. We start injecting, you know, valid, or what we thought
was valid Nginx syntax into the host header, and big surprise, Nginx fussed at us.
And said the front-end proxy from Sam's explanation earlier was not accepting that host header
because it had spaces and slashes in it.
So this is getting more and more, this is getting more and more difficult, right?
So what's the sitch here, just to summarize?
We have a blind SSRF via Nginx configuration file injection.
We've got callbacks that contain the victim's cookies, but we kind of wanted to avoid any social engineering
because this is bug bounty and that's how it is.
And we can't get full read SSRF because of the internal directive, and we couldn't find the GUID.
There were no interesting variables to leak, no access to code modules, and we couldn't use any spaces
or slashes in the host header.
So this was looking, this was looking really rough.
We were feeling down at this point.
And then Sam has this lovely idea.
I'm going to read.
I'm going to try and call the API directly from within the container.
And see if Python is okay with the spaces in the host header.
Which was a lovely idea.
So essentially what he's doing here is he's using those end user containers that we're able to spin up
and he's trying to communicate directly with the Chasm API from within the containers.
So he spun up a nice Ubuntu box here.
And boom, we could reach the internal API without Nginx getting in our way.
And lo and behold, Cherrypy is a little bit less finicky about their host headers
and will accept pretty much whatever you throw at it.
And a little extra, which we'll find out later.
So that was great.
We thought we had, you know, a way to inject into Nginx configurations.
And we proceeded to write up an exploit and take it to prod.
And nope.
The prod was running the multiple server configuration.
And we cannot access the Chasm API.
So at this point Sam and I are about to lose our minds.
We've been going back and forth on this vulnerability for a while.
And we were stuck.
We were really stuck.
So we tried a bunch of things here.
We tried HTTP request smuggling.
We dove in.
I don't know why we were so, like, obsessed with this vulnerability.
But we just really were.
And we started reading the Nginx.
When you start reading Nginx source code, you know you're in trouble.
So we started reading the Nginx source code.
And we couldn't find anything in the host header, parsing logic, or anything like that there.
But then we repeated this process for Cherrypy.
And we found something really interesting regarding ISO 8859-1 headers, which I had never heard of before.
So upon finding this, I sent a frantic message to Sam.
And I think you can see above me saying, man, this bug.
And then 15 minutes later, dude, I've got it.
So you can kind of feel the bug bounty hunter, you know, drain right there.
And the lows and the highs, right?
And so let me explain a little bit about ISO 8859-1.
If that looks familiar to you for some reason, that's because it's used often in SMTP headers.
So if you've ever clicked see original in your e-mails inside of, you know, Gmail, then you'd be looking at something like this.
And one of the things that neither of us knew and that I think is not very widely known is that headers are actually supposed to, according to the original RFC,
are supposed to be able to be parsed by a 8859-1, ISO 8859-1 encoding.
So that was exactly what we needed.
And the Cherrypy code you can see up at the top right says, yeah, if there's an equal sign and then a question mark in the beginning,
let's go ahead and try to decode this value as ISO 8859-1.
So the format of this is actually very similar to URL encoding,
which is super convenient for ISO 8859-1.
We can do something similar here.
We have to define this little header in the beginning, the 8859-1?q?part,
and then essentially we can URL encode our payload and then just replace the percentage signs with equal signs,
and you get a fairly close to valid ISO 8859-1 format.
So I think this is one of the takeaways from this talk,
is that if you're having trouble smuggling headers through some sort of intermediary proxy,
whether it be a WAF or a proxy in our scenario,
this could be some way to bypass it if any of your web servers in the stack are accepting and decoding or normalizing this ISO 8859 value.
So you can see right here just the character-to-character translation of what it looks like for that to be decoded.
And the value that you see in red is what is going to be passed to CherryPY in the end when the header is parsed.
So at this point, we have a way to get through the chasm proxy, and we have a way to inject host headers.
So we're almost there.
But we first have to get through Nginx configuration.
And so here, what we're actually injecting into is we're injecting to multiple places with the same string.
So we're injecting it to two proxy passcodes.
And then four add header directives, where the string is wrapped in single-quote characters.
And so this was something that we had to bypass.
Our end goal here really was to add our own location so that we could add arbitrary Nginx configuration.
And so we tried adding a new location point to the top two injection points.
And we didn't use any single-quote characters so that it would be okay for the remaining four injection spots.
Unfortunately, that didn't happen.
Unfortunately, this doesn't work because when Nginx sees a duplicate location, it rejects the configuration.
And so that didn't work.
We got this error, duplicate location slash new.
So we then tried something a little bit more creative where we actually used some quote magic is how I'll describe it
to inject into the top two positions using quotes and then a comment to break out successfully.
And then on the last four injection points,
because the single quotes would take priority over the double quotes, we won't escape from that string.
This should have worked, except for there's one evil dollar sign in a regular expression right in the middle of where we were trying to inject into.
And Nginx rejected this because a single dollar sign is an invalid variable name in Nginx.
As an aside, there is actually a solution to this where you can put this geo dollar default dollar sign into your Nginx configuration,
but it has to be done at a higher level than we had access to.
We only had access to the location block, and then we could also break out of that into the server block.
This had to be placed in the HTTP block, which we didn't have access to.
So this didn't work.
So at this point, you've reached the most boring part of our presentation.
I promise you it gets better from here.
So now is the time to wake up if you're sleeping.
So this is really exciting.
So it turns out you can't repeat static locations in Nginx configuration,
but you can repeat regular expressions.
And this is due to the way that Nginx parses the configuration file.
It will first compile a list of prefix paths, looking to quickly find a match.
But then if it falls back to regular expression, it will actually go through them one at a time.
And the first match it gets, it will send it through.
And this allows you to actually send through multiple identical regular expression location paths.
And so this was a huge bypass.
So you can repeat regular expression, and that's big.
It was big.
So at this point, you know, we built gadgets.
We weren't quite sure exactly how we were going to exploit the system yet.
So we built a full-read SSRF finally.
We built a local file inclusion where we could actually pull files off the system.
As an aside, they were very secure, and they built the Nginx proxy as a read-only file system.
So we couldn't write any files to it, which also made this more difficult than it should have been.
And we built a cross-site scripting payload as well.
And so we spent a lot of time looking for other interesting tags that we could use to either achieve remote code execution
or somehow cause interesting things to happen here.
And there really weren't very many.
But there's one other one, and this is a little bit obscure.
So client body and file only.
And so what this will do is when Nginx receives a post payload, Nginx will save the post payload to a file on disk.
And the expectation is you then have some sidecar Python or Perl process
or Lua that would then pick it up and do something with that post payload and then delete the file.
So it's more of a functionality than it is like a security feature tag directive, sorry.
And so why does this matter?
So we can actually combine this with our, so if we could find a secret key that we could save,
we could combine this with our LFI payload to steal some secret key that's somewhere in a post payload.
So that's exactly what we did.
So if you remember this where we had these secondary files,
these secondary servers where we had connections going to chasm agent,
one of the requests that went through to chasm agent was a health check.
And this went through about every 30 seconds.
And in that health check post body, there was a secret key that could be used to communicate with that chasm agent.
So we asked, what if we get access to that secret key?
And so we tried to get it.
So putting everything together, ignoring what's on the right side for right now, I'll get to that in a second.
So we carefully intercepted a single chasm agent request
using that client body and file only directive.
We then forwarded along so that the system wouldn't think that the secondary server was unhealthy.
If it thought the secondary server was unhealthy, it would take it offline and break the whole attack chain.
We'll then use our LFI payload to steal that secret.
So breaking this out a little bit better, what we're actually doing here,
we're putting this into a host header using our ISO 8859-1 encoding.
So the first in the yellow, we're using that to keep off working for this workspace.
We're then, for the top two injection points, we're using,
oh, sorry, for the bottom four injection points, sorry,
we're putting this into a comment so that everything is commented off correctly.
That's in blue.
In white is really the magic here where we're intercepting the hull of the health check
and then we're stealing the secret that contains in there.
Then we're using regular expression location
to create an LFI backdoor using a .secret end.
So any path that ends in .secret will be interpreted by this regular expression.
And we're going to use that to steal the file.
And then we're going to put an unused location at the end of all this to close it off nicely.
So we were also hit by a 255-character limit,
which is why this is a little bit code golfed.
So I do apologize. That's not super clear.
So running this, as you can see here, you end up with a series of files.
And we get our host token in each one of these files, which is super exciting.
So what can you do with chasm agent at this point?
So with chasm agent, it actually has a well-documented API
where you can either spin up new containers
or you can make privileged calls to other containers to run commands.
So in this case, what we actually wanted to do was
we wanted to have chasm agent make a Docker API call to run a command on itself.
And so,
because it was a privileged container,
that would allow us to access the underlying operating system.
So using our host token,
we sent the request on the right side here to the chasm agent to run a command.
This command mounted the underlying operating system as root,
and we simply touched a file to demonstrate that we had access.
And as you can see here, we were able to achieve
file write as root, which would have led to RCE
if we had actually exploited this for real.
At this point,
we were asked to stop.
The response we got back from the bug bounty program was,
well, that's worrying.
And we got our crit, we got our RCE.
So we were very excited.
So no good slide deck is complete without a vendor response slide.
So nginx doesn't consider this to be a vulnerability,
but they might introduce some checks in the future
to prevent certain characters that are obviously invalid
for host headers from going through.
Cherry PY actually addresses for the host header.
So other headers are still vulnerable, however.
Chasm workspaces has a configuration fix for this.
And if you're running this, we would recommend you apply it.
Additionally, in the latest version, it's been fixed as well.
And in terms of a timeline here,
this is just to give you an idea of roughly what this looks like.
We submitted this on 3-1.
On 3-6, we reported the additional ways
that we could exploit the system.
It then took us another 22 days to demonstrate
the Docker container breakout.
So some takeaways.
So from the blue team side, you have to ask yourself
how your organization is handling running a server
that has multiple domain names.
In general, you'll have a QA on prod.
How do you handle the host header there?
Are you trusting the end users to supply that?
Because that could be a little bit dangerous
if you end up doing something with it.
You also have to be a little bit careful
about Docker networking.
I mean, that's true in any networking environment,
but that's especially true with Docker
and mixed low and high privilege environments.
And then on the red team side,
we know you're spraying payloads everywhere,
so why not just add ISO 8.59-1 to that list?
And then similarly, build gadgets
and then constantly reevaluate what you have
and what you can do.
What we presented here, we tried to follow a logical plot.
We jumped around quite a bit
after actually exploiting this.
And then similarly, collaborate.
I think there were multiple times
where we both wanted to throw in the towel
and completely give up on this.
And then one of us found something
that just pushed us that much further,
which is really why we worked on this for a month, I think,
which is overkill.
So that's it.
Questions?
Hey, so you found, like, the Cherry PY,
you were able to use that different encoding.
So was that also affecting Nginx
for the host-header encoding when you did the...
Could you say it one more time?
We can't...
Yep. Oh, sorry, Tess.
Can you hear me better?
Okay, so when you did the actual exploiters
on the Cherry Pi server,
did the actual, like, encoding trick bypass on Nginx as well?
Can you hear him at all?
Oh, sorry.
Slow down a little bit. Go ahead.
Oh, yeah.
So when you were testing, like, the actual proof of concept,
you were on the Cherry Pi, correct?
Yeah.
I think the...
Yeah, so...
Yeah, so when you actually exploited it
on the Cherry Pi server,
but the actual exploit was done on...
Like, is it...
You were able to do the encoding trick on Nginx as well?
Yeah.
Okay.
Sorry, I think the speaker's...
Yeah, no.
Sorry about the...
Having a hard time hearing.
Yeah, so the...
The question was, I think, you know,
when we actually did the exploit,
were we able to do it through Nginx as well?
And I think the...
Yeah, so we were.
And so the cool thing about the ISO 8859-1 encoding
is that all of those characters are valid characters
to be in a host header on Nginx.
So it just...
What ended up happening was
all of those characters that were causing the problem
in the Nginx configuration just were encoded
and then flew right through the Nginx part.
And then were decoded...
Decoded by CherryPY on the back end,
and that's what allowed us to bypass that little,
you know, mismatch between the parsing of the headers
on the two...
On the reverse proxy there, so...
Yeah.
I actually see that as well.
Yeah, go for it.
So within Nginx itself,
the configuration parsing logic
is heavily dependent on certain control characters
which were blocked by trying to traverse through Nginx.
So that's why we had to do this.
Yeah.
Any other questions?
All right.
Well, you can find us on Twitter afterwards
if you have any other questions.
Our handles are down at the bottom.
And thank you guys so much for listening.