Eric, we've got a few questions already in the queue.
I'll start off by asking you one here.
I watched your talk, it was great. Thanks for doing the work you did and releasing that tool.
It's going to be helpful for the attackers.
Myself, as a Blue Teamer, some of the things we do as Trix is to look for...
You talked about how to detect it on the wire, essentially in network traffic.
Have you looked at anything like on an endpoint?
Like if there's any artifacts or anything that would indicate that there's a problem?
Maybe not necessarily that the domain is bad or that it's being fronted, but anything that would show there's a problem.
Yeah, I think it's going to depend on the tooling that you use when you actually implement this technique.
If you're using a test client or something like the demo C2 that I had, you'll obviously have an EXE executing.
If you bake it into something else, maybe you inject.
If you're running out of SVC host and it's supposed to have network connections and you're fronting on something that might look OK,
that's probably going to be the best bet for the Red Teamers.
But yeah, I think the last slide of my detection piece was just do the good old-fashioned police work and look at the endpoints and use that EDR.
And you mentioned that a couple years ago that some of the major players were essentially removed this capability
and Azure is only the last standing big cloud provider that does this.
Is it possible for Cloudflare to decide the same thing and they say, hey, we're going to block this? Can they do anything about it?
Yeah, definitely. They could turn it off tomorrow if they wanted to.
I think the way that would work is once they decrypt the server name indication,
they could check to see if it was sent to the IP that they have registered for an A record for that domain.
And if it doesn't match, then they could drop the connection.
But I don't think they're going to do that.
Nick Sullivan, their head of research, has been aware of this technique ever since Robin Wood posted it last year.
So it's not super new. And I think Cloudflare is kind of a progressive, freedom-loving company.
And I think besides, if you're not running 8chan or DailyStorm or something crazy, you're probably pretty safe with Cloudflare.
So I think it's going to live on. And unless governments really turn the screws, I think this has some shelf life in it.
OK. So anything like when you said you specifically talked about, you know, how you can detect it on the network, what would that look like?
So if I'm using this technique, you showed how it's happening. Again, my team is or my perspective is the blue side.
How would I see this? You know, tell me, you know, give me some hints.
How would I detect this and make sure that it's not happening on my network? Is there any quick and dirty tricks?
Yeah, I think the best way is going to look for those packets that have both the encrypted server name and the regular SNI.
If you're making a TLS connection and you use both of those, it's not normal, it's not expected, and it's probably not anything good.
So I would drop any packet that's trying to do that and then investigate that endpoint for sure.
If they're only using an encrypted server name indication, then it's a choice to either block all of it or allow all of it.
So that's where it gets tough. And there's no easy button in any product right now to enable you to do that.
And you mentioned custom rules. You could create your own custom rules, obviously.
Yeah, I think in Snort and Secure Kata, you're going to have to get into the content parser and really pick out those TLS extensions themselves.
I'm not a blue teamer. I haven't dug into that super in depth, but I suspect we'll see some rules pretty soon for this kind of stuff.
Maybe tomorrow.
Maybe.
Is there any other technologies besides WebSockets and HTTP that could be used to perform the same type of attack?
Yeah, definitely. So this works right now at the TLS layer. So anything that you can wrap in TLS would also work with this technique.
So I showed HTTP and I showed WebSockets working, but there's no reason you couldn't wrap arbitrary protocols in that.
And that's kind of what Cloak is doing, although it uses WebSockets to do that.
One thing you could also do is investigate QUIC, which is a UDP based protocol.
Cloudflare supports that, and I have a suspicion that this would also work using QUIC.
OK. Somebody's mentioned from the chat, you know, why if the ESNI is the problem, the traffic, they're asking why not just block it all or decrypt it all, I guess.
Blocking it all obviously would have some impacts, but speak to both, I guess.
Yeah, you could block it all. That is an option. As adoption grows, I think that's going to be a more difficult option.
Right now, you might be able to get away with it since it's only a smaller percentage of the traffic.
TLS 1.3 itself is 25 to 50 percent, I would guess, depending on the network.
With ESNI, it's still a draft, so it's probably much lower. It's in Firefox and things like that.
But you could probably get away with blocking all ESNI traffic right now and your users wouldn't yell at you too much.
As far as decrypting goes, I don't think that's really an option.
You're going to have to get in between the DNS and then put your own keys in there.
Maybe a vendor will come out with a solution to do that, but right now it's going to be messy for sure.
Probably a function of how big of a server you have to run, how much volume of data that's coming through.
You mentioned there was a little comment in there, in parentheses, the AI and ML reference.
Do you care to speak to that? Is there any sort of statistical analysis, machine learning that could be used to detect this stuff?
Yeah, I think there's definitely space in there for solutions that do their machine learning and their anomaly detection especially.
Like I said, if there's a computer, maybe an accounting or something, that all of a sudden every 30 minutes it reaches out to,
it doesn't really matter what domain it is, but all of a sudden there's a new pattern of behavior from that machine, that should be suspect.
And if you have a good solution, maybe you get an alert on that.
Okay. All right. Have you heard any response from, you know, any, it sounds like you've been talking to some, most of these places, people, businesses, and providers.
Have they mentioned that there's anything coming up or other than, you know, just expect expectation that once they see this on the, on the YouTube and Twitch, that it'll spark them to create some new rules?
Yeah, I haven't, I haven't really communicated with the firewall vendors much, but I suspect that as adoption grows and as this becomes more of a problem,
probably once the first major ransomware uses something similar to this, people will start taking notice.
But yeah, nothing, nothing right now. I don't have any insight on what's coming down the pipe, but I'd look out for it for sure.
Okay. So let's see.
I was especially interested in your problem you were seeing with Palo Alto and the mismatch on support for 1.3. Did that get resolved after you recorded this?
Yeah, so I don't think that's, it's not really an issue. It's just the default decryption profile that they have in place. If you just click on like, do HTTPS decryption for me, it just doesn't include 1.3 by default.
But it's, if you're setting that up, you're probably going to want to fine tune that. So all you do is you check the extra box and then it works.
But I was a little bit surprised when I tried to just do the easy button solution and turns out it just let everything through. It did log an error for every connection.
So your logs would fill up pretty quickly and you should notice something, but a little strange that it's not included in the default profile.
So what if I'm a, I'm a pen tester. I get paid for success. And I have my own little tool suite that I'm using. How would I utilize this technique? Like how is there, what did you have to go through to tailor your tool to work with?
Yeah, I think there are two options right now for what you can do. One is you can use the kind of cloak proxy shadow socks method that I demonstrated with Cobalt Strike.
So if your tool is proxy aware, you just bundle it with those two. It's going to be a little bigger and make a little more noise, but you can use this technique today with that.
If you want to integrate it, if you're using Go for your language that you write your tool set in, then you can, there are instructions on the GitHub on how to integrate it. It's not too bad.
Just a couple of changes. If it's another language, you're going to have to dig into the TLS library itself. There are two requirements. One, you've got to support the ESNI or the draft ESNI extension.
And then two, you've got to kind of adjust that to do things that maybe it's not designed to do by spec, like maintain the ESNI or maintain the regular SNI if you're using ESNI, if you so want to do that kind of decoy SNI stuff.
So digging into the actual TLS library of your language of choice might be required if you want to integrate it smoothly into your tool.
And asking for the red teamers on the call, because I'm sure they want to know if they're trying to be secretive and not use, you know, even though it's you're hiding in the in the noise by using Cloudflare.
What if I wanted to find something a little more rare, or make it so that it's less likely to get detected since, you know, most of the early rules are going to be based on essentially the examples that you had.
How do I check to see if there's what other tool or what other domain fronting options there are?
Yeah, I didn't actually show this in the talk, but there is a tool inside the Git repo that I call find fronts.
And you can feed it a CSV of domain names and it will spit out which ones are available to use with this technique.
But if you want to do it yourself, it's super easy.
All it's doing basically is curling the domain and then looking at the response headers.
And anything that comes back with a server name of Cloudflare has a set cookie with a, you know, underscore underscore CF, which is a Cloudflare value, or has an expect CT header that includes the Cloudflare domain pretty much indicates that that site is behind a Cloudflare worker or some other Cloudflare service, and it's available to be used with this technique.
So everything that's returned that so far has been great for this technique. So I'd do that.
Okay. And you used Cloudflare as the example. Have you tried it with other services like Azure?
I haven't worked on it with other services, mostly because most of them don't support the draft ESNI standard.
So you need a CDN that supports this standard and then kind of also allows arbitrary IPs to be used for arbitrary domains that are hosted on it.
Okay.
And another thing is that the bigger the better, right? So you want it to be painful to block and Cloudflare being the biggest CDN, they supported all the things and they were the biggest. So kind of a one stop shop with that one.
Yeah. And does your tool then, would that, you mentioned there's a couple of caveats there. Would your tool then work through that? You know, knowing that all those other conditions exist and tell you this is a legitimate website you could use?
No, the FindFronts isn't going to do that for you. The FindFronts is just looking for specifically Cloudflare protected domains.
So if you find another CDN that works with this technique, you're going to have to come up with your own method to detect which sites are available to front with those.
Okay. Thanks for clarifying.
Let's see, I got another request or question here. Let's see, the Tor project uses domain fronting. Are you aware of this being, of your tech, I think it's your technique being adopted or are they looking into it?
Yeah, I don't know that the Tor project is aware that this kind of branch of domain fronting is available yet. Hopefully this will spark their interest.
They use the Meek project to do domain fronting and they're kind of reliant on Azure right now since it's the only major CDN that still allows it.
But I would love to see them roll out with this technology and same with Signal. I think they do some domain fronting in restricted countries as well.
So besides being a red team tool, I'd also love to see it be used as a censorship bypass to give people free and open access to the internet.
Yeah, can you speak a little bit more about that? We have another another request that's, you know, you walked right into it. That's exactly what they want to know.
How would this be leveraged to essentially become a communication, you know, a tool for communication? I say secretly, but you know, to bypass censorship?
Yeah, I think you'd probably set up a system similar to one of the demos that I had where you have Cloak or the Cloak fork in the Noctilucent project, a Shadowsocks server.
You put that out on a VPS or a company sponsors a large instance, and then you run a Cloak client locally with a Shadowsocks client locally.
And then you have a Socks proxy running on your host that tunnels arbitrary TCP or UDP traffic out to the open internet and you can bypass any great wall or other restrictive government that's trying to block specific sites or really anything.
All your traffic is going to appear to come from that VPS. So it's like you're browsing from that VPS.
Okay.
All right, it looks like I don't see any new...
There was one other question that said, apart from the fact that using domain fronting must also host the site on the same domain, what are the other downsides to this? Are there any solutions for detecting and blocking it?
Yeah, so original domain fronting that was popular back in 2018, it required you to not only have your domain on the same service, but you'd have to run a service or a VPS on that provider.
So if you were using Azure, you had to have an Azure VPS running on that same thing.
If you were using Google Cloud, you'd have to have a Google app worker or something like that running on the same service.
And all those required full name to sign up, credit cards, phone numbers. It was a little burdensome, plus expensive if you had a lot of traffic.
I think the Tor project has some stats on how much money they were putting into domain fronting, but it wasn't trivial. It was an amount.
With Cloudflare, all you need to do is you sign up for a free account, which just requires a name and an email, and they will take disposable emails.
And then you point your domain, registrar, name servers to Cloudflare, and then you define your IPs in the Cloudflare web dashboard, and that's it.
Those IPs can point to anywhere you want. It can be DigitalOcean, Amazon, Google Cloud, it doesn't matter where they're hosted, as long as your DNS is run from Cloudflare.
I thought your example of the mail forwarding with the postcard in the envelope was a good analogy to help with the C-suite understand what the threat is and how it works.
Essentially, you accept the letter and blindly open it and hand it off to whoever it needs to go to without inspecting it.
Exactly. Yep.
Oh, so you did watch the show, huh?
I did. I was just joking, but I didn't watch it.
Let's see. I had another question. Oh, the speed. I found that very interesting that the speed was so high.
I was not expecting that. Do you think that is a function of Cloudflare's capabilities, or is it just a function of the protocol?
Because it's used on the internet, there's a focus on making sure the throughput is high.
Because you wouldn't want to slow people down to get their content.
And most of the time it's being used for video or those types of things anyway.
Yeah, I think it's a function of kind of every piece of the chain.
Web sockets are great for bidirectional communication with low overhead.
And then Cloudflare is just really good at delivering content.
I mean, that's their bread and butter. So even when you're pushing 100 megs per second fronted through them, they handle it no problem.
So yeah, everything from the web sockets to Cloudflare's infrastructure.
I wonder what would happen if I paid for the higher tier AWS. Could I get even more?
I don't know. Maybe that's a future research project is how fast can you do this?
But 100 megs a second is pretty solid for everything I was doing.
Yeah, yeah. Yeah, I was surprised at that when you showed that.
So any other ideas for future research? Are you, you know, is there any other techniques or tricks?
I saw the enhancements with the cloak was cool.
You know, as a blue teamer, we're always looking for this type of stuff, both, you know, we realize that there's techniques that could be, we won't see them.
And so we have to combine network analysis with user behavior with, you know, anomalies that are detected on the endpoint.
You know, like we were before we were on this, the chat, we're talking about, you know, somebody in finance running PowerShell.
And, you know, maybe the PowerShell is running in the middle of the night, which is unexpected, those types of detection capabilities.
And, you know, to me, what worries me is as when I was a consultant doing, you know, the pen testing stuff, I felt that a lot of the businesses were working with the small to medium sized businesses, which are, you know, 90% of the businesses in the country, or in the world, actually.
There's not a lot of, you know, not a lot of hope for them in detecting this level of stuff, because they just don't have the resources.
You know, it's difficult for them to, one, understand it, you did a great job of providing a way to understand, generally, what is happening.
But, you know, if the tools, if you go to a vendor and the vendors tools don't, can't do it, you know, out of the box or very simply, it's, you know, it's a daunting task for them.
Yeah, I think in this case, it might be more of a, you don't have to outrun the bear, you just have to outrun your friends situation where this technique is probably only going to be used by the most advanced actors or red team that you hire.
So, you know, first step one is detect that the fact that PowerShell is running on the accountant box in the middle of the night and then start worrying about the more advanced stuff.
But when you're ready for the more advanced stuff, it's there for sure.
Somebody's scratching at the door, so you may want to check.
Yeah.
Somebody made a comment about Cloudflare has something called Bandwidth Alliance, where you aren't charged for your VPS.
I don't know, it's just a comment, I guess.
Is there anybody, any reason why you chose Cloak over some other pluggable transports like V2Ray?
Yeah, there are a bunch of good projects out there. Chisel was another one that I looked at, which is also written in Go.
I liked Cloak because it seemed to have the widest variety of potential secondary services that you can kind of glue on to it.
I use Shadowsocks in my example, but the fact that you can tunnel UDP traffic through it, you could probably get WireGuard VPN running fronted through Cloudflare, which is just kind of crazy to think about, and at decent speeds as well.
I went with Cloak because it seemed to be the most censorship unfriendly, because of the fingerprinting, although I think I kind of broke that when I just used the standard Go TLS library, and then the widest variety of things you can use it with.
Yeah, I think that's, to me, that's what the biggest, any of these types of talks where you're kind of pushing the envelope and coming up with something new and building, you know, a concept is, is the where do we go from here?
As soon as somebody shows, hey, look what I found over here. And then there's this explosion that happens in the next six to nine months where development kicks off and then somebody will do a presentation next year, based on what you presented and kind of taking it to the next level.
Yeah, I hope to see, you know, C sharp TLS libraries that use this and C libraries and, you know, everybody kind of modifying their low level TLS libraries to enable these options. That'd be really cool.
Do you have any other, somebody's asking about research, future research, any avenues you think would be fruitful?
Yeah, I'd look into QUIC for sure. Q-U-I-C. It's the UDP based protocol that a lot of providers are, even Microsoft is now pushing QUIC and Cloudflare, of course, supports it. And I would be surprised if you couldn't do the same technique using QUIC.
So you wouldn't, you wouldn't be using TLS and TCP, but you'd be using UDP, using that quicker protocol, potentially higher speed, and potentially harder to detect if you're, if you're allowing UDP out to port 53. Maybe you can, you know, slip some traffic through that way.
Okay. All right, well, we're getting close to the end here. Got about seven, eight minutes. Do you have any other comments or ideas for people to...
Yeah, for sure.
Anything, anything you want to let us know? Any new tools or updates you want to improve on? Or know...
I'd be...
Go ahead. Sorry.
I'd be lying if I, if I didn't have a, if I told you I didn't have a DC 29 folder on my computer already with a, with a notes document in there thinking about, about the next talk and how to take this the next step.
But yeah, I think people should just keep up with the latest news, either on Twitter or Reddit, Netsack, or anything like that. When you see a little nugget, like I saw Robin Wood was the one who originally kind of showed that this was even possible.
And you can kind of run with that and productize it and take it to the next level and then explain how it works. And, and that's good enough for a talk.
So yeah, definitely stay, stay with your ear to the ground and keep looking for what's going to be the next thing. And who knows, you know, I was, I was at DEF CON, I think it was like 23 was my first DEF CON and watching speakers talk.
And I thought that, man, that's crazy. These guys are, you know, next level. And then you keep working at it for a couple of years and you find your little niche, you know, you don't have to, you don't have to understand everything. Just expand the universe a little bit. And here you are.
So, so speaking to that, let's say I'm, this is my first DEF CON, you know, I, for whatever reason, I couldn't afford it or couldn't go. And now, hey, I find myself quarantined at home and I'm watching you.
And I'm, and this is something I find interesting and is, I know you posted your contact information on the end of your presentation. Is it okay for people to call you and, or maybe not call you, but, you know, get ahold of you, ping you and say, you know, hey, I got this idea or can you explain something or is there any place they can go to find more basic information, you know, a little more of the one-on-one? Is there any, do you know of any known or, you know, common sites we can push them to?
Yeah, sure. I mean, you're always available to hit me up at Bad Sector Labs is my personal Twitter handle and 6gen.io if you need commercial support. But for basic stuff, I would, I would look at the kind of the introduction CTF, intro level CTFs, like over the wire is really great. And those, those will kind of walk you through kind of the basics of how do you get into this mindset of, you know, there's something that's supposed to work a certain way.
How can I get it to work a different way? Or how can I do things to it that the author didn't intend? And that kind of mindset is just, if you apply that to everything that you see that comes out with cybersecurity and even just the latest technologies, yeah, I think you'll go far. And honestly, reading the RFCs, as boring as it is, you know, that's, that's where a lot of gold is. So stay up to date on the latest RFCs, dig in, look at implementations. Are people doing it correctly? Probably not.
Break their implementations. And there you go.
Yeah, I think the RFCs is a key piece, because it may be not a new feature. It may be an old feature that people forgot. And nobody's looked, you know, under the covers to know, hey, look at this feature, I can use this. You also mentioned WireGuard. I think that's another, you know, cool project that's come about the last few years. Can you speak to that? You're using that in this kind of attack?
Yeah, I didn't, I didn't test it out with this tool. But I love WireGuard. And I think it's going to be the future of VPN, super fast, super easy to set up small code base auditable props to the author on that. That's, that's an amazing tool. But yeah, if you can tunnel UDP traffic, then you can probably put WireGuard behind this kind of fronting or hiding. And therefore, the sky's the limit, you know, once once you get out to the open internet, and then you can run your VPN through there.
Censorship is kind of, they're playing catch up for sure.
Yeah, I mean, to me, this is, you know, from a hacker's perspective, I think a good application of this would be tunneling out through a censored network and getting access to Netflix so I could watch, you know, the latest, the latest, whatever hot lava show or whatever.
And that's one of the last questions here is, so we'd mentioned if, I think we've talked about this a little bit, but they're asking what if government made the demanded that domain fronting illegal?
What would break? I guess, you know, obviously making it illegal is one thing and then you have to have the capability of stopping it because you can say it's illegal.
If people continue to do it, you know, but I'll let you speak to it.
Yeah, I think I think we saw this happen in 2018 when the Russian government turned the screws on on AWS and Google and they they eventually capitulated.
So it's a possibility that, you know, enough governments with enough power, turn the screws on Cloudflare that they say it's not worth fighting this and we're gonna, you know, succumb to the demands and they're not a government so they can't really fight that fight.
But I think that they're going to try their hardest to to maintain the ability for any of their IP range to go to any of their domains.
I think it's it's one of their business advantages, honestly. So turning that off might be detrimental to even just the basic operating of Cloudflare.
So while it's possible, I think they're going to fight it as hard as they can. But if the government makes you do it, I mean, we've seen it time and time again.
Eventually, I think these companies will succumb to what the government forces. This isn't like a Bitcoin or a magic blockchain solution to anything where, you know, it's going to survive no matter what happens. But we'll see.
Yeah, yeah. And things continue to evolve. So yeah, we're, they might, this might get fixed by the industry. And then a new technique comes up, you know, so. So we're about out of time. Thank you for speaking here at DEF CON. And it was a great talk. And I appreciate all the questions you had.
Yeah, thanks, guys. And thanks to all the goons who made this possible. I think a lot of people tuning in don't realize that it was a major effort to get everything coordinated and work. So thanks to you guys.
Thank you.
All right.