start	end	text
4640	8380	Eric, we've got a few questions already in the queue.
9580	12420	I'll start off by asking you one here.
13060	17900	I watched your talk, it was great. Thanks for doing the work you did and releasing that tool.
17900	20700	It's going to be helpful for the attackers.
21540	28000	Myself, as a Blue Teamer, some of the things we do as Trix is to look for...
28000	33260	You talked about how to detect it on the wire, essentially in network traffic.
34460	39400	Have you looked at anything like on an endpoint?
39400	42840	Like if there's any artifacts or anything that would indicate that there's a problem?
42840	49400	Maybe not necessarily that the domain is bad or that it's being fronted, but anything that would show there's a problem.
50040	55020	Yeah, I think it's going to depend on the tooling that you use when you actually implement this technique.
55020	62660	If you're using a test client or something like the demo C2 that I had, you'll obviously have an EXE executing.
63000	65920	If you bake it into something else, maybe you inject.
66800	74600	If you're running out of SVC host and it's supposed to have network connections and you're fronting on something that might look OK,
74600	77640	that's probably going to be the best bet for the Red Teamers.
77640	88980	But yeah, I think the last slide of my detection piece was just do the good old-fashioned police work and look at the endpoints and use that EDR.
89240	98620	And you mentioned that a couple years ago that some of the major players were essentially removed this capability
98620	103880	and Azure is only the last standing big cloud provider that does this.
104480	112360	Is it possible for Cloudflare to decide the same thing and they say, hey, we're going to block this? Can they do anything about it?
112640	116060	Yeah, definitely. They could turn it off tomorrow if they wanted to.
116120	122060	I think the way that would work is once they decrypt the server name indication,
122060	128540	they could check to see if it was sent to the IP that they have registered for an A record for that domain.
128540	131140	And if it doesn't match, then they could drop the connection.
131420	133200	But I don't think they're going to do that.
133200	139440	Nick Sullivan, their head of research, has been aware of this technique ever since Robin Wood posted it last year.
139480	147060	So it's not super new. And I think Cloudflare is kind of a progressive, freedom-loving company.
147060	154560	And I think besides, if you're not running 8chan or DailyStorm or something crazy, you're probably pretty safe with Cloudflare.
154560	160960	So I think it's going to live on. And unless governments really turn the screws, I think this has some shelf life in it.
160960	171760	OK. So anything like when you said you specifically talked about, you know, how you can detect it on the network, what would that look like?
171760	177840	So if I'm using this technique, you showed how it's happening. Again, my team is or my perspective is the blue side.
178100	181980	How would I see this? You know, tell me, you know, give me some hints.
181980	188940	How would I detect this and make sure that it's not happening on my network? Is there any quick and dirty tricks?
188940	196600	Yeah, I think the best way is going to look for those packets that have both the encrypted server name and the regular SNI.
196600	205320	If you're making a TLS connection and you use both of those, it's not normal, it's not expected, and it's probably not anything good.
205320	210080	So I would drop any packet that's trying to do that and then investigate that endpoint for sure.
210240	218780	If they're only using an encrypted server name indication, then it's a choice to either block all of it or allow all of it.
218780	226220	So that's where it gets tough. And there's no easy button in any product right now to enable you to do that.
226360	230160	And you mentioned custom rules. You could create your own custom rules, obviously.
230600	237780	Yeah, I think in Snort and Secure Kata, you're going to have to get into the content parser and really pick out those TLS extensions themselves.
237980	244520	I'm not a blue teamer. I haven't dug into that super in depth, but I suspect we'll see some rules pretty soon for this kind of stuff.
245500	246720	Maybe tomorrow.
246720	248280	Maybe.
251320	260820	Is there any other technologies besides WebSockets and HTTP that could be used to perform the same type of attack?
260940	268960	Yeah, definitely. So this works right now at the TLS layer. So anything that you can wrap in TLS would also work with this technique.
268960	276140	So I showed HTTP and I showed WebSockets working, but there's no reason you couldn't wrap arbitrary protocols in that.
276140	280600	And that's kind of what Cloak is doing, although it uses WebSockets to do that.
280700	285660	One thing you could also do is investigate QUIC, which is a UDP based protocol.
286480	291480	Cloudflare supports that, and I have a suspicion that this would also work using QUIC.
292260	308640	OK. Somebody's mentioned from the chat, you know, why if the ESNI is the problem, the traffic, they're asking why not just block it all or decrypt it all, I guess.
308640	312300	Blocking it all obviously would have some impacts, but speak to both, I guess.
312640	320540	Yeah, you could block it all. That is an option. As adoption grows, I think that's going to be a more difficult option.
320540	325980	Right now, you might be able to get away with it since it's only a smaller percentage of the traffic.
326040	331400	TLS 1.3 itself is 25 to 50 percent, I would guess, depending on the network.
331980	337120	With ESNI, it's still a draft, so it's probably much lower. It's in Firefox and things like that.
337120	342580	But you could probably get away with blocking all ESNI traffic right now and your users wouldn't yell at you too much.
343300	348380	As far as decrypting goes, I don't think that's really an option.
348380	353760	You're going to have to get in between the DNS and then put your own keys in there.
354640	359900	Maybe a vendor will come out with a solution to do that, but right now it's going to be messy for sure.
360080	366740	Probably a function of how big of a server you have to run, how much volume of data that's coming through.
369840	377360	You mentioned there was a little comment in there, in parentheses, the AI and ML reference.
378000	386940	Do you care to speak to that? Is there any sort of statistical analysis, machine learning that could be used to detect this stuff?
386940	394020	Yeah, I think there's definitely space in there for solutions that do their machine learning and their anomaly detection especially.
394020	401340	Like I said, if there's a computer, maybe an accounting or something, that all of a sudden every 30 minutes it reaches out to,
401340	407720	it doesn't really matter what domain it is, but all of a sudden there's a new pattern of behavior from that machine, that should be suspect.
407720	411020	And if you have a good solution, maybe you get an alert on that.
411020	424980	Okay. All right. Have you heard any response from, you know, any, it sounds like you've been talking to some, most of these places, people, businesses, and providers.
424980	440120	Have they mentioned that there's anything coming up or other than, you know, just expect expectation that once they see this on the, on the YouTube and Twitch, that it'll spark them to create some new rules?
440120	450260	Yeah, I haven't, I haven't really communicated with the firewall vendors much, but I suspect that as adoption grows and as this becomes more of a problem,
450260	455900	probably once the first major ransomware uses something similar to this, people will start taking notice.
456200	462700	But yeah, nothing, nothing right now. I don't have any insight on what's coming down the pipe, but I'd look out for it for sure.
462700	464460	Okay. So let's see.
464460	476780	I was especially interested in your problem you were seeing with Palo Alto and the mismatch on support for 1.3. Did that get resolved after you recorded this?
477300	489420	Yeah, so I don't think that's, it's not really an issue. It's just the default decryption profile that they have in place. If you just click on like, do HTTPS decryption for me, it just doesn't include 1.3 by default.
489420	495900	But it's, if you're setting that up, you're probably going to want to fine tune that. So all you do is you check the extra box and then it works.
495900	503920	But I was a little bit surprised when I tried to just do the easy button solution and turns out it just let everything through. It did log an error for every connection.
503920	510900	So your logs would fill up pretty quickly and you should notice something, but a little strange that it's not included in the default profile.
511540	527600	So what if I'm a, I'm a pen tester. I get paid for success. And I have my own little tool suite that I'm using. How would I utilize this technique? Like how is there, what did you have to go through to tailor your tool to work with?
528020	538220	Yeah, I think there are two options right now for what you can do. One is you can use the kind of cloak proxy shadow socks method that I demonstrated with Cobalt Strike.
538220	547380	So if your tool is proxy aware, you just bundle it with those two. It's going to be a little bigger and make a little more noise, but you can use this technique today with that.
547500	556440	If you want to integrate it, if you're using Go for your language that you write your tool set in, then you can, there are instructions on the GitHub on how to integrate it. It's not too bad.
556440	568600	Just a couple of changes. If it's another language, you're going to have to dig into the TLS library itself. There are two requirements. One, you've got to support the ESNI or the draft ESNI extension.
568860	585020	And then two, you've got to kind of adjust that to do things that maybe it's not designed to do by spec, like maintain the ESNI or maintain the regular SNI if you're using ESNI, if you so want to do that kind of decoy SNI stuff.
585020	593080	So digging into the actual TLS library of your language of choice might be required if you want to integrate it smoothly into your tool.
593080	609260	And asking for the red teamers on the call, because I'm sure they want to know if they're trying to be secretive and not use, you know, even though it's you're hiding in the in the noise by using Cloudflare.
609260	621760	What if I wanted to find something a little more rare, or make it so that it's less likely to get detected since, you know, most of the early rules are going to be based on essentially the examples that you had.
621760	628320	How do I check to see if there's what other tool or what other domain fronting options there are?
628700	635200	Yeah, I didn't actually show this in the talk, but there is a tool inside the Git repo that I call find fronts.
635220	641300	And you can feed it a CSV of domain names and it will spit out which ones are available to use with this technique.
641300	643520	But if you want to do it yourself, it's super easy.
643520	647740	All it's doing basically is curling the domain and then looking at the response headers.
647820	670880	And anything that comes back with a server name of Cloudflare has a set cookie with a, you know, underscore underscore CF, which is a Cloudflare value, or has an expect CT header that includes the Cloudflare domain pretty much indicates that that site is behind a Cloudflare worker or some other Cloudflare service, and it's available to be used with this technique.
670880	677520	So everything that's returned that so far has been great for this technique. So I'd do that.
677600	685460	Okay. And you used Cloudflare as the example. Have you tried it with other services like Azure?
685580	692760	I haven't worked on it with other services, mostly because most of them don't support the draft ESNI standard.
692760	701900	So you need a CDN that supports this standard and then kind of also allows arbitrary IPs to be used for arbitrary domains that are hosted on it.
701960	702060	Okay.
702060	712940	And another thing is that the bigger the better, right? So you want it to be painful to block and Cloudflare being the biggest CDN, they supported all the things and they were the biggest. So kind of a one stop shop with that one.
712940	728060	Yeah. And does your tool then, would that, you mentioned there's a couple of caveats there. Would your tool then work through that? You know, knowing that all those other conditions exist and tell you this is a legitimate website you could use?
729060	737120	No, the FindFronts isn't going to do that for you. The FindFronts is just looking for specifically Cloudflare protected domains.
737120	744600	So if you find another CDN that works with this technique, you're going to have to come up with your own method to detect which sites are available to front with those.
744700	746000	Okay. Thanks for clarifying.
747380	761740	Let's see, I got another request or question here. Let's see, the Tor project uses domain fronting. Are you aware of this being, of your tech, I think it's your technique being adopted or are they looking into it?
762520	771720	Yeah, I don't know that the Tor project is aware that this kind of branch of domain fronting is available yet. Hopefully this will spark their interest.
771720	779220	They use the Meek project to do domain fronting and they're kind of reliant on Azure right now since it's the only major CDN that still allows it.
779240	787500	But I would love to see them roll out with this technology and same with Signal. I think they do some domain fronting in restricted countries as well.
787500	794540	So besides being a red team tool, I'd also love to see it be used as a censorship bypass to give people free and open access to the internet.
794540	801540	Yeah, can you speak a little bit more about that? We have another another request that's, you know, you walked right into it. That's exactly what they want to know.
801540	813000	How would this be leveraged to essentially become a communication, you know, a tool for communication? I say secretly, but you know, to bypass censorship?
813000	824460	Yeah, I think you'd probably set up a system similar to one of the demos that I had where you have Cloak or the Cloak fork in the Noctilucent project, a Shadowsocks server.
824460	834000	You put that out on a VPS or a company sponsors a large instance, and then you run a Cloak client locally with a Shadowsocks client locally.
834000	851320	And then you have a Socks proxy running on your host that tunnels arbitrary TCP or UDP traffic out to the open internet and you can bypass any great wall or other restrictive government that's trying to block specific sites or really anything.
851320	857080	All your traffic is going to appear to come from that VPS. So it's like you're browsing from that VPS.
858780	859740	Okay.
861700	870360	All right, it looks like I don't see any new...
870360	882800	There was one other question that said, apart from the fact that using domain fronting must also host the site on the same domain, what are the other downsides to this? Are there any solutions for detecting and blocking it?
883560	897680	Yeah, so original domain fronting that was popular back in 2018, it required you to not only have your domain on the same service, but you'd have to run a service or a VPS on that provider.
897680	903460	So if you were using Azure, you had to have an Azure VPS running on that same thing.
903460	909280	If you were using Google Cloud, you'd have to have a Google app worker or something like that running on the same service.
909280	918660	And all those required full name to sign up, credit cards, phone numbers. It was a little burdensome, plus expensive if you had a lot of traffic.
919220	926320	I think the Tor project has some stats on how much money they were putting into domain fronting, but it wasn't trivial. It was an amount.
926320	935160	With Cloudflare, all you need to do is you sign up for a free account, which just requires a name and an email, and they will take disposable emails.
935160	945280	And then you point your domain, registrar, name servers to Cloudflare, and then you define your IPs in the Cloudflare web dashboard, and that's it.
945280	956540	Those IPs can point to anywhere you want. It can be DigitalOcean, Amazon, Google Cloud, it doesn't matter where they're hosted, as long as your DNS is run from Cloudflare.
957300	969440	I thought your example of the mail forwarding with the postcard in the envelope was a good analogy to help with the C-suite understand what the threat is and how it works.
969440	977180	Essentially, you accept the letter and blindly open it and hand it off to whoever it needs to go to without inspecting it.
977560	979020	Exactly. Yep.
979280	980800	Oh, so you did watch the show, huh?
980800	985200	I did. I was just joking, but I didn't watch it.
990950	1000090	Let's see. I had another question. Oh, the speed. I found that very interesting that the speed was so high.
1000090	1009750	I was not expecting that. Do you think that is a function of Cloudflare's capabilities, or is it just a function of the protocol?
1009750	1017230	Because it's used on the internet, there's a focus on making sure the throughput is high.
1017230	1020030	Because you wouldn't want to slow people down to get their content.
1020350	1024090	And most of the time it's being used for video or those types of things anyway.
1024690	1027530	Yeah, I think it's a function of kind of every piece of the chain.
1027650	1032890	Web sockets are great for bidirectional communication with low overhead.
1032950	1036790	And then Cloudflare is just really good at delivering content.
1036790	1044910	I mean, that's their bread and butter. So even when you're pushing 100 megs per second fronted through them, they handle it no problem.
1044910	1048530	So yeah, everything from the web sockets to Cloudflare's infrastructure.
1048950	1053790	I wonder what would happen if I paid for the higher tier AWS. Could I get even more?
1053790	1057670	I don't know. Maybe that's a future research project is how fast can you do this?
1057670	1061450	But 100 megs a second is pretty solid for everything I was doing.
1062290	1068610	Yeah, yeah. Yeah, I was surprised at that when you showed that.
1070650	1079530	So any other ideas for future research? Are you, you know, is there any other techniques or tricks?
1079530	1086090	I saw the enhancements with the cloak was cool.
1087890	1098330	You know, as a blue teamer, we're always looking for this type of stuff, both, you know, we realize that there's techniques that could be, we won't see them.
1098330	1109450	And so we have to combine network analysis with user behavior with, you know, anomalies that are detected on the endpoint.
1109450	1117170	You know, like we were before we were on this, the chat, we're talking about, you know, somebody in finance running PowerShell.
1117170	1124710	And, you know, maybe the PowerShell is running in the middle of the night, which is unexpected, those types of detection capabilities.
1124710	1139990	And, you know, to me, what worries me is as when I was a consultant doing, you know, the pen testing stuff, I felt that a lot of the businesses were working with the small to medium sized businesses, which are, you know, 90% of the businesses in the country, or in the world, actually.
1141270	1150910	There's not a lot of, you know, not a lot of hope for them in detecting this level of stuff, because they just don't have the resources.
1150910	1160570	You know, it's difficult for them to, one, understand it, you did a great job of providing a way to understand, generally, what is happening.
1161790	1174570	But, you know, if the tools, if you go to a vendor and the vendors tools don't, can't do it, you know, out of the box or very simply, it's, you know, it's a daunting task for them.
1175950	1189010	Yeah, I think in this case, it might be more of a, you don't have to outrun the bear, you just have to outrun your friends situation where this technique is probably only going to be used by the most advanced actors or red team that you hire.
1189130	1197910	So, you know, first step one is detect that the fact that PowerShell is running on the accountant box in the middle of the night and then start worrying about the more advanced stuff.
1197910	1201690	But when you're ready for the more advanced stuff, it's there for sure.
1201690	1206110	Somebody's scratching at the door, so you may want to check.
1206430	1206990	Yeah.
1209570	1218950	Somebody made a comment about Cloudflare has something called Bandwidth Alliance, where you aren't charged for your VPS.
1219510	1221310	I don't know, it's just a comment, I guess.
1222390	1228730	Is there anybody, any reason why you chose Cloak over some other pluggable transports like V2Ray?
1228730	1235510	Yeah, there are a bunch of good projects out there. Chisel was another one that I looked at, which is also written in Go.
1235610	1245870	I liked Cloak because it seemed to have the widest variety of potential secondary services that you can kind of glue on to it.
1245870	1261450	I use Shadowsocks in my example, but the fact that you can tunnel UDP traffic through it, you could probably get WireGuard VPN running fronted through Cloudflare, which is just kind of crazy to think about, and at decent speeds as well.
1262150	1277690	I went with Cloak because it seemed to be the most censorship unfriendly, because of the fingerprinting, although I think I kind of broke that when I just used the standard Go TLS library, and then the widest variety of things you can use it with.
1277690	1291990	Yeah, I think that's, to me, that's what the biggest, any of these types of talks where you're kind of pushing the envelope and coming up with something new and building, you know, a concept is, is the where do we go from here?
1291990	1312170	As soon as somebody shows, hey, look what I found over here. And then there's this explosion that happens in the next six to nine months where development kicks off and then somebody will do a presentation next year, based on what you presented and kind of taking it to the next level.
1312170	1326410	Yeah, I hope to see, you know, C sharp TLS libraries that use this and C libraries and, you know, everybody kind of modifying their low level TLS libraries to enable these options. That'd be really cool.
1326910	1334670	Do you have any other, somebody's asking about research, future research, any avenues you think would be fruitful?
1334670	1351230	Yeah, I'd look into QUIC for sure. Q-U-I-C. It's the UDP based protocol that a lot of providers are, even Microsoft is now pushing QUIC and Cloudflare, of course, supports it. And I would be surprised if you couldn't do the same technique using QUIC.
1351230	1370010	So you wouldn't, you wouldn't be using TLS and TCP, but you'd be using UDP, using that quicker protocol, potentially higher speed, and potentially harder to detect if you're, if you're allowing UDP out to port 53. Maybe you can, you know, slip some traffic through that way.
1371150	1385010	Okay. All right, well, we're getting close to the end here. Got about seven, eight minutes. Do you have any other comments or ideas for people to...
1386410	1387230	Yeah, for sure.
1387230	1393670	Anything, anything you want to let us know? Any new tools or updates you want to improve on? Or know...
1393670	1394210	I'd be...
1394210	1395110	Go ahead. Sorry.
1395110	1408650	I'd be lying if I, if I didn't have a, if I told you I didn't have a DC 29 folder on my computer already with a, with a notes document in there thinking about, about the next talk and how to take this the next step.
1408650	1423650	But yeah, I think people should just keep up with the latest news, either on Twitter or Reddit, Netsack, or anything like that. When you see a little nugget, like I saw Robin Wood was the one who originally kind of showed that this was even possible.
1423990	1431610	And you can kind of run with that and productize it and take it to the next level and then explain how it works. And, and that's good enough for a talk.
1431610	1445750	So yeah, definitely stay, stay with your ear to the ground and keep looking for what's going to be the next thing. And who knows, you know, I was, I was at DEF CON, I think it was like 23 was my first DEF CON and watching speakers talk.
1445750	1459510	And I thought that, man, that's crazy. These guys are, you know, next level. And then you keep working at it for a couple of years and you find your little niche, you know, you don't have to, you don't have to understand everything. Just expand the universe a little bit. And here you are.
1459510	1470970	So, so speaking to that, let's say I'm, this is my first DEF CON, you know, I, for whatever reason, I couldn't afford it or couldn't go. And now, hey, I find myself quarantined at home and I'm watching you.
1471630	1500930	And I'm, and this is something I find interesting and is, I know you posted your contact information on the end of your presentation. Is it okay for people to call you and, or maybe not call you, but, you know, get ahold of you, ping you and say, you know, hey, I got this idea or can you explain something or is there any place they can go to find more basic information, you know, a little more of the one-on-one? Is there any, do you know of any known or, you know, common sites we can push them to?
1501430	1530090	Yeah, sure. I mean, you're always available to hit me up at Bad Sector Labs is my personal Twitter handle and 6gen.io if you need commercial support. But for basic stuff, I would, I would look at the kind of the introduction CTF, intro level CTFs, like over the wire is really great. And those, those will kind of walk you through kind of the basics of how do you get into this mindset of, you know, there's something that's supposed to work a certain way.
1530090	1558650	How can I get it to work a different way? Or how can I do things to it that the author didn't intend? And that kind of mindset is just, if you apply that to everything that you see that comes out with cybersecurity and even just the latest technologies, yeah, I think you'll go far. And honestly, reading the RFCs, as boring as it is, you know, that's, that's where a lot of gold is. So stay up to date on the latest RFCs, dig in, look at implementations. Are people doing it correctly? Probably not.
1558650	1561730	Break their implementations. And there you go.
1561730	1586910	Yeah, I think the RFCs is a key piece, because it may be not a new feature. It may be an old feature that people forgot. And nobody's looked, you know, under the covers to know, hey, look at this feature, I can use this. You also mentioned WireGuard. I think that's another, you know, cool project that's come about the last few years. Can you speak to that? You're using that in this kind of attack?
1587950	1616890	Yeah, I didn't, I didn't test it out with this tool. But I love WireGuard. And I think it's going to be the future of VPN, super fast, super easy to set up small code base auditable props to the author on that. That's, that's an amazing tool. But yeah, if you can tunnel UDP traffic, then you can probably put WireGuard behind this kind of fronting or hiding. And therefore, the sky's the limit, you know, once once you get out to the open internet, and then you can run your VPN through there.
1617610	1621450	Censorship is kind of, they're playing catch up for sure.
1622050	1640450	Yeah, I mean, to me, this is, you know, from a hacker's perspective, I think a good application of this would be tunneling out through a censored network and getting access to Netflix so I could watch, you know, the latest, the latest, whatever hot lava show or whatever.
1640450	1654790	And that's one of the last questions here is, so we'd mentioned if, I think we've talked about this a little bit, but they're asking what if government made the demanded that domain fronting illegal?
1656130	1666750	What would break? I guess, you know, obviously making it illegal is one thing and then you have to have the capability of stopping it because you can say it's illegal.
1666750	1671370	If people continue to do it, you know, but I'll let you speak to it.
1671470	1679790	Yeah, I think I think we saw this happen in 2018 when the Russian government turned the screws on on AWS and Google and they they eventually capitulated.
1679790	1695570	So it's a possibility that, you know, enough governments with enough power, turn the screws on Cloudflare that they say it's not worth fighting this and we're gonna, you know, succumb to the demands and they're not a government so they can't really fight that fight.
1695570	1703390	But I think that they're going to try their hardest to to maintain the ability for any of their IP range to go to any of their domains.
1703390	1713150	I think it's it's one of their business advantages, honestly. So turning that off might be detrimental to even just the basic operating of Cloudflare.
1713270	1722210	So while it's possible, I think they're going to fight it as hard as they can. But if the government makes you do it, I mean, we've seen it time and time again.
1722210	1734490	Eventually, I think these companies will succumb to what the government forces. This isn't like a Bitcoin or a magic blockchain solution to anything where, you know, it's going to survive no matter what happens. But we'll see.
1734490	1756850	Yeah, yeah. And things continue to evolve. So yeah, we're, they might, this might get fixed by the industry. And then a new technique comes up, you know, so. So we're about out of time. Thank you for speaking here at DEF CON. And it was a great talk. And I appreciate all the questions you had.
1757310	1767790	Yeah, thanks, guys. And thanks to all the goons who made this possible. I think a lot of people tuning in don't realize that it was a major effort to get everything coordinated and work. So thanks to you guys.
1768070	1769230	Thank you.
1770830	1771750	All right.