WEBVTT 00:01.760 --> 00:06.200 Hello everyone, and welcome to Domain Fronting is Dead, Long Live Domain Fronting. 00:06.200 --> 00:11.940 Using TLS 1.3 to evade sensors, bypass network defenses, and blend in with the noise. 00:11.940 --> 00:19.700 My name is Eric Hunstad, and I'm the CTO and Adversary Emulation Lead at 6Gen, a full-spectrum cybersecurity company based in Annapolis, Maryland. 00:19.780 --> 00:25.040 Today, I'm going to go over what domain fronting is, including HTTP and HTTPS basics. 00:25.040 --> 00:30.800 I'll talk about TLS 1.3 and encrypted server name indication, and how this can be used for domain hiding. 00:31.120 --> 00:37.020 I'll show some demos of this in action, from basic bypasses, to bypassing an enterprise TLS decrypting firewall. 00:37.320 --> 00:41.560 And finally, I'll talk about some ideas for defenders who may want to detect this in their environment. 00:42.460 --> 00:43.920 Domain Fronting 101. 00:44.020 --> 00:50.900 If we're going to discuss a new method for domain fronting, let's all get up to speed on what domain fronting is, why it works, and what happened to it. 00:51.300 --> 00:55.680 Like many things in security, to understand the cool stuff, you have to dig into the basics. 00:55.680 --> 01:00.920 To understand domain fronting, we really need to understand HTTP and HTTPS. 01:01.060 --> 01:09.220 Specifically, we'll be focused on server name indication, which is a special extension added to TLS that allows multiple sites to be hosted on the same server. 01:09.220 --> 01:18.360 But TLS 1.3, this extension can be encrypted, and combined with encrypted secure DNS, we can do domain fronting 2.0, or domain hiding. 01:19.140 --> 01:21.320 Okay, let's start with HTTP. 01:21.440 --> 01:24.700 I'll skip over the super low-level stuff as it's not relevant here. 01:24.700 --> 01:29.660 The first external request involved in an HTTP connection is a DNS request. 01:29.700 --> 01:34.300 This DNS request gets sent out unencrypted via UDP to port 53. 01:34.860 --> 01:43.140 The DNS server responds, also unencrypted via UDP, with an answer that contains the IP address of the domain from the request. 01:43.440 --> 01:53.520 Now that the user's computer knows where the domain is hosted, it can issue an HTTP request to that IP address, with the host header indicating the domain the user wishes to browse. 01:53.520 --> 01:56.840 This request uses TCP, but it's unencrypted. 01:57.120 --> 02:01.160 The web server responds with content, in this case some simple HTML. 02:01.460 --> 02:11.960 This was fine for the early days of the Internet, when it was just a few research institutions, but as soon as people wanted to conduct business online, the obvious issues with completely unencrypted traffic had to be fixed. 02:12.500 --> 02:19.020 Plus, without encryption, all your data is free game for any number of actors that are positioned between you and the web server. 02:19.920 --> 02:24.760 HTTPS was the answer to this problem, and it starts off the same way, with a DNS request. 02:24.980 --> 02:31.680 The DNS request and response are still unencrypted when using HTTPS, unless you use a different system, which we'll talk about later on. 02:31.780 --> 02:34.360 Here's where HTTPS is different. 02:34.360 --> 02:39.180 The connection to the server starts with a TLS connection, specifically a client hello. 02:39.380 --> 02:45.040 In this packet, the server name extension is used to tell the web server which website the user is requesting. 02:45.160 --> 02:48.820 This allows the web server to return the proper certificate for that site. 02:48.820 --> 02:57.940 The server returns the certificate in a server hello packet, and then the client and server can agree on which encryption algorithms to use, and exchange session keys. 02:58.240 --> 03:07.760 All traffic after this is encrypted, but both the client hello and server hello are unencrypted, which leaks the domain and certificate to anyone between the client and the web server. 03:08.400 --> 03:11.940 And we know there are plenty of organizations interested in that data. 03:12.220 --> 03:15.520 With the basics covered, let's talk about domain fronting. 03:15.520 --> 03:24.500 Domain fronting is a technique to get around network defenses or sensors by connecting to an approved server, while hiding the true destination of the HTTPS request. 03:24.620 --> 03:32.580 One of the biggest restrictions of domain fronting is that the fronted domain and the true destination domain must be hosted by the same service provider. 03:32.620 --> 03:35.000 Typically, this is a content delivery network. 03:35.180 --> 03:37.420 So how does domain fronting work in practice? 03:37.420 --> 03:43.020 Well, it starts with the same DNS lookup as before, and the TLS handshake is the same as well. 03:43.020 --> 03:48.840 However, in the handshake, the client connects to the front domain, not the true destination domain. 03:48.840 --> 03:52.940 Once the TLS handshake is complete with the front domain, the fronting begins. 03:52.940 --> 04:00.780 The client sends an HTTP request to the front domain, wrapped in TLS, but with a host header of the true destination domain. 04:00.900 --> 04:12.160 When the content delivery provider receives this request and inspects the host header, determines that it's for a different domain hosted on the same provider, and dutifully forwards the request to the true destination domain. 04:12.160 --> 04:15.960 You can think of domain fronting like a postcard inside an envelope. 04:16.160 --> 04:23.540 On the outside of the envelope, the client writes the address of the CDN, but on the inside, the true destination domain is on the postcard. 04:23.760 --> 04:31.280 The network filters or sensors are like mailmen, who can see the outside of the envelope, and they deliver it, since the CDN is an approved address. 04:31.360 --> 04:36.880 However, when the CDN opens the letter, they deliver the postcard to the true destination internally. 04:37.140 --> 04:49.680 This was working well on all major CDNs until April of 2018, when the Russian government put pressure on cloud providers to stop it, since the popular messaging app Telegram was using both Google and AWS for domain fronting. 04:49.720 --> 04:59.920 Both Google and Amazon stated that the reasons were technical, and domain fronting was either never supported or a violation of Terms of Service, and other providers follow suit to include Cloudflare. 05:00.240 --> 05:06.740 Microsoft's Azure platform still allows domain fronting for now, as do a few other smaller CDNs. 05:07.020 --> 05:12.140 Domain fronting is a great censorship or network bypass technique, but it has some major problems. 05:12.280 --> 05:20.940 For any network censorship bypass to be effective, it has to be really painful to block, and since the largest provider shut it down, it lacks the punch that it once had. 05:21.120 --> 05:30.640 Also, the true destination domain has to be hosted on the same service as the front domain, which limits the potential front sites and imposes costs on anyone wishing to set up fronting. 05:30.860 --> 05:38.380 With the background out of the way, we can talk about TLS 1.3, encrypted server name indication, and what can be called domain hiding. 05:38.380 --> 05:40.980 First, let's talk about TLS 1.3. 05:40.980 --> 05:43.480 It's seeing steady growth on the internet today. 05:43.480 --> 05:48.920 This is data from Quali's SSL Labs, which tests the most popular 150,000 sites. 05:48.960 --> 05:56.260 As you can see, TLS 1.3 is on the rise, and is supported by 31.7% of those sites tested last month. 05:57.020 --> 06:03.360 Cloudflare is seeing 59% of all traffic using TLS 1.3, again, showing a steady increase. 06:03.360 --> 06:11.160 Before we dig into the mechanics of a TLS 1.3 handshake with ESNI, we first have to secure the DNS process. 06:11.220 --> 06:15.900 If a network defender or sensor can see our DNS requests, they can easily be blocked. 06:16.080 --> 06:20.500 Also, secure DNS is a cornerstone of ESNI, which we'll see in a moment. 06:20.620 --> 06:24.520 The first solution is to simply wrap a DNS query in a TLS connection. 06:24.580 --> 06:32.660 To make things even more difficult for network defenders or sensors, DNS via HTTPS is a newer standard that is seeing more widespread adoption. 06:33.580 --> 06:43.600 Once we have an encrypted way to perform DNS queries, we can use this to retrieve a public key for a domain which we can use to encrypt the server name in the client hello packet. 06:43.620 --> 06:54.020 In this screenshot from Wireshark, you can see the encrypted underscore server underscore name TLS extension, and the encrypted SNI value is just an encrypted hex string. 06:54.020 --> 06:59.760 No longer can defenders use the server name value to block traffic based on a plain text value. 06:59.760 --> 07:07.920 A TLS 1.3 connection with ESNI relies on having the server's public key in order to encrypt the server name in the client hello. 07:07.920 --> 07:15.080 And the current standard for how to get this key is to use DNS and query the underscore ESNI text record for that domain. 07:15.480 --> 07:23.160 This request returns a Base64 encoded public key the client can use to derive a session key that encrypts the server name. 07:23.420 --> 07:27.880 Providers that support this should rotate this key to provide some forward secrecy. 07:27.880 --> 07:37.100 Cloudflare, for example, rotates this key every hour, but allows a few hours of buffer in case a client has a slightly stale key from a previous request or from caching. 07:38.000 --> 07:46.820 Armed with the server's ESNI public key, the client derives a session key and uses that to encrypt the server name and sends the client hello. 07:47.220 --> 07:56.060 The server has the corresponding private key and can derive the same session key, which it uses to decrypt the server name and performs the same steps as standard TLS. 07:56.060 --> 08:02.880 This session key generation ensures that the key is tied to the session that generated it and prevents replay attacks. 08:03.440 --> 08:10.700 Another difference between TLS 1.3 and previous versions is that the certificate returned by the server is encrypted. 08:10.880 --> 08:24.720 This is possible because in TLS 1.3, the client hello includes a list of supported ciphers, and the client also makes a guess as to which key agreement algorithm the server is going to choose, and it preemptively sends a keyshare for that algorithm. 08:24.720 --> 08:29.420 So all the plain text portions of HTTPS we saw before have been mitigated. 08:29.420 --> 08:34.020 However, there are still weak spots where sensors or network defenders could stop the connection. 08:34.680 --> 08:43.840 Even if the DNS request itself is encrypted inside either TLS or HTTPS, the data returned by a resolver may be poisoned. 08:43.920 --> 08:53.120 Imagine a scenario where a country or enterprise returns their own key for all underscore ESNI requests, no matter the domain, to enable man-in-the-middle. 08:53.120 --> 08:58.540 DNSSEC solves this problem by signing the record set and allowing the chain of trust to be checked. 08:58.660 --> 09:02.980 The details of DNSSEC are outside the scope of this talk, but I encourage you to check it out. 09:03.500 --> 09:06.580 But what if encrypted DNS is completely blocked? 09:06.580 --> 09:12.220 The solution here is to bootstrap the connection by preloading the current ESNI keys by some other means. 09:12.280 --> 09:18.180 Maybe encrypted DNS is blocked, but what if I have a script that updates a GitHub gist with the current keys every hour? 09:18.180 --> 09:24.380 Or, you could just bake the keys directly into your binary if you know it's going to be executed in the next hour or two. 09:24.960 --> 09:30.700 Okay, let's say we get a valid ESNI key, but the IP address of the server we want to connect to is blocked. 09:30.920 --> 09:34.500 The IP may be shared with other sites, but it doesn't have to be. 09:34.700 --> 09:39.560 Is there a way to route from arbitrary domains and IPs to our true destination domain and IP? 09:39.720 --> 09:42.500 The answer is no, but let's see how close we can get. 09:43.380 --> 09:53.660 So far, I haven't introduced anything new, but we have covered a lot of ground and identified potential issues and solutions around domain hiding with TLS 1.3 and ESNI. 09:53.660 --> 09:57.840 Let's tackle that last point and get around IP or domain blocks. 09:58.740 --> 10:02.560 We'll do this by leveraging the world's largest CDN. 10:02.800 --> 10:06.640 Cloudflare was founded in 2009 and has had explosive growth. 10:06.640 --> 10:15.540 It's now the world's largest CDN, with the most internet exchange points, as well as being an authoritative DNS server hosting over 26 million domains. 10:15.540 --> 10:24.760 Cloudflare is on the forefront of internet technology and supports TLS 1.3, ESNI, WebSockets, QUIC, and DNSSEC. 10:24.920 --> 10:32.280 This new technique, which I'm calling domain hiding, accomplishes the same goals as domain fronting, but uses different technologies. 10:32.280 --> 10:43.980 A TLS 1.3 connection with an ESNI of the true destination is made to any Cloudflare IP, and the underlying HTTP request also has a host header of the true destination. 10:44.120 --> 10:50.840 This enables any Cloudflare-owned IP to act as a front for any domain hosted by Cloudflare DNS. 10:50.920 --> 10:56.060 This technique was first shown to be possible by Robin Wood, so props to him for the discovery. 10:56.980 --> 11:10.040 Today, I'm releasing Noctilucent, a project that enables ESNI in Go's crypto-slash-TLS library, as well as exposes two additional configuration options, ESNI Server Name and Preserve SNI. 11:10.040 --> 11:16.560 It can be used by any project currently using the standard crypto-slash-TLS library, as it's backwards compatible. 11:16.560 --> 11:28.420 It comes with a demo client application that allows you to control just about every part of the TLS connection, as well as the HTTP GET request, so you can play with what's possible using this technique. 11:28.420 --> 11:36.020 It also supports WebSockets, and through the magic of Go, it compiles cross-platform to a single binary with no dependencies. 11:37.180 --> 11:41.820 Here's what a standard TLS 1.3 connection looks like with Noctilucent. 11:41.820 --> 11:49.880 We've specified the TLS host in red, the server name indication in yellow, and the HTTP host header in green. 11:49.880 --> 11:51.860 The output shows what we expect. 11:51.860 --> 12:04.060 There's no ESNI, the SNI is set to cloudflare.com, we're connecting to cloudflare.com on port 443, and our GET request to cloudflare.com returns the expected 301 response. 12:04.060 --> 12:05.700 Nothing fancy yet. 12:06.720 --> 12:10.480 Let's enable ESNI and set the ESNI server name. 12:10.480 --> 12:22.180 Everything is the same as the normal TLS 1.3 connection we looked at on the last slide, except this time, the ESNI value is set to cloudflare.com, and there is no standard SNI. 12:22.180 --> 12:25.520 Progress, but no fronting or domain hiding quite yet. 12:25.520 --> 12:39.240 Now I'll change the ESNI server name and HTTP host header to a domain we control, in this case, DEFCON28.hackthis.computer, but leave the TLS host as cloudflare.com. 12:39.240 --> 12:49.760 Notice that this connection is still going to cloudflare.com, but the HTTP request successfully goes to DEFCON28.hackthis.computer, which returns hello DEFCON. 12:49.760 --> 12:55.620 To anyone between this host and cloudflare, it appears to be a TLS connection to cloudflare.com. 12:55.620 --> 12:58.140 We've successfully achieved domain hiding. 12:58.980 --> 13:00.980 Now the fun begins. 13:00.980 --> 13:07.680 Here we send an ESNI and standard SNI, where the standard SNI is cloudflare.com. 13:07.680 --> 13:14.040 Because cloudflare ignores the standard SNI when an encrypted SNI is used, we can set it to anything. 13:14.040 --> 13:29.820 The HTTP connection works as before, and this time, anyone between this host and cloudflare, looking at SNI data, thinks this connection is going to cloudflare.com, in addition to the underlying TLS connection actually connecting to a cloudflare.com IP. 13:30.220 --> 13:38.640 What I've demonstrated is the ability to arbitrarily front or hide any IP behind a domain that is using DNS provided by Cloudflare. 13:38.640 --> 13:43.880 The true destination IP doesn't have to be a Cloudflare worker or other Cloudflare service. 13:43.880 --> 13:48.740 In fact, all the examples today are IPs hosted by DigitalOcean. 13:48.920 --> 13:54.440 Best of all, the requirements to sign up for Cloudflare DNS are minimal, and it's free. 13:55.960 --> 13:58.340 What domains can we use for this hiding? 13:58.340 --> 13:59.780 It turns out a lot. 13:59.780 --> 14:06.220 Over 21% of the top 100,000 sites are behind Cloudflare and allow this to work. 14:06.220 --> 14:15.880 Some notable examples are shown below, including 3 of 11 domains which are both on the Palo Alto decryption whitelist and behind Cloudflare. 14:15.880 --> 14:18.400 I'll demonstrate why this is important in a minute. 14:19.360 --> 14:29.340 There's a little bit of everything on this list, from security-related sites, banks, sports, higher education, streaming services, and even government sites. 14:29.340 --> 14:30.460 And porn. 14:30.460 --> 14:32.100 So much porn. 14:32.680 --> 14:37.620 Here's a good example of how the variety of available domains to hide behind can be useful. 14:37.840 --> 14:44.920 This is a request to www.okta.com, a single sign-on and identity provider used by many enterprises. 14:44.940 --> 14:48.300 But the actual request is going to a server I control. 14:48.500 --> 14:56.300 To a sensor or network defender, this looks like standard Okta traffic, with the exception of the ESNI extension being used. 14:56.300 --> 15:02.080 The logical next step is to try out Noctilucent with some web filtering and see how it fares. 15:02.080 --> 15:08.780 It turns out most web filtering is done by inspecting the ESNI and making a filtering decision based on that. 15:08.780 --> 15:16.580 Here I set up the untangle firewall with federal government settings and explicitly blocked defcon28.hacktest.computer. 15:17.160 --> 15:22.360 On a host behind the untangle firewall, I made this request, which succeeded. 15:22.480 --> 15:29.720 I set the unencrypted SNI to bypass-untangle.com to see if it appears in the logs anywhere. 15:31.260 --> 15:40.320 After repeating the request 30 times or so, back on the firewall, we can see that the top domains include the decoy domain that we sent with the request. 15:41.140 --> 15:47.520 Simple SNI filters were easy to defeat, but what about HTTPS decrypting firewalls? 15:47.540 --> 15:56.740 These are often seen in enterprise environments and work by breaking and then re-encrypting HTTPS using a root certificate placed on endpoints behind the firewall. 15:56.740 --> 16:03.060 You can think of it as a corporate man-in-the-middle that allows the network defenders to inspect full, encrypted packet data. 16:03.260 --> 16:13.580 Kazakhstan actually attempted to implement this nationwide for a few weeks back in July of 2019 before major international backlash forced them to abandon what they called a, quote, test. 16:14.540 --> 16:23.460 I headed to AWS and spun up the latest and greatest in enterprise firewalls, Apollo Alto PAVM version 10.0.0. 16:23.460 --> 16:29.980 Released in June, this version touts the ability to decrypt TLS 1.3 with a two-time speed boost. 16:30.120 --> 16:36.320 Imagine my surprise when it was all set up and the default decryption profile didn't include TLS 1.3. 16:36.320 --> 16:41.140 Instead, it let it pass through while logging an error about a version mismatch. 16:42.080 --> 16:53.160 Once set up with a decryption profile that enabled decryption of TLS 1.3, I pulled the list of default domains that are allowed to bypass decryption for a variety of reasons, such as having a PIN certificate. 16:53.540 --> 16:58.540 Here, the firewall interface is on the left and a Windows VM behind the firewall is on the right. 16:58.740 --> 17:04.920 We can see that decryption is enabled and we have no other decryption rules that would allow traffic to bypass decryption. 17:04.920 --> 17:13.700 Normally, in an enterprise environment, categories like banking or healthcare bypass decryption due to privacy laws, but this setup is as strict as possible. 17:13.700 --> 17:15.520 No explicit exemptions. 17:15.920 --> 17:23.360 On the Windows VM, I'll run a PowerShell script that checks the certificate hashes of major sites against hard-coded known fingerprints. 17:23.580 --> 17:27.780 We see that it fails, indicating that all tested sites are being man-in-the-middled. 17:27.780 --> 17:35.640 Additionally, opening a browser and going to google.com shows a lock icon, but inspecting the certificate shows that it's the firewall certificate. 17:36.160 --> 17:51.280 Switching to the command line and using the Noctilucent test client, I'll set the TLS host to mozilla.org, the host header to defcon28.hackthis.computer, and the unencrypted SNI to mozilla.org, which is on the decryption exemption list. 17:51.280 --> 17:54.280 The SNI is preserved and set to mozilla.org. 17:54.280 --> 18:01.700 A connection is made to mozilla.org on port 443, and we see the hello defcon28 response we expect from the test server. 18:02.280 --> 18:07.400 We'll run an NS lookup of mozilla.org so we can look for the connection in the logs. 18:11.460 --> 18:17.180 Looking at the firewall traffic logs shows a connection to one of the IP addresses for mozilla.org. 18:21.120 --> 18:33.400 Moving to the decryption logs, we see DNS over HTTPS, which is the ESNI key lookup that is performed before each TLS connection, but no traffic to mozilla.org's IP. 18:36.060 --> 18:40.640 To make sure we didn't miss it, I'll add a filter for the two possible IPs. 18:41.380 --> 18:46.080 No results, because mozilla.org is exempted from decryption by default. 18:46.080 --> 18:53.300 We have successfully defeated an HTTPS decrypting firewall by using built-in default exemption and domain hiding. 18:53.380 --> 18:59.880 This bypass is great, but every time we want to send data, we'll add an entry to the traffic log, which could stand out. 18:59.880 --> 19:01.720 What if there was a better way to hide? 19:01.720 --> 19:07.720 This technique works at the TLS level, so any protocol that Cloudflare supports underneath would also work. 19:07.720 --> 19:12.960 We saw that regular HTTP requests work, but create a new connection for each request. 19:13.300 --> 19:21.060 WebSockets are a technology used by many streaming services that enable full duplex communication channels over a single TCP connection. 19:21.200 --> 19:30.900 And we can take this a step further by using one or more of these WebSocket connections to tunnel arbitrary TCP or UDP packets to a proxy beyond the firewall. 19:32.200 --> 19:41.120 Here's the Noctilucent test client sending and receiving via WebSockets to our test server while hiding behind www.okta.com. 19:41.240 --> 19:52.200 WebSockets are the perfect candidate to use as a wrapper for a proxy protocol that allows us to send arbitrary TCP or UDP traffic using a helper program while only making a single TCP connection. 19:52.200 --> 20:06.200 The Noctilucent project includes a fork of the Amazing Cloak project by Andy Wang that enables two new options in the client configuration, ESNI Server Name and Preserve SNI, which operate just like they did on the test client. 20:06.540 --> 20:23.740 Cloak is a universal, pluggable transport that cryptographically obfuscates proxy traffic as legitimate HTTPS traffic, disguises the proxy server as a normal web server, and multiplexes traffic through a fixed amount of TCP connection while providing multi-user usage control. 20:23.740 --> 20:28.260 With the additions from Noctilucent, it can now do domain hiding as well. 20:29.060 --> 20:36.620 Here's the same Palo Alto and Windows VMs from before, with a public IP of 3.14.44.10. 20:36.740 --> 20:42.460 This time, the Windows VM is running a Noctilucent Cloak client and a local Shadowsocks client. 20:42.460 --> 20:48.700 The Cloak client is connecting to a VPS running the standard Cloak server and a Shadowsocks server. 20:49.020 --> 20:57.120 Firefox is configured to use the local Socks proxy provided by the Shadowsocks client running locally, and I can browse the internet as I would normally. 20:57.580 --> 21:07.580 The certificates of websites in this proxy browser are legitimate, as they are not being intercepted by the Palo Alto firewall, and all traffic is appearing to come from the DigitalOcean VPS. 21:15.550 --> 21:17.770 The speed is surprisingly good. 21:17.830 --> 21:23.410 During testing, I was getting speeds up to 100 Mbps using four TCP connections with Cloak. 21:23.610 --> 21:34.850 If we look at the firewall logs, we only see the four connections to Mozilla.org, which are not decrypted, while the client has been browsing freely, with all traffic remaining encrypted and uncensored. 21:38.010 --> 21:45.470 Switching to the decryption logs, we see the DNS over HDPS requests, but no requests to Mozilla.org. 21:47.270 --> 21:52.050 I'll filter for the Mozilla.org IPs, but we'll get no results. 21:53.410 --> 22:02.710 What we see here is arbitrary traffic at high speeds using only four TCP connections to what Palo Alto thinks is Mozilla.org. 22:04.010 --> 22:07.390 Can we use this technique for our pre-existing Red Team tools? 22:07.390 --> 22:09.670 If they support proxying, you bet. 22:09.670 --> 22:14.990 Here's a Cobalt Strike listener with a SOCKS proxy setting for localhost port 9999. 22:15.470 --> 22:22.850 For fun, we'll set Cloak to use bitdefender.com as the front domain, which has the IP address shown here. 22:24.890 --> 22:34.210 After starting the Noctilucent Cloak client and local Shadowsocks server in SOCKS 4 mode, we'll double-click on an innocent-looking EXE. 22:34.210 --> 22:37.750 We get the callback immediately, and it works as expected. 22:39.310 --> 22:43.090 And of course, the first thing you do when you get access is pop a calc. 22:48.240 --> 22:54.960 Switching to the firewall and looking at the decryption logs, we only see entries for cloudflaredns.com. 22:59.250 --> 23:04.710 I'll paste in a filter for the two possible IPs for bitdefender.com, and we get no results. 23:08.600 --> 23:17.080 Looking at the traffic logs, we see four connections to the IP of bitdefender.com, as well as lots of DNS over HTTPS. 23:21.070 --> 23:26.530 Our Cobalt Strike beacon is still functioning as normal, and any network defender is none the wiser. 23:27.270 --> 23:36.930 Packaging up Cobalt Strike, Cloak, and Shadowsocks into a clean package, possibly communicating over namepipes instead of sockets, is an exercise left up to the viewer. 23:38.350 --> 23:46.650 Red teamers are no doubt wondering how much work it would be to integrate this with their existing tooling, and I can say if you're using Go, it's almost no work at all. 23:46.650 --> 23:49.430 Here's a new C2 framework called DemoC2. 23:49.430 --> 23:54.290 It was released just five days ago and is built using Go and Vue.js. 23:56.970 --> 24:03.190 I've set up an instance here with an HTTPS listener with a public IP on port 443. 24:03.670 --> 24:08.690 Going into the listener settings, we can view the source code for an HTTPS agent. 24:12.760 --> 24:17.820 It has the C2 host and port in the code, and the agent will connect directly to this host. 24:18.720 --> 24:23.140 Switching to an editor, I've made a few modifications to get this to work with Noctilucent. 24:23.280 --> 24:34.180 First, I set the host to bitdefender.com, and I've added two new variables, frontDomain and actualDomain, which are set to bitdefender.com and demos.hackess.computer. 24:35.100 --> 24:39.260 The actual code changes needed to get everything working is just 30 lines. 24:39.320 --> 24:43.540 First, we have to query for the ESNI keys and parse them. 24:43.540 --> 24:49.000 If we wanted to, we could bake in the current keys here to avoid the network traffic related to querying for them. 24:49.320 --> 24:53.920 Then I define a TLS config and set the minimum value to TLS 1.3. 24:54.120 --> 25:01.980 I'll define the two new options from Noctilucent, ESNI server name is set to the actual domain, and preserve SNI is set to true. 25:03.560 --> 25:09.600 I define an HTTP client by hand, so that I can tell it to use this TLS config when dialing the host. 25:10.620 --> 25:17.860 Besides that, the only change to the agent is having the HTTP client use the actual host when it makes its POST requests. 25:20.580 --> 25:27.180 On our Windows VM, I'll set a filter in Wireshark for the IP of Bitdefender, so we can inspect the client hello. 25:27.460 --> 25:31.360 I'll double-click this new innocent exe and switch back to Wireshark. 25:33.620 --> 25:41.760 In the client hello, we see the unencrypted server name is Bitdefender.com, and the encrypted server name is just a bunch of hex. 25:48.480 --> 25:51.400 Back on the C2 server, we have an agent calling back. 25:53.080 --> 25:55.020 Notice the external IP. 25:55.100 --> 26:00.880 When using this technique, the reported external IP will be from one of Cloudflare's ranges. 26:10.410 --> 26:13.950 Just like with CobaltStrike, everything works as normal. 26:15.450 --> 26:20.290 You can run WhoAmI, pop a calc, and check the public IP. 26:28.240 --> 26:33.460 Besides the external IP appearing to be from a Cloudflare range, the agent functions as expected. 26:43.520 --> 26:48.540 Back on the Windows VM, I'll search for traffic to the C2's server IP. 26:49.700 --> 26:51.140 No packets. 26:51.140 --> 26:57.180 There's no trickery here, but just a small modification, this Go-based agent is using domain hiding. 26:57.860 --> 27:03.700 So what is the blue team to do, given that we've just seemingly bypassed the most sophisticated network defenses? 27:03.760 --> 27:07.360 The easiest thing to do would be to block TLS 1.3. 27:07.520 --> 27:15.340 But as that's likely anywhere from 25% to 50% of the TLS traffic leaving your network, it might make a few users unhappy. 27:15.560 --> 27:22.680 You could block Cloudflare, but dropping 21% of the most popular 100,000 domains probably is not a good idea either. 27:22.980 --> 27:26.160 What about just targeting client hellos with ESNIs? 27:26.160 --> 27:31.380 It's possible, and one vendor has had the ability to do that for 6 months now. 27:31.380 --> 27:35.700 However, since you can't determine the destination site, it's all or none blocking. 27:35.700 --> 27:40.820 There's no way to only block selective domains if you block TLS connections that have an ESNI. 27:40.820 --> 27:46.340 I was surprised by the lack of support for this across major vendors, but perhaps it's just not an issue yet. 27:46.980 --> 27:55.940 The most sensible way to detect this specific technique is to alert on packets that contain both a server name and an encrypted server name TLS extension. 27:55.940 --> 28:07.020 There are no default helpers for TLS and SNORT besides SSL underscore state, which can help you get the client hello packet, and TLS dot version, which can get you TLS 1.3. 28:07.020 --> 28:11.400 So you'll have to get creative with the content parser and look for the extension types. 28:12.020 --> 28:22.600 SecureKata has a bit more horsepower than SNORT, but only has a selector for TLS dot SNI and not TLS dot ESNI, so custom rules will be required here as well. 28:23.400 --> 28:28.040 It's going to come down to tried and true techniques that work regardless of traffic content. 28:28.080 --> 28:36.600 Things like beaconing detection and anomaly detection can help show clients that all of a sudden start sending regular traffic to destinations that perhaps don't make sense. 28:36.800 --> 28:39.880 There are even some open source tools that do just this. 28:40.180 --> 28:45.400 A network defender should be asking if a traffic pattern to a given endpoint makes sense in context. 28:45.520 --> 28:50.480 Perhaps that Windows machine and accounting shouldn't be sending gigs of data to your single sign-on provider. 28:52.380 --> 28:57.140 TLS fingerprinting with JAW3 and JAW3S is another technique that could be effective. 28:57.340 --> 29:05.700 JAW3 looks at the specific options used in a client hello that can sometimes uniquely identify a client, and JAW3S does the same for servers. 29:06.180 --> 29:15.320 This will work until attackers tune their tools to be identical to browsers and common web servers, and work is already underway to do this in projects like Cloak and uTLS. 29:15.320 --> 29:25.360 As always, a layered approach that uses strong, instrumented and managed endpoint detection and response, and other techniques, is the best way to prevent compromise. 29:25.440 --> 29:30.500 Or as Microsoft so helpfully says, just don't let untrusted code ever run on your server. 29:31.520 --> 29:42.280 To wrap up today, I've discussed the basics of HTTP, HTTPS, domain fronting, and a new form of censorship-resistant communication, domain hiding. 29:42.280 --> 29:50.120 This technique is usable today with Noctilucent, either as a drop-in replacement for the Go TLS library, or as a proxy. 29:50.300 --> 29:57.340 The ESNI standard is still very much a draft, and this technique will continue to evolve as the changes in adoption grows. 29:57.340 --> 30:00.340 In fact, it's not even called ESNI anymore. 30:00.620 --> 30:05.100 Just as with standard domain fronting, this relies on the CDN permitting it. 30:05.100 --> 30:11.340 However, in this case, besides preserving the standard SNI header, we aren't really doing anything terribly out of spec. 30:11.340 --> 30:16.040 I look forward to seeing the snort and securicada rules to detect and block this technique. 30:16.100 --> 30:22.440 And like every good red teamer, I'll push my customers with the latest techniques, hopefully before the real adversary can. 30:22.880 --> 30:30.580 I want to give special thanks to Robin Wood, who first showed this technique was possible with a rough POC using a fork of OpenSSL. 30:30.760 --> 30:37.740 Andy Wang, the creator of Cloak, and Nick Sullivan from Cloudflare, who helped get me data on TLS 1.3 adoption. 30:38.640 --> 30:40.660 Noctilucent is available now on GitHub. 30:40.660 --> 30:47.580 If you have any questions or comments, feel free to hit me up on Twitter, where my personal handle is at badsectorlabs. 30:47.580 --> 30:48.940 Thanks for watching.