1
00:00:01,760 --> 00:00:06,200
Hello everyone, and welcome to Domain Fronting is Dead, Long Live Domain Fronting.

2
00:00:06,200 --> 00:00:11,940
Using TLS 1.3 to evade sensors, bypass network defenses, and blend in with the noise.

3
00:00:11,940 --> 00:00:19,700
My name is Eric Hunstad, and I'm the CTO and Adversary Emulation Lead at 6Gen, a full-spectrum cybersecurity company based in Annapolis, Maryland.

4
00:00:19,780 --> 00:00:25,040
Today, I'm going to go over what domain fronting is, including HTTP and HTTPS basics.

5
00:00:25,040 --> 00:00:30,800
I'll talk about TLS 1.3 and encrypted server name indication, and how this can be used for domain hiding.

6
00:00:31,120 --> 00:00:37,020
I'll show some demos of this in action, from basic bypasses, to bypassing an enterprise TLS decrypting firewall.

7
00:00:37,320 --> 00:00:41,560
And finally, I'll talk about some ideas for defenders who may want to detect this in their environment.

8
00:00:42,460 --> 00:00:43,920
Domain Fronting 101.

9
00:00:44,020 --> 00:00:50,900
If we're going to discuss a new method for domain fronting, let's all get up to speed on what domain fronting is, why it works, and what happened to it.

10
00:00:51,300 --> 00:00:55,680
Like many things in security, to understand the cool stuff, you have to dig into the basics.

11
00:00:55,680 --> 00:01:00,920
To understand domain fronting, we really need to understand HTTP and HTTPS.

12
00:01:01,060 --> 00:01:09,220
Specifically, we'll be focused on server name indication, which is a special extension added to TLS that allows multiple sites to be hosted on the same server.

13
00:01:09,220 --> 00:01:18,360
But TLS 1.3, this extension can be encrypted, and combined with encrypted secure DNS, we can do domain fronting 2.0, or domain hiding.

14
00:01:19,140 --> 00:01:21,320
Okay, let's start with HTTP.

15
00:01:21,440 --> 00:01:24,700
I'll skip over the super low-level stuff as it's not relevant here.

16
00:01:24,700 --> 00:01:29,660
The first external request involved in an HTTP connection is a DNS request.

17
00:01:29,700 --> 00:01:34,300
This DNS request gets sent out unencrypted via UDP to port 53.

18
00:01:34,860 --> 00:01:43,140
The DNS server responds, also unencrypted via UDP, with an answer that contains the IP address of the domain from the request.

19
00:01:43,440 --> 00:01:53,520
Now that the user's computer knows where the domain is hosted, it can issue an HTTP request to that IP address, with the host header indicating the domain the user wishes to browse.

20
00:01:53,520 --> 00:01:56,840
This request uses TCP, but it's unencrypted.

21
00:01:57,120 --> 00:02:01,160
The web server responds with content, in this case some simple HTML.

22
00:02:01,460 --> 00:02:11,960
This was fine for the early days of the Internet, when it was just a few research institutions, but as soon as people wanted to conduct business online, the obvious issues with completely unencrypted traffic had to be fixed.

23
00:02:12,500 --> 00:02:19,020
Plus, without encryption, all your data is free game for any number of actors that are positioned between you and the web server.

24
00:02:19,920 --> 00:02:24,760
HTTPS was the answer to this problem, and it starts off the same way, with a DNS request.

25
00:02:24,980 --> 00:02:31,680
The DNS request and response are still unencrypted when using HTTPS, unless you use a different system, which we'll talk about later on.

26
00:02:31,780 --> 00:02:34,360
Here's where HTTPS is different.

27
00:02:34,360 --> 00:02:39,180
The connection to the server starts with a TLS connection, specifically a client hello.

28
00:02:39,380 --> 00:02:45,040
In this packet, the server name extension is used to tell the web server which website the user is requesting.

29
00:02:45,160 --> 00:02:48,820
This allows the web server to return the proper certificate for that site.

30
00:02:48,820 --> 00:02:57,940
The server returns the certificate in a server hello packet, and then the client and server can agree on which encryption algorithms to use, and exchange session keys.

31
00:02:58,240 --> 00:03:07,760
All traffic after this is encrypted, but both the client hello and server hello are unencrypted, which leaks the domain and certificate to anyone between the client and the web server.

32
00:03:08,400 --> 00:03:11,940
And we know there are plenty of organizations interested in that data.

33
00:03:12,220 --> 00:03:15,520
With the basics covered, let's talk about domain fronting.

34
00:03:15,520 --> 00:03:24,500
Domain fronting is a technique to get around network defenses or sensors by connecting to an approved server, while hiding the true destination of the HTTPS request.

35
00:03:24,620 --> 00:03:32,580
One of the biggest restrictions of domain fronting is that the fronted domain and the true destination domain must be hosted by the same service provider.

36
00:03:32,620 --> 00:03:35,000
Typically, this is a content delivery network.

37
00:03:35,180 --> 00:03:37,420
So how does domain fronting work in practice?

38
00:03:37,420 --> 00:03:43,020
Well, it starts with the same DNS lookup as before, and the TLS handshake is the same as well.

39
00:03:43,020 --> 00:03:48,840
However, in the handshake, the client connects to the front domain, not the true destination domain.

40
00:03:48,840 --> 00:03:52,940
Once the TLS handshake is complete with the front domain, the fronting begins.

41
00:03:52,940 --> 00:04:00,780
The client sends an HTTP request to the front domain, wrapped in TLS, but with a host header of the true destination domain.

42
00:04:00,900 --> 00:04:12,160
When the content delivery provider receives this request and inspects the host header, determines that it's for a different domain hosted on the same provider, and dutifully forwards the request to the true destination domain.

43
00:04:12,160 --> 00:04:15,960
You can think of domain fronting like a postcard inside an envelope.

44
00:04:16,160 --> 00:04:23,540
On the outside of the envelope, the client writes the address of the CDN, but on the inside, the true destination domain is on the postcard.

45
00:04:23,760 --> 00:04:31,280
The network filters or sensors are like mailmen, who can see the outside of the envelope, and they deliver it, since the CDN is an approved address.

46
00:04:31,360 --> 00:04:36,880
However, when the CDN opens the letter, they deliver the postcard to the true destination internally.

47
00:04:37,140 --> 00:04:49,680
This was working well on all major CDNs until April of 2018, when the Russian government put pressure on cloud providers to stop it, since the popular messaging app Telegram was using both Google and AWS for domain fronting.

48
00:04:49,720 --> 00:04:59,920
Both Google and Amazon stated that the reasons were technical, and domain fronting was either never supported or a violation of Terms of Service, and other providers follow suit to include Cloudflare.

49
00:05:00,240 --> 00:05:06,740
Microsoft's Azure platform still allows domain fronting for now, as do a few other smaller CDNs.

50
00:05:07,020 --> 00:05:12,140
Domain fronting is a great censorship or network bypass technique, but it has some major problems.

51
00:05:12,280 --> 00:05:20,940
For any network censorship bypass to be effective, it has to be really painful to block, and since the largest provider shut it down, it lacks the punch that it once had.

52
00:05:21,120 --> 00:05:30,640
Also, the true destination domain has to be hosted on the same service as the front domain, which limits the potential front sites and imposes costs on anyone wishing to set up fronting.

53
00:05:30,860 --> 00:05:38,380
With the background out of the way, we can talk about TLS 1.3, encrypted server name indication, and what can be called domain hiding.

54
00:05:38,380 --> 00:05:40,980
First, let's talk about TLS 1.3.

55
00:05:40,980 --> 00:05:43,480
It's seeing steady growth on the internet today.

56
00:05:43,480 --> 00:05:48,920
This is data from Quali's SSL Labs, which tests the most popular 150,000 sites.

57
00:05:48,960 --> 00:05:56,260
As you can see, TLS 1.3 is on the rise, and is supported by 31.7% of those sites tested last month.

58
00:05:57,020 --> 00:06:03,360
Cloudflare is seeing 59% of all traffic using TLS 1.3, again, showing a steady increase.

59
00:06:03,360 --> 00:06:11,160
Before we dig into the mechanics of a TLS 1.3 handshake with ESNI, we first have to secure the DNS process.

60
00:06:11,220 --> 00:06:15,900
If a network defender or sensor can see our DNS requests, they can easily be blocked.

61
00:06:16,080 --> 00:06:20,500
Also, secure DNS is a cornerstone of ESNI, which we'll see in a moment.

62
00:06:20,620 --> 00:06:24,520
The first solution is to simply wrap a DNS query in a TLS connection.

63
00:06:24,580 --> 00:06:32,660
To make things even more difficult for network defenders or sensors, DNS via HTTPS is a newer standard that is seeing more widespread adoption.

64
00:06:33,580 --> 00:06:43,600
Once we have an encrypted way to perform DNS queries, we can use this to retrieve a public key for a domain which we can use to encrypt the server name in the client hello packet.

65
00:06:43,620 --> 00:06:54,020
In this screenshot from Wireshark, you can see the encrypted underscore server underscore name TLS extension, and the encrypted SNI value is just an encrypted hex string.

66
00:06:54,020 --> 00:06:59,760
No longer can defenders use the server name value to block traffic based on a plain text value.

67
00:06:59,760 --> 00:07:07,920
A TLS 1.3 connection with ESNI relies on having the server's public key in order to encrypt the server name in the client hello.

68
00:07:07,920 --> 00:07:15,080
And the current standard for how to get this key is to use DNS and query the underscore ESNI text record for that domain.

69
00:07:15,480 --> 00:07:23,160
This request returns a Base64 encoded public key the client can use to derive a session key that encrypts the server name.

70
00:07:23,420 --> 00:07:27,880
Providers that support this should rotate this key to provide some forward secrecy.

71
00:07:27,880 --> 00:07:37,100
Cloudflare, for example, rotates this key every hour, but allows a few hours of buffer in case a client has a slightly stale key from a previous request or from caching.

72
00:07:38,000 --> 00:07:46,820
Armed with the server's ESNI public key, the client derives a session key and uses that to encrypt the server name and sends the client hello.

73
00:07:47,220 --> 00:07:56,060
The server has the corresponding private key and can derive the same session key, which it uses to decrypt the server name and performs the same steps as standard TLS.

74
00:07:56,060 --> 00:08:02,880
This session key generation ensures that the key is tied to the session that generated it and prevents replay attacks.

75
00:08:03,440 --> 00:08:10,700
Another difference between TLS 1.3 and previous versions is that the certificate returned by the server is encrypted.

76
00:08:10,880 --> 00:08:24,720
This is possible because in TLS 1.3, the client hello includes a list of supported ciphers, and the client also makes a guess as to which key agreement algorithm the server is going to choose, and it preemptively sends a keyshare for that algorithm.

77
00:08:24,720 --> 00:08:29,420
So all the plain text portions of HTTPS we saw before have been mitigated.

78
00:08:29,420 --> 00:08:34,020
However, there are still weak spots where sensors or network defenders could stop the connection.

79
00:08:34,680 --> 00:08:43,840
Even if the DNS request itself is encrypted inside either TLS or HTTPS, the data returned by a resolver may be poisoned.

80
00:08:43,920 --> 00:08:53,120
Imagine a scenario where a country or enterprise returns their own key for all underscore ESNI requests, no matter the domain, to enable man-in-the-middle.

81
00:08:53,120 --> 00:08:58,540
DNSSEC solves this problem by signing the record set and allowing the chain of trust to be checked.

82
00:08:58,660 --> 00:09:02,980
The details of DNSSEC are outside the scope of this talk, but I encourage you to check it out.

83
00:09:03,500 --> 00:09:06,580
But what if encrypted DNS is completely blocked?

84
00:09:06,580 --> 00:09:12,220
The solution here is to bootstrap the connection by preloading the current ESNI keys by some other means.

85
00:09:12,280 --> 00:09:18,180
Maybe encrypted DNS is blocked, but what if I have a script that updates a GitHub gist with the current keys every hour?

86
00:09:18,180 --> 00:09:24,380
Or, you could just bake the keys directly into your binary if you know it's going to be executed in the next hour or two.

87
00:09:24,960 --> 00:09:30,700
Okay, let's say we get a valid ESNI key, but the IP address of the server we want to connect to is blocked.

88
00:09:30,920 --> 00:09:34,500
The IP may be shared with other sites, but it doesn't have to be.

89
00:09:34,700 --> 00:09:39,560
Is there a way to route from arbitrary domains and IPs to our true destination domain and IP?

90
00:09:39,720 --> 00:09:42,500
The answer is no, but let's see how close we can get.

91
00:09:43,380 --> 00:09:53,660
So far, I haven't introduced anything new, but we have covered a lot of ground and identified potential issues and solutions around domain hiding with TLS 1.3 and ESNI.

92
00:09:53,660 --> 00:09:57,840
Let's tackle that last point and get around IP or domain blocks.

93
00:09:58,740 --> 00:10:02,560
We'll do this by leveraging the world's largest CDN.

94
00:10:02,800 --> 00:10:06,640
Cloudflare was founded in 2009 and has had explosive growth.

95
00:10:06,640 --> 00:10:15,540
It's now the world's largest CDN, with the most internet exchange points, as well as being an authoritative DNS server hosting over 26 million domains.

96
00:10:15,540 --> 00:10:24,760
Cloudflare is on the forefront of internet technology and supports TLS 1.3, ESNI, WebSockets, QUIC, and DNSSEC.

97
00:10:24,920 --> 00:10:32,280
This new technique, which I'm calling domain hiding, accomplishes the same goals as domain fronting, but uses different technologies.

98
00:10:32,280 --> 00:10:43,980
A TLS 1.3 connection with an ESNI of the true destination is made to any Cloudflare IP, and the underlying HTTP request also has a host header of the true destination.

99
00:10:44,120 --> 00:10:50,840
This enables any Cloudflare-owned IP to act as a front for any domain hosted by Cloudflare DNS.

100
00:10:50,920 --> 00:10:56,060
This technique was first shown to be possible by Robin Wood, so props to him for the discovery.

101
00:10:56,980 --> 00:11:10,040
Today, I'm releasing Noctilucent, a project that enables ESNI in Go's crypto-slash-TLS library, as well as exposes two additional configuration options, ESNI Server Name and Preserve SNI.

102
00:11:10,040 --> 00:11:16,560
It can be used by any project currently using the standard crypto-slash-TLS library, as it's backwards compatible.

103
00:11:16,560 --> 00:11:28,420
It comes with a demo client application that allows you to control just about every part of the TLS connection, as well as the HTTP GET request, so you can play with what's possible using this technique.

104
00:11:28,420 --> 00:11:36,020
It also supports WebSockets, and through the magic of Go, it compiles cross-platform to a single binary with no dependencies.

105
00:11:37,180 --> 00:11:41,820
Here's what a standard TLS 1.3 connection looks like with Noctilucent.

106
00:11:41,820 --> 00:11:49,880
We've specified the TLS host in red, the server name indication in yellow, and the HTTP host header in green.

107
00:11:49,880 --> 00:11:51,860
The output shows what we expect.

108
00:11:51,860 --> 00:12:04,060
There's no ESNI, the SNI is set to cloudflare.com, we're connecting to cloudflare.com on port 443, and our GET request to cloudflare.com returns the expected 301 response.

109
00:12:04,060 --> 00:12:05,700
Nothing fancy yet.

110
00:12:06,720 --> 00:12:10,480
Let's enable ESNI and set the ESNI server name.

111
00:12:10,480 --> 00:12:22,180
Everything is the same as the normal TLS 1.3 connection we looked at on the last slide, except this time, the ESNI value is set to cloudflare.com, and there is no standard SNI.

112
00:12:22,180 --> 00:12:25,520
Progress, but no fronting or domain hiding quite yet.

113
00:12:25,520 --> 00:12:39,240
Now I'll change the ESNI server name and HTTP host header to a domain we control, in this case, DEFCON28.hackthis.computer, but leave the TLS host as cloudflare.com.

114
00:12:39,240 --> 00:12:49,760
Notice that this connection is still going to cloudflare.com, but the HTTP request successfully goes to DEFCON28.hackthis.computer, which returns hello DEFCON.

115
00:12:49,760 --> 00:12:55,620
To anyone between this host and cloudflare, it appears to be a TLS connection to cloudflare.com.

116
00:12:55,620 --> 00:12:58,140
We've successfully achieved domain hiding.

117
00:12:58,980 --> 00:13:00,980
Now the fun begins.

118
00:13:00,980 --> 00:13:07,680
Here we send an ESNI and standard SNI, where the standard SNI is cloudflare.com.

119
00:13:07,680 --> 00:13:14,040
Because cloudflare ignores the standard SNI when an encrypted SNI is used, we can set it to anything.

120
00:13:14,040 --> 00:13:29,820
The HTTP connection works as before, and this time, anyone between this host and cloudflare, looking at SNI data, thinks this connection is going to cloudflare.com, in addition to the underlying TLS connection actually connecting to a cloudflare.com IP.

121
00:13:30,220 --> 00:13:38,640
What I've demonstrated is the ability to arbitrarily front or hide any IP behind a domain that is using DNS provided by Cloudflare.

122
00:13:38,640 --> 00:13:43,880
The true destination IP doesn't have to be a Cloudflare worker or other Cloudflare service.

123
00:13:43,880 --> 00:13:48,740
In fact, all the examples today are IPs hosted by DigitalOcean.

124
00:13:48,920 --> 00:13:54,440
Best of all, the requirements to sign up for Cloudflare DNS are minimal, and it's free.

125
00:13:55,960 --> 00:13:58,340
What domains can we use for this hiding?

126
00:13:58,340 --> 00:13:59,780
It turns out a lot.

127
00:13:59,780 --> 00:14:06,220
Over 21% of the top 100,000 sites are behind Cloudflare and allow this to work.

128
00:14:06,220 --> 00:14:15,880
Some notable examples are shown below, including 3 of 11 domains which are both on the Palo Alto decryption whitelist and behind Cloudflare.

129
00:14:15,880 --> 00:14:18,400
I'll demonstrate why this is important in a minute.

130
00:14:19,360 --> 00:14:29,340
There's a little bit of everything on this list, from security-related sites, banks, sports, higher education, streaming services, and even government sites.

131
00:14:29,340 --> 00:14:30,460
And porn.

132
00:14:30,460 --> 00:14:32,100
So much porn.

133
00:14:32,680 --> 00:14:37,620
Here's a good example of how the variety of available domains to hide behind can be useful.

134
00:14:37,840 --> 00:14:44,920
This is a request to www.okta.com, a single sign-on and identity provider used by many enterprises.

135
00:14:44,940 --> 00:14:48,300
But the actual request is going to a server I control.

136
00:14:48,500 --> 00:14:56,300
To a sensor or network defender, this looks like standard Okta traffic, with the exception of the ESNI extension being used.

137
00:14:56,300 --> 00:15:02,080
The logical next step is to try out Noctilucent with some web filtering and see how it fares.

138
00:15:02,080 --> 00:15:08,780
It turns out most web filtering is done by inspecting the ESNI and making a filtering decision based on that.

139
00:15:08,780 --> 00:15:16,580
Here I set up the untangle firewall with federal government settings and explicitly blocked defcon28.hacktest.computer.

140
00:15:17,160 --> 00:15:22,360
On a host behind the untangle firewall, I made this request, which succeeded.

141
00:15:22,480 --> 00:15:29,720
I set the unencrypted SNI to bypass-untangle.com to see if it appears in the logs anywhere.

142
00:15:31,260 --> 00:15:40,320
After repeating the request 30 times or so, back on the firewall, we can see that the top domains include the decoy domain that we sent with the request.

143
00:15:41,140 --> 00:15:47,520
Simple SNI filters were easy to defeat, but what about HTTPS decrypting firewalls?

144
00:15:47,540 --> 00:15:56,740
These are often seen in enterprise environments and work by breaking and then re-encrypting HTTPS using a root certificate placed on endpoints behind the firewall.

145
00:15:56,740 --> 00:16:03,060
You can think of it as a corporate man-in-the-middle that allows the network defenders to inspect full, encrypted packet data.

146
00:16:03,260 --> 00:16:13,580
Kazakhstan actually attempted to implement this nationwide for a few weeks back in July of 2019 before major international backlash forced them to abandon what they called a, quote, test.

147
00:16:14,540 --> 00:16:23,460
I headed to AWS and spun up the latest and greatest in enterprise firewalls, Apollo Alto PAVM version 10.0.0.

148
00:16:23,460 --> 00:16:29,980
Released in June, this version touts the ability to decrypt TLS 1.3 with a two-time speed boost.

149
00:16:30,120 --> 00:16:36,320
Imagine my surprise when it was all set up and the default decryption profile didn't include TLS 1.3.

150
00:16:36,320 --> 00:16:41,140
Instead, it let it pass through while logging an error about a version mismatch.

151
00:16:42,080 --> 00:16:53,160
Once set up with a decryption profile that enabled decryption of TLS 1.3, I pulled the list of default domains that are allowed to bypass decryption for a variety of reasons, such as having a PIN certificate.

152
00:16:53,540 --> 00:16:58,540
Here, the firewall interface is on the left and a Windows VM behind the firewall is on the right.

153
00:16:58,740 --> 00:17:04,920
We can see that decryption is enabled and we have no other decryption rules that would allow traffic to bypass decryption.

154
00:17:04,920 --> 00:17:13,700
Normally, in an enterprise environment, categories like banking or healthcare bypass decryption due to privacy laws, but this setup is as strict as possible.

155
00:17:13,700 --> 00:17:15,520
No explicit exemptions.

156
00:17:15,920 --> 00:17:23,360
On the Windows VM, I'll run a PowerShell script that checks the certificate hashes of major sites against hard-coded known fingerprints.

157
00:17:23,580 --> 00:17:27,780
We see that it fails, indicating that all tested sites are being man-in-the-middled.

158
00:17:27,780 --> 00:17:35,640
Additionally, opening a browser and going to google.com shows a lock icon, but inspecting the certificate shows that it's the firewall certificate.

159
00:17:36,160 --> 00:17:51,280
Switching to the command line and using the Noctilucent test client, I'll set the TLS host to mozilla.org, the host header to defcon28.hackthis.computer, and the unencrypted SNI to mozilla.org, which is on the decryption exemption list.

160
00:17:51,280 --> 00:17:54,280
The SNI is preserved and set to mozilla.org.

161
00:17:54,280 --> 00:18:01,700
A connection is made to mozilla.org on port 443, and we see the hello defcon28 response we expect from the test server.

162
00:18:02,280 --> 00:18:07,400
We'll run an NS lookup of mozilla.org so we can look for the connection in the logs.

163
00:18:11,460 --> 00:18:17,180
Looking at the firewall traffic logs shows a connection to one of the IP addresses for mozilla.org.

164
00:18:21,120 --> 00:18:33,400
Moving to the decryption logs, we see DNS over HTTPS, which is the ESNI key lookup that is performed before each TLS connection, but no traffic to mozilla.org's IP.

165
00:18:36,060 --> 00:18:40,640
To make sure we didn't miss it, I'll add a filter for the two possible IPs.

166
00:18:41,380 --> 00:18:46,080
No results, because mozilla.org is exempted from decryption by default.

167
00:18:46,080 --> 00:18:53,300
We have successfully defeated an HTTPS decrypting firewall by using built-in default exemption and domain hiding.

168
00:18:53,380 --> 00:18:59,880
This bypass is great, but every time we want to send data, we'll add an entry to the traffic log, which could stand out.

169
00:18:59,880 --> 00:19:01,720
What if there was a better way to hide?

170
00:19:01,720 --> 00:19:07,720
This technique works at the TLS level, so any protocol that Cloudflare supports underneath would also work.

171
00:19:07,720 --> 00:19:12,960
We saw that regular HTTP requests work, but create a new connection for each request.

172
00:19:13,300 --> 00:19:21,060
WebSockets are a technology used by many streaming services that enable full duplex communication channels over a single TCP connection.

173
00:19:21,200 --> 00:19:30,900
And we can take this a step further by using one or more of these WebSocket connections to tunnel arbitrary TCP or UDP packets to a proxy beyond the firewall.

174
00:19:32,200 --> 00:19:41,120
Here's the Noctilucent test client sending and receiving via WebSockets to our test server while hiding behind www.okta.com.

175
00:19:41,240 --> 00:19:52,200
WebSockets are the perfect candidate to use as a wrapper for a proxy protocol that allows us to send arbitrary TCP or UDP traffic using a helper program while only making a single TCP connection.

176
00:19:52,200 --> 00:20:06,200
The Noctilucent project includes a fork of the Amazing Cloak project by Andy Wang that enables two new options in the client configuration, ESNI Server Name and Preserve SNI, which operate just like they did on the test client.

177
00:20:06,540 --> 00:20:23,740
Cloak is a universal, pluggable transport that cryptographically obfuscates proxy traffic as legitimate HTTPS traffic, disguises the proxy server as a normal web server, and multiplexes traffic through a fixed amount of TCP connection while providing multi-user usage control.

178
00:20:23,740 --> 00:20:28,260
With the additions from Noctilucent, it can now do domain hiding as well.

179
00:20:29,060 --> 00:20:36,620
Here's the same Palo Alto and Windows VMs from before, with a public IP of 3.14.44.10.

180
00:20:36,740 --> 00:20:42,460
This time, the Windows VM is running a Noctilucent Cloak client and a local Shadowsocks client.

181
00:20:42,460 --> 00:20:48,700
The Cloak client is connecting to a VPS running the standard Cloak server and a Shadowsocks server.

182
00:20:49,020 --> 00:20:57,120
Firefox is configured to use the local Socks proxy provided by the Shadowsocks client running locally, and I can browse the internet as I would normally.

183
00:20:57,580 --> 00:21:07,580
The certificates of websites in this proxy browser are legitimate, as they are not being intercepted by the Palo Alto firewall, and all traffic is appearing to come from the DigitalOcean VPS.

184
00:21:15,550 --> 00:21:17,770
The speed is surprisingly good.

185
00:21:17,830 --> 00:21:23,410
During testing, I was getting speeds up to 100 Mbps using four TCP connections with Cloak.

186
00:21:23,610 --> 00:21:34,850
If we look at the firewall logs, we only see the four connections to Mozilla.org, which are not decrypted, while the client has been browsing freely, with all traffic remaining encrypted and uncensored.

187
00:21:38,010 --> 00:21:45,470
Switching to the decryption logs, we see the DNS over HDPS requests, but no requests to Mozilla.org.

188
00:21:47,270 --> 00:21:52,050
I'll filter for the Mozilla.org IPs, but we'll get no results.

189
00:21:53,410 --> 00:22:02,710
What we see here is arbitrary traffic at high speeds using only four TCP connections to what Palo Alto thinks is Mozilla.org.

190
00:22:04,010 --> 00:22:07,390
Can we use this technique for our pre-existing Red Team tools?

191
00:22:07,390 --> 00:22:09,670
If they support proxying, you bet.

192
00:22:09,670 --> 00:22:14,990
Here's a Cobalt Strike listener with a SOCKS proxy setting for localhost port 9999.

193
00:22:15,470 --> 00:22:22,850
For fun, we'll set Cloak to use bitdefender.com as the front domain, which has the IP address shown here.

194
00:22:24,890 --> 00:22:34,210
After starting the Noctilucent Cloak client and local Shadowsocks server in SOCKS 4 mode, we'll double-click on an innocent-looking EXE.

195
00:22:34,210 --> 00:22:37,750
We get the callback immediately, and it works as expected.

196
00:22:39,310 --> 00:22:43,090
And of course, the first thing you do when you get access is pop a calc.

197
00:22:48,240 --> 00:22:54,960
Switching to the firewall and looking at the decryption logs, we only see entries for cloudflaredns.com.

198
00:22:59,250 --> 00:23:04,710
I'll paste in a filter for the two possible IPs for bitdefender.com, and we get no results.

199
00:23:08,600 --> 00:23:17,080
Looking at the traffic logs, we see four connections to the IP of bitdefender.com, as well as lots of DNS over HTTPS.

200
00:23:21,070 --> 00:23:26,530
Our Cobalt Strike beacon is still functioning as normal, and any network defender is none the wiser.

201
00:23:27,270 --> 00:23:36,930
Packaging up Cobalt Strike, Cloak, and Shadowsocks into a clean package, possibly communicating over namepipes instead of sockets, is an exercise left up to the viewer.

202
00:23:38,350 --> 00:23:46,650
Red teamers are no doubt wondering how much work it would be to integrate this with their existing tooling, and I can say if you're using Go, it's almost no work at all.

203
00:23:46,650 --> 00:23:49,430
Here's a new C2 framework called DemoC2.

204
00:23:49,430 --> 00:23:54,290
It was released just five days ago and is built using Go and Vue.js.

205
00:23:56,970 --> 00:24:03,190
I've set up an instance here with an HTTPS listener with a public IP on port 443.

206
00:24:03,670 --> 00:24:08,690
Going into the listener settings, we can view the source code for an HTTPS agent.

207
00:24:12,760 --> 00:24:17,820
It has the C2 host and port in the code, and the agent will connect directly to this host.

208
00:24:18,720 --> 00:24:23,140
Switching to an editor, I've made a few modifications to get this to work with Noctilucent.

209
00:24:23,280 --> 00:24:34,180
First, I set the host to bitdefender.com, and I've added two new variables, frontDomain and actualDomain, which are set to bitdefender.com and demos.hackess.computer.

210
00:24:35,100 --> 00:24:39,260
The actual code changes needed to get everything working is just 30 lines.

211
00:24:39,320 --> 00:24:43,540
First, we have to query for the ESNI keys and parse them.

212
00:24:43,540 --> 00:24:49,000
If we wanted to, we could bake in the current keys here to avoid the network traffic related to querying for them.

213
00:24:49,320 --> 00:24:53,920
Then I define a TLS config and set the minimum value to TLS 1.3.

214
00:24:54,120 --> 00:25:01,980
I'll define the two new options from Noctilucent, ESNI server name is set to the actual domain, and preserve SNI is set to true.

215
00:25:03,560 --> 00:25:09,600
I define an HTTP client by hand, so that I can tell it to use this TLS config when dialing the host.

216
00:25:10,620 --> 00:25:17,860
Besides that, the only change to the agent is having the HTTP client use the actual host when it makes its POST requests.

217
00:25:20,580 --> 00:25:27,180
On our Windows VM, I'll set a filter in Wireshark for the IP of Bitdefender, so we can inspect the client hello.

218
00:25:27,460 --> 00:25:31,360
I'll double-click this new innocent exe and switch back to Wireshark.

219
00:25:33,620 --> 00:25:41,760
In the client hello, we see the unencrypted server name is Bitdefender.com, and the encrypted server name is just a bunch of hex.

220
00:25:48,480 --> 00:25:51,400
Back on the C2 server, we have an agent calling back.

221
00:25:53,080 --> 00:25:55,020
Notice the external IP.

222
00:25:55,100 --> 00:26:00,880
When using this technique, the reported external IP will be from one of Cloudflare's ranges.

223
00:26:10,410 --> 00:26:13,950
Just like with CobaltStrike, everything works as normal.

224
00:26:15,450 --> 00:26:20,290
You can run WhoAmI, pop a calc, and check the public IP.

225
00:26:28,240 --> 00:26:33,460
Besides the external IP appearing to be from a Cloudflare range, the agent functions as expected.

226
00:26:43,520 --> 00:26:48,540
Back on the Windows VM, I'll search for traffic to the C2's server IP.

227
00:26:49,700 --> 00:26:51,140
No packets.

228
00:26:51,140 --> 00:26:57,180
There's no trickery here, but just a small modification, this Go-based agent is using domain hiding.

229
00:26:57,860 --> 00:27:03,700
So what is the blue team to do, given that we've just seemingly bypassed the most sophisticated network defenses?

230
00:27:03,760 --> 00:27:07,360
The easiest thing to do would be to block TLS 1.3.

231
00:27:07,520 --> 00:27:15,340
But as that's likely anywhere from 25% to 50% of the TLS traffic leaving your network, it might make a few users unhappy.

232
00:27:15,560 --> 00:27:22,680
You could block Cloudflare, but dropping 21% of the most popular 100,000 domains probably is not a good idea either.

233
00:27:22,980 --> 00:27:26,160
What about just targeting client hellos with ESNIs?

234
00:27:26,160 --> 00:27:31,380
It's possible, and one vendor has had the ability to do that for 6 months now.

235
00:27:31,380 --> 00:27:35,700
However, since you can't determine the destination site, it's all or none blocking.

236
00:27:35,700 --> 00:27:40,820
There's no way to only block selective domains if you block TLS connections that have an ESNI.

237
00:27:40,820 --> 00:27:46,340
I was surprised by the lack of support for this across major vendors, but perhaps it's just not an issue yet.

238
00:27:46,980 --> 00:27:55,940
The most sensible way to detect this specific technique is to alert on packets that contain both a server name and an encrypted server name TLS extension.

239
00:27:55,940 --> 00:28:07,020
There are no default helpers for TLS and SNORT besides SSL underscore state, which can help you get the client hello packet, and TLS dot version, which can get you TLS 1.3.

240
00:28:07,020 --> 00:28:11,400
So you'll have to get creative with the content parser and look for the extension types.

241
00:28:12,020 --> 00:28:22,600
SecureKata has a bit more horsepower than SNORT, but only has a selector for TLS dot SNI and not TLS dot ESNI, so custom rules will be required here as well.

242
00:28:23,400 --> 00:28:28,040
It's going to come down to tried and true techniques that work regardless of traffic content.

243
00:28:28,080 --> 00:28:36,600
Things like beaconing detection and anomaly detection can help show clients that all of a sudden start sending regular traffic to destinations that perhaps don't make sense.

244
00:28:36,800 --> 00:28:39,880
There are even some open source tools that do just this.

245
00:28:40,180 --> 00:28:45,400
A network defender should be asking if a traffic pattern to a given endpoint makes sense in context.

246
00:28:45,520 --> 00:28:50,480
Perhaps that Windows machine and accounting shouldn't be sending gigs of data to your single sign-on provider.

247
00:28:52,380 --> 00:28:57,140
TLS fingerprinting with JAW3 and JAW3S is another technique that could be effective.

248
00:28:57,340 --> 00:29:05,700
JAW3 looks at the specific options used in a client hello that can sometimes uniquely identify a client, and JAW3S does the same for servers.

249
00:29:06,180 --> 00:29:15,320
This will work until attackers tune their tools to be identical to browsers and common web servers, and work is already underway to do this in projects like Cloak and uTLS.

250
00:29:15,320 --> 00:29:25,360
As always, a layered approach that uses strong, instrumented and managed endpoint detection and response, and other techniques, is the best way to prevent compromise.

251
00:29:25,440 --> 00:29:30,500
Or as Microsoft so helpfully says, just don't let untrusted code ever run on your server.

252
00:29:31,520 --> 00:29:42,280
To wrap up today, I've discussed the basics of HTTP, HTTPS, domain fronting, and a new form of censorship-resistant communication, domain hiding.

253
00:29:42,280 --> 00:29:50,120
This technique is usable today with Noctilucent, either as a drop-in replacement for the Go TLS library, or as a proxy.

254
00:29:50,300 --> 00:29:57,340
The ESNI standard is still very much a draft, and this technique will continue to evolve as the changes in adoption grows.

255
00:29:57,340 --> 00:30:00,340
In fact, it's not even called ESNI anymore.

256
00:30:00,620 --> 00:30:05,100
Just as with standard domain fronting, this relies on the CDN permitting it.

257
00:30:05,100 --> 00:30:11,340
However, in this case, besides preserving the standard SNI header, we aren't really doing anything terribly out of spec.

258
00:30:11,340 --> 00:30:16,040
I look forward to seeing the snort and securicada rules to detect and block this technique.

259
00:30:16,100 --> 00:30:22,440
And like every good red teamer, I'll push my customers with the latest techniques, hopefully before the real adversary can.

260
00:30:22,880 --> 00:30:30,580
I want to give special thanks to Robin Wood, who first showed this technique was possible with a rough POC using a fork of OpenSSL.

261
00:30:30,760 --> 00:30:37,740
Andy Wang, the creator of Cloak, and Nick Sullivan from Cloudflare, who helped get me data on TLS 1.3 adoption.

262
00:30:38,640 --> 00:30:40,660
Noctilucent is available now on GitHub.

263
00:30:40,660 --> 00:30:47,580
If you have any questions or comments, feel free to hit me up on Twitter, where my personal handle is at badsectorlabs.

264
00:30:47,580 --> 00:30:48,940
Thanks for watching.