All right. Welcome back, everybody. I know it's getting a little late in the day, so thank you
all for turning out. I'd like to introduce our next speaker, Lee Brotherston. He'll be talking
about, as the slide deck suggests, Blue Team TLS Hugs. Let's all give him a warm welcome.
Hello. Okay. Hi. So I'm going to take the standard format of state a problem, say you're all
doing it wrong, and what we should do to fix it. A quick sort of upfront disclaimer. I mention
HTTPS a lot. It's not a talk about HTTPS, but it's the TLS implementation that most people know,
so it's an easy example. And, you know, it's everywhere. It's pretty well understood. And
TLS is being used everywhere. So a super quick primer, just in case anyone doesn't know what TLS
does and not SSL. It's not a cryptographic algorithm. It's a protocol for determining
what kind of cryptography you're using. And it's a protocol for determining what kind of
crypto you're going to use between two endpoints that don't know what each other support. So it's
a negotiation protocol for encryption rather than the encryption itself, although the encryption
naturally follows in the payload afterwards. So here's how it works, or kind of how it works.
This is really more how it works. So the client sends a client a hello packet, and that packet
contains a bunch of information. That's things like what cryptographic protocols it supports,
hashing algorithms it supports, what elliptic curves it has, if any. And then the server
replies with, from that list, I also accept these things, and they can determine what they're going
to do. They then exchange keying data, and then encrypted data, and then renegotiate, and that
carries on in a cycle until they're done. So that said, the current state of the art, we should
look at that to sort of see what's wrong and where we can go and do other things in the future.
Three things that defense people can do with TLS. You can ignore it, you can break it, so that's
man in the middle and that sort of thing, or you can embrace it. In the real world, we don't bother
with embracing it, and we either ignore or break it. The closest to doing it right is actually
the domestic situation in a lot of cases, not in a corporate environment. So I'll just explain
a little about what I mean by each of these. So ignoring TLS. If you've ever run something
like an IDS or something like that, you're going to have a lot of trouble with it. So
if you run something like that, and you see TLS encrypted traffic on the network, it's just
garbled up encrypted mess. I mean, that's the point. You don't see the payload. If you look at
the snort community rules as they stood at the time, this is the clear text and TLS-enabled
versions of each of the protocols, and that's the number of rules. You can see there's significantly
more for the clear text, because if you've got your IDS alerting on any request that's like a
dot-dot-slash-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot-dot
dot-dot-slash-dot-dot-dot-slash, that will alert fine in clear text, but not in TLS. The
TLS rules are just heartbleed and things like that. So you're largely blind to a whole sea of
exploits, and so effectively it's just ignored. So that leads to this. Does anyone know what this
is? I will give you a further hint. It is the universal firewall bypass port.
Because if we don't understand crypto, we just let it through, right? Because you've seen this,
I'm guessing. You can get out of a lot of corporate environments just by SSH-ing on port 443,
because what are you going to do with port 443? Well, we'll come to what you can do, but right
now we're talking it's often ignored and just left open. You can also often connect even when
they have a proxy and just use a connect command on port 443, and it will let you straight through.
So what do we do in the situation where we're not looking at TLS on the network?
Most of the time that comes down to using endpoint security solutions. So antivirus,
anti-malware, various memory protection techniques, and that sort of thing,
to capture all the bad on the endpoint as it's coming in, rather than looking at it on the
network. We also look at blacklists. So you've got a list of blacklisted IP
addresses, for example. You don't know what the data is going to them, but you know they're part
of a command and control network or something. So you know that there's bad data going there,
you avoid it, run away, and hope that that was a good blacklist. But there are problems with
blacklists too. Very often they're not up to date or are lacking information, but even if they have
the information, there's a lag between it coming from the real world to the blacklist, from the
blacklist to your environment, and your environment to you actioning it, by which time half the
addresses are out of date. So some of the time we break TLS. So I'm talking things like where you
have these proxies that run their own CA and they man in the middle of the traffic. To explain how
it works, you have a client, a proxy, and the web server you're connecting to, or any other
server, sorry. Your client makes a TLS handshake with the proxy, and then files a request, so
sends the URL that it would like. The proxy then does a TLS handshake with the server. The request
gets forwarded, goes to the server and it gets a response, which then gets forwarded, and sent back
to the client. The problem is that this stuff is in the clear. That's because the proxy has its own
SSL certificate, and what this means is that you have the actual CA on the right that uses all the
proper signatures and all the magic that's associated, and you've got your shonky, my first
work CA happening on the left-hand side.
And this brings a whole ton of issues other than it's a really horrible thing to have to try and manage.
So to explain how that works, it breaks the CA model in a bunch of ways, because although the CA model's
kind of broken to start with, this breaks it even more because it is not one of the certificate
authorities trusted by browsers, for example, and that causes a bunch of issues. I'm just going to go
through a few and then we'll get to it.
And we can get on to what you can actually do to fix it.
So there's simple compromised appliance. If that appliance in the middle got hacked, data's going through in the
clear. It could be manipulated. It can be sniffed and logged and all sorts of stuff. So think of online
banking or logging in to look at medical records or something like that. Just compromising the device
that has the cert is enough. If the key that's on there gets compromised, that's bad too, because if the
key is lost or goes to someone else, they will be able to take that key and impersonate any website to your
devices, because when you set up that sort of system, you set the CA as trusted in all the endpoint devices.
And also, key management is hard. It's really, really hard to get right. If you want the closest I can get to an
empirical study on key management is hard, if you look at ISO 27002, and I apologize for that, but if you look at the
compliance framework type stuff, but key management alone has a higher word count than the whole of the rest of the
cryptographic control section. And in the cryptographic control section, six of those are words key management
referring to go look at the key management section. There's also poor certificate validation. So when you connect
directly to something with a web browser or any other TLS-enabled client, it does a bunch of checks. It does things like
does the host name on the certificate match what I was expecting?
It checks the certificate chains, it looks for revoked certs, it looks to see if it's a trusted CA, all these things.
None of these you see when you man in the middle, because all you see is the one valid cert for everything that you just
installed on that device. That means that you are trusting that device to do all that validation for you. If any of you
have ever run those devices, you'll know that you can't trust them to do that. If nothing else, their list of trusted
CAs is horrifically out of date, typically.
And they fail a bunch of the other validation checks.
So let's talk about that trusted CA list. If you look in your browsers or your OS, in fact, as many browsers use the default OS
one too, you will see that there's the list of trusted CAs. So that is authorities who are able to sign certificates as
valid. But these things get revoked. You trust what the appliance trusts, right? But a few months ago, a few months ago,
WoSign got pulled by a bunch of browsers and pulled pretty quickly because of some of the practices they were employing.
But if you are not updating that appliance in real time, you can go to one of those sites that the certificates have been pulled from,
and your browser will be none the wiser. It will think they're all valid because it's just getting that cert from the proxy.
Oh, yeah, I don't know what that is. Anyway, there's also certificate unpinning.
If anyone knows what certificate pinning is, this is where your client is pre-configured to say, when I connect to such and such a service,
I expect to see this certificate or this certificate, which has been signed by this CA, or some combination.
It matches this fingerprint, or it's got this intermediate one. But whatever, you can pin to one degree or another what you're
expecting to see. This gets around the issue of things like, say, a rogue government with not many scruples somewhere,
convinces a CA to issue a certificate, a valid certificate, for a domain that's really hosted with another CA.
And then they can use that to man in the middle without flagging up warnings because it comes from a valid CA.
When you install a local CA, as is the case when you're having the man-in-the-middle proxy, most browsers disable pinning.
This is because if you're running one of these proxies, people like Google use pinning, and they don't want Gmail to just
fail everywhere on your corporate network. Disabling pinning is bad in a number of ways.
And finally, there's the malicious insider, just the member of staff who now, because they have access to the appliance or the
certificate or both, has access to do all sorts of bad on your network and to your devices.
So that is my brief TLS, we're doing it wrong. So I'm going to talk about trying to embrace it a little more and do the right thing.
So let's embrace it. I'm going to set some goals. So if we look at all the different devices that intercept the traffic and do
things to your traffic, they generally do this list. So it's things like content filtering, logging that content filtering, checking
certificates, malware detection, etc., etc. In my opinion, which is largely unscientific, but I'm going to give it anyway, go team, this is
the stuff they approximately do right, this is the stuff I don't think they do particularly well, this is the stuff they sort of do well and not so
well, and this is what we actually want. Not many people seem to want cryptographic checks, and by that I mean checking which algorithms
are used and not used. They're pretty happy to trust their browsers to do that. And granular content filtering, I have an argument about
this, that people don't really need that, and by granular I mean full URL content filtering instead of hostname.
For example, most people don't want to block one category of porn and not another category of porn. They just want to block Pornhub or not block
Pornhub. That's pretty much where they're at. So feel free to disagree, but that's what I'm saying. Anyway, so I'm going to try and get those
goals sorted. So I'm a believer that perfection is the enemy of good enough. People boil the ocean trying to get RFP done for 100% complete
solution. So I'm trying to offer that.
But a gets you a good chunk of the way there, but quickly and easily and without hassle. We have many cases in the real world where sort of half-assed
things do work and get the job done. So we'll work with that. So I had a thought. So there's first, there's packet, there's a bunch of packet
sniffing we can do. I did some studying a little while ago on TLS fingerprinting, and you can use that to determine all sorts of bad stuff. The long and the
short of that, there's another talk on that if you want to know more about that. So I'm going to try and get those goals sorted.
I'm going to learn about the in-depth stuff. But TLS fingerprinting is basically you can go through and monitor the TLS handshake and determine what client it is. That means that you can treat this like an IDS and say, oh, great, this host is connecting on Google Drive. This one's on Skype. This one's on Tor. This is using Chrome. This is using IE. If anyone uses a corporate environment, you'll know that seeing Chrome and IE is fine. Seeing Tor and things is not. And depending on your environment, Google Drive or Dropbox or something could be potentially
bad.
Someone losing data. But what I'm getting at really is that it gives you the opportunity to understand the encrypted connections in your environment to a good way. A good degree, sorry. You can also use it to spot bad things in general. So you can spot rogue clients. You're in a all Mac environment and you see IE. That's probably someone plugging their own laptop in. You can also spot malware. For example,
Drydex used TLS. You can monitor the TLS connections that it makes. And you can see that
not only which clients you have are infected, you can map out the command and control network
real time as they talk back to it. This is Cisco saying the same thing because I can't push the
slide advance button in a sensible order. But this is Cisco's blog that was saying,
hey, we can't see stuff because malware uses TLS. But they do use different TLS fingerprints to a lot
apps so you can often spot malware using it you can also do semi-automated fingerprinting so
it's so easy to make fingerprints that you can automate creating a fingerprint for anything you
see but have not seen before and then that can be reused when you see that fingerprint coming again
so if it's seen once and associated with something bad and then you see it come back again you can
often tell that it's the same something bad prime examples are botnets have their own metasploit has
its own so you don't have to know what it is in advance if you tie it back to a previous incident
you can still determine that it's bad and you can do pseudo anomaly based detection as well
which is largely the same thing but it's based on on past behaviors and finally you've got
fingerprint canaries so if you look think of being in the
situation of being a generic ride-sharing app a bankrupting blockbuster as a service or one of the
closed API social media services they all create their own API and their own client and they don't
open up to third parties if you were to one of these companies you could monitor all connections
to your API endpoint to see if it matches the fingerprint of your client if it doesn't match
your official client you just drop it on the floor and you've then removed curl burp suite and all
the other things that you need to do to make sure that it matches the fingerprint of your client if it doesn't match your official client you just drop it on the floor and you've then removed curl burp suite and all the other things
that you potentially don't want in touching your API endpoints unless
someone goes to the hassle of actually masquerading the TLS stack of that app
to give you an example of a similar technique and Mozilla released this
blog about checking for man-in-the-middle which is exactly what
I'm talking about here based on the connections coming in from the browsers
you can also sort of do incident response and attribution although
attribution is hard and that's like a whole area
shouldn't venture into but if you have an exploit that comes into a network
over TLS the tools that are used to develop it if they have their own
signatures you can reuse watch that signature propagating across the network
and you can watch the progress of an incident as people pivot horizontally
across the network and I'm going to just quickly demo what this actually means
before I stop talking about fingerprinting and drive you all insane
on that one topic just for reference though this is the best German word in
whatever it means it doesn't look when someone's what it doesn't work when
someone's watching so I've so I've I've recorded it so this is just a tool I
wrote running in the terminal and this is a standard Windows VM it's from a few
months ago so I apologize the versions are a little old but you'll you'll get
the idea so you go visit some site and it's just at a packet sniffer at the
end and it's picked up that you've got Firefox and signal because why not
connect into a bunch of things and then at the same time you can go I think it's
Google Drive or Dropbox I used my memory is terrible in this anyway yeah
so these all use either API's or their HTTPS under the hood or whatever so as
the file syncs up you'll see that it Dropbox comes up there as having
connected and finally the thing that most people like to see on their network
is Tor so I'll just leave that there.
Let it do the Tor thing so this is Tor browser which is just a bundled Tor and
Firefox all-in-one tool and as that connects to the Tor network without
doing any web browsing it starts screaming all over the network I'm Tor
if you were being a bit more active about that you could use that for
alerting and you could pick up on malicious network traffic anyway that's
that bit so oh thank you so enough on fingerprinting their server
responses too so the server hello packet also has a bunch of information
that's really useful for you first of all it lets you know what has been
negotiated so you can tell if they're using old TLS versions weak cipher
suites and that sort of thing I'm running a long time so I'm gonna go
super fast you can harvest all the certificates I'm going to refer back to
something I mentioned earlier you can log all the certificates that are
associated with the website at the time so in the future you can go back in time
and see what's the
you saw why is this useful well this again will sign and Google drop China's
route cert and I think other people may have done to this let's oh sorry and
super fish who had its horrible difficult situation this is super easy
to do you can use a plug-in in bro and that will do logging of certificates and
then when someone says hey this certificate
authority
has been acting bad you can go back retrospectively see which sites use
that cert and you know if your clients could have been compromised in the past
finally I'm going to talk about inline TLS shenanigans which is actually my new
thing so you can mess with the handshake it's in the clear it's not
cryptographically signed so you can modify what's inside it remember this I
suggest this know what I suggest is an alternative is a proxy that works at the
socket layer and passes data backwards and forwards
so it doesn't mess with the encryption, it doesn't re-sign things and fall foul of all those CA issues I mentioned.
It can still see the server hello stuff, so it knows what's been negotiated,
it knows what cryptographic protocol is being used, and it can see that stuff.
Most importantly, it can fingerprint stuff, and because anything vaguely current uses SNI,
you can pull the host name in the clear that it's connecting to.
That gives you enough for coarse-grained content filtering, and you can also tie this together with clients.
So you can say that Skype can access Skype servers, but Skype servers alone, i.e. can go to all sorts of other servers, whatever.
This is also safe because it's subtractive.
I can't put in a crappy cipher suite and trick browsers into degrading themselves to that terrible cipher suite.
They never offered it in the first place, they won't support it.
I can, however, pull crappy cipher suites out and force them to only negotiate the stronger cipher suites and other aspects.
And then you've got...
This is what the compliance is.
Most of the compliance standards, including PCI, will say things like,
you can't use old versions of SSL and TLS.
Well, if you're monitoring that handshake and have a proxy in the middle,
you can look for old versions of TLS, drop them on the floor,
and ensure you maintain compliance on your network, irrespective of what the applications are doing.
So there's a remaining problem.
If you're not manning the middling, it's super hard to put that little pop-up up that goes,
you've been blocked, you were a bad...
person.
So, I tried a few things.
I put a bad certificate in.
Shockingly, Chrome and Firefox were super not happy with that plan.
There is a response of server hello done, which is just,
I'm not sent a certificate, but I'm done.
It also doesn't sound super good to the user going,
can't provide a secure connection, and...
This is Chrome, sorry.
And in Firefox, saying secure connection failed also sounds bad.
And if it's the subcomponent of another...
page you've allowed through, it throws those SSL errors about mixed content.
So, it's not very good.
But I came up with a solution which is super crude, but works.
So, I will just demonstrate how this works really quickly,
and then I will show you how that looks in browsers for your reference.
So, demo again.
So, this is my tool at the top.
This is a proxy.
It's just written in Go.
I'm going to curl.
I've set it...
There's a proxy, but it can be a transparent proxy, too.
This is just for ease of demo purposes.
So, if I try and curl the CryptoVillage website,
it logs that I used curl, but I went to CryptoVillage.
It's not on the block list, and it went through at the bottom.
If I curl something that I'm less interested in,
in this case, one of our favorite ad networks,
what you'll find is that the server-aborted SSL handshake,
that's actually not...
Not as bad as it sounds.
That's literally just dropped the TCP connection on the floor,
and it got the drop on the proxy.
This looks crude in curl, because I'm just kind of showing you,
and it has that nasty error,
but it actually has the valid cert come back every time,
and the browser gets to do all their thing.
But more importantly, it just looks like the server is down to the browser when you go direct,
and when it's a subcomponent on a page, like an image or a script,
it just looks like the messing little square
in all the...
in all the browsers.
You don't get any SSL errors.
You don't get big barfs or anything like that.
So, I'm going to go as fast as I can.
How do we do?
I say we did course content filtering fine,
and application detection we got,
course logging we got,
granular logging not really.
I'll come back to that, but I probably won't have time, so I won't.
Certificate checks we did.
Malware detection we sort of did.
We didn't do it in the AV sense,
but you can have TLS signatures on malware,
so you can pick up a certain amount of it.
In-page exploits, totally didn't do that,
but I said I was only doing 90%.
DLP, half did.
We don't detect documents going out,
but we can detect things like Google Drive and Dropbox
and things that may hint at data exfiltration,
and data exfiltration falls under that banner.
One more thing.
Do I have time?
Oh, I'm going to try and do it quick.
Course grain login.
This is an encrypted payload.
It's 227 bytes long.
There is a set ratio of algorithm clear text to ciphertext.
There is a set ratio of algorithm clear text to ciphertext.
There is a set ratio of algorithm clear text to ciphertext.
You can take it.
So, if you know the size of something,
you can guess certain amounts of the content.
You know the hostname that it's connected to,
so you've got 206 left.
You know the user agent, because I fingerprinted the OS,
so you can roughly guess what the user agent is,
so now you're down to 151.
You know what that user agent accepts,
so you're down to 95.
You know how it encodes,
so now you're down to 58.
And, you've got the IP address,
so you can probably guess that you've got 26 left.
left you know that you're going to have the words get and http so you can guess that the remainder
of the bytes are probably the url and narrowed down to a degree where you might be unless you
pad so just like pad everything conclusion i'm running low on time so uh this is stuff uh related
to this um probably the main thing to do is just look at the slide deck one uh speaker deck.com
my name then you can find this um i was going to say any questions i've got maybe 10 no no time for
questions i will be in a corridor or a bar or something and you're more than welcome to grab
me and i'll do this again thank you