Hello DEF CON, welcome to our talk DNS section.
Today will be an interesting talk about DNS and the workings of the internet and cryptography.
So what this talk is about is about an email privacy breach in the largest French cloud provider
which is also the first practical attack on NSEC, DNS Secure Zone Working.
And finally it's a cautionary tale about the use of hash functions.
Why this matters is because DNS is everywhere, it's part of the internet, it's heavily used,
it's very important and there are tons of potentially very valuable interesting data
all over the place. The other reason why it matters is that zone working has not been demonstrated
in the wild to this extent before and it's the first time that we actually recover very valuable
interesting data from it. Who we are? Well this is Adrien. Hello! And I'm Rémy. We are from Ecole
Normale Supérieure, PSL University in Paris, France, in the beautiful place itself. And this
is not the first time that we talk at DEFCON. This has been done in collaboration with Amaury
Barral and David Nekash. So without further ado let's start with the first part of our talk.
So to understand DNS it's very important to understand our talk. We're going to start with
that. The basic question that you ask with DNS is who is behind a website such as skytalk-vids.com
for instance. So DNS, the domain name system, is a system to name remote resources, to give you
access to them, and it does that by holding a distributed database. A system that contains
record resources and domain names and that allows resolvers
to figure out the translation between a domain name and an IP address.
The DNS tree itself is subdivided into subtrees that are called zones and are
administrated by different people, different entities, whose role it is to keep it up to date.
So when you want to, say, build a website, create a new website, well, you actually need to go
through several steps. Once you have a device and you have a server and you have a website running
on it, you need to add your device, your IP address, to a DNS server. And to do that you
call a registrar and you run a DNS server that advertises your IP in connection to the domain
name that you are interested in. And you have to maintain all of this, of course.
Most people do not want to go to such trouble to get their website running,
and therefore they just pay someone to do the job for them,
including getting all this working together. This is the spirit behind cloud hosting,
which does provide the device, the hosting, the maintenance, and also the DNS registration and DNS
publication. And this is the main thing we are going to talk about.
Our target today is OVHcloud. They are the largest French cloud provider, by far,
and second in Europe. And they sell all kinds of cloud-related services.
In particular, they do sell domain names. And the basic plan for domain names include email
redirects. Namely, it means you do not get a full email account, but you can redirect,
contact at your OVHcloud domain to your usual Gmail account.
And as a side note, they do host Wikileaks since 2010.
So let's create an email redirect on the example domain. Here, dnssection.ovh,
which we have just bought for the occasion. So in the admin panel, we just add a redirect from
test.dnssection.ovh to target.yopmail.com and click confirm. And if we go look at the DNS zone,
which we can also find in the admin panel, then we realize there is a new DNS record in there.
If we zoom out this, then we can see that for test.dnssection.ovh,
then you have a txt record with a target email, so target.yopmail.com.
So this means that anyone in the world can just query test.dnssection.ovh and get the target of
your email redirect. Obviously, it's not easy to get the whole database because you cannot ask
the DNS server for the whole list of subdomains of at.dnssection.ovh.
So why does it matter? Why do we care? Well, assume for a moment that we do manage to find a
way to access this redirection database. It does contain a lot of interesting information. It's
actually essentially client information for the users of this cloud hosting service.
It includes emails, of course, but also names and potentially more information,
billing information, for instance, with which we can be creative. We can actually
start wondering, okay, what can we do with this information? Well, we can perhaps spam, we can
try password dumps, we can perform social engineering on the basis of this information,
we can perhaps find interesting information on the businesses that they hold, and so forth.
So there's really a lot of ideas about what we can do with this information,
and we will show you that in the latest part of this talk.
So, well, let's try the naive way to do brute force to recover all the information using just
a DNS query directly on the hosts that we are interested in. So the way we would do that is we
get a list of OVH cloud-handled domains, we select those that we are most interested in,
get a sublist, and we DNS query these domains. It works rather well for the domains that are
hosted by OVH. And in doing so, we have two ways essentially to get an interesting redirect email.
Either we look for the public email addresses, the ones that are displayed on the website,
and we try to see whether they redirect to something interesting. This is one way. The
other way is we don't really know what the real address is or whether they are redirects to begin
with. So we, well, we try addresses we think are likely to exist, such as abuse at the domain,
admin at contact at the domain. And in doing so, we, if we get a response,
we get not only the existence of this particular email address that we just guessed, but also the
corresponding target email redirect, which does not appear either on the website or in that case
elsewhere. Of course, if you just try to do this, you're going to run into trouble
because it's very likely that it is not stealthy to just query massively random DNS
servers with random email addresses. So in practice, if you want to do this, you better be
clever and avoid rate limiting by using several clients. But just for the sake of demonstrations,
here we can do this from a single host using very simple low-tech devices, including
bash and dig and just the file system. So really, we just run a first script that
queries the DNS server to get a lot of interesting information about the domains. And then we dig the
txt record, the one that contains the redirect in our case. So well, there's a demo for that.
Okay, so let's look at this in action. You have here the bash script we just showed you.
And we are going to run this very script right now. It will show addresses we just blurred them
for privacy reasons. So on the left, you have the addresses that we query,
and we recover the txt record on the right with the redirection email. This video is real-time,
so it shows you that this is quite fast, in fact, quite efficient. And we do this from a single host
here. But of course, you can parallelize this effort by calling several addresses at the same
time. So what do we get? It does work. It does work. We do get information. So if we consider
a subset of about 14,000 potentially vulnerable domains in the frtld, what we get is about
15,000 email redirects, of which about 10,000 unique email addresses, something we didn't have
before. So this is extremely interesting. But you might say, well, we used public emails or
easily guessable emails, and we found private redirection emails. So well, perhaps, I mean,
it's interesting, but what are we not seeing? What are we missing from the picture? And that's
where DNS section comes into play, really. So what we are going to do is we are going to use
DNSSEC. DNSSEC is in itself a very interesting topic and a very large one. If we want to go into
details, we might as well do a talk entirely on the topic and perhaps even several talks. So
very short summary, what you should know about it is that DNS itself is insecure if you use it like
it, just like this. So you need extensions, which are provided for most of them by DNSSEC.
Every modern good implementation of a DNS server does have support for DNSSEC and so do
resolvers. Essentially, what DNSSEC does is it provides a route of trust and a tree derivation
scheme that use public key cryptography, digital signatures to ensure authenticity. So you know
that you are really talking to the server you think you are talking to and that the information
provided there is at least the one that it commits to sending. So it requires cryptographic
elements and it requires sometimes some lockpicking skills, as perhaps the recent enough
incident depicted here shows, if you know what I'm talking about. If not, look into what happens
in a key rollover session of DNSSEC. So we can look at this, we can look at the information
sent to us by a DNSSEC compatible server. One tool that we can use to recover information
about DNSSEC is DNSViz. It is available in your favorite browser at dnsviz.net.
So here we apply it to dnssection.ovh, which is the website we use for demonstration
purposes during this talk. It has been slowed down so I can comment on it. It is actually much faster
than this. It queries all the interesting information about the DNS records here
and we can see the full chain of trust for the domain.
So in particular we can see all the cryptographic information, the algorithm in use and we can also
see some of the common sub-records that appear in the DNS zone. SOA, TXT, MX, NS and A records.
You can also see which algorithms have been used. In this particular case, dnssection.ovh
uses DNSSEC with algorithm 8, RSA plus SHA 265 with NSEC3. And you can go up to the OVH
and above zones and see how things are derivated from the root of trust. Here you can see .ovh
switching from algorithm 8 to algorithm 13, one of the 2020 goals for AFNIC.
DNSviz also checks for errors. Try it on Defcon.org, you will be surprised.
Unless of course the goons have watched our talk in advance and have already fixed it.
Let us dive in a fundamental obstacle of DNSSEC. Issues with negative responses.
For positive responses, it's quite easy. If you want to authenticate that the record example.com
is at IP 1.2.3.4 exists, then it's quite easy. You just affix your signature to it.
But if you want to authenticate that there is no record for bad.example.com,
this is much trickier. You obviously cannot put every negative possibility signing in the zone.
This is where NSSEC comes to the rescue.
NSSEC is much more trickier and complicated than what I'm going to explain here, but all you need
to understand is the principle depicted in this slide. So basically NSSEC signs
intervals where there is no domain. For example, it could sign there is no domain
between apple.example.com and carrot.example.com in lexicographical order and therefore it means
that bad.example.com cannot exist because B is between A and C.
And it does create a big issue so that you can now enumerate all records in the DNS zone.
How to do that is quite easy. Just pick a random name, for example fgfrd.example.com
and query the DNS server for it. Obviously fgrd does not exist, it's a random name,
it will enter the interval associated with it and so here it tells you that there is nothing
between carrot.example.com and good.example.com. Very good, we have just found two subdomains
of example.com. We just repeat with a successor of good, here good A, get another name,
another interval, and loop until we've found every positive record in the zone.
So it means that it is trivially doable for an attacker to get the whole DNS zone
of an NSEC-enabled domain. So now you might be thinking okay that's what we're gonna do,
we're just gonna use NSEC zone walking. No, that's not what we're gonna do,
because NSEC zone walking doesn't work. It doesn't work anymore in the real
world for the simple reason that no one uses NSEC at all anymore. Yes, you can be sad about this,
it's not what we're gonna do. We're gonna do something more clever,
although not very different, so that's why we talked about it.
What we're gonna do is zone walking with NSEC 3. NSEC 3 has been designed and implemented and
deployed for the very reason of providing resistance against zone enumeration, the way
we just described for NSEC. So it's a patch on NSEC. In a nutshell, what it does is instead of
having the records in plain text, they hash them using several repetitions of the SHA-1 algorithm,
usually with sort or without sort. So the idea is it should hide the contents of the DNS records,
and assuming you cannot do anything with hash values, well, you don't get access to them,
you cannot enumerate. The truth is you can still dump the hash itself, so you can still
do zone walking, you can still extract the hashes, it still kind of works. And NSEC 3
is really what is deployed today in the real world, so that's what we are going to attack.
As Remy just explained, the assumption behind NSEC 3 is that reversing even partially the hash
is difficult. Well, in practice, it turns out it's not really true. You've probably heard about
many people trying to break hashes, especially for passwords, and breaking hashes is even basically
how you mine Bitcoin. So I think I'm hearing laughs in the Bitcoin mining farm when they hear
that SHA-1 cannot be reversed. So in practice, for DNSSEC, you can find off-the-shelf readily
available tools to crack the NSEC 3 hashes. However, to the best of our knowledge, it has
never been used to dig valuable data. They only use that for the same demo, which is finding the
list of subdomains of .com or .org, but most of the time the list of valuable domains is
better found using Google or your favorite search engine. Okay, so let's show you the NSEC 3
Volcker tool, which is used to get the hashes from the domain zone. This is actually a collection
of scripts. Here we call the script collect of NSEC 3 Volcker on our dnssection.ovh domain.
It starts by getting the NSEC 3 parameters, which are the salt and the number of SHA-1 iterations.
Then, it is going to crawl through it and try 100 domains to find the intervals as we have explained
in the previous slides. This video has been slowed down a lot. In practice, with a good network,
this is actually really fast. You even get a progress bar from the tool. Here, the zone is
quite small, so we get all the hashes very quickly. If you want to dump a TLD zone, I mean
top-level domain zone, such as .org, then it takes much much more time. Another tool, NSEC 3 map,
also exists and has advanced features, such as parallel queries and automatic conversion
for Hashcat and John the Ripper. Unfortunately, it has been written in Python 2, which is in the
process of going extinct. So, use whichever tool you feel more comfortable with.
Well, so we have just dumped all the hashes of dnssection.ovh. What do we do now? Well,
we bring out the GPU rig. Unfortunately, we don't have a whole datacenter.
We only have a nice Tesla P100. If you know nothing about GPUs, let's just say this is not
a potato GPU. Most of you have heard about Hashcat before. This is the standard tool
to break hashes. So, let us just feed the output of NSEC 3 to Hashcat and try different
kind of attacks, such as dictionary attacks and brute force attacks.
Here, we show you a little demo of Hashcat. In this case, this is a simple brute force of all
lowercase alphanumeric emails of exactly five characters, as you can see on the command line.
First, Hashcat displays some information about the GPU and compiles the OpenCL NSEC3 kernel
tailored for your GPU. Then, it starts the hash cracking process.
In this case, we only run it on a small subset of our findings. The video has been sped up
to avoid audience boredom. You can see a few hashes being reversed on the screen.
Again, we blurred the result for privacy reasons. As a side note, we have found a bug in Hashcat.
Dots were not handled correctly. Many emails contained dots in the left part,
so this proved to be a serious concern for us. We made a dirty fix for ourselves,
and the Hashcat team has since fixed it properly. Thanks a lot to them.
Let us consider about 16,000 interesting DNSSEC hashed records. Well, using our GPU,
we were able to unhash as much as 88% of them, so that's quite a good result.
Let's see a little breakdown of our results. So, in three quarters of the cases,
we were able to reverse the hash, and we did find an interesting email redirection.
In 14%, we did reverse the hash, but it was something else, such as an FPF record.
And in the remaining 12%, well, we were not able to unhash.
By God's thing, the remaining ones are mostly domain key records,
which are not descriptible for email redirections.
A little disclaimer, we are not here to dox people, so obviously, all people names and
domain names in the following example have been modified. Let's look at a few statistics.
So, first, we can just classify the target of the redirection by the email provider they use.
I think you can guess which email provider is used by most webmasters, and this is obviously
Gmail.com. And then, if you look at the left part of the email, well, it does leak the name
of the people quite often, because, well, most emails are your name at something,
not your country. So it happens that, in practice, the name of the person is found in the email in
about half the time. However, this doesn't mean that we found some private information
on some website, where people happily say that they do run the website.
So we try to find out how often we could not find the name in the target of the redirection
on the website. Well, this is in about two-thirds of the time. So, in two-thirds of the time, when
we have a name in the target of a redirect, then this was not something we could have easily found
on website. So we have leaked some private information. A little more complicated is
how often this email would not appear in a Google search. And in practice, this is about half of the
time. So this means that there are a lot of emails that are not publicly available, and we did,
again, find a lot of private info using them. The last thing was a little more difficult to
identify. And this is about a business connection, conflict of interest, or fake
competitors. And using the redirection we found, well, we were able to find such things in about
one-quarter of the cases. A little homework for you, that we have not tried, is run the found
emails for the haven't-been-pwned database, and find how many of them do have an entry in it.
What if we try to dox a scam on an adult website? Well, first, please don't tell my wife about it.
Well, we did try to find some names or some business connections, but they are very clever,
and it's actually a fail. Their email never discloses their name.
But we still have the email, so who's the scammer and who's the scammer.
Anything more serious? Well, we did find some famous people emails, like politicians
and showbiz people, which have their own Wikipedia page. We also found a few emails of
activists, and on a much lighter note, we did find a lawyer website with a redirect to,
guess what, mylittlepony.hisbirthdate at gmail.com. And also, we found 50 people who
had added redirects for the no-reply at other domain. Why would they do that?
A little caveat, this is actually some manual analysis. We went through hundreds of websites
phishing for the names and the emails. So typically, this means going through the contact
page, googling the name, googling the emails, and dealing with obscene stuff such as adult
flash websites, which are really difficult to navigate through. So this is all by best effort,
and we might have missed some public data. So okay, we found a lot of interesting information.
Some of it apparently does not appear either on the website or in Google, and very likely was
information that was not meant to be made public. So we thought it was an interesting issue. We
tried to bring it up to OVHcloud. So we called the hotline, and we're going to tell you about
the disclosure process. So we called the hotline, and we said, okay, we think we have a problem
with your handling of private data. They say, send an email to abuseat. Okay, so we write the email,
and we include the technical details, some of which we just shared with you right now,
and we never got a reply. So well, we were a bit puzzled. Perhaps we did not understand
what they meant by send an email. So we called them again, and calling them again, they say, yes,
do send an email to abuseat. You did the right thing. Just do it again. Perhaps they missed it.
So okay, well, we write a second email, and we never got a reply.
So at that point, what we tried is to contact people working there, try to ping the right
person, and to try and get them to forward our email internally to the right people.
Well, to this day, we are still waiting for a response. So OVH, if you are looking at us,
we tried. Okay, so if you want to fix it, what should you do if you do not want to be
targeted by the kind of enumeration and privacy disclosure that we just described?
Turns out that this has been a goal of NSEC for many years now. This was the reason why
NSEC 3 was proposed, and obviously NSEC 3 failed. But the real, the correct answer
to this is use public key cryptography. It's been described in two RFCs how to do this with
NSEC, in a way that's compatible with the NSEC. And there are two leading implementations of this
idea. One is NSEC 5, which is already six years old now. And the other is to use NSEC 3 but replace
the hash by a public key signature, a digital signature. The problem is NSEC 5 was met with
skepticism in the first draft, and the draft has not been finalized, and it has not been standardized,
and it brings latency into the game. But most importantly, NSEC has a bad track record. This is
already the fifth iteration of it, and you have not heard of NSEC 2 or NSEC 4 in this talk, and
you might wonder why. Well, perhaps they never got to see the light for good reasons. The alternative
is NSEC 3, NSEC 3 with digital signatures. And today that means ECDSA algorithm 13 in the DNS
SEC documentation, mainly because it is the only one that can be used in this list,
because it signs fast and the signatures are small enough that it is actually practical to use.
But it's on a fixed curve with a fixed hash function, and it's a bit of an old algorithm,
to be honest, which makes, amongst other issues, resolvers carry the burden. The fact that
resolvers now have to perform signature verification, which is actually quite slow.
It also requires very careful implementation and handling of algorithms and keys, because ECDSA
is famously brittle. An experience in the matter shows that DNS servers are, well,
historically bad at it. We gave you a reference on the slide that you can look into
about the handling of RSA keys in the past. Remy has just explained you how to fix DNSSEC,
but most people do not want to handle their DNS zone by themselves, and thus use cloud hosting.
So, assume for a second that you are an OVHclose customer, and you are using their redirection
features. How can you protect yourself right now? Well, there are two different things to protect.
First, the target of the email redirection. An easy way to hide the real target email
is to use double redirects. Let me explain. First, redirect test.dnssection.ovh
to test.dnssection.gmail.com in the OVHcloud admin panel. Then, in your Gmail interface,
redirect test.dnssection.gmail.com to your personal email address. An attacker will only
be able to see the first redirect, and thus will not learn anything useful.
Second, you want to protect your domain email list. This is actually quite tricky.
What about disabling DNSSEC entirely? Well, there is a reason DNSSEC exists,
and dropping the security properties of DNSSEC might not be worth it. Or you may also have
services for which DNSSEC use is a must. Another possibility to protect the list
would be to use long and unpredictable emails, so the hash is almost impossible to reverse.
For online websites, this works quite well. The web server does not care much. However,
a long and non-human friendly email is not going to make a good first impression
on your business card. If your domain email list is especially sensitive, the best way out is
probably to stop using vulnerable OVHcloud redirection for now. Upgrade to an OVHcloud plan
with real email addresses. So, well, that brings us to a conclusion for this talk,
which will be rather fast, because you heard most of what we had to say already.
The lesson to remember, do not store private information in your DNS zone if you do not have
to. This is likely to leak unless, well, countermeasures are widely deployed. DNSSEC and
SEC3 attacks exist. They are practical and they can reveal information that was not obtained in
any other way that we know of. So try to push your local representative for DNS server to adopt
modern countermeasures against zone enumeration. NSEC5 is not yet final. Try to push for it.
And adoption of ECDSA as a signature for zone enumeration is also on the way. But, you know,
it's taking time. So try and get there. That's all we had to say. And do have a look
at the Proof of Concept website that we put up for you to play with.
Thank you again. Have a great day and try this at home. Very carefully, of course. Bye-bye.