start	end	text
3210	8360	Hello DEF CON, welcome to our talk DNS section.
9280	17080	Today will be an interesting talk about DNS and the workings of the internet and cryptography.
17480	26040	So what this talk is about is about an email privacy breach in the largest French cloud provider
26730	32560	which is also the first practical attack on NSEC, DNS Secure Zone Working.
32660	38180	And finally it's a cautionary tale about the use of hash functions.
39400	46240	Why this matters is because DNS is everywhere, it's part of the internet, it's heavily used,
46240	51780	it's very important and there are tons of potentially very valuable interesting data
51780	59140	all over the place. The other reason why it matters is that zone working has not been demonstrated
59140	65260	in the wild to this extent before and it's the first time that we actually recover very valuable
65260	78100	interesting data from it. Who we are? Well this is Adrien. Hello! And I'm Rémy. We are from Ecole
78100	84880	Normale Supérieure, PSL University in Paris, France, in the beautiful place itself. And this
84880	91620	is not the first time that we talk at DEFCON. This has been done in collaboration with Amaury
91620	100340	Barral and David Nekash. So without further ado let's start with the first part of our talk.
104780	110980	So to understand DNS it's very important to understand our talk. We're going to start with
110980	119620	that. The basic question that you ask with DNS is who is behind a website such as skytalk-vids.com
119620	127060	for instance. So DNS, the domain name system, is a system to name remote resources, to give you
127060	133800	access to them, and it does that by holding a distributed database. A system that contains
134360	139100	record resources and domain names and that allows resolvers
139650	145600	to figure out the translation between a domain name and an IP address.
146440	152860	The DNS tree itself is subdivided into subtrees that are called zones and are
152860	159720	administrated by different people, different entities, whose role it is to keep it up to date.
159720	167680	So when you want to, say, build a website, create a new website, well, you actually need to go
167680	172680	through several steps. Once you have a device and you have a server and you have a website running
172680	181400	on it, you need to add your device, your IP address, to a DNS server. And to do that you
181400	189880	call a registrar and you run a DNS server that advertises your IP in connection to the domain
189880	195010	name that you are interested in. And you have to maintain all of this, of course.
195700	200700	Most people do not want to go to such trouble to get their website running,
200700	205420	and therefore they just pay someone to do the job for them,
205420	212680	including getting all this working together. This is the spirit behind cloud hosting,
212680	220740	which does provide the device, the hosting, the maintenance, and also the DNS registration and DNS
221260	228120	publication. And this is the main thing we are going to talk about.
231840	238140	Our target today is OVHcloud. They are the largest French cloud provider, by far,
238140	244220	and second in Europe. And they sell all kinds of cloud-related services.
245500	252200	In particular, they do sell domain names. And the basic plan for domain names include email
252800	257770	redirects. Namely, it means you do not get a full email account, but you can redirect,
257770	263250	contact at your OVHcloud domain to your usual Gmail account.
263330	267330	And as a side note, they do host Wikileaks since 2010.
269610	276810	So let's create an email redirect on the example domain. Here, dnssection.ovh,
276810	284150	which we have just bought for the occasion. So in the admin panel, we just add a redirect from
284150	294830	test.dnssection.ovh to target.yopmail.com and click confirm. And if we go look at the DNS zone,
294830	301450	which we can also find in the admin panel, then we realize there is a new DNS record in there.
301690	308470	If we zoom out this, then we can see that for test.dnssection.ovh,
308470	313850	then you have a txt record with a target email, so target.yopmail.com.
314670	324730	So this means that anyone in the world can just query test.dnssection.ovh and get the target of
324730	331170	your email redirect. Obviously, it's not easy to get the whole database because you cannot ask
331170	338730	the DNS server for the whole list of subdomains of at.dnssection.ovh.
344260	350960	So why does it matter? Why do we care? Well, assume for a moment that we do manage to find a
350960	357740	way to access this redirection database. It does contain a lot of interesting information. It's
357740	364000	actually essentially client information for the users of this cloud hosting service.
364000	369760	It includes emails, of course, but also names and potentially more information,
369760	374500	billing information, for instance, with which we can be creative. We can actually
374500	380380	start wondering, okay, what can we do with this information? Well, we can perhaps spam, we can
380380	386300	try password dumps, we can perform social engineering on the basis of this information,
386300	391620	we can perhaps find interesting information on the businesses that they hold, and so forth.
391620	396320	So there's really a lot of ideas about what we can do with this information,
396320	400280	and we will show you that in the latest part of this talk.
402420	411400	So, well, let's try the naive way to do brute force to recover all the information using just
411860	419020	a DNS query directly on the hosts that we are interested in. So the way we would do that is we
419020	426640	get a list of OVH cloud-handled domains, we select those that we are most interested in,
426640	434540	get a sublist, and we DNS query these domains. It works rather well for the domains that are
434540	443440	hosted by OVH. And in doing so, we have two ways essentially to get an interesting redirect email.
443440	450000	Either we look for the public email addresses, the ones that are displayed on the website,
450000	455560	and we try to see whether they redirect to something interesting. This is one way. The
455560	461840	other way is we don't really know what the real address is or whether they are redirects to begin
461840	470100	with. So we, well, we try addresses we think are likely to exist, such as abuse at the domain,
470100	477980	admin at contact at the domain. And in doing so, we, if we get a response,
477980	483820	we get not only the existence of this particular email address that we just guessed, but also the
483820	492300	corresponding target email redirect, which does not appear either on the website or in that case
493000	498960	elsewhere. Of course, if you just try to do this, you're going to run into trouble
499500	508180	because it's very likely that it is not stealthy to just query massively random DNS
509160	515180	servers with random email addresses. So in practice, if you want to do this, you better be
515180	522400	clever and avoid rate limiting by using several clients. But just for the sake of demonstrations,
523280	530120	here we can do this from a single host using very simple low-tech devices, including
530800	539140	bash and dig and just the file system. So really, we just run a first script that
539140	547140	queries the DNS server to get a lot of interesting information about the domains. And then we dig the
547140	555060	txt record, the one that contains the redirect in our case. So well, there's a demo for that.
560710	568270	Okay, so let's look at this in action. You have here the bash script we just showed you.
571060	581510	And we are going to run this very script right now. It will show addresses we just blurred them
581510	587730	for privacy reasons. So on the left, you have the addresses that we query,
587730	597410	and we recover the txt record on the right with the redirection email. This video is real-time,
597410	605270	so it shows you that this is quite fast, in fact, quite efficient. And we do this from a single host
605270	611790	here. But of course, you can parallelize this effort by calling several addresses at the same
611790	625810	time. So what do we get? It does work. It does work. We do get information. So if we consider
626010	634910	a subset of about 14,000 potentially vulnerable domains in the frtld, what we get is about
634910	642990	15,000 email redirects, of which about 10,000 unique email addresses, something we didn't have
642990	650150	before. So this is extremely interesting. But you might say, well, we used public emails or
650150	657830	easily guessable emails, and we found private redirection emails. So well, perhaps, I mean,
657830	663930	it's interesting, but what are we not seeing? What are we missing from the picture? And that's
663930	676130	where DNS section comes into play, really. So what we are going to do is we are going to use
676130	684070	DNSSEC. DNSSEC is in itself a very interesting topic and a very large one. If we want to go into
684070	690550	details, we might as well do a talk entirely on the topic and perhaps even several talks. So
691170	696770	very short summary, what you should know about it is that DNS itself is insecure if you use it like
696770	703750	it, just like this. So you need extensions, which are provided for most of them by DNSSEC.
703970	710770	Every modern good implementation of a DNS server does have support for DNSSEC and so do
710770	717230	resolvers. Essentially, what DNSSEC does is it provides a route of trust and a tree derivation
717230	723610	scheme that use public key cryptography, digital signatures to ensure authenticity. So you know
723610	728610	that you are really talking to the server you think you are talking to and that the information
728610	736730	provided there is at least the one that it commits to sending. So it requires cryptographic
736730	744890	elements and it requires sometimes some lockpicking skills, as perhaps the recent enough
744890	751650	incident depicted here shows, if you know what I'm talking about. If not, look into what happens
751650	760090	in a key rollover session of DNSSEC. So we can look at this, we can look at the information
760090	773490	sent to us by a DNSSEC compatible server. One tool that we can use to recover information
773490	783370	about DNSSEC is DNSViz. It is available in your favorite browser at dnsviz.net.
785370	792550	So here we apply it to dnssection.ovh, which is the website we use for demonstration
792550	801010	purposes during this talk. It has been slowed down so I can comment on it. It is actually much faster
801010	811740	than this. It queries all the interesting information about the DNS records here
811740	816260	and we can see the full chain of trust for the domain.
817360	825400	So in particular we can see all the cryptographic information, the algorithm in use and we can also
825400	837160	see some of the common sub-records that appear in the DNS zone. SOA, TXT, MX, NS and A records.
838120	845820	You can also see which algorithms have been used. In this particular case, dnssection.ovh
845820	861250	uses DNSSEC with algorithm 8, RSA plus SHA 265 with NSEC3. And you can go up to the OVH
861250	869650	and above zones and see how things are derivated from the root of trust. Here you can see .ovh
869650	877250	switching from algorithm 8 to algorithm 13, one of the 2020 goals for AFNIC.
879010	886230	DNSviz also checks for errors. Try it on Defcon.org, you will be surprised.
886530	892570	Unless of course the goons have watched our talk in advance and have already fixed it.
902710	910410	Let us dive in a fundamental obstacle of DNSSEC. Issues with negative responses.
911090	916610	For positive responses, it's quite easy. If you want to authenticate that the record example.com
916610	925710	is at IP 1.2.3.4 exists, then it's quite easy. You just affix your signature to it.
925710	930770	But if you want to authenticate that there is no record for bad.example.com,
930770	938130	this is much trickier. You obviously cannot put every negative possibility signing in the zone.
938610	941090	This is where NSSEC comes to the rescue.
942990	949330	NSSEC is much more trickier and complicated than what I'm going to explain here, but all you need
949330	956490	to understand is the principle depicted in this slide. So basically NSSEC signs
957450	963590	intervals where there is no domain. For example, it could sign there is no domain
963590	972050	between apple.example.com and carrot.example.com in lexicographical order and therefore it means
972050	978430	that bad.example.com cannot exist because B is between A and C.
980070	989210	And it does create a big issue so that you can now enumerate all records in the DNS zone.
989210	996390	How to do that is quite easy. Just pick a random name, for example fgfrd.example.com
996390	1004350	and query the DNS server for it. Obviously fgrd does not exist, it's a random name,
1004350	1011010	it will enter the interval associated with it and so here it tells you that there is nothing
1011010	1018450	between carrot.example.com and good.example.com. Very good, we have just found two subdomains
1018450	1026050	of example.com. We just repeat with a successor of good, here good A, get another name,
1026050	1032950	another interval, and loop until we've found every positive record in the zone.
1032950	1040210	So it means that it is trivially doable for an attacker to get the whole DNS zone
1040210	1050210	of an NSEC-enabled domain. So now you might be thinking okay that's what we're gonna do,
1050210	1055290	we're just gonna use NSEC zone walking. No, that's not what we're gonna do,
1055890	1060750	because NSEC zone walking doesn't work. It doesn't work anymore in the real
1061410	1068690	world for the simple reason that no one uses NSEC at all anymore. Yes, you can be sad about this,
1068690	1072170	it's not what we're gonna do. We're gonna do something more clever,
1072170	1075710	although not very different, so that's why we talked about it.
1077970	1085130	What we're gonna do is zone walking with NSEC 3. NSEC 3 has been designed and implemented and
1085130	1091530	deployed for the very reason of providing resistance against zone enumeration, the way
1091530	1098490	we just described for NSEC. So it's a patch on NSEC. In a nutshell, what it does is instead of
1098490	1106190	having the records in plain text, they hash them using several repetitions of the SHA-1 algorithm,
1106190	1115070	usually with sort or without sort. So the idea is it should hide the contents of the DNS records,
1115890	1120890	and assuming you cannot do anything with hash values, well, you don't get access to them,
1120890	1126630	you cannot enumerate. The truth is you can still dump the hash itself, so you can still
1126630	1133470	do zone walking, you can still extract the hashes, it still kind of works. And NSEC 3
1133470	1140690	is really what is deployed today in the real world, so that's what we are going to attack.
1145050	1152170	As Remy just explained, the assumption behind NSEC 3 is that reversing even partially the hash
1152170	1158870	is difficult. Well, in practice, it turns out it's not really true. You've probably heard about
1158870	1165610	many people trying to break hashes, especially for passwords, and breaking hashes is even basically
1165610	1176750	how you mine Bitcoin. So I think I'm hearing laughs in the Bitcoin mining farm when they hear
1176750	1186910	that SHA-1 cannot be reversed. So in practice, for DNSSEC, you can find off-the-shelf readily
1186910	1194890	available tools to crack the NSEC 3 hashes. However, to the best of our knowledge, it has
1194890	1203910	never been used to dig valuable data. They only use that for the same demo, which is finding the
1203910	1211530	list of subdomains of .com or .org, but most of the time the list of valuable domains is
1211530	1224470	better found using Google or your favorite search engine. Okay, so let's show you the NSEC 3
1224470	1232530	Volcker tool, which is used to get the hashes from the domain zone. This is actually a collection
1232530	1242330	of scripts. Here we call the script collect of NSEC 3 Volcker on our dnssection.ovh domain.
1246680	1255540	It starts by getting the NSEC 3 parameters, which are the salt and the number of SHA-1 iterations.
1256980	1265560	Then, it is going to crawl through it and try 100 domains to find the intervals as we have explained
1265560	1278360	in the previous slides. This video has been slowed down a lot. In practice, with a good network,
1278360	1287700	this is actually really fast. You even get a progress bar from the tool. Here, the zone is
1287700	1295860	quite small, so we get all the hashes very quickly. If you want to dump a TLD zone, I mean
1295860	1309290	top-level domain zone, such as .org, then it takes much much more time. Another tool, NSEC 3 map,
1309290	1316190	also exists and has advanced features, such as parallel queries and automatic conversion
1316190	1324450	for Hashcat and John the Ripper. Unfortunately, it has been written in Python 2, which is in the
1324450	1331950	process of going extinct. So, use whichever tool you feel more comfortable with.
1334640	1342000	Well, so we have just dumped all the hashes of dnssection.ovh. What do we do now? Well,
1342000	1347820	we bring out the GPU rig. Unfortunately, we don't have a whole datacenter.
1347820	1354500	We only have a nice Tesla P100. If you know nothing about GPUs, let's just say this is not
1354720	1364240	a potato GPU. Most of you have heard about Hashcat before. This is the standard tool
1364240	1371660	to break hashes. So, let us just feed the output of NSEC 3 to Hashcat and try different
1371660	1376720	kind of attacks, such as dictionary attacks and brute force attacks.
1379380	1388220	Here, we show you a little demo of Hashcat. In this case, this is a simple brute force of all
1388220	1396020	lowercase alphanumeric emails of exactly five characters, as you can see on the command line.
1400470	1409970	First, Hashcat displays some information about the GPU and compiles the OpenCL NSEC3 kernel
1409970	1418370	tailored for your GPU. Then, it starts the hash cracking process.
1419150	1427830	In this case, we only run it on a small subset of our findings. The video has been sped up
1427830	1435630	to avoid audience boredom. You can see a few hashes being reversed on the screen.
1435870	1449400	Again, we blurred the result for privacy reasons. As a side note, we have found a bug in Hashcat.
1449700	1457220	Dots were not handled correctly. Many emails contained dots in the left part,
1457220	1464720	so this proved to be a serious concern for us. We made a dirty fix for ourselves,
1464720	1470580	and the Hashcat team has since fixed it properly. Thanks a lot to them.
1473220	1482640	Let us consider about 16,000 interesting DNSSEC hashed records. Well, using our GPU,
1482640	1488140	we were able to unhash as much as 88% of them, so that's quite a good result.
1488800	1494340	Let's see a little breakdown of our results. So, in three quarters of the cases,
1494340	1500240	we were able to reverse the hash, and we did find an interesting email redirection.
1500360	1506600	In 14%, we did reverse the hash, but it was something else, such as an FPF record.
1506640	1511700	And in the remaining 12%, well, we were not able to unhash.
1512460	1517720	By God's thing, the remaining ones are mostly domain key records,
1517720	1521780	which are not descriptible for email redirections.
1523080	1530480	A little disclaimer, we are not here to dox people, so obviously, all people names and
1530480	1540740	domain names in the following example have been modified. Let's look at a few statistics.
1541520	1551000	So, first, we can just classify the target of the redirection by the email provider they use.
1551140	1557700	I think you can guess which email provider is used by most webmasters, and this is obviously
1557700	1567420	Gmail.com. And then, if you look at the left part of the email, well, it does leak the name
1567420	1575320	of the people quite often, because, well, most emails are your name at something,
1575320	1585380	not your country. So it happens that, in practice, the name of the person is found in the email in
1586060	1592480	about half the time. However, this doesn't mean that we found some private information
1593100	1597020	on some website, where people happily say that they do run the website.
1597420	1604640	So we try to find out how often we could not find the name in the target of the redirection
1604640	1610880	on the website. Well, this is in about two-thirds of the time. So, in two-thirds of the time, when
1610880	1618180	we have a name in the target of a redirect, then this was not something we could have easily found
1618180	1625940	on website. So we have leaked some private information. A little more complicated is
1625940	1634340	how often this email would not appear in a Google search. And in practice, this is about half of the
1634340	1641900	time. So this means that there are a lot of emails that are not publicly available, and we did,
1641900	1650260	again, find a lot of private info using them. The last thing was a little more difficult to
1651120	1657020	identify. And this is about a business connection, conflict of interest, or fake
1657020	1665600	competitors. And using the redirection we found, well, we were able to find such things in about
1665600	1674340	one-quarter of the cases. A little homework for you, that we have not tried, is run the found
1674340	1681980	emails for the haven't-been-pwned database, and find how many of them do have an entry in it.
1683860	1691940	What if we try to dox a scam on an adult website? Well, first, please don't tell my wife about it.
1692340	1699120	Well, we did try to find some names or some business connections, but they are very clever,
1699120	1703720	and it's actually a fail. Their email never discloses their name.
1703800	1708460	But we still have the email, so who's the scammer and who's the scammer.
1710720	1716480	Anything more serious? Well, we did find some famous people emails, like politicians
1717040	1724960	and showbiz people, which have their own Wikipedia page. We also found a few emails of
1724960	1731920	activists, and on a much lighter note, we did find a lawyer website with a redirect to,
1731920	1740820	guess what, mylittlepony.hisbirthdate at gmail.com. And also, we found 50 people who
1740820	1747000	had added redirects for the no-reply at other domain. Why would they do that?
1748740	1755020	A little caveat, this is actually some manual analysis. We went through hundreds of websites
1755020	1759780	phishing for the names and the emails. So typically, this means going through the contact
1759780	1764980	page, googling the name, googling the emails, and dealing with obscene stuff such as adult
1764980	1771940	flash websites, which are really difficult to navigate through. So this is all by best effort,
1771940	1783440	and we might have missed some public data. So okay, we found a lot of interesting information.
1783440	1791080	Some of it apparently does not appear either on the website or in Google, and very likely was
1791080	1797620	information that was not meant to be made public. So we thought it was an interesting issue. We
1797620	1804480	tried to bring it up to OVHcloud. So we called the hotline, and we're going to tell you about
1804480	1809780	the disclosure process. So we called the hotline, and we said, okay, we think we have a problem
1810560	1819920	with your handling of private data. They say, send an email to abuseat. Okay, so we write the email,
1819920	1825020	and we include the technical details, some of which we just shared with you right now,
1825020	1833300	and we never got a reply. So well, we were a bit puzzled. Perhaps we did not understand
1833300	1839980	what they meant by send an email. So we called them again, and calling them again, they say, yes,
1839980	1844540	do send an email to abuseat. You did the right thing. Just do it again. Perhaps they missed it.
1844540	1850700	So okay, well, we write a second email, and we never got a reply.
1852600	1859540	So at that point, what we tried is to contact people working there, try to ping the right
1859540	1865900	person, and to try and get them to forward our email internally to the right people.
1867020	1872480	Well, to this day, we are still waiting for a response. So OVH, if you are looking at us,
1872480	1882290	we tried. Okay, so if you want to fix it, what should you do if you do not want to be
1882810	1889130	targeted by the kind of enumeration and privacy disclosure that we just described?
1889690	1896710	Turns out that this has been a goal of NSEC for many years now. This was the reason why
1896710	1904530	NSEC 3 was proposed, and obviously NSEC 3 failed. But the real, the correct answer
1905090	1912410	to this is use public key cryptography. It's been described in two RFCs how to do this with
1912410	1919390	NSEC, in a way that's compatible with the NSEC. And there are two leading implementations of this
1919390	1928730	idea. One is NSEC 5, which is already six years old now. And the other is to use NSEC 3 but replace
1928730	1938310	the hash by a public key signature, a digital signature. The problem is NSEC 5 was met with
1938310	1944550	skepticism in the first draft, and the draft has not been finalized, and it has not been standardized,
1944550	1950850	and it brings latency into the game. But most importantly, NSEC has a bad track record. This is
1950850	1958070	already the fifth iteration of it, and you have not heard of NSEC 2 or NSEC 4 in this talk, and
1958070	1966070	you might wonder why. Well, perhaps they never got to see the light for good reasons. The alternative
1966070	1975450	is NSEC 3, NSEC 3 with digital signatures. And today that means ECDSA algorithm 13 in the DNS
1975990	1984310	SEC documentation, mainly because it is the only one that can be used in this list,
1985330	1992450	because it signs fast and the signatures are small enough that it is actually practical to use.
1993130	1999570	But it's on a fixed curve with a fixed hash function, and it's a bit of an old algorithm,
1999570	2006830	to be honest, which makes, amongst other issues, resolvers carry the burden. The fact that
2006830	2013070	resolvers now have to perform signature verification, which is actually quite slow.
2013470	2019570	It also requires very careful implementation and handling of algorithms and keys, because ECDSA
2019570	2027570	is famously brittle. An experience in the matter shows that DNS servers are, well,
2027570	2033510	historically bad at it. We gave you a reference on the slide that you can look into
2034030	2044870	about the handling of RSA keys in the past. Remy has just explained you how to fix DNSSEC,
2044870	2052490	but most people do not want to handle their DNS zone by themselves, and thus use cloud hosting.
2052730	2059630	So, assume for a second that you are an OVHclose customer, and you are using their redirection
2059630	2067790	features. How can you protect yourself right now? Well, there are two different things to protect.
2067870	2073790	First, the target of the email redirection. An easy way to hide the real target email
2073790	2083970	is to use double redirects. Let me explain. First, redirect test.dnssection.ovh
2083970	2094890	to test.dnssection.gmail.com in the OVHcloud admin panel. Then, in your Gmail interface,
2094890	2103510	redirect test.dnssection.gmail.com to your personal email address. An attacker will only
2103510	2109530	be able to see the first redirect, and thus will not learn anything useful.
2110210	2116750	Second, you want to protect your domain email list. This is actually quite tricky.
2117850	2126070	What about disabling DNSSEC entirely? Well, there is a reason DNSSEC exists,
2126070	2132130	and dropping the security properties of DNSSEC might not be worth it. Or you may also have
2132130	2139010	services for which DNSSEC use is a must. Another possibility to protect the list
2139010	2145930	would be to use long and unpredictable emails, so the hash is almost impossible to reverse.
2146570	2154870	For online websites, this works quite well. The web server does not care much. However,
2155070	2160150	a long and non-human friendly email is not going to make a good first impression
2160150	2168290	on your business card. If your domain email list is especially sensitive, the best way out is
2168290	2175570	probably to stop using vulnerable OVHcloud redirection for now. Upgrade to an OVHcloud plan
2175570	2184370	with real email addresses. So, well, that brings us to a conclusion for this talk,
2184370	2190390	which will be rather fast, because you heard most of what we had to say already.
2190390	2198330	The lesson to remember, do not store private information in your DNS zone if you do not have
2198330	2207270	to. This is likely to leak unless, well, countermeasures are widely deployed. DNSSEC and
2207270	2212930	SEC3 attacks exist. They are practical and they can reveal information that was not obtained in
2212930	2222110	any other way that we know of. So try to push your local representative for DNS server to adopt
2222110	2231270	modern countermeasures against zone enumeration. NSEC5 is not yet final. Try to push for it.
2231270	2238810	And adoption of ECDSA as a signature for zone enumeration is also on the way. But, you know,
2238810	2245310	it's taking time. So try and get there. That's all we had to say. And do have a look
2245310	2250150	at the Proof of Concept website that we put up for you to play with.
2251450	2258990	Thank you again. Have a great day and try this at home. Very carefully, of course. Bye-bye.