Hey, folks. Thanks for being here at the Crypto and Privacy Village. We have another fantastic
talk lined up. Our very own Jake Williams, Malware Jake from Rendition Info Sec. He's going
to be talking about the semantic SSL debacle, lessons learned and future steps. So please give
it up for Jake. And now you're going to give me a minute to readjust this mic because
apparently we only have one, which is cool. And it's not cordless. So if I trip on this,
somebody be prepared. Does anybody know first aid? No? Okay. I can't stand behind a podium
for half an hour or an hour or whatever we're going to do here. I think we've got like 45
minutes or something. So this is something that was really, really interesting to me when it
first hit. I'm really, really interested in SSL in the first place. I think SSL is a
dumpster fire. And hopefully by the end of this talk you share some of those same opinions
and if not, you at least understand why I think SSL is a dumpster fire and kind of what we need
to be thinking about as InfoSec people. And I look around here and I'm guessing that maybe I've
got 10ish percent decision makers and a lot of influencers, a lot of technical influencers,
right? So my goal here is for the decision makers to make sure you understand what's going on
and for the influencers to give you some of the arguments to go back and talk to folks and
communicate effectively with your decision makers so they can make the right decisions.
So I want to talk about the Symantec SSL issue. Originally I wasn't going to do a lot of this. I
was just going to kind of assume everybody knew. And over the last couple of weeks I've been
teaching SANS around the world, actually in three different countries over the last month. And I
found out that by polling InfoSec people, I found out that almost nobody understands this. So I
revamped the slide deck a lot. Then this week Google pulled a nasty on me. And they reached a
tentative agreement with Symantec on how the
data sets were going to move forward. So then I had to overhaul a bunch of stuff in the slide deck.
Because obviously then I've got, you know, can't talk about like, well, we don't know how this is
going to end. We actually have some idea how it's going to end. Of course we'll have to see how
that all plays out. We at least have an idea, right? So I've been pulling data, too. I'm a data
guy. I really, really like data. I'll make my data sets publicly available later. If anybody's
really interested, follow me on Twitter. I promise you I'll tweet out links to the data sets when I
get them up in GitHub. But basically over the last several months we've been pulling certificates.
And I'll talk more about some of our methods here. We've been going out and pulling SSL
certificates from the Alexa top 1 million. Now I know that this is not necessarily the best data
set. But it's a consistent data set. So back in April this whole thing started in March. Back in
April we went ahead and grabbed early April, grabbed the Alexa top 1 million. We haven't
updated that since because I want a consistent set of sites. I want to be able to watch the drift
across basically how certificates change and what are people doing with certificates that are
different. Because I would think that this industry wide announcement of intent to deprecate
trust in Symantec would force change in people. For those that don't know me, I'm a big
psychology guy, too. I actually minored in psych back in the day in my undergrad. And I love
psychology. So I'm really interested in the psychological aspect of this. Some of these messages
came out and people are renewing certificates. Did they do stuff differently? And unfortunately the
answer there is not a lot actually. It's kind of scary. I want to talk about some of the stuff
that we're dealing with here. This is not the first time we've dealt with the intent to deprecate a
cert. And it's definitely not the first time that we've dealt with not only an intent to deprecate a
cert but the need for mass cert revocation. And yet we're treating it like it's the first time.
Someone have a couple closing thoughts and kind of fire away on a couple of ideas. I'll give you a
little trigger warning here. If you think at the end I've got a solution for this, I do not. I'm a
smart guy. I like that. I tend to want to provide a solution, provide a problem, talk about the
problem space and then show up with a solution. I'm not smart enough apparently because I've been
racking my brain, I've been talking to a lot of other really smart people and nobody that I've
talked to has a good solution for this. So the issue. Why are we even having the talk in the
first place? The start of this is that Symantec, their subordinate CAs, Symantec and or their
subordinate CAs, they misissued a number of certificates. And originally the
number was found through an audit and originally it was about 12 certificates and now we know
that it's somewhere on the order of 30,000 certificates. And part of the problem that Google
had was that over time the number kept growing and growing and growing and there were addendums
upon addendums upon addendums to the final report. Obviously this is never a good thing. You
have a final report and again we're not talking about a final report. We all work in InfoSec I
presume most of you work in InfoSec somewhere. And you know that when you perform an audit, for
goodness sakes, there's always data that skeletons in the closet, stuff that was swept under the
rug, stuff that you find during an audit that you didn't find otherwise. But it's rare that I
ever go back and I have to issue an addendum to my final report and it's rare that I have to issue
an addendum to the addendum to the final report and I've never issued an addendum to the
addendum to the addendum to the final report which is what Symantec continued to do here.
Separately from this, this is not a regular audit. Part of this key here is that when you operate a
certificate authority, you are bound by a number of regulations to track data. This would be like
going into a bank audit and asking them, for instance, how much money is in the vault and
they're like, hold on a second while we go count that stuff. That's not okay. And it's not okay
here either. We're talking about the web of trust for the entire Internet. What else? Symantec failed to
track the activities of its affiliated CAs and the really cool part here is if you read into this or
read through this a little bit, they have this geo, basically this geo trust or geo root
initiative and basically in this geo root initiative, they are signing basically certificates
for what they call registration authorities, RAs. And these RAs, basically their job is to
validate that you are who you say you are. So picture for a moment you have this whole proxy
arrangement, right? I can vouch for you. I can vouch for you. I can vouch for you. I can vouch for
you. But I might vouch for you without even knowing you and it's the registration authority
that's actually validating that you are who you say you are and they say, hey, Jake. Of course I
know this person. I say, hey, Jake, go vouch for this guy and I just do it. Now here's where
things get interesting because over the years Symantec has bought a lot of RAs. They've bought a
lot of other certificate authorities. VeriSign, for instance, is one that they bought. Thought is
another one that they bought that used to be just absolutely huge, right? What's really, really
interesting is that Symantec in one of their audits failed to disclose one of their
registration agents, one of their RAs. And in the later audit they found out as Google challenged
them and said, hey, you forgot to put this in your transparency report, they were like, that's
not ours. And Google is like, oh, no, you're signing certs for these people. Now when you step
back and you think how bad is this, right? Now I'm paraphrasing a lot here. Go read the docs. I
understand. This stuff is tricky. But I think it's really interesting. And I think it's really
dry. But it's important and it's dry. Hopefully I'll get enough out there that you can speak
intelligently to this anyway. Symantec cross signed a root certificate used by the U.S. federal
government. And look, initially this is probably not bad if you live in the U.S. and 100% trust
the U.S. government. Of course I do. Right? I mean, I live in the U.S. and I don't trust the U.S.
government. It's ridiculous, right? Symantec issued test certificates and certs for unregistered
domains. Now some folks, and particularly Symantec, have tried to move the narrative along that
one of the reasons Google is so mad about this is that one of the test certificates that they
issued, I don't even know what a test certificate, well, what Symantec says a test certificate is, is
a certificate they legitimately signed for a domain that they do not own that they use for
testing. One of them that they signed was for Google. And you can imagine this made Google a
little annoyed because what would one do with this? Who the knows, right? And so they, now
Symantec comes back and says, look, these certs never, ever made it out of the wild and yet Google
figured it out. So anyway, there's like some interesting crosstalk going on there and some, I
will tell you that as you read this, you cannot come, as a rational person, as a rational analyst,
you cannot come to any other conclusion reading Symantec's responses than they were 100% asleep at
the wheel. And rather than saying, hey, guys, we were asleep at the wheel, we don't know what we're
going to do now. Help us out. At every turn, we're going to have to move the narrative around. And
in turn, they deny, deny, deny. It's like watching politicians debate. It's absolutely nuts.
So our CA is officially too big to fail. Look, the issues with Symantec were serious. Google
earlier dumped WoSign and StartCom. They said, look, we're not going to trust these guys
anymore. And they gave us about a month to knock this stuff out, a month and a half to knock out
WoSign and StartCom. Admittedly, WoSign and StartCom are relatively small. And if you read the
Google document, basically from Chrome or the Chrome team basically saying that they're ready to
dump support for Symantec, but we're going to phase it out over a period of more than a year,
you can see throughout that they talk about how big Symantec is, what the impact is going to be if
they just go bam and drop it. And I step back and I have to say, look, Symantec is apparently too
big to fail in the context of, in the context at least as far as Google and then Mozilla, which is
following suit with Google, is concerned. I think this is a big issue because if it's a security
issue, it's a security issue no matter how big the CA is. I might actually argue that if the CA is
that large, the certificate authority is that large, the security issue is actually more serious
because they're signing more certificates. And hence, we need to be ready to move faster with
large certificate authorities than we need to with small certificate authorities that sign a very,
very small number of certs. I'll finally say that InfoSec, as we're talking about this, when I do
security audits and I'm looking at different types of security audits and I'm looking at different
best practices for organizations, I rate organizations that, let's say, have a $10 billion
a year market cap differently than somebody that has a $2 billion a year market cap. You have a
different level of resources. Here, Symantec, my friends, has no excuse. And I'm going to be very
careful so I don't get sued because I'm sure there's someone from Symantec in here taking notes
and ready to go. That's cool, no problem. But look, the originally proposed solution, basically,
there was the, there was a, there was a, there was a, there was a, there was a, there was a, there
was the goal to go phase out over a number of releases and basically phase out trust for the
Symantec certs that were signed during what we'll call the trouble period. So by the beginning of
the year, by about February, it seems, at least, and there's some missing transparency, I think,
here from Google as well as Symantec, but it seems that everybody's confident that whatever the
issues were, well, not whatever they were, but among the many, many issues, and I don't have all
of them listed on the original slide. There's a lot of them listed on the original slide that I'm
there. There are many, many more. Those are the most serious. Based on the many issues that they
found, they said, hey, we think we're good going forward, but that leaves us in a position where,
during the trouble period, we know we misissued around 30,000 certs. Well, at the time, it was 100
certs and then it was many certs and then it eventually was 30,000 certs. And so the question
then becomes, what do you do with those? Because some of those have lifetimes of five years easily.
All right? You can go in and you can select if you're going to go in and you can select if you're
willing to pay the $15 a year or whatever it happens to be, depending on who your reseller is.
You can go ahead and pay that and you can get a ridiculously long certificate life. Should you do
that? That's a whole different security question, whole different security discussion. Google
appeared to be interested then in, even as they reissued certs, they wanted to lower the
validity. Symantec came back and said, look, this is punitive because nobody else is going to be
bound by these timelines. And they're right. Google, to their credit, came back and said, hey,
we get it, totally, but you guys have been negligent, I think, as well. Wait, wait, wait. If
there's a lawyer in the crowd for Symantec, I didn't use the word negligent. I was paraphrasing
there. Maybe negligent isn't the word. You have been demonstrably careless in, that sounds
better. I'm not a lawyer, but mine reviewed this talk. So anyway, so the basically
demonstrably careless in doing this, we think you're going to do it again. Kind of like I've
got a dieting problem, I ate too many cookies, and odds are good I'm going to do it tonight,
too. So look, seriously, that kind of thing going. And so they drop this down, and they say,
look, you know, consistently you're going to have lower and lower validity periods here, and
you're going to have to live with that. Symantec then says, look, we've got predatory vendors
jumping up and down, basically contacting customers, and look, if I'm a vendor, I totally
would be doing this. If I'm an SSL vendor, I would totally be contacting Symantec customers for
two reasons. One, the majority of Symantec customers have no idea this is going on, or very
limited idea this is going on, at least initially. And to be fair, if the proposed solution
rolls through, it's going to be difficult for you to, more difficult for you to stay with
Symantec than it is to go with literally anybody else. And if I'm a vendor, that seems like a
really good market differentiation, and I would totally be pitching that to people there. Look,
this is either dirty pool or good business, depending on where you sit and the kind of
how you feel, and I don't know, if I'm Symantec, I call foul. If I'm every other vendor, I'm
like, heck yeah, I'm totally getting in on that game. Regardless, know that it happened, know
that it's still happening because the proposed solution, proposed now, new proposed solution was
just ratified this week. So know that it's nowhere near done, and I would still expect other
vendors to capitalize on this and try to capitalize it. If you are a Symantec customer, be
very, very careful about what you do next. Make a good investment. If you are a Symantec
customer, make an informed decision. My personal take, and I'll go ahead and, in case anybody
needs to cut out quickly, and you are a Symantec customer, my personal take is I just wouldn't
continue with them. And this has nothing to do with whether or not I believe they're doing good
CA work today. My honest to goodness opinion there is no, they're not, but that's a whole
separate issue. The reality is that they're already on probation, right? And I feel like
everything else that happens from here is probably going to be viewed with a, or a
responded to basically with a sledgehammer rather than a coddling like they've had so far.
Regardless, just so you know what's happening, let's talk about the cross signed federal route
because this is cool. I like this, right? Well, no, actually I don't like this at all, but
Symantec cross signed a bridge certificate for the U.S. federal government. So the period of
nearly five years, by the way, Symantec, when they sign a route, this is not a certificate that
they sign. I want to be very clear about this so everybody understands this. It's not a regular
certificate. It's not like they have for DOJ.gov. It's not like they have for DOJ.gov,
they signed a certificate so you go to the website and get a little trusted icon there. They
signed a route for the federal government. Cross signed a route for the federal government and
failed to report it. Now, any CA that's trusted is required, absolutely required by the CA
standards, by the browser alliance. Actually, that sounds like the force or something, but you
get the idea there, right? So Mozilla, Opera, or Mozilla, Apple, Google, and Microsoft to
report when they sign new routes.
They signed a route for the federal government, for the U.S. federal government. And this
certificate could be used without any further interaction from Symantec to sign certificates for
anything. And you have to understand the severity of that. Because as much as I trust the federal
government in the U.S., right, I think looking out here, the majority of you are probably from the
U.S. That's fine. Step back and ask yourself, what would you be doing if you were to sign a route for
anything if this was China? If Symantec had signed a bridge certificate for China, how would you
feel about that? Well, so hold on a second. China has roots. The Chinese government doesn't
have. Now you're arguing with me about is it China? Now, I'm getting there. So you're jumping ahead
with me here. So, okay, look. If it was the Chinese government or the Somali government, they
don't have a CA. The Somali government were owned a. . .
And then we had certificate pirates running around. Sounds actually like a new whiskey pirates
kind of thing. But regardless, look, bottom line, try not to put your tinfoil hats on here, but this
is serious because we're talking about the same government that oversees NSA, CIA, and lots of
organizations you probably don't trust. And when I step back and ask, realistically, if you got
redirected to a website that now had a brand-new certificate signed by a trusted CA, would you know?
Well, that's like one guy that would know because he constantly is looking at certs and doing
research on certs. And that's like the one guy, I.E. Brotherson. And that's about it. I can't
imagine any. . . I wouldn't notice. I'll be the first to tell you. Most of our organizations
wouldn't notice either. And these can facilitate man-in-the-middle attacks. And so once the
transgression was discovered, and again, this is important to know this ran for five years, and
once it was discovered, they didn't fix it. They waited. And they went back and forth about like
the, hey, we're working on this. . . . . . . . . . . . . . . . . . . .
. . . what? I thought nobody planned on revoking this. This is too easy. It's literally, you
just revoke it. There's a process in place for this. It's called CRL. It took them five months to
correct the issue. I have a couple of ideas. Why did it happen. General laziness? I don't know.
Give it to Fed's time to react to a PKI, change. I don't know why this is necessary in the first
place. The federal government on the browsers that they are, sorry, sorry on the sites they use,
they issue their own certs, anyway. If you've ever been in the military or worked with any of
of those folks, you know you have to go install root, which is
weird anyway, but you got to go install the root. They didn't
need to deal with this. Give the feds time to react to the PKI
change. Maybe Symantec doesn't want to be a CA and the five
months seemed like a great way to bridge that. It's kind of like
a polite no-bid. Somebody comes up, asks you to work a job and
your normal rate is here and you're like, I'm going to go bid
it way up here. You don't want to tell them no, but maybe
Symantec is like that. They're like, we're done with the CA
business, but I'm going to piss off all our customers, so we'll
just get beat down. I don't know. Or possibly it's to give
feds time to abuse the soon to be lost capability. And I'm not
going to tell you which one I think it is, because I don't
want to put a tin foil hat on up here. But if you knew you had a
capability and you had a very, very limited time left to use
it, what would you do with it? Anyway. The issue with test
certificates. So Symantec issued me a test certificate.
They issued test certificates for legitimate in the wild
domains. They also did it for thousands of domains that were
unregistered. And this is a huge separate issue. Because if I'm
able to get a cert for a site that you may register later, or
domain that you may register later, and then I already have
the valid cert, you can see I could be abused for man in the
middle activity. I might actually have a cert for a
domain that you own and you don't have any cert for the
domain. Imagine that. That's wow. Symantec says these test
certificates were never actively in use because they were not
in use. Google, of course, was able to discover it. There's no
legitimate purpose for this. And particularly for a site as big as
Google.com. There's no legitimate use for this. There's
no legitimate use for this. By the way, I feel it's probably
appropriate to mention that Symantec, the owner of a very,
very large root CA who has cross signed certificates against
guidance without reporting previously, now also owns some
very interesting HTTPS interception software stuff.
Anyway, I'll just throw that out there. I think they reached a
deal with Blue Coat last year. I think they reached a deal with
Blue Coat last year. Wow. Gosh. That just hit me. Anyway. So
improperly revoking certificates. So Symantec, even though these
weren't related to Google, Symantec this month, so I want
to put this in perspective. Because if you don't already
feel like Symantec is a dumpster fire, and again, lawyer
in the room, I know you're here. I know you're here. I'm not
calling Symantec a dumpster fire. I'm just saying if these
people aren't convinced that it's a dumpster fire, by now,
maybe this will convince you. Because while Symantec's already
on probation, because this whole thing kicks off in February, in
early July, a gentleman named Hanno Bock, I think Bock. I
don't know. It sounds German. Anyway, he went and bought some
certificates for a couple of domains that he owns. He went
out and registered a couple of crazy named domains and then went
and bought some certificates. And so there are four domains
that he owns. And he bought some through Symantec and he bought
some through Komodo. And I think there was one other registrant,
one other certificate provider he bought them through. And then
he did something cool. He went and he revoked the certificate.
He said, hey, my keys have been compromised. I need you to
revoke the certificate. He used a different e-mail to contact
them. Basically said, hey, here's my private keys. I need
you to revoke these because they've been compromised. Now,
a CA, according to the browser alliance, has 24 hours upon
notification of compromise of a private key to revoke the
certificate. The kicker is they're supposed to validate the
private key. And see if you can figure out where this is going.
Oh. Oh, sorry. A gap was identified. This is Symantec's
response. A gap was identified in the public and private key
matching process, also known as we forgot to validate the key.
And what this means is that while Symantec is unaware of
anybody having done this in the wild, that up until mid-July, up
until two weeks ago, you could have sent them something that
had some random binary string and said, hey, this is the
private key for blah domain. It's been compromised. I need you
to revoke the cert. Now, there's math that you can do to validate
this. Symantec knows all about this. That's how they create the
certs in the first place. I presume they know about this.
That's how they create the certs. Actually, you know what?
Forget it. I don't even presume they know this anymore. But,
look, that's how they create the certs in the first place. And
you can mathematically validate that this private key is the
real private key. And so picture the power that you have when you
had in your hands to shut down operations. Because suppose for a
minute that we take a large provider like Amazon or
something and we go and validate their cert, their private key,
do it a private key compromise. Now, I think Symantec would have
been smarter not to do it for Symantec, but you can picture
other, or sorry, Symantec wouldn't do it for Amazon. They
seem pretty big. Seems like somebody might have gone,
really? Amazon? Seriously? They don't want a key reissue or
anything? I think they'd have been smarter to see that, right?
But there are some mid-range things that you can do to
manage sites that I think you could do some really serious
damage with here. Obviously nobody has gone back and tested
this again. I'm not encouraging you to test it again because it
sounds all like fraudulent activity to me. But if you're
so inclined, who am I to stop you? So, no joke, though, by the
way, this is a neat attack vector. And it's something that
you ought to try with whoever, wait, you should talk to your
general counsel and then you should try this with whoever
your SSL providers are. Because this is something that you should
be serious because once they revoke the cert, everybody gets an
SSL warning, you know, this is a brand damage issue or possibly a
business loss issue. So, again, it's something that you should
check, something you should know. So I want to talk about
trust issues because I have lots of trust issues. This gentleman
in the front row does as well. And I want to talk about who is
your browser trusting anyway? Because the entire SSL model is
based on trust in a lot of certs that I think if you looked at the
people who own those certs, I think you would look and go, for
instance, and I want to put this into perspective, when I consult
with clients, and feel free to steal my analogies here, when I
consult with clients, I open up the list, we talk about this a
lot for the browser trust, and we open up the list of
certificates and we look in Firefox and we look in Chrome and
we just say, look, would you hand them $1,000 and trust that
in six months you're going to get it back? And they're like, no, I
wouldn't. I'm like, good. Then pull trust. Because what you're
asking for is trust. And I'm like, no, I wouldn't. I'm like,
no, I wouldn't. There is something way worse than that.
You are asserting that these people can sign trusted
certificates for literally anything. Many nation states, as
this gentleman correctly pointed out, have virtual control over
businesses operating on their borders. This does not align
well with very complex threat models that most organizations
have. Now, I know what you're thinking. If you're a Chinese
telecom, I'm just going to use China as an example, because
it's always China. Actually, recently it's been always Russia,
but whatever. Chinese telecom, I'm going to use China as an
example. If you're a Chinese telecom and you go sign a
certificate, how does my traffic, if I'm just a U.S.
company, how does it even get there in the first place? Before
we talk about that, let's talk about a couple of concerning
ones, because Hong Kong Post does not look like somebody that I
want to have. Why does the Hong Kong Post Office operate a C.A.
that Firefox in the United States trusts? And is this
necessary? And should you leave this talk and remove this from
your browser for your set of accepted certificates? Right?
Totally. And I feel fine saying that. I don't think it's a good
idea. If there's a lawyer from the Hong Kong Post in here, game
on, right? So, look, I'm just not down with this. Is it being
used? Yeah, absolutely. 38 of the top 1 million sites are
secured with these. That's not the issue. If you're looking at
that and you're hearing only 38 and 1, that's not the issue.
It's not that it's 38 and 1 million. They can sign certs for
anybody. That's the whole point of the trust, right? You are
trusting them to sign a cert for literally anyone, right? And so,
of course, then if we get that man in the middle, we're going
to have a lot of trouble. Well, you get the idea. Okay. What
about the Chung Wah Telecom in Taiwan? Right? Again, this is
another CA trusted by Mozilla and all the major browsers.
Microsoft actually goes a step further because they're cool and
they trust the China Financial Certification Authority, the
CFCA. I don't even know who these people are, but they sound
bad. And I don't mean bad from a I don't like China, just China is
part of my threat model because like half the intrusions I do
are China, right? And it's just a reality. I operate here and I
operate in Southeast Asia where, as you can imagine, China has
got a lot of vested interest in what's going on. And, yeah, I
mean, so is it fear mongering, right? No. No, this isn't fear
mongering. People have accused me of this. It's definitely not.
I'm a low FUD, fear, uncertainty, and doubt kind of
person. You combine this trusted CA with the ability to do man in
the middle. If you're in a foreign country, ISP, man in the
middle, should definitely be a thing you are concerned with.
DNS cache poisoning. I don't know if Dan Kaminsky is here. I haven't
seen him tweeting yet since he's been here. If he is, don't fall
asleep near Dan. That's a losing move. Or pass out near Dan.
It's a losing move. Dan's an awesome guy, though. And Dan
walked through a number of years ago DNS cache poisoning. And, of
course, vendors rushed out to fix that over the last several
years. And so he presented it here like in what? Was it 2008?
2007? All right. So it's been nearly a decade since Dan
presented how to do DNS cache poisoning. And trust me, it works.
And so as we look at this, with DNS cache poisoning, they can
route traffic basically from you in the U.S. out to wherever it is
they need to. Or, realistically, they don't even need to route it
outside of the U.S. They just pick up a server in Amazon, the
Amazon Cloud or in Azure and throw one of their trusted certs
there and game on. But if you want to get into the really,
really scary stuff, take a look, my friends, at BGP hijacking.
Because this is where stuff gets crazy scary. And you talk a
bout the SSL trust thing being an absolute dumpster fire, BGP is
far worse. And if you're not looking at this, go take a look.
I don't know how I missed the E there. That's probably the
British spelling. Anyway, BGP route hijacking is probably the
easiest of these to track. If you're not following BGP stream
on Twitter, go check this out. It is an absolutely awesome
account. And it tracks when somebody either intentionally or
accidentally steals your data. So if you're not following BGP
rules, it just automatically steals prefixes in the BGP
system. Basically when some router somewhere says, oh, hey,
I own these, go and route that stuff over here. Sometimes I'm
sure that's a mistype. Sometimes I'm sure that's an actual error.
It's not a typo. Just a flat out error. And typos happen.
Right? Amazon took down a big chunk of S3 earlier this year
with a typo. That was obviously them having a typo. But there's
also some really interesting stuff. We saw all the traffic in
Europe. Germany is a big fiber switching hub. We saw basically
all the traffic that goes through the route through St.
Petersburg for about an hour earlier this year. That was just
accidental, I'm sure. It was no big deal. And so we've seen
stuff route from Japan or Korea into China. We've seen a lot of
really weird temporary disruptions in the force, if you
will. And again, this is based on a trust model. When you
combine this with certificate authorities, you have the
ultimate power for man in the middle. Because a lot of our,
when we talk about controls for this stuff, for BGP route
hijacking, for DNS cache poisoning, it's that you don't
own a legitimate cert. But here you do own a legitimate cert or
the equivalent of one because, again, we're trusting some of
the wrong people, in my opinion, to sign this. So they could
steal secure only cookies, harvest credentials, exploit
user trust, serve malicious updates to software. We've seen
that recently? Yeah, we have. That's right. So look,
technology is the ultimate power for man in the middle.
This is where I had to rework a huge chunk of my presentation
because freaking Google. Anyway, so they came to an agreement and
the tentative solution says that basically, first off, it's
important because it allows both sides to claim victory.
Symantec can say, you didn't totally shut us down. We can
still issue certs, sort of. And Google can say, we found issues,
we corrected the issues, there was some punitive action taken.
Everybody else in the room, you're definitely not going to
get away with this. Don't try this if you're another
certificate authority. This is ridiculous, right? They're letting
Symantec off with at best a slap on the wrist, but game on. The
proposed solution has three phases. First off is register a
subordinate CA. Then there's a partial followed by a complete
distrust of all Symantec certificates. So Symantec is
going to partner with another CA no later than December 1, 2017.
This original timeline, we were supposed to be done trusting
Symantec certs. So as of February, we're supposed to be
done by December. So we're supposed to be done by December.
Here we've negotiated a lot. And look, over time, between
February, or sorry, between March and now, things have only
gotten worse, meaning we only know more and more and more
about mis-issuance and other issues with Symantec. So the
picture has gotten worse. And then simultaneously, we've moved
the timeline back. And this does not sound like a winning piece
for security. This sounds like a lot when we talk about like
security plus and CISSP, where we've compromised basically
convenience for security.
And that's effectively what we've done, but cool.
Regardless, Symantec's going to partner with another CA, and the
subordinate CA is going to be entirely responsible for issuing
certificates. And I do give Google mad props here, because
they say, look, if Symantec gets out of line and starts issuing
certificates, or if they press a button there and they issue
certificates on your behalf, it's your butt. We'll delist you
too. So that's good. I like that. And they basically say, hey,
look, you're entirely accountable. Now, I don't know what
this actually means. I don't know what this actually means, but I'm
I say accountability for the win, but I don't know what
accountability means in this case, because realistically,
Symantec isn't being held accountable for this. They're
being stripped of their ability to issue certs on their own, but
they're being given a way out to continue making money issuing
certificates. And I don't see this as a real punishment, you
know, compared to, you know, compared to the crime. So, and
by the way, lawyers, it's not a crime. It was a phrase of
speech, right? So tentative solution phase two, partial
distrust. April 28th, we're going to talk about the
fact that by 2018, Symantec certificates issued before June
1st are going to start displaying SSL errors. The
thought is that by 2018, most of these certificates are going to
expire anyway, and so this shouldn't be an issue. I'll
talk a little about the reality of this a little bit later. In
October 2018, Symantec certificates issued before
December 1st, 2017, now will start showing SSL errors. So
the idea here is phase one comes in, we've got the older piece,
and then phase two comes in and we basically completely
distrust Symantec certs, except for the ones signed by or issued
by that new subordinate CA. So I did some analysis here, and I
started with the Alexa top one million sites, and we went and
retrieved SSL certificates, and we can start to look at
certificate changes over time. We grabbed the full certificates,
we wanted to allow the analysis of the data that we didn't think
was relevant before, and I'm happy I did this because what I
thought was relevant at first was not the full set of data
that I wanted. There's a number of limitations of the
standard approach, though. Number one is that I didn't make
any attempts to discover HTTPS on anything but 443. So if you're
running on some nonstandard port, I got nothing. Only HTTPS
certs were examined. There are certainly other certs that
exist, secure SMTP, you name it. Websites with SSL certs that
aren't in the Alexa top one million, I know a lot of these
exist. My mom registered something on GoDaddy recently,
and she has an HTTPS cert. I don't have it because she's not
in the Alexa top one million. She's not that interesting.
Symantec is the most popular search engine in the industry.
They're in the U.S., or is most popular in the U.S., but the
Alexa top one million is international. And I think
they're probably, Symantec is probably underrepresented in my
data set, but I don't know how to fix this. So I kind of
kicked around some ideas originally, and then we finally
said forget it. We'll just go with what we know is an
imperfect solution. Like all academics, document the
limitations and go, go, go. How prevalent is SSL? Over the top
one million, 60 percent. So 625,000 had SSL certs, and
as of two days ago, three days ago, and we've been continuously
updating this as we go, 112,000 of those were signed with
Symantec certs. So originally the thought was, and I think the
hyperbole that we saw initially was one in six certs in the
Internet is signed by Symantec. It turns out that's actually
close to true. And this may be the case that it's even
disproportionately the top one million, because that's better
than one in six. Symantec's actually not the biggest in the
Alexa top one million. I thought it was the biggest in the Alexa
top one million, but I thought they would be. Komodo actually
accounts for slightly more certs, GlobalSign and GoDaddy. I
cannot fathom getting a GoDaddy cert, but okay. I guess Danica
Patrick comes in and signs those herself. Regardless, when we
took a look at the web, basically Net Trust, I think, or
NetCraft, SSL survey, you can see our numbers are kind of
falling in line, give or take, and NetCraft doesn't use the
Alexa top one. I mean, I'm sure they do use the Alexa top one
million, but they've got a ton of other sites that they're
indexing as well. Meh, I didn't know that. So we're going to
talk about that. I didn't do that because I was lazy. But you
can see here Symantec has a bigger slice of the pie
according to what they're reporting than what we're
reporting. We're showing about one in six. They're showing
about, it looks like, one in three-ish, give or take. So we
talk about how prevalent is Symantec and how do they break
out, because Symantec has a lot of CAs. It's not just Symantec.
This is one of the problems. If you take nothing else away from
this talk, go back and talk to your business, not just your
business, but your business partners or your business
partners, all the people that you coordinate with. And I know
some of you in here are working for Fortune 500s and you're
coordinating with these little mom and pop shops. And you
probably have this taken care of and they do not. It's your job
as good Internet citizens and for your security to actually go
and kind of follow up on this. Symantec operates a lot of CAs.
It's something like 15 or 16 different CAs that we found as
we started doing this research that are signed by Symantec
Roots. This is a huge deal because all of them are
going to be impacted. As we look at this, probably the
biggest two are GeoTrust and Rapid SSL, neither of which bear
the Symantec name directly. So there are a lot of people here.
You can see Symantec is way down on the list. And we've
talked to several clients who said, no problem. Thanks for
bringing this to my attention. We don't use Symantec. And we're
like, ha, ha, ha, ha. Actually, you do, as it turns out. And
they're like, we use Rapid SSL. I'm like, let me explain to you
how PKI works. Come over to the white board here real quick. And
we kind of draw out and show them how it works. And they're
like, oh, I'm going to show them how indeed this is signed by
Symantec. And it is going away, too, at exactly the same pace.
So my question, my theory here is that Google was really going
to shake Symantec customers. Because if I'm in the Fortune
500, if I'm in, forget Fortune 500, if I'm in any business, I'm
not buying a Symantec cert at this point because of
uncertainty. I don't like uncertainty. If I can go with
somebody else that I have a good feeling about who's not already
in the penalty box, or I can go with Symantec, I would go with
somebody else. That's just how I feel. So, I'm not going to go
with Symantec. It's just me. It's not advice, by the way. It's
just me. And we did some analysis here. We did our first polls in
the very beginning of April. We look at 112,000 as of two days
ago. 115,000 in April. It's 2.3% market shift. And
initially I thought, wow, that's huge, right? 2.3% of your
business is actually pretty significant. I'm going to tell
you that I thought it was going to be more. I thought by this
time we'd be looking at somewhere in the 10 to 20%
market shift, to be honest. I thought 2.3%. And it turns out I
was extra, extra disappointed because it's not really 2.3%.
I'll talk about that in just a second. The numbers are kind of
lying here. I'll tell you this is less than I would have
predicted. I don't have data on normal SSL certificate market
shifts. I don't think anybody does. Well, some of the CAs
probably do. But I don't know over time how do people change
certificate authorities. I mean, the lazy thing to do is literally
just to pay them again and get a brand-new cert issued. You don't
change anything. It just kind of works. It's the easiest
possible thing to do. It's why I think so many people have
domains registered with GoDaddy. They went and they registered
once and the easiest possible thing to do is let them charge
their credit card again. Game on. A lot of people moved away
from Symantec, but other people moved to Symantec. And this, my
friends, is absolutely mind-blowing. I think, although I
have no data to support it, this is the problem. Because the
majority of them that moved came in via rapid SSL. And I think
that's probably what we're going to have to deal with in the next
look at some of the other things that we're seeing there is
people are, if they're cognizant at all of the Symantec issue,
they don't understand that these are also Symantec. So, literally,
if you take nothing away from this talk, one, these are
Symantec, right? And two, Symantec signed a federal
government cert that they could use for man-in-the-middle for a
period of five years, then failed to do anything about it
for five months, right? That's probably the other thing that I
would take away from this, because I don't know. Anyway. So,
moving away from Symantec, we saw 4,400 domains move in that
period. Where did they go? Largely Komodo and Let's
Encrypt. I'm actually really excited about Let's Encrypt.
They sign a lot of certs for malicious domains. I mean,
that's just a reality. A lot. By the way, if you're looking to
do good cyber threat intelligence type stuff, early
warning, go pull the certs that Let's Encrypt is, they're very
transparent about it, the domains that they're signing
certs for, and look, because you can see a lot of them, like,
grep for the words PayPal or iPhone. And, brother, there are
some crazy numbers of domains that they're signing certs
there. So, while Let's Encrypt does a lot of malicious signing
there, they do it the way they're supposed to for the
browser alliance. They're verifying that you own the
domain. The fact that the domain itself you probably
shouldn't own is a separate issue. They don't deal with
that. So, as far as the movement to Symantec, where do
people leave from, right? Turns out a lot of them left from
Komodo. A few of them graduated from Let's Encrypt. But,
overall, woe sign, right, now that they're untrusted. Some
people finally got on the bandwagon between April and
July. So, let's take a look at that. So, let's take a look at
what happened in July. Realized they've been running on an
untrusted cert for a number of months and said, hey, where
should we go? The next untrusted cert, right? Anyway, so
shouldn't certificate lifetimes be going down, right? We saw in
the beginning where the initial plan, the initial plan, as it
were, was to lower the lifetime, right, of these certs.
I would expect, then, that as people go and renew these certs,
that Symantec would be, like, jumping through hoops to tell
people, like, hey, don't register for three years,
right, because we can't support that because, because, you know,
the Google plan says so. When we started, the certificate
lifetime was 656 days. In the July data set, it drops to a
whopping 652. I am thoroughly disappointed here. I really am.
I'm extremely disappointed that we're not seeing a decrease in
the lifetime of the certs. And, again, I can only ascribe two
things to this, right? One, people don't understand they
have Symantec certs, although I have to say, as people are
reissuing certs through normal expirations, it really irks me
that Symantec isn't advising customers, and maybe they are,
I don't know. Customers could be ignoring them. But it would seem
to me that Symantec would be, like, hey, this is on the
horizon, and we think you might need to deal with the
possibility that this is going to happen. So. .
No, I know what they're telling their customers. So,
the comment here is they're telling their customers, we're
too big to fail. We're not going to fail. Don't worry about it.
I've seen that correspondence through my own customers. It's,
yes. Anyway. So, I think, you know, I think, you know, we're
doing this better. How can we not screw this up again, right?
Look, it's easy to ignore the potential impact. People come
back and say there's not a single example of malicious activity as
a result of this. That's fine. It's fine for you to say you
don't have proof of any of this activity, but we never will,
right? This is kind of one of these spots where we don't know
what the damage is, and we likely never will. I don't think
we can say it's okay, nothing happened here, because, again,
we don't know if anything actually did happen. One, and
two, the risk here is the entire Internet ecosystem. And I'm not
trying to do hyperbole here. It really is the entire Internet
ecosystem. This is not our first certificate fire drill. We dealt
with this in Heartbleed where we had to invalidate lots of
certs. And we sucked at it. We did. And here again, we're in a
position where one of Symantec's arguments consistently over the
last several months has been we can't invalidate and reissue
certs. The global ecosystem is not set up to do this. And
they're right. That's the one thing I'm going to back up on.
They're right. Our ecosystem is not ready for this. We need to
plan for dealing with mass certificate revocation and
reissue. Because if it's the CA that's being untrusted, you need
then to validate yourself with another CA. This takes time, it
takes resources, and CA's don't just scale orders of magnitude
larger overnight. So possible solutions. These are admittedly
flawed, but they're possible solutions. We could have a
trusted third party pre-validate organizations in case a CA shuts
down unexpectedly. Kind of like a global brain trust of CA's. We
kind of have something ish global browser alliance type thing. Could
we have this too? I think that's ripe for abuse as well. So I don't
necessarily recommend that. But it's a thought. We could buy
multiple certificates and have them on standby in case our CA gets
shut down. So I could buy from Symantec and Komodo. I think this
is also a bad plan. I've actually had customers recommend, well,
hedge our bets, we bought an extra private key from Komodo. No,
no, no. Where is that thing? Is it online someplace? Is it an HSM?
Because I'm worried about keeping control. If it's hard to control
your keys as it is, doubling the number of keys makes it twice as hard.
That's just math. We could have third party CA SWAT teams that can
jump in and run a certificate authority when issues are discovered
or surge support is needed. So if Symantec or if the browser
alliance, if there were this magic, you know, magic magic
group that had a SWAT team, they could have jumped in here and done
this. Of course, realistically, all these cost money, all of them
are going to increase costs. I don't have an answer here. Somebody
smarter than me is probably sitting in this room who does have the
answer. Totally let me know what that is after we're done with the talk.
So I'll give you a couple parting thoughts here. It's 2017. We should
not be talking about this. We should not be talking about this. We
definitely shouldn't be talking about this months after the initial
release. Symantec should not be telling their customers it's okay,
we're too big to fail. We should not be talking about this months
after the release. I would as a personal thing, not as a legal
thing, Mr. Lawyer, whoever you are. As a personal thing, I think
it's abhorrent that Symantec is talking about the too big to fail
piece and not talking about the very real security issues that have
been introduced. I think that that's just bad for the community
overall. I will also say that Google's handling of Symantec is not
good either. I think Google deserves to be in the penalty box here.
They identified a security issue. They have the power to take action
and they are valuing user convenience and customer convenience over
security. And we in InfoSec routinely tell people don't do that. There's
a hardware hacking village with the voting machines right down the hall
here where that's the whole reason it's there is because of this garbage.
Right? Symantec should have been rapidly untrusted just like woe sign.
Forget this too big to fail stuff. It didn't work for the government,
didn't work for the banks, it's not working for CAs either. And then
finally, review the CAs that you're working for. You're working for the
government, you trust in your browsers. This is nuts. Most of them don't need to be
there. And the way you can convince your management of this is run Bro. Bro
network security monitor, NSM. Go through, you can configure Bro with Bro
scripts to log all of your SSL certificates. And find out who's signing those things.
And if you go through over a 30, 60, 90 day period, whatever works for you, 30 is fine for me.
And you look and you find out there's only five or six of those trusted CAs signing the
certs that you are actually using, get rid of the rest. That's what I'm saying.
Maybe you cause one pop up someplace. I will tell you, and I can't share the stories here,
but I will tell you, and somebody else sitting in the audience saw an example of this,
where we actually are fairly confident that based on some good net witness data that we got
popped like this. And it was a big company, it was a big target, and they were having trouble
getting in and we're pretty sure that's how they got in. Because something served a certificate
that was trusted but not trusted. And I'll just leave it there. And I'll just leave it there.
That's it. I got like five minutes tops for questions. Sir, speak into the mic,
because otherwise they can't. There's a mic right there. I think it's a mic, but.
Hello? Yeah, that's good. Okay, it works. So there was a slide that mentioned that
performed a modular comparison between the private key and the public key in the cert. Yep.
To validate, but they did not do something else. What is that something else? Do we know?
Oh, no, that was it. They actually didn't validate the private key. So the question is did
they, what else didn't they do besides validating the private key? We're talking about here.
Yes, this one. It says perform the modular comparison, which I thought was the whole key of
this mathematical comparison between the public and the private key. What did they not do?
I, it's my understanding they were doing some mathematical shortcut. I know whatever they did
is not a full comparison of the key. Because what he sent them was not the private key.
. Yeah, so they validated the private key, but they didn't do anything else. So
they validated the module as part of. Okay, so just to preserve this on the video here.
Again, basically that what we're thinking here, although I don't know that we're 100% sure in
this case, is that they generated another key with the same modulus. They validated the modulus
rather than validating the key itself. Again, you can't fake that. If you can, I totally want to
work with you, because that's going to be awesome, right? We're going to get stuff taken down. Sir?
Do you know if the, if the, if the, if the, if the, if the, if the, if the, if the, if the,
if the revocation of the U.S. Federal Government Certificate actually had any effect? My
understanding the revocation process is kind of not working as we wanted it to. Yeah, so did the
revocation of the U.S. Government Cert have any effect. The revocation process definitely is not
working the way it's supposed to. The cert eventually expired. So it's less of an issue now
than it was. But no, you're right. The revocation process is not working the way it's supposed
to. I don't think it had any effect. It eventually expired off anyway. But it was a cross-examination
signed route and that was one of the arguments that Symantec made
is that a lot of government sites were signed with this and
revoking could potentially cause issues there. Again, I don't
think they understand the math behind this. Or some of the
people making arguments don't understand the math behind it.
Sir? What are your thoughts on cert pinning as a possible
solution? So what are my thoughts on cert pinning as a
possible solution for some of these issues? I will say that
cert pinning is neat, but from a blue team standpoint, I need
SSL interception. I need SSL interception. If you're doing
cert pinning, I consistently then am overriding that in group
policy, I'm overriding that in browser settings because I need
to be able to do SSL interception. I can't run a large
organization today and do data loss prevention without being
able to see into your Gmail. This is just a reality. If I
can't see it, the attackers know that they can use that.
John Strand and his team over at Black Hills put together Gcat.
I don't know if anybody has seen this, but it's like Netcat except
it does basically data exfil. It's a free framework for doing
data exfil and command and control over Gmail. What's that?
I thought somebody mentioned Gcat. I did drop smack a couple
years ago. Same thing with Dropbox. So if you're doing cert
pinning, that's great. It solves one problem but creates a brand
new one because we can't see into that and attackers are going to
use those types of techniques.
I don't have the right answer here, unfortunately.
Couple of things. With the modulus, I'm not sure you can
actually generate another key unless you've got the totient.
If you've got the totient, you can factor in the first place
and get anything, right? I am not 100 percent sure on the
off. If anybody has PNQ, they can make any key. Totally
understood, right? And again, this is a spot where his blog
post is not 100 percent detailed for obvious attack
repeaters. I don't have the right answer here, unfortunately.
But he confirms that he sent them not the private key and they
revoked it anyway. That's the key. Not to overload the word the
key, but that's the point is they revoked a key they shouldn't
have. Symantec admits as much. I think they were doing some
mathematical short cut there that is not the full analysis.
Then RSA security, if we get quantum or advances in factoring,
we have to be better handling certificate revocation generally
and shouldn't that have been baked in in the beginning? Yeah
sure. So if we have advances in factoring, kind of repeat this
for everybody, if we have advances in factoring, you know
RSA basically, a lot of the security kind of rotates around
that. We have to get better at revoking this stuff in the first
place. So I agree with that wholeheartedly. You're right.
Obviously the SHA-1 apocalypse taught us a lot about that. We
accelerated the death of SHA-1 for certificate signing because
of advances in, well not factoring specifically there, but
advances in combinatorial mathematics there. We're looking
at the collision vectors in SHA-1. So again, I think you're
100% right on that. We have to have a better way to do this and
we don't. Okay, I think it's a good spot to cut it. I'll hang
out here if anyone wants to talk, like in the back or out
in the hall or whatever. Thanks for coming. Appreciate your
time. See you next year.