Thank you for being here on this wonderful Sunday afternoon
at DEF CON and the Crypto Privacy Village.
Our last talk of the day is going to be by Scott Arsazewski,
and he's talking about the why and how
of secure automatic patch management.
So let's give a nice, warm welcome.
Thank you.
So a little bit about me.
I mostly do application security work,
which basically is a fancy way of saying
I'm a programmer who kind of knows security.
I'm kind of known in the community
for getting Libsodium into the PHP core in 7.2.
I do a little bit of security research here and there.
You might have seen my name on Full Disclosure a couple times.
Today I'm going to talk about
why and how to implement secure,
automatic updates or patch management,
as you'd probably hear in a corporate memo.
And the best place to start
is to look at the life cycle of vulnerabilities,
as I call it.
So generally, one of the vulnerabilities found in software,
that isn't the start of the story.
The start of the story is when a programmer
either made a mistake or built something
without understanding the complete implications
of what they were building,
and they introduced a vulnerability.
At some point in time,
it was discovered by somebody else,
whether it was a white hat or internal.
It gets patched, and then the patch gets released.
That's the general life cycle,
like egg, larva, butterfly, and then death,
you'd see for security bugs.
Most people and most processes in the application security
part of the industry focus on the first three steps.
Preventing them from getting written in the first place,
that's where like BlankSec and a lot of the usable security,
initiatives are focused.
You have like automated testing tools,
static analysis that focus on discovering them
if they already exist.
And then you have the people who specialize
in finding and patching the vulnerabilities.
But once you get to step three,
it's kind of out of the hands of the vendor
because they only control the code.
They don't control the people who have deployed it.
And this has actually been
kind of a catastrophic problem in the past.
A couple years ago,
there was a severe vulnerability
in Drupal.
And within seven hours,
if you didn't have it patched,
you should assume that your system was compromised.
And I believe that was a remote code execution vulnerability
that was wormable.
There's another incident earlier this year,
a news article was published about Magento,
which is an e-commerce platform,
where somebody who had done research in that area
found that there were over 6,000 shops
that had been unpatched after two years.
And they estimated 21,000
credit cards were stolen
from people exploiting these out-of-date systems.
And then there was the recent incident with WannaCry,
which I'm sure everybody's still sick of hearing about,
so I won't touch on that too much.
So there's a lot of reasons for automatic updates.
The most one is you want to eliminate old days,
as some people call them.
If you have, if you've read the,
I think it's the Verizon data breach incident response report
from several years ago,
they kept saying,
that 99% of the breaches they saw
were from people using old vulnerabilities,
some of which were 18 months old.
So if you manage to deploy automatic updates,
the upside is that if you're,
if somebody wants to break into your system
and everybody's running the latest version,
they need a zero day.
And a lot of the script kiddies and carters out in the world
don't have the skills or the patience
to build, to find and develop an exploit for one.
So that increases the cost of an attack.
And there's also a usability win.
Avi Duglin, big person on Stack Overflow
entering security questions,
does AppSec Israel,
has a saying,
security at the cost of usability
comes at the cost of security.
And we've seen that from people who, like,
will implement AES 128 in ECB mode
because they don't know what the hell a nonce is.
They'll just use the defaults.
So, you know, two use cases,
Chrome updates itself,
Magento, you have to go through a long process
to get the patch file from them,
and then you have to manually apply it.
It's not even like some forum software
where you press the update button
and then enter your admin password.
You have to go into their corporate portal
and download the patch after clicking through menus.
So, the good of an automatic update system
is no more unpatched one day vulnerabilities in the wild.
The bad is, what if the update server gets hacked?
And I ask these questions to a lot of open,
open source projects,
and either their eyes glaze over
or they get very quiet
because they don't have an answer for this.
A very bad example would be WordPress.
A proud system that powers 27% of the websites on the internet,
their automatic update system is only secured by HTTPS.
If you manage to hack their update server,
you get 27% of all websites on the internet for free.
If I was a nation state or a criminal operation,
I would say that is worth burning a single zero day
to get into that server.
And,
and,
and this discussion also leads to a dangerous rabbit hole
like trusting trust and, you know, PKI stuff
and the subject of many cypherpunks mailing list debates
back in the day.
So, let's build a system that does automatic updates
but isn't vulnerable to this attack
or very similar related attacks.
And specifically, compromised update servers,
like we mentioned,
somebody hacking into the version control software
and replacing the code in the code base.
The suppliers or vendors themselves having their computers hacked.
And or the signing keys being stolen
because we're going to be using signatures.
So, here is a system that I believe would provide secure automatic updates
and be very difficult to attack.
The first element is you have to deploy it over a secure channel,
like TLS or noise.
Some operating systems like Debian, they just ship
a release.pgp file and that's it.
I'll get into it later,
but basically, if you only have a signed,
like offline signed thing
and not like a transport layer integrity check
on what was there,
you can replay old messages.
It's a very trivial replay attack
and you can prevent systems from realizing
there's even an update.
And then you need to make sure the updates are authentic
and the first thing you need to do is,
implement cryptographic signatures,
which is what Microsoft does,
what Google Chrome does.
Basically, every major browser vendor
that doesn't touch the PHP community
because they don't really do crypto,
already does code signing
because they're usually delivering binaries.
You need reproducible builds,
so that way the binary you're delivering
is reproducible from the source code
at a given checkpoint.
And then there's this property I'm going to be talking about
in more detail, called user-based consistency verification.
So, if you have a user-based consistency verification,
if you've read the work of Taylor Hornby,
crackstation.net, DFU security,
he initially proposed this as a
triangle of secure code delivery
and me and him have gone back and forth
on what is needed, what is not needed,
and how to implement it several times.
So this might be familiar with you
if you're familiar with his work.
And then you need the actual update process
to have a few things in mind,
like rollback in case something breaks.
That's a usability feature.
If you don't have that, people are just going to disable it
because they're afraid
something's going to break production.
And that, for some businesses, could be
hundreds of thousands of dollars an hour.
And then privilege separation,
that's always a good thing to have.
Don't give processes more power than they need
in case a vulnerability exists.
So, for cryptographic signatures,
we're only going to use things like
ED25519,
deterministic ECDSA,
or if we're going to go post-quantum,
something like Sphinx, which is a stateless
hash-based signature protocol.
Signed by the company or the
person who delivers the actual
software, not the update server.
It's an offline signature.
Make software reproducible.
And the user-based consistency
verification is
make sure that everybody sees
the same thing.
And this is very important.
If you had
somebody capable of hacking into your
computer and stealing your private key,
you're probably dealing with somebody on the same level
as a nation state adversary.
And if they're doing this to an open source project,
for example, they're probably
doing it as part of either a sting operation
or some kind of clandestine thing.
And
they're going to be very
unwilling to do anything where they have to
announce their presence.
And in addition, anything they do,
if they manage to pull off the first attack where they steal the key
and hack the update server and push out their
malware, it's detectable.
Because you have a permanent record
of it ever happening.
So,
moving along
a little bit quicker.
Because we're going to have a system where all
your users see the same thing,
you're going to have some communication
either between other peers
or between other servers
that mirror the data
that basically keeps
a log or ledger
of all the updates
that have ever happened.
And we've actually built something
called Chronicle.
It's an open source
service. It's on GitHub.
It's a hash chain data structure.
It's not like a Merkle tree.
It's almost like a linked list.
It has a couple
things that make it useful. You can quickly
cross sign your latest hash
onto other Chronicles and you can replicate
many of them onto your local one.
The crypto side of things are a little bit
more interesting. It's a JSON
API that uses a
specification I wrote called
Sapient, which is a backronym for secure
API engineering toolkit.
It basically
adds specific headers that have
base64 encoded signatures
that authenticate the entire message
body.
So that's how it communicates.
If you're trying to communicate with it
and you just have an API
adapter that speaks JSON but
doesn't know what Sapient is, it will be
backwards compatible. You just won't have the security
of having the signature verified.
Each message has
two hashes. A current hash
uses the previous hash as the key
for a Blake 2 hash.
Then you have a summary hash, which is literally
just, you concatenate every single
current hash.
That's used for
the verification
steps that you can do later.
So
when an automatic update is signed,
if you wanted to build an automatic update system,
what you would do is you would sign it using
your offline key, push your signature
and some other metadata like what the
version number is, what was the latest
git commit hash, and you would
submit that to a chronicle.
And after every end
messages or every end days or however
it's configured, it would cross-sign to their peers.
And as soon as a message is there,
anybody who's replicating that
would download, verify a copy
and they'd have a copy in their local cache.
So when you go to download
an update, your software should verify
the signature and then
verify that everything is
what they'd expect to see in the chronicle at that time.
If any of those fail,
it should fail hard
and alert them and
probably scream bloody
murder crying home.
And then software updates should be reproducible.
That's more of a C programmer's
area of interest than
crypto, I would think.
So I didn't really go into that too much.
We have a library called Pharaoh
because PHP has these deliverables called
PHP Archives. It's kind of like how Java
has JAR files. All it does
is diff two of them and tell you what's different.
And then, as I mentioned,
everything should be delivered over HTTPS.
There's metadata leaks,
you know, privacy concerns there
for not using that.
In addition, you don't want people
to just replay an old message
and keep people on a vulnerable
version of your software.
Privilege separation
and then availability.
So with chronicle, I designed it
so where you could have a replication
like a replicated copy
of a chronicle used to sign
projects updates.
The idea was that hosting companies
that have like 2,000 WordPress
clients, instead of hitting
our server or hitting some other server
for like a plug-in vendor, could just
have a copy they keep locally for all their
customers and benefit from lower
network latency and if there's downtime
upstream, their customers are unaffected.
You could do something similar for the actual
updates themselves.
You can just mirror them with trusted mirrors
that have their own public keys penned.
You could use BitTorrent to distribute it.
There's a lot of options.
Building a system like this does come
with some challenges, of course.
The supplier of the code has to be
disciplined enough to keep their
secret key from getting stolen.
They also need to make sure that their
API doesn't break to prevent people
from wanting the update to be there.
And that's a very important one
because one of the pain points
of some operating systems like Debian
and some of the other more enterprise-y
long-term support Linuxes
is that they don't have the
capability to do that.
They still ship an old version of
libsodium, still ship old versions of
PHP and they lock them to really weird
patch versions.
An old version of Debian shipped with
PHP 5.3.3 but they backported a patch
that fixed password hashing from
5.3.7.
So if you were trying to detect what
version you're running to where if you
could use the new password hash,
that became extremely unreliable.
So this has been a problem for the
PHP community in particular.
Operating systems pinning to
patch numbers instead of minor numbers
and then just backporting patches
and lying about what version you're on.
So there needs to be discipline among
the people who are developing the software.
And then anybody who's using like
a library in their own software
needs to not use anything undocumented
in case there's a breaking change.
Sometimes I make a breaking change
into the protected or private methods
in the stuff I write and release it
as a patch version.
And so far I've not had anybody complain about that
but a lot of people tell me they're afraid
to do that because they're afraid of the complaints
and having to do rollbacks and all that other
fun stuff.
So this isn't just a theoretical talk.
I've actually built this twice.
I have an open source CMS
called Airship.
It's free software.
I'm releasing version 2 with PHP 7.2
early next year.
It uses ED25519 signatures
but it uses a Merkle tree structure
instead of a hash chain.
I went with a hash chain with Chronicle
just for speed reasons.
It approximates the use case of
Git and certificate transparency
kind of meshed together.
I was going to release something
this past week.
It's kind of like a firewall slash IDS toolkit
but one of the things it does
is actually do automatic updates
for example those Magento sites
that haven't been updated in two years.
The idea is that people who use
Word, which is the name of the project
it has normal
boring, uninteresting
WAF IDS features.
It will keep all your software up to date.
If there's a security update
it will query our servers and say
hey, is this authentic?
And since we do all our signing offline
on a Raspberry Pi that never connects
to the internet,
the likelihood of that ever getting stolen
is pretty low.
Won't say zero, but pretty low.
So that's what we're doing there.
I'm working to get
the similar system built for the PHP
package manager called Composer.
Right now that's been kind of stalled
from complete lack of response
from the Composer team,
but I'm assuming that's because they're busy.
So the PHP community is moving
forward on this slowly.
I expect in probably a couple years
the talk I'll be giving is
how we did this rather than
what I want to do.
If anybody wants to build this
in another framework,
there's this thing called the update framework
and I forgot to make a slide for this,
but it's called the automatic updates.
All you would need to do is
make sure your builds are reproducible
and query a chronicle and you've got
a secure automatic update system
as described in the model here.
And that is all I have.
Does anybody have any questions?
Yes.
So one of the slides mentioned
that a more trusted user than the Apache user
should be responsible for the software updates
for the web applications.
Sorry, that was a little bit quick.
Okay.
So one of the slides mentioned
that a more trusted user than the Apache
is supposed to be responsible
for the software updates for the web apps.
Don't let Apache...
Oh, okay.
So that comes into a chicken and egg conversation.
So if you're in Harlem,
anything a hacked PHP script can do,
your normal PHP script can do.
And the point of privilege separation
would be to prevent the...
Somebody who found a vulnerability
that allows code execution
to be able to write to your file system.
So you usually prevent them from having
write access to your web route.
Automatic updates, by necessity,
require something to be able to write to the web route.
So you would need to have a separate user
that runs that process to make sure
that your normal web user isn't overwriting index.php,
if they found a way to do that
from some weird code path
and something's un-serialized
and something gets called under
the underscore destruct magic function.
Now their shell is at the very bottom
of index.php and it runs
if you pass a magic variable to get.
It's a defense in depth.
Airship doesn't do this.
We have a mechanism
to where anybody who wants
to do privilege separation can do that.
But by default,
we made the transition
trade-off to where automatic updates
and getting vulnerabilities fixed
was considered a higher priority
than preventing traditional views
on web hosting where everything
is locked away to where
the web, the process,
the user that has the privilege
to run the code isn't able to write to it.
Other people
will make different trade-offs
and it depends on your threat model.
Ours is more general and we tried
to go for usability on that one.
Is there any way to implement this
on shared hosting?
Yes.
But you would need
some kind of cooperation from them.
The other thing you could do is
see if they can run your web
processes as like nobody or as Apache
or some other user.
That's the only way I can see
to reliably do it because OS permissions
which is what the whole privilege
separation is supposed to do is tied into
OS users and I can't really think
of a way that would work without
having more than one user.
Yes.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
. .
. .
. .
. .
. . .
. . .
. . .
. . .
. . . .
. . . .
. . . . .
Well, thank you for having me.