1
00:00:00,000 --> 00:00:03,140
Thank you for being here on this wonderful Sunday afternoon

2
00:00:03,140 --> 00:00:06,480
at DEF CON and the Crypto Privacy Village.

3
00:00:06,700 --> 00:00:12,860
Our last talk of the day is going to be by Scott Arsazewski,

4
00:00:13,600 --> 00:00:16,200
and he's talking about the why and how

5
00:00:16,200 --> 00:00:18,320
of secure automatic patch management.

6
00:00:18,580 --> 00:00:21,400
So let's give a nice, warm welcome.

7
00:00:28,340 --> 00:00:29,620
Thank you.

8
00:00:30,880 --> 00:00:32,400
So a little bit about me.

9
00:00:32,720 --> 00:00:34,700
I mostly do application security work,

10
00:00:35,200 --> 00:00:37,480
which basically is a fancy way of saying

11
00:00:37,480 --> 00:00:38,900
I'm a programmer who kind of knows security.

12
00:00:40,140 --> 00:00:41,520
I'm kind of known in the community

13
00:00:41,520 --> 00:00:45,380
for getting Libsodium into the PHP core in 7.2.

14
00:00:48,600 --> 00:00:50,840
I do a little bit of security research here and there.

15
00:00:51,120 --> 00:00:53,700
You might have seen my name on Full Disclosure a couple times.

16
00:00:55,080 --> 00:00:56,440
Today I'm going to talk about

17
00:00:57,280 --> 00:00:59,240
why and how to implement secure,

18
00:00:59,620 --> 00:01:01,540
automatic updates or patch management,

19
00:01:01,780 --> 00:01:03,660
as you'd probably hear in a corporate memo.

20
00:01:05,180 --> 00:01:06,560
And the best place to start

21
00:01:06,560 --> 00:01:09,160
is to look at the life cycle of vulnerabilities,

22
00:01:09,420 --> 00:01:10,140
as I call it.

23
00:01:11,800 --> 00:01:14,460
So generally, one of the vulnerabilities found in software,

24
00:01:14,580 --> 00:01:15,900
that isn't the start of the story.

25
00:01:16,060 --> 00:01:19,280
The start of the story is when a programmer

26
00:01:19,280 --> 00:01:21,640
either made a mistake or built something

27
00:01:21,640 --> 00:01:23,800
without understanding the complete implications

28
00:01:23,800 --> 00:01:24,860
of what they were building,

29
00:01:25,020 --> 00:01:26,280
and they introduced a vulnerability.

30
00:01:27,240 --> 00:01:28,840
At some point in time,

31
00:01:29,620 --> 00:01:31,180
it was discovered by somebody else,

32
00:01:31,300 --> 00:01:32,900
whether it was a white hat or internal.

33
00:01:34,120 --> 00:01:36,400
It gets patched, and then the patch gets released.

34
00:01:36,840 --> 00:01:39,300
That's the general life cycle,

35
00:01:39,460 --> 00:01:43,240
like egg, larva, butterfly, and then death,

36
00:01:43,880 --> 00:01:45,420
you'd see for security bugs.

37
00:01:46,600 --> 00:01:50,180
Most people and most processes in the application security

38
00:01:50,180 --> 00:01:52,940
part of the industry focus on the first three steps.

39
00:01:53,580 --> 00:01:55,240
Preventing them from getting written in the first place,

40
00:01:55,460 --> 00:01:59,200
that's where like BlankSec and a lot of the usable security,

41
00:01:59,620 --> 00:02:00,380
initiatives are focused.

42
00:02:01,360 --> 00:02:03,560
You have like automated testing tools,

43
00:02:03,800 --> 00:02:06,080
static analysis that focus on discovering them

44
00:02:06,080 --> 00:02:06,760
if they already exist.

45
00:02:07,340 --> 00:02:09,500
And then you have the people who specialize

46
00:02:09,500 --> 00:02:11,120
in finding and patching the vulnerabilities.

47
00:02:12,580 --> 00:02:15,060
But once you get to step three,

48
00:02:15,200 --> 00:02:17,240
it's kind of out of the hands of the vendor

49
00:02:17,240 --> 00:02:19,660
because they only control the code.

50
00:02:19,860 --> 00:02:21,720
They don't control the people who have deployed it.

51
00:02:22,180 --> 00:02:23,760
And this has actually been

52
00:02:23,760 --> 00:02:26,140
kind of a catastrophic problem in the past.

53
00:02:26,600 --> 00:02:27,920
A couple years ago,

54
00:02:27,980 --> 00:02:29,600
there was a severe vulnerability

55
00:02:29,620 --> 00:02:30,500
in Drupal.

56
00:02:31,080 --> 00:02:33,520
And within seven hours,

57
00:02:33,640 --> 00:02:35,180
if you didn't have it patched,

58
00:02:35,180 --> 00:02:37,340
you should assume that your system was compromised.

59
00:02:38,320 --> 00:02:41,540
And I believe that was a remote code execution vulnerability

60
00:02:41,540 --> 00:02:42,460
that was wormable.

61
00:02:43,640 --> 00:02:46,140
There's another incident earlier this year,

62
00:02:46,200 --> 00:02:48,200
a news article was published about Magento,

63
00:02:48,400 --> 00:02:50,020
which is an e-commerce platform,

64
00:02:50,340 --> 00:02:53,240
where somebody who had done research in that area

65
00:02:53,240 --> 00:02:55,080
found that there were over 6,000 shops

66
00:02:55,080 --> 00:02:56,960
that had been unpatched after two years.

67
00:02:57,460 --> 00:02:59,600
And they estimated 21,000

68
00:02:59,620 --> 00:03:00,920
credit cards were stolen

69
00:03:01,460 --> 00:03:04,380
from people exploiting these out-of-date systems.

70
00:03:04,880 --> 00:03:07,120
And then there was the recent incident with WannaCry,

71
00:03:07,380 --> 00:03:09,420
which I'm sure everybody's still sick of hearing about,

72
00:03:09,560 --> 00:03:10,840
so I won't touch on that too much.

73
00:03:11,960 --> 00:03:14,640
So there's a lot of reasons for automatic updates.

74
00:03:15,260 --> 00:03:17,460
The most one is you want to eliminate old days,

75
00:03:17,600 --> 00:03:19,020
as some people call them.

76
00:03:21,060 --> 00:03:23,220
If you have, if you've read the,

77
00:03:23,260 --> 00:03:27,380
I think it's the Verizon data breach incident response report

78
00:03:27,380 --> 00:03:28,500
from several years ago,

79
00:03:28,700 --> 00:03:29,500
they kept saying,

80
00:03:29,500 --> 00:03:32,120
that 99% of the breaches they saw

81
00:03:32,120 --> 00:03:35,560
were from people using old vulnerabilities,

82
00:03:35,740 --> 00:03:37,340
some of which were 18 months old.

83
00:03:39,160 --> 00:03:42,420
So if you manage to deploy automatic updates,

84
00:03:42,560 --> 00:03:44,720
the upside is that if you're,

85
00:03:45,140 --> 00:03:46,580
if somebody wants to break into your system

86
00:03:46,580 --> 00:03:48,140
and everybody's running the latest version,

87
00:03:48,340 --> 00:03:49,160
they need a zero day.

88
00:03:49,580 --> 00:03:52,020
And a lot of the script kiddies and carters out in the world

89
00:03:52,020 --> 00:03:53,840
don't have the skills or the patience

90
00:03:53,840 --> 00:03:56,220
to build, to find and develop an exploit for one.

91
00:03:56,660 --> 00:03:58,460
So that increases the cost of an attack.

92
00:03:59,500 --> 00:04:00,780
And there's also a usability win.

93
00:04:02,280 --> 00:04:05,240
Avi Duglin, big person on Stack Overflow

94
00:04:05,240 --> 00:04:06,460
entering security questions,

95
00:04:07,120 --> 00:04:08,080
does AppSec Israel,

96
00:04:08,740 --> 00:04:09,480
has a saying,

97
00:04:09,680 --> 00:04:11,120
security at the cost of usability

98
00:04:11,120 --> 00:04:12,600
comes at the cost of security.

99
00:04:13,020 --> 00:04:15,180
And we've seen that from people who, like,

100
00:04:15,180 --> 00:04:17,860
will implement AES 128 in ECB mode

101
00:04:17,860 --> 00:04:20,320
because they don't know what the hell a nonce is.

102
00:04:20,460 --> 00:04:21,660
They'll just use the defaults.

103
00:04:23,080 --> 00:04:25,240
So, you know, two use cases,

104
00:04:25,540 --> 00:04:26,620
Chrome updates itself,

105
00:04:26,980 --> 00:04:28,940
Magento, you have to go through a long process

106
00:04:28,940 --> 00:04:30,260
to get the patch file from them,

107
00:04:30,400 --> 00:04:31,920
and then you have to manually apply it.

108
00:04:32,220 --> 00:04:34,240
It's not even like some forum software

109
00:04:34,240 --> 00:04:35,380
where you press the update button

110
00:04:35,380 --> 00:04:36,520
and then enter your admin password.

111
00:04:36,920 --> 00:04:39,320
You have to go into their corporate portal

112
00:04:39,320 --> 00:04:41,720
and download the patch after clicking through menus.

113
00:04:44,800 --> 00:04:46,940
So, the good of an automatic update system

114
00:04:46,940 --> 00:04:50,460
is no more unpatched one day vulnerabilities in the wild.

115
00:04:51,680 --> 00:04:54,660
The bad is, what if the update server gets hacked?

116
00:04:56,680 --> 00:04:58,800
And I ask these questions to a lot of open,

117
00:04:58,800 --> 00:04:59,380
open source projects,

118
00:04:59,500 --> 00:05:00,840
and either their eyes glaze over

119
00:05:00,840 --> 00:05:02,300
or they get very quiet

120
00:05:02,300 --> 00:05:03,620
because they don't have an answer for this.

121
00:05:04,380 --> 00:05:06,400
A very bad example would be WordPress.

122
00:05:07,000 --> 00:05:11,000
A proud system that powers 27% of the websites on the internet,

123
00:05:11,240 --> 00:05:14,300
their automatic update system is only secured by HTTPS.

124
00:05:14,820 --> 00:05:16,540
If you manage to hack their update server,

125
00:05:16,800 --> 00:05:19,440
you get 27% of all websites on the internet for free.

126
00:05:19,900 --> 00:05:22,920
If I was a nation state or a criminal operation,

127
00:05:23,260 --> 00:05:25,640
I would say that is worth burning a single zero day

128
00:05:25,640 --> 00:05:26,520
to get into that server.

129
00:05:28,040 --> 00:05:28,500
And,

130
00:05:28,580 --> 00:05:28,780
and,

131
00:05:28,780 --> 00:05:32,080
and this discussion also leads to a dangerous rabbit hole

132
00:05:32,080 --> 00:05:35,760
like trusting trust and, you know, PKI stuff

133
00:05:35,760 --> 00:05:41,800
and the subject of many cypherpunks mailing list debates

134
00:05:41,800 --> 00:05:42,880
back in the day.

135
00:05:44,000 --> 00:05:47,380
So, let's build a system that does automatic updates

136
00:05:47,380 --> 00:05:49,380
but isn't vulnerable to this attack

137
00:05:49,380 --> 00:05:51,260
or very similar related attacks.

138
00:05:51,840 --> 00:05:54,840
And specifically, compromised update servers,

139
00:05:54,960 --> 00:05:55,500
like we mentioned,

140
00:05:56,340 --> 00:05:58,420
somebody hacking into the version control software

141
00:05:58,420 --> 00:06:00,880
and replacing the code in the code base.

142
00:06:02,860 --> 00:06:06,300
The suppliers or vendors themselves having their computers hacked.

143
00:06:06,900 --> 00:06:09,020
And or the signing keys being stolen

144
00:06:09,020 --> 00:06:10,520
because we're going to be using signatures.

145
00:06:11,960 --> 00:06:17,660
So, here is a system that I believe would provide secure automatic updates

146
00:06:17,660 --> 00:06:18,980
and be very difficult to attack.

147
00:06:19,980 --> 00:06:22,980
The first element is you have to deploy it over a secure channel,

148
00:06:23,100 --> 00:06:24,200
like TLS or noise.

149
00:06:25,420 --> 00:06:28,140
Some operating systems like Debian, they just ship

150
00:06:28,140 --> 00:06:31,400
a release.pgp file and that's it.

151
00:06:33,100 --> 00:06:34,320
I'll get into it later,

152
00:06:34,420 --> 00:06:37,100
but basically, if you only have a signed,

153
00:06:37,220 --> 00:06:38,900
like offline signed thing

154
00:06:38,900 --> 00:06:43,200
and not like a transport layer integrity check

155
00:06:43,200 --> 00:06:44,460
on what was there,

156
00:06:44,680 --> 00:06:46,220
you can replay old messages.

157
00:06:46,620 --> 00:06:47,920
It's a very trivial replay attack

158
00:06:47,920 --> 00:06:49,340
and you can prevent systems from realizing

159
00:06:49,340 --> 00:06:50,240
there's even an update.

160
00:06:51,620 --> 00:06:53,840
And then you need to make sure the updates are authentic

161
00:06:53,840 --> 00:06:58,120
and the first thing you need to do is,

162
00:06:58,120 --> 00:06:59,360
implement cryptographic signatures,

163
00:06:59,840 --> 00:07:01,880
which is what Microsoft does,

164
00:07:02,000 --> 00:07:02,940
what Google Chrome does.

165
00:07:03,840 --> 00:07:05,800
Basically, every major browser vendor

166
00:07:05,800 --> 00:07:07,600
that doesn't touch the PHP community

167
00:07:08,120 --> 00:07:10,600
because they don't really do crypto,

168
00:07:11,860 --> 00:07:13,140
already does code signing

169
00:07:13,140 --> 00:07:14,600
because they're usually delivering binaries.

170
00:07:15,520 --> 00:07:17,040
You need reproducible builds,

171
00:07:17,300 --> 00:07:18,800
so that way the binary you're delivering

172
00:07:18,800 --> 00:07:20,560
is reproducible from the source code

173
00:07:20,560 --> 00:07:21,320
at a given checkpoint.

174
00:07:21,920 --> 00:07:24,200
And then there's this property I'm going to be talking about

175
00:07:24,200 --> 00:07:26,860
in more detail, called user-based consistency verification.

176
00:07:27,540 --> 00:07:28,100
So, if you have a user-based consistency verification,

177
00:07:28,100 --> 00:07:30,000
if you've read the work of Taylor Hornby,

178
00:07:31,520 --> 00:07:33,020
crackstation.net, DFU security,

179
00:07:33,640 --> 00:07:35,700
he initially proposed this as a

180
00:07:36,400 --> 00:07:37,920
triangle of secure code delivery

181
00:07:38,540 --> 00:07:40,340
and me and him have gone back and forth

182
00:07:40,340 --> 00:07:42,920
on what is needed, what is not needed,

183
00:07:43,040 --> 00:07:44,440
and how to implement it several times.

184
00:07:44,600 --> 00:07:45,720
So this might be familiar with you

185
00:07:45,720 --> 00:07:46,740
if you're familiar with his work.

186
00:07:47,300 --> 00:07:49,920
And then you need the actual update process

187
00:07:49,920 --> 00:07:51,160
to have a few things in mind,

188
00:07:51,240 --> 00:07:53,360
like rollback in case something breaks.

189
00:07:53,840 --> 00:07:55,000
That's a usability feature.

190
00:07:55,260 --> 00:07:57,440
If you don't have that, people are just going to disable it

191
00:07:58,340 --> 00:07:58,900
because they're afraid

192
00:07:59,580 --> 00:08:00,820
something's going to break production.

193
00:08:01,240 --> 00:08:02,920
And that, for some businesses, could be

194
00:08:02,920 --> 00:08:04,640
hundreds of thousands of dollars an hour.

195
00:08:05,060 --> 00:08:06,120
And then privilege separation,

196
00:08:07,240 --> 00:08:08,520
that's always a good thing to have.

197
00:08:08,660 --> 00:08:10,660
Don't give processes more power than they need

198
00:08:10,660 --> 00:08:11,760
in case a vulnerability exists.

199
00:08:14,220 --> 00:08:15,800
So, for cryptographic signatures,

200
00:08:16,160 --> 00:08:17,500
we're only going to use things like

201
00:08:18,020 --> 00:08:18,460
ED25519,

202
00:08:19,680 --> 00:08:20,560
deterministic ECDSA,

203
00:08:21,180 --> 00:08:22,760
or if we're going to go post-quantum,

204
00:08:22,760 --> 00:08:24,820
something like Sphinx, which is a stateless

205
00:08:25,420 --> 00:08:26,700
hash-based signature protocol.

206
00:08:28,100 --> 00:08:29,700
Signed by the company or the

207
00:08:29,700 --> 00:08:31,500
person who delivers the actual

208
00:08:32,100 --> 00:08:33,460
software, not the update server.

209
00:08:33,820 --> 00:08:35,020
It's an offline signature.

210
00:08:36,400 --> 00:08:37,820
Make software reproducible.

211
00:08:38,360 --> 00:08:40,140
And the user-based consistency

212
00:08:40,140 --> 00:08:41,320
verification is

213
00:08:41,960 --> 00:08:44,120
make sure that everybody sees

214
00:08:44,120 --> 00:08:44,780
the same thing.

215
00:08:46,200 --> 00:08:47,680
And this is very important.

216
00:08:49,320 --> 00:08:50,280
If you had

217
00:08:50,280 --> 00:08:52,140
somebody capable of hacking into your

218
00:08:52,140 --> 00:08:53,500
computer and stealing your private key,

219
00:08:53,680 --> 00:08:56,060
you're probably dealing with somebody on the same level

220
00:08:56,060 --> 00:08:57,160
as a nation state adversary.

221
00:08:58,100 --> 00:08:59,780
And if they're doing this to an open source project,

222
00:09:00,000 --> 00:09:01,900
for example, they're probably

223
00:09:01,900 --> 00:09:03,800
doing it as part of either a sting operation

224
00:09:03,800 --> 00:09:05,960
or some kind of clandestine thing.

225
00:09:07,200 --> 00:09:07,640
And

226
00:09:08,420 --> 00:09:09,420
they're going to be very

227
00:09:10,060 --> 00:09:11,920
unwilling to do anything where they have to

228
00:09:11,920 --> 00:09:12,760
announce their presence.

229
00:09:13,660 --> 00:09:15,760
And in addition, anything they do,

230
00:09:15,860 --> 00:09:18,060
if they manage to pull off the first attack where they steal the key

231
00:09:18,060 --> 00:09:20,000
and hack the update server and push out their

232
00:09:20,000 --> 00:09:21,920
malware, it's detectable.

233
00:09:22,320 --> 00:09:24,040
Because you have a permanent record

234
00:09:24,040 --> 00:09:24,980
of it ever happening.

235
00:09:27,360 --> 00:09:27,800
So,

236
00:09:29,300 --> 00:09:30,580
moving along

237
00:09:30,580 --> 00:09:31,060
a little bit quicker.

238
00:09:31,840 --> 00:09:34,100
Because we're going to have a system where all

239
00:09:34,100 --> 00:09:35,480
your users see the same thing,

240
00:09:35,840 --> 00:09:37,040
you're going to have some communication

241
00:09:37,040 --> 00:09:40,280
either between other peers

242
00:09:40,280 --> 00:09:42,060
or between other servers

243
00:09:42,060 --> 00:09:43,860
that mirror the data

244
00:09:44,860 --> 00:09:46,220
that basically keeps

245
00:09:46,220 --> 00:09:48,740
a log or ledger

246
00:09:48,740 --> 00:09:50,160
of all the updates

247
00:09:50,160 --> 00:09:50,880
that have ever happened.

248
00:09:51,800 --> 00:09:53,780
And we've actually built something

249
00:09:53,780 --> 00:09:55,100
called Chronicle.

250
00:09:55,800 --> 00:09:57,220
It's an open source

251
00:09:58,100 --> 00:09:59,200
service. It's on GitHub.

252
00:10:00,080 --> 00:10:02,040
It's a hash chain data structure.

253
00:10:02,420 --> 00:10:03,580
It's not like a Merkle tree.

254
00:10:03,800 --> 00:10:05,740
It's almost like a linked list.

255
00:10:07,320 --> 00:10:08,220
It has a couple

256
00:10:08,220 --> 00:10:10,000
things that make it useful. You can quickly

257
00:10:10,000 --> 00:10:12,020
cross sign your latest hash

258
00:10:12,020 --> 00:10:13,860
onto other Chronicles and you can replicate

259
00:10:13,860 --> 00:10:15,360
many of them onto your local one.

260
00:10:17,940 --> 00:10:19,980
The crypto side of things are a little bit

261
00:10:19,980 --> 00:10:21,820
more interesting. It's a JSON

262
00:10:21,820 --> 00:10:23,900
API that uses a

263
00:10:24,480 --> 00:10:25,740
specification I wrote called

264
00:10:25,740 --> 00:10:27,720
Sapient, which is a backronym for secure

265
00:10:28,100 --> 00:10:28,940
API engineering toolkit.

266
00:10:30,040 --> 00:10:31,100
It basically

267
00:10:32,040 --> 00:10:33,660
adds specific headers that have

268
00:10:33,660 --> 00:10:35,820
base64 encoded signatures

269
00:10:35,820 --> 00:10:38,020
that authenticate the entire message

270
00:10:38,020 --> 00:10:38,220
body.

271
00:10:39,620 --> 00:10:41,440
So that's how it communicates.

272
00:10:41,540 --> 00:10:43,980
If you're trying to communicate with it

273
00:10:43,980 --> 00:10:45,860
and you just have an API

274
00:10:45,860 --> 00:10:47,920
adapter that speaks JSON but

275
00:10:47,920 --> 00:10:50,020
doesn't know what Sapient is, it will be

276
00:10:50,020 --> 00:10:51,900
backwards compatible. You just won't have the security

277
00:10:51,900 --> 00:10:53,420
of having the signature verified.

278
00:10:54,600 --> 00:10:55,920
Each message has

279
00:10:55,920 --> 00:10:57,920
two hashes. A current hash

280
00:10:58,100 --> 00:11:00,220
uses the previous hash as the key

281
00:11:00,220 --> 00:11:01,160
for a Blake 2 hash.

282
00:11:01,760 --> 00:11:03,560
Then you have a summary hash, which is literally

283
00:11:03,560 --> 00:11:05,420
just, you concatenate every single

284
00:11:06,000 --> 00:11:06,620
current hash.

285
00:11:08,500 --> 00:11:09,440
That's used for

286
00:11:09,880 --> 00:11:11,680
the verification

287
00:11:11,680 --> 00:11:13,040
steps that you can do later.

288
00:11:14,840 --> 00:11:15,320
So

289
00:11:16,000 --> 00:11:17,480
when an automatic update is signed,

290
00:11:18,060 --> 00:11:19,860
if you wanted to build an automatic update system,

291
00:11:20,000 --> 00:11:21,820
what you would do is you would sign it using

292
00:11:21,820 --> 00:11:24,040
your offline key, push your signature

293
00:11:24,040 --> 00:11:25,620
and some other metadata like what the

294
00:11:25,620 --> 00:11:28,020
version number is, what was the latest

295
00:11:28,100 --> 00:11:29,860
git commit hash, and you would

296
00:11:29,860 --> 00:11:30,880
submit that to a chronicle.

297
00:11:31,340 --> 00:11:33,520
And after every end

298
00:11:34,030 --> 00:11:35,720
messages or every end days or however

299
00:11:35,720 --> 00:11:37,780
it's configured, it would cross-sign to their peers.

300
00:11:38,300 --> 00:11:40,100
And as soon as a message is there,

301
00:11:40,300 --> 00:11:41,720
anybody who's replicating that

302
00:11:41,720 --> 00:11:44,160
would download, verify a copy

303
00:11:44,160 --> 00:11:45,720
and they'd have a copy in their local cache.

304
00:11:48,720 --> 00:11:50,080
So when you go to download

305
00:11:50,080 --> 00:11:52,100
an update, your software should verify

306
00:11:52,100 --> 00:11:54,080
the signature and then

307
00:11:54,080 --> 00:11:55,940
verify that everything is

308
00:11:55,940 --> 00:11:58,080
what they'd expect to see in the chronicle at that time.

309
00:11:58,100 --> 00:12:00,800
If any of those fail,

310
00:12:00,920 --> 00:12:02,140
it should fail hard

311
00:12:02,140 --> 00:12:03,640
and alert them and

312
00:12:04,420 --> 00:12:06,040
probably scream bloody

313
00:12:06,040 --> 00:12:06,800
murder crying home.

314
00:12:07,540 --> 00:12:09,940
And then software updates should be reproducible.

315
00:12:10,620 --> 00:12:12,100
That's more of a C programmer's

316
00:12:12,100 --> 00:12:13,620
area of interest than

317
00:12:13,620 --> 00:12:15,060
crypto, I would think.

318
00:12:15,400 --> 00:12:17,220
So I didn't really go into that too much.

319
00:12:17,880 --> 00:12:19,420
We have a library called Pharaoh

320
00:12:20,050 --> 00:12:21,780
because PHP has these deliverables called

321
00:12:21,780 --> 00:12:23,940
PHP Archives. It's kind of like how Java

322
00:12:23,940 --> 00:12:26,180
has JAR files. All it does

323
00:12:26,180 --> 00:12:27,860
is diff two of them and tell you what's different.

324
00:12:30,360 --> 00:12:31,740
And then, as I mentioned,

325
00:12:31,840 --> 00:12:33,400
everything should be delivered over HTTPS.

326
00:12:34,380 --> 00:12:35,500
There's metadata leaks,

327
00:12:35,720 --> 00:12:37,060
you know, privacy concerns there

328
00:12:37,060 --> 00:12:37,880
for not using that.

329
00:12:38,080 --> 00:12:39,980
In addition, you don't want people

330
00:12:39,980 --> 00:12:42,000
to just replay an old message

331
00:12:42,000 --> 00:12:43,060
and keep people on a vulnerable

332
00:12:43,880 --> 00:12:45,020
version of your software.

333
00:12:47,100 --> 00:12:47,720
Privilege separation

334
00:12:47,720 --> 00:12:48,820
and then availability.

335
00:12:49,500 --> 00:12:51,780
So with chronicle, I designed it

336
00:12:51,780 --> 00:12:53,520
so where you could have a replication

337
00:12:54,020 --> 00:12:55,560
like a replicated copy

338
00:12:55,560 --> 00:12:57,560
of a chronicle used to sign

339
00:12:58,100 --> 00:12:58,660
projects updates.

340
00:12:59,660 --> 00:13:01,740
The idea was that hosting companies

341
00:13:01,740 --> 00:13:03,780
that have like 2,000 WordPress

342
00:13:03,780 --> 00:13:05,700
clients, instead of hitting

343
00:13:05,700 --> 00:13:07,720
our server or hitting some other server

344
00:13:07,720 --> 00:13:09,860
for like a plug-in vendor, could just

345
00:13:09,860 --> 00:13:11,720
have a copy they keep locally for all their

346
00:13:11,720 --> 00:13:13,580
customers and benefit from lower

347
00:13:14,100 --> 00:13:15,780
network latency and if there's downtime

348
00:13:15,780 --> 00:13:17,500
upstream, their customers are unaffected.

349
00:13:18,000 --> 00:13:19,860
You could do something similar for the actual

350
00:13:19,860 --> 00:13:20,420
updates themselves.

351
00:13:21,440 --> 00:13:23,640
You can just mirror them with trusted mirrors

352
00:13:23,640 --> 00:13:25,440
that have their own public keys penned.

353
00:13:25,600 --> 00:13:27,320
You could use BitTorrent to distribute it.

354
00:13:28,100 --> 00:13:29,040
There's a lot of options.

355
00:13:31,080 --> 00:13:32,720
Building a system like this does come

356
00:13:32,720 --> 00:13:33,960
with some challenges, of course.

357
00:13:34,520 --> 00:13:36,420
The supplier of the code has to be

358
00:13:36,420 --> 00:13:37,360
disciplined enough to keep their

359
00:13:37,360 --> 00:13:38,400
secret key from getting stolen.

360
00:13:39,460 --> 00:13:41,200
They also need to make sure that their

361
00:13:41,200 --> 00:13:43,640
API doesn't break to prevent people

362
00:13:43,640 --> 00:13:46,420
from wanting the update to be there.

363
00:13:48,020 --> 00:13:49,480
And that's a very important one

364
00:13:49,480 --> 00:13:51,340
because one of the pain points

365
00:13:51,340 --> 00:13:52,800
of some operating systems like Debian

366
00:13:52,800 --> 00:13:55,660
and some of the other more enterprise-y

367
00:13:55,660 --> 00:13:58,000
long-term support Linuxes

368
00:13:58,000 --> 00:13:58,080
is that they don't have the

369
00:13:58,100 --> 00:13:58,960
capability to do that.

370
00:13:58,960 --> 00:13:59,820
They still ship an old version of

371
00:13:59,820 --> 00:14:01,340
libsodium, still ship old versions of

372
00:14:01,340 --> 00:14:05,260
PHP and they lock them to really weird

373
00:14:05,260 --> 00:14:06,000
patch versions.

374
00:14:06,680 --> 00:14:08,560
An old version of Debian shipped with

375
00:14:08,560 --> 00:14:12,180
PHP 5.3.3 but they backported a patch

376
00:14:12,180 --> 00:14:14,380
that fixed password hashing from

377
00:14:14,380 --> 00:14:15,580
5.3.7.

378
00:14:15,860 --> 00:14:18,100
So if you were trying to detect what

379
00:14:18,680 --> 00:14:19,900
version you're running to where if you

380
00:14:19,900 --> 00:14:21,180
could use the new password hash,

381
00:14:21,870 --> 00:14:23,520
that became extremely unreliable.

382
00:14:23,940 --> 00:14:26,800
So this has been a problem for the

383
00:14:26,800 --> 00:14:28,080
PHP community in particular.

384
00:14:29,660 --> 00:14:31,060
Operating systems pinning to

385
00:14:31,060 --> 00:14:32,340
patch numbers instead of minor numbers

386
00:14:32,340 --> 00:14:34,400
and then just backporting patches

387
00:14:34,400 --> 00:14:35,620
and lying about what version you're on.

388
00:14:36,180 --> 00:14:38,440
So there needs to be discipline among

389
00:14:38,440 --> 00:14:39,820
the people who are developing the software.

390
00:14:40,980 --> 00:14:42,700
And then anybody who's using like

391
00:14:42,700 --> 00:14:44,280
a library in their own software

392
00:14:44,280 --> 00:14:46,060
needs to not use anything undocumented

393
00:14:46,060 --> 00:14:47,620
in case there's a breaking change.

394
00:14:49,100 --> 00:14:50,540
Sometimes I make a breaking change

395
00:14:50,540 --> 00:14:52,300
into the protected or private methods

396
00:14:52,300 --> 00:14:54,360
in the stuff I write and release it

397
00:14:54,360 --> 00:14:54,900
as a patch version.

398
00:14:55,260 --> 00:14:57,840
And so far I've not had anybody complain about that

399
00:14:58,100 --> 00:14:59,700
but a lot of people tell me they're afraid

400
00:14:59,700 --> 00:15:01,480
to do that because they're afraid of the complaints

401
00:15:01,480 --> 00:15:03,480
and having to do rollbacks and all that other

402
00:15:03,480 --> 00:15:03,960
fun stuff.

403
00:15:05,680 --> 00:15:08,580
So this isn't just a theoretical talk.

404
00:15:08,820 --> 00:15:10,360
I've actually built this twice.

405
00:15:11,200 --> 00:15:12,580
I have an open source CMS

406
00:15:12,580 --> 00:15:13,340
called Airship.

407
00:15:13,500 --> 00:15:14,380
It's free software.

408
00:15:14,920 --> 00:15:17,420
I'm releasing version 2 with PHP 7.2

409
00:15:17,420 --> 00:15:18,320
early next year.

410
00:15:18,800 --> 00:15:21,620
It uses ED25519 signatures

411
00:15:21,620 --> 00:15:23,600
but it uses a Merkle tree structure

412
00:15:23,600 --> 00:15:24,600
instead of a hash chain.

413
00:15:24,600 --> 00:15:26,480
I went with a hash chain with Chronicle

414
00:15:26,480 --> 00:15:27,400
just for speed reasons.

415
00:15:28,820 --> 00:15:30,580
It approximates the use case of

416
00:15:30,580 --> 00:15:32,160
Git and certificate transparency

417
00:15:32,160 --> 00:15:33,160
kind of meshed together.

418
00:15:34,360 --> 00:15:36,060
I was going to release something

419
00:15:36,060 --> 00:15:36,960
this past week.

420
00:15:37,380 --> 00:15:39,840
It's kind of like a firewall slash IDS toolkit

421
00:15:39,840 --> 00:15:41,080
but one of the things it does

422
00:15:41,080 --> 00:15:42,760
is actually do automatic updates

423
00:15:42,760 --> 00:15:45,120
for example those Magento sites

424
00:15:45,120 --> 00:15:47,200
that haven't been updated in two years.

425
00:15:48,100 --> 00:15:50,460
The idea is that people who use

426
00:15:50,460 --> 00:15:52,240
Word, which is the name of the project

427
00:15:52,240 --> 00:15:53,980
it has normal

428
00:15:54,980 --> 00:15:56,120
boring, uninteresting

429
00:15:56,120 --> 00:15:57,800
WAF IDS features.

430
00:15:59,620 --> 00:16:01,640
It will keep all your software up to date.

431
00:16:01,780 --> 00:16:02,740
If there's a security update

432
00:16:02,740 --> 00:16:04,220
it will query our servers and say

433
00:16:04,220 --> 00:16:05,540
hey, is this authentic?

434
00:16:06,140 --> 00:16:08,220
And since we do all our signing offline

435
00:16:08,220 --> 00:16:10,200
on a Raspberry Pi that never connects

436
00:16:10,200 --> 00:16:10,680
to the internet,

437
00:16:11,600 --> 00:16:13,680
the likelihood of that ever getting stolen

438
00:16:13,680 --> 00:16:14,420
is pretty low.

439
00:16:14,680 --> 00:16:16,100
Won't say zero, but pretty low.

440
00:16:17,080 --> 00:16:19,400
So that's what we're doing there.

441
00:16:20,240 --> 00:16:22,160
I'm working to get

442
00:16:22,160 --> 00:16:24,260
the similar system built for the PHP

443
00:16:24,600 --> 00:16:25,880
package manager called Composer.

444
00:16:26,720 --> 00:16:28,340
Right now that's been kind of stalled

445
00:16:28,340 --> 00:16:29,740
from complete lack of response

446
00:16:29,740 --> 00:16:30,660
from the Composer team,

447
00:16:30,860 --> 00:16:32,680
but I'm assuming that's because they're busy.

448
00:16:34,200 --> 00:16:36,500
So the PHP community is moving

449
00:16:36,500 --> 00:16:37,640
forward on this slowly.

450
00:16:38,220 --> 00:16:40,620
I expect in probably a couple years

451
00:16:40,620 --> 00:16:42,040
the talk I'll be giving is

452
00:16:42,040 --> 00:16:43,580
how we did this rather than

453
00:16:43,580 --> 00:16:44,360
what I want to do.

454
00:16:46,560 --> 00:16:48,060
If anybody wants to build this

455
00:16:48,060 --> 00:16:49,100
in another framework,

456
00:16:49,900 --> 00:16:51,460
there's this thing called the update framework

457
00:16:51,460 --> 00:16:52,800
and I forgot to make a slide for this,

458
00:16:54,600 --> 00:16:55,280
but it's called the automatic updates.

459
00:16:55,720 --> 00:16:57,200
All you would need to do is

460
00:16:57,200 --> 00:16:59,120
make sure your builds are reproducible

461
00:16:59,120 --> 00:17:01,060
and query a chronicle and you've got

462
00:17:01,060 --> 00:17:02,260
a secure automatic update system

463
00:17:02,260 --> 00:17:04,700
as described in the model here.

464
00:17:05,820 --> 00:17:07,560
And that is all I have.

465
00:17:07,700 --> 00:17:08,800
Does anybody have any questions?

466
00:17:10,960 --> 00:17:11,420
Yes.

467
00:17:24,600 --> 00:17:25,800
So one of the slides mentioned

468
00:17:25,800 --> 00:17:28,760
that a more trusted user than the Apache user

469
00:17:28,760 --> 00:17:31,100
should be responsible for the software updates

470
00:17:31,100 --> 00:17:31,700
for the web applications.

471
00:17:32,040 --> 00:17:33,960
Sorry, that was a little bit quick.

472
00:17:34,280 --> 00:17:34,600
Okay.

473
00:17:34,640 --> 00:17:37,120
So one of the slides mentioned

474
00:17:37,120 --> 00:17:39,100
that a more trusted user than the Apache

475
00:17:39,100 --> 00:17:40,740
is supposed to be responsible

476
00:17:40,740 --> 00:17:42,440
for the software updates for the web apps.

477
00:17:44,220 --> 00:17:45,600
Don't let Apache...

478
00:17:57,920 --> 00:17:58,840
Oh, okay.

479
00:18:04,460 --> 00:18:24,580
So that comes into a chicken and egg conversation.

480
00:18:24,600 --> 00:18:24,720
So if you're in Harlem,

481
00:18:24,920 --> 00:18:27,260
anything a hacked PHP script can do,

482
00:18:27,380 --> 00:18:29,160
your normal PHP script can do.

483
00:18:29,380 --> 00:18:30,680
And the point of privilege separation

484
00:18:30,680 --> 00:18:33,400
would be to prevent the...

485
00:18:34,240 --> 00:18:35,700
Somebody who found a vulnerability

486
00:18:35,700 --> 00:18:36,680
that allows code execution

487
00:18:36,680 --> 00:18:38,420
to be able to write to your file system.

488
00:18:38,740 --> 00:18:40,340
So you usually prevent them from having

489
00:18:40,340 --> 00:18:41,720
write access to your web route.

490
00:18:42,900 --> 00:18:44,740
Automatic updates, by necessity,

491
00:18:44,880 --> 00:18:46,900
require something to be able to write to the web route.

492
00:18:47,200 --> 00:18:48,740
So you would need to have a separate user

493
00:18:48,740 --> 00:18:50,180
that runs that process to make sure

494
00:18:50,180 --> 00:18:53,440
that your normal web user isn't overwriting index.php,

495
00:18:54,600 --> 00:18:55,420
if they found a way to do that

496
00:18:55,420 --> 00:18:56,700
from some weird code path

497
00:18:56,700 --> 00:18:59,000
and something's un-serialized

498
00:18:59,000 --> 00:19:00,960
and something gets called under

499
00:19:00,960 --> 00:19:02,940
the underscore destruct magic function.

500
00:19:03,240 --> 00:19:05,040
Now their shell is at the very bottom

501
00:19:05,040 --> 00:19:06,380
of index.php and it runs

502
00:19:06,380 --> 00:19:09,320
if you pass a magic variable to get.

503
00:19:11,440 --> 00:19:14,100
It's a defense in depth.

504
00:19:15,060 --> 00:19:16,140
Airship doesn't do this.

505
00:19:17,740 --> 00:19:19,020
We have a mechanism

506
00:19:19,020 --> 00:19:19,980
to where anybody who wants

507
00:19:19,980 --> 00:19:21,460
to do privilege separation can do that.

508
00:19:21,980 --> 00:19:23,000
But by default,

509
00:19:23,780 --> 00:19:24,580
we made the transition

510
00:19:24,600 --> 00:19:26,080
trade-off to where automatic updates

511
00:19:26,080 --> 00:19:27,500
and getting vulnerabilities fixed

512
00:19:27,500 --> 00:19:29,420
was considered a higher priority

513
00:19:29,420 --> 00:19:31,900
than preventing traditional views

514
00:19:31,900 --> 00:19:33,960
on web hosting where everything

515
00:19:33,960 --> 00:19:35,740
is locked away to where

516
00:19:36,220 --> 00:19:38,140
the web, the process,

517
00:19:38,440 --> 00:19:39,860
the user that has the privilege

518
00:19:39,860 --> 00:19:42,120
to run the code isn't able to write to it.

519
00:19:43,840 --> 00:19:44,600
Other people

520
00:19:44,600 --> 00:19:45,600
will make different trade-offs

521
00:19:45,600 --> 00:19:47,840
and it depends on your threat model.

522
00:19:48,220 --> 00:19:49,640
Ours is more general and we tried

523
00:19:49,640 --> 00:19:50,980
to go for usability on that one.

524
00:19:51,840 --> 00:19:54,240
Is there any way to implement this

525
00:19:54,600 --> 00:19:55,100
on shared hosting?

526
00:19:57,480 --> 00:19:57,920
Yes.

527
00:20:00,280 --> 00:20:01,080
But you would need

528
00:20:01,080 --> 00:20:02,380
some kind of cooperation from them.

529
00:20:02,680 --> 00:20:04,660
The other thing you could do is

530
00:20:04,660 --> 00:20:06,220
see if they can run your web

531
00:20:06,220 --> 00:20:08,560
processes as like nobody or as Apache

532
00:20:08,560 --> 00:20:10,100
or some other user.

533
00:20:11,780 --> 00:20:12,880
That's the only way I can see

534
00:20:12,880 --> 00:20:14,800
to reliably do it because OS permissions

535
00:20:14,800 --> 00:20:16,220
which is what the whole privilege

536
00:20:16,220 --> 00:20:18,220
separation is supposed to do is tied into

537
00:20:18,220 --> 00:20:20,500
OS users and I can't really think

538
00:20:20,500 --> 00:20:21,760
of a way that would work without

539
00:20:21,760 --> 00:20:22,920
having more than one user.

540
00:20:25,460 --> 00:20:26,860
Yes.

541
00:20:29,040 --> 00:20:30,440
.

542
00:20:33,560 --> 00:20:34,960
.

543
00:20:34,960 --> 00:20:35,400
.

544
00:20:43,140 --> 00:20:44,540
.

545
00:20:44,540 --> 00:20:45,520
.

546
00:20:45,520 --> 00:20:47,040
.

547
00:20:47,040 --> 00:20:47,120
.

548
00:20:47,120 --> 00:20:47,160
.

549
00:20:47,160 --> 00:20:47,180
.

550
00:20:47,180 --> 00:20:47,280
.

551
00:20:47,280 --> 00:20:47,300
.

552
00:21:07,200 --> 00:21:07,240
.

553
00:21:14,060 --> 00:21:14,100
.

554
00:21:14,240 --> 00:21:14,280
.

555
00:21:14,280 --> 00:21:14,300
.

556
00:21:14,300 --> 00:21:14,320
.

557
00:21:14,320 --> 00:21:14,340
.

558
00:21:14,340 --> 00:21:14,360
.

559
00:21:14,360 --> 00:21:14,380
.

560
00:21:14,380 --> 00:21:14,400
.

561
00:21:14,400 --> 00:21:14,420
.

562
00:21:14,420 --> 00:21:14,440
.

563
00:21:14,980 --> 00:21:15,020
.

564
00:21:17,720 --> 00:21:17,760
.

565
00:21:22,860 --> 00:21:22,900
.

566
00:21:24,820 --> 00:21:25,060
.

567
00:21:34,640 --> 00:21:34,880
.

568
00:21:35,340 --> 00:21:35,580
.

569
00:21:38,080 --> 00:21:38,320
.

570
00:21:38,320 --> 00:21:38,340
.

571
00:21:38,340 --> 00:21:38,460
.

572
00:21:38,460 --> 00:21:38,480
.

573
00:21:38,480 --> 00:21:38,500
.

574
00:21:38,500 --> 00:21:38,520
.

575
00:21:38,520 --> 00:21:38,540
.

576
00:21:38,540 --> 00:21:38,560
.

577
00:21:47,960 --> 00:21:48,200
.

578
00:21:48,200 --> 00:21:48,440
.

579
00:21:51,520 --> 00:21:52,360
.

580
00:21:55,360 --> 00:21:56,200
.

581
00:21:57,820 --> 00:21:58,660
.

582
00:21:58,660 --> 00:21:59,140
.

583
00:21:59,140 --> 00:21:59,560
.

584
00:21:59,560 --> 00:21:59,580
.

585
00:21:59,580 --> 00:21:59,600
.

586
00:21:59,600 --> 00:21:59,620
.

587
00:21:59,620 --> 00:21:59,640
.

588
00:21:59,640 --> 00:21:59,680
.

589
00:22:08,080 --> 00:22:08,920
.

590
00:22:19,360 --> 00:22:19,800
. .

591
00:22:24,960 --> 00:22:25,200
. .

592
00:22:25,200 --> 00:22:25,360
. .

593
00:22:25,360 --> 00:22:25,500
. .

594
00:22:25,500 --> 00:22:25,660
. .

595
00:22:25,660 --> 00:22:25,820
. . .

596
00:22:25,820 --> 00:22:25,880
. . .

597
00:22:25,880 --> 00:22:26,160
. . .

598
00:22:26,160 --> 00:22:26,260
. . .

599
00:22:26,260 --> 00:22:26,380
. . . .

600
00:22:26,380 --> 00:22:26,960
. . . .

601
00:22:26,960 --> 00:22:29,920
. . . . .

602
00:22:33,640 --> 00:22:34,960
Well, thank you for having me.