1
00:00:00,000 --> 00:00:03,751
I'd like to introduce Chris Sumner

2
00:00:03,751 --> 00:00:07,999
(Applause.) (Whistles

3
00:00:10,709 --> 00:00:12,834
CHRIS SUMNER: All right.

4
00:00:14,501 --> 00:00:18,209
Either way, very much appreciated.

5
00:00:20,626 --> 00:00:21,999
As the gentleman there said,

6
00:00:21,999 --> 00:00:26,751
a small charitable organization called

7
00:00:26,751 --> 00:00:29,999
or privacy in the U.S., and part

8
00:00:29,999 --> 00:00:34,999
at behavioral residue, that's the sort

9
00:00:34,999 --> 00:00:37,542
in an online context to see

10
00:00:37,542 --> 00:00:41,501
without actually knowing that you're

11
00:00:41,501 --> 00:00:44,999
So in this experiment, we're looking

12
00:00:44,999 --> 00:00:50,125
with a social bot on Twitter,

13
00:00:50,125 --> 00:00:53,834
So before I begin, I was also

14
00:00:53,834 --> 00:00:57,667
for DEF CON, so I know there are

15
00:00:57,667 --> 00:01:00,626
and there's

16
00:01:00,626 --> 00:01:02,459
to this one.

17
00:01:02,459 --> 00:01:04,459
So what I wanted

18
00:01:04,459 --> 00:01:08,709
of this talk, which some

19
00:01:08,709 --> 00:01:11,459
So if you are familiar

20
00:01:11,459 --> 00:01:16,292
Tim Hwang, familiar with astro turfing,

21
00:01:16,292 --> 00:01:19,292
a gentleman called Yaz,

22
00:01:19,292 --> 00:01:25,999
of the presentation may be how can

23
00:01:25,999 --> 00:01:27,999
But if these terms are fairly new,

24
00:01:27,999 --> 00:01:30,250
will be pretty interesting.

25
00:01:30,250 --> 00:01:34,709
And the other thing is that we oh yeah,

26
00:01:34,709 --> 00:01:35,999
That's what I added later.

27
00:01:37,083 --> 00:01:39,667
It also contains about five minutes

28
00:01:39,667 --> 00:01:44,501
and specifically machine learning,

29
00:01:44,959 --> 00:01:48,209
So just so you are aware, there

30
00:01:48,209 --> 00:01:49,209
Okay.

31
00:01:49,209 --> 00:01:52,250
So moving on, because we're

32
00:01:52,250 --> 00:01:56,083
is all about trying to win money, or

33
00:01:56,083 --> 00:02:00,584
and win money, or just hack things,

34
00:02:00,584 --> 00:02:04,083
to laser, you know,

35
00:02:04,083 --> 00:02:07,999
who is susceptible to a social bot

36
00:02:08,083 --> 00:02:11,999
It was more to improve the odds,

37
00:02:11,999 --> 00:02:16,667
at least improve the odds

38
00:02:16,667 --> 00:02:18,918
So that's what we were really aiming

39
00:02:18,918 --> 00:02:21,167
So if you're expecting laser precision

40
00:02:21,167 --> 00:02:24,626
in an organization,

41
00:02:24,626 --> 00:02:27,375
But if you want to move

42
00:02:27,375 --> 00:02:31,292
a bit more of an advantage,

43
00:02:31,542 --> 00:02:36,083
Okay, so with that out of the way,

44
00:02:36,083 --> 00:02:39,167
Then I pretty much, I like the dude,

45
00:02:39,167 --> 00:02:41,876
that looked pretty good.

46
00:02:41,876 --> 00:02:45,584
So moving on, starting

47
00:02:45,584 --> 00:02:50,292
this gentleman is called Tim Hwang,

48
00:02:50,292 --> 00:02:55,626
a competition called

49
00:02:55,999 --> 00:02:59,709
And essentially how that works

50
00:02:59,709 --> 00:03:04,542
with social bots competing

51
00:03:04,542 --> 00:03:07,501
a small amount

52
00:03:07,501 --> 00:03:09,999
props essentially.

53
00:03:09,999 --> 00:03:13,626
So they were given a target audience

54
00:03:13,626 --> 00:03:17,083
to go and apply their bots to, and

55
00:03:17,083 --> 00:03:21,792
all of these Twitter users had

56
00:03:21,792 --> 00:03:26,083
they had tweeted or had some interest

57
00:03:26,999 --> 00:03:29,709
Okay, so that's a lot of Twitter users.

58
00:03:29,709 --> 00:03:32,083
But the social bot

59
00:03:32,083 --> 00:03:35,792
they got one for a follow back,

60
00:03:35,792 --> 00:03:39,834
for social response, and

61
00:03:39,834 --> 00:03:45,999
points if they were killed by Twitter, so,

62
00:03:45,999 --> 00:03:49,999
So that was the scoring mechanism

63
00:03:50,417 --> 00:03:54,209
It was described as Blood Spot

64
00:03:54,209 --> 00:03:56,999
and Network Analysis Nerds.

65
00:03:57,501 --> 00:03:59,667
So I'm not going to go into this

66
00:03:59,667 --> 00:04:02,292
because there's

67
00:04:02,292 --> 00:04:05,083
about five minutes,

68
00:04:05,083 --> 00:04:09,999
and it's really pretty good and

69
00:04:09,999 --> 00:04:11,999
So all of the references and what we're

70
00:04:11,999 --> 00:04:14,083
is in the speaker notes,

71
00:04:14,083 --> 00:04:15,999
about writing notes.

72
00:04:17,375 --> 00:04:19,999
One team won, obviously.

73
00:04:20,459 --> 00:04:24,999
And it was a team led

74
00:04:24,999 --> 00:04:29,918
on Twitter and his bot got 198

75
00:04:29,918 --> 00:04:32,999
is pretty good, almost 40%.

76
00:04:33,999 --> 00:04:37,834
So we're going to come back

77
00:04:37,834 --> 00:04:39,918
in a short while.

78
00:04:39,918 --> 00:04:42,999
He was in the audience when we talked

79
00:04:42,999 --> 00:04:45,083
If you're in the audience today,

80
00:04:45,083 --> 00:04:48,167
and we'll get you on stage

81
00:04:48,876 --> 00:04:52,751
So most research in this area

82
00:04:52,751 --> 00:04:56,918
at identifying social bots or bots

83
00:04:56,918 --> 00:05:00,375
or other social networks,

84
00:05:00,375 --> 00:05:02,999
in looking

85
00:05:02,999 --> 00:05:06,083
with who responds to social bots

86
00:05:06,083 --> 00:05:09,125
with potential strangers online.

87
00:05:09,209 --> 00:05:12,501
So as the goon mentioned

88
00:05:12,501 --> 00:05:16,292
last year we were looking

89
00:05:16,292 --> 00:05:19,959
We weren't trying

90
00:05:19,959 --> 00:05:22,999
We were looking at psychopathy

91
00:05:22,999 --> 00:05:26,459
to improve the odds in your favor

92
00:05:26,459 --> 00:05:28,999
whether they had certain traits

93
00:05:28,999 --> 00:05:32,751
whether you could reduce

94
00:05:32,751 --> 00:05:36,334
and that was kind

95
00:05:36,334 --> 00:05:36,999
Not saying that app's what it should be

96
00:05:36,999 --> 00:05:39,083
but that's what we were looking at.

97
00:05:39,083 --> 00:05:41,584
As part of that research,

98
00:05:41,584 --> 00:05:44,751
to have a look at social bots as well.

99
00:05:44,751 --> 00:05:46,167
That was

100
00:05:46,167 --> 00:05:50,792
of the social bot competition,

101
00:05:50,792 --> 00:05:52,999
We asked a subset

102
00:05:52,999 --> 00:05:55,542
in this experiment

103
00:05:55,542 --> 00:05:59,959
in an experiment where they'd receive

104
00:05:59,959 --> 00:06:04,250
in the future, and you might think, okay,

105
00:06:04,250 --> 00:06:07,125
to begin with,

106
00:06:07,125 --> 00:06:08,999
all the time.

107
00:06:08,999 --> 00:06:11,751
So we felt that that was quite

108
00:06:11,751 --> 00:06:15,501
of roughly 700 people take the bots

109
00:06:15,501 --> 00:06:17,209
out of 3,000.

110
00:06:17,501 --> 00:06:19,999
So we had two main questions.

111
00:06:19,999 --> 00:06:21,334
The first was,

112
00:06:21,334 --> 00:06:25,834
predisposed to interacting

113
00:06:25,918 --> 00:06:29,125
And the other was, is it possible

114
00:06:29,125 --> 00:06:33,083
of getting a response

115
00:06:33,083 --> 00:06:36,083
for any social bot, so that

116
00:06:36,083 --> 00:06:40,999
in context can avoid Twitter jail,

117
00:06:41,083 --> 00:06:45,501
So a good question

118
00:06:45,501 --> 00:06:46,375
Now, for the younger people

119
00:06:46,375 --> 00:06:49,292
you maybe not remember "Different

120
00:06:49,292 --> 00:06:50,626
to find

121
00:06:50,626 --> 00:06:51,999
because it was

122
00:06:51,999 --> 00:06:53,876
from my childhood..

123
00:06:54,667 --> 00:07:00,501
So one group that are really interested

124
00:07:00,501 --> 00:07:04,375
in this is marketing and sales types.

125
00:07:04,834 --> 00:07:08,083
So the reason they're interested

126
00:07:08,083 --> 00:07:12,167
all of their work was based on likes

127
00:07:12,167 --> 00:07:15,667
for example, or follow account,

128
00:07:15,667 --> 00:07:17,999
to engagement

129
00:07:17,999 --> 00:07:21,999
for marketing services have got wise

130
00:07:21,999 --> 00:07:26,292
to generate fake likes and friend counts

131
00:07:26,292 --> 00:07:31,334
but it's less hard or it's less easy

132
00:07:31,334 --> 00:07:36,667
So it behooves them to actually try

133
00:07:36,667 --> 00:07:39,626
The next group that are interested are

134
00:07:39,626 --> 00:07:42,250
about that in some more detail, trying

135
00:07:42,250 --> 00:07:46,999
a message or their message

136
00:07:47,417 --> 00:07:51,250
Another group, and this uses

137
00:07:52,918 --> 00:07:54,999
It's a fantastic tool.

138
00:07:54,999 --> 00:07:57,999
You may be interested in taking

139
00:07:57,999 --> 00:08:02,083
from a particular corporation

140
00:08:02,083 --> 00:08:07,209
of those groups are more likely

141
00:08:07,209 --> 00:08:10,834
request or your social bot request.

142
00:08:10,834 --> 00:08:12,999
So, again, picking

143
00:08:12,999 --> 00:08:15,334
to focusing

144
00:08:15,334 --> 00:08:16,999
to respond.

145
00:08:17,667 --> 00:08:20,501
Then the other group that are interested

146
00:08:20,501 --> 00:08:23,375
of research are privacy researchers,

147
00:08:23,375 --> 00:08:26,167
is called Erhardt Graeff at M.I.T.

148
00:08:26,167 --> 00:08:28,167
and he's got

149
00:08:28,167 --> 00:08:32,417
Should Do Before Social Bots Take

150
00:08:32,417 --> 00:08:34,999
the Political Economy

151
00:08:34,999 --> 00:08:38,083
and he's concerned one

152
00:08:38,083 --> 00:08:41,167
about in his paper

153
00:08:41,167 --> 00:08:44,209
to harvest a lot

154
00:08:44,209 --> 00:08:46,709
by engaging

155
00:08:46,709 --> 00:08:49,375
to say are gullible,

156
00:08:49,375 --> 00:08:52,999
to provide certain information which

157
00:08:52,999 --> 00:08:57,417
if they knew it was a fake account

158
00:08:57,417 --> 00:08:59,125
So he was quite worried about that.

159
00:08:59,125 --> 00:09:01,751
We'll talk about that

160
00:09:01,751 --> 00:09:04,501
And the final group,

161
00:09:04,584 --> 00:09:06,834
So Facebook, for example, have

162
00:09:06,834 --> 00:09:10,459
and it's kind of this constant battle

163
00:09:10,459 --> 00:09:13,834
and the fake account creators are

164
00:09:13,834 --> 00:09:17,999
the Facebook immune system,

165
00:09:17,999 --> 00:09:19,792
And I also have

166
00:09:19,792 --> 00:09:22,292
a pretty good job if you look at spam,

167
00:09:22,292 --> 00:09:24,250
in 2011 they've done

168
00:09:24,250 --> 00:09:26,501
of throttling that down.

169
00:09:26,501 --> 00:09:27,876
So these are

170
00:09:27,876 --> 00:09:32,999
and obviously they've been applying

171
00:09:33,667 --> 00:09:35,999
So we set

172
00:09:35,999 --> 00:09:39,375
to actually go in and have a look at this.

173
00:09:39,375 --> 00:09:41,792
So let me go over some of the some

174
00:09:41,792 --> 00:09:44,999
in this this sphere of research.

175
00:09:45,959 --> 00:09:48,876
So the talk's going

176
00:09:48,876 --> 00:09:51,626
of some of the current research.

177
00:09:51,626 --> 00:09:52,999
Then we'll talk

178
00:09:52,999 --> 00:09:55,459
discuss the findings that we had,

179
00:09:55,459 --> 00:09:57,876
up with some conclusions.

180
00:09:57,876 --> 00:10:00,042
That's how it's going to go

181
00:10:01,167 --> 00:10:05,959
So the only other people who we found

182
00:10:05,959 --> 00:10:10,959
on Twitter provided this working

183
00:10:10,959 --> 00:10:12,918
A social bot is a piece

184
00:10:12,918 --> 00:10:15,083
a user account

185
00:10:15,083 --> 00:10:17,959
and passes itself off as human.

186
00:10:17,959 --> 00:10:22,000
So kind of well, duh, but that

187
00:10:22,000 --> 00:10:24,999
working definition.

188
00:10:25,334 --> 00:10:27,709
You might also hear the term Sybil.

189
00:10:27,709 --> 00:10:29,999
Sybil was coined by a chap

190
00:10:29,999 --> 00:10:34,250
and it looks also I mean that's kind

191
00:10:34,250 --> 00:10:36,918
and it derives its name

192
00:10:36,918 --> 00:10:40,459
like 16 different personalities

193
00:10:40,459 --> 00:10:42,459
of the same name.

194
00:10:42,584 --> 00:10:45,626
Some of you might be going, well,

195
00:10:45,626 --> 00:10:48,876
because we have chap bots on IRC

196
00:10:48,876 --> 00:10:50,751
and you're right as well

197
00:10:50,751 --> 00:10:54,417
because this paper here was published

198
00:10:54,417 --> 00:10:55,999
ones, too.

199
00:10:56,209 --> 00:11:00,083
So I mentioned popularity being

200
00:11:00,083 --> 00:11:03,417
We've certainly had a lot of that

201
00:11:03,417 --> 00:11:05,999
accounts, Justin Bieber's got

202
00:11:05,999 --> 00:11:08,792
of fake followers apparently.

203
00:11:08,999 --> 00:11:11,834
But it does make you look

204
00:11:11,834 --> 00:11:12,999
Who's going to trust

205
00:11:12,999 --> 00:11:14,709
ten followers?

206
00:11:14,751 --> 00:11:16,709
Maybe not that many.

207
00:11:16,876 --> 00:11:19,167
The other thing of course is spam.

208
00:11:19,167 --> 00:11:20,999
So, you know, you get DMs.

209
00:11:20,999 --> 00:11:22,626
I'm wondering, you know,

210
00:11:22,626 --> 00:11:26,083
a subtle hint that maybe I should lose

211
00:11:27,083 --> 00:11:30,083
But, you know,

212
00:11:30,083 --> 00:11:33,083
that stuff is, casa berries,

213
00:11:33,083 --> 00:11:37,167
of what you saw a lot on Twitter

214
00:11:37,167 --> 00:11:39,918
clicks on that stuff, right.

215
00:11:39,999 --> 00:11:42,292
The other thing

216
00:11:42,292 --> 00:11:44,375
that are quite amusing.

217
00:11:44,375 --> 00:11:46,999
So in this one, for example,

218
00:11:46,999 --> 00:11:49,999
tweets to his followers,

219
00:11:49,999 --> 00:11:53,626
had a successful auction yesterday,

220
00:11:53,626 --> 00:11:57,083
the universe promptly replies,

221
00:11:57,083 --> 00:11:59,083
and do your thing.

222
00:11:59,083 --> 00:12:04,125
So one bot that I'm particularly sad

223
00:12:04,125 --> 00:12:10,667
by a gentleman called Neil Codner

224
00:12:10,667 --> 00:12:13,083
So every time you mention

225
00:12:13,083 --> 00:12:16,999
tweet you back and give you a ton

226
00:12:16,999 --> 00:12:20,626
such that you'd want to go

227
00:12:20,626 --> 00:12:22,999
just to receive the grief.

228
00:12:23,083 --> 00:12:26,626
But that no longer does that, which

229
00:12:26,626 --> 00:12:30,250
If Twitter could somehow work

230
00:12:30,250 --> 00:12:33,999
But then there was another chap,

231
00:12:33,999 --> 00:12:39,083
or Australia, who got tired

232
00:12:39,083 --> 00:12:42,083
So he created a bot, the clue is kind

233
00:12:42,083 --> 00:12:45,999
It's a Choring Test was the name

234
00:12:45,999 --> 00:12:49,083
So anytime somebody was talking

235
00:12:49,083 --> 00:12:53,334
or denying climate change,

236
00:12:53,334 --> 00:12:57,125
out bits of research and

237
00:12:57,125 --> 00:12:59,834
and set them straight.

238
00:12:59,876 --> 00:13:02,459
So you could apply that to religion,

239
00:13:02,459 --> 00:13:05,584
like to argue on your behalf

240
00:13:05,584 --> 00:13:07,167
to do, right?

241
00:13:08,083 --> 00:13:10,083
(Laughter.) It's a time saving device.

242
00:13:10,083 --> 00:00:33,000
Thanks for making it this early hour,

243
00:13:11,999 --> 00:13:15,209
Another gentleman, and this

244
00:13:15,209 --> 00:13:17,999
was Project Rio Boy

245
00:13:17,999 --> 00:13:20,709
he was one of the first

246
00:13:20,709 --> 00:13:24,292
at social bots and creating social bots

247
00:13:24,292 --> 00:13:26,959
of more like humans, really.

248
00:13:27,167 --> 00:13:29,959
So well worth checking out.

249
00:13:29,959 --> 00:13:30,751
I'm not going to go

250
00:13:30,751 --> 00:13:32,876
because we'll be short on time.

251
00:13:32,918 --> 00:13:35,209
Then politics.

252
00:13:35,209 --> 00:13:39,751
So I always mispronounce Roelof's

253
00:13:39,751 --> 00:13:44,375
the guys that wrote Martigo,

254
00:13:44,375 --> 00:13:49,334
in 2009 called Virtual Plots Real

255
00:13:49,334 --> 00:13:55,083
the concept here of what if both left

256
00:13:55,083 --> 00:14:00,792
false but credible information

257
00:14:00,792 --> 00:14:05,626
Well, a year later, on Twitter, we saw,

258
00:14:05,626 --> 00:14:09,292
between Martha Coakley

259
00:14:09,292 --> 00:14:14,083
before that vote, there was a lot of,

260
00:14:14,083 --> 00:14:17,999
or dragging Martha's name

261
00:14:17,999 --> 00:14:21,959
and that escalated pretty quickly so that

262
00:14:21,959 --> 00:14:25,459
of an overwhelming amount

263
00:14:25,459 --> 00:14:30,334
about Martha Coakley and the result,

264
00:14:30,334 --> 00:14:35,292
to what happened on Twitter,

265
00:14:35,292 --> 00:14:38,209
There was certainly a lot

266
00:14:38,209 --> 00:14:41,999
on the use of Twitter to sort

267
00:14:41,999 --> 00:14:45,999
and that got a name,

268
00:14:45,999 --> 00:14:49,751
and the name for that is swiftboating.

269
00:14:49,751 --> 00:14:57,667
(Laughter.) That actually went

270
00:14:57,667 --> 00:15:02,334
But (Laughter.) As does everything I do.

271
00:15:02,751 --> 00:15:08,999
So, but that gained its name not

272
00:15:09,334 --> 00:15:11,584
military veterans where you've got this

273
00:15:11,584 --> 00:15:14,209
of thing where you question

274
00:15:14,209 --> 00:15:17,792
and that apparently has some effect

275
00:15:17,792 --> 00:15:19,667
in certain states.

276
00:15:19,834 --> 00:15:23,876
So the other thing we've been seeing

277
00:15:23,876 --> 00:15:26,999
or bloggers or what have you were paid

278
00:15:26,999 --> 00:15:30,417
to write positive blogs

279
00:15:30,417 --> 00:15:32,667
up to his election.

280
00:15:32,667 --> 00:15:35,834
It's not a brilliant example of this,

281
00:15:36,125 --> 00:15:38,209
And that's got a term.

282
00:15:38,209 --> 00:15:39,999
It's called astroturfing

283
00:15:39,999 --> 00:15:43,584
a fake grass roots campaign

284
00:15:43,584 --> 00:15:47,209
to build support for a message,

285
00:15:47,209 --> 00:15:49,542
of agenda essentially.

286
00:15:50,542 --> 00:15:54,375
There was a group

287
00:15:54,375 --> 00:15:58,542
working on a project called Truthy,

288
00:15:58,542 --> 00:16:02,876
of political and other sorts

289
00:16:02,876 --> 00:16:06,459
of means and activity like that online.

290
00:16:06,584 --> 00:16:07,999
That's worth having a look.

291
00:16:07,999 --> 00:16:09,459
They've got some neat videos.

292
00:16:09,459 --> 00:16:10,959
You can put

293
00:16:10,959 --> 00:16:13,709
and look at where it originated from.

294
00:16:13,709 --> 00:16:15,083
So they're actually doing some really

295
00:16:15,083 --> 00:16:18,417
because they feel that what's

296
00:16:18,417 --> 00:16:21,501
a real impact in the real world and

297
00:16:21,501 --> 00:16:24,167
pay some attention to this space.

298
00:16:24,167 --> 00:16:25,918
So really interesting.

299
00:16:25,999 --> 00:16:29,083
Then we've got sockpuppets.

300
00:16:29,292 --> 00:16:33,542
It was HBGary's e mail hack when was

301
00:16:33,542 --> 00:16:37,876
like that kind of switched me

302
00:16:37,876 --> 00:16:40,751
and it mentioned that U.S.

303
00:16:40,751 --> 00:16:43,083
spy operations were

304
00:16:43,083 --> 00:16:45,292
for sort

305
00:16:45,292 --> 00:16:49,125
and there was some discussion

306
00:16:49,125 --> 00:16:52,999
in like the Arab spring context

307
00:16:52,999 --> 00:16:56,125
into a certain, you know, direction.

308
00:16:56,667 --> 00:16:59,334
The interesting thing

309
00:16:59,334 --> 00:17:01,999
is that this was cited

310
00:17:01,999 --> 00:17:06,876
and Kenneth Geers' 2009 paper as well,

311
00:17:06,876 --> 00:17:11,292
all over the world and encompassing

312
00:17:11,292 --> 00:17:14,751
could be programmed

313
00:17:14,751 --> 00:17:18,209
political, military or terrorist agenda.

314
00:17:18,375 --> 00:17:20,375
So there's some incentive there.

315
00:17:20,834 --> 00:17:25,792
Next, at Christmas this year,

316
00:17:25,792 --> 00:17:29,584
in the UK that was sending

317
00:17:29,584 --> 00:17:33,334
to users who were having problems.

318
00:17:33,334 --> 00:17:35,918
Please follow us and DM your mobile

319
00:17:35,918 --> 00:17:37,999
and password, thanks.

320
00:17:37,999 --> 00:17:39,792
That was the legitimate account.

321
00:17:39,876 --> 00:17:43,334
(Laughter.) That got picked

322
00:17:43,334 --> 00:17:47,959
in Australia called Troy Hunt,

323
00:17:47,959 --> 00:17:52,542
on Twitter or read his blogs,

324
00:17:52,542 --> 00:17:55,999
he was tweeting them

325
00:17:55,999 --> 00:18:00,999
from this account, my EE,

326
00:18:00,999 --> 00:18:03,667
it's subtly different.

327
00:18:03,667 --> 00:18:07,542
So he's looking then Troy Hunt

328
00:18:07,834 --> 00:18:11,999
I'm not saying Troy Hunt's susceptible,

329
00:18:11,999 --> 00:18:15,334
saying something like it was nice

330
00:18:15,334 --> 00:18:18,626
an engaging approach to attacking

331
00:18:18,626 --> 00:18:22,999
the MIEKER folks said, no,

332
00:18:23,918 --> 00:18:26,999
But the point is that

333
00:18:26,999 --> 00:18:30,709
on behalf of a brand and get

334
00:18:30,709 --> 00:18:34,667
of users aren't actually going

335
00:18:34,667 --> 00:18:37,083
or the non legitimate one.

336
00:18:37,584 --> 00:18:40,792
Then after our Black Hat talk

337
00:18:40,792 --> 00:18:44,334
up to us and started asking questions

338
00:18:44,334 --> 00:18:48,876
of misdirecting emergency resources

339
00:18:48,876 --> 00:18:52,250
and that made me think, oh, yeah,

340
00:18:52,250 --> 00:18:56,918
at Manchester University,

341
00:18:56,918 --> 00:18:58,083
I may be wrong about that.

342
00:18:58,083 --> 00:18:59,999
I'll add that to the speaker notes.

343
00:18:59,999 --> 00:19:01,876
But he was looking

344
00:19:01,876 --> 00:19:06,083
to the London riots and what they'd

345
00:19:06,083 --> 00:19:10,501
and the rumors were like there's lots

346
00:19:10,501 --> 00:19:13,918
don't go into this High Street,

347
00:19:13,918 --> 00:19:18,083
around Twitter at the, you know,

348
00:19:18,083 --> 00:19:20,999
almost nobody

349
00:19:20,999 --> 00:19:23,999
a riot and no stores were on fire.

350
00:19:23,999 --> 00:19:26,083
So they were looking

351
00:19:26,083 --> 00:19:29,209
to perpetuate and the truth catch up,

352
00:19:29,209 --> 00:19:31,125
which made me chuckle,

353
00:19:31,125 --> 00:19:33,209
a rumor gets halfway

354
00:19:33,209 --> 00:19:36,999
before the truth has a chance

355
00:19:37,751 --> 00:19:42,125
So there's another way you could sort

356
00:19:42,459 --> 00:19:44,626
(Clapping.) I mentioned Tim Hwang.

357
00:19:44,626 --> 00:19:46,918
He's

358
00:19:46,918 --> 00:19:50,959
Social because part of that,

359
00:19:50,959 --> 00:19:56,209
the social bots experiment was really

360
00:19:56,209 --> 00:20:00,834
after that it's morphed majorly,

361
00:20:00,834 --> 00:20:02,209
So he was like, oh, can you distort

362
00:20:02,209 --> 00:20:05,542
maybe stitch two disparate

363
00:20:05,999 --> 00:20:09,125
So that was kind of his position

364
00:20:09,125 --> 00:20:11,999
of research it a lit bit more.

365
00:20:11,999 --> 00:20:13,918
The other thing he was interested

366
00:20:13,918 --> 00:20:15,876
contagion and happiness.

367
00:20:15,876 --> 00:20:17,999
So you take a group

368
00:20:17,999 --> 00:20:21,792
when their happiness levels drop

369
00:20:21,792 --> 00:20:26,375
then you start injecting happy tweets

370
00:20:26,375 --> 00:20:30,209
analysis and turn

371
00:20:30,250 --> 00:20:33,542
So that's one

372
00:20:33,542 --> 00:20:35,250
I'm not sure how far they've got to.

373
00:20:35,250 --> 00:20:38,542
He also mentioned this concept

374
00:20:38,542 --> 00:20:40,417
So spread information

375
00:20:40,417 --> 00:20:44,334
see where they're challenged

376
00:20:44,334 --> 00:20:47,751
the most influential,

377
00:20:47,751 --> 00:20:49,751
and target them.

378
00:20:50,876 --> 00:20:55,501
And this chap here at University

379
00:20:55,501 --> 00:20:59,375
in quite some depth in relation

380
00:20:59,375 --> 00:21:04,501
the social bot steal 250 gigabytes

381
00:21:04,501 --> 00:21:07,167
to the paper from M.I.T.

382
00:21:08,209 --> 00:21:10,834
So he was one of the first

383
00:21:10,834 --> 00:21:13,083
to actually understand some

384
00:21:13,083 --> 00:21:15,999
behind who's engaging

385
00:21:15,999 --> 00:21:19,792
and to that end there are two groups

386
00:21:19,792 --> 00:21:22,709
and Trustworthy Cyberspace Initiative

387
00:21:22,709 --> 00:21:26,209
the Insider Threat Project

388
00:21:26,667 --> 00:21:28,209
So we set about looking

389
00:21:28,209 --> 00:21:33,083
to find people who would talk

390
00:21:33,709 --> 00:21:35,125
So our methodology.

391
00:21:35,125 --> 00:21:38,083
We used the winning bot code

392
00:21:38,083 --> 00:21:42,375
at ArrowFate's bot and we did that

393
00:21:42,375 --> 00:21:44,918
which I'll talk about.

394
00:21:44,918 --> 00:21:47,167
But essentially it was the same

395
00:21:47,167 --> 00:21:48,167
license.

396
00:21:48,334 --> 00:21:50,999
We had 610 participants.

397
00:21:50,999 --> 00:21:53,292
We got their Twitter information,

398
00:21:53,292 --> 00:21:57,125
and their Klout score and linguistic

399
00:21:57,125 --> 00:21:59,999
We divided them into two groups

400
00:21:59,999 --> 00:22:01,334
a bot.

401
00:22:01,918 --> 00:22:05,125
We did that just so that we could get

402
00:22:05,209 --> 00:22:06,959
That was all.

403
00:22:07,459 --> 00:22:10,792
We gave our bot an image

404
00:22:10,792 --> 00:22:15,334
along the lines of I've got my own teeth,

405
00:22:15,334 --> 00:22:18,501
like a young dude in New Zealand.

406
00:22:18,501 --> 00:22:20,709
So we switched some sort

407
00:22:20,834 --> 00:22:24,584
How the bot works

408
00:22:24,584 --> 00:22:28,542
in our case dog fashion,

409
00:22:28,542 --> 00:22:31,250
to WordPress and then tweet.

410
00:22:31,250 --> 00:22:33,501
So you'd get a picture like,

411
00:22:33,501 --> 00:22:36,083
and then you'd get a tweet

412
00:22:36,083 --> 00:22:37,999
like new blog post.

413
00:22:37,999 --> 00:22:40,250
So this is creating some content

414
00:22:40,501 --> 00:22:43,751
Then, next we use

415
00:22:43,751 --> 00:22:47,083
which actually then took let's see.

416
00:22:47,292 --> 00:22:50,999
If the weather went above 20° C

417
00:22:50,999 --> 00:22:54,083
is supposed to live, then she'd,

418
00:22:54,083 --> 00:22:58,375
a tweet and that would look something

419
00:22:58,375 --> 00:23:00,626
I wonder if I can switch the heating

420
00:23:00,626 --> 00:23:00,751
It's 21° C.

421
00:23:00,751 --> 00:23:05,999
And so this is creating some content

422
00:23:06,918 --> 00:23:12,626
Then we've got our target users, 305,

423
00:23:12,626 --> 00:23:14,292
If any followed back,

424
00:23:14,292 --> 00:23:17,083
in an interaction CSV file.

425
00:23:17,083 --> 00:23:18,584
That's how technical it got.

426
00:23:18,584 --> 00:23:19,834
No databases, just CSV.

427
00:23:21,959 --> 00:23:23,083
Yeah.

428
00:23:23,083 --> 00:23:25,167
Let's hear it for CSV files.

429
00:23:27,999 --> 00:23:32,959
(Applause.)    (Inaudible.)    CHRIS

430
00:23:33,083 --> 00:23:36,918
So then we started tweeting,

431
00:23:36,918 --> 00:23:38,959
shit basically.

432
00:23:39,083 --> 00:23:41,626
And this was not what we tweeted,

433
00:23:41,626 --> 00:23:45,501
is what ArrowFate's bot tweeted,

434
00:23:45,501 --> 00:23:48,834
as slightly misogynistic,

435
00:23:48,834 --> 00:23:52,542
felt like we might get into a little bit

436
00:23:52,542 --> 00:23:54,501
like this instead.

437
00:23:55,250 --> 00:23:58,999
(Laughter.) And then, you know,

438
00:23:58,999 --> 00:24:02,209
So these were obvious cues that

439
00:24:02,209 --> 00:24:06,792
the person she's pretending to be

440
00:24:06,792 --> 00:24:11,250
we swapped any references to cats

441
00:24:11,250 --> 00:24:12,959
That's essentially all we did.

442
00:24:12,959 --> 00:24:14,792
So you might think, yes, that's banal,

443
00:24:14,792 --> 00:24:18,334
to keep it as close as possible

444
00:24:18,334 --> 00:24:20,334
So we only switched dog for cats

445
00:24:20,334 --> 00:24:22,999
construed as slightly misogynistic.

446
00:24:23,209 --> 00:24:27,083
So all of the responses to that were,

447
00:24:27,083 --> 00:24:29,709
in an interactions file.

448
00:24:29,709 --> 00:24:32,083
Once we'd targeted all of the users, i.e.

449
00:24:32,083 --> 00:24:33,999
followed them all,

450
00:24:33,999 --> 00:24:37,125
to see if we could generate a response.

451
00:24:37,125 --> 00:24:39,959
This is how ArrowFate's bot actually

452
00:24:39,959 --> 00:24:43,626
So we had questions along the line

453
00:24:44,083 --> 00:24:50,626
What's better, a dog or a cat?

454
00:24:51,626 --> 00:24:56,083
And then we'd look at trying

455
00:24:56,083 --> 00:25:01,999
or maintain a conversation

456
00:25:01,999 --> 00:25:04,459
So ArrowFate's blog would just

457
00:25:04,459 --> 00:25:06,709
of tweets,

458
00:25:06,709 --> 00:25:08,751
an Eliezer engine.

459
00:25:08,751 --> 00:25:10,334
So you'd get something like this.

460
00:25:10,334 --> 00:25:11,334
Hello.

461
00:25:11,334 --> 00:25:13,709
And it would say, how

462
00:25:13,709 --> 00:25:15,626
What I liked about ArrowFate's bot

463
00:25:15,626 --> 00:25:18,459
sometimes, so we wanted

464
00:25:18,459 --> 00:25:21,375
of the random tweets

465
00:25:21,375 --> 00:25:24,125
of the coin as to what you'd get.

466
00:25:24,125 --> 00:25:29,250
You could get a "it's interesting"

467
00:25:29,250 --> 00:25:33,167
(Laughter.) So it could go kind

468
00:25:33,167 --> 00:25:36,083
But we also had to consider

469
00:25:36,083 --> 00:25:36,959
So, you know,

470
00:25:36,959 --> 00:25:40,167
in there so we could actually look

471
00:25:40,167 --> 00:25:42,751
out before she tweeted it out.

472
00:25:42,751 --> 00:25:44,918
And this is why, this is ArrowFate's bot.

473
00:25:45,083 --> 00:25:47,999
Do you have any pets, and if so, what?

474
00:25:49,626 --> 00:25:53,083
No, I don't currently have any pets

475
00:25:53,083 --> 00:25:55,999
a few years ago,

476
00:25:55,999 --> 00:25:57,999
LOL that rules.

477
00:25:57,999 --> 00:25:59,250
(Applause and laughter.).

478
00:26:01,083 --> 00:26:05,751
So props to ArrowFate

479
00:26:05,751 --> 00:26:09,083
after reading that blog post.

480
00:26:09,083 --> 00:26:10,375
So we've got a ton of limitations,

481
00:26:10,375 --> 00:26:12,209
about and you're right.

482
00:26:12,209 --> 00:26:15,501
We use basic measures of personality,

483
00:26:15,501 --> 00:26:19,626
We had a pretty basic and dumb social

484
00:26:19,626 --> 00:26:22,918
So maybe the questions had

485
00:26:22,918 --> 00:26:25,083
As the experiment progressed,

486
00:26:25,083 --> 00:26:28,959
and arguably gave it more credibility

487
00:26:28,959 --> 00:26:31,999
if the people knew it was

488
00:26:31,999 --> 00:26:33,834
playing along.

489
00:26:33,834 --> 00:26:35,918
So, but either way,

490
00:26:35,918 --> 00:26:38,334
pretty much identical

491
00:26:38,334 --> 00:26:41,999
which has already got some research

492
00:26:42,125 --> 00:26:45,167
Although we looked at the dimension

493
00:26:45,250 --> 00:26:46,999
So what did we find?

494
00:26:47,083 --> 00:26:50,125
Well, we got 20% return

495
00:26:50,125 --> 00:26:53,209
the ArrowFate bot had got 40%,

496
00:26:53,209 --> 00:26:55,876
for a number of reasons.

497
00:26:55,876 --> 00:26:56,999
We targeted a diverse range

498
00:26:56,999 --> 00:27:00,083
they targeted people who were liking

499
00:27:00,125 --> 00:27:01,999
We also had an old lady.

500
00:27:01,999 --> 00:27:03,083
They had a dude

501
00:27:03,083 --> 00:27:06,459
so there are a number

502
00:27:06,459 --> 00:27:10,667
But we had 124 interactions,

503
00:27:10,667 --> 00:27:16,999
which could be automated for sure,

504
00:27:16,999 --> 00:27:19,083
So the most

505
00:27:19,083 --> 00:27:22,999
interactions with 1 user,

506
00:27:22,999 --> 00:27:25,375
and there's kind

507
00:27:25,375 --> 00:27:29,083
to why we had 65 people replied just

508
00:27:29,167 --> 00:27:31,209
So the difference between us

509
00:27:31,209 --> 00:27:34,292
if we look at it on a sort

510
00:27:34,918 --> 00:27:37,959
We also had some interesting funny

511
00:27:37,959 --> 00:27:41,626
which ArrowFate had also noticed

512
00:27:41,626 --> 00:27:45,626
So you get we had this interaction

513
00:27:45,626 --> 00:27:51,250
and only nouns to describe yourself,

514
00:27:51,250 --> 00:27:56,375
it rhymes with runt, almost ten times,

515
00:27:56,375 --> 00:28:01,999
with the Eliezer engine,

516
00:28:02,083 --> 00:28:04,959
(Laughter.) Which gave me a chuckle

517
00:28:04,959 --> 00:28:05,876
And that was actually one

518
00:28:05,876 --> 00:28:07,999
of conducting this research

519
00:28:07,999 --> 00:28:09,834
at the interactions.

520
00:28:13,751 --> 00:28:19,292
(Laughter.) I'm laughing.

521
00:28:20,083 --> 00:28:23,459
(Laughter.) Bring it on, essentially,

522
00:28:23,667 --> 00:28:26,792
But we also got spotted as well,

523
00:28:26,792 --> 00:28:29,375
what do you do for a living.

524
00:28:29,667 --> 00:28:33,083
This is a pretty clear, you know,

525
00:28:33,083 --> 00:28:35,999
Write software

526
00:28:35,999 --> 00:28:37,834
And she responds

527
00:28:37,834 --> 00:28:42,584
you're right and when you're right,

528
00:28:43,834 --> 00:28:47,292
(Laughter.) Granny failing showing test

529
00:28:47,292 --> 00:28:50,167
is still far off or something like that.

530
00:28:50,167 --> 00:28:51,167
Brilliant.

531
00:28:51,167 --> 00:28:52,167
Okay.

532
00:28:52,250 --> 00:28:56,250
So looking at personality,

533
00:28:56,250 --> 00:29:00,999
he indicated people might not be

534
00:29:00,999 --> 00:29:03,083
from the '80s.

535
00:29:03,083 --> 00:29:06,250
Well, that's Ferris Bueller

536
00:29:07,876 --> 00:29:12,083
We found that extroverted users were

537
00:29:12,083 --> 00:29:17,876
statistically significantly more likely

538
00:29:17,876 --> 00:29:21,542
and that's using

539
00:29:21,542 --> 00:29:24,999
invented by a professor in Texas.

540
00:29:25,959 --> 00:29:29,626
We found that Klout score was

541
00:29:29,751 --> 00:29:31,999
There were no other personality traits,

542
00:29:31,999 --> 00:29:33,918
at a bunch of them.

543
00:29:33,918 --> 00:29:39,125
Klout score, friends, and followers

544
00:29:39,125 --> 00:29:43,334
relationship, albeit relatively weak.

545
00:29:43,459 --> 00:29:44,459
So what.

546
00:29:44,999 --> 00:29:47,709
Well, in terms of eLearning

547
00:29:47,709 --> 00:29:51,083
most eLearning around phishing

548
00:29:51,083 --> 00:29:54,999
of a one size fits all approach,

549
00:29:54,999 --> 00:29:58,834
about that, that's targeted more

550
00:29:58,834 --> 00:30:00,999
There's some papers that look

551
00:30:00,999 --> 00:30:03,417
and personality, and

552
00:30:03,417 --> 00:30:05,459
a better time

553
00:30:05,459 --> 00:30:09,292
over their learning experience,

554
00:30:09,292 --> 00:30:12,876
So I think there's some mileage

555
00:30:12,876 --> 00:30:15,709
relates in the context

556
00:30:15,709 --> 00:30:18,626
of corporate eLearning experience.

557
00:30:18,999 --> 00:30:22,375
So moving on quickly

558
00:30:22,501 --> 00:30:25,667
This was our baseline performance,

559
00:30:25,667 --> 00:30:27,999
true positives in green.

560
00:30:28,000 --> 00:30:31,250
We want to avoid this bit really

561
00:30:31,250 --> 00:30:34,999
the people that are probably going

562
00:30:34,999 --> 00:30:39,626
try and aim for this, which

563
00:30:39,834 --> 00:30:42,000
We're going to go light on some

564
00:30:42,292 --> 00:30:45,999
And the real aim here for us isn't

565
00:30:45,999 --> 00:30:49,584
is to achieve perfection,

566
00:30:49,584 --> 00:30:51,999
More realistically, what we want

567
00:30:51,999 --> 00:30:55,000
the false positive rate

568
00:30:55,000 --> 00:30:58,501
people and hopefully getting more

569
00:30:58,501 --> 00:31:00,250
to responding.

570
00:31:00,375 --> 00:31:03,334
So for the next five minutes we're going

571
00:31:03,334 --> 00:31:05,999
of sort of the machine learning stuff

572
00:31:05,999 --> 00:31:08,083
up five minutes after that.

573
00:31:08,083 --> 00:31:09,250
And so it's my pleasure to introduce Dr.

574
00:31:09,250 --> 00:31:11,999
Randall Wald, who I met here

575
00:31:11,999 --> 00:31:13,918
in the Q & A room.

576
00:31:18,999 --> 00:31:21,709
(Applause.)    RANDALL WALD: Okay.

577
00:31:21,709 --> 00:31:23,999
As Chris was saying,

578
00:31:23,999 --> 00:31:27,292
we wanted to actually work

579
00:31:27,292 --> 00:31:31,250
understand what's really going

580
00:31:31,250 --> 00:31:35,125
us make better predictions

581
00:31:35,125 --> 00:31:37,125
So it's very basic.

582
00:31:37,584 --> 00:31:39,792
This is what data looks like.

583
00:31:39,792 --> 00:31:41,125
You've got instances.

584
00:31:41,125 --> 00:31:43,501
In this case we've got three instances

585
00:31:43,501 --> 00:31:46,999
You're going to see, you know, Alice,

586
00:31:46,999 --> 00:31:51,918
and instance each instance has many

587
00:31:51,918 --> 00:31:53,918
In this case,

588
00:31:53,918 --> 00:31:56,542
the class attribute,

589
00:31:56,542 --> 00:31:59,959
and then also

590
00:31:59,959 --> 00:32:04,334
which are the least information we can

591
00:32:04,334 --> 00:32:06,626
So the concept here

592
00:32:06,626 --> 00:32:09,083
is labeled where we know

593
00:32:09,083 --> 00:32:12,626
whether or not they responded

594
00:32:12,626 --> 00:32:15,501
to use this to help us build

595
00:32:15,501 --> 00:32:19,584
in the future say I have a new person,

596
00:32:19,584 --> 00:32:22,959
to them, should my bot try

597
00:32:22,959 --> 00:32:24,334
Should I try to work with them?

598
00:32:24,334 --> 00:32:26,584
I'm going to use the features,

599
00:32:26,584 --> 00:32:29,999
out about this person to build

600
00:32:29,999 --> 00:32:33,584
whether or not that individual

601
00:32:33,584 --> 00:32:37,083
so they're going to be a good target

602
00:32:37,501 --> 00:32:40,292
So we did two separate experiments

603
00:32:40,292 --> 00:32:43,083
The first was to figure

604
00:32:43,083 --> 00:32:46,667
because the data we've been working

605
00:32:46,667 --> 00:32:48,999
all the users,

606
00:32:48,999 --> 00:32:50,999
properties of them.

607
00:32:50,999 --> 00:32:54,083
For example, the number of friends,

608
00:32:54,083 --> 00:32:57,709
how long they've been on Twitter,

609
00:32:57,709 --> 00:32:59,876
that sort of thing.

610
00:32:59,876 --> 00:33:01,999
We also took the content

611
00:33:01,999 --> 00:33:05,375
out like how many what type

612
00:33:05,375 --> 00:33:06,999
Are they using expletives a lot?

613
00:33:06,999 --> 00:33:08,375
Are they using I and me a lot?

614
00:33:08,375 --> 00:33:10,250
Are they using we?

615
00:33:10,250 --> 00:33:13,959
Different types of words like, that try

616
00:33:13,959 --> 00:33:18,999
is most relevant to the problem

617
00:33:18,999 --> 00:33:20,334
Will they reply with the bot?

618
00:33:20,334 --> 00:33:22,417
So we wanted to figure out, first,

619
00:33:22,417 --> 00:33:25,542
and then once we figured out,

620
00:33:25,542 --> 00:33:27,999
to build our models to use that

621
00:33:27,999 --> 00:33:31,959
whether an individual will interact

622
00:33:31,999 --> 00:33:35,209
So all of our experiments here

623
00:33:35,209 --> 00:33:37,751
the WEKA open source tool.

624
00:33:37,751 --> 00:33:41,083
It is available

625
00:33:41,083 --> 00:33:43,999
I don't know if it works on BSD,

626
00:33:43,999 --> 00:33:46,751
desktop operating systems.

627
00:33:46,999 --> 00:33:48,209
Java.

628
00:33:48,209 --> 00:33:49,918
You can extend it,

629
00:33:49,918 --> 00:33:53,626
adding additional tools to it,

630
00:33:53,626 --> 00:33:55,501
I encourage you all to download

631
00:33:55,501 --> 00:33:58,083
in learning more

632
00:33:58,209 --> 00:33:59,584
Our results.

633
00:33:59,876 --> 00:34:01,083
First, the interactive.

634
00:34:01,083 --> 00:34:02,876
We wanted

635
00:34:02,876 --> 00:34:06,667
likely to interact, so we used a bunch

636
00:34:06,667 --> 00:34:09,999
out with these three properties

637
00:34:09,999 --> 00:34:15,292
if they're going to interact with the bot,

638
00:34:15,292 --> 00:34:17,542
The first is the Klout score and

639
00:34:17,542 --> 00:34:19,542
number of followers.

640
00:34:19,542 --> 00:34:21,250
So those were, as Chris was saying,

641
00:34:21,250 --> 00:34:26,125
the features that were most relevant

642
00:34:26,125 --> 00:34:28,999
We also wanted to look

643
00:34:28,999 --> 00:34:30,999
We found something a little different.

644
00:34:30,999 --> 00:34:33,459
So same concept, all the things,

645
00:34:33,459 --> 00:34:35,417
is number one.

646
00:34:35,417 --> 00:34:37,542
But percent follow Friday

647
00:34:37,542 --> 00:34:39,083
feature here.

648
00:34:39,083 --> 00:34:41,626
So that's basically what percentage

649
00:34:41,626 --> 00:34:44,876
#followFriday, that sort of thing,

650
00:34:44,876 --> 00:34:47,999
because we have these two different

651
00:34:47,999 --> 00:34:49,999
a little differently.

652
00:34:50,125 --> 00:34:51,876
Once we figured out these features,

653
00:34:51,876 --> 00:34:54,459
to build models that could actually

654
00:34:54,459 --> 00:34:57,834
into whether they're going to interact

655
00:34:57,834 --> 00:34:59,501
to reply or not.

656
00:34:59,626 --> 00:35:02,999
So we built a number

657
00:35:02,999 --> 00:35:06,999
features, etc., and found

658
00:35:06,999 --> 00:35:11,083
rate of 61%, true negative of 71%, or

659
00:35:11,083 --> 00:35:15,501
we see here we have true positives,

660
00:35:15,501 --> 00:35:17,834
and false negatives.

661
00:35:17,834 --> 00:35:19,999
Now, you're going to look at this

662
00:35:19,999 --> 00:35:24,209
but your model still gave more false

663
00:35:24,375 --> 00:35:28,209
If the model predicted that someone

664
00:35:28,209 --> 00:35:31,999
it's still going

665
00:35:31,999 --> 00:35:35,626
However, compared to the gray boxes,

666
00:35:35,626 --> 00:35:38,417
to everyone and try

667
00:35:38,417 --> 00:35:41,792
we're able to eliminate

668
00:35:41,792 --> 00:35:45,167
otherwise have gotten the message,

669
00:35:45,167 --> 00:35:47,292
to respond anyway,

670
00:35:47,292 --> 00:35:50,083
of hitting people we care about.

671
00:35:50,083 --> 00:35:51,584
So, remember,

672
00:35:51,584 --> 00:35:53,959
a message, somebody is not going

673
00:35:53,959 --> 00:35:56,751
a chance they're going

674
00:35:56,751 --> 00:35:59,167
So you want to minimize the number

675
00:35:59,167 --> 00:36:02,292
to maximize the effectiveness

676
00:36:02,292 --> 00:36:05,876
And we did related models

677
00:36:05,876 --> 00:36:06,999
That's a little more challenging

678
00:36:06,999 --> 00:36:09,459
with a performance a little worse here.

679
00:36:09,459 --> 00:36:12,375
And we found also that different models

680
00:36:12,375 --> 00:36:15,083
And this data mining is important

681
00:36:15,083 --> 00:36:17,999
we always just use the best model.

682
00:36:17,999 --> 00:36:19,167
What's the challenge

683
00:36:19,167 --> 00:36:21,375
and looking how they perform.

684
00:36:21,375 --> 00:36:23,334
Different data sets are going

685
00:36:23,334 --> 00:36:25,626
the optimal model will vary.

686
00:36:25,626 --> 00:36:28,417
We see also here similar performance

687
00:36:28,417 --> 00:36:32,334
more false positives than true positives.

688
00:36:32,334 --> 00:36:34,334
But we're significantly able

689
00:36:34,334 --> 00:36:37,459
of our model, improve the chances

690
00:36:37,459 --> 00:36:40,959
to hit those people we care about,

691
00:36:40,959 --> 00:36:42,792
to the bot,

692
00:36:42,792 --> 00:36:46,542
a better bot that will be able

693
00:36:46,542 --> 00:36:49,375
before it gets reported on Twitter.

694
00:36:49,375 --> 00:36:52,834
So overall we found

695
00:36:52,834 --> 00:36:55,792
is that we have our a model able

696
00:36:55,792 --> 00:37:02,375
the features that are most important

697
00:37:02,375 --> 00:37:06,501
We also found that the two data sets,

698
00:37:06,501 --> 00:37:08,999
So actually Klout score, which

699
00:37:08,999 --> 00:37:12,375
is important for interaction and reply.

700
00:37:12,542 --> 00:37:14,999
These whether the number

701
00:37:14,999 --> 00:37:17,292
mattered more

702
00:37:17,292 --> 00:37:20,292
followFriday matters more for replies.

703
00:37:20,459 --> 00:37:23,999
And even though our models were not

704
00:37:23,999 --> 00:37:27,083
we still have a large number

705
00:37:27,083 --> 00:37:31,584
to build models that let us build bots

706
00:37:31,584 --> 00:37:34,792
likely to be susceptible to the bot.

707
00:37:34,792 --> 00:37:37,667
So with that, I'm going to give it back

708
00:37:37,751 --> 00:37:40,250
CHRIS SUMNER: Thanks, Randall.

709
00:37:40,999 --> 00:37:44,501
So wrapping

710
00:37:44,792 --> 00:37:48,167
So, again,

711
00:37:48,167 --> 00:37:53,709
the greatest risk

712
00:37:53,834 --> 00:37:58,125
We can look at that in maybe just

713
00:37:58,334 --> 00:38:02,999
And bot masters could use machine

714
00:38:02,999 --> 00:38:04,999
of their bots.

715
00:38:04,999 --> 00:38:06,834
So the key here isn't that

716
00:38:06,834 --> 00:38:08,999
at to improve their bot,

717
00:38:08,999 --> 00:38:10,959
to our particular bot.

718
00:38:10,959 --> 00:38:13,626
When Wagner looked

719
00:38:13,626 --> 00:38:16,959
the sort

720
00:38:16,959 --> 00:38:20,876
because they had a larger majority

721
00:38:20,876 --> 00:38:23,292
larger minority class.

722
00:38:23,292 --> 00:38:26,292
However, so you create a bot,

723
00:38:26,292 --> 00:38:28,501
You use machine learning

724
00:38:28,501 --> 00:38:31,375
and then you target it

725
00:38:31,375 --> 00:38:34,667
of how we think this could be

726
00:38:34,667 --> 00:38:36,501
We're not suggesting people do that.

727
00:38:36,501 --> 00:38:38,501
But marketers have got

728
00:38:38,501 --> 00:38:40,999
into that sort of behavior.

729
00:38:40,999 --> 00:38:43,292
And marketers, if you ever go

730
00:38:43,292 --> 00:38:45,834
conference, are actually sort

731
00:38:45,834 --> 00:38:48,626
to improve engagement and understand

732
00:38:48,626 --> 00:38:50,876
all of that sort of stuff.

733
00:38:50,876 --> 00:38:53,959
They are kind of light years ahead

734
00:38:54,083 --> 00:38:55,751
Propagandists

735
00:38:55,751 --> 00:38:59,375
to find people who are more likely

736
00:38:59,375 --> 00:39:02,584
maybe propagate false messages

737
00:39:02,584 --> 00:39:06,375
mentioned that Wecker's got

738
00:39:06,375 --> 00:39:08,999
Well, you can build that model

739
00:39:08,999 --> 00:39:11,999
so that you get a bunch

740
00:39:11,999 --> 00:39:15,334
the likelihood in a different color here

741
00:39:15,334 --> 00:39:17,501
is going to respond.

742
00:39:17,501 --> 00:39:19,292
So find the most gullible potentially

743
00:39:19,292 --> 00:39:22,999
in your organization and reduce your

744
00:39:23,626 --> 00:39:25,918
The other thing, this

745
00:39:25,918 --> 00:39:29,167
this could be used in terms

746
00:39:29,167 --> 00:39:32,083
So help people understand

747
00:39:32,083 --> 00:39:35,292
or more at risk from a social bot.

748
00:39:35,292 --> 00:39:37,584
We see that with browsers and now

749
00:39:37,584 --> 00:39:39,999
in a social network environment.

750
00:39:40,375 --> 00:39:43,709
So you probably want

751
00:39:45,584 --> 00:39:47,999
(Laughter.) So training.

752
00:39:47,999 --> 00:39:49,083
I mentioned training.

753
00:39:49,125 --> 00:39:51,459
You know, focus

754
00:39:51,459 --> 00:39:52,876
at risk.

755
00:39:52,876 --> 00:39:55,667
So maybe it's your sales team who are

756
00:39:55,667 --> 00:39:58,501
more extroverted naturally.

757
00:39:58,626 --> 00:40:02,667
Then in terms of future research here,

758
00:40:02,667 --> 00:40:05,417
of the big five, specifically

759
00:40:05,417 --> 00:40:07,834
which has many facets.

760
00:40:08,209 --> 00:40:11,209
Specifically, I'm thinking impulsivity.

761
00:40:11,209 --> 00:40:13,250
Impulsivity seems to be related

762
00:40:13,250 --> 00:40:16,209
to phishing messages,

763
00:40:16,209 --> 00:40:20,167
cognitive reflective test to see

764
00:40:20,209 --> 00:40:22,042
So here's an example of one

765
00:40:22,042 --> 00:40:24,999
in the cognitive reflective test.

766
00:40:24,999 --> 00:40:27,834
A bat and a ball cost $1.10 in total.

767
00:40:27,834 --> 00:40:29,999
A bat costs $1 more than the ball.

768
00:40:29,999 --> 00:40:31,626
How much does the ball cost?

769
00:40:31,626 --> 00:40:37,417
So get in that getting that wrong

770
00:40:37,417 --> 00:40:40,918
I think there are three questions

771
00:40:40,999 --> 00:40:44,918
So also maybe

772
00:40:44,918 --> 00:40:45,999
Ours was a one size fits

773
00:40:45,999 --> 00:40:48,334
but you could actually start looking

774
00:40:48,334 --> 00:40:52,709
of the call group you're following

775
00:40:52,751 --> 00:40:56,125
But it's not all negative necessarily.

776
00:40:57,959 --> 00:41:01,375
There's a tall German gentleman

777
00:41:01,375 --> 00:41:05,501
who does a lot of work in this field

778
00:41:05,501 --> 00:41:08,999
of his videos

779
00:41:08,999 --> 00:41:11,626
He mentioned that OK cupid had

780
00:41:11,626 --> 00:41:14,167
they created their own bot.

781
00:41:14,167 --> 00:41:16,834
And what they did when

782
00:41:16,834 --> 00:41:21,501
is they created a replica

783
00:41:21,501 --> 00:41:25,959
talk to other bots and now there's

784
00:41:25,959 --> 00:41:30,999
of OK cupid where you've just got what

785
00:41:30,999 --> 00:41:35,417
on bot (Laughter.) (Applause.)

786
00:41:35,417 --> 00:41:40,459
whether that's true or not,

787
00:41:40,459 --> 00:41:40,792
I have no reason

788
00:41:40,792 --> 00:41:42,999
because he's taller than I am.

789
00:41:42,999 --> 00:41:46,626
(Laughter.)    CHRIS SUMNER: So

790
00:41:46,626 --> 00:41:50,083
this gets back to Roelof Temmingh

791
00:41:50,083 --> 00:41:53,209
illustrations

792
00:41:53,209 --> 00:41:57,999
activity of the computer should reveal

793
00:41:57,999 --> 00:41:59,792
But maybe that's going

794
00:41:59,792 --> 00:42:02,501
to need more and more time

795
00:42:02,501 --> 00:42:04,501
or maybe you don't care.

796
00:42:04,501 --> 00:42:07,999
But, you know, the key thing here

797
00:42:07,999 --> 00:42:10,834
learning to improve the performance

798
00:42:10,834 --> 00:42:13,667
but it also shows that that's actually

799
00:42:13,667 --> 00:42:17,167
the security community and folks

800
00:42:17,167 --> 00:42:20,083
about tackling because this sort

801
00:42:20,083 --> 00:42:24,167
on social media could be making users

802
00:42:24,167 --> 00:42:28,542
to respond and perhaps need more

803
00:42:28,751 --> 00:42:32,751
So that, ladies and gentlemen,

804
00:42:32,751 --> 00:42:34,999
We'll take questions

805
00:42:34,999 --> 00:42:36,250
(Applause.) Thanks a lot.

806
00:42:36,250 --> 00:42:39,501
So before you all leave,

807
00:42:39,501 --> 00:42:39,501
You know,

808
00:42:39,501 --> 00:42:39,501
we're doing those pub crawly things,

809
00:42:39,501 --> 00:42:40,876
of alcohol and strippers.

810
00:42:40,876 --> 00:42:41,876
Yeah.

811
00:42:41,876 --> 00:42:43,999
Maybe you shouldn't be bringing

812
00:42:43,999 --> 00:42:44,999
Just saying.

813
00:42:44,999 --> 00:42:45,999
You're an adult.

814
00:42:45,999 --> 00:42:49,667
You can make your own evaluated

815
00:42:49,667 --> 00:42:51,918
But    What did they call the party,

816
00:42:51,918 --> 00:42:53,999
Well, they did call it the petting zoo.

817
00:42:53,999 --> 00:42:56,834
But if there's a petting zoo in Vegas

818
00:42:56,834 --> 00:42:57,834
Anywhere.

819
00:42:57,834 --> 00:42:57,834
(Laughter.)    Then maybe, just maybe,

820
00:42:57,834 --> 00:42:59,999
about what content

821
00:42:59,999 --> 00:43:00,999
That's all.

822
00:43:00,999 --> 00:43:02,167
Thanks for the updates!

823
00:43:02,167 --> 00:43:03,167
Goodbye!

824
00:43:03,167 --> 00:43:06,501
(Applause.)    Could I ask

825
00:43:06,501 --> 00:43:07,501
Chris?

826
00:43:07,501 --> 00:43:09,626
Yeah, if you want to come

827
00:43:09,626 --> 00:43:09,626
Well, I just wanted

828
00:43:09,626 --> 00:43:12,834
is that a WhiteHouse.gov,

829
00:43:12,834 --> 00:43:15,999
Or is that a data mining project

830
00:43:15,999 --> 00:43:15,999
CHRIS SUMNER: I'm afraid I'm not

831
00:43:15,999 --> 00:43:17,999
because the next speaker is coming on.

832
00:43:17,999 --> 00:43:17,999
So, if you want to hop up,

833
00:43:17,999 --> 00:43:19,334
and maybe understand more.

834
00:43:19,334 --> 00:43:20,501
Thank you very much.

835
00:43:20,501 --> 00:43:21,501
Appreciate it.

836
00:43:21,501 --> 00:43:22,999
CHRIS SUMNER: No problem.

837
00:43:22,999 --> 00:43:25,667
***** This is being provided

838
00:43:25,667 --> 00:43:25,667
Communication Access Realtime

839
00:43:25,667 --> 00:43:25,667
in order

840
00:43:25,667 --> 00:43:25,667
accessibility and may not be

841
00:43:25,667 --> 00:43:26,709
of the Proceedings.