1
00:00:00,000 --> 00:00:04,580
So, before we begin, I just want to give a couple of shout-outs today.

2
00:00:04,740 --> 00:00:07,520
Just a little preview of the Packet Hacking Village.

3
00:00:09,640 --> 00:00:11,300
Later on today, we've got Chef.

4
00:00:11,700 --> 00:00:13,100
He'll be speaking later. What time?

5
00:00:15,180 --> 00:00:15,660
1610.

6
00:00:16,720 --> 00:00:17,840
We've got 1610.

7
00:00:18,060 --> 00:00:23,420
And also, I want to give a very warm welcome to DEVCON from Japan.

8
00:00:23,960 --> 00:00:27,740
Megumi, you'll be giving a training at a workshops area.

9
00:00:27,740 --> 00:00:30,980
And welcome to DEVCON, and thank you so much.

10
00:00:31,420 --> 00:00:32,100
Thank you.

11
00:00:32,780 --> 00:00:36,880
So, right now, we're going to start off with a third talk at a speaker workshop.

12
00:00:37,220 --> 00:00:42,640
And without much ado, it is absolutely my pleasure to introduce you to Princess Leia.

13
00:00:45,940 --> 00:00:46,840
Hi, everyone.

14
00:00:47,000 --> 00:00:48,600
Thank you so much for your patience.

15
00:00:49,080 --> 00:00:53,840
Sometimes the Linux works with displays, and sometimes it tells me to go fuck myself.

16
00:00:54,000 --> 00:00:57,040
So, the fact that it's up here right now, I'm very happy.

17
00:00:57,140 --> 00:00:57,720
I may look old.

18
00:00:57,740 --> 00:01:02,860
I may look over here, because my presenter view is about this big, so I can't actually read it on my laptop.

19
00:01:03,100 --> 00:01:05,280
So, I apologize in advance if I'm glancing over a bit.

20
00:01:06,280 --> 00:01:10,660
So, this is iron sights for your data, predictive analytics for your blue team.

21
00:01:12,260 --> 00:01:14,420
Before I start, I should tell you a little bit about myself.

22
00:01:14,840 --> 00:01:18,300
I actually have a master's in education in ABD and research psychology.

23
00:01:18,980 --> 00:01:24,080
I almost finished the PhD, and then I realized I didn't want to be a professor for the rest of my life.

24
00:01:24,440 --> 00:01:25,720
And I noped out of there.

25
00:01:26,760 --> 00:01:34,320
I also realized I really enjoyed doing the data part, not so much the research on some of the topics I was doing in research psychology.

26
00:01:34,620 --> 00:01:36,280
So, I just went back into data analytics.

27
00:01:36,800 --> 00:01:40,260
I've been a data analyst in higher education off and on for about 13 years.

28
00:01:41,160 --> 00:01:43,500
I love data, and so I play with it in my spare time.

29
00:01:44,160 --> 00:01:45,480
I'm also a fiber artist.

30
00:01:45,560 --> 00:01:47,400
I knit, crochet, learning to sew.

31
00:01:48,300 --> 00:01:49,040
That's my cat.

32
00:01:50,440 --> 00:01:53,360
And my Twitter is at sweetgirl, or you can email me.

33
00:01:54,920 --> 00:01:57,940
And if any of you want to try to say my last name, it's Figueroa.

34
00:01:58,840 --> 00:02:00,300
You can just call me Princess Leia.

35
00:02:02,320 --> 00:02:02,640
Okay.

36
00:02:02,960 --> 00:02:08,220
So, before we start into looking into predictive analytics, one of the things we needed to look at is where we're currently at.

37
00:02:08,720 --> 00:02:14,440
So, one of the things I use is the database incidence response database that's put out by Verizon.

38
00:02:14,680 --> 00:02:15,640
It's one of the nicer ones.

39
00:02:15,740 --> 00:02:16,600
It's a little more comprehensive.

40
00:02:17,580 --> 00:02:21,440
In 2016, there were more than 64,000 reported security incidents.

41
00:02:21,760 --> 00:02:23,280
I'd like to point out that all of these are.

42
00:02:23,360 --> 00:02:25,680
These are more responsibly disclosed items.

43
00:02:25,780 --> 00:02:28,000
These are not the ones that the companies pretend didn't happen.

44
00:02:28,860 --> 00:02:32,840
More than 3,100 of them were confirmed data breaches where people's data were leaked.

45
00:02:33,800 --> 00:02:36,500
2015, you're looking at 79,000 incidents.

46
00:02:37,320 --> 00:02:40,380
Not as many confirmed data breaches, but there's still a lot.

47
00:02:40,540 --> 00:02:41,560
Same thing in 2014.

48
00:02:41,960 --> 00:02:46,040
I anticipate 2017 is going to have an even higher number of data breaches attached to it.

49
00:02:48,860 --> 00:02:50,980
So, but it can't possibly be that bad, right?

50
00:02:51,120 --> 00:02:52,420
It's not on the news all the time.

51
00:02:53,940 --> 00:02:56,160
Well, here's a year in review from last year.

52
00:03:02,620 --> 00:03:07,320
I'd also like to point out, if you have Blue Cross Blue Shield, make sure that they tell you if they have a data breach.

53
00:03:07,760 --> 00:03:10,400
Over 5 million people's records were leaked last year.

54
00:03:12,580 --> 00:03:14,280
Or any health care, for that matter.

55
00:03:18,380 --> 00:03:22,420
So, these are just the big ones where there was enough data breach that it could possibly.

56
00:03:23,360 --> 00:03:25,240
Have accounted for some kind of issue.

57
00:03:25,640 --> 00:03:28,460
That doesn't include all of the data breaches.

58
00:03:28,580 --> 00:03:29,780
I just picked the largest ones.

59
00:03:33,280 --> 00:03:34,660
So, yeah, it's pretty bad.

60
00:03:35,200 --> 00:03:36,500
And I apologize for Linux.

61
00:03:36,720 --> 00:03:39,480
This was really cute where he freaked out and fell down.

62
00:03:40,700 --> 00:03:41,020
Linux.

63
00:03:43,260 --> 00:03:44,340
So, what the hell do we do?

64
00:03:46,360 --> 00:03:47,420
Well, you can run around and scream.

65
00:03:48,260 --> 00:03:50,960
I know a lot of you work with people that are like, oh, God, I don't know what.

66
00:03:51,080 --> 00:03:52,860
We're just going to let them have all the data because they have it anywhere.

67
00:03:52,960 --> 00:03:53,340
It's no point.

68
00:03:55,320 --> 00:03:57,240
Or you can weaponize your data.

69
00:03:58,880 --> 00:04:03,300
So, the underlying philosophy behind this talk is that past behavior often predicts future behavior.

70
00:04:03,540 --> 00:04:07,460
And I know, I'm sure some of you who have ever taken psychology at any point have heard this.

71
00:04:09,880 --> 00:04:12,740
It's often, in other words, people often repeat the same behaviors.

72
00:04:12,920 --> 00:04:17,500
And a lot of people are like, well, it's not necessarily people, but people are behind all the data breaches.

73
00:04:18,000 --> 00:04:19,200
They design the botnets.

74
00:04:19,300 --> 00:04:20,260
They design everything else.

75
00:04:20,360 --> 00:04:21,420
So, we're looking at that philosophy.

76
00:04:22,420 --> 00:04:23,280
So, what the heck?

77
00:04:23,360 --> 00:04:24,260
What does this have to do with security?

78
00:04:25,240 --> 00:04:30,560
A lot of times, data breaches, attacks, threats, or vulnerabilities often come up in cycles.

79
00:04:30,820 --> 00:04:37,820
The same types of vulnerabilities that caused X company to get hit are most likely the ones that are going to cause it to get hit again.

80
00:04:39,120 --> 00:04:43,420
Attacks and attackers and attack vectors typically follow patterns you can recognize.

81
00:04:44,160 --> 00:04:47,280
Usually, there's one big name associated with some type of attack.

82
00:04:47,420 --> 00:04:50,760
Or it's an attack that has been recycled and just, you know, made it pretty.

83
00:04:50,940 --> 00:04:52,100
And then it continues on.

84
00:04:52,200 --> 00:04:53,340
But you can trace it back to the record.

85
00:04:53,340 --> 00:04:53,820
And you can trace it back to the original source.

86
00:04:54,200 --> 00:04:59,540
So, learning about where the stuff came from can help you understand what's going to happen next.

87
00:05:02,700 --> 00:05:04,260
So, how do I weaponize my data?

88
00:05:04,720 --> 00:05:07,740
Well, before you can weaponize it, you have to understand how to approach it.

89
00:05:08,680 --> 00:05:10,580
This is actually a two-pronged approach.

90
00:05:10,800 --> 00:05:12,420
It's data mining and predictive analytics.

91
00:05:15,540 --> 00:05:17,240
So, why bother mining your data?

92
00:05:17,440 --> 00:05:18,400
There's a lot of it.

93
00:05:18,520 --> 00:05:21,600
I know any of you who run logs have pages and pages and pages.

94
00:05:23,340 --> 00:05:26,460
And just massive amounts of data storage.

95
00:05:26,900 --> 00:05:28,440
You know, why should you use predictive analytics?

96
00:05:28,660 --> 00:05:30,000
Well, I'll tell you one thing.

97
00:05:30,820 --> 00:05:31,860
Facebook uses it.

98
00:05:32,000 --> 00:05:33,140
Big Brother uses it.

99
00:05:33,580 --> 00:05:35,320
The three-letter agencies use it.

100
00:05:35,620 --> 00:05:37,340
Pretty much everybody else uses it.

101
00:05:37,420 --> 00:05:38,200
So, why shouldn't you?

102
00:05:42,060 --> 00:05:43,580
No, more seriously, why bother?

103
00:05:43,820 --> 00:05:47,680
Well, one thing is forecasts can predict attack surfaces, vectors, and actors.

104
00:05:48,300 --> 00:05:52,420
Just like in weather prediction, forecasts can kind of help you prepare for something bad.

105
00:05:52,560 --> 00:05:53,320
If you know a hurricane, you can predict it.

106
00:05:53,320 --> 00:05:55,680
If you know a hurricane's coming, you're not going to sit in your house that's directly in the path.

107
00:05:57,100 --> 00:05:57,860
Well, you might.

108
00:05:58,040 --> 00:06:00,760
But most people won't do that.

109
00:06:00,960 --> 00:06:05,260
You know, and there's too many breach reports, not enough time, too many ice giants, not enough Nordic gods.

110
00:06:06,520 --> 00:06:07,500
It's not perfect.

111
00:06:07,720 --> 00:06:10,460
Knowing what your data is can put you in a more defensible position.

112
00:06:10,660 --> 00:06:15,780
Because, like most people, you probably have information coming at you from every source.

113
00:06:16,080 --> 00:06:23,300
And not having a way to make it easier to understand and codify it makes it a lot harder to actually see what's going on.

114
00:06:23,300 --> 00:06:24,520
It's just a bunch of noise.

115
00:06:25,920 --> 00:06:28,460
This is an ongoing effort that requires a team effort.

116
00:06:29,000 --> 00:06:33,280
But, if you do this kind of thing, it can make it so you can effectively fight back.

117
00:06:36,600 --> 00:06:38,900
So, I know some of you are here, like, but I have this.

118
00:06:39,640 --> 00:06:41,520
Palantir, Maltego, Gabana, Splunk.

119
00:06:41,680 --> 00:06:43,940
You have all these fancy tools for working with your data.

120
00:06:44,280 --> 00:06:46,940
You can use them, but only after you look at your data.

121
00:06:47,620 --> 00:06:51,600
Actually, if you use one of those things first without cleaning your data and trying to figure out what it is,

122
00:06:51,660 --> 00:06:53,280
it's basically like you're using a firehook.

123
00:06:53,280 --> 00:06:54,080
You're using a firehook to fill a shot glass.

124
00:06:55,580 --> 00:06:57,720
Everyone is going to get wet and nothing is going to get done.

125
00:07:02,000 --> 00:07:06,180
So, before we talk about weaponized data and doing the data mining and predictive analytics,

126
00:07:06,340 --> 00:07:09,560
this is what an actual weaponized data feedback loop would look like.

127
00:07:09,760 --> 00:07:10,920
You start with the data collection.

128
00:07:11,420 --> 00:07:14,760
Data mining allows you to clean it and codify what you need,

129
00:07:14,880 --> 00:07:20,080
which allows you to form a predictive analytics or actually perform predictive analytics on the data.

130
00:07:20,160 --> 00:07:21,420
And you can forecast it.

131
00:07:21,540 --> 00:07:22,720
Once you forecast it,

132
00:07:22,720 --> 00:07:26,520
you have a narrower scope to look at and you can start collecting data within that area.

133
00:07:26,720 --> 00:07:28,400
Don't stop collecting the other data, though.

134
00:07:28,620 --> 00:07:30,320
Seriously, just you need it.

135
00:07:30,780 --> 00:07:33,760
But that allows you to look further at the topics that you need.

136
00:07:34,000 --> 00:07:38,060
Say, for instance, you have a problem with passwords.

137
00:07:38,440 --> 00:07:42,260
Somehow logins are getting compromised left and right.

138
00:07:42,520 --> 00:07:45,100
You have other issues in the network, but that's the big one.

139
00:07:45,500 --> 00:07:47,140
With this, it'll let you know, hey,

140
00:07:47,300 --> 00:07:51,860
I've noticed that it's always the top officials in this business that are having

141
00:07:51,860 --> 00:07:53,060
their passwords brute forced.

142
00:07:54,020 --> 00:07:55,800
That allows you to focus better on them.

143
00:07:56,180 --> 00:07:58,760
Maybe get them a password manager or something along those lines.

144
00:07:58,840 --> 00:08:00,220
But that tells you where you need to focus.

145
00:08:00,540 --> 00:08:04,780
The feedback loop also makes sure the forecast becomes a lot more accurate.

146
00:08:05,080 --> 00:08:07,720
Without the feedback loop, you're just shouting into the void.

147
00:08:11,080 --> 00:08:14,440
So before you start, you need to have a framework for collecting data.

148
00:08:14,780 --> 00:08:16,820
A good framework includes incident tracking.

149
00:08:17,520 --> 00:08:18,920
You need victim demographics.

150
00:08:19,300 --> 00:08:20,960
I know people don't think that's that important.

151
00:08:21,860 --> 00:08:23,980
Sometimes you can find out that there's something going on.

152
00:08:24,100 --> 00:08:26,220
Maybe it's a specific subset of users

153
00:08:26,220 --> 00:08:30,940
who have this Windows box at their home that are being attacked.

154
00:08:33,500 --> 00:08:36,100
Sorry, it's like in my face. Is that better?

155
00:08:36,460 --> 00:08:37,440
OK, sorry.

156
00:08:39,580 --> 00:08:41,160
My mouth doesn't open that big.

157
00:08:42,640 --> 00:08:46,160
OK, a good framework for data collection should include incident tracking.

158
00:08:46,500 --> 00:08:49,020
Each incident should have its own special identifier.

159
00:08:49,560 --> 00:08:50,640
If you have

160
00:08:51,720 --> 00:08:54,040
redundant identifiers, it'll just make the data dirtier.

161
00:08:54,160 --> 00:08:55,520
You also need victim demographics.

162
00:08:56,100 --> 00:08:57,140
Trust me on this.

163
00:08:57,280 --> 00:08:59,240
It can be a pain in the butt, but if you have

164
00:08:59,920 --> 00:09:04,120
something that pops up consistently in victim demographics, then that can help

165
00:09:04,120 --> 00:09:07,220
you pinpoint what's going on or where some issues are or where you need to focus.

166
00:09:08,340 --> 00:09:09,900
You also need an incident description.

167
00:09:10,600 --> 00:09:12,020
I've looked through some of these,

168
00:09:12,660 --> 00:09:16,100
a lot of them, and my favorite was person was hacked.

169
00:09:17,500 --> 00:09:20,300
OK, that's not telling me anything. What happened?

170
00:09:20,640 --> 00:09:23,860
I don't care if it takes you four pages to describe it.

171
00:09:23,920 --> 00:09:25,040
If you have a good data person,

172
00:09:25,160 --> 00:09:28,500
they can figure out what you're saying, but telling me the person was hacked means nothing.

173
00:09:30,120 --> 00:09:30,780
Discovering response.

174
00:09:31,280 --> 00:09:33,320
What did you do? What happened as a result?

175
00:09:33,580 --> 00:09:34,740
And impact assessment.

176
00:09:35,440 --> 00:09:38,980
Sometimes you don't know the impact and you can say that, but this kind of data

177
00:09:38,980 --> 00:09:43,340
collection makes it a lot easier for people to use the data in a manner that

178
00:09:43,340 --> 00:09:47,780
allows for predictive analytics. Now, there's some good frameworks out there.

179
00:09:48,720 --> 00:09:50,200
I like the Verizon one because,

180
00:09:50,200 --> 00:09:53,600
like I said, I live in data and it's nice and set up and there's little columns

181
00:09:53,600 --> 00:09:57,440
and everything is nice and neat. Data is not neat, but you can tie and force it into boxes.

182
00:09:58,340 --> 00:10:00,500
There are less Verizon-y ones like CERT,

183
00:10:01,380 --> 00:10:05,780
FAIR, NIST, RMF, Octave, the SANS, 20 critical security control,

184
00:10:05,960 --> 00:10:11,880
Terra and others, or you can make your own that has all of those issues or all of those stipulations.

185
00:10:15,020 --> 00:10:15,820
Data collection.

186
00:10:17,200 --> 00:10:20,000
I'm sure you have incident reports and system logs.

187
00:10:20,200 --> 00:10:21,060
And application logs.

188
00:10:22,340 --> 00:10:24,820
All of these allow for different kinds of analysis.

189
00:10:25,660 --> 00:10:28,060
For instance, incident reports let you do trend analysis.

190
00:10:28,760 --> 00:10:30,640
System logs let you do broad analysis.

191
00:10:31,100 --> 00:10:33,280
And application logs can allow focused analyses.

192
00:10:33,640 --> 00:10:34,860
All of these can fit in.

193
00:10:34,980 --> 00:10:36,300
And it's basically, you know,

194
00:10:36,340 --> 00:10:39,000
the idea behind Facebook or any of the other big ones.

195
00:10:39,120 --> 00:10:40,500
The more data you get, the better it is.

196
00:10:42,060 --> 00:10:45,080
Many tools exist to help you extract it.

197
00:10:45,220 --> 00:10:46,460
Logstash is free and flexible.

198
00:10:47,220 --> 00:10:49,660
You may have Splunk, which I've heard great things about.

199
00:10:49,660 --> 00:10:50,660
But I can't afford.

200
00:10:51,460 --> 00:10:54,280
Or Security Onion, which is kind of the kitchen sink of all operators.

201
00:10:54,660 --> 00:10:56,680
There are other processing tools.

202
00:10:57,380 --> 00:10:59,420
But, you know, you can ask your operations team.

203
00:10:59,640 --> 00:11:00,660
They should know about this.

204
00:11:03,380 --> 00:11:06,680
Now, data mining and predictive analytics are sometimes used interchangeably,

205
00:11:06,720 --> 00:11:08,780
which is wrong, and they should not do that.

206
00:11:08,900 --> 00:11:09,940
But nobody asked the data people.

207
00:11:10,520 --> 00:11:13,360
Data mining produces decisions based on normal reports.

208
00:11:13,980 --> 00:11:17,860
Predictive analytics uses data mining reports to move forward.

209
00:11:18,040 --> 00:11:19,160
So this is kind of how it goes.

210
00:11:19,660 --> 00:11:22,480
You find the project, collect the data, produce data analysis.

211
00:11:22,760 --> 00:11:23,580
There's statistics.

212
00:11:23,840 --> 00:11:27,380
Then you do predictive analytics modeling and you deploy it.

213
00:11:27,940 --> 00:11:30,800
What it doesn't show is there's a nice little feedback loop that goes back

214
00:11:30,800 --> 00:11:34,680
to the definition and data collection, because sometimes when you're doing

215
00:11:35,500 --> 00:11:37,700
this type of data collection and predictive analytics,

216
00:11:37,860 --> 00:11:40,000
you realize that you were looking in the complete wrong area.

217
00:11:40,320 --> 00:11:44,760
Like, you're like, I am so sure that people are coming from outside the network

218
00:11:44,760 --> 00:11:48,520
and attacking, but no, there's something inside the network that is causing all

219
00:11:48,520 --> 00:11:49,020
of these issues.

220
00:11:49,020 --> 00:11:51,260
And you never spotted it before because of all of the noise.

221
00:11:55,100 --> 00:11:58,820
Data mining finds valuable information that is hidden in large volumes of data.

222
00:11:59,660 --> 00:12:03,760
It's the five major elements ETL, storage and manage data,

223
00:12:04,020 --> 00:12:06,640
provide data access, analyze it and present it.

224
00:12:07,720 --> 00:12:09,880
You can use different levels of analysis.

225
00:12:12,920 --> 00:12:16,660
Predictive analytics extracts data from the existing sets.

226
00:12:17,140 --> 00:12:18,520
Basically, predictive analytics

227
00:12:19,020 --> 00:12:20,460
allows you to identify trends.

228
00:12:20,900 --> 00:12:24,620
You can kind of see this in when they're talking about housing markets or rental

229
00:12:24,620 --> 00:12:27,180
or any of those big things, but this actually is more important

230
00:12:27,180 --> 00:12:29,740
for security because you can predict some of these things.

231
00:12:31,160 --> 00:12:34,400
It's not an absolute science, though, so if someone says I'm using predictive

232
00:12:34,400 --> 00:12:37,440
analytics and I can tell you exactly which targets are going to get hit at what

233
00:12:37,440 --> 00:12:39,900
time, they're full of bullshit and don't pay their contractor fee.

234
00:12:43,920 --> 00:12:46,360
This is how predictive analytics should work.

235
00:12:46,360 --> 00:12:50,420
You've got it is a continual iterative cycle.

236
00:12:51,300 --> 00:12:52,420
You have reporting analysis.

237
00:12:52,640 --> 00:12:54,160
What happened? Why it happened?

238
00:12:55,420 --> 00:12:58,800
Monitoring, which a lot of people in security spend a lot of time doing.

239
00:12:58,960 --> 00:12:59,960
What's happening now?

240
00:13:00,180 --> 00:13:03,060
Predictive analytics is what's going to happen in the future.

241
00:13:03,240 --> 00:13:07,080
This allows you to start an action which sometimes change what's going on.

242
00:13:07,220 --> 00:13:11,540
So if you're using predictive analytics in it and find an attack factor or able

243
00:13:11,540 --> 00:13:15,980
to successfully kind of shut down that area, sometimes the

244
00:13:16,360 --> 00:13:19,740
same attackers will try a different area using the same sort of attack vector.

245
00:13:19,880 --> 00:13:21,940
And that can allow you to predict that kind of action.

246
00:13:23,580 --> 00:13:26,320
It's a weird concept, but just think of it as a spiral.

247
00:13:27,620 --> 00:13:32,300
You can't stop doing this once you start, which is kind of a little disheartening.

248
00:13:32,440 --> 00:13:35,460
But at the same time, it does make your security team a little better.

249
00:13:38,480 --> 00:13:41,020
So forecasts, models allow forecasting.

250
00:13:41,160 --> 00:13:43,540
A lot of times it's very much like the weather forecaster.

251
00:13:43,660 --> 00:13:44,640
Sometimes they're spot on.

252
00:13:46,360 --> 00:13:46,980
Sometimes they're off.

253
00:13:47,140 --> 00:13:51,500
And that's because sometimes things change in the model that you can't account for.

254
00:13:51,780 --> 00:13:54,400
Let's say that there was a threat actor and something changed.

255
00:13:54,680 --> 00:13:58,700
Maybe the threat actor that you were looking at was arrested and you didn't know that.

256
00:13:58,720 --> 00:13:59,740
And so you're looking at this area.

257
00:13:59,880 --> 00:14:01,340
But someone else had found out about the

258
00:14:01,340 --> 00:14:03,360
work they did and decided to attack from a different point.

259
00:14:05,820 --> 00:14:09,620
It does allow mineable data for future forecasts as well, because once you pull

260
00:14:09,620 --> 00:14:11,980
the prediction down, you can also pull what actually happened

261
00:14:11,980 --> 00:14:14,960
and compare those and create a model that takes those two into account.

262
00:14:15,340 --> 00:14:16,160
So eventually,

263
00:14:16,360 --> 00:14:20,400
the model does become more accurate, but it's never going to be 100 percent perfect.

264
00:14:23,820 --> 00:14:27,140
So before you begin using any type of predictive analytics and data mining,

265
00:14:27,340 --> 00:14:29,680
you need to select a tool and learn to use it effectively.

266
00:14:29,920 --> 00:14:31,660
And you'll want to use it in that feedback loop.

267
00:14:33,460 --> 00:14:35,280
Sorry, it's really sorry, the microphone.

268
00:14:35,620 --> 00:14:39,260
Anyway, the best approach is to combine both data mining and predictive analytics

269
00:14:39,260 --> 00:14:42,300
tools with a nice, friendly GUI. It really gets messy.

270
00:14:42,800 --> 00:14:44,300
I play with some of this stuff.

271
00:14:44,460 --> 00:14:45,960
It's really, really messy on the back end.

272
00:14:45,960 --> 00:14:50,140
And I know most of you live in, you know, bash or everything else.

273
00:14:50,200 --> 00:14:52,540
Trust me, it gets messy with data because data.

274
00:14:53,420 --> 00:14:56,620
The top tools in use are enterprise tools like IBM and SAS.

275
00:14:57,180 --> 00:14:58,720
Where I work, I'm a SAS shop.

276
00:14:58,900 --> 00:15:00,880
I don't really like it because it's clunky and expensive.

277
00:15:02,260 --> 00:15:08,280
For my free time, I use a lot of R. R is a nice open open source project.

278
00:15:08,460 --> 00:15:10,580
It's free. They do have things you can pay for,

279
00:15:10,660 --> 00:15:12,400
but it's friendly and I like open source.

280
00:15:12,660 --> 00:15:14,940
There's also Weka, KNIME and RapidMiner.

281
00:15:14,940 --> 00:15:16,980
There's also probably other tools that you can use.

282
00:15:17,540 --> 00:15:19,120
I prefer R just because

283
00:15:20,700 --> 00:15:23,240
it's one of the ones I can install at work and work on in my free time.

284
00:15:27,340 --> 00:15:29,420
So this is the blue team or predictive analytics

285
00:15:30,700 --> 00:15:34,240
from earlier. Basically, the blue team actually focuses defense.

286
00:15:34,360 --> 00:15:35,880
So you add an extra step of focusing

287
00:15:35,880 --> 00:15:38,240
defense and data collection and monitoring it.

288
00:15:41,520 --> 00:15:43,120
Now, remember your front end tools,

289
00:15:43,120 --> 00:15:46,900
Kibana, Splunk, Maltego, Clalenter, Graphite, Greylog.

290
00:15:47,080 --> 00:15:48,660
Once you have all the data cleaned up,

291
00:15:48,680 --> 00:15:51,420
you can actually feed it into this and it actually makes it a lot easier.

292
00:15:51,720 --> 00:15:54,260
And it cleans out some of that noise,

293
00:15:54,360 --> 00:15:56,600
that background noise, things that you don't need to worry about

294
00:15:57,560 --> 00:16:00,200
and allows you to focus your data so it's a lot more efficient.

295
00:16:03,520 --> 00:16:06,500
So data is everywhere, it's useful, it's pretty powerful.

296
00:16:08,300 --> 00:16:10,340
Do you have any questions? I'm sorry that went really quickly.

297
00:16:10,420 --> 00:16:11,280
I'm super nervous.

298
00:16:14,740 --> 00:16:15,740
Any questions?

299
00:16:29,700 --> 00:16:34,640
Yeah, top three things not to do when you're migrating from just being a command

300
00:16:34,640 --> 00:16:38,400
line jockey to starting to use the analytic and predictive tools.

301
00:16:38,620 --> 00:16:42,600
Top three. Don't assume that your command line skills

302
00:16:43,120 --> 00:16:45,260
work as well in one of these tools.

303
00:16:45,580 --> 00:16:49,300
So one of the things I like about R, sorry, I'm short and this thing is so tall.

304
00:16:49,480 --> 00:16:49,700
Anyway,

305
00:16:50,800 --> 00:16:54,360
R has a very nice GUI that you can use.

306
00:16:54,440 --> 00:16:55,340
It's called RStudio.

307
00:16:55,480 --> 00:16:58,680
So it has a pseudo command line so you can still use those skills,

308
00:16:59,340 --> 00:17:02,980
but it allows you to kind of convert into R language because it doesn't always work

309
00:17:02,980 --> 00:17:06,120
exactly like it does in Linux command line or anything like that.

310
00:17:06,660 --> 00:17:08,820
Another one is assuming that

311
00:17:11,220 --> 00:17:12,800
these tools make

312
00:17:13,120 --> 00:17:16,980
the same kind of logical sense that you're used to in working in the command line.

313
00:17:17,180 --> 00:17:20,360
They don't always necessarily make the same sense and sometimes you have to do

314
00:17:20,360 --> 00:17:23,520
some fancy acrobatics to get it to see what you wanted to look at.

315
00:17:24,180 --> 00:17:27,240
And the third is to assume that everything you have is valuable data.

316
00:17:27,520 --> 00:17:29,620
It's all valuable in the sense that it's data.

317
00:17:30,220 --> 00:17:32,360
But trying to get it all to process

318
00:17:32,360 --> 00:17:37,240
through without doing pre-cleaning is a huge mistake because it'll end up

319
00:17:37,240 --> 00:17:42,040
bogging down your time and preventing you from focusing on what's important.

320
00:17:49,480 --> 00:17:49,880
So I think that's a really cool thing.

321
00:17:49,880 --> 00:17:52,840
Hi, approximately how many analysts?

322
00:17:53,360 --> 00:17:55,360
What what is the ratio that you would

323
00:17:55,780 --> 00:18:01,060
suggest for number of people looking at the data to the quantity of the data?

324
00:18:02,060 --> 00:18:03,200
That really depends.

325
00:18:03,360 --> 00:18:04,460
So if you have someone who's very

326
00:18:04,460 --> 00:18:08,160
comfortable with the data, you can usually get away with one or two analysts.

327
00:18:08,560 --> 00:18:11,260
As long as you have a lot of data, like a lot, a lot.

328
00:18:11,360 --> 00:18:12,140
You could use.

329
00:18:12,240 --> 00:18:12,640
Honestly,

330
00:18:13,120 --> 00:18:19,760
as long as you have support from the team and you have a date like a DBR in place.

331
00:18:20,440 --> 00:18:24,380
So I would suggest the DBR and then at least one to two analysts.

332
00:18:24,520 --> 00:18:25,660
You have a lot, maybe three.

333
00:18:26,740 --> 00:18:29,080
But honestly, you have to make sure

334
00:18:29,080 --> 00:18:31,880
that the analysts get along and work in the same kind of flow.

335
00:18:32,140 --> 00:18:38,120
This sounds kind of weird and stuff, but if you work in I work with a group

336
00:18:38,120 --> 00:18:40,500
of five analysts, there's five of us, two of them.

337
00:18:40,560 --> 00:18:42,120
I won't work on any of their code projects

338
00:18:42,120 --> 00:18:47,320
because I'm pretty sure there is crack smoking going on while they're working.

339
00:18:47,440 --> 00:18:51,740
These projects is I don't understand their code and it's supposed to be code that I

340
00:18:51,740 --> 00:18:54,700
can read, so I work with the others, but as long as you have people who can work

341
00:18:54,700 --> 00:18:58,620
in conjunction and can work with the DBR, you don't need as many analysts as you

342
00:18:58,620 --> 00:19:03,900
think a good data analyst will be able to pull out important things.

343
00:19:04,000 --> 00:19:10,220
And so here's this is kind of a good analyst can look at data and see trends

344
00:19:10,220 --> 00:19:11,780
before they start pulling.

345
00:19:12,400 --> 00:19:18,760
So in the same way that they still use humans to look at child porn in order

346
00:19:18,760 --> 00:19:22,760
to recognize these scenes, the human eye recognizes it better than any computer

347
00:19:22,760 --> 00:19:25,920
model, a good analyst will recognize that there's a pattern in something,

348
00:19:26,100 --> 00:19:27,740
pull it out and get to work on it.

349
00:19:27,860 --> 00:19:29,560
So you need someone who really,

350
00:19:30,160 --> 00:19:33,800
really likes data and not someone who just wants a good paycheck.

351
00:19:37,000 --> 00:19:39,140
And just a quick follow up question.

352
00:19:42,120 --> 00:19:50,800
Is what do you guys usually use for data storage in terms of how much data you have?

353
00:19:52,240 --> 00:19:55,700
And a follow up to that, to that, this is my last one.

354
00:19:56,480 --> 00:19:58,420
What time window do you usually look at?

355
00:19:58,660 --> 00:20:00,340
We have a seven year time window on data.

356
00:20:01,220 --> 00:20:06,960
So I work for a college and so we have a seven year time window and we are talking,

357
00:20:07,120 --> 00:20:08,180
I think we have

358
00:20:10,020 --> 00:20:11,680
a for our particular

359
00:20:12,120 --> 00:20:13,300
storage for the data that we work with.

360
00:20:13,380 --> 00:20:14,480
I think we have 2000 terabytes.

361
00:20:16,440 --> 00:20:16,880
And

362
00:20:19,960 --> 00:20:23,980
it's a lot of students, every single data piece related to every single student back

363
00:20:23,980 --> 00:20:28,860
to the start of time, so some data we do keep from, you know, from 40 years ago

364
00:20:29,540 --> 00:20:33,160
that's been digitized and put up, but we do keep seven years.

365
00:20:33,300 --> 00:20:37,220
I would recommend for like doing this kind of analytics, keep a report,

366
00:20:37,720 --> 00:20:40,040
but keep your data for a year before you dump it.

367
00:20:40,420 --> 00:20:41,960
But you need to make sure that you

368
00:20:42,120 --> 00:20:45,760
have an adequate, like a summary report of the data that was dumped so people can

369
00:20:45,760 --> 00:20:49,520
use that model to continue working. You don't necessarily need that.

370
00:20:49,600 --> 00:20:52,600
We have some federal rules that are like you must keep data for X number of years.

371
00:20:55,180 --> 00:20:56,340
Any other questions?

372
00:21:00,900 --> 00:21:04,400
I had a follow up question related to how many analysts.

373
00:21:04,760 --> 00:21:11,440
Is it a thing where you can kind of put in as much as you want and get out as much as you put

374
00:21:11,440 --> 00:21:20,220
in? Or is there like a minimum amount of eyes that you need to have on something to get any meaningful use out of analysis?

375
00:21:20,540 --> 00:21:23,460
If you have a good analyst, you can get away with one.

376
00:21:23,620 --> 00:21:25,600
But to be safe, you want probably two.

377
00:21:25,900 --> 00:21:27,140
But a good analyst.

378
00:21:27,320 --> 00:21:27,380
The bus problem?

379
00:21:29,440 --> 00:21:31,320
Yeah. And a good.

380
00:21:31,440 --> 00:21:36,400
So in my case, so on the stuff that I work on, we do a lot of predictive modeling and stuff.

381
00:21:36,540 --> 00:21:41,420
I always keep something on the server with all of my latest code, because, you know,

382
00:21:41,440 --> 00:21:44,020
if something happens to me, someone needs to be able to come in and pick it up.

383
00:21:44,280 --> 00:21:45,860
Not all analysts work that way.

384
00:21:46,000 --> 00:21:48,840
We have that's as I said, part of our team kind of works fast and loose.

385
00:21:49,580 --> 00:21:52,880
And I'm sure they're coding in Klingon or something because I have no idea what

386
00:21:52,880 --> 00:21:56,200
this is because I'm looking through it and I should recognize the code and it has nothing.

387
00:21:56,300 --> 00:21:58,780
And when I try to run it, SAS is like, I don't know.

388
00:21:59,020 --> 00:22:00,320
Peace out. We're not doing this.

389
00:22:01,400 --> 00:22:04,140
But really, a good analyst who is passionate

390
00:22:04,140 --> 00:22:09,360
about data is going to be able to give you a lot more than a team of 20 analysts who don't really care.

391
00:22:11,780 --> 00:22:14,080
Yeah. You want people who care about the data.

392
00:22:14,220 --> 00:22:15,940
And I know that sounds weird, but you have people like me.

393
00:22:15,980 --> 00:22:18,980
That's like, oh, I get to play with data and it's a little odd,

394
00:22:19,120 --> 00:22:21,940
but it's the same way people like playing with bugs or anything else like that.

395
00:22:25,200 --> 00:22:26,260
Any other questions?

396
00:22:30,340 --> 00:22:32,700
Well, I know I talked really fast, I was super nervous.

397
00:22:32,740 --> 00:22:33,060
I'm sorry.

398
00:22:37,280 --> 00:22:38,240
But thank you.

399
00:22:41,440 --> 00:22:42,400
Thank you very much.