1
00:00:00,000 --> 00:00:05,540
Damon, I don't know, you've spoken at Source Boston, you've spoken at DevCon.

2
00:00:05,600 --> 00:00:08,480
Didn't you give a one-on-one in DevCon two years ago?

3
00:00:09,480 --> 00:00:13,600
I think that's how I first was introduced to you.

4
00:00:13,860 --> 00:00:19,500
Well, okay, without much ado, I'm just going to go to the next slide.

5
00:00:19,760 --> 00:00:21,780
My pleasure to introduce to you Damon Small.

6
00:00:23,020 --> 00:00:23,900
Thank you, Ming.

7
00:00:25,600 --> 00:00:26,340
All right.

8
00:00:27,080 --> 00:00:34,620
So, as you can see, I did not go through my normal pre-presentation routine that I normally go through,

9
00:00:34,700 --> 00:00:35,220
but that's fine.

10
00:00:35,420 --> 00:00:37,680
This will make this much more interesting, I think.

11
00:00:38,340 --> 00:00:40,220
So, yeah, my name is Damon Small.

12
00:00:40,400 --> 00:00:41,880
Some folks know me as Chef.

13
00:00:42,540 --> 00:00:43,960
Thanks for coming out to see me.

14
00:00:44,120 --> 00:00:50,500
And the title of my slide is Layer 8 and Why People are the Most Important Security Tool.

15
00:00:50,820 --> 00:00:55,120
And I'm realizing, as usual, I'm a little short for that podium,

16
00:00:55,300 --> 00:00:56,320
so I'll wander around.

17
00:00:56,340 --> 00:00:58,180
Sorry in advance to the cameraman.

18
00:00:59,920 --> 00:01:03,840
So, first of all, I want to say special thanks to, well, the Wall of Sheep,

19
00:01:03,920 --> 00:01:05,700
who sponsors the Packet Hacking Village.

20
00:01:06,140 --> 00:01:10,100
It's been a privilege of mine to be a volunteer staffer for the last three years.

21
00:01:10,320 --> 00:01:12,500
I love DevCon, and I love this venue.

22
00:01:13,120 --> 00:01:14,040
It's a lot of fun.

23
00:01:14,440 --> 00:01:15,860
A lot of work, but a lot of fun.

24
00:01:15,960 --> 00:01:19,060
Of course, my employer, who is helping me be out here,

25
00:01:19,480 --> 00:01:24,140
so shout out to my boss and the rest of the NCC folks that I work with,

26
00:01:24,240 --> 00:01:26,020
and my wife and editor.

27
00:01:26,080 --> 00:01:26,320
My wife and editor.

28
00:01:26,340 --> 00:01:29,220
My wife gets to go through all these slide decks that I do,

29
00:01:29,360 --> 00:01:32,820
and she takes out the mistakes and makes them suck a little bit less.

30
00:01:33,440 --> 00:01:35,600
So, thanks, hon, for reading through it.

31
00:01:37,280 --> 00:01:38,680
Last but not least, you.

32
00:01:39,500 --> 00:01:43,540
Every time I get accepted to present somewhere, after the excitement wears off,

33
00:01:43,620 --> 00:01:46,660
you then wonder, it's like, oh, God, is anyone going to show up?

34
00:01:47,500 --> 00:01:49,180
Especially, you know, it's in the afternoon.

35
00:01:49,380 --> 00:01:50,640
It's been a long day.

36
00:01:50,760 --> 00:01:53,560
So, people have shown up, so thank you very much to all of you.

37
00:01:56,340 --> 00:02:01,720
Thank you for sitting through this, and I mentioned that my employer was helping me be out here,

38
00:02:01,800 --> 00:02:03,140
which I'm very, very grateful for.

39
00:02:03,320 --> 00:02:05,120
So, you'll have to sit through a short commercial.

40
00:02:05,760 --> 00:02:07,400
I work for the NCC group.

41
00:02:07,880 --> 00:02:11,940
We're a large global cybersecurity company, 35 offices.

42
00:02:12,880 --> 00:02:15,120
That number in the lower right is incorrect.

43
00:02:15,340 --> 00:02:18,140
We released our annual financials a couple weeks ago,

44
00:02:18,300 --> 00:02:21,780
and we actually did 244 million pounds in revenue last year.

45
00:02:21,880 --> 00:02:23,060
So, it was a good year.

46
00:02:24,220 --> 00:02:25,520
We have a lot of services.

47
00:02:25,720 --> 00:02:26,320
We do a lot of work.

48
00:02:26,320 --> 00:02:26,660
We do a lot of things.

49
00:02:26,840 --> 00:02:31,240
We've got 2,000 geeks globally, and a lot of offices.

50
00:02:33,300 --> 00:02:34,540
And enough about them.

51
00:02:34,860 --> 00:02:35,800
Let's talk about me.

52
00:02:36,800 --> 00:02:37,740
My name is Damon.

53
00:02:37,840 --> 00:02:40,780
Like I said, tech director for NCC group in North America.

54
00:02:40,940 --> 00:02:45,880
I've been in IT since 1995, and I started focusing on InfoSec in 2001.

55
00:02:46,280 --> 00:02:48,780
The dot-com bubble had burst around 2000,

56
00:02:49,300 --> 00:02:52,420
so that was a good time to kind of change my career a little bit,

57
00:02:52,580 --> 00:02:56,080
and I think I got lucky and thought,

58
00:02:56,320 --> 00:02:59,540
back then, that, well, security is probably going to be a problem for a while,

59
00:02:59,680 --> 00:03:02,560
so that's a good area to go into, and I think I was right.

60
00:03:03,360 --> 00:03:04,900
Louisiana native, go Tigers.

61
00:03:05,760 --> 00:03:07,040
Any LSU fans here?

62
00:03:07,800 --> 00:03:08,540
Oh, really?

63
00:03:08,740 --> 00:03:09,200
Right on.

64
00:03:09,320 --> 00:03:09,920
Okay, cool.

65
00:03:10,200 --> 00:03:12,080
That was unexpected.

66
00:03:13,000 --> 00:03:15,040
I studied music at LSU.

67
00:03:15,080 --> 00:03:16,240
I was in Tiger Band and all that.

68
00:03:16,380 --> 00:03:17,620
I went to grad school in 2005.

69
00:03:18,120 --> 00:03:20,580
I spent most of my career in Blue Team,

70
00:03:21,260 --> 00:03:26,020
and I've worked in the healthcare, DOD, aerospace, and oil and gas industries,

71
00:03:26,320 --> 00:03:26,580
as far as I know.

72
00:03:26,960 --> 00:03:29,220
There's a nice picture of me, which reminds me of a joke.

73
00:03:29,280 --> 00:03:31,000
How do you know when someone is a DEFCON speaker?

74
00:03:32,460 --> 00:03:33,240
They'll tell you.

75
00:03:35,720 --> 00:03:36,220
There you go.

76
00:03:37,200 --> 00:03:40,940
shout out to my buddy Dennis who's spoken on the big stage three times

77
00:03:42,180 --> 00:03:45,260
alright so let's get into it how did this slide deck

78
00:03:45,720 --> 00:03:49,780
come to be so I'll tell you a little story about that I'm a bit of a digital

79
00:03:49,780 --> 00:03:53,920
pack rat and I ran across a white paper that I had written about a decade

80
00:03:53,920 --> 00:03:57,700
ago I believe it was and I found this old white paper and I thought

81
00:03:57,700 --> 00:04:01,800
well this is going to be comical because this is old material and it's probably not aged very

82
00:04:01,800 --> 00:04:05,780
well and it's not going to make any sense and I read the white paper and I was shocked to discover

83
00:04:05,780 --> 00:04:08,420
that it was actually still quite relevant

84
00:04:09,800 --> 00:04:13,740
particularly in the context of the wall of sheep because if you've been over across

85
00:04:13,740 --> 00:04:17,720
the hall and you've seen what we do with packet detective and capture

86
00:04:17,720 --> 00:04:21,440
the packet and some of the training we do we're very much about

87
00:04:21,440 --> 00:04:25,500
you know basic skills that are important for everybody

88
00:04:25,500 --> 00:04:29,880
in infosec regardless of what your role is and especially since our capture

89
00:04:29,880 --> 00:04:33,740
the packet contest is you know it's a CTF basically but you're

90
00:04:33,740 --> 00:04:35,280
finding telemetry on a network

91
00:04:35,780 --> 00:04:39,880
as it's flying across so as we get into this I think you'll see why I thought hey

92
00:04:39,880 --> 00:04:41,880
the wall of sheep is the perfect venue for this

93
00:04:44,580 --> 00:04:49,480
so yeah the techniques are simple and old but remain very useful I like telling

94
00:04:49,480 --> 00:04:52,620
stories so I thought this was an interesting narrative and

95
00:04:53,640 --> 00:04:57,600
WannaCry had just happened when I started putting this slide deck together and a lot of the

96
00:04:57,600 --> 00:05:03,580
techniques I'll talk about from the incident from a decade ago are still quite relevant

97
00:05:04,380 --> 00:05:05,240
so quick

98
00:05:05,240 --> 00:05:11,220
poll of the audience how many folks in here on some regular basis will use tools like

99
00:05:11,220 --> 00:05:17,420
TCP dump or win dump to examine network traffic okay a few hands

100
00:05:18,080 --> 00:05:22,000
what about something like some of the PS tools stuff like

101
00:05:24,340 --> 00:05:30,080
PS exact or things like that yeah some other hands I mean so those are tools that have been around for a while right

102
00:05:30,080 --> 00:05:35,020
they're not new but we'll see how I think I think they'll be interesting

103
00:05:35,240 --> 00:05:43,620
as we get into this so to start off I mean if you work in any kind of IT department you know you've got you

104
00:05:43,620 --> 00:05:49,260
know people time and money are limited and you face a myriad of decisions every day to try and

105
00:05:49,260 --> 00:05:54,880
figure out what tools what technical controls are you going to use you've got to make purchasing

106
00:05:54,880 --> 00:06:00,220
decisions to try and figure out you know which are the right widgets for your environment and

107
00:06:00,220 --> 00:06:04,660
you have to remain fiscally responsible though right so you can't you know you can't buy everything

108
00:06:05,900 --> 00:06:12,540
But I wonder sometimes if in business, if we spend too much time worrying about which widget to get

109
00:06:12,540 --> 00:06:20,060
and we don't spend enough time looking at the human beings that we have in our IT teams and our InfoSec teams.

110
00:06:20,380 --> 00:06:28,000
So to start off, the first case study I'll go through, the name of the white paper was actually called Finding the Needle.

111
00:06:28,000 --> 00:06:37,120
And so to set it up, it was December 2004, and a primary firewall of a health care provider failed.

112
00:06:37,340 --> 00:06:39,320
And it failed over to the backup firewall.

113
00:06:39,640 --> 00:06:44,380
And, of course, the networking team was keenly interested in trying to figure out why the failure had occurred.

114
00:06:44,500 --> 00:06:48,240
And they noticed a large volume of traffic on port 135.

115
00:06:49,000 --> 00:06:54,700
And there was also a periodic burst of Internet-bound traffic on various ephemeral ports.

116
00:06:54,920 --> 00:06:57,680
So there was a lot of traffic on the perimeter.

117
00:06:57,700 --> 00:06:57,980
On multiple.

118
00:06:58,000 --> 00:07:02,920
On 135 and lots of other high TCP port traffic.

119
00:07:03,140 --> 00:07:07,740
So they were, you know, trying to figure out, okay, based on that information, what is going on?

120
00:07:09,480 --> 00:07:15,540
Of course, you know, a lot of folks immediately suspected something malicious, some malicious software.

121
00:07:15,700 --> 00:07:19,180
But we had several technological limitations to deal with.

122
00:07:19,300 --> 00:07:25,300
Namely, some of the, and I heard the previous speaker was talking about spoofed IP addresses and things.

123
00:07:25,400 --> 00:07:27,600
So it's kind of funny that that came up.

124
00:07:27,600 --> 00:07:30,340
But some of the source IP addresses were spoofed.

125
00:07:30,340 --> 00:07:31,980
So, obviously, they weren't valid packets.

126
00:07:32,120 --> 00:07:34,340
So we couldn't figure out, you know, from where they were coming.

127
00:07:35,400 --> 00:07:41,400
This health care provider also had a lot of third-party supported systems that they themselves didn't have access to.

128
00:07:41,560 --> 00:07:43,540
So they were, you know, tenants on their network.

129
00:07:43,640 --> 00:07:45,080
But they didn't have control over them.

130
00:07:46,000 --> 00:07:49,480
At the time, this shop had something like 8,000 workstations.

131
00:07:49,500 --> 00:07:55,160
But, you know, health care environments are very difficult to manage.

132
00:07:55,360 --> 00:07:57,580
And, you know, antivirus wasn't necessary.

133
00:07:57,580 --> 00:08:00,320
It wasn't necessarily fully functional on all of those machines.

134
00:08:00,540 --> 00:08:02,360
And, again, many of those were third-party supported.

135
00:08:03,320 --> 00:08:06,380
Some of this should sound familiar to my two buddies down there.

136
00:08:06,460 --> 00:08:06,940
Yeah, it doesn't?

137
00:08:07,020 --> 00:08:07,140
Okay.

138
00:08:09,100 --> 00:08:12,160
So, and lastly, the networking team at the time.

139
00:08:12,260 --> 00:08:14,620
And, again, this was, what year did I say this was?

140
00:08:14,740 --> 00:08:15,800
This was a long time ago.

141
00:08:16,040 --> 00:08:17,860
They had protocol analyzers.

142
00:08:17,980 --> 00:08:22,600
But they didn't, the protocol analyzers were really good at giving you trends over time.

143
00:08:22,820 --> 00:08:26,520
But they weren't really good at giving you visibility into real-time traffic.

144
00:08:27,580 --> 00:08:28,020
On the network.

145
00:08:28,240 --> 00:08:31,780
So they were having a hard time sorting out what was going on.

146
00:08:31,900 --> 00:08:36,440
So due to these limitations, they couldn't really troubleshoot any further than just saying,

147
00:08:36,560 --> 00:08:39,180
yep, something had gone horribly wrong.

148
00:08:39,400 --> 00:08:44,280
So they, in that last bullet point there, they begrudgingly engaged the InfoSec team.

149
00:08:44,480 --> 00:08:45,860
And I say begrudgingly.

150
00:08:48,240 --> 00:08:56,540
At the time, network teams weren't really keen.

151
00:08:57,580 --> 00:09:01,480
To let anybody else look at their infrastructure or work on it.

152
00:09:01,680 --> 00:09:10,140
And the InfoSec team, you know, back a decade or more ago, was viewed as the guys that managed, you know, access and identity.

153
00:09:10,500 --> 00:09:11,640
It was really it.

154
00:09:11,960 --> 00:09:14,760
And we had told them, it was like, look, we've got some pretty cool tools.

155
00:09:14,840 --> 00:09:15,820
We could help you out.

156
00:09:15,940 --> 00:09:18,160
But, you know, they're like, no, no, no, you're not a network person.

157
00:09:18,260 --> 00:09:19,120
You can't get on there.

158
00:09:19,200 --> 00:09:25,760
So after they figured out they were kind of stuck, they engaged us and said, okay, will you help us out?

159
00:09:26,020 --> 00:09:27,340
And we were happy to.

160
00:09:27,580 --> 00:09:29,680
So what we needed to know was this.

161
00:09:30,320 --> 00:09:36,180
From where the traffic came to where it was going, what was causing it and how to get rid of it.

162
00:09:36,400 --> 00:09:38,520
Those are very basic questions, right?

163
00:09:38,620 --> 00:09:41,080
I mean, that shouldn't be a huge problem.

164
00:09:41,840 --> 00:09:44,160
But we couldn't answer those four questions.

165
00:09:44,240 --> 00:09:56,160
So I'm going to step through how the creativity of the human beings involved in this using very basic tools and techniques were able to answer those questions where the network team was unable to.

166
00:09:56,620 --> 00:09:57,560
So the first thing.

167
00:09:57,580 --> 00:09:59,080
We had some detective methods.

168
00:10:00,840 --> 00:10:03,520
We had commercial antivirus, of course, on the network.

169
00:10:03,660 --> 00:10:07,400
But I've already mentioned that it wasn't fully functional on some workstations.

170
00:10:07,400 --> 00:10:11,800
So we couldn't depend on it for 100% visibility into our entire environment.

171
00:10:12,540 --> 00:10:15,700
We had an IPS and it was giving us some information.

172
00:10:15,960 --> 00:10:17,820
I'll touch on that a little more.

173
00:10:18,100 --> 00:10:27,460
But interestingly, the most useful thing we had was wind dump, which, of course, is the Windows load of TCP dump that we use on Unix.

174
00:10:27,580 --> 00:10:28,460
All the time in Linux.

175
00:10:28,780 --> 00:10:36,740
So all we did was we got access to a span port near the perimeter so we knew we would see all the traffic.

176
00:10:37,620 --> 00:10:41,780
And we just simply ran wind dump looking for destination port 135.

177
00:10:43,260 --> 00:10:44,060
That was it.

178
00:10:44,140 --> 00:10:45,100
So simple.

179
00:10:45,600 --> 00:10:49,700
But it was a capability that we didn't have in other parts of the organization.

180
00:10:50,040 --> 00:10:56,820
So from that, we were able to figure out which hosts were generating all that traffic.

181
00:10:57,580 --> 00:11:02,760
And we were able to navigate around port 135, which as it turns out was the malicious software trying to propagate to other hosts.

182
00:11:02,860 --> 00:11:16,380
So now that we started gathering a list of who the bad guys were, who the bad hosts were, we were able to target individual machines that were infected and try and clean the mess up.

183
00:11:16,540 --> 00:11:21,680
We found various self-propagating worms that took advantage of the RBCD COM buffer overflow.

184
00:11:22,220 --> 00:11:25,400
And there was also an LSAS buffer overflow at the time.

185
00:11:25,480 --> 00:11:27,560
So there were like two different platforms.

186
00:11:27,560 --> 00:11:30,720
self-propagating softwares, exploiting two different

187
00:11:30,720 --> 00:11:35,240
vulnerabilities on our network. Now I'll pause just to say that

188
00:11:35,240 --> 00:11:39,860
that is a bad day. You know, one piece of malicious software is

189
00:11:39,860 --> 00:11:45,440
no good and two is twice as no good. Looked like Blaster and

190
00:11:45,440 --> 00:11:49,460
Sasser and patches existed but recall that, you know, this was

191
00:11:49,460 --> 00:11:56,580
2004. It's hard to get patches installed on a large environment

192
00:11:56,580 --> 00:12:01,560
today. It was, you know, even harder back then. And in 2004,

193
00:12:01,760 --> 00:12:04,180
we were still going through the narrative of trying to decide,

194
00:12:04,680 --> 00:12:08,140
oh, you know, we patch every quarter, you know, whether we

195
00:12:08,140 --> 00:12:13,380
need to or not. That's enough, right? And even now we still

196
00:12:13,380 --> 00:12:15,820
struggle with trying to figure out, you know, well, I mean, it's

197
00:12:15,820 --> 00:12:17,440
got to be monthly, right? Because that's when patch

198
00:12:17,440 --> 00:12:21,080
Tuesday is. So, yeah, so we weren't, the point is we were

199
00:12:21,080 --> 00:12:25,860
not patched at all. So just, I'm sure most of us, if not all

200
00:12:25,860 --> 00:12:26,560
of us in here, are familiar with patching. But, you know,

201
00:12:26,580 --> 00:12:29,820
we're familiar with botnet worms. But just to give a little

202
00:12:29,820 --> 00:12:33,560
context before I move forward here, I'll say that, you know, a

203
00:12:33,560 --> 00:12:36,900
lot of times they'll do command and control through, in these

204
00:12:36,900 --> 00:12:39,500
cases, these specific examples, they were using internet relay

205
00:12:39,500 --> 00:12:42,920
chat. I think there was a period of time where the only people

206
00:12:42,920 --> 00:12:47,640
that ever used internet relay chat was malicious software. So

207
00:12:47,640 --> 00:12:50,540
the worm logs into an IRC channel and receives instructions

208
00:12:50,540 --> 00:12:54,500
from the bad guy. And then the bad guy can use these zombie

209
00:12:54,500 --> 00:12:56,420
computers for other bad things, right?

210
00:12:56,580 --> 00:13:00,680
I mean, I'm pretty sure we're all familiar with this. But

211
00:13:00,680 --> 00:13:04,720
interestingly, the traffic from the worm propagating and also

212
00:13:04,720 --> 00:13:08,220
logging into the IRC channel caused so much traffic that it

213
00:13:08,220 --> 00:13:11,680
crippled our network. So we had some information. We knew where

214
00:13:11,680 --> 00:13:15,060
the bad traffic was coming from. But we still needed to gather

215
00:13:15,060 --> 00:13:23,160
more data. So, like I said, we had problems with antivirus not

216
00:13:23,160 --> 00:13:26,560
giving us complete data. It really wasn't helping us. You

217
00:13:27,080 --> 00:13:31,660
know, we enabled the IRC log-in event rules on the IPS. You may

218
00:13:31,660 --> 00:13:34,600
be wondering why weren't they enabled in the first place. And

219
00:13:34,600 --> 00:13:39,220
you know, who knows? Tuning devices is hard. And by turning

220
00:13:39,220 --> 00:13:42,620
on the IRC log-in event rules on the IPS, we found three

221
00:13:42,620 --> 00:13:45,540
different IRC channels that the malicious softwares were using.

222
00:13:45,680 --> 00:13:48,440
Of course, I've obfuscated the IP addresses, but, you know,

223
00:13:48,440 --> 00:13:52,240
there they are, the three different ones that existed. But

224
00:13:52,240 --> 00:13:56,000
because we were blocking the IRC log-in events, we actually

225
00:13:56,000 --> 00:13:59,680
created another problem for ourselves, which was the IPS

226
00:14:00,360 --> 00:14:05,300
existed on the external interface of the firewall such

227
00:14:05,300 --> 00:14:09,440
that the firewall had translated the addresses to a routable IP

228
00:14:09,440 --> 00:14:13,020
address before the IPS saw it. So at the point at which the IPS

229
00:14:13,730 --> 00:14:18,080
blocked the traffic, all I saw was the NAT address. And to

230
00:14:18,080 --> 00:14:22,720
figure out what the RFC 1918 address was, we had to go through

231
00:14:22,720 --> 00:14:24,920
the firewall logs and look at the NAT tables, and that was a

232
00:14:24,920 --> 00:14:25,980
huge pain. Very time-consuming. We had to go through the firewall logs and look at the NAT tables, and that was a huge pain. Very time-consuming. So, you know,

233
00:14:25,980 --> 00:14:33,060
consuming. And it was then that we realized that note to self

234
00:14:33,060 --> 00:14:36,660
you should probably put the IPS on the internal interface of the

235
00:14:36,660 --> 00:14:41,240
firewall or otherwise positioned such that you can see the 1918

236
00:14:41,240 --> 00:14:44,720
addresses before they're translated. Again it was 2004.

237
00:14:45,400 --> 00:14:49,240
IPSs were new and exciting back then so we were still sorting a

238
00:14:49,240 --> 00:14:54,080
lot of things out. So what did we do? Wind dump to the rescue

239
00:14:54,080 --> 00:14:59,040
again. So we already knew what those three IRC channels were so

240
00:14:59,040 --> 00:15:03,460
it was trivial to as you see there to set up a string that

241
00:15:03,460 --> 00:15:08,940
would look for any traffic going to any three of those hosts. And

242
00:15:08,940 --> 00:15:12,480
we quickly so we started seeing all kinds of traffic and it was

243
00:15:12,480 --> 00:15:15,480
like great so we're seeing more traffic but if you've ever done

244
00:15:15,480 --> 00:15:19,780
wind dump you know stuff flies by and yes you can write it to

245
00:15:19,780 --> 00:15:23,940
disk and grep it and those sorts of things. But we still

246
00:15:23,940 --> 00:15:27,060
weren't having very much luck on quickly figuring out where the

247
00:15:27,060 --> 00:15:30,680
infected hosts were. So here's another tool. Anybody use snort

248
00:15:30,680 --> 00:15:34,520
or something similar? I'm sure you do. Yep. So it's really easy

249
00:15:34,520 --> 00:15:37,480
you can see the syntax there. I mean it's so easy I could write

250
00:15:37,480 --> 00:15:41,160
that. And all we did is say that look we set up a variable

251
00:15:41,160 --> 00:15:45,300
that included the three IP addresses of the botnets we are

252
00:15:45,300 --> 00:15:49,100
interested in. And we told snort to just alert us any time it saw

253
00:15:49,100 --> 00:15:53,180
traffic going to there. And it was a lot easier to generate a

254
00:15:53,180 --> 00:15:53,920
list of infected hosts. So we set up a variable that included the

255
00:15:53,940 --> 00:15:57,180
infected host using this technique rather than going

256
00:15:57,180 --> 00:16:00,460
through a bunch of you know wind dump logs. So interestingly as

257
00:16:00,460 --> 00:16:03,960
we were going through the logs of the alerts we would capture

258
00:16:05,180 --> 00:16:10,380
packets. Again shout out to wall of sheep. And we started

259
00:16:10,380 --> 00:16:14,640
seeing all sorts of interesting things. Which is the militia

260
00:16:14,640 --> 00:16:20,880
software actually logging in to the IRC channel. And getting

261
00:16:20,880 --> 00:16:23,920
ready to receive commands from the back end of the network. And

262
00:16:23,920 --> 00:16:24,320
we can see the data. We can see the data that we have. And we

263
00:16:24,320 --> 00:16:25,100
can see the data that we have. So I'm not going to walk through

264
00:16:25,100 --> 00:16:28,240
this. But this is one of those artifacts that during an incident

265
00:16:28,240 --> 00:16:31,120
response or during an investigation you will stumble

266
00:16:31,120 --> 00:16:35,060
across data like this and I don't know. I thought it was

267
00:16:35,060 --> 00:16:37,460
kind of cool. And that was one of those moments when I was

268
00:16:37,460 --> 00:16:40,280
reading it you know more than ten years after I had written it.

269
00:16:40,340 --> 00:16:46,480
I was like huh. That's interesting. We can also see

270
00:16:46,480 --> 00:16:53,900
yeah so the militia software continues to get logged in. And

271
00:16:53,920 --> 00:16:59,640
there are on the second line there are 4,253 invisible users.

272
00:16:59,880 --> 00:17:03,680
So I mean I reckon that means that's how many other devices

273
00:17:03,680 --> 00:17:07,000
had logged in. So I mean by today's standards that's not a

274
00:17:07,000 --> 00:17:10,540
large bot net. But it was still you know pretty interesting to

275
00:17:10,540 --> 00:17:18,440
see that. And here towards the end of the conversation it

276
00:17:19,040 --> 00:17:23,580
appears that the bad guy sent a command to our host to begin

277
00:17:23,920 --> 00:17:28,840
scanning. That 10.32 network was part of our network. And it

278
00:17:28,840 --> 00:17:32,920
was scanning looking for port 445 being open presumably to

279
00:17:32,920 --> 00:17:36,660
find other hosts that were potentially vulnerable. In this

280
00:17:36,660 --> 00:17:39,320
case I believe that would have been the LSAS buffer overflow.

281
00:17:40,280 --> 00:17:44,260
So we actually saw as we were looking at this traffic you know

282
00:17:44,260 --> 00:17:48,500
the actual evidence of the malicious activity going on. And

283
00:17:48,500 --> 00:17:53,740
of course for you know a bad guy you know a malicious user to be

284
00:17:53,920 --> 00:17:56,900
successful at what he's doing he's got to you know find other

285
00:17:56,900 --> 00:18:01,280
hosts and exploit them. So we see the scan starting here. And

286
00:18:01,280 --> 00:18:06,440
then we see where the software actually found other malicious

287
00:18:06,440 --> 00:18:11,640
hosts. And there they are. So that was useful because you know

288
00:18:11,640 --> 00:18:14,940
while the exploit was still taking place we already knew who

289
00:18:14,940 --> 00:18:22,100
those hosts were. And then we see the bad guy continuing on to

290
00:18:23,920 --> 00:18:26,200
take over. It looked like this might have been a SYN flood

291
00:18:26,200 --> 00:18:30,560
against port 6667 and other ports. So maybe this is somebody

292
00:18:30,560 --> 00:18:34,960
that you know the malicious actor was interested in taking

293
00:18:34,960 --> 00:18:38,140
down or something. But you know and meanwhile you know our

294
00:18:38,140 --> 00:18:42,180
firewall is falling over. And now you know we got computers on

295
00:18:42,180 --> 00:18:48,140
our network launching SYN floods against other people. So that

296
00:18:48,140 --> 00:18:51,800
sucked. But so we figured out a bunch of things using very very

297
00:18:51,800 --> 00:18:53,900
simple tools. We figured out a lot of things. We figured out a

298
00:18:53,920 --> 00:18:56,560
lot of information that other you know as I said other parts of

299
00:18:56,560 --> 00:19:00,500
our organization were not able to figure out. So now that we

300
00:19:00,500 --> 00:19:02,860
knew all this stuff now it's time to kind of clean up the

301
00:19:02,860 --> 00:19:09,180
mess. And try and get rid of it. So again antivirus was

302
00:19:09,180 --> 00:19:14,240
largely nonfunctional. We had third party devices. Oh and it

303
00:19:14,240 --> 00:19:19,980
being a hospital system we had biomedical devices. Now if we

304
00:19:19,980 --> 00:19:23,720
had more time this is where I'd go on my rant about FDA 510

305
00:19:23,720 --> 00:19:23,900
cases. I'd go on my rant about FDA 510 cases. I'd go on my rant

306
00:19:23,920 --> 00:19:24,280
about FDA 510 cases. I'd go on my rant about FDA 510 cases. I's

307
00:19:24,280 --> 00:19:25,840
more aboutress on healthcare and why biomedical device

308
00:19:25,840 --> 00:19:28,800
manufacturers don't want to patch their device because

309
00:19:28,800 --> 00:19:30,960
they think the federal Gotan' birth don't want them to do it.

310
00:19:31,280 --> 00:19:35,100
And it's not true. But suffice to say they were devices that

311
00:19:35,100 --> 00:19:35,820
had SOLIDWORKS. S central communication that

312
00:19:35,820 --> 00:19:41,520
had Windows based computers in them. Um Actually I will tell

313
00:19:41,520 --> 00:19:45,060
half of this story. So while this was happening, you know I

314
00:19:45,060 --> 00:19:50,220
have the microphone and changed my mind. Um so we had a it was

315
00:19:50,220 --> 00:19:53,800
U H F or V H F radios on the radio and we are seeing the

316
00:19:53,800 --> 00:19:58,420
traffic coming from some device and we knew what Ethernet port

317
00:19:58,420 --> 00:20:01,560
it was on so therefore the engineers were able to tell us

318
00:20:01,560 --> 00:20:03,640
what room it was in. So I get on the radio and it's like hey

319
00:20:03,640 --> 00:20:08,620
someone in hospital blah blah blah go to room 67 and turn off

320
00:20:08,620 --> 00:20:11,200
the computer that's in there because it's making a lot of

321
00:20:11,200 --> 00:20:14,360
noise and breaking things. And the guy radios back and said

322
00:20:14,360 --> 00:20:18,540
okay I'm in this room and there's no computer here. I was

323
00:20:18,540 --> 00:20:21,480
like dude that's not possible. I'm looking at the traffic. I

324
00:20:21,480 --> 00:20:25,100
know the Ethernet port is in that room. There's got to be a

325
00:20:25,100 --> 00:20:27,760
computer in there turn it off. And he's like look. And I said

326
00:20:27,760 --> 00:20:30,440
where are you? Physically where are you? He's like well I'm in

327
00:20:30,440 --> 00:20:33,640
radiology and I'm telling you there's no bloody computer in

328
00:20:33,640 --> 00:20:36,580
here. I'm like alright fine. So I told the engineer alright

329
00:20:36,580 --> 00:20:39,220
turn off that Ethernet port. I don't know what that device is

330
00:20:39,220 --> 00:20:41,900
but turn it off. We'd been talking to the hospital

331
00:20:41,900 --> 00:20:44,380
administration the whole time during this thing so they knew

332
00:20:44,380 --> 00:20:48,100
that stuff was going on. So we disabled the port, took the

333
00:20:48,100 --> 00:20:51,460
machine offline and within about 90 seconds my computer

334
00:20:51,480 --> 00:20:55,400
my phone rings and it's the director of the ED at the

335
00:20:55,400 --> 00:21:03,380
hospital. And he says hi our only x-ray machine just went

336
00:21:03,380 --> 00:21:06,720
offline so I'm putting the entire hospital on drive-by.

337
00:21:07,080 --> 00:21:10,220
Drive-by meaning they can't accept new patients because they

338
00:21:10,220 --> 00:21:13,640
don't have an x-ray machine. So if an ambulance was on the way

339
00:21:13,640 --> 00:21:16,540
to that hospital they get to turn around and go somewhere else

340
00:21:16,540 --> 00:21:20,040
because they can't admit new patients. And so the director

341
00:21:20,040 --> 00:21:25,140
continued and said what did you just do? And I'm like well I

342
00:21:25,140 --> 00:21:31,340
mean nothing. I turned off a network port but you know it was

343
00:21:31,340 --> 00:21:34,080
a Windows box and you said it was an x-ray machine. That can't

344
00:21:34,080 --> 00:21:37,040
be an x-ray machine. And he said no no actually it runs

345
00:21:37,040 --> 00:21:43,600
Windows XP. Huh. So I get back on the radio and I ask the

346
00:21:43,600 --> 00:21:47,620
desktop guys like so you said there was a big box in there

347
00:21:47,620 --> 00:21:50,020
apparently it's an x-ray machine. Does it have a

348
00:21:50,040 --> 00:21:53,540
screen on it or something? And he goes over there and poke poke

349
00:21:53,540 --> 00:21:59,260
poke huh it runs Windows XP. I'm like oh god we just turned off

350
00:21:59,260 --> 00:22:03,220
the only x-ray machine in this entire hospital. So then the

351
00:22:03,220 --> 00:22:08,440
trick was you know do you turn it back on so you could admit

352
00:22:08,440 --> 00:22:11,600
patients because I you know I'm a computer geek. I'm not in the

353
00:22:11,600 --> 00:22:14,680
business of making life and death decisions right. That's why

354
00:22:14,680 --> 00:22:18,740
I went into computer science and not medicine. So we turned the

355
00:22:18,740 --> 00:22:20,020
machine back on and I told him I was going to turn it back on.

356
00:22:20,040 --> 00:22:22,640
I was like look man if I see it start propagating outside the

357
00:22:22,640 --> 00:22:26,120
organization we'll have no choice but to turn it off. So you

358
00:22:26,120 --> 00:22:28,960
know your call if you want to put somebody in there. So

359
00:22:28,960 --> 00:22:33,140
anyway that that was like you know early in my career and one

360
00:22:33,140 --> 00:22:36,000
of the worst moments I had because you think as you know in

361
00:22:36,000 --> 00:22:38,900
computer you know if you're in IT or infosec or whatever you

362
00:22:38,900 --> 00:22:42,280
know I'm dealing with you know packets and computers and you

363
00:22:42,280 --> 00:22:44,820
know if a computer doesn't isn't doing what I want it to do you

364
00:22:44,820 --> 00:22:46,760
just turn the damn thing off. You don't have to worry about it

365
00:22:46,760 --> 00:22:48,880
and then you know so I turned the damn thing off and now I got

366
00:22:48,880 --> 00:22:50,020
you know people in the ambulance and you know people in the

367
00:22:50,040 --> 00:22:55,640
ambulance is getting rerouted and so I chose to leave health

368
00:22:55,640 --> 00:23:00,340
care of course and got into consulting instead. So alright I

369
00:23:00,340 --> 00:23:03,760
told that story. Thanks for sitting through that. So now that

370
00:23:03,760 --> 00:23:08,940
we knew who the infected hosts were talking to and we knew who

371
00:23:08,940 --> 00:23:13,960
the infected hosts were now we needed to try and figure out you

372
00:23:13,960 --> 00:23:17,340
know what actual software is doing this and how do we get rid

373
00:23:17,340 --> 00:23:19,840
of it. So if you look at some of these logs from Snort

374
00:23:20,780 --> 00:23:24,560
the blue bits of course that's an IP address so we knew which

375
00:23:24,560 --> 00:23:28,940
hosts were infected and then it also told us which port was

376
00:23:28,940 --> 00:23:33,720
being used on the red bits there. So we had a port so we

377
00:23:33,720 --> 00:23:37,000
have a little more information now so now it's time to find the

378
00:23:37,000 --> 00:23:42,140
offending process. Anyone ever use F port? It's a great tool

379
00:23:42,140 --> 00:23:43,940
there's other tools that will give you the same information

380
00:23:43,940 --> 00:23:47,820
this is one of many ways to find this information out. But we had

381
00:23:47,820 --> 00:23:50,020
access to the network. So we had access to the network and we

382
00:23:50,040 --> 00:23:53,320
had access to the remote machines. Again disclaimer this

383
00:23:53,320 --> 00:23:57,260
was 2004. Everybody had local admin on every machine because

384
00:23:57,260 --> 00:24:02,180
why not. So it was easy to run F port and figure out so I knew

385
00:24:02,180 --> 00:24:07,160
that what was that number 1303. So if you look at all the

386
00:24:07,160 --> 00:24:15,580
processes look process 1420 sure enough was using port 1303. And

387
00:24:15,580 --> 00:24:19,820
look at that the executable associated with it is W32 USB2

388
00:24:20,980 --> 00:24:25,980
.exe. Not to be confused with the actual windows process W32

389
00:24:25,980 --> 00:24:32,080
USB.exe. So of course they gave it a file name that would make it

390
00:24:32,080 --> 00:24:36,000
easy to overlook and not know what you're seeing. So now we

391
00:24:36,000 --> 00:24:40,120
knew what piece of software was doing all this. Great. So how do

392
00:24:40,120 --> 00:24:42,920
we stop it? Well thankfully malicious software back then was

393
00:24:42,920 --> 00:24:47,500
much much less sophisticated than it is today. So we could use

394
00:24:47,500 --> 00:24:50,020
another tool called PS kill. Probably one of the most popular

395
00:24:50,020 --> 00:24:55,360
ways to process kill. You give it a UNC path to an IP address.

396
00:24:55,560 --> 00:25:00,160
You give it the name of the process. Again W32 USB2 and it

397
00:25:00,160 --> 00:25:05,400
kills it. It's a lot harder to do now. A lot of malicious

398
00:25:05,400 --> 00:25:08,880
software it will protect itself and reinstall itself and you

399
00:25:08,880 --> 00:25:12,240
can't necessarily just stop the process and delete it. But again

400
00:25:12,240 --> 00:25:16,240
the point is these were very simple techniques we were using

401
00:25:16,240 --> 00:25:19,900
to overcome technical deficiencies that we had. So

402
00:25:19,900 --> 00:25:23,180
we went through this and to be fair this is very much a manual

403
00:25:23,180 --> 00:25:26,300
process. In a sufficiently large organization this would not

404
00:25:26,300 --> 00:25:30,020
likely scale very well. You could script it I imagine and

405
00:25:30,020 --> 00:25:34,680
still be useful. But as we went through this process we found

406
00:25:34,680 --> 00:25:37,720
several. So I had mentioned that there were two vulnerabilities

407
00:25:37,720 --> 00:25:42,520
that were being exploited. All told there was one, two, three,

408
00:25:42,560 --> 00:25:46,280
four different infected files with a variety of different

409
00:25:46,280 --> 00:25:49,580
malicious softwares. As you see there's spy bot, woot bot,

410
00:25:49,900 --> 00:25:55,200
randex spy bot, rbot and the list goes on. I think the

411
00:25:55,200 --> 00:25:59,180
message here is there was a lot of these types of softwares being

412
00:25:59,180 --> 00:26:04,200
used around that time. And I'm not sure that we ever figured

413
00:26:04,200 --> 00:26:07,700
out who patient zero was. How that stuff got inside our

414
00:26:07,700 --> 00:26:11,960
network. But the point is it was a variety of things that were

415
00:26:11,960 --> 00:26:19,540
eating us up not just one thing. So the lesson here knowing

416
00:26:19,900 --> 00:26:22,840
which screw to turn. So there's a story. I have no idea if it's

417
00:26:22,840 --> 00:26:25,460
true or not. But it's very interesting and I like to

418
00:26:25,460 --> 00:26:31,420
believe it's true. Where a machinist was hired to fix an

419
00:26:31,420 --> 00:26:35,100
electric generator that wasn't working. And this was you know

420
00:26:36,580 --> 00:26:40,740
way back in the 20th century somewhere. And so the machinist

421
00:26:40,740 --> 00:26:44,860
walked in, looked at the device, turned a screw and then

422
00:26:44,860 --> 00:26:48,420
the generator started working again and he submitted his

423
00:26:48,420 --> 00:26:49,880
invoice for $10,000. So that's a lot of work. And so the

424
00:26:49,900 --> 00:26:54,080
client was like I mean $10,000 you turned a screw. You know you

425
00:26:54,080 --> 00:26:57,140
got to itemize that invoice and tell me what are you charging me

426
00:26:57,140 --> 00:26:59,240
for because that seems like an awful lot of money. So he said

427
00:26:59,240 --> 00:27:04,320
fine. I charged you $1 for turning the screw and I charged

428
00:27:04,320 --> 00:27:12,040
you $9,999 for knowing which screw to turn. So as I've walked

429
00:27:12,040 --> 00:27:15,300
through what we did, hopefully none of you are sitting there

430
00:27:15,300 --> 00:27:18,440
thinking wow that was really impressive that they knew how to

431
00:27:18,440 --> 00:27:19,880
use very, very basic software. So I'm going to give you a

432
00:27:19,900 --> 00:27:20,960
little bit of an overview of the security tools that everybody

433
00:27:20,960 --> 00:27:24,220
in this room probably knows how to use. But we were in an incident

434
00:27:24,220 --> 00:27:29,120
response situation where our other response techniques had

435
00:27:29,120 --> 00:27:34,220
failed and we felt like that we just happened to know which

436
00:27:34,220 --> 00:27:39,380
screw to turn. And the huge win in our case was not that

437
00:27:39,380 --> 00:27:43,280
anything we did was necessarily amazing but it was very

438
00:27:43,280 --> 00:27:49,160
effective and it overcame several challenges. Understanding the nature

439
00:27:49,900 --> 00:27:53,380
and the context of the incident based on the collective experience

440
00:27:53,380 --> 00:27:56,620
of all of us involved is what resulted in the malware

441
00:27:56,620 --> 00:28:01,260
eventually being eradicated. And as I mentioned in the beginning

442
00:28:01,260 --> 00:28:04,940
of this, if you go over to packet detective or capture the

443
00:28:04,940 --> 00:28:07,940
packet, a lot of these techniques are very, very similar

444
00:28:07,940 --> 00:28:12,560
to what you're still doing today. And so it's I think even

445
00:28:12,560 --> 00:28:15,800
though they're not the exciting, you know, kind of layer

446
00:28:15,800 --> 00:28:18,980
seven sorts of things that are becoming ever more popular,

447
00:28:19,900 --> 00:28:26,360
they're still very, very relevant. So one more case study that goes

448
00:28:26,360 --> 00:28:31,200
over some very similar things involved web dev vulnerabilities

449
00:28:31,200 --> 00:28:35,100
existed. It was back in 2003. I think it was on the weekend. I

450
00:28:35,100 --> 00:28:38,220
got a call from one of my buddies that had a new server he

451
00:28:38,220 --> 00:28:41,300
had deployed in the DMZ. And I answered the phone and he was

452
00:28:41,300 --> 00:28:45,420
like, hey, man, do you want to see a dead body? Recall we worked

453
00:28:45,420 --> 00:28:49,360
for a hospital system so I wasn't sure if that was a euphemism or

454
00:28:49,900 --> 00:28:52,620
something. So I didn't know where it was. I was like, what do

455
00:28:52,620 --> 00:28:56,160
you mean? And he's like, yeah, I got a server in the DMZ and

456
00:28:56,160 --> 00:28:58,960
there's a problem with it. I noticed that the C drive was

457
00:28:58,960 --> 00:29:02,960
filled up. I'm like, okay, what filled it up? Well, I don't

458
00:29:02,960 --> 00:29:07,600
know. So, yeah. Then it became interesting. He looked at the C

459
00:29:07,600 --> 00:29:11,240
drive of his machine that he had deployed. It was probably very

460
00:29:11,240 --> 00:29:14,040
difficult to see, but none of that stuff had anything to do

461
00:29:14,040 --> 00:29:19,560
with making sick people healthy. So we immediately thought, okay,

462
00:29:19,900 --> 00:29:24,400
well, what does the network profile of this box look like? So

463
00:29:24,400 --> 00:29:29,760
we scanned it and there was three, we pretty much knew what

464
00:29:29,760 --> 00:29:35,540
everything, oh, look, FTP was on there. Again, 2003, don't be a

465
00:29:35,540 --> 00:29:40,320
hater. So we knew what everything was except those bits

466
00:29:40,320 --> 00:29:47,420
in red. We weren't sure what the ports, 1978, 2003 and 3460. So

467
00:29:47,420 --> 00:29:49,820
those were the ones we knew just a quick in map and we

468
00:29:49,900 --> 00:29:52,020
knew, okay, those are the things we need to look at first

469
00:29:52,020 --> 00:29:56,540
to figure out what's going on. So F port, again, saved us. We ran

470
00:29:56,540 --> 00:30:01,360
F port on it and figured out that port 2003 was being used by

471
00:30:01,360 --> 00:30:08,000
a process called WIMNGNT, not to be confused with the

472
00:30:08,000 --> 00:30:15,020
legitimate Windows process called WIMN, oh, now I can't

473
00:30:15,020 --> 00:30:18,820
remember. It's spelled different. Trust me. Oh, there it

474
00:30:18,820 --> 00:30:19,880
is. Right there. Yeah. I should have done that. I should have

475
00:30:19,900 --> 00:30:26,080
read my own slides. WIMNGNT, not NT. Okay. So, all right, that

476
00:30:26,080 --> 00:30:28,340
didn't, we knew that was sketchy. We didn't know what it was. So

477
00:30:28,340 --> 00:30:30,840
what did we do? Well, of course, we telnet it over to

478
00:30:30,840 --> 00:30:34,900
port 2003 to see what it was. And apparently the bad guy

479
00:30:34,900 --> 00:30:39,000
wasn't done configuring his server, but we had, you know, it

480
00:30:39,000 --> 00:30:45,480
responded with a banner and it had, it was, you know, once it

481
00:30:45,480 --> 00:30:48,520
was set up, it would have had server statistics and, you know,

482
00:30:48,520 --> 00:30:49,880
this was a rogue FTP server. So, you know, it was, you know,

483
00:30:49,880 --> 00:30:55,060
this was a FTP site that was on our server in the hospital. So

484
00:30:55,060 --> 00:30:57,380
that was kind of neat. And I'm glad, thank goodness I'm a

485
00:30:57,380 --> 00:31:00,600
digital pack rat and I saved all of this information. Here I

486
00:31:00,600 --> 00:31:03,700
am, you know, 15 years later talking about it. We continued

487
00:31:03,700 --> 00:31:07,080
to look around the server to see what was interesting there. We

488
00:31:07,080 --> 00:31:10,960
found a file called rb.bat and apparently that was the install

489
00:31:10,960 --> 00:31:16,300
script that was used. And you can see it installs some things,

490
00:31:16,380 --> 00:31:17,680
starts some services up.

491
00:31:20,480 --> 00:31:23,520
And so, yeah, I'm not going to talk through the technical

492
00:31:23,520 --> 00:31:25,680
details of it. It was just interesting that we found

493
00:31:25,680 --> 00:31:29,660
artifacts left over by the bad guy. And so this is probably a

494
00:31:29,660 --> 00:31:32,660
well-known technique probably for most folks in here. But the

495
00:31:32,660 --> 00:31:36,980
hacker was kind enough to patch our system once he or she was

496
00:31:36,980 --> 00:31:41,860
done. So you can see here that the script stops the vulnerable

497
00:31:42,470 --> 00:31:47,280
service, modifies the registry such that that service was no

498
00:31:47,280 --> 00:31:49,860
longer vulnerable to the web dev buffer. So, you can see here,

499
00:31:49,880 --> 00:31:55,060
over flow, imported some information and then restarted

500
00:31:55,060 --> 00:31:59,000
the service at some point. And I think we all know why they do

501
00:31:59,000 --> 00:32:02,040
that. But, again, back in 2004, that was a bit non-intuitive.

502
00:32:02,280 --> 00:32:05,660
You know, why did the bad person take the trouble to patch

503
00:32:05,660 --> 00:32:08,360
our server after he attacked it? Well, of course, the answer is

504
00:32:08,360 --> 00:32:12,860
because the bad guy didn't want anybody else to also compromise

505
00:32:12,860 --> 00:32:17,280
that server and gain access to the server's resources. So, at

506
00:32:17,280 --> 00:32:19,620
least we knew that machine would only

507
00:32:19,880 --> 00:32:25,220
get owned once, I guess, with, you know. But, of course, any

508
00:32:25,220 --> 00:32:29,900
malicious software can only continue to exist if it finds

509
00:32:29,900 --> 00:32:32,160
other victims. So as we continued to look through the

510
00:32:32,160 --> 00:32:37,100
logs and whatnot that had been left behind, it began looking

511
00:32:37,100 --> 00:32:40,420
for new victims. So you can see it was looking for other hosts

512
00:32:40,420 --> 00:32:44,260
that had web dev enabled. I obfuscated those because those

513
00:32:44,260 --> 00:32:49,080
were not RFC 1918 IP addresses. They were routable IP addresses,

514
00:32:49,140 --> 00:32:49,860
which meant, you know, it was a little bit of a challenge to

515
00:32:49,880 --> 00:32:52,340
make sure that the hospital that I worked for was attacking

516
00:32:52,340 --> 00:32:55,660
other people out on the open Internet, which means that, you

517
00:32:55,660 --> 00:32:58,180
know, you're going to start getting, you know, angry phone

518
00:32:58,180 --> 00:33:00,580
calls from ISPs and whatnot wondering what you're doing.

519
00:33:01,840 --> 00:33:07,120
Excuse me. Lastly, that hacker was nice enough to leave behind

520
00:33:07,120 --> 00:33:11,020
his calling card. I started Googling some of the key words

521
00:33:11,020 --> 00:33:15,260
on there, and they started redirecting to a lot of TLDs

522
00:33:15,260 --> 00:33:18,000
that I have no interest in communicating with, so.

523
00:33:21,320 --> 00:33:24,100
If anyone would like to research that further and let me

524
00:33:24,100 --> 00:33:27,440
know who these people are or who they were, but like I said, you

525
00:33:27,440 --> 00:33:30,980
know, it was a lot of, yeah, a lot of domains I didn't want to

526
00:33:30,980 --> 00:33:36,640
visit. So that, again, none of those techniques were

527
00:33:36,640 --> 00:33:40,180
mind-blowing or complex or anything, but it was just very

528
00:33:40,180 --> 00:33:43,360
simple techniques that a couple guys on the weekend, we went

529
00:33:43,360 --> 00:33:48,060
through and figured some interesting things out. But

530
00:33:48,060 --> 00:33:49,720
there is another side of the coin.

531
00:33:49,880 --> 00:33:52,960
So I've talked about how people can be the most useful security

532
00:33:52,960 --> 00:33:58,180
tool, but sometimes we're the problem as well. And this is a

533
00:33:58,180 --> 00:34:00,520
much shorter part of this slide deck, so don't panic. We're

534
00:34:00,520 --> 00:34:05,040
getting close to the end. Thank you for still being here. It

535
00:34:05,040 --> 00:34:08,960
was on a red team exercise, and we had a very complex nine-step

536
00:34:08,960 --> 00:34:13,080
attack chain. And I'll show you a picture of the attack chain

537
00:34:13,080 --> 00:34:17,940
just to make the point that, ooh, this was really hard. But

538
00:34:19,880 --> 00:34:23,580
during that nine-step attack chain, we got very, very lucky

539
00:34:23,580 --> 00:34:26,960
because the reason the attack chain worked was because of human

540
00:34:26,960 --> 00:34:35,100
error on the part of the client. For example, we gained access to a

541
00:34:35,100 --> 00:34:38,480
workstation on which the administrator left credentials

542
00:34:38,480 --> 00:34:44,020
in a spreadsheet on his desktop. And this was last year. This was

543
00:34:44,020 --> 00:34:48,840
2016 when this happened. Okay. Don't leave your creds in a

544
00:34:48,840 --> 00:34:49,860
spreadsheet. This was in 2016. Don't leave your creds in a

545
00:34:49,880 --> 00:34:52,200
encrypted. Unencrypted. I didn't put this in the slide deck,

546
00:34:52,260 --> 00:34:59,040
but also, do not name the spreadsheet credentials.CSV. You

547
00:34:59,040 --> 00:35:03,280
you don't have to be a very clever red teamer to figure out

548
00:35:03,280 --> 00:35:09,300
that that's a file I'm going to open. And then we went through

549
00:35:09,300 --> 00:35:12,800
the attack chain. Once we had domain admin, during the read

550
00:35:12,800 --> 00:35:17,340
out, we asked the client, look, we were in domain admin for like

551
00:35:17,780 --> 00:35:21,600
three days before anyone said anything. You should probably

552
00:35:21,600 --> 00:35:29,020
monitor the DA group. And they said, we do. All right. So we

553
00:35:29,040 --> 00:35:32,680
asked, why didn't you notice that we had created an account

554
00:35:32,680 --> 00:35:35,660
for ourselves in the group? And they said, well, because the

555
00:35:35,660 --> 00:35:41,860
monitoring system that checked DA had failed. So you need a

556
00:35:41,860 --> 00:35:43,880
monitoring system for the monitoring system. And that

557
00:35:43,880 --> 00:35:46,160
sounds funny and everything, but it's true. I mean, if you've

558
00:35:46,160 --> 00:35:49,500
got one box that does a critical thing. So that was human error

559
00:35:49,500 --> 00:35:52,340
as well. They had a bad design. And whomever was responsible

560
00:35:52,340 --> 00:35:54,780
for that system hadn't checked it, apparently, because it

561
00:35:54,780 --> 00:35:59,020
wasn't doing its job. So this, I mean, this is a very good

562
00:35:59,040 --> 00:36:00,880
example. This has been sanitized. But, I mean, this

563
00:36:00,880 --> 00:36:04,120
looks amazing. But, again, I'm making the point that if it

564
00:36:04,120 --> 00:36:07,680
weren't for a couple of key failures by people, then we

565
00:36:07,680 --> 00:36:10,540
would not have been successful. So that's the cynical point of

566
00:36:10,540 --> 00:36:13,700
view, is that people, a lot of times, you know, there are

567
00:36:13,700 --> 00:36:16,360
processes. There are technical controls. There are things in

568
00:36:16,360 --> 00:36:20,040
place that will stop bad things from happening. But we're not

569
00:36:20,040 --> 00:36:23,160
perfect. And we, you know, sometimes we make mistakes. And

570
00:36:23,160 --> 00:36:29,020
this is what happens. So to start to wrap it up, I'm going

571
00:36:29,040 --> 00:36:30,440
to give you a couple of examples. And I just copied and

572
00:36:30,440 --> 00:36:33,740
pasted this directly from the white paper I mentioned that I

573
00:36:33,740 --> 00:36:38,420
wrote. So we had several technological challenges. We

574
00:36:38,420 --> 00:36:41,800
used commercial and freely obtainable tools. And we

575
00:36:41,800 --> 00:36:45,540
overcame the obstacles in a resourceful and cost-efficient

576
00:36:45,540 --> 00:36:49,640
manner. So the point is, problems can be solved

577
00:36:49,640 --> 00:36:56,040
creatively. I really do speak English, I promise. Using

578
00:36:56,040 --> 00:36:59,020
limited resources. And so we have to regularly evaluate the

579
00:36:59,040 --> 00:37:04,920
commercial products to use inside our organizations. But

580
00:37:04,920 --> 00:37:07,960
properly trained personnel can be more valuable to an

581
00:37:07,960 --> 00:37:12,840
organization than any hardware or software device. So what can

582
00:37:12,840 --> 00:37:16,680
we do as information security professionals? Now this is where

583
00:37:16,680 --> 00:37:22,540
the conversation usually begins. And you may or may not agree with

584
00:37:22,540 --> 00:37:25,400
me. And if you don't agree with me, that's awesome. Because then

585
00:37:25,400 --> 00:37:28,920
you can tell me your opinion and maybe I can learn something to

586
00:37:29,040 --> 00:37:32,900
change. But I think we should spend on people first and

587
00:37:32,900 --> 00:37:36,200
technology second. If you build a team around a set of

588
00:37:36,200 --> 00:37:39,420
technologies that you deploy, then when those technologies

589
00:37:39,420 --> 00:37:44,500
change, you may be left with the wrong team for your

590
00:37:44,500 --> 00:37:49,140
organization. So I advocate for building the team first and

591
00:37:49,140 --> 00:37:53,100
making sure they're enabled to adapt as the technology changes

592
00:37:53,880 --> 00:37:58,320
because it's the people that are more important, I think. When

593
00:37:59,040 --> 00:38:02,340
you're ready to spend on technology, make sure you consider the

594
00:38:02,340 --> 00:38:05,420
skills of the people you already have. You might make a decision to

595
00:38:05,420 --> 00:38:09,280
deploy a certain technology based on your support model. Or maybe

596
00:38:09,280 --> 00:38:13,240
you know if you're going to deploy a certain technology, then you

597
00:38:13,240 --> 00:38:16,480
will have to include training to get your people the appropriate

598
00:38:16,480 --> 00:38:20,760
skills to support it. And that's, you know, identify those gaps.

599
00:38:20,980 --> 00:38:24,240
Understand what's going to, it's not just buying the widget. With

600
00:38:24,240 --> 00:38:28,820
all due respect to any sales folks that might be here, just

601
00:38:29,040 --> 00:38:31,720
because someone says, oh, yeah, this product's easy to support

602
00:38:31,720 --> 00:38:38,200
and doesn't require any training, that's rarely true, right? And

603
00:38:38,200 --> 00:38:41,220
work outside your organizational silos. I think this is less of a

604
00:38:41,220 --> 00:38:44,300
problem these days, but I mentioned in the beginning slides that

605
00:38:44,300 --> 00:38:47,820
our networking team was reluctant to engage the InfoSec team

606
00:38:48,380 --> 00:38:51,680
because, you know, we were in a different silo. I think that's

607
00:38:51,680 --> 00:38:55,880
less of a problem these days, but, you know, just be aware of

608
00:38:55,880 --> 00:38:59,020
the capabilities that exist in your organization outside of the

609
00:38:59,040 --> 00:39:04,000
one in which you work. So, that's my story. I'm sticking to

610
00:39:04,000 --> 00:39:09,600
it. Many thanks. Oh, right. So, I know putting up a QR code in

611
00:39:09,600 --> 00:39:13,980
front of an InfoSec crowd is probably a wildly unpopular idea.

612
00:39:14,600 --> 00:39:18,900
You can trust me. There's nothing malicious in there. So,

613
00:39:19,000 --> 00:39:21,880
go ahead and scan it. Everybody scan it right now. So, anyway,

614
00:39:21,920 --> 00:39:25,040
that's me. That's my e-mail address on Twitter. I'm Damon

615
00:39:25,040 --> 00:39:29,020
Small. Comments or questions before we wrap up?

616
00:39:29,040 --> 00:39:32,540
I guess this is where Ming gets up and says comments or

617
00:39:32,540 --> 00:39:36,160
questions. So, any comments and questions, please. Thank you,

618
00:39:36,220 --> 00:39:42,400
Ming. Nothing. All right. So, I'll be around. I will be over

619
00:39:42,400 --> 00:39:45,520
there with the wall of sheep most of the time. I'm happy to

620
00:39:45,520 --> 00:39:48,020
talk about this more. If anyone has any comments or questions

621
00:39:48,020 --> 00:39:51,800
later on you want to ask me or catch me, please do. Thank you

622
00:39:51,800 --> 00:39:55,640
for sitting through this. Enjoy the rest of DEF CON 25. Cheers.

623
00:39:59,040 --> 00:40:00,380
Thank you.