1
00:00:01,080 --> 00:00:03,400
Okay, I guess we'll go ahead and get started.

2
00:00:04,400 --> 00:00:10,500
I am Julian Grizzard and I'm a graduate student at Georgia Tech and studying computer security.

3
00:00:10,780 --> 00:00:17,100
And today I'm gonna give you a talk on Surgical Recovery from Current-Level Rootkit Installations.

4
00:00:17,740 --> 00:00:20,360
And this is on Linux-based systems.

5
00:00:22,890 --> 00:00:29,010
So first off, the latest slides, tools, and our spine architecture work is available online.

6
00:00:29,110 --> 00:00:31,110
The links to this is on your CD.

7
00:00:31,110 --> 00:00:35,090
And so definitely download the latest, spent a lot of changes.

8
00:00:36,850 --> 00:00:40,630
All right, so first off, some background on Rootkits.

9
00:00:40,630 --> 00:00:42,270
What does a Rootkit do?

10
00:00:42,710 --> 00:00:52,290
So a Rootkit is a toolkit that an attacker will use to retain access to your system and hide activity that he's doing on your system.

11
00:00:52,790 --> 00:00:57,070
A Rootkit doesn't give you access to a system if you don't have prior access.

12
00:00:57,070 --> 00:01:02,990
You have to exploit the system by some known vulnerability or something you figured out, or social engineering or whatever.

13
00:01:03,630 --> 00:01:14,350
And then the attacker will usually install a Rootkit in order to come back into the system at a later point in case the password changes or his vulnerability's patched.

14
00:01:18,630 --> 00:01:25,230
So how many people have personally had experience with a Rootkit installed on a system they administered?

15
00:01:25,790 --> 00:01:28,430
We got some people, oh wow, we got lots of people out there.

16
00:01:28,430 --> 00:01:35,090
Interesting, and so what is the recommended thing to do when you get a Rootkit installed on your system?

17
00:01:37,070 --> 00:01:39,490
That's right, format and reinstall.

18
00:01:39,790 --> 00:01:44,050
So I'm actually going against that grain, you know what I'm saying?

19
00:01:44,050 --> 00:01:48,130
Okay, maybe there's more efficient ways to recover from a Rootkit.

20
00:01:48,130 --> 00:01:53,650
I mean, what if you got a server that every minute of downtime costs millions of dollars?

21
00:01:53,690 --> 00:01:56,310
I mean, how can you format and reinstall that system?

22
00:01:58,210 --> 00:02:02,110
So we're going to look at different ways to recover from a Rootkit.

23
00:02:02,910 --> 00:02:11,850
So first off, I want to give you guys some background on operating systems because what a Rootkit does is it replaces parts of your operating system.

24
00:02:11,850 --> 00:02:18,490
So it's important to understand how an operating system works in order to understand the design of Rootkits.

25
00:02:19,030 --> 00:02:22,210
So this is a monolithic operating system.

26
00:02:22,490 --> 00:02:25,850
That's just a fancy term for this particular architecture.

27
00:02:25,850 --> 00:02:28,050
Linux looks like this.

28
00:02:28,070 --> 00:02:29,830
Windows, all the windows look like this.

29
00:02:29,830 --> 00:02:31,430
Most systems look like this.

30
00:02:32,270 --> 00:02:34,830
So you've got two parts to an operating system.

31
00:02:34,830 --> 00:02:47,950
You've got the kernel space and this is the code that runs right on top of your hardware, controls all your devices, schedules your processes, does your file access, network stack, these kinds of things.

32
00:02:50,580 --> 00:02:52,800
And you have user space.

33
00:02:52,800 --> 00:02:55,740
User space is where the programs that you really care about run.

34
00:02:55,740 --> 00:03:02,880
So P1 could be your word processor, P2 could be a LS, bash command, and so on and so forth.

35
00:03:04,420 --> 00:03:09,680
Now I want to give you a contrasted operating system design that I'll come back to later.

36
00:03:09,840 --> 00:03:14,200
But I just want to give you a preview and compare and contrast to current OSs.

37
00:03:14,200 --> 00:03:17,540
This is the microkernel operating system design.

38
00:03:18,260 --> 00:03:22,280
The difference here is you have a very thin layer for the kernel space.

39
00:03:22,700 --> 00:03:33,420
You have minimal, I mean, only code you have in the kernel is the minimum amount of code needed to guarantee isolation among processes and that kind of thing.

40
00:03:33,420 --> 00:03:43,120
All the drivers, video, disk, hardware, network, all this stuff is pushed into user space sitting alongside your user space processes.

41
00:03:43,220 --> 00:03:47,980
And each bubble here is guaranteed isolated by the microkernel.

42
00:03:50,430 --> 00:03:53,450
So what guarantees this isolation?

43
00:03:53,850 --> 00:04:01,650
Well, on the Intel x86 architecture, a lot of you have probably heard of this ring architecture where you have level 0, 1, 2, 3.

44
00:04:01,930 --> 00:04:06,190
And it turns out modern operating systems only use two levels.

45
00:04:06,190 --> 00:04:11,370
They use level 0 for the kernel, which gives you unlimited access to the hardware.

46
00:04:11,490 --> 00:04:18,970
If you have a program running at level 0, you can do anything to any bit anywhere in the computer.

47
00:04:19,630 --> 00:04:24,170
And the user space processes run at level 3, which is shown in yellow here.

48
00:04:24,730 --> 00:04:28,890
And if you want access, they don't have direct access to the hardware.

49
00:04:28,890 --> 00:04:33,430
If you want access to the hardware, you have to make a system call, ask the kernel to do it for you.

50
00:04:36,090 --> 00:04:42,970
So I want to show you just real quick how this is implemented in the hardware.

51
00:04:42,990 --> 00:04:49,970
There's a 16-bit code segment register on Intel Pentiums and x86 processors.

52
00:04:50,570 --> 00:04:57,970
And the lower two bits of that register specify what privilege level you're running at, which is which ring you're in.

53
00:04:58,710 --> 00:05:02,890
And this is some test code to test and see which ring you're in.

54
00:05:05,110 --> 00:05:08,950
So I'm going to demo this code running real quick.

55
00:05:08,950 --> 00:05:11,490
Just a quick demo here.

56
00:05:13,840 --> 00:05:17,640
So I have a Linux box running over here to my right.

57
00:05:18,320 --> 00:05:19,940
I'm going to log into that.

58
00:05:30,190 --> 00:05:36,290
All right, so I have this getcpl program that gets your current privilege level.

59
00:05:36,450 --> 00:05:41,870
And it returns three, because I'm running a user space process forking off the command line.

60
00:05:43,350 --> 00:05:45,090
Now let's check out the kernel.

61
00:05:53,880 --> 00:05:57,820
All right, so I inserted a kernel module called...

62
00:06:00,360 --> 00:06:02,440
Well, it's way too big on there.

63
00:06:03,780 --> 00:06:08,420
I inserted a kernel module called getcpl, which all it does is...

64
00:06:10,200 --> 00:06:19,460
Right here, all it does is pops the CS register from into a variable, and ends it with a three to get the ring level.

65
00:06:19,960 --> 00:06:28,060
And if you look at the bar log sys log, you can see that, in fact, the kernel is running at ring zero.

66
00:06:37,490 --> 00:06:41,430
All right, I'm going to get my next demo set up, and we'll go back to the presentation.

67
00:06:50,220 --> 00:06:51,900
And so these slides here are just showing that.

68
00:06:51,900 --> 00:07:02,800
All right, so if you're looking at the operating system, and you want to make a user level rootkit, how do you do it?

69
00:07:02,800 --> 00:07:06,500
Well, you attack files on the hard disk, shown in red here.

70
00:07:06,960 --> 00:07:11,080
And so you replace ps, netstat, ls, top, password.

71
00:07:11,120 --> 00:07:16,280
These kinds of files, you replace those with malicious files that hide your activities.

72
00:07:16,280 --> 00:07:19,480
Hide files, hide network connections, these kinds of things.

73
00:07:19,840 --> 00:07:26,060
And so when you replace those files, they get loaded into your processes, the green bubbles up there.

74
00:07:27,020 --> 00:07:29,300
So that is pretty much the user level rootkit.

75
00:07:29,320 --> 00:07:30,680
It's pretty easy to detect.

76
00:07:30,680 --> 00:07:36,660
You just run tripwire, check your system against your known good state, and it's not too difficult.

77
00:07:36,660 --> 00:07:37,900
We know how to do that.

78
00:07:38,480 --> 00:07:43,900
Definitely tripwire has its issues, but there's a way to deal with those.

79
00:07:44,480 --> 00:07:52,800
Now kernel level rootkit attacks, they modify the running kernel code and the running data structures, which is this block shown here in red.

80
00:07:53,280 --> 00:07:58,180
So anything in there is suspect to being attacked by a kernel level rootkit.

81
00:07:58,520 --> 00:08:02,820
And all a kernel is, is a running program on your computer.

82
00:08:02,860 --> 00:08:11,320
So in essence, these kernel level rootkits are modifying code on the fly and redirecting that code to malicious code.

83
00:08:11,820 --> 00:08:20,200
In particular, the system calls is the gateway from user space to kernel, and that's a good target.

84
00:08:20,200 --> 00:08:22,500
And I'll go over those two examples.

85
00:08:23,600 --> 00:08:33,240
So first I want to go over a system call and why that is the most popular attack target for kernel level rootkits.

86
00:08:33,380 --> 00:08:43,020
So it's important to understand how a system call works in order to A, detect that the system call has been tampered with, and B, recover from it.

87
00:08:43,600 --> 00:08:47,120
So what happens, what do you use a system call for?

88
00:08:47,120 --> 00:08:51,300
Okay, say you're a user process and you want to read files from a hard disk.

89
00:08:51,440 --> 00:08:55,260
Then you can't do that directly because you don't have access to the hardware.

90
00:08:55,260 --> 00:09:02,660
So you issue a system call to the kernel to open, read the contents of those files, and so on and so forth.

91
00:09:03,240 --> 00:09:06,120
So for instance, let's say we have a sys read system call.

92
00:09:06,120 --> 00:09:07,080
What happens?

93
00:09:07,260 --> 00:09:22,800
Well, the first thing that happens is your user space process issues an int 0x80 instruction, which says a software interrupt, which indexes into 0x80th entry of the interrupt descriptor table.

94
00:09:23,380 --> 00:09:24,860
And that's shown right here.

95
00:09:25,100 --> 00:09:37,320
So as soon as you issue that interrupt, you go over to the kernel, the kernel looks up where the IDT is in memory, looks up that 0x80th entry, and executes the code pointed to by that entry.

96
00:09:37,760 --> 00:09:41,660
The code pointed to by that entry is the system call handler.

97
00:09:42,680 --> 00:09:53,380
And the system call handler is responsible for setting up parameters of your system call, generic parameters, it works, that's the first thing that's called for any system call.

98
00:09:53,680 --> 00:09:57,520
And then it calls the system call you specifically requested.

99
00:09:57,960 --> 00:10:04,660
So you see this pointer coming out of the code in the system call handler, which points to the system call table.

100
00:10:05,020 --> 00:10:13,720
So it references, let's say for sys read, you look up this entry number three in the system call table over here.

101
00:10:14,380 --> 00:10:17,900
And that redirects you down here to the sys read system call.

102
00:10:18,480 --> 00:10:24,960
And the sys read then takes the parameters given to it by a system call handler and executes your request.

103
00:10:28,890 --> 00:10:31,070
So what's the attack points here?

104
00:10:31,110 --> 00:10:34,310
Anything in this chain is an attack point.

105
00:10:34,690 --> 00:10:48,850
You could redirect the IDT entry, you could redirect the system call handler reference to the system call table, you could redirect individual system calls, you could overwrite the system call with a jump instruction to go somewhere else.

106
00:10:48,850 --> 00:10:50,890
All these are susceptible.

107
00:10:59,310 --> 00:11:03,290
So, how do you recover from this system call attack?

108
00:11:03,470 --> 00:11:11,290
Well here's the algorithm, it's basically installing a new system call table into the running kernel.

109
00:11:11,290 --> 00:11:13,910
This is exactly what a rootkit will do, in fact.

110
00:11:13,910 --> 00:11:18,450
But we're gonna use a good system call table that has known good state.

111
00:11:19,070 --> 00:11:31,650
So the best place to get that is to keep a copy of your kernel on another disk on safe storage and you can rip out these system calls using what I did was modify GDB to get that code out.

112
00:11:32,250 --> 00:11:51,570
And so you, it's pretty simple, you just, well, it's somewhat simple, it is kernel stuff, but basically you copy each system call into kernel memory and then you create a new system call table pointing to all those system calls and then you can set the handler to point to that system call table.

113
00:11:52,330 --> 00:12:11,310
And there's, you can go full blown, you can, if we go back to this picture, you can set this pointer right here which is on the left for you guys and you can point that to a new system call handler if you want and that way you'll know the whole chain is correct.

114
00:12:15,360 --> 00:12:21,980
Okay, so it's a little bit, just want to give you guys some details on, it's not quite that simple, you can't just copy stuff over.

115
00:12:22,080 --> 00:12:32,640
When you copy code around in memory, x86 code has these call instructions with relative offset parameters, so you have to adjust them, basically relink them into the kernel.

116
00:12:33,180 --> 00:12:44,820
So you could recompile the code if you know where it's gonna be loaded, but I chose just to recompute those relative offsets as I was inserting each system call into memory.

117
00:12:45,300 --> 00:12:57,940
And you adjust the relative offset based on where it's placed in memory because you allocate dynamic memory for it and you compare that to where it was originally in memory and you just subtract the difference.

118
00:12:58,880 --> 00:13:05,140
Of course, it's a little bit more confusing with the way the Intel architecture works, but that's the gist of it.

119
00:13:06,800 --> 00:13:11,080
So another thing is there's multiple ways into the kernel.

120
00:13:11,440 --> 00:13:12,620
I'll just give you some background.

121
00:13:12,620 --> 00:13:21,060
This is another tidbit of kernel-level rootkit stuff is you can insert a kernel module into the kernel and muck with the system call table.

122
00:13:21,300 --> 00:13:23,760
And that's one way to penetrate the kernel.

123
00:13:24,480 --> 00:13:29,260
Another way you can get into the kernel is through the dev.kmem.device file.

124
00:13:29,580 --> 00:13:36,900
This device file gives you access to the kernel memory as a file.

125
00:13:37,400 --> 00:13:41,540
So if you're a user space process, you can open that dev.kmem like it's a regular file.

126
00:13:41,660 --> 00:13:49,440
You can seek to it and you pass it a kernel memory address and it seeks that memory and then you can read that data or write to it.

127
00:13:51,140 --> 00:13:58,220
Is anybody familiar with the SuckIt rootkit from those people that have had installations on their computers?

128
00:13:59,040 --> 00:14:01,120
We got some people, all right.

129
00:14:02,460 --> 00:14:03,760
So there's a few more.

130
00:14:03,760 --> 00:14:11,260
Okay, so if you want to install your rootkit from dev.kmem without doing a kernel module, there's a couple of things that are tricky.

131
00:14:12,300 --> 00:14:14,500
First, how do you find the system call table?

132
00:14:14,640 --> 00:14:15,960
You don't have the symbols.

133
00:14:16,400 --> 00:14:22,160
And two, how do you allocate memory to insert new code into the kernel?

134
00:14:26,470 --> 00:14:30,830
All right, so first I'll just go over how you find the system call handler right quick.

135
00:14:31,490 --> 00:14:35,090
This is some code pulled out of SuckIt.

136
00:14:35,090 --> 00:14:41,190
First thing you do is you pop the IDT register into memory.

137
00:14:41,890 --> 00:14:45,530
And what that is, is that tells you the address.

138
00:14:45,530 --> 00:14:47,610
Let's go back a few slides real quick.

139
00:14:47,610 --> 00:14:50,670
That tells you where this table is in memory.

140
00:14:54,210 --> 00:14:56,730
All right, so now you know where that table is.

141
00:14:56,790 --> 00:15:08,870
So then you read the 0x80th entry of that table and you can compute where the system call handler is in memory.

142
00:15:08,870 --> 00:15:12,870
And then you can parse the system call handler to find out where the system call table is.

143
00:15:15,980 --> 00:15:24,460
All right, so how do you get dynamically allocated memory from the kernel from user space?

144
00:15:24,460 --> 00:15:31,040
If you write a kernel module, you just use kmalloc, but you don't have access to that code from user space.

145
00:15:31,460 --> 00:15:37,820
Well, the way SuckIt does it is sets up kmalloc as a system call.

146
00:15:38,480 --> 00:15:42,120
So there's a number of unused system calls in the system call tabler.

147
00:15:42,980 --> 00:15:47,420
SuckIt takes over one of those system calls and points it to kmalloc.

148
00:15:47,420 --> 00:15:57,400
So now you can issue system call 180 or something like that and pass it a size, memory allocation size and you'll get kernel memory returns.

149
00:16:01,140 --> 00:16:04,020
Okay, so now I wanna do my next demo.

150
00:16:24,770 --> 00:16:36,390
Okay, so the scenario here is I've already got root access to this machine, so I'm wanting to install my rootkit to retain access and to hide some files.

151
00:16:36,650 --> 00:16:47,170
In this folder right here, I have some bank accounts I wanna store on this computer and don't worry, these are not legit.

152
00:16:49,990 --> 00:16:51,270
At least I hope not.

153
00:16:53,230 --> 00:17:02,710
Okay, so we've got this data that I don't wanna store on my own computer, but I wanna store it on this machine that I've compromised so I can trade it with somebody else, something like that.

154
00:17:06,140 --> 00:17:17,360
So then I've already downloaded my SuckIt rootkit onto the system and I'm going to install it and it's done.

155
00:17:17,800 --> 00:17:26,320
So now that directory that I was just showing you up here, this hidden info.sk12 is now hidden.

156
00:17:26,700 --> 00:17:32,440
The way this works is anything that ends in .sk12 will be hidden.

157
00:17:32,440 --> 00:17:40,580
For instance, if I touch a file called hide.me, you can see that it's there.

158
00:17:40,640 --> 00:17:46,720
If I move hide.me to hide.me.sk12, it's gone.

159
00:17:51,840 --> 00:17:58,720
So now I'm the administrator of this computer and I believe I may have a rootkit on my system.

160
00:17:59,360 --> 00:18:07,280
So what I'm going to do is try and install a new system call table to see if there's any differences in the file system.

161
00:18:10,120 --> 00:18:13,860
Okay, so this takes a little bit of time because it's not optimized.

162
00:18:14,100 --> 00:18:17,160
It's good for demo though because I can explain a few things.

163
00:18:17,520 --> 00:18:24,100
So what you see here is each little block of code, it's copying a system call into kernel memory.

164
00:18:25,500 --> 00:18:33,780
It issues a kmalloc for each one of these, copies it in, and then you see these asterisks, asterisks, asterisks, found, dot, dot, dot.

165
00:18:33,780 --> 00:18:41,880
It's finding these call instructions and it's recomputing where they need to be relinked to and setting that.

166
00:18:43,960 --> 00:18:48,740
So right now we're at about 130 system calls that have been copied into memory.

167
00:18:48,740 --> 00:18:53,440
Turns out Linux has 255-ish system calls.

168
00:18:54,400 --> 00:18:57,640
And so we'll be through this here in a second.

169
00:19:12,400 --> 00:19:14,500
Does anybody have any questions so far?

170
00:19:14,500 --> 00:19:15,900
We've got a few more minutes.

171
00:19:29,980 --> 00:19:36,680
From what I've found, I mean, that's a possibility that they have absolute calls, but you could relink those in as well.

172
00:19:36,780 --> 00:19:40,060
From what I've found, they don't have absolute calls.

173
00:19:50,470 --> 00:19:51,870
Excellent, excellent question.

174
00:19:51,870 --> 00:19:53,730
I will get to that very shortly.

175
00:19:56,050 --> 00:20:02,050
All right, so the new system call table has been installed.

176
00:20:02,050 --> 00:20:04,890
Now notice these hidden files come back up.

177
00:20:05,810 --> 00:20:09,050
So as you can see, all Sucket was doing was redirecting some system calls.

178
00:20:09,050 --> 00:20:11,630
So we install a known good system call table into memory.

179
00:20:11,710 --> 00:20:13,990
All of a sudden, these files come back up.

180
00:20:24,140 --> 00:20:28,320
Does anybody have any other questions while we got a stopping point in the presentation?

181
00:20:43,830 --> 00:20:45,330
How does it catch it?

182
00:20:48,680 --> 00:20:50,480
Right, let's go back to that figure.

183
00:20:53,860 --> 00:21:01,840
All right, so you're saying this code right here is being overwritten or something like that.

184
00:21:02,440 --> 00:21:07,820
Okay, so first off, the tools I was just showing you do not do rootkit detection.

185
00:21:07,820 --> 00:21:11,780
Well, some of them do, but the recover rootkit itself does not.

186
00:21:12,240 --> 00:21:22,740
I do have the ability to read any system calls, or I could dump this code for sysread to a file and see if it's been overwritten.

187
00:21:22,740 --> 00:21:25,400
For instance, to answer your question, that would be the way to do it.

188
00:21:25,480 --> 00:21:26,860
It's changing memory.

189
00:21:29,740 --> 00:21:39,340
So the way the recovery tools work is, right now it actually just replaces this pointer to a new system call table.

190
00:21:40,100 --> 00:21:43,200
And so that's gonna shoot over here into memory somewhere.

191
00:21:43,520 --> 00:21:46,800
You got a new system call table over here, you got a new sysread over here.

192
00:21:46,940 --> 00:21:52,420
So even if this sysread right here was overwritten, that's not a problem because we aren't depending on that.

193
00:21:52,780 --> 00:21:53,800
Good question.

194
00:21:59,250 --> 00:22:03,830
Okay, so back to our gentleman's over here's question.

195
00:22:04,330 --> 00:22:25,370
Okay, so it turns out if you use devkmem to modify the kernel, as I do with my recovery tools, or even if you use a kernel module to do the recovery, either one of these approaches relies on system calls to do the recovery.

196
00:22:25,490 --> 00:22:27,410
So there's an obvious problem here.

197
00:22:27,410 --> 00:22:44,030
If you've installed a rootkit that's redirected system calls and you're depending on those system calls that's redirected, then you may be depending on code that's going to prevent your recovery tools from working.

198
00:22:45,110 --> 00:22:46,730
Does that make sense to everybody?

199
00:22:46,730 --> 00:22:48,030
It's a little bit confusing.

200
00:22:48,430 --> 00:22:55,910
Okay, so how do we deal with this?

201
00:22:56,710 --> 00:23:03,110
Well, I propose we need a new system architecture.

202
00:23:03,310 --> 00:23:13,290
And let me take a minute to explain the vision I have here, which goes much beyond rootkits and much beyond anything like that.

203
00:23:13,290 --> 00:23:18,010
I've got this vision of intrusion recovery system.

204
00:23:18,670 --> 00:23:28,790
So this is like, you know, you've got your intrusion detection system, you've got your intrusion prevention system, but bottom line is we still have compromises every day.

205
00:23:28,790 --> 00:23:30,950
It's still a problem for the foreseeable future.

206
00:23:31,190 --> 00:23:39,090
So we need to think about ways to more quickly recover from compromises given that they're going to be a problem.

207
00:23:39,090 --> 00:23:40,790
Systems are so complex.

208
00:23:41,510 --> 00:23:52,690
So that's why I've come up with this notion of an intrusion recovery system, which has a new system architecture that's not currently used in widely available OSs.

209
00:23:52,690 --> 00:24:02,030
And it's capable of verifying the integrity of the running system and repairing that state if it has some problems.

210
00:24:02,690 --> 00:24:16,570
And it has, you know, obviously there's got to be some kind of, this has got to be the top most secure piece of your system because you rely on it for, you know, your system running good.

211
00:24:16,750 --> 00:24:21,930
So you have to have some kind of way to store known good state for your system.

212
00:24:22,430 --> 00:24:28,250
Because if you store the known good state in your system that is accessible by root, then that can be changed.

213
00:24:28,310 --> 00:24:32,370
So this new architecture supports these kinds of things.

214
00:24:32,450 --> 00:24:37,770
It supports what I call a state hold, which is kind of like a strong hold, but it stores your state instead.

215
00:24:38,710 --> 00:24:40,490
And it does these other things.

216
00:24:40,490 --> 00:24:43,890
And I'll go over, this is my research right now.

217
00:24:43,890 --> 00:24:45,350
So it's certainly in development.

218
00:24:45,470 --> 00:24:47,970
And I'm looking for any kind of feedback you guys have.

219
00:24:47,970 --> 00:24:51,110
It's probably got some problems, but I'll show you what I've got.

220
00:24:51,110 --> 00:24:53,710
And we'll see what you say.

221
00:24:54,630 --> 00:25:03,370
So back to the beginning, I showed you, you've got the monolithic operating systems and then you've got these microkernel operating systems.

222
00:25:04,230 --> 00:25:07,070
Well, I'm gonna go back to this microkernel operating system.

223
00:25:07,490 --> 00:25:10,990
Turns out this looks really good for security applications.

224
00:25:11,630 --> 00:25:19,050
If you look at the microkernel, it has about 20,000 lines of code that run the kernel.

225
00:25:19,670 --> 00:25:23,530
Compare that to the Linux kernel, which has on the order of 4 million.

226
00:25:23,950 --> 00:25:30,370
So we can hopefully better guess or better rely on the microkernel for correctness.

227
00:25:30,470 --> 00:25:38,050
Matter of fact, there's some research microkernels out there that try and formally prove that the microkernel is correct.

228
00:25:40,980 --> 00:25:47,240
The system calls to the microkernel, there's only 14 or so, just depends on the implementation.

229
00:25:47,440 --> 00:25:49,460
Maybe on the order of 10.

230
00:25:49,460 --> 00:25:52,980
So you've got a small number of system calls that do basic stuff.

231
00:25:52,980 --> 00:25:59,540
You do IPCs, you do inter-process communication, you do thread management, that kind of thing.

232
00:25:59,540 --> 00:26:00,800
That's it, actually.

233
00:26:00,880 --> 00:26:04,760
All the hardware access, these kinds of things are pushed to user level.

234
00:26:09,140 --> 00:26:18,580
So based on this microkernel design, I've laid out this spine architecture for the intrusion recovery system.

235
00:26:20,340 --> 00:26:31,360
At the base of the architecture you have the microkernel, which I'm using L4 Fiasco implementation, which is developed by a university in Dresden, Germany.

236
00:26:33,060 --> 00:26:40,040
And on top of that runs the guest kernel, the L4 Linux implementation runs on top of that.

237
00:26:40,040 --> 00:26:43,920
So you can run a full-blown version of Linux on top of the microkernel.

238
00:26:44,660 --> 00:26:49,980
So you're probably more familiar with the term virtual machine than you are microkernel.

239
00:26:49,980 --> 00:26:53,480
Turns out these are very highly related topics.

240
00:26:53,900 --> 00:26:56,440
Essentially the microkernel does the same thing for us.

241
00:26:56,660 --> 00:27:01,900
And so we have this guest kernel, which is L4 Linux running on top of it.

242
00:27:02,580 --> 00:27:08,760
At the next level you've got your user processes, and those run just like they do on Linux.

243
00:27:08,760 --> 00:27:10,180
You don't even have to modify them.

244
00:27:11,480 --> 00:27:17,440
And then over here on the first, only Fiasco runs in kernel mode.

245
00:27:17,440 --> 00:27:24,020
So only Fiasco runs in that ring zero, and everything else runs in ring three.

246
00:27:24,460 --> 00:27:35,060
That means even if you have root access on the kernel machine, you're not guaranteed access to all bits in the computer system.

247
00:27:39,310 --> 00:27:46,470
So on the right here I have at each level a component of the intrusion recovery system.

248
00:27:46,470 --> 00:27:49,470
I call them L0, L1, L2, L3.

249
00:27:49,930 --> 00:27:51,550
And L0 is just the hardware.

250
00:27:51,550 --> 00:27:55,850
There's really no support for that right now, just except the ring stuff.

251
00:27:56,390 --> 00:28:04,010
Turns out Intel and IBM, these kinds of companies, are very interested in hardware support for virtual machines.

252
00:28:04,010 --> 00:28:06,710
So in the future this will definitely be huge.

253
00:28:08,050 --> 00:28:11,530
L1 is a component I have running at the microkernel.

254
00:28:11,890 --> 00:28:20,810
The importance of this component is it's able to verify that the guest kernel and the L2 component are operating correctly.

255
00:28:21,550 --> 00:28:24,190
And similarly for L2 to L3.

256
00:28:24,270 --> 00:28:30,190
And why do I have these various levels in the system?

257
00:28:30,190 --> 00:28:43,930
Well it turns out if you're looking at the entire Linux operating system from L1, you're gonna have a hard time interpreting all the data structures, interpreting what's good, what's bad.

258
00:28:43,930 --> 00:28:53,670
So it's much easier to run components in each level in order to understand the semantics of that level.

259
00:28:54,770 --> 00:29:04,410
And the vertical hierarchy is a way of guaranteeing that those components are not trojaned or stopped or bypassed, these kinds of things.

260
00:29:07,250 --> 00:29:09,150
So there's a lot more details here.

261
00:29:09,150 --> 00:29:10,470
I just wanna give you a few.

262
00:29:10,470 --> 00:29:12,310
I'll just show you the memory real quick.

263
00:29:12,310 --> 00:29:14,530
How does the memory work for instance?

264
00:29:14,890 --> 00:29:22,190
So down here at the bottom, you have the microkernel L1 which owns all the memory.

265
00:29:24,510 --> 00:29:30,050
And it can allocate some of the physical, by memory I mean the RAM.

266
00:29:30,050 --> 00:29:34,710
It can allocate some of the RAM to the guest kernel which is shown in red here.

267
00:29:35,030 --> 00:29:42,090
And you can see that portions of the memory are still owned by the microkernel and the Linux guest kernel cannot touch that memory.

268
00:29:42,750 --> 00:29:50,710
And then in turn, the guest kernel, the Linux kernel can map some of its memory to user space processes.

269
00:29:53,050 --> 00:29:56,050
All right, so this is the system.

270
00:29:56,050 --> 00:29:57,290
How do we attack it?

271
00:29:57,290 --> 00:30:00,090
So you can attack any of these things in user space.

272
00:30:00,090 --> 00:30:05,370
If you have root access, you have access to all these components highlighted in red here.

273
00:30:06,070 --> 00:30:13,950
So again, the L2 and L3 components of the intrusion recovery system are suspect to attack.

274
00:30:14,070 --> 00:30:17,630
And that's why we need L1 to verify that they're operating correctly.

275
00:30:18,950 --> 00:30:26,350
So right now, I'm still working on algorithms to determine when L2 and L3 are operating correctly.

276
00:30:26,350 --> 00:30:40,230
So far, I only have monitoring the PC instruction counter and verifying the hash of the code at L2 to verify and check that the program counter runs in that code periodically.

277
00:30:41,130 --> 00:30:45,970
If anybody's got some ideas on other ways to verify, I'd definitely like to talk to you.

278
00:30:47,630 --> 00:30:52,650
Okay, so now I wanna look at another kernel-level rootkit target.

279
00:30:53,490 --> 00:31:01,110
So it turns out these same kernel-level rootkits will work on this new spine architecture.

280
00:31:01,310 --> 00:31:04,790
They will attack the guest kernel instead of the host kernel.

281
00:31:05,410 --> 00:31:11,890
So I wanna go through another target to give you guys some more background on how kernel-level rootkits work.

282
00:31:12,510 --> 00:31:15,870
This target is the virtual file system.

283
00:31:16,350 --> 00:31:26,090
Now, the virtual file system in Linux is a wrapper that gives you access to all file systems in Linux.

284
00:31:26,090 --> 00:31:28,790
And so you can insert your own virtual file systems.

285
00:31:28,790 --> 00:31:39,270
If you have an ext3 formatted partition, then you insert that in the VFS and you don't have to readjust your system calls to point to that.

286
00:31:39,270 --> 00:31:41,570
Everything goes through the virtual file system.

287
00:31:42,130 --> 00:31:43,670
Let me show you how it works.

288
00:31:45,070 --> 00:31:49,210
So first off, a system call is issued.

289
00:31:49,210 --> 00:31:53,450
Let's say I'll go over the proc file system in this example.

290
00:31:53,770 --> 00:31:55,610
Let me back up a minute.

291
00:31:56,110 --> 00:31:59,070
So you have lots of different file systems in Linux.

292
00:31:59,070 --> 00:32:02,550
You have your file system for your hard disk.

293
00:32:02,550 --> 00:32:07,990
You have remote disk storage stored on another network drive.

294
00:32:07,990 --> 00:32:20,590
You have the proc file system, which is a file system built by the kernel and gives you information from the kernel, such as task IDs, and so on and so forth.

295
00:32:21,510 --> 00:32:28,230
So if you want to, remember, if you wanna get access to the file system, any file system, you have to issue a system call.

296
00:32:29,210 --> 00:32:37,110
So in this example, I want to read the contents of the proc directory to see what processes are running on my system.

297
00:32:37,790 --> 00:32:41,670
So first, my program will issue a sys read dir.

298
00:32:42,750 --> 00:32:51,990
That sys read dir indexes into the virtual file system, which then, and you've told it to open up this proc, some directory in proc.

299
00:32:52,570 --> 00:32:59,430
So the virtual file system knows, based on the path you gave it, where it needs to go to service your request.

300
00:32:59,590 --> 00:33:01,810
For proc, that's a special file system.

301
00:33:01,810 --> 00:33:04,730
It indexes into its root structure.

302
00:33:05,570 --> 00:33:09,650
And there's a lot of pointers in that root structure.

303
00:33:09,810 --> 00:33:14,170
The F underscore op stands for file operations, I believe.

304
00:33:14,710 --> 00:33:22,390
And so for the proc, you look up what are the file operations for that file system.

305
00:33:22,850 --> 00:33:25,790
And so there's a F op structure, which is standard.

306
00:33:25,790 --> 00:33:30,630
And for each file system, you have to implement a number of function pointers.

307
00:33:30,630 --> 00:33:34,430
In this case, read dir is the one I'm going to highlight.

308
00:33:34,930 --> 00:33:37,950
So you've indexed all the way in here to read dir.

309
00:33:38,050 --> 00:33:42,310
And finally, you get to the proc read dir code.

310
00:33:43,150 --> 00:33:49,830
And so that'll then return information back to your system call, and then return that information back to your user program.

311
00:33:53,530 --> 00:33:56,370
Okay, so what's the attack points here?

312
00:33:56,370 --> 00:33:59,870
Again, it's the same basic idea.

313
00:33:59,950 --> 00:34:06,230
Anytime you've got a function pointer, anytime you've got a reference, any code along the path can be redirected.

314
00:34:11,760 --> 00:34:15,840
All right, so what's the recovery algorithm for this?

315
00:34:15,840 --> 00:34:19,620
Well, now's when we sort of step back and make it more general.

316
00:34:19,860 --> 00:34:28,700
So generally, the recovery methods for kernel-level rootkits is consistency checking on the data structures in the kernel and the code running.

317
00:34:28,700 --> 00:34:34,720
And you can store a copy of this, what it should look like in your protected spine architecture.

318
00:34:35,500 --> 00:34:44,280
And you can use that to check and see if it's been modified in a manner that's against your policy.

319
00:34:44,600 --> 00:34:49,320
And if so, then you repair it back to its known good state.

320
00:34:50,840 --> 00:34:53,800
And the IRS can do this automatically.

321
00:34:53,800 --> 00:35:05,800
And that's sort of the vision, is you have this system that's constantly monitoring your computer for malicious rootkit installations.

322
00:35:05,800 --> 00:35:09,960
If that happens, it undoes the damage from it.

323
00:35:09,960 --> 00:35:16,200
And furthermore, have longer-term goals of, okay , so yeah, you've got a rootkit installed.

324
00:35:16,200 --> 00:35:17,620
You undo this damage.

325
00:35:17,620 --> 00:35:19,740
The attacker still got in your system somehow.

326
00:35:20,300 --> 00:35:24,320
You also have to figure out how he got out, kick him out, change passwords.

327
00:35:24,320 --> 00:35:26,620
It's more than just repairing the system.

328
00:35:26,620 --> 00:35:27,860
It's a lot more complicated.

329
00:35:29,420 --> 00:35:31,560
All right, so let's go for another demo.

330
00:35:52,660 --> 00:36:02,920
Now, for the purposes of this demo, I'm not using the automated repair of the system because you wouldn't be able to see that anything happened.

331
00:36:05,490 --> 00:36:13,450
But one thing I wanna show you real quick is I am, in fact, running the spine architecture now.

332
00:36:16,380 --> 00:36:35,480
And you can see that I'm running the Linux 2.6.11-L4 kernel, which is, the way it works for the Linux kernel is you port the Linux kernel to the L4 architecture as opposed to the I386, the alpha, so on and so forth.

333
00:36:37,100 --> 00:36:44,980
So, in fact, we can go in here and check out our getcpl again to see if our privilege levels have changed.

334
00:36:45,960 --> 00:36:50,520
And so, the user processes still run at ring three, which is expected.

335
00:36:51,680 --> 00:36:55,220
And let's see what the kernel is now running at.

336
00:37:00,100 --> 00:37:02,860
And so, can everybody see this in the back?

337
00:37:02,860 --> 00:37:04,420
Or should I call it out for you guys?

338
00:37:05,640 --> 00:37:06,740
Move it up?

339
00:37:09,600 --> 00:37:10,920
Sorry about that.

340
00:37:13,320 --> 00:37:14,520
Is that better?

341
00:37:15,980 --> 00:37:16,860
Thanks.

342
00:37:17,400 --> 00:37:20,620
Okay, so what I've just done, let me scroll back up for you guys then.

343
00:37:22,000 --> 00:37:30,580
Um, so right here I just did the getcpl and it returned ring three, running in ring three.

344
00:37:31,620 --> 00:37:36,400
And I've just inserted the getcpl into the running kernel.

345
00:37:36,660 --> 00:37:44,500
And you can tell the var log, sys log, and see that it was printed out as running in ring three now.

346
00:37:44,500 --> 00:37:47,100
So now the kernel is running in ring three.

347
00:37:47,100 --> 00:37:53,600
If anybody out there is familiar with user mode Linux, it's a very similar concept to that, but it's much, much faster.

348
00:37:55,360 --> 00:37:58,440
In terms of performance, I'll take a slight venture.

349
00:37:58,560 --> 00:38:05,760
Just in case you guys are wondering, you figure about a 10% performance penalty for running in this architecture.

350
00:38:06,360 --> 00:38:11,900
So it runs about 90% the speed of Linux running directly on your hardware.

351
00:38:18,780 --> 00:38:28,000
Okay, so for the second demo, I'm going to install the Adore Next Generation rootkit.

352
00:38:28,000 --> 00:38:31,760
And this rootkit works on the Linux 2.6 kernel.

353
00:38:34,620 --> 00:38:40,860
And what this rootkit does is exactly what I was just showing you with the virtual file system.

354
00:38:40,860 --> 00:38:51,020
It redirects the virtual file system to malicious structures so it can hide files and processes.

355
00:38:57,710 --> 00:39:02,950
Okay, so I've just loaded the rootkit.

356
00:39:04,170 --> 00:39:10,990
Notice first off that my Adore NG directory that I was just in is now hidden.

357
00:39:10,990 --> 00:39:12,450
That's part of the configuration.

358
00:39:12,450 --> 00:39:15,550
It hides whatever your rootkit directory is.

359
00:39:16,090 --> 00:39:23,990
And then I'm going to launch this rootme binary, which doesn't really do anything, but it runs.

360
00:39:24,310 --> 00:39:28,330
And you can see right here in the process list that rootme is running.

361
00:39:29,430 --> 00:39:32,750
Now I've got to scroll back up here and see what my directory was.

362
00:39:34,150 --> 00:39:37,910
So I'm going to go back into the Adore NG directory.

363
00:39:37,910 --> 00:39:41,530
I'm tabbing over right now, and that's another system call.

364
00:39:41,530 --> 00:39:47,310
It doesn't find that there's an Adore NG directory, so I have to type the full path.

365
00:39:48,470 --> 00:39:56,010
And there's a user-level utility to go with this Adore NG rootkit, which can hide processes for you.

366
00:39:56,590 --> 00:40:00,210
So I want to hide this process 644.

367
00:40:09,250 --> 00:40:10,970
So now rootme's gone.

368
00:40:12,490 --> 00:40:22,690
All right, so now I'm going to put on my system administrator hat again, and I suspect that a rootkit's been installed.

369
00:40:23,330 --> 00:40:30,270
This could be done automatically with the intrusion recovery system, but again, for the purposes of this demo, I will do it manually.

370
00:40:32,830 --> 00:40:40,250
And so what I'm going to do is a similar algorithm to the system call table repair.

371
00:40:40,590 --> 00:40:44,170
I am going to, but this time I made it a little bit simpler.

372
00:40:44,170 --> 00:40:47,070
I just fixed the pointers back to where they're supposed to be.

373
00:40:52,560 --> 00:41:00,660
Okay, so I've just repaired the proc file system, and now you can see the rootme file, our process is back.

374
00:41:02,680 --> 00:41:06,360
And you can see the adore-ng directory is back.

375
00:41:06,820 --> 00:41:10,380
So in fact, we've been able to recover from these kernel-level rootkits.

376
00:41:15,210 --> 00:41:18,610
Okay, let's go back to wrap up things here.

377
00:41:31,600 --> 00:41:44,300
All right, so I've shown you the ability to recover from some kernel-level rootkits, and presented you with an architecture that's perhaps more robust, and you can somewhat rely on for this recovery.

378
00:41:45,160 --> 00:41:53,100
So, I mean, definitely there's some limitations to this system, I'm still developing it now, so any feedback you guys have would be great.

379
00:41:53,680 --> 00:41:58,120
Some of the limitations, you know, first off, how can you trust this microkernel?

380
00:41:58,120 --> 00:42:02,340
What if the attacker installs a rootkit into the microkernel?

381
00:42:02,340 --> 00:42:12,360
Well, you know, that's a hard question, but it's relying on the smallness of the microkernel to be correct to prevent that capability.

382
00:42:13,020 --> 00:42:23,920
You know, one of the things that some of you guys are probably thinking about already is, what about direct memory access, where you program a controller to overwrite the microkernel?

383
00:42:23,920 --> 00:42:28,880
Well, that's definitely a problem right now, that's a limitation.

384
00:42:29,020 --> 00:42:38,760
In future, Intel architectures that support better isolation, that hopefully will not be a problem.

385
00:42:39,260 --> 00:42:46,120
So, and finally, you know, just like everything else with security, there's no be-all, end-all solution.

386
00:42:46,320 --> 00:42:54,720
However, I believe this IRS can make your systems more reliable and save us a lot more time and trouble.

387
00:42:55,960 --> 00:43:01,800
So thanks to a lot of people that have helped me out, you know, there's lots more people on here than I have listed.

388
00:43:02,380 --> 00:43:05,000
These people in particular have been helpful.

389
00:43:06,140 --> 00:43:10,280
I got some links for you guys, if you wanna find out some more information about what I've been talking about.

390
00:43:10,600 --> 00:43:15,900
Various architectures, system projects here, you know, check rootkit, our research.

391
00:43:17,200 --> 00:43:23,920
And at this time, we got a few starter questions, if anybody wants to comment on, and if not, if you got your own questions, that's fine.

392
00:43:23,920 --> 00:43:26,280
We got about five minutes or so.

393
00:43:26,480 --> 00:43:30,500
And also after the talk, I will be heading out that door over there.

394
00:43:30,500 --> 00:43:34,720
So if you guys wanna catch me, definitely link up with me outside.

395
00:43:35,560 --> 00:43:43,880
So first off, let me just ask these three questions, and I'll take either answer these comments to these questions or your own questions.

396
00:43:44,240 --> 00:43:48,280
So I've already asked this question, how many people have personally dealt with the recovery of rootkit?

397
00:43:48,280 --> 00:43:51,880
If anybody has a testimony, we might can share that in a minute.

398
00:43:51,960 --> 00:43:54,660
Let me read all three of these and then I'll start calling on people.

399
00:43:55,260 --> 00:44:02,900
Secondly is, I would definitely be interested to see if anybody's seen any rootkits that use DMA, direct memory access.

400
00:44:02,900 --> 00:44:04,320
I've seen a lot of talk about it.

401
00:44:04,320 --> 00:44:10,480
Maybe I just have not seen the public talk about it or public implementation.

402
00:44:12,020 --> 00:44:22,260
And finally, if anybody has not followed the conventional wisdom of wiping their system clean and reinstalling, I'd be interested to hear that.

403
00:44:22,260 --> 00:44:25,260
So this time I'll open it up for questions and comments.

404
00:44:28,320 --> 00:44:29,540
Over there.

405
00:44:36,430 --> 00:44:41,170
Okay, so the question was with the L4 architecture, can you run multiple instances of your kernel?

406
00:44:41,170 --> 00:44:42,770
And the answer is yes.

407
00:44:42,770 --> 00:44:47,610
The L4 architecture is in fact pretty much virtual machine architecture.

408
00:44:47,610 --> 00:44:50,210
So you can run multiple instances of Linux.

409
00:44:54,510 --> 00:44:56,130
System pipes and log files?

410
00:44:56,130 --> 00:44:57,910
I have not addressed that yet.

411
00:44:58,650 --> 00:45:00,070
Good question.

412
00:45:07,100 --> 00:45:09,080
Certainly, but I mean,

413
00:45:12,320 --> 00:45:18,180
so it's suspicious, but that's, it's kind of like running a virtual machine, you know.

414
00:45:18,180 --> 00:45:20,240
You can tell when you're running on VMware.

415
00:45:20,240 --> 00:45:22,540
But, you know, legitimate systems run on there.

416
00:45:22,540 --> 00:45:26,420
So that definitely would be able to tell something that's not a normal system.

417
00:45:26,420 --> 00:45:27,240
Absolutely.

418
00:45:43,210 --> 00:45:43,850
Right.

419
00:45:43,850 --> 00:45:51,490
Actually, a lot of the tools that I've written for the system call stuff, it gives you all kinds of information on the system call table.

420
00:45:51,490 --> 00:45:57,250
So for instance, you can check and, you can check out all the, each individual system call with these tools.

421
00:45:57,250 --> 00:45:59,330
And that is much more surgical.

422
00:45:59,730 --> 00:46:00,770
Definitely.

423
00:46:01,350 --> 00:46:02,850
Good, good comment.

424
00:46:03,570 --> 00:46:04,730
Question here.

425
00:46:27,920 --> 00:46:28,940
Hmm.

426
00:46:28,940 --> 00:46:35,040
So the question was, one, this system would be useful just for detection, correct?

427
00:46:35,220 --> 00:46:36,600
And absolutely.

428
00:46:36,880 --> 00:46:39,920
Why am I doing intrusion recovery system as well?

429
00:46:39,920 --> 00:46:43,820
Because I'm a research student and we gotta do new and exciting things.

430
00:46:45,420 --> 00:46:46,540
So absolutely.

431
00:46:46,540 --> 00:46:50,020
And lots of people have talked about using similar architectures for intrusion detection.

432
00:46:50,020 --> 00:46:51,120
That's an excellent idea.

433
00:46:51,120 --> 00:46:53,620
That'll be the way it works 10 years from now.

434
00:46:53,920 --> 00:46:55,420
Got guys in the back?

435
00:46:55,520 --> 00:46:56,540
People over here?

436
00:47:01,920 --> 00:47:03,420
Oh yeah, absolutely.

437
00:47:03,660 --> 00:47:08,800
Matter of fact, I was gonna do a demo on that for this presentation, but didn't have enough time.

438
00:47:09,160 --> 00:47:12,200
Recovery from user level rootkits, I've done some work on that.

439
00:47:12,300 --> 00:47:23,040
It's pretty much tying a tripwire or aid kind of thing to a recovery program, which just checks your file system periodically and replaces the binaries back to normal.

440
00:47:23,040 --> 00:47:24,680
That's absolutely a part of the system.

441
00:47:24,800 --> 00:47:26,600
Cause you need the full system.

442
00:47:26,860 --> 00:47:29,560
We got a question way back there in the back and then I'll come back up here.

443
00:47:35,700 --> 00:47:37,240
That's an excellent question.

444
00:47:37,240 --> 00:47:40,860
So Xen is another very similar architecture.

445
00:47:40,860 --> 00:47:42,920
In fact, I got a link to that in the slides.

446
00:47:43,040 --> 00:47:44,840
Xen is very hot right now.

447
00:47:44,840 --> 00:47:49,340
It's a virtual machine architecture, open source.

448
00:47:49,380 --> 00:47:51,800
I believe it's England people are doing that.

449
00:47:51,820 --> 00:47:53,460
And it's very fast.

450
00:47:53,460 --> 00:47:56,320
Matter of fact, so Xen and L4 compete.

451
00:47:56,580 --> 00:48:01,760
I am going to compare Xen to L4 in my thesis to see which one would be better.

452
00:48:02,560 --> 00:48:04,140
That's just an implementation difference.

453
00:48:04,140 --> 00:48:05,760
Either one is certainly suitable.

454
00:48:05,760 --> 00:48:07,420
I just chose L4.

455
00:48:07,760 --> 00:48:11,500
We had a question up here and then I'll quit after this guy.

456
00:48:19,820 --> 00:48:23,280
Okay, so the question is, what's more background on this microkernel?

457
00:48:23,500 --> 00:48:35,240
And the reason I chose L4 fiasco implementation is basically it's one of the more mature L4 implementations developed by a university at Dresden.

458
00:48:35,520 --> 00:48:39,080
And the team there has been really good at helping me out.

459
00:48:39,080 --> 00:48:45,240
You know, any kind of questions I've got, big team there, lots of questions, you know, they've got the mailing list and everything.

460
00:48:46,680 --> 00:48:51,560
Comparing it to other, there's also two or three other implementations I could have used.

461
00:48:51,680 --> 00:48:55,860
I happen to know the guys a little bit at Dresden, so that's really the only reason.

462
00:48:55,940 --> 00:48:56,720
It's a good question.

463
00:48:56,860 --> 00:48:59,880
And okay, so I'm going to wrap up and let the next speaker get in.

464
00:48:59,880 --> 00:49:02,560
I'll be out here in the hall if you guys got any more questions.

465
00:49:02,980 --> 00:49:03,240
Thank you.