WEBVTT 00:00.000 --> 00:01.140 So hello, everyone. 00:01.140 --> 00:02.180 How are you doing? 00:02.180 --> 00:02.680 I'm Nidal. 00:02.680 --> 00:05.020 I'm the head of the Data Duplication Village. 00:05.360 --> 00:08.680 We have a repeat speaker here. 00:08.680 --> 00:09.800 This is Andy Klein. 00:09.800 --> 00:18.040 He is the individual who has come to us several times, three, four times now, to the Data Duplication Village and spoken for us. 00:18.100 --> 00:25.260 He works for a company that does hundreds, if not thousands, if not tens of thousands of hard drives. 00:25.900 --> 00:26.600 More. 00:26.740 --> 00:27.860 Every year. 00:28.320 --> 00:35.400 And he's come to talk to us about those hard drives, their reliability, and how they compare with SSDs and a number of different things. 00:35.400 --> 00:38.900 But instead of giving away everything, I'm going to introduce you to Andy Klein. 00:38.960 --> 00:40.520 Andy, thank you for joining us. 00:41.560 --> 00:42.780 Hey, thank you, Scott. 00:42.780 --> 00:44.080 I appreciate it. 00:44.220 --> 00:53.560 And like you said, for the last few years, I've been coming out and talking to the folks at the Data Duplication Village about hard drives and SSDs and so on and so forth. 00:53.840 --> 00:57.260 And let me give you a little idea of what we'll talk about today. 00:57.820 --> 01:03.320 And by the way, any time you have a question, please just stick your hand up in the air and we can get that out of the way. 01:04.180 --> 01:04.900 I'm old. 01:04.900 --> 01:06.220 You don't need to know that. 01:07.200 --> 01:10.100 We're going to talk a little bit about the data we collect. 01:10.760 --> 01:12.140 It's about hard drives. 01:12.140 --> 01:16.000 It's hard drive failure data, hard drive capabilities, and so on and so forth. 01:16.260 --> 01:18.180 We'll talk about failure rates. 01:18.200 --> 01:20.620 I'm going to do SSDs for a little bit. 01:20.660 --> 01:23.300 They're kind of popular these days I've heard. 01:23.760 --> 01:31.040 And then I have a bunch of other oddball statistics that I'll just throw in there about hard drives and things that we found out. 01:31.180 --> 01:34.280 And then finally, we're going to put all of this in action. 01:34.280 --> 01:37.860 In other words, how we make use of it moving forward. 01:37.860 --> 01:40.520 So, kind of fun to do. 01:41.140 --> 01:41.640 All right. 01:41.640 --> 01:44.620 So, a little bit about the environment. 01:44.780 --> 01:45.520 All right? 01:45.540 --> 01:50.500 Right now, we're currently monitoring about 250,000 or so hard drives. 01:50.500 --> 01:52.840 So, that's not a bad day at the office, right? 01:53.500 --> 01:57.460 About three exabytes of data that we're dealing with. 01:57.460 --> 02:01.340 We've been doing this for the last ten years. 02:01.440 --> 02:02.880 Hence the title, right? 02:02.880 --> 02:05.120 So, we started back in 2013. 02:05.340 --> 02:07.300 We record the data every day. 02:07.300 --> 02:09.160 You'll see that in just a second. 02:09.160 --> 02:17.340 And then we can analyze the data and figure out what kinds of patterns we might see in the information that were there. 02:17.340 --> 02:19.740 So, I forgot to bring my hammer. 02:20.760 --> 02:23.400 We have a bunch of different data centers there. 02:23.400 --> 02:27.520 The fun one is the data center on a boat. 02:27.520 --> 02:28.200 Excuse me. 02:28.200 --> 02:29.380 A barge. 02:30.200 --> 02:32.900 So, this is actually kind of cool. 02:32.900 --> 02:34.260 I just brought this along. 02:34.260 --> 02:37.400 It's got nothing to do with the fact except it's full of hard drives. 02:37.960 --> 02:40.640 But it's actually water cooled. 02:40.720 --> 02:44.480 There's no freezers or anything like that in there. 02:45.040 --> 02:51.180 It sits on the San Joaquin River in Stockton, California, which is kind of nice. 02:51.960 --> 02:53.900 Saves us a bunch of electricity. 02:53.900 --> 02:55.520 Doesn't kill fish. 02:55.860 --> 02:57.720 And so on and so forth. 02:57.840 --> 02:58.260 Okay? 02:58.260 --> 02:59.600 Which is all cool. 02:59.920 --> 03:03.940 But this is where things need to go in general. 03:03.940 --> 03:04.420 Right? 03:04.540 --> 03:09.000 Data centers like the guys we are doing use a lot of electricity. 03:09.700 --> 03:12.720 Do a lot of interesting things to the environment. 03:12.840 --> 03:14.700 This is our first step . 03:14.900 --> 03:17.540 So, it's one of the nice things that we're doing around here. 03:17.880 --> 03:18.200 All right. 03:18.200 --> 03:19.700 So, let's get to the data. 03:19.700 --> 03:20.380 Okay? 03:20.400 --> 03:23.400 Again, I said we've been storing it since 2013. 03:23.700 --> 03:25.840 We use a package called smart run tools. 03:25.840 --> 03:30.720 It just takes something called smart stats off of every single drive. 03:30.720 --> 03:38.460 Every hard drive, every SSD out there for the last 20 some 30 some odd years produces these statistics. 03:38.700 --> 03:41.740 And then this package will read that. 03:41.740 --> 03:42.180 Okay? 03:42.180 --> 03:46.460 What we did is those statistics and I'll show you some of them a little bit later. 03:46.460 --> 03:47.720 All right? 03:47.880 --> 03:52.720 Every day we snap the statistics off of every single drive. 03:52.720 --> 03:54.460 And that's what we've been keeping. 03:55.180 --> 03:55.580 All right? 03:55.580 --> 04:00.620 And it gives us the ability to track a drive and its behavior. 04:00.620 --> 04:04.140 So, for example, one of the statistics is temperature. 04:04.280 --> 04:08.320 I can monitor the temperature of every single drive I have. 04:08.320 --> 04:10.320 All 250,000 of them. 04:10.320 --> 04:10.780 Okay? 04:10.780 --> 04:13.980 I can tell when something's getting interesting inside there. 04:13.980 --> 04:18.380 I can monitor the media inside of every single drive. 04:18.380 --> 04:23.700 And I can understand that we're having trouble failing in writing in a certain area. 04:23.700 --> 04:25.560 So, we can do all of that kind of stuff. 04:25.560 --> 04:26.820 And that's what we've been doing. 04:27.300 --> 04:27.700 All right? 04:27.700 --> 04:30.300 The data is open source. 04:30.940 --> 04:33.980 So, you can go and play with this if you choose to. 04:33.980 --> 04:34.600 All right? 04:34.600 --> 04:36.500 It's a bit to download. 04:36.760 --> 04:39.100 But you're more than welcome to do so. 04:39.100 --> 04:40.140 And people do. 04:41.060 --> 04:43.880 Interestingly enough, the drive manufacturers do. 04:44.840 --> 04:46.980 And you'll see why in a minute. 04:47.800 --> 04:52.760 So, what does it look like just to be just to kind of get everybody on the same page? 04:52.960 --> 04:55.440 You can see I lost my colors. 04:55.440 --> 04:56.200 Oh, my goodness. 04:56.200 --> 04:57.060 It's horrible. 04:58.640 --> 05:01.220 The so, here's the data. 05:01.220 --> 05:02.340 It's just laid out, right? 05:02.340 --> 05:03.840 Row, data with columns. 05:03.840 --> 05:04.720 It's no big deal. 05:04.720 --> 05:05.980 Nothing spectacular. 05:06.460 --> 05:08.760 You can see smart one normalized. 05:08.760 --> 05:13.340 You get a normalized value and a raw value for every smart statistic out there. 05:13.480 --> 05:16.660 There are potentially 255 of them. 05:16.660 --> 05:18.240 Pairs like that. 05:19.040 --> 05:22.740 No more any given drive doesn't use any more than 40. 05:23.100 --> 05:25.000 And that's a pretty high number. 05:25.020 --> 05:27.440 And we'll look at some of them a little bit later. 05:27.440 --> 05:30.300 Like smart one is a read error rate. 05:30.300 --> 05:32.320 Smart five is reallocated sectors. 05:32.320 --> 05:34.060 They'll start to be interesting. 05:34.540 --> 05:36.780 Smart nine is power on hours. 05:36.780 --> 05:38.360 How much that drive has been running. 05:38.360 --> 05:40.000 All those kinds of things. 05:40.220 --> 05:40.840 Right? 05:41.040 --> 05:45.800 Now, the data there, you can see there's a failure in one of the columns. 05:45.800 --> 05:48.060 That's how the data marks a failure. 05:48.380 --> 05:50.300 That drive failed that day. 05:50.300 --> 05:52.120 It was it's gone. 05:52.120 --> 05:53.800 It won't be there tomorrow. 05:53.800 --> 05:54.660 All right? 05:54.660 --> 05:56.540 And that's how we know. 05:56.540 --> 06:05.860 We know that that particular drive, that particular model, SD800, so on and so forth, okay, failed that day. 06:06.440 --> 06:11.840 Now, each row for each day is what we call a drive day. 06:12.080 --> 06:18.460 And that's important to understand when we start to do aggregation and we start to figure out failure rates. 06:19.820 --> 06:20.760 All right? 06:20.760 --> 06:24.580 Speaking of failure, so what is a failure? 06:24.900 --> 06:25.540 All right? 06:25.540 --> 06:29.020 Now, most people would say my drive failed when it stops running. 06:29.020 --> 06:31.420 That's a failure, right? 06:31.740 --> 06:33.120 We have two states. 06:33.120 --> 06:37.100 We have what we call a reactive failure, which is exactly that. 06:37.100 --> 06:38.260 I can't talk to it. 06:38.260 --> 06:39.540 It doesn't give me data back. 06:39.620 --> 06:41.060 It's broke. 06:41.200 --> 06:41.680 Okay? 06:41.680 --> 06:42.920 For all of that. 06:42.920 --> 06:45.580 And then we have something called a proactive failure. 06:45.580 --> 06:45.980 Okay? 06:45.980 --> 06:49.180 We're monitoring these on a regular basis. 06:49.340 --> 06:54.280 You might get what's called ATA errors coming at you, which are electrical type errors. 06:54.280 --> 06:57.940 You might fail an FS check, a file system check. 06:58.220 --> 07:02.100 Or combine that with the smart data that we were just talking about. 07:02.100 --> 07:03.720 All those various statistics. 07:03.800 --> 07:07.320 And you go, no, that drive is on its way to failure. 07:07.540 --> 07:10.000 And I think I need to get it out of there. 07:10.280 --> 07:11.840 Now, why would we do that? 07:11.840 --> 07:13.140 Why would we care? 07:13.580 --> 07:14.060 Right? 07:14.060 --> 07:23.500 The reason we care, one of the primary reasons is if I can take a drive out before it rolls over and dies, I can clone it. 07:24.040 --> 07:26.620 Cloning works really well right about here. 07:26.620 --> 07:30.940 You get about a 95% success rate to clone a drive. 07:31.060 --> 07:32.660 And why is cloning important? 07:32.940 --> 07:36.840 Because you know how long it takes to rebuild the drive when you put it in? 07:37.160 --> 07:38.660 Like a rate array. 07:38.660 --> 07:40.100 And you pull one out. 07:40.500 --> 07:42.960 Not bad with a 1 terabyte drive. 07:42.980 --> 07:44.580 Maybe it's a day or two. 07:44.840 --> 07:47.100 Gets kind of, okay. 07:47.100 --> 07:51.560 With a 4 terabyte drive, because that certainly gets to be 4 or 5 days or 6 days. 07:51.980 --> 07:55.500 We're dealing with 116 terabyte drives right now. 07:55.500 --> 07:57.220 And that can take weeks. 07:58.000 --> 08:00.240 So, I don't want weeks. 08:00.240 --> 08:08.120 I can clone it in a day and a half and have it back in the array and running and keep up what's called durability on the whole system. 08:08.120 --> 08:08.720 Right? 08:08.720 --> 08:13.980 So, you can start to see the data we collect and the decisions we make. 08:13.980 --> 08:19.720 But either one of those will produce that little one in that column and that drive is gone. 08:19.720 --> 08:20.200 Okay? 08:22.140 --> 08:26.800 Now, how do we compute this thing called annualized failure rate? 08:27.120 --> 08:28.900 Which is what we publish. 08:28.900 --> 08:31.900 You'll see that number as we go through the thing. 08:32.320 --> 08:32.940 Right? 08:32.940 --> 08:34.400 We take a group of drives. 08:34.400 --> 08:35.640 A cohort of drives. 08:35.640 --> 08:40.500 In this case, all of the models that were active as of 6-30-2023. 08:40.500 --> 08:41.000 Okay? 08:41.000 --> 08:42.680 All of the different drive models. 08:42.840 --> 08:44.500 And we give it a period. 08:44.900 --> 08:45.880 Q2. 08:46.480 --> 08:50.280 And then we just obtain the numbers for those things and we do some math. 08:50.700 --> 08:51.980 Drive days. 08:52.100 --> 08:56.040 Number of drives running each day over that period. 08:56.100 --> 08:56.660 Failures. 08:56.660 --> 08:57.680 How many failures. 08:57.680 --> 08:59.060 These are real numbers, by the way. 08:59.060 --> 09:00.660 I'm not making these up. 09:00.760 --> 09:01.300 Okay? 09:01.300 --> 09:06.960 And at the end of Q3, Q2, excuse me, we had 240,000 drives we were looking at. 09:07.260 --> 09:07.920 Right? 09:08.400 --> 09:14.300 And then we do a little math there and we come up with a number and it says, that's our annualized failure rate. 09:14.300 --> 09:17.720 And then we publish that number and we hand people the data. 09:18.600 --> 09:19.080 Right? 09:19.080 --> 09:20.360 And say, here. 09:20.400 --> 09:25.780 Now, sometimes people say, hey, why don't you use drive count? 09:25.780 --> 09:27.400 Because that's what I would do. 09:27.740 --> 09:29.540 I had ten drives, one failed. 09:29.540 --> 09:31.600 I have a ten percent failure rate. 09:31.680 --> 09:32.940 We can't do that. 09:32.940 --> 09:39.100 Because that makes the assumption that you had ten drives over the entire period you were looking at. 09:39.480 --> 09:40.320 We don't. 09:40.320 --> 09:43.780 We're constantly adding drives in and taking drives out. 09:43.780 --> 09:44.140 You know? 09:44.140 --> 09:47.840 So, we have to use drive days instead of drive count. 09:48.280 --> 09:56.520 I tried publishing the reports one time without drive count and everybody yelled at me because they really want that number. 09:56.660 --> 09:57.280 Okay? 09:57.620 --> 09:59.360 But it's irrelevant. 10:02.660 --> 10:04.460 So, what do we do with the data? 10:04.460 --> 10:06.340 We put it in some quarterly reports. 10:06.340 --> 10:07.900 Some of you may have seen it. 10:07.900 --> 10:09.320 We do it for hard drives. 10:09.320 --> 10:12.200 We do it for SSDs, which we'll talk about in just a minute. 10:12.940 --> 10:15.560 We give you the data for the most recent quarter. 10:15.560 --> 10:20.900 We give you the data on an annual basis and a lifetime basis for all of the different drives that are out there. 10:20.900 --> 10:22.020 It's kind of useful. 10:22.020 --> 10:28.860 If you want to go there, go to the blog, type search for drive stats, you'll get effectively eight years' worth of reports. 10:29.820 --> 10:32.220 And you can see what the heck this is all about. 10:32.220 --> 10:33.220 It's free. 10:33.220 --> 10:34.400 We don't charge you. 10:34.400 --> 10:37.460 We don't track you or any of that silliness. 10:37.460 --> 10:40.540 If you would like to get a newsletter, you can join in. 10:41.040 --> 10:42.960 Sorry, that's what the marketing people do. 10:42.960 --> 10:44.340 But that's about it. 10:44.760 --> 10:45.380 Okay? 10:45.720 --> 10:48.280 So, a report kind of looks like this. 10:49.000 --> 10:49.540 Okay? 10:49.540 --> 10:50.680 It's a table. 10:50.720 --> 10:53.700 We write some fancy words around it and make it look pretty. 10:53.900 --> 10:56.440 I pulled out a couple of things in here. 10:56.440 --> 11:00.360 Because as people start to look at this, I want you to understand what you're looking at. 11:00.360 --> 11:01.000 Right? 11:01.820 --> 11:06.540 One of them, the first one there, the HGST one, right? 11:06.540 --> 11:08.940 It's got a 13% failure rate. 11:08.940 --> 11:10.420 And people go, that's terrible. 11:10.420 --> 11:12.180 I would never want one of them. 11:12.380 --> 11:12.820 Okay? 11:12.820 --> 11:14.600 And that's not a good conclusion. 11:14.600 --> 11:20.060 Because there's only a total of 8,000 drive days. 11:20.420 --> 11:20.800 Right? 11:20.800 --> 11:22.620 That's not a lot over a quarter. 11:22.620 --> 11:27.560 You really want 50,000, 60, 70,000, start to get a number that feels good. 11:27.560 --> 11:28.220 All right? 11:28.220 --> 11:30.160 So, not much there. 11:30.840 --> 11:32.980 I highlighted the Seagate one. 11:32.980 --> 11:35.380 Because it's our old timer. 11:36.120 --> 11:42.160 And the data duplication village guys have been using the 6 terabyte drives for the last few years. 11:42.160 --> 11:44.180 Or asking you to bring them in. 11:44.180 --> 11:46.120 As the case may be. 11:46.420 --> 11:49.740 And that happens to be look how old that is. 11:49.740 --> 11:51.100 98 months. 11:51.180 --> 11:52.340 That's not bad. 11:52.340 --> 11:53.060 Right? 11:53.300 --> 11:55.120 8 times 12, 96. 11:55.120 --> 11:58.980 Little over 8 years on average in a data center. 11:59.220 --> 11:59.740 Okay? 11:59.740 --> 12:01.560 Being beaten every day. 12:02.320 --> 12:03.120 All right? 12:03.420 --> 12:05.080 A good little drive. 12:05.720 --> 12:07.320 It does bring up one point. 12:07.320 --> 12:08.340 And I want to point this out. 12:08.340 --> 12:11.000 If you ever do look at these reports, okay? 12:11.000 --> 12:17.380 And you see that model number, that ST6000DX000, right? 12:17.660 --> 12:20.280 And you can go, wow, that sounds pretty good. 12:20.280 --> 12:21.940 I'm going to go get me one of them. 12:21.940 --> 12:22.660 Right? 12:23.060 --> 12:29.740 And you go to the store and it's a ST6000DX005. 12:29.980 --> 12:31.580 They're not the same. 12:32.900 --> 12:33.500 Okay? 12:33.500 --> 12:34.760 Different model. 12:34.760 --> 12:35.120 Okay? 12:35.120 --> 12:36.800 They may look the same. 12:36.800 --> 12:39.600 They may have the same specs and everything like that. 12:39.600 --> 12:44.520 There's a reason that Seagate and all of the other manufacturers change the model numbers. 12:44.640 --> 12:46.600 And they don't tell us. 12:47.180 --> 12:52.560 So, if it's on there and it's a certain number, you like it, great. 12:52.880 --> 12:53.300 Okay? 12:53.300 --> 12:59.560 But if you can't find it, the next model over, okay? 12:59.560 --> 13:01.160 Your call on that. 13:01.460 --> 13:04.120 In other words, the models don't transfer. 13:04.120 --> 13:05.960 It doesn't go from one to the other. 13:06.880 --> 13:11.280 So, in the most recent quarter, we had a total of 31 different models. 13:11.620 --> 13:13.980 These are the ones we're using in our data center. 13:13.980 --> 13:14.900 These are active. 13:14.900 --> 13:18.140 They are drives that we are storing customer data on. 13:18.140 --> 13:18.780 Okay? 13:18.780 --> 13:20.880 So, they are in use. 13:21.540 --> 13:24.340 And they're treated all the same. 13:24.700 --> 13:26.900 They're all given air conditioning every day. 13:26.900 --> 13:27.400 Right? 13:27.400 --> 13:30.220 There's a guy who comes out and dusts every once in a while. 13:30.220 --> 13:30.900 No. 13:32.800 --> 13:36.140 Another chart you'll see is what's called lifetime. 13:36.360 --> 13:38.000 That previous one was quarterly. 13:38.000 --> 13:39.160 This is lifetime. 13:39.160 --> 13:44.840 This is a whole lot more accurate for the whole drive, for the picture of the whole drive, right? 13:45.400 --> 13:47.520 A couple of fun ones there. 13:48.200 --> 13:50.060 There's that HGST one. 13:50.320 --> 13:56.340 There you can begin to see this thing called a confidence interval, all the way in that last two columns there. 13:56.820 --> 13:59.440 That is the it's a statistical thing. 13:59.440 --> 14:00.940 You don't have to care what it is. 14:00.940 --> 14:07.020 All you have to care about is the delta between those two numbers, .5 or less. 14:07.340 --> 14:09.240 Maybe .6 or less. 14:09.420 --> 14:09.880 Okay? 14:09.880 --> 14:16.900 If it is, you got a pretty solid number in that AFR column, annualized failure rate column. 14:16.900 --> 14:19.140 If it's not, you don't. 14:19.380 --> 14:20.740 And it's that simple. 14:20.960 --> 14:25.200 So, the first one, the HGST, that difference is ten. 14:25.440 --> 14:27.360 Not .5, ten. 14:27.360 --> 14:27.820 Okay? 14:27.820 --> 14:31.800 Which means that AFR 656 is, eh. 14:32.240 --> 14:32.800 Okay? 14:32.800 --> 14:33.780 We just did some math. 14:33.780 --> 14:34.240 That's all. 14:34.240 --> 14:35.100 All we did. 14:35.100 --> 14:35.740 Okay? 14:35.860 --> 14:37.860 Statistically, that's not a valid number. 14:37.860 --> 14:39.400 We publish it, but it's not. 14:39.400 --> 14:42.300 The one right below it, .2. 14:42.600 --> 14:43.400 Okay. 14:43.640 --> 14:44.200 All right. 14:44.200 --> 14:48.420 That failure rate of .59% for that drive is pretty solid. 14:49.100 --> 14:49.720 All right? 14:49.720 --> 14:53.480 You can probably look at that and go, that fails at less than 1%. 14:53.480 --> 14:54.900 That's a cool drive. 14:54.900 --> 14:56.000 I like that. 14:56.040 --> 14:56.700 Right? 14:57.560 --> 15:01.180 You can look at the one, the last one there for Seagate. 15:01.220 --> 15:03.160 Oh, my goodness, look, no drive failures. 15:03.160 --> 15:04.080 That's great. 15:04.140 --> 15:05.740 Yeah, it's great. 15:05.740 --> 15:08.780 But, again, a pretty wide number there. 15:08.780 --> 15:11.200 So, it doesn't have enough history. 15:11.500 --> 15:19.780 And I tell you this because people look at these numbers all the time and then they go, you know, I want you to, you know, they complain. 15:20.040 --> 15:22.640 And the confidence interval is there for a purpose. 15:22.700 --> 15:23.740 So, take a look at it. 15:23.740 --> 15:24.560 Use it. 15:24.560 --> 15:26.260 Make it part of it. 15:26.460 --> 15:33.040 Now, one of the things that people talk about is, you know, what can I use this for? 15:33.420 --> 15:33.980 All right? 15:33.980 --> 15:35.320 The data itself. 15:35.320 --> 15:35.900 Okay? 15:35.900 --> 15:37.220 That we published. 15:37.260 --> 15:42.100 And I always tell them, it's entirely up to you, obviously, what you want to do with it. 15:42.180 --> 15:45.000 But there's nothing out there like this. 15:45.520 --> 15:49.160 The only other data you get is manufacturer's data. 15:49.300 --> 15:57.660 And every manufacturer I've ever seen either has an annualized failure rate of .9 or 1%, depending on how they feel. 15:58.440 --> 15:58.780 Okay? 15:58.780 --> 16:01.480 And it's never higher and it's never lower. 16:01.980 --> 16:02.700 All right? 16:03.260 --> 16:06.820 So, this is a little bit helpful in trying to talk to you. 16:06.820 --> 16:12.180 Now, you can make all of the arguments about, hey, different environment, all of that kind of cool stuff, right? 16:12.760 --> 16:13.360 Right? 16:13.360 --> 16:15.240 We do treat these things nice . 16:15.240 --> 16:18.420 We don't put them in an external case and stuff them under the bed. 16:18.480 --> 16:19.160 Okay? 16:19.600 --> 16:24.020 You know, so, but they do a lot of work for us. 16:25.060 --> 16:25.820 All right. 16:25.920 --> 16:35.340 Just to make your life a little easier, I put together a list of things that we call, you know, gold medal winners. 16:35.400 --> 16:40.200 And I'm going to actually start from the bottom of the list just to kind of show you some things here. 16:40.320 --> 16:44.160 The bottom three, the ones with the little three there are friends. 16:45.140 --> 16:48.460 Those drives are those are really great numbers. 16:48.560 --> 16:49.300 Okay? 16:49.740 --> 16:51.100 Excuse me. 17:00.080 --> 17:02.320 Those are really great numbers. 17:02.320 --> 17:04.200 But you can't buy them. 17:04.240 --> 17:05.860 You can't buy them new. 17:05.860 --> 17:07.400 You can buy rehabs. 17:08.200 --> 17:10.180 Rehabs all bets are off. 17:10.700 --> 17:11.780 Okay? 17:12.120 --> 17:20.760 So, and if somebody's trying to sell you one of those as new, they must have had them in a warehouse somewhere for the last eight years. 17:21.420 --> 17:26.040 You go up a little bit to the 12 terabyte drives. 17:26.100 --> 17:28.620 You can kind of see the numbers there. 17:28.620 --> 17:30.520 You can still buy all of those. 17:31.560 --> 17:36.000 But two of them are from HGSTs. 17:42.110 --> 17:46.790 The two from HGST are actually rebranded as Western Digital now. 17:47.590 --> 17:48.710 Okay? 17:49.030 --> 17:55.050 Western Digital bought a chunk of the HGST business five or six years ago. 17:55.050 --> 17:56.590 Seven or eight years ago. 17:57.450 --> 18:00.690 The top ones are all pretty legit. 18:00.810 --> 18:03.630 Fourteens and sixteens. 18:03.630 --> 18:08.010 I haven't seen that very top one available for sale in the US. 18:08.070 --> 18:11.450 So, maybe you want to go down to the second one. 18:11.790 --> 18:12.370 Okay? 18:13.130 --> 18:15.210 But, those are solid drives. 18:15.210 --> 18:16.930 They've been performing for us. 18:16.950 --> 18:19.690 They got more than two million drive days. 18:19.690 --> 18:20.990 So, they've been working. 18:22.010 --> 18:27.530 Um, you know, make sure, whatever choice, use the data however you want to help make decisions. 18:28.450 --> 18:29.170 Alright. 18:29.470 --> 18:31.370 Uh, a little bit about SSDs. 18:31.370 --> 18:38.450 So, last year I talked about this with, um, at the group. 18:38.930 --> 18:45.850 And, um, we, we had this question about SSDs and reliability versus HDDs. 18:46.210 --> 18:47.030 Right? 18:47.030 --> 18:50.350 Everybody said SSDs are much better. 18:50.350 --> 18:53.430 But nobody had any proof. 18:53.630 --> 18:54.190 Right? 18:54.190 --> 18:58.950 So, so we went and we started to take a look and see what we could do. 18:58.970 --> 19:01.770 But you always want to compare apples to apples. 19:01.770 --> 19:06.870 So, the only place we had SSDs were as boot drives. 19:07.570 --> 19:11.390 But we also had HDDs, hard drives, as boot drives. 19:11.390 --> 19:12.130 Right? 19:12.370 --> 19:14.630 So, we wanted to compare them. 19:14.630 --> 19:22.770 Now, boot drives you might think of as turn on the machine, spins about 30 seconds or whatever, and then it just sits there and does nothing. 19:23.050 --> 19:24.230 Ours don't do that. 19:24.230 --> 19:27.830 Ours are actually used for daily, for temporary storage. 19:27.870 --> 19:34.030 Um, the drive stacks actually get downloaded onto them and then get offloaded about once a week. 19:34.230 --> 19:36.510 Um, so these are drives that are working. 19:36.870 --> 19:37.730 Right? 19:37.730 --> 19:43.130 Um, they, they are reading and writing on a regular basis and deleting files and all of that kind of stuff. 19:43.130 --> 19:50.290 And you can see the counts aren't spectacular, um, but it's about 4,000 drives we did the analysis with. 19:51.450 --> 19:54.370 And last year I presented this chart. 19:55.450 --> 19:59.650 And I asked the question, I says, well, what are we gonna do? 19:59.650 --> 20:16.770 So, what we had done here, let me give you a little background, is we tracked the failure rates over time for the red, which was the hard drives, remember, the boot drives, um, and the SSDs in the first year they were in existence, the second year, and so on and so forth, 20:16.770 --> 20:17.750 as they aged. 20:17.750 --> 20:18.430 Okay? 20:18.870 --> 20:24.550 And we got to year four and we didn't have any more data for the SSDs because they had only been around about four years. 20:24.790 --> 20:27.150 So we said, well, what do you think's gonna happen? 20:27.150 --> 20:30.490 Because they kinda tracked pretty closely together there, right? 20:30.730 --> 20:32.930 We said, do you think it's gonna go up? 20:33.410 --> 20:34.270 Follow it up? 20:34.270 --> 20:36.650 Do you think it's gonna be, meh, in the middle? 20:36.650 --> 20:38.630 Or maybe it's gonna level out. 20:38.870 --> 20:39.270 Right? 20:39.650 --> 20:44.170 And the answer, you know, and we had people guessing and everything like that. 20:44.170 --> 20:48.130 And the actual answer is, they were all wrong, they went down. 20:49.390 --> 20:50.110 Okay? 20:50.450 --> 21:01.050 Um, the SSDs, and I will tell you, I'm getting ready to do the next report, uh, for SSDs, and the number continues to be that low. 21:01.050 --> 21:06.350 So, SSDs do in fact seem to be more reliable than HDDs. 21:07.010 --> 21:15.190 Which everybody said it would, but at least we have a little bit of math now, uh, in a practical environment, um, to, to put it to rest. 21:15.190 --> 21:25.810 Now, at some point, we'll put SSDs in the data center, we'll start using them, we'll compare them and all of that, but that's, right now it doesn't make it sense from a cost perspective to do that. 21:25.810 --> 21:28.190 So, SSD is pretty solid though. 21:28.830 --> 21:29.470 Yeah. 21:29.510 --> 21:30.330 Okay? 21:30.410 --> 21:32.370 So you can see the numbers right there. 21:33.630 --> 21:39.330 The other thing we looked at over the last few months is the smart stats, remember? 21:39.330 --> 21:44.770 The statistics we catch for each, we capture for each of the hard drives. 21:44.930 --> 21:47.850 Tell us things like media, temperature, and so on. 21:47.870 --> 21:50.010 Well, SSDs do the same thing. 21:50.010 --> 21:51.710 They report that data. 21:51.890 --> 21:52.290 Okay? 21:52.290 --> 21:53.690 And we can record it. 21:53.830 --> 21:56.830 And nobody was making any sense of that. 21:56.830 --> 21:57.930 That we could find out. 21:57.930 --> 22:02.550 People tried, um, and we tried to say, well, what are we gonna do? 22:03.990 --> 22:06.350 So, we took three of them. 22:06.350 --> 22:07.970 Three SSD models. 22:07.970 --> 22:08.110 Alright? 22:08.110 --> 22:09.410 And I listed them there. 22:09.510 --> 22:13.670 Uh, the Seagate, the WDC one, and the Crucial one. 22:13.670 --> 22:16.530 They were all 250 gig SSD drives. 22:16.530 --> 22:16.790 Okay? 22:16.790 --> 22:19.090 So they're all identical from that perspective. 22:19.210 --> 22:22.550 Um, and we analyzed the smart stats. 22:22.550 --> 22:24.910 And we said, what the heck do we have? 22:25.130 --> 22:25.870 Alright? 22:25.870 --> 22:27.910 What are we looking at when we look at it? 22:27.910 --> 22:29.070 Is it any good? 22:29.290 --> 22:29.770 Okay? 22:29.770 --> 22:35.550 They've been doing, the hard drives have been using smart stats for, oh, I don't know, you know, 25 years. 22:35.550 --> 22:39.670 Um, is, uh, you know, so what about SSDs? 22:39.690 --> 22:43.870 And we found some things that were common between all three models. 22:43.870 --> 22:44.870 Right? 22:44.870 --> 22:46.210 Power on hours. 22:46.210 --> 22:46.950 Yay. 22:47.130 --> 22:49.890 Uh, power cycle count on and off. 22:49.950 --> 22:51.010 Uh, temperature. 22:51.010 --> 22:52.090 That was good. 22:52.090 --> 22:53.710 And then a couple of them. 22:53.990 --> 22:57.690 SDD, SSD wear leveling and unexpected power loss. 22:57.690 --> 23:05.010 Those were the only five attributes that were common between those three SSDs. 23:05.410 --> 23:10.470 And so for example, the Seagate has 20 pairs. 23:10.610 --> 23:12.070 Only five are common with the other one. 23:12.070 --> 23:15.010 That means 15 are unique to the Seagate. 23:16.550 --> 23:17.670 Why? 23:17.690 --> 23:18.730 Okay? 23:19.590 --> 23:24.950 So we tried to figure out what were interesting ones. 23:25.030 --> 23:28.830 Um, and I want to talk about a few of those today just so you can see. 23:29.050 --> 23:35.430 It's a little bit of a wild west right now as it relates to SSDs and smart stats. 23:36.290 --> 23:37.070 Right? 23:37.070 --> 23:38.230 Here's a good one. 23:38.230 --> 23:39.850 Lifetime percentage. 23:39.950 --> 23:43.690 Now, and most of us would say, oh, that's, I get that. 23:43.690 --> 23:44.170 Okay? 23:44.170 --> 23:47.830 That means how long do I expect this thing to live? 23:47.830 --> 23:51.530 Where is it perhaps in its life cycle? 23:51.810 --> 23:52.570 Okay? 23:52.570 --> 23:54.690 That's a really, really cool thing. 23:54.690 --> 23:56.390 And they calculate that. 23:56.830 --> 24:01.750 All three of the vendors, the three different vendors, use a different smart stat. 24:01.850 --> 24:02.790 Okay? 24:04.930 --> 24:11.310 The WDC one and the Seagate both basically compute it the same way. 24:11.310 --> 24:11.750 Okay? 24:11.750 --> 24:16.170 Something called program erase cycles, available blocks, math. 24:16.190 --> 24:16.970 Okay? 24:17.090 --> 24:22.610 And then they get a number, an SSD, a number that starts at a hundred and comes down. 24:22.910 --> 24:23.410 Okay? 24:23.410 --> 24:25.770 So a hundred is new, new, new, new. 24:25.770 --> 24:27.770 Ninety-nine, still pretty new. 24:27.770 --> 24:29.290 Ninety, really good. 24:29.290 --> 24:29.990 Okay? 24:29.990 --> 24:37.270 In the case of the Western Digital one, when it gets to zero, you got to, it's over. 24:37.270 --> 24:39.770 It's past its life percentage time. 24:39.770 --> 24:41.670 You really should yank that drive. 24:41.670 --> 24:44.590 In the case of the Seagate, it's ten. 24:44.590 --> 24:46.530 That's their threshold value. 24:46.570 --> 24:50.550 And the number zero, that means the drive's in read-only mode. 24:50.670 --> 24:51.850 Right? 24:52.190 --> 24:56.070 Now, okay, so that's, it's close enough. 24:56.130 --> 24:59.830 The folks at Crucial decided that they weren't going to count down. 24:59.830 --> 25:01.310 They were going to count up. 25:02.090 --> 25:02.950 Okay? 25:03.130 --> 25:04.010 Okay? 25:04.350 --> 25:06.870 So it's lifetime used. 25:06.870 --> 25:11.650 So the number comes up as zero when you buy it, and it goes up as you use it. 25:11.990 --> 25:12.650 All right? 25:12.650 --> 25:15.230 So I don't know. 25:15.230 --> 25:16.010 You know? 25:16.930 --> 25:31.650 So one of the things that points out, by the way, is that tools like smart CTL, which is what we use to do this, right, now have to pull out, I have to pull out three sets of numbers and do my analysis if I want to compare things. 25:31.650 --> 25:34.710 And then I have to remember, in the case of the Crucial one, to flip. 25:34.790 --> 25:35.170 Right? 25:35.170 --> 25:36.310 And that's kind of cool. 25:37.210 --> 25:52.590 The other thing we saw is that all three vendors, and they're not the only ones, it's not, Micron does the same thing, Dell seems to do the same thing, all, all of the different vendors seem to have their own tool that you're going to get. 25:52.790 --> 26:06.130 Now, by the way, that's really cool if you buy a Crucial drive and a Crucial SSD, you got three or four of them, you really like it, it's a great brand, you go, and you use that tool to do your analysis and see the health of that drive and excellent, right? 26:06.370 --> 26:06.970 Right? 26:07.210 --> 26:13.990 But when you're in a business that has actually more than one type of SSD, it starts to get to be a pain. 26:13.990 --> 26:19.530 And when you're in a business that has 20 different types of SSDs, it's a pain in the neck. 26:21.350 --> 26:27.810 So one of these days I hope they kind of get their act together and maybe make these a little more consistent. 26:28.730 --> 26:33.370 Another one, some interesting ones that I found as I was digging through these, right? 26:34.070 --> 26:39.070 WDC has something called drive life protection status. 26:39.890 --> 26:40.610 Right? 26:40.690 --> 27:01.330 And this is amazing because it, it has this thing like, it's like a trajectory, it says, okay, this implies that they have, they can calculate their drive life, and then it implies that there's some curve that they've come up with, okay, and embedded into that so that they can do this projection. 27:01.890 --> 27:02.510 Okay? 27:02.950 --> 27:07.710 But nowhere have I been able to find what that curve is. 27:07.790 --> 27:11.190 So is it tied to, for example, the warranty period? 27:11.230 --> 27:13.590 Or is it, whatever, right? 27:13.590 --> 27:14.730 Is it tied to load? 27:14.730 --> 27:24.010 So you are, you, they have a curve like this, and if you're using your SSD too much, it could be above it, but, okay, what's too much? 27:24.470 --> 27:25.310 Alright? 27:26.770 --> 27:31.010 Endurance remaining, okay, that's kind of, kind of fun. 27:32.170 --> 27:40.310 The, with SSDs and stuff like that, it's a twisted way of looking at the protection status, but there's no curve. 27:40.590 --> 27:44.450 The last one is actually really cool. 27:44.510 --> 27:45.250 Okay? 27:45.630 --> 27:49.170 This is, the folks at Crucial have this built in. 27:49.170 --> 27:54.310 It's a little thing called RAIN, which is kind of like RAID, except on their SSD. 27:54.630 --> 27:55.150 Okay? 27:56.050 --> 28:02.210 And what they do is they hash out your data, and they store the parity off. 28:02.330 --> 28:15.810 And so if you as a customer suddenly start to use your SSD, and the data can't be found, okay, in a normal read, they can use the hash to recover it, and that's all completely invisible to you. 28:16.550 --> 28:17.210 Right? 28:17.550 --> 28:19.450 Really, really awesome. 28:19.690 --> 28:20.070 Okay? 28:20.070 --> 28:28.070 They didn't say whether that parity was actually part of that 250 gigabits, or it's added and you don't see it. 28:28.070 --> 28:28.890 Okay? 28:29.090 --> 28:36.490 But the overheads for something like that, it's got to be 10, 15% of storage to pull it off. 28:36.490 --> 28:40.890 But it's a really cool feature because, again, as a user, you wouldn't even know it. 28:41.250 --> 28:41.810 Okay? 28:41.810 --> 28:46.290 They go read something, they couldn't read it, they use the hash and recover it. 28:46.650 --> 28:47.290 Man. 28:47.930 --> 28:49.450 Really enjoyed that one. 28:53.050 --> 28:58.410 One of the questions that came up while I was doing a bunch of research was this one. 28:58.410 --> 29:02.210 How long can you store an SSD without power? 29:02.290 --> 29:04.270 You take it, put it in the closet. 29:04.490 --> 29:04.850 Right? 29:04.850 --> 29:05.750 We all do this. 29:05.750 --> 29:07.170 We do it with hard drives. 29:07.170 --> 29:08.990 We expect them to last forever. 29:09.270 --> 29:09.430 Okay? 29:09.430 --> 29:12.430 We expect to pull that hard drive out after five years. 29:12.430 --> 29:14.750 Plug it in and spin that puppy right out. 29:14.970 --> 29:18.210 And, I don't know, sometimes that doesn't happen. 29:18.630 --> 29:22.390 Sometimes you got to get out the hammer and hit the top. 29:23.110 --> 29:25.930 Sometimes you got to get a new hard drive. 29:26.130 --> 29:27.770 So, SSDs. 29:27.770 --> 29:28.150 Okay? 29:28.150 --> 29:35.770 The number, the number's been all over the place, but I'll just say the current thinking is about a year. 29:36.310 --> 29:36.950 Okay? 29:36.950 --> 29:38.130 Is the maximum. 29:38.690 --> 29:46.470 Now, that seems to be tied to how long the drive, busy the drive is. 29:46.470 --> 29:47.730 How tired it is. 29:47.730 --> 29:49.630 How much it's been used before. 29:50.070 --> 29:52.670 Crucial actually documents that. 29:52.690 --> 29:53.270 Okay? 29:53.270 --> 29:54.550 And has a little number. 29:54.550 --> 29:57.850 So, all the way over on the, that side. 29:58.190 --> 30:00.690 I get my left and right screwed up all the time. 30:01.070 --> 30:08.470 You can see, if you look at smart value 202 and it's a zero, once again, they like to count down and count up. 30:09.510 --> 30:13.030 Zero, the maximum storage period, a year. 30:13.410 --> 30:13.750 Okay? 30:13.750 --> 30:17.810 And as that number goes up, they continue to reduce it. 30:17.810 --> 30:21.090 So, if that number's 50, six months. 30:21.090 --> 30:24.130 If it's 100, a month and you're stretching it. 30:24.410 --> 30:25.170 Okay? 30:25.690 --> 30:37.490 The other two vendors, okay, don't have similar attributes that they use for the same kind of thing, but I can't find any documentation which says that's the way it works. 30:38.090 --> 30:39.390 All right? 30:39.590 --> 30:46.130 So, I can't trust it, but it seems to be that those two pieces are built that way for it. 30:46.130 --> 30:59.390 So, if you're thinking of taking a hard, an SSD and putting it in archive for a little bit of time, you may want to take a look at that number for the SSD and see how long you think you're going to put it there for. 30:59.390 --> 31:05.010 If it's been used a lot, you may not be able to get away with putting it in there for a year. 31:05.650 --> 31:06.250 Right? 31:10.210 --> 31:10.850 All right. 31:10.850 --> 31:14.250 So, a little bit more about, oops, okay. 31:14.250 --> 31:16.190 No more about SSDs. 31:16.190 --> 31:18.650 Apparently it's, hey, computer's pissed off. 31:18.970 --> 31:22.030 Um, so we'll get back to hard drives. 31:22.030 --> 31:24.270 Just some fun stuff about hard drives. 31:25.130 --> 31:30.410 A few months ago, there was an article published by Secure Data Recovery. 31:30.410 --> 31:32.110 It's another company in the business. 31:32.790 --> 31:42.290 And they came up with this number which said, hey, we looked at 2007 failed drives and the average age of a failed drive was two years and ten months. 31:42.690 --> 31:47.470 And that, I looked at that and said, oh my, that seems really low. 31:48.050 --> 31:51.270 I was thinking, it's got to be higher than that, right? 31:51.510 --> 31:54.110 So, I took more data. 31:54.470 --> 31:54.910 Okay? 31:54.910 --> 32:00.850 And over the years, we've had about 17,000 or so failures and I said, what's the average age they failed at? 32:01.090 --> 32:03.230 And it came up to two years and six months. 32:03.670 --> 32:09.590 And I went, you just look at something and it just doesn't seem right, right? 32:09.870 --> 32:11.570 So, then I said, well, wait a minute. 32:11.570 --> 32:15.450 I have a bunch of models that haven't finished failing yet. 32:15.690 --> 32:16.310 Okay? 32:16.970 --> 32:18.670 In other words, they're still in production. 32:18.670 --> 32:20.550 So, they could fail tomorrow. 32:20.830 --> 32:23.070 So, I took out all of their drives. 32:23.070 --> 32:26.330 I only had to use the drives that we didn't have in production anymore. 32:26.330 --> 32:29.150 So, our data set, those numbers weren't going to change. 32:29.250 --> 32:33.650 And it rose all the way up to two years and seven months, right? 32:33.950 --> 32:35.750 Which still feels low. 32:35.810 --> 32:38.350 It just doesn't feel right. 32:38.350 --> 32:55.730 And so, just to kind of give you an idea of what we're trying to look at here, the ones in pink that aren't pink, okay, that have zero drives remaining, those are the ones that I used for that data set that got me two years and seven months, right? 32:56.390 --> 33:01.870 The other drives still have plenty, we still have plenty of drives in service. 33:02.150 --> 33:05.650 So, we can't really use those as a calculation. 33:05.710 --> 33:12.790 And if you look at them, for example, the four terabyte drives, the average age of those is seven years and one month. 33:12.790 --> 33:20.290 So, anything that fails out of that set is certainly going to be older than two years and six months. 33:20.510 --> 33:31.770 So, what I'm saying is, as those drives start to come out, I expect that two years and six months or two years and seven months to go up to some other number. 33:31.770 --> 33:35.150 And we'll track it over time and see what it looks like, okay? 33:35.650 --> 33:42.250 This is just one of these things that you look at the data, you get an answer, you go, I just don't believe the answer. 33:43.230 --> 33:45.970 And then you go from there. 33:46.110 --> 33:46.630 All right. 33:46.630 --> 33:49.310 A couple of little fun things that we like to do. 33:50.330 --> 33:53.190 Failure rates by, in this case, drive size. 33:53.190 --> 33:55.730 So, we take all of the drives of a given size. 33:58.010 --> 34:00.650 Our ten terabyte drives are doing terrible. 34:01.350 --> 34:02.150 Okay. 34:02.850 --> 34:03.890 It happens. 34:03.890 --> 34:06.030 They get old, they hit the wall. 34:06.710 --> 34:10.390 You know, we obviously had a lot of strategies in place to deal with all of this. 34:10.390 --> 34:11.610 It's not an issue. 34:12.510 --> 34:17.090 We've been dealing, you know, we've been around for 16 years. 34:17.090 --> 34:20.210 So, we've seen all kinds of different interesting patterns. 34:20.950 --> 34:24.950 But I bring this up because this is what we publish and this is what we show. 34:24.950 --> 34:26.830 And this is the data that's out there. 34:27.370 --> 34:30.930 It also gives us an interesting idea to see what's happening. 34:30.930 --> 34:34.910 So, the one that was the surprise was the eight terabyte drives. 34:35.370 --> 34:40.030 So, the last couple of quarters, they suddenly started to jump up. 34:41.470 --> 34:45.550 And so, then we had to go in and dig in and see if it was a particular drive model. 34:45.550 --> 34:46.150 It wasn't. 34:46.150 --> 34:47.690 It was all of the drives. 34:48.630 --> 34:55.510 And we had to go dig in and see if it had to do something with software changes that we made or location they are. 34:55.510 --> 34:56.990 We had to get all of that out. 34:56.990 --> 34:57.390 No. 34:57.390 --> 35:02.350 What's just happening is they're just getting to the end of their life and the failure rate is starting to rise. 35:02.810 --> 35:03.210 Right? 35:03.210 --> 35:05.690 But we publish this kind of stuff on a regular basis. 35:05.690 --> 35:07.930 We also do this by manufacturer. 35:08.490 --> 35:09.190 Okay? 35:10.070 --> 35:15.050 And I always put this up and everybody looks at it and goes, Seagate sucks. 35:15.470 --> 35:16.350 Okay? 35:17.730 --> 35:19.450 And no, they don't suck. 35:19.450 --> 35:20.530 They're a nice guys, too. 35:20.530 --> 35:21.550 They're all nice guys. 35:21.550 --> 35:23.830 We have a good relationship with all of them. 35:23.990 --> 35:24.430 Right? 35:24.510 --> 35:29.570 But you can see the numbers are rising and they've always been a little high. 35:30.090 --> 35:32.790 And so, I wanted to do a little bit of math for you. 35:33.570 --> 35:39.050 Just to show you that sometimes it isn't just the failure rate that matters . 35:39.050 --> 35:39.830 All right? 35:40.290 --> 35:42.630 And we publish this model. 35:42.630 --> 35:50.650 It's a simplified version of what we do internally to figure out who we should buy drives from and how much we're going to pay. 35:51.410 --> 35:52.290 Right? 35:52.510 --> 35:55.330 And I'm actually going to apply real data to it. 35:55.330 --> 35:56.250 But that's the model. 35:56.250 --> 35:57.910 You can kind of walk through it. 35:57.910 --> 35:59.530 We pay so much for a drive. 35:59.530 --> 36:01.410 Model 1, model 2, model 3. 36:01.410 --> 36:02.730 Three different prices. 36:02.770 --> 36:03.350 Right? 36:03.350 --> 36:05.050 We buy so many drives. 36:05.050 --> 36:07.030 They have three different failure rates. 36:07.190 --> 36:10.290 The cheapest one has the highest failure rate and so on. 36:10.370 --> 36:12.870 And you can see the number of failed drives. 36:12.870 --> 36:15.770 To replace a drive, that's 300 bucks for labor. 36:15.770 --> 36:17.890 We assume all of them are under warranty. 36:17.890 --> 36:20.510 Five-year warranty because it's a five-year period. 36:20.510 --> 36:22.950 So, we're not going to pay and buy any more drives. 36:22.950 --> 36:25.890 But if we did, it would be the same for everybody. 36:26.430 --> 36:30.010 And then there's at the bottom, there's a total replacement total cost. 36:30.010 --> 36:32.210 So, the drives plus the maintenance. 36:32.210 --> 36:35.830 Again, a little bit simplified, but it'll work for us. 36:35.830 --> 36:38.130 And you can see model 1. 36:38.130 --> 36:39.250 Oh, my goodness. 36:39.530 --> 36:43.970 I saved almost $200,000 by buying those drives. 36:44.530 --> 36:46.310 Over the lifetime of those drives. 36:46.310 --> 36:47.750 That's really, really cool. 36:47.750 --> 36:48.210 Right? 36:48.450 --> 36:51.010 So, let's use real data. 36:53.070 --> 36:57.550 And so, I put Seagate, Toshiba, and WDC. 36:57.630 --> 37:00.150 These are 14 terabyte drives. 37:00.550 --> 37:01.030 Right? 37:01.030 --> 37:03.810 This is in our data center, real numbers. 37:04.130 --> 37:06.030 And I wanted to break it down. 37:06.030 --> 37:14.190 So, the first thing I did is I started down here and I said, I'm going to make, I'm going to do this math backwards. 37:14.510 --> 37:20.690 I'm going to make the answer the same and let all of the other factors start to change going back up to the top. 37:20.690 --> 37:23.610 In other words, I want to get a number at the top. 37:23.610 --> 37:24.190 Right? 37:24.370 --> 37:31.090 So, it's going to cost me almost $2.6 million to buy 10,000 drives and maintain them for five years. 37:33.640 --> 37:40.220 And then I put in the failure rates, the actual failure rates for the lifetime for those drives. 37:40.320 --> 37:40.880 Okay? 37:41.220 --> 37:42.020 And I put them in. 37:42.020 --> 37:43.020 And those are real. 37:43.020 --> 37:44.620 I didn't make them up at all. 37:45.200 --> 37:46.080 Right? 37:47.800 --> 37:53.820 And what I come up with is the price I'd be willing to pay to buy those drives. 37:55.860 --> 37:56.440 Right? 37:56.440 --> 38:08.640 So, I'd be willing to give Seagate $230 and I'd but there are other factors which come into that. 38:08.640 --> 38:10.820 Like I said 10,000 drives. 38:11.080 --> 38:15.560 So, if I go to WDC and I say, hey, I need 10,000 drives, what number can you give me? 38:15.800 --> 38:18.460 And they go, I don't have 10,000 of those. 38:18.460 --> 38:20.100 I have 12,000. 38:20.500 --> 38:20.980 Okay? 38:21.840 --> 38:30.400 And then you go to Toshiba and Toshiba says, well, I have 8,000 but I can't get them to you until a month from now. 38:30.460 --> 38:31.260 Okay? 38:31.260 --> 38:32.780 But I'll cut the price down. 38:32.780 --> 38:35.620 And so, you have all of these other factors which make it in. 38:35.620 --> 38:41.780 But the point is, okay, this is why we have drives from different vendors. 38:41.780 --> 38:42.260 Okay? 38:42.260 --> 38:45.000 When you do all of that kind of math to figure it all out. 38:46.200 --> 38:46.940 All right. 38:46.940 --> 38:50.580 So, again, we publish this data on a regular basis. 38:51.500 --> 38:53.740 We have all this smart stats. 38:53.920 --> 38:56.000 We have more data coming. 38:56.000 --> 38:56.500 Okay? 38:56.500 --> 39:01.600 Which will give us the opportunity to share more with the community because that's what it's all about. 39:02.540 --> 39:04.500 We have something called the vault ID. 39:04.500 --> 39:06.860 I didn't talk much about what a vault is. 39:06.860 --> 39:09.780 A vault is just an organizational unit for us. 39:09.860 --> 39:12.980 1,200 drives in 20 servers. 39:13.400 --> 39:16.580 Each drive on each server is a tome and so on and so forth. 39:16.580 --> 39:18.320 I'm not going to spend any time with that. 39:18.320 --> 39:27.700 But what we get is we can isolate problems then down to a vault, which is, again, a logical unit or a pod, which is a storage server. 39:27.700 --> 39:30.040 And start to see and do the analysis. 39:30.040 --> 39:32.980 Now, our ops guys have been doing this for years. 39:33.100 --> 39:33.340 Okay? 39:33.340 --> 39:35.700 I just didn't have the data to share with you. 39:35.700 --> 39:37.660 So, we'll be able to share that. 39:37.840 --> 39:40.080 We'll also have data center. 39:40.220 --> 39:42.820 So, you know that boat that I showed you earlier? 39:42.820 --> 39:46.820 We'll be able to say, is the failure rate any better or worse out in the boat? 39:47.180 --> 39:51.200 Versus our other data center in Phoenix because it's kind of warm down there. 39:51.200 --> 39:54.120 Or our other data center in Sacramento. 39:54.120 --> 39:55.860 Or the one in Amsterdam. 39:55.860 --> 39:58.040 Is there a problem because the electricity is different? 39:58.040 --> 39:59.320 Who knows, right? 40:00.220 --> 40:05.240 And then we'll be able to look at things like failure rates by server type and data center. 40:05.260 --> 40:08.000 And even things like temperature. 40:08.860 --> 40:09.300 Right? 40:09.300 --> 40:11.540 So, on that boat. 40:12.080 --> 40:12.620 Okay? 40:12.620 --> 40:16.280 That's using water cooling instead of air conditioning. 40:16.280 --> 40:19.020 What kinds of things are going on out there? 40:19.140 --> 40:20.840 So, those are the kinds of things coming. 40:21.240 --> 40:21.640 All right? 40:21.640 --> 40:23.400 Lots of fun stuff that we look at. 40:23.400 --> 40:24.660 We've been doing it for years. 40:24.660 --> 40:25.920 The data is free. 40:26.140 --> 40:27.220 It's out there. 40:27.220 --> 40:29.980 You guys can play with it if you want. 40:29.980 --> 40:31.880 You can read the reports if you want. 40:31.880 --> 40:32.560 Okay? 40:32.960 --> 40:35.580 You can also do analysis for us and send it to us. 40:35.580 --> 40:38.260 Because I'd love to share that with folks. 40:38.300 --> 40:39.880 That's what this is all about. 40:39.880 --> 40:41.660 So, all right. 40:41.660 --> 40:44.300 If there's any questions, I'll gladly take them now. 40:44.300 --> 40:46.300 Or I'll be here for a few minutes. 40:46.300 --> 40:49.100 And we can take it from there. 40:49.540 --> 40:49.600 So 40:55.480 --> 40:57.640 , any questions in the audience? 40:57.640 --> 40:59.600 Any questions in the audience? 40:59.600 --> 41:00.000 Thanks, Andy. 41:00.000 --> 41:01.820 If you have any questions, just come on up to this mic. 41:01.820 --> 41:04.920 That way we can hear you and they can hear you on the DEF CON TV as well. 41:06.800 --> 41:08.900 I have one quick question. 41:09.100 --> 41:11.860 What drive do you use at home for backups? 41:12.960 --> 41:14.620 I have a Seagate drive. 41:15.160 --> 41:17.420 So, those are HTDs, I assume? 41:17.580 --> 41:18.060 Yeah. 41:18.060 --> 41:19.100 Well, yes. 41:19.100 --> 41:20.740 They're all HTD Seagates, yeah. 41:20.740 --> 41:21.000 Okay. 41:21.000 --> 41:21.840 What size? 41:24.480 --> 41:26.520 I think it's a six terabyte. 41:26.520 --> 41:27.200 Okay. 41:27.200 --> 41:28.300 Thank you. 41:30.520 --> 41:35.040 My question is about the SSD and its storage of data. 41:35.440 --> 41:41.660 Was that based on power to the chip or the bit being written to the year of life? 41:43.040 --> 41:44.720 I'm not sure I understand the question. 41:44.720 --> 41:51.800 So, your chart showed that if the SSD wasn't used for a year, it could start to lose data. 41:52.180 --> 41:59.520 Was that meaning that the bit was written to or power applied to the chip? 41:59.520 --> 42:00.560 Power applied. 42:00.560 --> 42:00.920 Okay. 42:00.920 --> 42:01.120 Yeah. 42:01.120 --> 42:01.680 Thank you. 42:01.680 --> 42:02.400 Yeah. 42:04.700 --> 42:05.780 Great. 42:05.780 --> 42:11.890 Have you done any special analysis on how vibration maybe influences lifetime of drives? 42:12.880 --> 42:13.420 Yeah. 42:13.420 --> 42:14.900 So, that's a good question. 42:14.900 --> 42:16.720 Vibration is a tough thing to measure. 42:16.720 --> 42:22.940 Technically, there's a smart stat for that, but it doesn't seem to be populated very well for it. 42:23.580 --> 42:29.020 And so, what you end up doing is, in our case, that's part of what we'll see with the server type. 42:29.540 --> 42:35.380 Because we use different mechanisms in the different servers to lock the drives down or hold them in place. 42:35.640 --> 42:38.260 And that may be one of the things we see. 42:38.460 --> 42:43.300 The other thing I have is anecdotal information from our maintenance records. 42:43.460 --> 42:49.780 All of this data, by the way, that I produce off the drives gets validated against maintenance records before we publish it. 42:49.960 --> 42:55.980 And the maintenance records sometimes have that as a reason, drive vibration. 42:56.580 --> 43:03.460 And because drive vibration actually not only bothers the current drive, it usually bothers the neighbors as well. 43:03.940 --> 43:10.400 So, it's a hard one to say because we should have a stat that gives us that, but the stat seems to be garbage. 43:11.160 --> 43:14.420 And so, we have to rely on just other things. 43:14.900 --> 43:20.960 When we built, when we were building storage servers, the storage pots, okay, we built multiple versions of them. 43:20.960 --> 43:25.440 We had like four different ways that we tried to minimize vibration. 43:25.660 --> 43:29.080 That's one of the things we're going to take a look at now that we have the storage ID. 43:29.760 --> 43:30.980 So, it'll be fun. 43:30.980 --> 43:32.020 So, good question. 43:32.020 --> 43:33.680 I don't have a really good answer, sorry. 43:34.040 --> 43:34.480 Good to know. 43:34.480 --> 43:35.140 Thank you. 43:40.190 --> 43:58.930 Kind of in that same vein, have you looked at other correlations, not just with things happening in the building, but things happening to employees, to personnel, kind of looking for other explanations for drive failure over time? 43:58.930 --> 43:59.330 Yeah. 43:59.330 --> 44:04.650 So, that's a really good thing outside of the technology, the human element a little bit. 44:05.590 --> 44:17.770 Interestingly enough, the maintenance records show that the most because they're all trained to do the maintenance records a certain way, and they actually have a script that they have to follow and things of that nature. 44:17.770 --> 44:19.650 And when they don't, you can see it. 44:20.070 --> 44:26.990 And so, Larry, who's the guy who runs our global data centers, him and I have that conversation from time to time. 44:27.070 --> 44:33.470 I'll be reading through it and I'll go, you know, Barney doesn't seem to be doing real good on that stuff. 44:33.470 --> 44:35.370 He says, yeah, Barney's no longer with us. 44:36.430 --> 44:40.130 So, they do have a process that they're supposed to follow. 44:40.350 --> 44:49.090 It's relatively consistent, but the reality is that sometimes things just don't go in a straight line, you know. 44:49.090 --> 44:52.690 And so, it's an interesting point, but no, we don't. 44:52.690 --> 44:54.030 That's the best we can do. 44:54.030 --> 44:54.670 Gotcha. 44:55.290 --> 44:56.490 All right. 44:56.490 --> 44:57.390 Any other questions? 44:57.390 --> 45:05.690 If there's not, I will gladly, like I said, I'll be down here if you have anything else you want to ask and I'll gladly share anything we want. 45:05.810 --> 45:06.450 Check it out. 45:06.450 --> 45:08.230 It's fun data, okay. 45:08.570 --> 45:10.030 There's no harm, no foul. 45:10.030 --> 45:12.110 You can use it, you can don't use it, whatever. 45:13.530 --> 45:14.490 Thank you, Andy. 45:14.790 --> 45:15.810 Thank you, Andy. 45:15.810 --> 45:19.570 And the Data Duplication Village is on the other side of that wall, if you haven't been there before. 45:19.570 --> 45:20.350 Thanks, everyone.