[00:00.000 --> 00:06.080] This is Jeremy Gosney. I'm coming to you live from Password Village HQ2 in sunny Austin, Texas. [00:06.080 --> 00:08.400] Today's high is 104 degrees. [00:08.880 --> 00:15.200] So today I'm going to be talking to you about what it's like cracking at Terrahash's scale. [00:16.720 --> 00:18.920] I'm not going to make this a vendor pitch in any way. [00:18.920 --> 00:25.900] I'm going to focus on the technical challenges that we encounter as we support clients [00:25.900 --> 00:28.920] who have hundreds of GPUs in their clusters. [00:30.620 --> 00:35.080] When I first started on this, I thought 100 GPUs was a really big deal. [00:35.640 --> 00:39.660] Now we tend to view 100 GPUs as like, meh. [00:40.660 --> 00:43.300] It's good, but it's kind of middle of the road. [00:47.260 --> 00:51.880] There's a lot of distributed solutions for HashCat out there. [00:52.360 --> 00:58.440] But as far as I know, Terrahash is the only one that aims to operate at warehouse scale. [00:58.440 --> 01:05.340] What I mean by that is, more so with the most recent version, but with every version that passes, [01:05.340 --> 01:10.020] we aim to embrace the concepts of data center as a computer. [01:10.560 --> 01:16.820] While we currently don't have any clients who have an entire warehouse full of password cracking hardware, [01:16.820 --> 01:19.300] that's the point where we're trying to go. [01:20.880 --> 01:27.720] We try to enable the software to be at that level, even if our clients are not quite at that level. [01:30.560 --> 01:38.400] I'm going to go through the history of how I got started in massive distributed cracking. [01:38.520 --> 01:46.540] And I'm going to go through the technical challenges that we faced as we continued to evolve our hashtag distributed solution. [01:47.580 --> 01:49.600] I'm going to start the slide deck. [01:57.330 --> 01:58.810] All right. [02:01.480 --> 02:07.300] So, in the beginning, I kind of got into this because I was messing around with distributed computing. [02:07.300 --> 02:16.720] I was doing stuff with MPI, some other MOSIX software, rocks clusters, things along those natures. [02:17.820 --> 02:19.840] But all that was all CPU-based. [02:19.840 --> 02:29.900] And then when GPU cracking came about, we started looking for solutions that would enable us to distribute GPU workloads. [02:29.940 --> 02:34.840] So, I found a piece of software at a Hebrew university called VirtualCL. [02:35.160 --> 02:44.060] And I did a presentation on a 25 GPU proof of concept cluster that me and Bitweasel put together at passwords12 in Oslo. [02:44.300 --> 02:49.740] And when I woke up the next morning after giving that presentation, we had gone viral. [02:49.840 --> 02:56.060] On the front page of Slashdot, Gizmodo, Boing Boing, NBC News, The Register, we were everywhere. [02:57.140 --> 03:01.480] And in hindsight, I really should have used a lot more than 25 GPUs. [03:01.480 --> 03:05.300] I mean, we had more GPUs, that's just what we chose to dedicate to this proof of concept. [03:05.300 --> 03:12.540] And had I known in advance in any way that the general public would react how they did to this proof of concept, [03:12.540 --> 03:15.160] I would have used like three times the number of GPUs. [03:17.060 --> 03:22.940] So just, you know, some of the headlines that came out the next morning that I woke up to, [03:22.940 --> 03:25.980] which was pretty surreal because I didn't really see this as a big deal. [03:27.360 --> 03:35.500] So, VirtualCL was cool at the time because it was the first solution that I knew of that enabled us to [03:36.300 --> 03:42.820] sort of transparently distribute OpenCL jobs across the arbitrary cluster. [03:42.820 --> 03:47.080] So what VirtualCL does is it provides a virtual OpenCL platform. [03:48.620 --> 03:54.080] And the target software, in theory, doesn't need to be aware it's communicating with a virtual platform. [03:54.080 --> 04:00.400] It just sees any, you know, it sees VirtualCL as any other, you know, OpenCL platform, [04:00.400 --> 04:03.160] just like AMD, NVIDIA, Intel, what have you. [04:04.560 --> 04:10.200] So the target software just simply links against libOpenCL, which is provided by VCL, [04:10.200 --> 04:15.140] which then communicates to a broker daemon, and then that broker daemon distributes to the agent daemons, [04:15.140 --> 04:17.400] and the agent daemons, which are installed on the compute nodes, [04:18.100 --> 04:24.440] actually communicate with the real OpenCL library, be it AMD, NVIDIA, whatever. [04:26.040 --> 04:29.680] So in theory, all this is transparent to the end application. [04:29.680 --> 04:32.420] But in practice, it didn't quite work that way. [04:33.380 --> 04:38.440] We had to have a special fork of OCLHashCat called VCLHashCat, [04:38.440 --> 04:42.160] which had some workarounds for VCL quirks and such. [04:43.280 --> 04:47.760] It also required InfiniBand, because it was really latency sensitive. [04:47.980 --> 04:53.140] It ended up being a very chatty protocol, and it required essentially real-time communication. [04:53.400 --> 04:55.000] Bandwidth wasn't so much of an issue. [04:55.000 --> 04:59.780] It pretty much consistently pulled, you know, less than one gig of bandwidth, [04:59.780 --> 05:03.600] but it was really sensitive to latency. [05:03.960 --> 05:08.040] Even just a couple milliseconds was acceptable, [05:08.040 --> 05:10.760] but as you start getting up into double digits of latency, [05:10.760 --> 05:13.620] you had a noticeable impact on hash rate. [05:14.480 --> 05:16.920] And of course, InfiniBand hardware is expensive. [05:16.920 --> 05:19.300] It's way more expensive than Ethernet. [05:20.240 --> 05:23.520] VCL was also closed source, and it required frequent updates. [05:23.520 --> 05:28.120] So this is back in the time, this is 2012 through 2013, [05:28.120 --> 05:33.300] where every GPU driver that was released broke some shit. [05:33.300 --> 05:36.480] And then we had to go back in and implement workarounds and fixes, [05:36.480 --> 05:40.540] whatever the fuck AMD or NVIDIA broke in their driver that day. [05:40.540 --> 05:42.660] And VCL was no exception. [05:42.660 --> 05:48.600] So with VCLHashCat, we had to have our own workarounds for VirtualCL, [05:48.600 --> 05:51.780] and then the VCL team, or VirtualCL team, [05:51.780 --> 05:56.560] had to turn around and implement any workarounds for any new driver releases [05:56.560 --> 05:58.440] every time they were released. [05:58.440 --> 06:02.280] Which is a pretty substantial job. [06:02.560 --> 06:05.180] The problem is, VirtualCL was created by grad school students. [06:05.180 --> 06:08.850] This was a grad school project, and they created this their senior year. [06:09.360 --> 06:13.680] And when they graduated in spring 2013, that was it. [06:13.680 --> 06:15.300] The project died. [06:15.860 --> 06:20.720] So when the next version of FGLRX was released, [06:20.720 --> 06:24.560] I think that was version 13.9 in September of 2013, [06:25.600 --> 06:30.280] we go to install a VirtualCL cluster for our client. [06:30.280 --> 06:32.800] This is actually a 64 GPU cluster. [06:33.060 --> 06:35.740] We were just starting out, so this was actually a pretty big deal for us. [06:35.740 --> 06:38.480] 64 GPUs was pretty fucking cool at the time. [06:39.780 --> 06:44.480] But we get the OS installed, we get the hardware built, all that shit, [06:44.480 --> 06:48.980] and then we go to install VirtualCL and FGLRX, [06:49.500 --> 06:51.440] and VirtualCL just shits the bed. [06:51.440 --> 06:54.800] So I panic and email Hebrew University, [06:54.800 --> 06:56.760] and I'm like, you guys really need to update VirtualCL [06:58.240 --> 07:02.340] for Catalyst and FGLRX 13.9. [07:02.780 --> 07:06.680] And they're like, well, we can't. [07:06.680 --> 07:09.200] Everyone who was working on this project has gone. [07:10.660 --> 07:14.660] And the intellectual property remains with Hebrew University and MOSIX, [07:14.660 --> 07:18.520] so they can just have those people work on it outside of that. [07:18.940 --> 07:21.380] I offered to take over the project for them, [07:22.420 --> 07:25.040] because it was kind of essential to what we were doing, [07:25.040 --> 07:26.420] and I didn't want to see the project die. [07:26.420 --> 07:27.680] I thought it was really neat. [07:28.940 --> 07:32.580] But the professor over there who was in charge of the project [07:32.580 --> 07:35.180] was not amenable to that idea. [07:35.180 --> 07:37.000] He actually had pretty grave concerns [07:37.000 --> 07:39.460] when we first started using VirtualCL for password cracking, [07:39.460 --> 07:43.140] because he envisioned us turning the world into a giant botnet. [07:43.540 --> 07:45.860] And of course, that wasn't really on our agenda, [07:45.860 --> 07:47.500] at least not at that time. [07:47.500 --> 07:52.340] But yeah, he really didn't like the idea [07:52.340 --> 07:57.660] of his VirtualCL shit ending up in our hands. [07:58.280 --> 08:02.000] So we were left high and dry without a software solution, [08:02.000 --> 08:04.640] and so we desperately needed something. [08:04.700 --> 08:07.500] So we conceived the idea of Hashtag. [08:09.700 --> 08:12.380] Initially, we thought about making our own VirtualCL clone, [08:12.380 --> 08:16.680] since Hebrew University was not going to give us the VirtualCL source code. [08:16.680 --> 08:18.180] And we're like, well, fuck it. [08:18.180 --> 08:21.600] We'll start our own VirtualCL with hookers and blackjack. [08:22.320 --> 08:26.380] But the more we actually sat down and thought about what that entailed, [08:26.380 --> 08:30.700] we determined the level of effort for that was just way too high. [08:30.760 --> 08:34.100] And the timeframe in which we needed it was way too short. [08:34.500 --> 08:37.800] We literally had hardware in-house that we're building for clients [08:37.800 --> 08:41.680] who now have no software solution because of VirtualCL. [08:45.970 --> 08:47.950] So we decided to make a distributed wrapper. [08:47.950 --> 08:50.470] And that's when we came up with the idea of Hashtag. [08:51.190 --> 08:53.570] Like I said, we had to iterate really quickly on this. [08:53.570 --> 08:59.190] So we went from a whiteboard session to production in less than two months. [08:59.230 --> 09:03.330] And this was basically me and Tom Steele locking ourselves in the office [09:04.850 --> 09:07.430] and ordering over $100 for the Taco Bell [09:07.950 --> 09:11.310] and just banging this out as fast and furious as we could. [09:12.630 --> 09:16.010] It had a traditional client-server-agent architecture. [09:16.650 --> 09:18.570] And it was Hashtag focused, [09:18.570 --> 09:20.690] but it actually had a generic plugin interface. [09:20.990 --> 09:25.230] We had plugins for... and this was before there was just one Hashtag. [09:25.230 --> 09:33.150] So we had Hashtag, OCL Hashtag Plus, OCL Hashtag Lite plugins. [09:33.150 --> 09:35.090] And then we had John the Ripper plugin. [09:36.110 --> 09:38.970] And then we had just a generic interface [09:38.970 --> 09:41.650] to where any cracking tool could be made to work with Hashtag [09:41.650 --> 09:46.890] as long as it adhered to a standard format that we had defined. [09:47.350 --> 09:50.310] And then it also had the ability to run arbitrary commands when idle. [09:51.010 --> 09:53.690] This was less for our clients and actually more for me [09:53.690 --> 09:56.270] because we were running Hashtag in-house as well [09:56.270 --> 09:59.410] for our cracking-as-a-service type stuff. [10:00.470 --> 10:04.290] And this is back in a time when GPU mining was still profitable. [10:04.370 --> 10:06.730] So when we weren't working on a password cracking job, [10:06.730 --> 10:09.750] I wanted this thing to be generating coins in the background. [10:09.750 --> 10:14.930] So the arbitrary idle exec command was literally just put in place [10:14.930 --> 10:19.490] so that people could mine Litecoin or Namecoin or whatever [10:19.490 --> 10:22.590] when their cluster was idle. [10:24.090 --> 10:26.270] So, you know, totally different time. [10:26.670 --> 10:28.170] But, I mean, it actually worked. [10:28.170 --> 10:32.530] It did exactly what it was supposed to do and it was good. [10:33.530 --> 10:36.270] And the whole goddamn thing was implemented [10:36.270 --> 10:38.570] in this fancy new language called Node.js. [10:38.570 --> 10:44.370] Again, this is early 2013, so Node.js was the new hotness [10:44.370 --> 10:46.830] and everyone was starting to move that direction [10:46.830 --> 10:48.610] with microservices and everything. [10:48.850 --> 10:52.470] It's not on this slide, but we implemented Hashtag version 1 [10:52.470 --> 10:54.270] entirely as microservices. [10:55.130 --> 10:59.110] And I shit you not, there was like 23 individual packages [10:59.110 --> 11:03.210] that comprised Hashtag and almost all of those were microservices. [11:05.350 --> 11:07.890] And we went that direction because that's kind of like [11:08.570 --> 11:09.970] programming zeitgeist at the time, right? [11:09.970 --> 11:12.930] Like everyone was pushing microservices and pushing asynchronous [11:13.370 --> 11:17.170] and, you know, NoSQL and all that other horseshit, right? [11:17.170 --> 11:20.190] And in the end, it created more headaches than not [11:20.190 --> 11:22.810] if we just had a monolithic fucking program. [11:23.490 --> 11:26.970] You know, maintaining like one server binary [11:26.970 --> 11:31.570] and one agent binary is way simpler than like, you know, [11:31.570 --> 11:34.270] maintaining 23 individual microservices and making sure [11:34.270 --> 11:35.990] that they're all synced and communicating properly [11:35.990 --> 11:39.230] and, you know, that the versions are correct, you know, [11:39.230 --> 11:40.810] across all 23 packages. [11:40.950 --> 11:43.310] And there's just a lot of unnecessary headache [11:43.310 --> 11:46.270] that was introduced with that, that, you know, [11:46.270 --> 11:48.510] we attempted to manage. [11:48.870 --> 11:51.430] But even then, like our clients would still kind of mess it up. [11:51.430 --> 11:56.010] You know, like they would upgrade like 10 of 20 packages [11:56.010 --> 11:58.030] and then like the other ones would just be stuck at old versions [11:58.030 --> 12:01.770] or, you know, one demon would die and another demon would know [12:01.770 --> 12:03.470] that it's dead, so then it would shit the bed. [12:03.470 --> 12:09.470] And anyway, so yeah, no JS, microservices, [12:09.470 --> 12:11.270] no SQL, MongoDB, all that shit. [12:11.370 --> 12:13.310] I tried to use all like the latest and greatest everything [12:13.310 --> 12:15.950] because at the time, that seemed like the best idea. [12:17.010 --> 12:19.710] And also, Hashtag was more than just workload distribution. [12:19.710 --> 12:21.290] So a lot of our clients at the time thought [12:21.290 --> 12:25.450] that Hashtag was just a GUI front-end for Hashcat. [12:25.850 --> 12:26.990] And we're like, no, it's not. [12:26.990 --> 12:29.530] It's not just a web UI for Hashcat. [12:29.530 --> 12:33.370] It actually does like, you know, multi-user and, you know, [12:33.470 --> 12:35.970] workload distribution with like a really complex [12:35.970 --> 12:37.630] four-dimensional queuing mechanism. [12:37.630 --> 12:38.830] Like you're trying to explain all this to the client [12:38.830 --> 12:40.370] and they're just like, you know, eyes glazed over [12:40.370 --> 12:42.310] and like, so it's a web UI for Hashcat. [12:42.470 --> 12:45.110] Like fucking yes, okay, it's a web UI for Hashcat. [12:46.090 --> 12:47.590] But it actually did a hell of a lot more than that. [12:47.590 --> 12:50.290] In fact, when you installed Hashtag on your box, [12:50.290 --> 12:53.070] it basically hijacked it and configured it [12:53.070 --> 12:54.990] exactly the way that we needed it to be configured. [12:54.990 --> 12:56.810] It would actually provision the entire server [12:56.810 --> 13:00.570] from just installing like the initial meta package. [13:01.410 --> 13:03.350] So it's a complete stack of packages. [13:03.350 --> 13:05.430] They get stack, Hashtag, that's where it comes from. [13:05.910 --> 13:08.190] So it handled driver installation, [13:08.190 --> 13:10.170] driver configuration, driver updates. [13:10.170 --> 13:11.630] And again, this is back at a time, [13:11.630 --> 13:13.090] and even nowadays still kind of a challenge, [13:13.090 --> 13:18.990] but this is back at a time when GPU drivers [13:18.990 --> 13:20.130] were a pain in the ass to install. [13:20.130 --> 13:21.210] And then once you got them installed, [13:21.210 --> 13:22.570] it was a pain in the ass to update. [13:23.150 --> 13:24.530] So the fact that we kind of handled this [13:24.530 --> 13:27.230] automagically was a pretty big value add. [13:27.970 --> 13:29.490] Also handled XOR configuration, [13:30.870 --> 13:32.070] headless servers. [13:32.350 --> 13:33.550] This is actually during a time [13:33.550 --> 13:35.270] where when you had a headless server, [13:35.270 --> 13:36.650] you had to have fucking dummy plugs [13:36.650 --> 13:38.530] connected to each of the GPUs. [13:38.530 --> 13:40.150] There's probably very few people watching this [13:40.150 --> 13:41.470] who actually remember that. [13:42.330 --> 13:44.250] But yeah, if you had a completely headless server, [13:44.250 --> 13:45.410] each one of those GPUs had to have [13:45.510 --> 13:47.190] a special dummy plug attached to it [13:47.650 --> 13:49.330] that would connect the necessary pins [13:49.330 --> 13:50.630] to trick it to think that there's actually [13:50.770 --> 13:51.850] a monitor attached. [13:51.990 --> 13:53.610] And we actually had to run XOR [13:53.610 --> 13:55.710] because even though these are headless, [14:00.570 --> 14:01.410] they don't have GPU clock rates [14:01.830 --> 14:03.670] or set like power tune profiles [14:03.670 --> 14:04.610] or any of that. [14:04.610 --> 14:06.610] So even though this is a headless server, [14:06.610 --> 14:07.770] we still had to have dummy plugs [14:07.770 --> 14:09.090] to think there's a monitor attached, [14:09.090 --> 14:10.130] still had to run XOR, [14:10.130 --> 14:13.190] and still had to configure XOR properly [14:14.010 --> 14:16.750] to enable us to control the fans [14:16.750 --> 14:17.970] and control the clocks [14:17.970 --> 14:19.630] and all the other shit that we needed to do. [14:20.030 --> 14:22.110] And then along those lines, [14:22.110 --> 14:24.290] Hashtag also did GPU tuning for you [14:24.290 --> 14:26.090] and also automatic overclocking. [14:26.090 --> 14:33.470] And this was kind of a double-edged sword. [14:33.510 --> 14:36.710] So when the R9 290X came out, [14:36.710 --> 14:38.490] at first it was like entirely unusable [14:39.430 --> 14:42.190] because of the way that AMD aggressively [14:42.190 --> 14:45.910] implemented power tune firmware, [14:45.910 --> 14:48.570] I think it was 2.0 at the time. [14:50.470 --> 14:53.490] But the GPUs would throttle so aggressively [14:53.490 --> 14:56.070] that it was like way slower than the 7970. [14:57.130 --> 14:59.050] And it was just essentially unusable. [14:59.050 --> 15:02.310] So then I wrote od6config. [15:05.310 --> 15:06.470] Sorry, did I say PowerTune? [15:06.470 --> 15:08.670] PowerTune is fucking NVIDIA. Overdrive. I'm sorry. [15:09.450 --> 15:10.770] Oh, no, no. PowerTune is AMD. [15:10.810 --> 15:12.570] PowerMizer is NVIDIA. All right, whatever. [15:13.870 --> 15:16.890] So od6config enabled us [15:16.890 --> 15:19.970] to properly configure the R9 290X. [15:20.930 --> 15:22.250] But it was good, [15:22.250 --> 15:23.590] but it was still kind of mediocre. [15:24.190 --> 15:26.010] But at that time, when I measured it, [15:26.090 --> 15:27.910] it was only drawing 300 watts, [15:27.910 --> 15:28.710] which is important [15:29.450 --> 15:31.890] because our server platforms like the Brutalis [15:32.450 --> 15:35.130] can only handle GPUs that drop to 300 watts. [15:35.130 --> 15:36.970] You draw more than that, you're going to blow a fuse [15:36.970 --> 15:38.730] on the PCI slots on the motherboard. [15:40.550 --> 15:44.610] So we had a automatic overclock profile [15:44.610 --> 15:46.730] for the R9 290X that we pushed out to our clients [15:47.830 --> 15:52.410] that made the GPU draw just about 300 watts. [15:52.410 --> 15:56.510] And then a little bit later down the road, [15:56.510 --> 15:58.310] about a year, year and a half later, [15:58.310 --> 16:01.210] AMD pushed out a new driver update [16:01.210 --> 16:03.530] that actually caused the power consumption [16:03.530 --> 16:06.730] to go up by a good 50 to 75 watts. [16:06.730 --> 16:09.270] And we did not know this. [16:09.270 --> 16:11.030] This kind of went undetected by us [16:11.030 --> 16:15.450] until we started getting a rash of clients [16:16.030 --> 16:19.530] who were seeing dead PCIe slots on their motherboards, [16:19.530 --> 16:21.930] and all of them had R9 290X-based solutions. [16:22.910 --> 16:25.230] And I finally busted out a kilowatt [16:25.230 --> 16:27.190] and figured out what the fuck was going on. [16:28.290 --> 16:30.510] But we actually had the ability, [16:30.510 --> 16:35.290] through all these mechanisms that we had in Hashtag [16:35.290 --> 16:38.670] at the time to enable the automatic overclocking [16:38.670 --> 16:40.490] and configuration and all that, [16:40.490 --> 16:43.350] to push out a new underclock profile [16:43.350 --> 16:45.870] to all of our clients to try to drop [16:45.870 --> 16:47.450] the power consumption down to keep it [16:47.450 --> 16:50.170] from blowing PCIe slots on the motherboards. [16:51.770 --> 16:52.970] So it's both good and bad. [16:52.970 --> 16:55.470] The fact that we're able to automatically overclock GPUs [16:56.150 --> 16:59.530] to help get the most speed out of the GPUs is awesome. [17:00.530 --> 17:02.590] But then in that one specific scenario, [17:02.590 --> 17:05.310] it was bad because we started blowing PCIe slot fuses. [17:05.450 --> 17:06.930] But then it was good that we had this functionality [17:06.930 --> 17:09.330] because then we could correct it through software, [17:09.330 --> 17:12.090] over the air, and automatically correct it [17:12.090 --> 17:14.010] on all of our client systems all at the same time. [17:15.590 --> 17:16.670] Which saved us a lot of money [17:16.670 --> 17:19.110] because we were spending hundreds of thousands of dollars [17:19.110 --> 17:20.390] on new motherboards. [17:21.790 --> 17:22.430] So yeah. [17:22.430 --> 17:24.210] And then, of course, Hashtag also had [17:24.210 --> 17:25.830] cluster resource monitoring. [17:28.330 --> 17:30.170] So there were some drawbacks. [17:30.170 --> 17:31.830] Like I said, the entire Godend thing [17:31.830 --> 17:33.050] was written in Node.js, [17:33.050 --> 17:35.350] and while Node.js is asynchronous, [17:35.350 --> 17:38.430] it also runs on a single CPU thread. [17:38.670 --> 17:40.430] That may have changed at some point, [17:40.430 --> 17:41.930] because we're talking like 2013, [17:41.930 --> 17:42.910] that's seven years ago, [17:42.910 --> 17:45.570] and I haven't even touched Node.js since. [17:45.730 --> 17:47.710] But at the time, everything ran [17:49.110 --> 17:49.430] on a single CPU thread, [17:49.430 --> 17:50.010] which was stupid, [17:50.010 --> 17:51.110] because our cluster controllers [17:51.110 --> 17:54.570] had a minimum of 12 CPU cores, [17:54.570 --> 17:55.450] 24 threads, [17:55.450 --> 17:58.230] and we're sitting here stuck in one thread [17:58.850 --> 18:00.790] on the server daemon. [18:01.610 --> 18:02.890] So yeah, any long-running methods [18:02.890 --> 18:04.570] would just totally block [18:04.570 --> 18:05.950] the entire application, [18:05.950 --> 18:09.430] and we had massive scalability issues. [18:10.210 --> 18:13.510] I think the very first alpha [18:13.510 --> 18:15.010] that we released to clients [18:15.010 --> 18:17.510] could only support four nodes. [18:17.510 --> 18:18.750] The second one, [18:18.750 --> 18:21.830] we were able to get up to nine or ten nodes. [18:22.290 --> 18:23.670] Basically, any time a client ordered [18:23.830 --> 18:25.230] a cluster larger than the previous [18:25.230 --> 18:26.950] largest cluster we had ever built, [18:26.950 --> 18:28.950] we ran into some kind of scalability issue, [18:28.950 --> 18:30.130] identified some bottlenecks, [18:30.130 --> 18:31.870] and had to work to remove those bottlenecks. [18:32.870 --> 18:35.350] Unfortunately, 90% of the time, [18:35.350 --> 18:36.370] the bottleneck was the fact [18:36.370 --> 18:38.030] that JavaScript was too slow [18:38.030 --> 18:40.110] for the task that we were trying to do. [18:40.290 --> 18:41.650] So we had to reimplement [18:42.830 --> 18:44.110] those bottleneck methods [18:44.110 --> 18:45.210] in a faster language, [18:45.210 --> 18:47.570] and then just shell out to them. [18:48.350 --> 18:49.590] So keyspace calculation [18:49.590 --> 18:51.070] was reimplemented in C. [18:51.070 --> 18:53.230] Now, calculating keyspace for a mask attack [18:53.230 --> 18:54.550] is really lightweight, [18:54.550 --> 18:56.830] but when you talk about massive dictionaries [18:56.830 --> 18:59.090] or rule files, [18:59.090 --> 19:00.670] trying to count the number of lines [19:00.670 --> 19:02.330] in a file with JavaScript [19:02.330 --> 19:04.510] is painfully slow. [19:05.350 --> 19:06.710] So that's largely why [19:06.710 --> 19:09.270] that entire portion was reimplemented in C, [19:09.270 --> 19:11.190] and we just shelled out to that binary [19:11.810 --> 19:13.170] to eliminate that bottleneck. [19:14.310 --> 19:16.430] We didn't dramatically reduce it. [19:17.850 --> 19:18.590] Regex. [19:18.590 --> 19:20.110] Anything Regex, [19:20.110 --> 19:21.650] we actually implemented, [19:21.650 --> 19:22.850] we had reimplemented it in Perl, [19:22.850 --> 19:25.250] because Perl is amazing at Regex. [19:25.250 --> 19:28.570] JavaScript is not very good at Regex. [19:28.950 --> 19:30.610] The primary thing [19:30.610 --> 19:31.810] that we were doing with Regex here [19:31.810 --> 19:34.990] was validating the hash lists. [19:34.990 --> 19:37.510] So a client uploads a hash list, [19:37.510 --> 19:39.250] and some of our clients [19:39.250 --> 19:40.210] wouldn't even bother throwing [19:40.210 --> 19:42.010] 200 million hashes at hashtag [19:42.010 --> 19:43.570] in one job. [19:43.970 --> 19:45.550] But then hashtag has to sit there [19:45.550 --> 19:50.610] and receive the massively large hash file, [19:50.610 --> 19:53.330] and then validate each individual hash, [19:53.330 --> 19:55.710] and then stuff them in a database. [19:56.130 --> 19:58.270] That was a massive bottleneck. [19:58.270 --> 19:59.010] So we ended up reimplementing [19:59.590 --> 20:00.730] all that in Perl [20:00.730 --> 20:02.830] to try to get that down. [20:02.830 --> 20:04.310] And then the agent, [20:04.310 --> 20:05.710] the entire fucking agent, [20:05.710 --> 20:07.410] we ended up just implementing [20:07.410 --> 20:08.730] the whole thing in Perl. [20:12.710 --> 20:13.930] And the only reason [20:13.930 --> 20:14.790] we picked Perl at the time [20:14.790 --> 20:15.930] is because that's the only language [20:15.930 --> 20:17.250] we could iterate fast enough in [20:17.250 --> 20:19.090] to rewrite the entire damn thing. [20:21.770 --> 20:23.090] But yeah. [20:23.430 --> 20:24.910] Like I said, we also used MongoDB [20:26.090 --> 20:29.750] because NoSQL was the rage at the time. [20:30.070 --> 20:32.590] And out of all the NoSQL databases we tested, [20:32.590 --> 20:34.450] Mongo was the most performant. [20:35.010 --> 20:36.790] But then we had clients do things [20:36.790 --> 20:39.190] like throw 200 million hashes at hashtag. [20:39.950 --> 20:41.490] And that would cause [20:42.970 --> 20:44.330] some not only bottlenecks [20:45.290 --> 20:47.110] and blocking processes, [20:47.110 --> 20:51.390] but some really UI weirdness as well. [20:51.530 --> 20:53.270] They would go through the process [20:53.270 --> 20:54.490] of creating a job, [20:54.490 --> 20:55.850] uploading the hash list, [20:55.850 --> 20:56.890] and they would just sit there [20:56.890 --> 20:57.790] with the little spinner [20:58.690 --> 21:01.350] for a while until it timed out. [21:01.830 --> 21:03.510] So they'd go to create the job again [21:03.510 --> 21:04.650] because it timed out. [21:04.650 --> 21:06.510] And then they'd go back 10 minutes later [21:09.190 --> 21:10.590] and it was actually active and running. [21:10.590 --> 21:12.430] But because they went through the process four times [21:12.430 --> 21:13.390] trying to get it to start, [21:13.390 --> 21:14.910] now they have like four of the same process [21:14.910 --> 21:16.090] running on the cluster. [21:16.470 --> 21:19.190] So we threw Redis in front of Mongo [21:19.190 --> 21:21.930] as an in-memory caching layer. [21:22.290 --> 21:24.330] And that actually had a significant impact, [21:24.330 --> 21:25.670] like positive impact. [21:25.850 --> 21:27.910] And it enabled us to ingest hashes [21:28.110 --> 21:29.230] a lot faster. [21:30.930 --> 21:34.570] But still not a very great solution overall. [21:36.190 --> 21:38.510] The Web UI client was implemented [21:38.510 --> 21:41.310] as a single-page application in Angular. [21:41.750 --> 21:43.470] It was really clunky. [21:43.470 --> 21:44.670] It was really heavy. [21:44.670 --> 21:46.090] It was really slow. [21:46.110 --> 21:48.350] It constantly caused the browser to freeze [21:48.350 --> 21:50.830] if you left it up for more than 30 minutes. [21:50.930 --> 21:52.910] And it just had a ton of bugs. [21:53.390 --> 21:54.790] And I think probably the most annoying thing [21:54.790 --> 21:56.370] is if you're like a hashcat power user [21:56.370 --> 21:57.770] and you sit down and try to use [21:57.770 --> 21:59.670] the hashtag Web UI, [21:59.670 --> 22:02.610] it was just cumbersome as fuck. [22:03.010 --> 22:04.550] There's no way to move quickly [22:04.570 --> 22:06.390] in that UI. [22:06.550 --> 22:07.810] And it gets very repetitive [22:07.810 --> 22:09.290] and very tedious. [22:10.370 --> 22:12.390] So there was a lot of drawbacks [22:12.390 --> 22:14.950] to having that Angular Web UI. [22:15.250 --> 22:16.770] I absolutely hated it. [22:16.770 --> 22:20.370] And anyone who was really experienced [22:20.370 --> 22:22.310] with cracking and hashcat and stuff, [22:22.310 --> 22:23.550] they also hated it [22:23.550 --> 22:26.910] and begged us not to have that again. [22:27.570 --> 22:29.270] And we were just severely limited [22:29.270 --> 22:30.830] by what we could do in a browser. [22:31.410 --> 22:34.130] For non-hash formats at the time, [22:34.130 --> 22:36.390] we were scraping everything server-side. [22:36.390 --> 22:38.170] So you would actually upload a PDF [22:38.170 --> 22:39.610] or upload a .docx [22:39.610 --> 22:41.870] or whatever, [22:41.870 --> 22:45.590] and we would scrape the file server-side [22:45.590 --> 22:49.190] to get the necessary bits, [22:49.190 --> 22:51.630] like Office to hashcat [22:51.630 --> 22:52.870] and PDF to hashcat, [22:52.870 --> 22:53.910] shit like that. [22:54.530 --> 22:56.190] The problem then became, [22:56.190 --> 22:58.310] what do we do with things like [22:58.310 --> 22:59.510] TrueCrypt volumes [22:59.510 --> 23:02.990] or really large 7-zip volumes [23:02.990 --> 23:04.690] or RAR files? [23:04.690 --> 23:06.110] Are you really going to upload [23:06.750 --> 23:09.410] a 100-gigabyte disk image [23:09.410 --> 23:10.370] through the browser [23:10.370 --> 23:12.170] just so we can scrape off [23:12.850 --> 23:14.390] two kilobytes? [23:14.390 --> 23:17.010] No, that doesn't make any sense. [23:19.690 --> 23:21.710] So yeah, the browser was an issue. [23:21.710 --> 23:24.450] And then we implemented access controls, [23:24.450 --> 23:25.970] but there was nothing fine-grained. [23:25.970 --> 23:28.770] We assumed that because everyone [23:28.770 --> 23:30.790] operating the hashcat cluster [23:30.790 --> 23:32.210] was on the same team, [23:32.990 --> 23:35.710] that they all had equal rights [23:35.710 --> 23:37.190] and equal access to the cluster, [23:37.190 --> 23:38.770] and therefore all users are admins. [23:38.770 --> 23:42.010] We didn't have different roles or anything. [23:42.010 --> 23:44.350] It was just, if you have an account, [23:44.350 --> 23:46.650] then you obviously have a right to be there, [23:46.650 --> 23:48.410] so therefore you are an admin. [23:50.970 --> 23:52.590] And I think some of our clients [23:52.590 --> 23:53.390] didn't have a problem with that, [23:53.390 --> 23:54.970] because everyone on this team [23:54.970 --> 23:57.290] does have equal rights and access [23:57.290 --> 23:58.510] and privilege to the cluster, [23:58.510 --> 23:59.730] so this is totally fine. [23:59.730 --> 24:01.110] Other clients were like, [24:01.110 --> 24:02.390] this is absolutely unacceptable. [24:04.810 --> 24:06.210] And I eventually got to the point [24:06.210 --> 24:09.010] where the old version one code [24:09.010 --> 24:10.970] was completely unmanageable. [24:11.370 --> 24:12.550] Even just implementing [24:12.550 --> 24:14.350] what should have been a minor bug fix [24:14.350 --> 24:18.210] ended up requiring major refactoring. [24:19.890 --> 24:20.990] It just wasn't, [24:20.990 --> 24:21.850] because we threw it together [24:21.850 --> 24:22.870] in less than two months, [24:22.870 --> 24:25.450] so we didn't think of everything in advance [24:25.450 --> 24:27.130] that would come up, [24:29.730 --> 24:30.770] and try to implement new features, [24:30.770 --> 24:32.190] and try to work around [24:32.190 --> 24:35.550] some of the bugs in there. [24:35.550 --> 24:36.550] So we decided we needed [24:36.550 --> 24:37.970] an entire ground-up rewipe [24:37.970 --> 24:39.910] for Hashtag version two. [24:41.630 --> 24:43.190] And at some point, [24:43.190 --> 24:43.970] I got the idea [24:44.950 --> 24:47.330] to create a commercial fork of Hashcat [24:47.330 --> 24:49.650] with native distributed capabilities. [24:49.850 --> 24:50.670] And at the time, [24:50.670 --> 24:52.690] I was so fucking stoked on this idea. [24:52.690 --> 24:54.610] It seemed like the best idea [24:55.170 --> 24:57.090] in the world to me at the time. [24:58.050 --> 25:01.330] So we set out to do just that. [25:02.150 --> 25:03.730] Now this is right [25:04.530 --> 25:05.210] before [25:05.890 --> 25:07.810] Hashtag went open source, [25:07.810 --> 25:09.930] and then we started work on this [25:09.930 --> 25:11.510] literally the minute [25:11.510 --> 25:13.890] that Hashtag went open source. [25:14.450 --> 25:16.210] So we implemented in Golang [25:16.210 --> 25:17.030] in C [25:18.010 --> 25:20.290] with Postgres database. [25:20.510 --> 25:21.730] Now we chose Golang [25:21.730 --> 25:23.290] because of high concurrency [25:23.950 --> 25:26.910] with goroutines, right? [25:27.090 --> 25:28.970] So with Node.js, [25:28.970 --> 25:31.150] everything ran on a single CPU thread. [25:31.150 --> 25:31.930] And we wanted [25:31.930 --> 25:34.490] the opposite of that for version two, right? [25:34.490 --> 25:35.990] We're going to learn the lesson on that. [25:35.990 --> 25:37.810] So we wanted [25:38.290 --> 25:39.770] just lots and lots of goroutines [25:39.770 --> 25:40.610] to run everything [25:40.610 --> 25:43.450] so that we'd use all 24 threads [25:43.450 --> 25:44.910] plus on the box. [25:46.870 --> 25:48.510] And Golang actually also [25:49.490 --> 25:51.130] allows for pretty rapid development. [25:51.130 --> 25:53.010] So we thought it was a pretty good choice. [25:54.850 --> 25:55.930] But we still stuck [25:55.930 --> 25:57.370] with the traditional client server [25:57.370 --> 25:58.690] agent architecture. [25:58.730 --> 26:01.570] We have numerous clients [26:01.570 --> 26:03.150] and then one server [26:03.150 --> 26:05.250] and then however many agents. [26:06.350 --> 26:07.570] Meaning that that server [26:07.570 --> 26:09.230] is both a bottleneck [26:09.230 --> 26:10.630] and a single point of failure. [26:11.910 --> 26:13.550] So the first thing we had to do [26:13.550 --> 26:15.430] is we had to split Hashcat in two. [26:15.990 --> 26:16.930] We took everything [26:16.930 --> 26:18.230] that had to do with [26:18.230 --> 26:21.290] actually starting a cracking job [26:21.290 --> 26:23.910] and executing an OpenCL kernel [26:23.910 --> 26:24.550] on a GPU [26:25.930 --> 26:26.150] with the agent [26:26.150 --> 26:27.610] and nothing else. [26:27.610 --> 26:29.830] The agent was just barebones [26:29.830 --> 26:31.030] trimmed down Hashcat [26:32.830 --> 26:34.210] launch attack on GPU [26:34.210 --> 26:35.990] and that was it, nothing more. [26:36.410 --> 26:37.490] And everything else [26:37.490 --> 26:39.750] we placed into the server library. [26:40.290 --> 26:42.350] And then for any code [26:42.350 --> 26:43.910] that wasn't already in Hashcat [26:43.910 --> 26:45.770] such as the JSON API [26:45.770 --> 26:47.510] and the [26:48.510 --> 26:49.110] protobufs [26:49.110 --> 26:50.630] for agent server communication [26:50.630 --> 26:51.810] workload distribution [26:55.930 --> 26:56.730] fine-grained [26:56.730 --> 26:59.390] multi-role access control. [26:59.470 --> 27:00.850] All that was implemented in Go [27:00.850 --> 27:03.650] and then we just linked against the C libraries. [27:03.650 --> 27:05.270] And this effort actually [27:05.270 --> 27:07.150] laid the foundation for [27:08.130 --> 27:10.410] the Hashcat 4.0 refactor [27:10.410 --> 27:12.150] with libHashcat. [27:12.610 --> 27:14.910] And then because of how clunky [27:14.910 --> 27:16.510] our old web UI was [27:16.510 --> 27:19.130] we actually ditched it entirely [27:19.130 --> 27:20.890] for a cross-platform CLI [27:20.890 --> 27:23.410] which made our power users [27:23.410 --> 27:24.590] like really happy [27:25.930 --> 27:28.290] but our less experienced users [27:28.290 --> 27:30.330] were not very thrilled [27:30.330 --> 27:31.390] with that decision. [27:31.390 --> 27:34.310] They still wanted a GUI [27:34.310 --> 27:36.830] which, fair enough. [27:38.090 --> 27:40.090] So our plan initially [27:40.090 --> 27:42.350] was to maintain both Hashcat [27:42.350 --> 27:44.590] and Hashstack trees in parallel. [27:44.830 --> 27:45.330] And then [27:46.570 --> 27:47.930] if there was any features [27:47.930 --> 27:49.230] that we added to Hashstack [27:49.230 --> 27:50.890] or any new kernels that we wrote [27:50.890 --> 27:52.850] we would just backport those [27:52.850 --> 27:55.510] as we chose to Hashcat. [27:58.550 --> 27:59.770] This didn't really work out [27:59.770 --> 28:00.790] very well in practice though. [28:00.790 --> 28:02.430] So when we released version 2 [28:02.430 --> 28:04.750] it was in sync with Hashcat 3.5 [28:05.230 --> 28:06.370] but it didn't stay that way [28:06.370 --> 28:07.370] for very long. [28:08.410 --> 28:09.510] And of course we had to add [28:09.610 --> 28:10.530] a cross-platform GUI [28:10.530 --> 28:11.290] to the roadmap as well [28:11.290 --> 28:12.790] because we had significant [28:12.790 --> 28:13.870] feedback from our clients [28:13.870 --> 28:15.390] that they really wanted [28:15.390 --> 28:16.290] the goddamn GUI [28:16.290 --> 28:16.990] and they were not happy [28:16.990 --> 28:18.370] that we took away the web UI. [28:19.890 --> 28:20.770] It turns out [28:20.770 --> 28:22.130] this was the single dumbest [28:22.130 --> 28:23.350] motherfucking thing I've ever [28:23.350 --> 28:24.450] done in my life. [28:25.590 --> 28:27.390] I completely underestimated [28:27.390 --> 28:28.450] how much work was involved [28:28.450 --> 28:30.410] in creating a commercial fork [28:30.410 --> 28:31.370] of Hashcat. [28:32.010 --> 28:33.230] Like, genuinely [28:36.470 --> 28:36.970] I... [28:37.770 --> 28:39.510] Yeah, I really don't know [28:39.510 --> 28:40.910] how to convey how fucking stupid [28:40.910 --> 28:42.310] of an idea this was. [28:43.110 --> 28:45.570] We spent like 100% [28:45.570 --> 28:46.650] of our time just backporting [28:47.210 --> 28:48.930] things from Hashcat into Hashtag [28:48.930 --> 28:50.110] instead of actually adding new [28:50.110 --> 28:51.650] features or writing new kernels. [28:52.290 --> 28:54.030] Like, literally months of work [28:54.550 --> 28:55.610] you know, just backporting [28:56.930 --> 28:58.130] Hashcat commits. [28:58.870 --> 29:00.690] And it got really frustrating. [29:00.910 --> 29:01.490] You know, it's like, [29:01.490 --> 29:02.650] hey, are we working on these [29:02.650 --> 29:03.690] tickets? Are we working on these [29:03.690 --> 29:04.930] features? No, we're working on [29:04.930 --> 29:05.770] backports. [29:05.990 --> 29:06.950] You know, I got so sick of [29:06.950 --> 29:07.930] hearing working on backports, [29:07.930 --> 29:09.330] working on backports, like [29:10.290 --> 29:11.230] when are we ever going to be [29:11.230 --> 29:12.230] done with backports? [29:13.430 --> 29:15.590] And then when Adam refactored [29:15.590 --> 29:17.950] Hashcat for 4.0, that also [29:17.950 --> 29:19.390] meant that we had to refactor [29:19.390 --> 29:20.990] Hashtag as well. [29:21.150 --> 29:22.550] And we hadn't even finished [29:22.550 --> 29:23.750] implementing all the backports [29:23.750 --> 29:25.410] at the speed that Adam was, you [29:25.410 --> 29:26.250] know, committing things to [29:26.250 --> 29:27.050] Hashcat. [29:27.170 --> 29:28.710] So this was just like, [29:28.710 --> 29:29.250] absolutely [29:30.510 --> 29:32.130] unmaintainable, right? [29:32.130 --> 29:34.250] Like, fuck me. [29:35.550 --> 29:36.870] And then there was also this [29:36.870 --> 29:38.010] other little minor problem [29:38.950 --> 29:40.610] where we picked the wrong [29:40.610 --> 29:41.570] HTTP library [29:42.230 --> 29:43.450] for the API. [29:44.130 --> 29:45.310] And this basically [29:45.310 --> 29:46.750] limited the service capability [29:46.750 --> 29:49.770] to about 12 nodes. [29:49.790 --> 29:51.290] Now, 12 nodes [29:51.290 --> 29:52.830] might seem like a lot. [29:52.830 --> 29:54.510] That's 96 GPUs. [29:54.510 --> 29:55.270] You know, that's a pretty [29:55.270 --> 29:57.670] respectable size cluster. [29:58.890 --> 30:01.310] The problem is this [30:01.310 --> 30:03.050] particular client had [30:03.050 --> 30:05.950] ordered 320 GPUs, [30:05.950 --> 30:07.230] 40 nodes. [30:07.510 --> 30:09.190] And you can't really [30:09.190 --> 30:11.250] deliver a 40 node client [30:11.250 --> 30:14.290] to a cluster with a [30:14.290 --> 30:15.970] server that can only [30:15.970 --> 30:17.230] really handle [30:17.230 --> 30:22.360] 12 nodes. [30:22.360 --> 30:23.300] So this required [30:24.680 --> 30:26.280] not a complete rewrite, [30:26.440 --> 30:27.840] but a damn near complete [30:27.840 --> 30:28.320] rewrite time. [30:28.320 --> 30:29.500] Fuck, because we use that [30:29.500 --> 30:30.300] HTTP library [30:31.780 --> 30:32.620] extensively throughout the [30:32.620 --> 30:34.400] code. [30:34.400 --> 30:35.440] And there was no, like, drop [30:35.440 --> 30:36.140] in replacement. [30:36.140 --> 30:37.200] So basically anywhere where [30:37.200 --> 30:38.800] we were dealing with [30:38.800 --> 30:39.880] HTTP, [30:39.880 --> 30:41.220] we had to rip out the old [30:41.220 --> 30:42.800] library and then, [30:42.800 --> 30:43.200] you know, [30:44.060 --> 30:44.960] add in all the stuff for [30:44.960 --> 30:45.840] the new library. [30:45.960 --> 30:47.000] But they did resolve the [30:47.000 --> 30:47.540] issue. [30:48.320 --> 30:49.500] It was just really painful [30:49.500 --> 30:50.540] at the time to have to [30:50.540 --> 30:51.400] mess with that. [30:53.420 --> 30:54.600] But the server [30:54.600 --> 30:56.740] still continued to be a [30:56.740 --> 30:57.780] single point of failure [30:57.780 --> 30:58.820] and still proved to be a [30:58.820 --> 31:00.180] source of bottlenecks. [31:00.180 --> 31:01.540] As we, you know, [31:01.540 --> 31:02.150] tried to build [31:03.220 --> 31:04.360] larger and larger [31:05.160 --> 31:06.000] clusters, [31:06.580 --> 31:07.000] we [31:07.000 --> 31:07.640] kept running into [31:07.640 --> 31:09.160] problems with the [31:09.640 --> 31:10.060] server [31:10.200 --> 31:10.660] being a bottleneck [31:10.760 --> 31:11.380] and a single point of [31:11.380 --> 31:12.340] failure. [31:12.340 --> 31:13.080] And we also had clients [31:13.080 --> 31:14.200] coming to us asking us [31:14.640 --> 31:15.520] if they could order [31:15.520 --> 31:16.400] multiple cluster [31:16.400 --> 31:17.780] controllers and cluster [31:17.780 --> 31:18.800] them together, [31:18.800 --> 31:19.240] you know, either [31:19.240 --> 31:19.960] high availability [31:19.960 --> 31:21.860] or, like, [31:21.860 --> 31:22.700] load balance them [31:22.700 --> 31:23.540] or something. [31:24.560 --> 31:24.980] And [31:26.680 --> 31:27.660] we didn't have a [31:27.660 --> 31:28.380] solution for that, [31:28.380 --> 31:29.140] but I started thinking [31:29.280 --> 31:32.300] about a solution for that. [31:32.300 --> 31:33.220] And again, like, [31:33.220 --> 31:33.980] 40% of our clients [31:33.980 --> 31:34.660] were just completely [31:34.660 --> 31:35.620] pissed that we had no [31:35.620 --> 31:40.280] GUI. [31:40.280 --> 31:40.480] So [31:40.480 --> 31:41.340] it got to the point [31:41.340 --> 31:42.020] where it's like, [31:41.080 --> 31:42.080] let's just [31:42.540 --> 31:44.080] rethink, like, [31:44.080 --> 31:45.000] the entire fucking thing [31:45.000 --> 31:46.140] and just, like, [31:46.140 --> 31:47.000] go as far away from [31:47.000 --> 31:48.080] these mistakes as we [31:48.080 --> 31:49.160] could possibly get. [31:49.220 --> 31:50.080] Like, we never, ever [31:50.080 --> 31:50.780] want to make any of [31:50.780 --> 31:51.860] these mistakes ever again. [31:51.860 --> 31:52.420] We've learned a lot of [31:52.420 --> 31:53.960] lessons from, you know, [31:53.960 --> 31:54.720] what we've done over [31:54.720 --> 31:55.920] the last seven years. [31:57.340 --> 31:58.640] So, but we need [31:58.640 --> 31:59.360] more than just a [31:59.360 --> 32:00.120] complete ground-up rewrite. [32:00.120 --> 32:01.220] Like, we need an entire [32:01.220 --> 32:01.640] paradigm shift. [32:01.640 --> 32:03.100] Like, we need to [32:03.100 --> 32:04.020] rethink this entire [32:05.480 --> 32:05.880] problem [32:06.640 --> 32:07.040] entirely [32:07.600 --> 32:08.000] and [32:08.000 --> 32:08.620] come up with [32:08.620 --> 32:09.300] something that's [32:09.300 --> 32:09.980] completely different [32:09.980 --> 32:10.460] from what's ever been [32:10.460 --> 32:11.120] done. [32:11.940 --> 32:12.820] So we need to [32:12.820 --> 32:13.740] eliminate the single [32:13.740 --> 32:14.400] point of failure of [32:14.400 --> 32:15.180] the server. [32:15.300 --> 32:16.640] We need to enable, [32:16.640 --> 32:17.600] like, actual infinite [32:17.600 --> 32:18.440] horizontal scaling [32:18.440 --> 32:19.440] because, again, [32:19.440 --> 32:20.480] we're, you know, [32:20.480 --> 32:21.740] we always try to [32:21.740 --> 32:23.440] strive for, like, [32:23.440 --> 32:24.460] warehouse as a [32:24.460 --> 32:25.360] computer, right? [32:25.360 --> 32:26.900] We're looking at [32:26.900 --> 32:27.560] warehouse scale [32:27.560 --> 32:28.300] computing as, like, [32:28.300 --> 32:28.880] the target that we're [32:28.880 --> 32:31.360] trying to hit. [32:31.360 --> 32:32.660] There should never be [32:32.660 --> 32:34.860] any, you know, [32:34.860 --> 32:35.560] limiting factors [32:35.560 --> 32:36.540] outside of budget [32:36.540 --> 32:38.280] for, like, how many [32:38.280 --> 32:38.860] nodes we can [32:38.860 --> 32:39.280] support in [32:39.280 --> 32:39.560] HashtagWrite. [32:40.220 --> 32:41.160] And then, of course, [32:41.160 --> 32:42.040] the other big requirement [32:42.040 --> 32:42.700] is we have to actually [32:42.700 --> 32:43.560] move at the pace of [32:43.560 --> 32:44.740] Hashtag development. [32:46.040 --> 32:47.620] We have to, [32:47.620 --> 32:48.380] you know, as Adam [32:48.380 --> 32:49.500] commits things to [32:49.500 --> 32:50.420] Hashtag, we need to [32:50.420 --> 32:51.040] have those [32:51.960 --> 32:53.280] not instantly available [32:53.280 --> 32:54.440] in Hashtag, but within [32:54.600 --> 32:55.280] a very reasonable [32:55.280 --> 32:56.260] time frame. [32:58.880 --> 32:59.320] So [33:00.000 --> 33:01.000] the very first [33:01.000 --> 33:01.760] idea that we came up [33:01.760 --> 33:02.620] with was let's not [33:02.620 --> 33:03.540] do a traditional [33:04.540 --> 33:06.000] client-server-agent [33:06.740 --> 33:07.460] model. [33:08.440 --> 33:09.540] And let's not even [33:09.540 --> 33:10.340] try to cluster the [33:10.340 --> 33:11.400] cluster controllers. [33:11.740 --> 33:12.780] Let's just eliminate [33:12.780 --> 33:13.620] the cluster controllers [33:13.620 --> 33:14.800] entirely and make [33:14.800 --> 33:15.580] everything peer-to-peer [33:15.580 --> 33:16.740] where all nodes [33:16.740 --> 33:17.160] are equal [33:17.860 --> 33:18.620] and there are no [33:18.620 --> 33:19.860] dedicated servers. [33:20.860 --> 33:21.100] And [33:21.920 --> 33:22.960] the biggest requirement [33:22.960 --> 33:23.860] to this is that [33:23.860 --> 33:24.660] work can be submitted [33:24.660 --> 33:25.760] to any node. [33:26.460 --> 33:27.540] So let's say you have, [33:27.540 --> 33:28.620] like, you know, 20 users [33:28.620 --> 33:29.780] and you have, [33:29.780 --> 33:31.120] you know, [33:31.120 --> 33:32.000] 10 boxes. [33:32.000 --> 33:32.780] Like, there shouldn't [33:32.780 --> 33:33.720] be just one node that [33:33.720 --> 33:34.900] all of them have to hit. [33:34.900 --> 33:35.700] They should be able [33:35.700 --> 33:36.420] to hit any node [33:36.420 --> 33:37.160] in the cluster [33:37.160 --> 33:38.400] and, you know, have [33:38.400 --> 33:39.160] the same view [33:39.160 --> 33:40.600] from any node. [33:41.880 --> 33:42.780] And it also needs to [33:42.780 --> 33:43.400] be more user-friendly [33:43.400 --> 33:44.140] than ever. [33:44.400 --> 33:45.400] And also needs to [33:45.400 --> 33:46.340] run on Windows. [33:48.100 --> 33:48.960] I'll tell you why [33:48.960 --> 33:49.640] in a minute. [33:52.240 --> 33:53.380] So set out to, [33:53.380 --> 33:54.120] like, figure out [33:54.120 --> 33:54.660] how we're going to [33:54.660 --> 33:55.320] implement this. [33:55.320 --> 33:55.820] Because we have the [33:55.820 --> 33:57.280] high-level requirements, [33:57.280 --> 33:58.040] right? Like, we know [33:58.040 --> 33:58.760] it needs to be [33:58.760 --> 33:59.380] peer-to-peer. [33:59.380 --> 34:00.140] We know all the nodes [34:00.140 --> 34:01.060] need to be equal. [34:01.240 --> 34:02.520] You know, all that shit. [34:03.040 --> 34:03.800] But how are we actually [34:03.800 --> 34:04.740] going to do this? [34:05.600 --> 34:05.960] So [34:06.840 --> 34:07.560] pretty obviously [34:07.560 --> 34:08.060] we're going to need [34:08.060 --> 34:08.540] some sort of [34:08.540 --> 34:09.880] distributed state machine. [34:10.040 --> 34:10.680] Right? [34:11.460 --> 34:12.400] And that's just [34:12.400 --> 34:13.280] straightforward [34:13.280 --> 34:14.520] and self-explanatory. [34:14.780 --> 34:16.360] But, you know, [34:16.360 --> 34:17.180] like, we just [34:17.180 --> 34:17.600] throw a message [34:17.600 --> 34:18.040] queue at it [34:18.040 --> 34:18.520] and some PubSub [34:18.520 --> 34:20.400] and, you know, [34:20.400 --> 34:21.500] Bob's your uncle. [34:22.160 --> 34:22.860] But what about [34:22.860 --> 34:23.860] disagreements? [34:24.060 --> 34:25.460] Or, you know, [34:25.460 --> 34:26.060] maybe even more [34:26.060 --> 34:26.760] than disagreements, [34:26.760 --> 34:27.360] what about, like, [34:27.360 --> 34:28.680] an actual, you know, [34:28.680 --> 34:29.060] rogue [34:29.900 --> 34:31.100] cluster member? [34:31.220 --> 34:33.100] Or, you know, [34:33.980 --> 34:34.380] a broken [34:34.820 --> 34:35.520] cluster node [34:35.520 --> 34:36.560] that's maybe not [34:36.560 --> 34:36.980] necessarily [34:36.980 --> 34:37.340] intentionally [34:37.340 --> 34:38.180] malicious, [34:38.180 --> 34:39.000] but it certainly, [34:39.000 --> 34:39.300] you know, [34:39.300 --> 34:41.180] has, you know, [34:41.180 --> 34:41.820] the appearance of [34:41.820 --> 34:42.700] malicious behavior. [34:42.700 --> 34:43.500] Right? [34:44.620 --> 34:45.600] So I started [34:45.600 --> 34:46.320] coming up with [34:46.940 --> 34:47.560] solutions for [34:47.560 --> 34:48.060] this. [34:48.680 --> 34:49.280] And [34:49.980 --> 34:50.960] I somehow [34:50.960 --> 34:51.740] got into my [34:51.740 --> 34:53.220] head, like, [34:53.220 --> 34:54.200] dude, [34:54.200 --> 34:54.860] what if [34:54.860 --> 34:55.600] computers could [34:55.600 --> 34:56.580] vote? [34:57.000 --> 34:57.840] Right? [34:57.840 --> 34:58.980] Like, [35:01.520 --> 35:02.360] you just [35:02.360 --> 35:02.840] make it to [35:02.840 --> 35:03.300] where, like, [35:03.300 --> 35:03.580] you know, [35:03.580 --> 35:04.420] if there's a [35:04.420 --> 35:05.260] dispute, [35:06.100 --> 35:06.440] then one of [35:06.440 --> 35:06.920] the nodes, [35:06.920 --> 35:07.520] like, [35:07.520 --> 35:08.140] requests a vote, [35:08.140 --> 35:08.740] right? [35:09.060 --> 35:10.160] And so they take a vote [35:10.160 --> 35:11.060] and majority wins. [35:11.060 --> 35:11.820] And if there's a tie, [35:11.820 --> 35:12.360] like we have an even [35:12.360 --> 35:13.200] number of nodes in the [35:13.200 --> 35:14.600] cluster, then, [35:14.600 --> 35:14.700] like, [35:14.700 --> 35:15.420] literally fucking [35:15.420 --> 35:16.640] flip a coin, [35:16.640 --> 35:17.140] you know, [35:17.140 --> 35:17.640] randomly, [35:17.640 --> 35:18.500] or play rock, [35:18.500 --> 35:18.720] paper, [35:18.720 --> 35:19.220] scissors. [35:19.360 --> 35:19.980] Like, [35:19.980 --> 35:20.780] why not? [35:22.100 --> 35:22.960] And if a peer [35:22.960 --> 35:23.660] doesn't accept the [35:23.660 --> 35:24.440] results, like, [35:24.440 --> 35:24.880] then the rest of [35:24.880 --> 35:25.400] the cluster just [35:25.400 --> 35:26.380] shuns that peer. [35:26.800 --> 35:27.400] And I started [35:27.400 --> 35:27.820] thinking about, [35:27.820 --> 35:27.940] like, [35:27.940 --> 35:28.920] is this actually, [35:28.920 --> 35:29.040] like, [35:29.100 --> 35:29.380] a thing? [35:29.380 --> 35:29.500] Like, [35:29.500 --> 35:30.080] can we actually do [35:30.080 --> 35:30.380] this? [35:30.380 --> 35:30.860] Am I, [35:30.860 --> 35:31.160] like, [35:31.160 --> 35:31.760] completely fucking [35:31.760 --> 35:32.400] stoned? [35:34.600 --> 35:35.420] It turns out that [35:35.420 --> 35:35.780] I really didn't have [35:35.780 --> 35:36.160] to put that much [35:36.160 --> 35:36.640] more thought into [35:36.640 --> 35:37.340] it because this [35:37.340 --> 35:38.120] already exists. [35:38.120 --> 35:38.320] Like, [35:38.320 --> 35:39.180] what I thought up [35:39.660 --> 35:40.820] is actually a [35:40.820 --> 35:41.700] thing called the [35:41.700 --> 35:42.480] wrapped consensus [35:42.480 --> 35:43.940] protocol. [35:43.940 --> 35:45.040] Now, [35:45.040 --> 35:45.920] it's cool that it [35:45.920 --> 35:46.420] already exists because [35:46.420 --> 35:46.880] that, [35:46.880 --> 35:47.220] you know, [35:47.220 --> 35:47.460] means we don't [35:47.460 --> 35:47.700] have to do a [35:47.700 --> 35:48.080] whole lot to [35:48.080 --> 35:48.520] implement it. [35:48.520 --> 35:49.140] But it's also cool [35:49.140 --> 35:49.980] because it kind of [35:50.480 --> 35:51.200] validated, [35:51.200 --> 35:51.940] you know, [35:51.940 --> 35:52.860] my ideas and theories [35:52.860 --> 35:53.560] on how to make this [35:53.560 --> 35:54.260] work. [35:55.080 --> 35:55.980] And told me that I [35:55.980 --> 35:56.260] wasn't just [35:56.260 --> 35:56.780] completely fucking [35:56.780 --> 35:57.400] insane. [35:58.380 --> 35:59.600] And we found [35:59.600 --> 36:00.800] several implementations [36:00.800 --> 36:01.420] of the wrapped [36:01.420 --> 36:02.460] consensus protocol. [36:03.640 --> 36:04.860] And the one that [36:04.860 --> 36:05.900] we ended up liking [36:05.900 --> 36:06.540] the most and [36:06.540 --> 36:07.500] settled on is [36:07.500 --> 36:08.100] Akka, [36:08.100 --> 36:08.960] which runs on [36:08.960 --> 36:09.620] the JVM. [36:09.620 --> 36:09.980] It's written in [36:09.980 --> 36:10.660] Scala. [36:11.140 --> 36:11.920] Now, [36:12.580 --> 36:14.180] I hate Java. [36:14.180 --> 36:15.020] I have shunned [36:15.020 --> 36:16.980] Java for damn [36:16.980 --> 36:17.640] near my entire [36:17.640 --> 36:18.500] life. [36:18.740 --> 36:19.100] You know, [36:19.100 --> 36:20.300] I completely [36:20.300 --> 36:21.040] condemn [36:22.280 --> 36:23.700] Oracle JVM. [36:26.660 --> 36:27.700] But, [36:27.700 --> 36:28.860] there is [36:28.860 --> 36:29.900] OpenJDK. [36:29.900 --> 36:30.420] We don't have [36:30.420 --> 36:31.360] to use Oracle. [36:31.580 --> 36:33.020] And we don't [36:33.020 --> 36:33.400] have to write [36:33.400 --> 36:34.160] in Java. [36:34.160 --> 36:34.620] We can use [36:34.620 --> 36:35.060] Scala and [36:35.060 --> 36:35.860] Kotlin. [36:36.780 --> 36:37.420] And since [36:37.420 --> 36:37.840] I've started [36:37.840 --> 36:38.200] to work on [36:38.200 --> 36:38.480] this, [36:38.480 --> 36:39.200] I actually [36:39.200 --> 36:40.020] really like [36:40.020 --> 36:40.920] Kotlin. [36:41.780 --> 36:42.600] I don't [36:42.600 --> 36:43.500] really have too [36:43.500 --> 36:43.940] many negative [36:43.940 --> 36:44.480] things to say [36:44.480 --> 36:44.860] about it. [36:44.860 --> 36:45.160] It's actually [36:45.160 --> 36:46.040] kind of fun [36:46.040 --> 36:47.180] to work with. [36:48.420 --> 36:48.720] And then [36:48.720 --> 36:49.620] Akka also [36:49.620 --> 36:50.420] has some [36:51.400 --> 36:52.220] plugins or [36:52.220 --> 36:54.000] modules that, [36:54.780 --> 36:55.760] you know, [36:55.760 --> 36:56.600] help us even [36:56.600 --> 36:57.160] further. [36:57.160 --> 36:57.620] Such as [36:57.620 --> 36:57.880] Akka [36:57.880 --> 36:58.880] persistence. [36:58.880 --> 36:59.260] What Akka [36:59.260 --> 37:00.040] persistence does, [37:00.040 --> 37:00.440] it takes our [37:00.440 --> 37:01.080] distributed state [37:01.080 --> 37:02.100] machine, [37:02.100 --> 37:03.000] which is [37:03.000 --> 37:03.200] normally [37:03.200 --> 37:03.700] resident in [37:03.700 --> 37:04.020] memory, [37:04.020 --> 37:04.480] and persists [37:04.480 --> 37:05.320] it to [37:05.320 --> 37:06.120] disk. [37:06.600 --> 37:07.040] And then [37:07.040 --> 37:08.020] Akka [37:08.020 --> 37:08.380] distributed [37:08.380 --> 37:08.860] data, of [37:08.860 --> 37:09.180] course, would [37:09.180 --> 37:09.660] be like what [37:09.660 --> 37:10.500] implements the [37:10.500 --> 37:11.220] distributed state [37:11.220 --> 37:11.960] machine. [37:12.420 --> 37:13.400] And then on [37:13.400 --> 37:13.880] top of all [37:13.880 --> 37:14.840] of that, [37:14.840 --> 37:16.120] we built a [37:16.120 --> 37:16.700] custom [37:16.700 --> 37:17.900] multicast-based [37:17.900 --> 37:19.200] protocol to [37:19.200 --> 37:20.040] enable cluster [37:20.040 --> 37:21.000] nodes to [37:21.000 --> 37:21.240] automatically [37:21.240 --> 37:21.540] discover each [37:21.540 --> 37:21.940] other on [37:21.940 --> 37:22.700] the network [37:22.700 --> 37:23.360] and then [37:23.360 --> 37:23.480] automatically [37:23.480 --> 37:24.220] join the [37:25.020 --> 37:25.560] cluster. [37:25.560 --> 37:25.980] Now, we [37:25.980 --> 37:26.820] made this opt-in [37:26.820 --> 37:27.080] because obviously [37:27.080 --> 37:27.620] that's kind of [37:36.020 --> 37:37.020] a security [37:27.620 --> 37:28.120] risk if [37:28.120 --> 37:28.800] anybody can [37:28.800 --> 37:29.420] just join the [37:29.420 --> 37:30.640] cluster at [37:30.640 --> 37:31.400] will. [37:33.880 --> 37:34.920] But for [37:34.920 --> 37:35.780] some scenarios [37:35.780 --> 37:36.320] like where [37:36.320 --> 37:36.820] all of your [37:36.820 --> 37:37.300] nodes are on [37:37.400 --> 37:37.800] a dedicated [37:38.660 --> 37:39.380] subnet that's [37:39.380 --> 37:40.140] properly firewalled [37:40.140 --> 37:40.600] off and you [37:40.600 --> 37:41.120] have proper [37:41.120 --> 37:41.780] layer 3 and [37:41.780 --> 37:42.420] layer 4 access [37:42.420 --> 37:44.440] controls, [37:45.760 --> 37:47.060] this would be a [37:47.060 --> 37:47.420] solution that would [37:47.420 --> 37:48.540] be really handy. [37:48.540 --> 37:49.720] So, again, [37:49.720 --> 37:50.700] thinking about [37:50.700 --> 37:51.440] warehouse-scale [37:51.440 --> 37:52.620] computing or [37:53.200 --> 37:54.080] data center as [37:54.080 --> 37:55.280] a computer, [37:55.280 --> 37:55.840] let's say you [37:55.840 --> 37:56.160] just have racks [37:56.160 --> 37:56.720] and racks and [37:56.720 --> 37:57.460] racks of compute [37:57.460 --> 37:58.180] nodes, you would [37:58.180 --> 38:00.120] just unbox one, [38:00.120 --> 38:00.740] literally throw it [38:00.740 --> 38:01.180] in the rack and [38:01.180 --> 38:02.560] power it on, and [38:02.560 --> 38:03.220] it would discover [38:03.220 --> 38:04.740] the cluster as [38:04.740 --> 38:05.040] soon as it's [38:05.040 --> 38:05.680] powered on and [38:05.680 --> 38:06.220] join the cluster [38:06.220 --> 38:06.840] and start doing [38:06.840 --> 38:07.500] work. [38:07.660 --> 38:08.380] That's kind of [38:08.380 --> 38:08.860] the vision that we [38:08.860 --> 38:09.840] have for this. [38:11.420 --> 38:12.060] And, of course, [38:12.060 --> 38:12.420] since it's all [38:12.420 --> 38:13.580] peer-to-peer, [38:13.580 --> 38:15.120] that enables it [38:15.120 --> 38:15.900] to actually be [38:15.900 --> 38:16.400] just literally [38:16.400 --> 38:17.280] that simple. [38:19.720 --> 38:20.580] But, of course, [38:20.580 --> 38:21.760] the default is [38:21.760 --> 38:22.260] you manually [38:22.260 --> 38:22.980] specify which [38:22.980 --> 38:23.800] peers you want [38:23.800 --> 38:24.220] to be part of [38:24.220 --> 38:25.040] the swarm. [38:29.110 --> 38:29.410] Okay. [38:29.410 --> 38:31.070] And then, in [38:31.070 --> 38:32.050] April this year, [38:32.050 --> 38:32.470] we had the [38:32.470 --> 38:33.110] unique opportunity [38:33.110 --> 38:33.410] to present [38:33.410 --> 38:34.170] itself to [38:34.170 --> 38:34.710] purchase [38:34.710 --> 38:35.950] loft crack. [38:37.210 --> 38:38.090] Dildog approached [38:38.090 --> 38:38.610] me and we [38:38.610 --> 38:39.430] entered negotiations [38:39.430 --> 38:41.610] and we [38:43.050 --> 38:43.630] purchased it [38:43.630 --> 38:43.990] earlier this [38:43.990 --> 38:44.610] year primarily [38:46.290 --> 38:47.370] for the [38:47.370 --> 38:49.310] GUI, which [38:49.310 --> 38:50.330] is written [38:50.330 --> 38:50.890] in C++ [38:50.890 --> 38:52.710] using QT [38:52.710 --> 38:53.350] as the [38:53.350 --> 38:53.690] windowing [38:53.690 --> 38:54.570] framework. [38:54.810 --> 38:55.170] And while [38:55.170 --> 38:55.510] it's not [38:55.510 --> 38:55.830] currently [38:55.830 --> 38:56.670] cross-platform, [38:56.670 --> 38:57.070] that's a [38:57.070 --> 38:57.870] really easy [38:57.870 --> 38:59.250] platform, or [38:59.370 --> 39:00.470] that's a really [39:00.470 --> 39:00.910] easy language [39:00.910 --> 39:01.310] and library [39:01.310 --> 39:01.910] to make [39:01.910 --> 39:08.130] cross-platform. [39:08.130 --> 39:09.310] So, instead [39:09.310 --> 39:09.810] of [39:09.810 --> 39:10.070] hashtag [39:10.070 --> 39:11.230] version 3, [39:11.230 --> 39:11.750] we now [39:11.750 --> 39:12.590] have loft [39:13.970 --> 39:14.410] crack version [39:14.410 --> 39:14.930] 8. [39:14.930 --> 39:16.710] And while [39:16.710 --> 39:17.110] loft crack [39:17.110 --> 39:17.250] 7 uses [39:17.250 --> 39:17.810] John the [39:17.810 --> 39:18.810] Ripper as [39:18.810 --> 39:19.430] a backend, [39:19.430 --> 39:20.470] we've ripped [39:20.470 --> 39:20.690] all that out [39:20.690 --> 39:21.630] and we are [39:21.630 --> 39:22.790] in the process [39:22.830 --> 39:23.830] of integrating [39:23.830 --> 39:24.830] what was [39:22.790 --> 39:23.390] hashtag version [39:23.390 --> 39:24.770] 3 as the [39:24.770 --> 39:25.450] new backend [39:25.450 --> 39:26.130] for loft crack [39:26.130 --> 39:26.710] 8. [39:28.670 --> 39:29.710] And the [39:29.710 --> 39:31.010] final thing [39:31.010 --> 39:31.330] we're doing [39:31.330 --> 39:32.170] with this is [39:32.170 --> 39:32.870] we are dropping [39:32.870 --> 39:33.590] the [39:33.590 --> 39:34.070] TeraHash [39:34.070 --> 39:34.330] hardware [39:34.330 --> 39:35.310] requirement. [39:35.490 --> 39:36.710] Traditionally, [39:36.710 --> 39:37.550] you know, we [39:37.550 --> 39:39.650] do not let [39:39.650 --> 39:40.810] people purchase [39:40.810 --> 39:42.270] hashtag as a [39:42.270 --> 39:42.950] standalone product. [39:42.950 --> 39:43.550] We say, like, [39:43.550 --> 39:44.070] you know, the only [39:44.070 --> 39:44.750] way to get [39:44.750 --> 39:45.650] hashtag is to [39:46.010 --> 39:46.410] buy a [39:46.410 --> 39:48.670] TeraHash [39:48.670 --> 39:49.310] appliance and [39:49.790 --> 39:50.790] it comes [39:50.790 --> 39:51.790] preinstalled [39:49.310 --> 39:49.690] and that's the [39:49.690 --> 39:50.050] only way to [39:50.050 --> 39:50.870] get it. [39:51.430 --> 39:52.230] But loft crack [39:52.230 --> 39:52.810] has always been [39:52.810 --> 39:53.890] sold standalone. [39:54.350 --> 39:56.030] So we are changing [39:56.030 --> 39:56.730] the model now [39:57.310 --> 39:58.250] to where, [39:59.150 --> 39:59.730] you know, [39:59.730 --> 40:00.170] keeping with [40:00.170 --> 40:00.570] loft crack [40:00.570 --> 40:01.030] being sold [40:01.030 --> 40:01.490] standalone, [40:01.490 --> 40:01.710] we're going to [40:01.710 --> 40:02.290] be selling [40:03.710 --> 40:04.330] loft crack [40:04.330 --> 40:05.070] 8 as a [40:05.070 --> 40:05.970] standalone product [40:05.970 --> 40:06.590] without the [40:06.590 --> 40:10.470] TeraHash [40:10.470 --> 40:10.530] hardware [40:10.530 --> 40:10.770] requirement. [40:10.770 --> 40:12.010] So, yeah, [40:12.010 --> 40:13.150] that's where [40:13.630 --> 40:14.470] we are now [40:14.470 --> 40:15.150] with development. [40:15.150 --> 40:15.290] That's how the [40:17.310 --> 40:18.310] product has [40:15.290 --> 40:15.750] evolved over [40:15.750 --> 40:16.270] time. [40:16.270 --> 40:16.870] And that's some [40:16.870 --> 40:17.410] of the [40:17.410 --> 40:18.750] challenges that [40:18.750 --> 40:19.490] we have faced [40:19.490 --> 40:21.670] trying to [40:22.830 --> 40:23.690] obtain our [40:23.690 --> 40:24.390] goal of [40:24.390 --> 40:24.890] true [40:24.890 --> 40:25.750] warehouse scale [40:25.750 --> 40:26.730] computing. [40:26.830 --> 40:27.730] Thank you and [40:27.730 --> 40:28.270] hope you've enjoyed [40:28.270 --> 40:29.010] this.