WEBVTT 00:00.000 --> 00:01.540 This is Jeremy Gosney. 00:01.540 --> 00:06.080 I'm coming to you live from Password Village HQ2 in sunny Austin, Texas. 00:06.080 --> 00:08.400 Today's high is 104 degrees. 00:08.880 --> 00:15.200 So today I'm going to be talking to you about what it's like cracking at Terrahash's scale. 00:16.720 --> 00:18.920 I'm not going to make this a vendor pitch in any way. 00:18.920 --> 00:28.920 I'm going to focus on the technical challenges that we encounter as we support clients who have hundreds of GPUs in their clusters. 00:30.620 --> 00:35.080 When I first started on this, I thought 100 GPUs was a really big deal. 00:35.640 --> 00:39.660 Now we tend to view 100 GPUs as like, meh. 00:40.660 --> 00:43.300 It's good, but it's kind of middle of the road. 00:47.260 --> 00:51.880 There's a lot of distributed solutions for HashCat out there. 00:52.360 --> 00:58.440 But as far as I know, Terrahash is the only one that aims to operate at warehouse scale. 00:58.440 --> 01:10.020 What I mean by that is, more so with the most recent version, but with every version that passes, we aim to embrace the concepts of data center as a computer. 01:10.560 --> 01:19.300 While we currently don't have any clients who have an entire warehouse full of password cracking hardware, that's the point where we're trying to go. 01:20.880 --> 01:27.720 We try to enable the software to be at that level, even if our clients are not quite at that level. 01:30.560 --> 01:38.400 I'm going to go through the history of how I got started in massive distributed cracking. 01:38.520 --> 01:46.540 And I'm going to go through the technical challenges that we faced as we continued to evolve our hashtag distributed solution. 01:47.580 --> 01:49.600 I'm going to start the slide deck. 01:57.330 --> 01:58.810 All right. 02:01.480 --> 02:07.300 So, in the beginning, I kind of got into this because I was messing around with distributed computing. 02:07.300 --> 02:16.720 I was doing stuff with MPI, some other MOSIX software, rocks clusters, things along those natures. 02:17.820 --> 02:19.840 But all that was all CPU-based. 02:19.840 --> 02:29.900 And then when GPU cracking came about, we started looking for solutions that would enable us to distribute GPU workloads. 02:29.940 --> 02:34.840 So, I found a piece of software at a Hebrew university called VirtualCL. 02:35.160 --> 02:44.060 And I did a presentation on a 25 GPU proof of concept cluster that me and Bitweasel put together at passwords12 in Oslo. 02:44.300 --> 02:49.740 And when I woke up the next morning after giving that presentation, we had gone viral. 02:49.840 --> 02:56.060 On the front page of Slashdot, Gizmodo, Boing Boing, NBC News, The Register, we were everywhere. 02:57.140 --> 03:01.480 And in hindsight, I really should have used a lot more than 25 GPUs. 03:01.480 --> 03:05.300 I mean, we had more GPUs, that's just what we chose to dedicate to this proof of concept. 03:05.300 --> 03:15.160 And had I known in advance in any way that the general public would react how they did to this proof of concept, I would have used like three times the number of GPUs. 03:17.060 --> 03:25.980 So just, you know, some of the headlines that came out the next morning that I woke up to, which was pretty surreal because I didn't really see this as a big deal. 03:27.360 --> 03:42.820 So, VirtualCL was cool at the time because it was the first solution that I knew of that enabled us to sort of transparently distribute OpenCL jobs across the arbitrary cluster. 03:42.820 --> 03:47.080 So what VirtualCL does is it provides a virtual OpenCL platform. 03:48.620 --> 03:54.080 And the target software, in theory, doesn't need to be aware it's communicating with a virtual platform. 03:54.080 --> 04:03.160 It just sees any, you know, it sees VirtualCL as any other, you know, OpenCL platform, just like AMD, NVIDIA, Intel, what have you. 04:04.560 --> 04:17.400 So the target software just simply links against libOpenCL, which is provided by VCL, which then communicates to a broker daemon, and then that broker daemon distributes to the agent daemons, and the agent daemons, which are installed on the compute nodes, 04:18.100 --> 04:24.440 actually communicate with the real OpenCL library, be it AMD, NVIDIA, whatever. 04:26.040 --> 04:29.680 So in theory, all this is transparent to the end application. 04:29.680 --> 04:32.420 But in practice, it didn't quite work that way. 04:33.380 --> 04:42.160 We had to have a special fork of OCLHashCat called VCLHashCat, which had some workarounds for VCL quirks and such. 04:43.280 --> 04:47.760 It also required InfiniBand, because it was really latency sensitive. 04:47.980 --> 04:53.140 It ended up being a very chatty protocol, and it required essentially real-time communication. 04:53.400 --> 04:55.000 Bandwidth wasn't so much of an issue. 04:55.000 --> 05:03.600 It pretty much consistently pulled, you know, less than one gig of bandwidth, but it was really sensitive to latency. 05:03.960 --> 05:13.620 Even just a couple milliseconds was acceptable, but as you start getting up into double digits of latency, you had a noticeable impact on hash rate. 05:14.480 --> 05:16.920 And of course, InfiniBand hardware is expensive. 05:16.920 --> 05:19.300 It's way more expensive than Ethernet. 05:20.240 --> 05:23.520 VCL was also closed source, and it required frequent updates. 05:23.520 --> 05:33.300 So this is back in the time, this is 2012 through 2013, where every GPU driver that was released broke some shit. 05:33.300 --> 05:40.540 And then we had to go back in and implement workarounds and fixes, whatever the fuck AMD or NVIDIA broke in their driver that day. 05:40.540 --> 05:42.660 And VCL was no exception. 05:42.660 --> 05:58.440 So with VCLHashCat, we had to have our own workarounds for VirtualCL, and then the VCL team, or VirtualCL team, had to turn around and implement any workarounds for any new driver releases every time they were released. 05:58.440 --> 06:02.280 Which is a pretty substantial job. 06:02.560 --> 06:05.180 The problem is, VirtualCL was created by grad school students. 06:05.180 --> 06:08.850 This was a grad school project, and they created this their senior year. 06:09.360 --> 06:13.680 And when they graduated in spring 2013, that was it. 06:13.680 --> 06:15.300 The project died. 06:15.860 --> 06:30.280 So when the next version of FGLRX was released, I think that was version 13.9 in September of 2013, we go to install a VirtualCL cluster for our client. 06:30.280 --> 06:32.800 This is actually a 64 GPU cluster. 06:33.060 --> 06:35.740 We were just starting out, so this was actually a pretty big deal for us. 06:35.740 --> 06:38.480 64 GPUs was pretty fucking cool at the time. 06:39.780 --> 06:51.440 But we get the OS installed, we get the hardware built, all that shit, and then we go to install VirtualCL and FGLRX, and VirtualCL just shits the bed. 06:51.440 --> 07:02.340 So I panic and email Hebrew University, and I'm like, you guys really need to update VirtualCL for Catalyst and FGLRX 13.9. 07:02.780 --> 07:06.680 And they're like, well, we can't. 07:06.680 --> 07:09.200 Everyone who was working on this project has gone. 07:10.660 --> 07:18.520 And the intellectual property remains with Hebrew University and MOSIX, so they can just have those people work on it outside of that. 07:18.940 --> 07:26.420 I offered to take over the project for them, because it was kind of essential to what we were doing, and I didn't want to see the project die. 07:26.420 --> 07:27.680 I thought it was really neat. 07:28.940 --> 07:35.180 But the professor over there who was in charge of the project was not amenable to that idea. 07:35.180 --> 07:43.140 He actually had pretty grave concerns when we first started using VirtualCL for password cracking, because he envisioned us turning the world into a giant botnet. 07:43.540 --> 07:47.500 And of course, that wasn't really on our agenda, at least not at that time. 07:47.500 --> 07:57.660 But yeah, he really didn't like the idea of his VirtualCL shit ending up in our hands. 07:58.280 --> 08:04.640 So we were left high and dry without a software solution, and so we desperately needed something. 08:04.700 --> 08:07.500 So we conceived the idea of Hashtag. 08:09.700 --> 08:16.680 Initially, we thought about making our own VirtualCL clone, since Hebrew University was not going to give us the VirtualCL source code. 08:16.680 --> 08:18.180 And we're like, well, fuck it. 08:18.180 --> 08:21.600 We'll start our own VirtualCL with hookers and blackjack. 08:22.320 --> 08:30.700 But the more we actually sat down and thought about what that entailed, we determined the level of effort for that was just way too high. 08:30.760 --> 08:34.100 And the timeframe in which we needed it was way too short. 08:34.500 --> 08:41.680 We literally had hardware in-house that we're building for clients who now have no software solution because of VirtualCL. 08:45.970 --> 08:47.950 So we decided to make a distributed wrapper. 08:47.950 --> 08:50.470 And that's when we came up with the idea of Hashtag. 08:51.190 --> 08:53.570 Like I said, we had to iterate really quickly on this. 08:53.570 --> 08:59.190 So we went from a whiteboard session to production in less than two months. 08:59.230 --> 09:11.310 And this was basically me and Tom Steele locking ourselves in the office and ordering over $100 for the Taco Bell and just banging this out as fast and furious as we could. 09:12.630 --> 09:16.010 It had a traditional client-server-agent architecture. 09:16.650 --> 09:20.690 And it was Hashtag focused, but it actually had a generic plugin interface. 09:20.990 --> 09:25.230 We had plugins for... and this was before there was just one Hashtag. 09:25.230 --> 09:33.150 So we had Hashtag, OCL Hashtag Plus, OCL Hashtag Lite plugins. 09:33.150 --> 09:35.090 And then we had John the Ripper plugin. 09:36.110 --> 09:46.890 And then we had just a generic interface to where any cracking tool could be made to work with Hashtag as long as it adhered to a standard format that we had defined. 09:47.350 --> 09:50.310 And then it also had the ability to run arbitrary commands when idle. 09:51.010 --> 09:59.410 This was less for our clients and actually more for me because we were running Hashtag in-house as well for our cracking-as-a-service type stuff. 10:00.470 --> 10:04.290 And this is back in a time when GPU mining was still profitable. 10:04.370 --> 10:09.750 So when we weren't working on a password cracking job, I wanted this thing to be generating coins in the background. 10:09.750 --> 10:22.590 So the arbitrary idle exec command was literally just put in place so that people could mine Litecoin or Namecoin or whatever when their cluster was idle. 10:24.090 --> 10:26.270 So, you know, totally different time. 10:26.670 --> 10:28.170 But, I mean, it actually worked. 10:28.170 --> 10:32.530 It did exactly what it was supposed to do and it was good. 10:33.530 --> 10:38.570 And the whole goddamn thing was implemented in this fancy new language called Node.js. 10:38.570 --> 10:48.610 Again, this is early 2013, so Node.js was the new hotness and everyone was starting to move that direction with microservices and everything. 10:48.850 --> 10:54.270 It's not on this slide, but we implemented Hashtag version 1 entirely as microservices. 10:55.130 --> 11:03.210 And I shit you not, there was like 23 individual packages that comprised Hashtag and almost all of those were microservices. 11:05.350 --> 11:09.970 And we went that direction because that's kind of like programming zeitgeist at the time, right? 11:09.970 --> 11:17.170 Like everyone was pushing microservices and pushing asynchronous and, you know, NoSQL and all that other horseshit, right? 11:17.170 --> 11:22.810 And in the end, it created more headaches than not if we just had a monolithic fucking program. 11:23.490 --> 11:38.950 You know, maintaining like one server binary and one agent binary is way simpler than like, you know, maintaining 23 individual microservices and making sure that they're all synced and communicating properly and, you know, that the versions are correct, 11:38.950 --> 11:40.810 you know, across all 23 packages. 11:40.950 --> 11:48.510 And there's just a lot of unnecessary headache that was introduced with that, that, you know, we attempted to manage. 11:48.870 --> 11:51.430 But even then, like our clients would still kind of mess it up. 11:51.430 --> 12:03.470 You know, like they would upgrade like 10 of 20 packages and then like the other ones would just be stuck at old versions or, you know, one demon would die and another demon would know that it's dead, so then it would shit the bed. 12:03.470 --> 12:11.270 And anyway, so yeah, no JS, microservices, no SQL, MongoDB, all that shit. 12:11.370 --> 12:15.950 I tried to use all like the latest and greatest everything because at the time, that seemed like the best idea. 12:17.010 --> 12:19.710 And also, Hashtag was more than just workload distribution. 12:19.710 --> 12:25.450 So a lot of our clients at the time thought that Hashtag was just a GUI front-end for Hashcat. 12:25.850 --> 12:26.990 And we're like, no, it's not. 12:26.990 --> 12:29.530 It's not just a web UI for Hashcat. 12:29.530 --> 12:37.630 It actually does like, you know, multi-user and, you know, workload distribution with like a really complex four-dimensional queuing mechanism. 12:37.630 --> 12:42.310 Like you're trying to explain all this to the client and they're just like, you know, eyes glazed over and like, so it's a web UI for Hashcat. 12:42.470 --> 12:45.110 Like fucking yes, okay, it's a web UI for Hashcat. 12:46.090 --> 12:47.590 But it actually did a hell of a lot more than that. 12:47.590 --> 12:54.990 In fact, when you installed Hashtag on your box, it basically hijacked it and configured it exactly the way that we needed it to be configured. 12:54.990 --> 13:00.570 It would actually provision the entire server from just installing like the initial meta package. 13:01.410 --> 13:03.350 So it's a complete stack of packages. 13:03.350 --> 13:05.430 They get stack, Hashtag, that's where it comes from. 13:05.910 --> 13:10.170 So it handled driver installation, driver configuration, driver updates. 13:10.170 --> 13:20.130 And again, this is back at a time, and even nowadays still kind of a challenge, but this is back at a time when GPU drivers were a pain in the ass to install. 13:20.130 --> 13:22.570 And then once you got them installed, it was a pain in the ass to update. 13:23.150 --> 13:27.230 So the fact that we kind of handled this automagically was a pretty big value add. 13:27.970 --> 13:32.070 Also handled XOR configuration, headless servers. 13:32.350 --> 13:38.530 This is actually during a time where when you had a headless server, you had to have fucking dummy plugs connected to each of the GPUs. 13:38.530 --> 13:41.470 There's probably very few people watching this who actually remember that. 13:42.330 --> 13:51.850 But yeah, if you had a completely headless server, each one of those GPUs had to have a special dummy plug attached to it that would connect the necessary pins to trick it to think that there's actually a monitor attached. 13:51.990 --> 13:55.710 And we actually had to run XOR because even though these are headless, 14:00.570 --> 14:04.610 they don't have GPU clock rates or set like power tune profiles or any of that. 14:04.610 --> 14:19.630 So even though this is a headless server, we still had to have dummy plugs to think there's a monitor attached, still had to run XOR, and still had to configure XOR properly to enable us to control the fans and control the clocks and all the other shit that we needed to do. 14:20.030 --> 14:26.090 And then along those lines, Hashtag also did GPU tuning for you and also automatic overclocking. 14:26.090 --> 14:33.470 And this was kind of a double-edged sword. 14:33.510 --> 14:48.570 So when the R9 290X came out, at first it was like entirely unusable because of the way that AMD aggressively implemented power tune firmware, I think it was 2.0 at the time. 14:50.470 --> 14:56.070 But the GPUs would throttle so aggressively that it was like way slower than the 7970. 14:57.130 --> 14:59.050 And it was just essentially unusable. 14:59.050 --> 15:02.310 So then I wrote od6config. 15:05.310 --> 15:06.470 Sorry, did I say PowerTune? 15:06.470 --> 15:07.290 PowerTune is fucking NVIDIA. 15:07.390 --> 15:08.010 Overdrive. 15:08.010 --> 15:08.670 I'm sorry. 15:09.450 --> 15:09.830 Oh, no, no. 15:09.830 --> 15:10.770 PowerTune is AMD. 15:10.810 --> 15:11.790 PowerMizer is NVIDIA. 15:11.790 --> 15:12.570 All right, whatever. 15:13.870 --> 15:19.970 So od6config enabled us to properly configure the R9 290X. 15:20.930 --> 15:23.590 But it was good, but it was still kind of mediocre. 15:24.190 --> 15:35.130 But at that time, when I measured it, it was only drawing 300 watts, which is important because our server platforms like the Brutalis can only handle GPUs that drop to 300 watts. 15:35.130 --> 15:38.730 You draw more than that, you're going to blow a fuse on the PCI slots on the motherboard. 15:40.550 --> 15:52.410 So we had a automatic overclock profile for the R9 290X that we pushed out to our clients that made the GPU draw just about 300 watts. 15:52.410 --> 16:06.730 And then a little bit later down the road, about a year, year and a half later, AMD pushed out a new driver update that actually caused the power consumption to go up by a good 50 to 75 watts. 16:06.730 --> 16:09.270 And we did not know this. 16:09.270 --> 16:21.930 This kind of went undetected by us until we started getting a rash of clients who were seeing dead PCIe slots on their motherboards, and all of them had R9 290X-based solutions. 16:22.910 --> 16:27.190 And I finally busted out a kilowatt and figured out what the fuck was going on. 16:28.290 --> 16:50.170 But we actually had the ability, through all these mechanisms that we had in Hashtag at the time to enable the automatic overclocking and configuration and all that, to push out a new underclock profile to all of our clients to try to drop the power consumption down to keep it from blowing PCIe slots on the motherboards. 16:51.770 --> 16:52.970 So it's both good and bad. 16:52.970 --> 16:59.530 The fact that we're able to automatically overclock GPUs to help get the most speed out of the GPUs is awesome. 17:00.530 --> 17:05.310 But then in that one specific scenario, it was bad because we started blowing PCIe slot fuses. 17:05.450 --> 17:14.010 But then it was good that we had this functionality because then we could correct it through software, over the air, and automatically correct it on all of our client systems all at the same time. 17:15.590 --> 17:20.390 Which saved us a lot of money because we were spending hundreds of thousands of dollars on new motherboards. 17:21.790 --> 17:22.430 So yeah. 17:22.430 --> 17:25.830 And then, of course, Hashtag also had cluster resource monitoring. 17:28.330 --> 17:30.170 So there were some drawbacks. 17:30.170 --> 17:38.430 Like I said, the entire Godend thing was written in Node.js, and while Node.js is asynchronous, it also runs on a single CPU thread. 17:38.670 --> 17:45.570 That may have changed at some point, because we're talking like 2013, that's seven years ago, and I haven't even touched Node.js since. 17:45.730 --> 18:00.790 But at the time, everything ran on a single CPU thread, which was stupid, because our cluster controllers had a minimum of 12 CPU cores, 24 threads, and we're sitting here stuck in one thread on the server daemon. 18:01.610 --> 18:09.430 So yeah, any long-running methods would just totally block the entire application, and we had massive scalability issues. 18:10.210 --> 18:17.510 I think the very first alpha that we released to clients could only support four nodes. 18:17.510 --> 18:21.830 The second one, we were able to get up to nine or ten nodes. 18:22.290 --> 18:31.870 Basically, any time a client ordered a cluster larger than the previous largest cluster we had ever built, we ran into some kind of scalability issue, identified some bottlenecks, and had to work to remove those bottlenecks. 18:32.870 --> 18:40.110 Unfortunately, 90% of the time, the bottleneck was the fact that JavaScript was too slow for the task that we were trying to do. 18:40.290 --> 18:47.570 So we had to reimplement those bottleneck methods in a faster language, and then just shell out to them. 18:48.350 --> 18:51.070 So keyspace calculation was reimplemented in C. 18:51.070 --> 19:04.510 Now, calculating keyspace for a mask attack is really lightweight, but when you talk about massive dictionaries or rule files, trying to count the number of lines in a file with JavaScript is painfully slow. 19:05.350 --> 19:13.170 So that's largely why that entire portion was reimplemented in C, and we just shelled out to that binary to eliminate that bottleneck. 19:14.310 --> 19:16.430 We didn't dramatically reduce it. 19:17.850 --> 19:18.590 Regex. 19:18.590 --> 19:25.250 Anything Regex, we actually implemented, we had reimplemented it in Perl, because Perl is amazing at Regex. 19:25.250 --> 19:28.570 JavaScript is not very good at Regex. 19:28.950 --> 19:34.990 The primary thing that we were doing with Regex here was validating the hash lists. 19:34.990 --> 19:43.570 So a client uploads a hash list, and some of our clients wouldn't even bother throwing 200 million hashes at hashtag in one job. 19:43.970 --> 19:55.710 But then hashtag has to sit there and receive the massively large hash file, and then validate each individual hash, and then stuff them in a database. 19:56.130 --> 19:58.270 That was a massive bottleneck. 19:58.270 --> 20:02.830 So we ended up reimplementing all that in Perl to try to get that down. 20:02.830 --> 20:08.730 And then the agent, the entire fucking agent, we ended up just implementing the whole thing in Perl. 20:12.710 --> 20:19.090 And the only reason we picked Perl at the time is because that's the only language we could iterate fast enough in to rewrite the entire damn thing. 20:21.770 --> 20:23.090 But yeah. 20:23.430 --> 20:29.750 Like I said, we also used MongoDB because NoSQL was the rage at the time. 20:30.070 --> 20:34.450 And out of all the NoSQL databases we tested, Mongo was the most performant. 20:35.010 --> 20:39.190 But then we had clients do things like throw 200 million hashes at hashtag. 20:39.950 --> 20:51.390 And that would cause some not only bottlenecks and blocking processes, but some really UI weirdness as well. 20:51.530 --> 21:01.350 They would go through the process of creating a job, uploading the hash list, and they would just sit there with the little spinner for a while until it timed out. 21:01.830 --> 21:04.650 So they'd go to create the job again because it timed out. 21:04.650 --> 21:10.590 And then they'd go back 10 minutes later and it was actually active and running. 21:10.590 --> 21:16.090 But because they went through the process four times trying to get it to start, now they have like four of the same process running on the cluster. 21:16.470 --> 21:21.930 So we threw Redis in front of Mongo as an in-memory caching layer. 21:22.290 --> 21:25.670 And that actually had a significant impact, like positive impact. 21:25.850 --> 21:29.230 And it enabled us to ingest hashes a lot faster. 21:30.930 --> 21:34.570 But still not a very great solution overall. 21:36.190 --> 21:41.310 The Web UI client was implemented as a single-page application in Angular. 21:41.750 --> 21:43.470 It was really clunky. 21:43.470 --> 21:44.670 It was really heavy. 21:44.670 --> 21:46.090 It was really slow. 21:46.110 --> 21:50.830 It constantly caused the browser to freeze if you left it up for more than 30 minutes. 21:50.930 --> 21:52.910 And it just had a ton of bugs. 21:53.390 --> 22:02.610 And I think probably the most annoying thing is if you're like a hashcat power user and you sit down and try to use the hashtag Web UI, it was just cumbersome as fuck. 22:03.010 --> 22:06.390 There's no way to move quickly in that UI. 22:06.550 --> 22:09.290 And it gets very repetitive and very tedious. 22:10.370 --> 22:14.950 So there was a lot of drawbacks to having that Angular Web UI. 22:15.250 --> 22:16.770 I absolutely hated it. 22:16.770 --> 22:26.910 And anyone who was really experienced with cracking and hashcat and stuff, they also hated it and begged us not to have that again. 22:27.570 --> 22:30.830 And we were just severely limited by what we could do in a browser. 22:31.410 --> 22:36.390 For non-hash formats at the time, we were scraping everything server-side. 22:36.390 --> 22:53.910 So you would actually upload a PDF or upload a .docx or whatever, and we would scrape the file server-side to get the necessary bits, like Office to hashcat and PDF to hashcat, shit like that. 22:54.530 --> 23:04.690 The problem then became, what do we do with things like TrueCrypt volumes or really large 7-zip volumes or RAR files? 23:04.690 --> 23:14.390 Are you really going to upload a 100-gigabyte disk image through the browser just so we can scrape off two kilobytes? 23:14.390 --> 23:17.010 No, that doesn't make any sense. 23:19.690 --> 23:21.710 So yeah, the browser was an issue. 23:21.710 --> 23:25.970 And then we implemented access controls, but there was nothing fine-grained. 23:25.970 --> 23:38.770 We assumed that because everyone operating the hashcat cluster was on the same team, that they all had equal rights and equal access to the cluster, and therefore all users are admins. 23:38.770 --> 23:42.010 We didn't have different roles or anything. 23:42.010 --> 23:48.410 It was just, if you have an account, then you obviously have a right to be there, so therefore you are an admin. 23:50.970 --> 23:59.730 And I think some of our clients didn't have a problem with that, because everyone on this team does have equal rights and access and privilege to the cluster, so this is totally fine. 23:59.730 --> 24:02.390 Other clients were like, this is absolutely unacceptable. 24:04.810 --> 24:10.970 And I eventually got to the point where the old version one code was completely unmanageable. 24:11.370 --> 24:18.210 Even just implementing what should have been a minor bug fix ended up requiring major refactoring. 24:19.890 --> 24:35.550 It just wasn't, because we threw it together in less than two months, so we didn't think of everything in advance that would come up, and try to implement new features, and try to work around some of the bugs in there. 24:35.550 --> 24:39.910 So we decided we needed an entire ground-up rewipe for Hashtag version two. 24:41.630 --> 24:49.650 And at some point, I got the idea to create a commercial fork of Hashcat with native distributed capabilities. 24:49.850 --> 24:52.690 And at the time, I was so fucking stoked on this idea. 24:52.690 --> 24:57.090 It seemed like the best idea in the world to me at the time. 24:58.050 --> 25:01.330 So we set out to do just that. 25:02.150 --> 25:13.890 Now this is right before Hashtag went open source, and then we started work on this literally the minute that Hashtag went open source. 25:14.450 --> 25:20.290 So we implemented in Golang in C with Postgres database. 25:20.510 --> 25:26.910 Now we chose Golang because of high concurrency with goroutines, right? 25:27.090 --> 25:31.150 So with Node.js, everything ran on a single CPU thread. 25:31.150 --> 25:34.490 And we wanted the opposite of that for version two, right? 25:34.490 --> 25:35.990 We're going to learn the lesson on that. 25:35.990 --> 25:44.910 So we wanted just lots and lots of goroutines to run everything so that we'd use all 24 threads plus on the box. 25:46.870 --> 25:51.130 And Golang actually also allows for pretty rapid development. 25:51.130 --> 25:53.010 So we thought it was a pretty good choice. 25:54.850 --> 25:58.690 But we still stuck with the traditional client server agent architecture. 25:58.730 --> 26:05.250 We have numerous clients and then one server and then however many agents. 26:06.350 --> 26:10.630 Meaning that that server is both a bottleneck and a single point of failure. 26:11.910 --> 26:15.430 So the first thing we had to do is we had to split Hashcat in two. 26:15.990 --> 26:27.610 We took everything that had to do with actually starting a cracking job and executing an OpenCL kernel on a GPU with the agent and nothing else. 26:27.610 --> 26:35.990 The agent was just barebones trimmed down Hashcat launch attack on GPU and that was it, nothing more. 26:36.410 --> 26:39.750 And everything else we placed into the server library. 26:40.290 --> 26:51.810 And then for any code that wasn't already in Hashcat such as the JSON API and the protobufs for agent server communication workload distribution 26:55.930 --> 26:59.390 fine -grained multi-role access control. 26:59.470 --> 27:03.650 All that was implemented in Go and then we just linked against the C libraries. 27:03.650 --> 27:12.150 And this effort actually laid the foundation for the Hashcat 4.0 refactor with libHashcat. 27:12.610 --> 27:31.390 And then because of how clunky our old web UI was we actually ditched it entirely for a cross-platform CLI which made our power users like really happy but our less experienced users were not very thrilled with that decision. 27:31.390 --> 27:36.830 They still wanted a GUI which, fair enough. 27:38.090 --> 27:44.590 So our plan initially was to maintain both Hashcat and Hashstack trees in parallel. 27:44.830 --> 27:55.510 And then if there was any features that we added to Hashstack or any new kernels that we wrote we would just backport those as we chose to Hashcat. 27:58.550 --> 28:00.790 This didn't really work out very well in practice though. 28:00.790 --> 28:07.370 So when we released version 2 it was in sync with Hashcat 3.5 but it didn't stay that way for very long. 28:08.410 --> 28:18.370 And of course we had to add a cross-platform GUI to the roadmap as well because we had significant feedback from our clients that they really wanted the goddamn GUI and they were not happy that we took away the web UI. 28:19.890 --> 28:24.450 It turns out this was the single dumbest motherfucking thing I've ever done in my life. 28:25.590 --> 28:31.370 I completely underestimated how much work was involved in creating a commercial fork of Hashcat. 28:32.010 --> 28:33.230 Like, genuinely 28:36.470 --> 28:36.970 I... 28:37.770 --> 28:42.310 Yeah, I really don't know how to convey how fucking stupid of an idea this was. 28:43.110 --> 28:51.650 We spent like 100% of our time just backporting things from Hashcat into Hashtag instead of actually adding new features or writing new kernels. 28:52.290 --> 28:58.130 Like, literally months of work you know, just backporting Hashcat commits. 28:58.870 --> 29:00.690 And it got really frustrating. 29:00.910 --> 29:03.030 You know, it's like, hey, are we working on these tickets? 29:03.030 --> 29:04.230 Are we working on these features? 29:04.230 --> 29:05.770 No, we're working on backports. 29:05.990 --> 29:12.230 You know, I got so sick of hearing working on backports, working on backports, like when are we ever going to be done with backports? 29:13.430 --> 29:20.990 And then when Adam refactored Hashcat for 4.0, that also meant that we had to refactor Hashtag as well. 29:21.150 --> 29:27.050 And we hadn't even finished implementing all the backports at the speed that Adam was, you know, committing things to Hashcat. 29:27.170 --> 29:32.130 So this was just like, absolutely unmaintainable, right? 29:32.130 --> 29:34.250 Like, fuck me. 29:35.550 --> 29:43.450 And then there was also this other little minor problem where we picked the wrong HTTP library for the API. 29:44.130 --> 29:49.770 And this basically limited the service capability to about 12 nodes. 29:49.790 --> 29:52.830 Now, 12 nodes might seem like a lot. 29:52.830 --> 29:54.510 That's 96 GPUs. 29:54.510 --> 29:57.670 You know, that's a pretty respectable size cluster. 29:58.890 --> 30:07.230 The problem is this particular client had ordered 320 GPUs, 40 nodes. 30:07.510 --> 30:18.070 And you can't really deliver a 40 node client to a cluster with a server that can only really handle 12 nodes. 30:22.080 --> 30:28.180 So this required not a complete rewrite, but a damn near complete rewrite time. 30:28.180 --> 30:33.140 Fuck , because we use that HTTP library extensively throughout the code . 30:33.880 --> 30:36.060 And there was no, like, drop in replacement. 30:36.060 --> 30:45.840 So basically anywhere where we were dealing with HTTP , we had to rip out the old library and then, you know, add in all the stuff for the new library. 30:45.960 --> 30:47.540 But they did resolve the issue. 30:48.320 --> 30:51.400 It was just really painful at the time to have to mess with that. 30:53.420 --> 30:59.720 But the server still continued to be a single point of failure and still proved to be a source of bottlenecks. 30:59.760 --> 31:11.800 As we, you know, tried to build larger and larger clusters , we kept running into problems with the server being a bottleneck and a single point of failure . 31:11.920 --> 31:23.540 And we also had clients coming to us asking us if they could order multiple cluster controllers and cluster them together, you know, either high availability or, like, load balance them or something. 31:24.560 --> 31:30.440 And we didn't have a solution for that, but I started thinking about a solution for that. 31:31.980 --> 31:35.920 And again, like, 40 % of our clients were just completely pissed that we had no GUI . 31:37.080 --> 31:49.160 So it got to the point where it's like, let's just rethink, like, the entire fucking thing and just, like, go as far away from these mistakes as we could possibly get. 31:49.220 --> 31:51.860 Like, we never, ever want to make any of these mistakes ever again. 31:51.860 --> 31:55.920 We've learned a lot of lessons from, you know, what we've done over the last seven years. 31:57.340 --> 32:00.020 So, but we need more than just a complete ground-up rewrite. 32:00.020 --> 32:01.540 Like , we need an entire paradigm shift. 32:01.540 --> 32:11.120 Like , we need to rethink this entire problem entirely and come up with something that's completely different from what's ever been done. 32:11.940 --> 32:15.180 So we need to eliminate the single point of failure of the server. 32:15.300 --> 32:25.180 We need to enable, like, actual infinite horizontal scaling because, again, we're, you know, we always try to strive for, like, warehouse as a computer , right? 32:25.180 --> 32:29.520 We're looking at warehouse scale computing as , like, the target that we're trying to hit. 32:30.000 --> 32:39.560 There should never be any, you know, limiting factors outside of budget for, like, how many nodes we can support in HashtagWrite . 32:40.220 --> 32:44.740 And then, of course, the other big requirement is we have to actually move at the pace of Hashtag development. 32:46.040 --> 32:56.260 We have to, you know, as Adam commits things to Hashtag, we need to have those not instantly available in Hashtag, but within a very reasonable time frame. 32:58.880 --> 33:07.460 So the very first idea that we came up with was let's not do a traditional client-server-agent model. 33:08.440 --> 33:11.400 And let's not even try to cluster the cluster controllers. 33:11.740 --> 33:19.860 Let's just eliminate the cluster controllers entirely and make everything peer-to-peer where all nodes are equal and there are no dedicated servers. 33:20.860 --> 33:25.760 And the biggest requirement to this is that work can be submitted to any node. 33:26.460 --> 33:32.000 So let's say you have, like, you know, 20 users and you have, you know, 10 boxes. 33:32.000 --> 33:34.900 Like, there shouldn't be just one node that all of them have to hit. 33:34.900 --> 33:40.600 They should be able to hit any node in the cluster and, you know, have the same view from any node. 33:41.880 --> 33:44.140 And it also needs to be more user-friendly than ever. 33:44.400 --> 33:46.340 And also needs to run on Windows. 33:48.100 --> 33:49.640 I'll tell you why in a minute. 33:52.240 --> 33:55.320 So set out to, like, figure out how we're going to implement this. 33:55.320 --> 33:57.540 Because we have the high-level requirements, right? 33:57.540 --> 33:59.380 Like, we know it needs to be peer-to-peer. 33:59.380 --> 34:01.060 We know all the nodes need to be equal. 34:01.240 --> 34:02.520 You know, all that shit. 34:03.040 --> 34:04.740 But how are we actually going to do this? 34:05.600 --> 34:09.880 So pretty obviously we're going to need some sort of distributed state machine. 34:10.040 --> 34:10.680 Right? 34:11.460 --> 34:14.520 And that's just straightforward and self-explanatory. 34:14.780 --> 34:21.500 But, you know, like, we just throw a message queue at it and some PubSub and, you know, Bob's your uncle. 34:22.160 --> 34:23.860 But what about disagreements? 34:24.060 --> 34:31.100 Or, you know, maybe even more than disagreements, what about, like, an actual, you know, rogue cluster member? 34:31.220 --> 34:42.700 Or, you know, a broken cluster node that's maybe not necessarily intentionally malicious, but it certainly, you know, has, you know, the appearance of malicious behavior. 34:42.700 --> 34:43.500 Right? 34:44.620 --> 34:48.060 So I started coming up with solutions for this. 34:48.680 --> 34:56.580 And I somehow got into my head, like, dude, what if computers could vote? 34:57.000 --> 34:57.840 Right? 34:57.840 --> 35:08.740 Like, you just make it to where, like, you know, if there's a dispute , then one of the nodes, like, requests a vote, right? 35:09.060 --> 35:11.060 And so they take a vote and majority wins. 35:11.060 --> 35:19.220 And if there's a tie, like we have an even number of nodes in the cluster, then, like, literally fucking flip a coin, you know, randomly, or play rock, paper, scissors. 35:19.360 --> 35:20.780 Like, why not? 35:22.100 --> 35:26.380 And if a peer doesn't accept the results, like, then the rest of the cluster just shuns that peer. 35:26.800 --> 35:29.380 And I started thinking about, like, is this actually, like, a thing? 35:29.380 --> 35:30.380 Like, can we actually do this? 35:30.380 --> 35:32.400 Am I, like , completely fucking stoned? 35:34.600 --> 35:38.120 It turns out that I really didn't have to put that much more thought into it because this already exists. 35:38.120 --> 35:42.800 Like, what I thought up is actually a thing called the wrapped consensus protocol . 35:43.620 --> 35:48.520 Now , it's cool that it already exists because that, you know , means we don't have to do a whole lot to implement it. 35:48.520 --> 35:54.260 But it's also cool because it kind of validated, you know, my ideas and theories on how to make this work. 35:55.080 --> 35:57.400 And told me that I wasn't just completely fucking insane. 35:58.380 --> 36:02.460 And we found several implementations of the wrapped consensus protocol. 36:03.640 --> 36:09.620 And the one that we ended up liking the most and settled on is Akka, which runs on the JVM. 36:09.620 --> 36:10.660 It's written in Scala. 36:11.140 --> 36:14.180 Now, I hate Java. 36:14.180 --> 36:18.500 I have shunned Java for damn near my entire life. 36:18.740 --> 36:23.700 You know, I completely condemn Oracle JVM. 36:26.660 --> 36:29.900 But, there is OpenJDK. 36:29.900 --> 36:31.360 We don't have to use Oracle. 36:31.580 --> 36:34.160 And we don't have to write in Java. 36:34.160 --> 36:35.860 We can use Scala and Kotlin. 36:36.780 --> 36:40.920 And since I've started to work on this, I actually really like Kotlin. 36:41.780 --> 36:44.860 I don't really have too many negative things to say about it. 36:44.860 --> 36:47.180 It's actually kind of fun to work with. 36:48.420 --> 36:57.160 And then Akka also has some plugins or modules that, you know, help us even further. 36:57.160 --> 36:58.880 Such as Akka persistence. 36:58.880 --> 37:06.120 What Akka persistence does, it takes our distributed state machine, which is normally resident in memory, and persists it to disk. 37:06.600 --> 37:11.960 And then Akka distributed data, of course, would be like what implements the distributed state machine. 37:12.420 --> 37:24.220 And then on top of all of that, we built a custom multicast -based protocol to enable cluster nodes to automatically discover each other on the network and then automatically join the cluster. 37:25.020 --> 37:31.400 Now, we made this opt-in because obviously that's kind of a security risk if anybody can just join the cluster at will. 37:33.880 --> 37:48.420 But for some scenarios like where all of your nodes are on a dedicated subnet that's properly firewalled off and you have proper layer 3 and layer 4 access controls, this would be a solution that would be really handy. 37:48.540 --> 38:07.500 So , again, thinking about warehouse -scale computing or data center as a computer , let's say you just have racks and racks and racks of compute nodes, you would just unbox one, literally throw it in the rack and power it on, and it would discover the cluster as soon as it's powered on and join the cluster and start doing work. 38:07.660 --> 38:09.840 That's kind of the vision that we have for this. 38:11.420 --> 38:17.280 And, of course, since it's all peer-to-peer, that enables it to actually be just literally that simple. 38:19.720 --> 38:25.040 But, of course, the default is you manually specify which peers you want to be part of the swarm. 38:29.110 --> 38:29.410 Okay. 38:29.410 --> 38:35.950 And then, in April this year, we had the unique opportunity to present itself to purchase loft crack. 38:37.210 --> 38:54.570 Dildog approached me and we entered negotiations and we purchased it earlier this year primarily for the GUI, which is written in C++ using QT as the windowing framework. 38:54.810 --> 39:02.470 And while it's not currently cross-platform, that's a really easy platform , or that's a really easy language and library to make cross-platform . 39:04.270 --> 39:12.590 So, instead of hashtag version 3 , we now have loft crack version 8. 39:13.970 --> 39:26.710 And while loft crack 7 uses John the Ripper as a backend , we've ripped all that out and we are in the process of integrating what was hashtag version 3 as the new backend for loft crack 8. 39:28.670 --> 39:35.310 And the final thing we're doing with this is we are dropping the TeraHash hardware requirement. 39:35.490 --> 39:42.830 Traditionally, you know, we do not let people purchase hashtag as a standalone product. 39:42.830 --> 39:50.870 We say, like, you know, the only way to get hashtag is to buy a TeraHash appliance and it comes preinstalled and that's the only way to get it. 39:51.430 --> 39:53.890 But loft crack has always been sold standalone. 39:54.350 --> 40:07.730 So we are changing the model now to where, you know, keeping with loft crack being sold standalone, we're going to be selling loft crack 8 as a standalone product without the TeraHash hardware requirement. 40:10.030 --> 40:14.270 So , yeah, that's where we are now with development. 40:14.270 --> 40:16.270 That's how the product has evolved over time. 40:16.270 --> 40:26.730 And that's some of the challenges that we have faced trying to obtain our goal of true warehouse scale computing. 40:26.830 --> 40:29.010 Thank you and hope you've enjoyed this.