1
00:00:00,000 --> 00:00:01,540
This is Jeremy Gosney.

2
00:00:01,540 --> 00:00:06,080
I'm coming to you live from Password Village HQ2 in sunny Austin, Texas.

3
00:00:06,080 --> 00:00:08,400
Today's high is 104 degrees.

4
00:00:08,880 --> 00:00:15,200
So today I'm going to be talking to you about what it's like cracking at Terrahash's scale.

5
00:00:16,720 --> 00:00:18,920
I'm not going to make this a vendor pitch in any way.

6
00:00:18,920 --> 00:00:28,920
I'm going to focus on the technical challenges that we encounter as we support clients who have hundreds of GPUs in their clusters.

7
00:00:30,620 --> 00:00:35,080
When I first started on this, I thought 100 GPUs was a really big deal.

8
00:00:35,640 --> 00:00:39,660
Now we tend to view 100 GPUs as like, meh.

9
00:00:40,660 --> 00:00:43,300
It's good, but it's kind of middle of the road.

10
00:00:47,260 --> 00:00:51,880
There's a lot of distributed solutions for HashCat out there.

11
00:00:52,360 --> 00:00:58,440
But as far as I know, Terrahash is the only one that aims to operate at warehouse scale.

12
00:00:58,440 --> 00:01:10,020
What I mean by that is, more so with the most recent version, but with every version that passes, we aim to embrace the concepts of data center as a computer.

13
00:01:10,560 --> 00:01:19,300
While we currently don't have any clients who have an entire warehouse full of password cracking hardware, that's the point where we're trying to go.

14
00:01:20,880 --> 00:01:27,720
We try to enable the software to be at that level, even if our clients are not quite at that level.

15
00:01:30,560 --> 00:01:38,400
I'm going to go through the history of how I got started in massive distributed cracking.

16
00:01:38,520 --> 00:01:46,540
And I'm going to go through the technical challenges that we faced as we continued to evolve our hashtag distributed solution.

17
00:01:47,580 --> 00:01:49,600
I'm going to start the slide deck.

18
00:01:57,330 --> 00:01:58,810
All right.

19
00:02:01,480 --> 00:02:07,300
So, in the beginning, I kind of got into this because I was messing around with distributed computing.

20
00:02:07,300 --> 00:02:16,720
I was doing stuff with MPI, some other MOSIX software, rocks clusters, things along those natures.

21
00:02:17,820 --> 00:02:19,840
But all that was all CPU-based.

22
00:02:19,840 --> 00:02:29,900
And then when GPU cracking came about, we started looking for solutions that would enable us to distribute GPU workloads.

23
00:02:29,940 --> 00:02:34,840
So, I found a piece of software at a Hebrew university called VirtualCL.

24
00:02:35,160 --> 00:02:44,060
And I did a presentation on a 25 GPU proof of concept cluster that me and Bitweasel put together at passwords12 in Oslo.

25
00:02:44,300 --> 00:02:49,740
And when I woke up the next morning after giving that presentation, we had gone viral.

26
00:02:49,840 --> 00:02:56,060
On the front page of Slashdot, Gizmodo, Boing Boing, NBC News, The Register, we were everywhere.

27
00:02:57,140 --> 00:03:01,480
And in hindsight, I really should have used a lot more than 25 GPUs.

28
00:03:01,480 --> 00:03:05,300
I mean, we had more GPUs, that's just what we chose to dedicate to this proof of concept.

29
00:03:05,300 --> 00:03:15,160
And had I known in advance in any way that the general public would react how they did to this proof of concept, I would have used like three times the number of GPUs.

30
00:03:17,060 --> 00:03:25,980
So just, you know, some of the headlines that came out the next morning that I woke up to, which was pretty surreal because I didn't really see this as a big deal.

31
00:03:27,360 --> 00:03:42,820
So, VirtualCL was cool at the time because it was the first solution that I knew of that enabled us to sort of transparently distribute OpenCL jobs across the arbitrary cluster.

32
00:03:42,820 --> 00:03:47,080
So what VirtualCL does is it provides a virtual OpenCL platform.

33
00:03:48,620 --> 00:03:54,080
And the target software, in theory, doesn't need to be aware it's communicating with a virtual platform.

34
00:03:54,080 --> 00:04:03,160
It just sees any, you know, it sees VirtualCL as any other, you know, OpenCL platform, just like AMD, NVIDIA, Intel, what have you.

35
00:04:04,560 --> 00:04:17,400
So the target software just simply links against libOpenCL, which is provided by VCL, which then communicates to a broker daemon, and then that broker daemon distributes to the agent daemons, and the agent daemons, which are installed on the compute nodes,

36
00:04:18,100 --> 00:04:24,440
actually communicate with the real OpenCL library, be it AMD, NVIDIA, whatever.

37
00:04:26,040 --> 00:04:29,680
So in theory, all this is transparent to the end application.

38
00:04:29,680 --> 00:04:32,420
But in practice, it didn't quite work that way.

39
00:04:33,380 --> 00:04:42,160
We had to have a special fork of OCLHashCat called VCLHashCat, which had some workarounds for VCL quirks and such.

40
00:04:43,280 --> 00:04:47,760
It also required InfiniBand, because it was really latency sensitive.

41
00:04:47,980 --> 00:04:53,140
It ended up being a very chatty protocol, and it required essentially real-time communication.

42
00:04:53,400 --> 00:04:55,000
Bandwidth wasn't so much of an issue.

43
00:04:55,000 --> 00:05:03,600
It pretty much consistently pulled, you know, less than one gig of bandwidth, but it was really sensitive to latency.

44
00:05:03,960 --> 00:05:13,620
Even just a couple milliseconds was acceptable, but as you start getting up into double digits of latency, you had a noticeable impact on hash rate.

45
00:05:14,480 --> 00:05:16,920
And of course, InfiniBand hardware is expensive.

46
00:05:16,920 --> 00:05:19,300
It's way more expensive than Ethernet.

47
00:05:20,240 --> 00:05:23,520
VCL was also closed source, and it required frequent updates.

48
00:05:23,520 --> 00:05:33,300
So this is back in the time, this is 2012 through 2013, where every GPU driver that was released broke some shit.

49
00:05:33,300 --> 00:05:40,540
And then we had to go back in and implement workarounds and fixes, whatever the fuck AMD or NVIDIA broke in their driver that day.

50
00:05:40,540 --> 00:05:42,660
And VCL was no exception.

51
00:05:42,660 --> 00:05:58,440
So with VCLHashCat, we had to have our own workarounds for VirtualCL, and then the VCL team, or VirtualCL team, had to turn around and implement any workarounds for any new driver releases every time they were released.

52
00:05:58,440 --> 00:06:02,280
Which is a pretty substantial job.

53
00:06:02,560 --> 00:06:05,180
The problem is, VirtualCL was created by grad school students.

54
00:06:05,180 --> 00:06:08,850
This was a grad school project, and they created this their senior year.

55
00:06:09,360 --> 00:06:13,680
And when they graduated in spring 2013, that was it.

56
00:06:13,680 --> 00:06:15,300
The project died.

57
00:06:15,860 --> 00:06:30,280
So when the next version of FGLRX was released, I think that was version 13.9 in September of 2013, we go to install a VirtualCL cluster for our client.

58
00:06:30,280 --> 00:06:32,800
This is actually a 64 GPU cluster.

59
00:06:33,060 --> 00:06:35,740
We were just starting out, so this was actually a pretty big deal for us.

60
00:06:35,740 --> 00:06:38,480
64 GPUs was pretty fucking cool at the time.

61
00:06:39,780 --> 00:06:51,440
But we get the OS installed, we get the hardware built, all that shit, and then we go to install VirtualCL and FGLRX, and VirtualCL just shits the bed.

62
00:06:51,440 --> 00:07:02,340
So I panic and email Hebrew University, and I'm like, you guys really need to update VirtualCL for Catalyst and FGLRX 13.9.

63
00:07:02,780 --> 00:07:06,680
And they're like, well, we can't.

64
00:07:06,680 --> 00:07:09,200
Everyone who was working on this project has gone.

65
00:07:10,660 --> 00:07:18,520
And the intellectual property remains with Hebrew University and MOSIX, so they can just have those people work on it outside of that.

66
00:07:18,940 --> 00:07:26,420
I offered to take over the project for them, because it was kind of essential to what we were doing, and I didn't want to see the project die.

67
00:07:26,420 --> 00:07:27,680
I thought it was really neat.

68
00:07:28,940 --> 00:07:35,180
But the professor over there who was in charge of the project was not amenable to that idea.

69
00:07:35,180 --> 00:07:43,140
He actually had pretty grave concerns when we first started using VirtualCL for password cracking, because he envisioned us turning the world into a giant botnet.

70
00:07:43,540 --> 00:07:47,500
And of course, that wasn't really on our agenda, at least not at that time.

71
00:07:47,500 --> 00:07:57,660
But yeah, he really didn't like the idea of his VirtualCL shit ending up in our hands.

72
00:07:58,280 --> 00:08:04,640
So we were left high and dry without a software solution, and so we desperately needed something.

73
00:08:04,700 --> 00:08:07,500
So we conceived the idea of Hashtag.

74
00:08:09,700 --> 00:08:16,680
Initially, we thought about making our own VirtualCL clone, since Hebrew University was not going to give us the VirtualCL source code.

75
00:08:16,680 --> 00:08:18,180
And we're like, well, fuck it.

76
00:08:18,180 --> 00:08:21,600
We'll start our own VirtualCL with hookers and blackjack.

77
00:08:22,320 --> 00:08:30,700
But the more we actually sat down and thought about what that entailed, we determined the level of effort for that was just way too high.

78
00:08:30,760 --> 00:08:34,100
And the timeframe in which we needed it was way too short.

79
00:08:34,500 --> 00:08:41,680
We literally had hardware in-house that we're building for clients who now have no software solution because of VirtualCL.

80
00:08:45,970 --> 00:08:47,950
So we decided to make a distributed wrapper.

81
00:08:47,950 --> 00:08:50,470
And that's when we came up with the idea of Hashtag.

82
00:08:51,190 --> 00:08:53,570
Like I said, we had to iterate really quickly on this.

83
00:08:53,570 --> 00:08:59,190
So we went from a whiteboard session to production in less than two months.

84
00:08:59,230 --> 00:09:11,310
And this was basically me and Tom Steele locking ourselves in the office and ordering over $100 for the Taco Bell and just banging this out as fast and furious as we could.

85
00:09:12,630 --> 00:09:16,010
It had a traditional client-server-agent architecture.

86
00:09:16,650 --> 00:09:20,690
And it was Hashtag focused, but it actually had a generic plugin interface.

87
00:09:20,990 --> 00:09:25,230
We had plugins for... and this was before there was just one Hashtag.

88
00:09:25,230 --> 00:09:33,150
So we had Hashtag, OCL Hashtag Plus, OCL Hashtag Lite plugins.

89
00:09:33,150 --> 00:09:35,090
And then we had John the Ripper plugin.

90
00:09:36,110 --> 00:09:46,890
And then we had just a generic interface to where any cracking tool could be made to work with Hashtag as long as it adhered to a standard format that we had defined.

91
00:09:47,350 --> 00:09:50,310
And then it also had the ability to run arbitrary commands when idle.

92
00:09:51,010 --> 00:09:59,410
This was less for our clients and actually more for me because we were running Hashtag in-house as well for our cracking-as-a-service type stuff.

93
00:10:00,470 --> 00:10:04,290
And this is back in a time when GPU mining was still profitable.

94
00:10:04,370 --> 00:10:09,750
So when we weren't working on a password cracking job, I wanted this thing to be generating coins in the background.

95
00:10:09,750 --> 00:10:22,590
So the arbitrary idle exec command was literally just put in place so that people could mine Litecoin or Namecoin or whatever when their cluster was idle.

96
00:10:24,090 --> 00:10:26,270
So, you know, totally different time.

97
00:10:26,670 --> 00:10:28,170
But, I mean, it actually worked.

98
00:10:28,170 --> 00:10:32,530
It did exactly what it was supposed to do and it was good.

99
00:10:33,530 --> 00:10:38,570
And the whole goddamn thing was implemented in this fancy new language called Node.js.

100
00:10:38,570 --> 00:10:48,610
Again, this is early 2013, so Node.js was the new hotness and everyone was starting to move that direction with microservices and everything.

101
00:10:48,850 --> 00:10:54,270
It's not on this slide, but we implemented Hashtag version 1 entirely as microservices.

102
00:10:55,130 --> 00:11:03,210
And I shit you not, there was like 23 individual packages that comprised Hashtag and almost all of those were microservices.

103
00:11:05,350 --> 00:11:09,970
And we went that direction because that's kind of like programming zeitgeist at the time, right?

104
00:11:09,970 --> 00:11:17,170
Like everyone was pushing microservices and pushing asynchronous and, you know, NoSQL and all that other horseshit, right?

105
00:11:17,170 --> 00:11:22,810
And in the end, it created more headaches than not if we just had a monolithic fucking program.

106
00:11:23,490 --> 00:11:38,950
You know, maintaining like one server binary and one agent binary is way simpler than like, you know, maintaining 23 individual microservices and making sure that they're all synced and communicating properly and, you know, that the versions are correct,

107
00:11:38,950 --> 00:11:40,810
you know, across all 23 packages.

108
00:11:40,950 --> 00:11:48,510
And there's just a lot of unnecessary headache that was introduced with that, that, you know, we attempted to manage.

109
00:11:48,870 --> 00:11:51,430
But even then, like our clients would still kind of mess it up.

110
00:11:51,430 --> 00:12:03,470
You know, like they would upgrade like 10 of 20 packages and then like the other ones would just be stuck at old versions or, you know, one demon would die and another demon would know that it's dead, so then it would shit the bed.

111
00:12:03,470 --> 00:12:11,270
And anyway, so yeah, no JS, microservices, no SQL, MongoDB, all that shit.

112
00:12:11,370 --> 00:12:15,950
I tried to use all like the latest and greatest everything because at the time, that seemed like the best idea.

113
00:12:17,010 --> 00:12:19,710
And also, Hashtag was more than just workload distribution.

114
00:12:19,710 --> 00:12:25,450
So a lot of our clients at the time thought that Hashtag was just a GUI front-end for Hashcat.

115
00:12:25,850 --> 00:12:26,990
And we're like, no, it's not.

116
00:12:26,990 --> 00:12:29,530
It's not just a web UI for Hashcat.

117
00:12:29,530 --> 00:12:37,630
It actually does like, you know, multi-user and, you know, workload distribution with like a really complex four-dimensional queuing mechanism.

118
00:12:37,630 --> 00:12:42,310
Like you're trying to explain all this to the client and they're just like, you know, eyes glazed over and like, so it's a web UI for Hashcat.

119
00:12:42,470 --> 00:12:45,110
Like fucking yes, okay, it's a web UI for Hashcat.

120
00:12:46,090 --> 00:12:47,590
But it actually did a hell of a lot more than that.

121
00:12:47,590 --> 00:12:54,990
In fact, when you installed Hashtag on your box, it basically hijacked it and configured it exactly the way that we needed it to be configured.

122
00:12:54,990 --> 00:13:00,570
It would actually provision the entire server from just installing like the initial meta package.

123
00:13:01,410 --> 00:13:03,350
So it's a complete stack of packages.

124
00:13:03,350 --> 00:13:05,430
They get stack, Hashtag, that's where it comes from.

125
00:13:05,910 --> 00:13:10,170
So it handled driver installation, driver configuration, driver updates.

126
00:13:10,170 --> 00:13:20,130
And again, this is back at a time, and even nowadays still kind of a challenge, but this is back at a time when GPU drivers were a pain in the ass to install.

127
00:13:20,130 --> 00:13:22,570
And then once you got them installed, it was a pain in the ass to update.

128
00:13:23,150 --> 00:13:27,230
So the fact that we kind of handled this automagically was a pretty big value add.

129
00:13:27,970 --> 00:13:32,070
Also handled XOR configuration, headless servers.

130
00:13:32,350 --> 00:13:38,530
This is actually during a time where when you had a headless server, you had to have fucking dummy plugs connected to each of the GPUs.

131
00:13:38,530 --> 00:13:41,470
There's probably very few people watching this who actually remember that.

132
00:13:42,330 --> 00:13:51,850
But yeah, if you had a completely headless server, each one of those GPUs had to have a special dummy plug attached to it that would connect the necessary pins to trick it to think that there's actually a monitor attached.

133
00:13:51,990 --> 00:13:55,710
And we actually had to run XOR because even though these are headless,

134
00:14:00,570 --> 00:14:04,610
they don't have GPU clock rates or set like power tune profiles or any of that.

135
00:14:04,610 --> 00:14:19,630
So even though this is a headless server, we still had to have dummy plugs to think there's a monitor attached, still had to run XOR, and still had to configure XOR properly to enable us to control the fans and control the clocks and all the other shit that we needed to do.

136
00:14:20,030 --> 00:14:26,090
And then along those lines, Hashtag also did GPU tuning for you and also automatic overclocking.

137
00:14:26,090 --> 00:14:33,470
And this was kind of a double-edged sword.

138
00:14:33,510 --> 00:14:48,570
So when the R9 290X came out, at first it was like entirely unusable because of the way that AMD aggressively implemented power tune firmware, I think it was 2.0 at the time.

139
00:14:50,470 --> 00:14:56,070
But the GPUs would throttle so aggressively that it was like way slower than the 7970.

140
00:14:57,130 --> 00:14:59,050
And it was just essentially unusable.

141
00:14:59,050 --> 00:15:02,310
So then I wrote od6config.

142
00:15:05,310 --> 00:15:06,470
Sorry, did I say PowerTune?

143
00:15:06,470 --> 00:15:07,290
PowerTune is fucking NVIDIA.

144
00:15:07,390 --> 00:15:08,010
Overdrive.

145
00:15:08,010 --> 00:15:08,670
I'm sorry.

146
00:15:09,450 --> 00:15:09,830
Oh, no, no.

147
00:15:09,830 --> 00:15:10,770
PowerTune is AMD.

148
00:15:10,810 --> 00:15:11,790
PowerMizer is NVIDIA.

149
00:15:11,790 --> 00:15:12,570
All right, whatever.

150
00:15:13,870 --> 00:15:19,970
So od6config enabled us to properly configure the R9 290X.

151
00:15:20,930 --> 00:15:23,590
But it was good, but it was still kind of mediocre.

152
00:15:24,190 --> 00:15:35,130
But at that time, when I measured it, it was only drawing 300 watts, which is important because our server platforms like the Brutalis can only handle GPUs that drop to 300 watts.

153
00:15:35,130 --> 00:15:38,730
You draw more than that, you're going to blow a fuse on the PCI slots on the motherboard.

154
00:15:40,550 --> 00:15:52,410
So we had a automatic overclock profile for the R9 290X that we pushed out to our clients that made the GPU draw just about 300 watts.

155
00:15:52,410 --> 00:16:06,730
And then a little bit later down the road, about a year, year and a half later, AMD pushed out a new driver update that actually caused the power consumption to go up by a good 50 to 75 watts.

156
00:16:06,730 --> 00:16:09,270
And we did not know this.

157
00:16:09,270 --> 00:16:21,930
This kind of went undetected by us until we started getting a rash of clients who were seeing dead PCIe slots on their motherboards, and all of them had R9 290X-based solutions.

158
00:16:22,910 --> 00:16:27,190
And I finally busted out a kilowatt and figured out what the fuck was going on.

159
00:16:28,290 --> 00:16:50,170
But we actually had the ability, through all these mechanisms that we had in Hashtag at the time to enable the automatic overclocking and configuration and all that, to push out a new underclock profile to all of our clients to try to drop the power consumption down to keep it from blowing PCIe slots on the motherboards.

160
00:16:51,770 --> 00:16:52,970
So it's both good and bad.

161
00:16:52,970 --> 00:16:59,530
The fact that we're able to automatically overclock GPUs to help get the most speed out of the GPUs is awesome.

162
00:17:00,530 --> 00:17:05,310
But then in that one specific scenario, it was bad because we started blowing PCIe slot fuses.

163
00:17:05,450 --> 00:17:14,010
But then it was good that we had this functionality because then we could correct it through software, over the air, and automatically correct it on all of our client systems all at the same time.

164
00:17:15,590 --> 00:17:20,390
Which saved us a lot of money because we were spending hundreds of thousands of dollars on new motherboards.

165
00:17:21,790 --> 00:17:22,430
So yeah.

166
00:17:22,430 --> 00:17:25,830
And then, of course, Hashtag also had cluster resource monitoring.

167
00:17:28,330 --> 00:17:30,170
So there were some drawbacks.

168
00:17:30,170 --> 00:17:38,430
Like I said, the entire Godend thing was written in Node.js, and while Node.js is asynchronous, it also runs on a single CPU thread.

169
00:17:38,670 --> 00:17:45,570
That may have changed at some point, because we're talking like 2013, that's seven years ago, and I haven't even touched Node.js since.

170
00:17:45,730 --> 00:18:00,790
But at the time, everything ran on a single CPU thread, which was stupid, because our cluster controllers had a minimum of 12 CPU cores, 24 threads, and we're sitting here stuck in one thread on the server daemon.

171
00:18:01,610 --> 00:18:09,430
So yeah, any long-running methods would just totally block the entire application, and we had massive scalability issues.

172
00:18:10,210 --> 00:18:17,510
I think the very first alpha that we released to clients could only support four nodes.

173
00:18:17,510 --> 00:18:21,830
The second one, we were able to get up to nine or ten nodes.

174
00:18:22,290 --> 00:18:31,870
Basically, any time a client ordered a cluster larger than the previous largest cluster we had ever built, we ran into some kind of scalability issue, identified some bottlenecks, and had to work to remove those bottlenecks.

175
00:18:32,870 --> 00:18:40,110
Unfortunately, 90% of the time, the bottleneck was the fact that JavaScript was too slow for the task that we were trying to do.

176
00:18:40,290 --> 00:18:47,570
So we had to reimplement those bottleneck methods in a faster language, and then just shell out to them.

177
00:18:48,350 --> 00:18:51,070
So keyspace calculation was reimplemented in C.

178
00:18:51,070 --> 00:19:04,510
Now, calculating keyspace for a mask attack is really lightweight, but when you talk about massive dictionaries or rule files, trying to count the number of lines in a file with JavaScript is painfully slow.

179
00:19:05,350 --> 00:19:13,170
So that's largely why that entire portion was reimplemented in C, and we just shelled out to that binary to eliminate that bottleneck.

180
00:19:14,310 --> 00:19:16,430
We didn't dramatically reduce it.

181
00:19:17,850 --> 00:19:18,590
Regex.

182
00:19:18,590 --> 00:19:25,250
Anything Regex, we actually implemented, we had reimplemented it in Perl, because Perl is amazing at Regex.

183
00:19:25,250 --> 00:19:28,570
JavaScript is not very good at Regex.

184
00:19:28,950 --> 00:19:34,990
The primary thing that we were doing with Regex here was validating the hash lists.

185
00:19:34,990 --> 00:19:43,570
So a client uploads a hash list, and some of our clients wouldn't even bother throwing 200 million hashes at hashtag in one job.

186
00:19:43,970 --> 00:19:55,710
But then hashtag has to sit there and receive the massively large hash file, and then validate each individual hash, and then stuff them in a database.

187
00:19:56,130 --> 00:19:58,270
That was a massive bottleneck.

188
00:19:58,270 --> 00:20:02,830
So we ended up reimplementing all that in Perl to try to get that down.

189
00:20:02,830 --> 00:20:08,730
And then the agent, the entire fucking agent, we ended up just implementing the whole thing in Perl.

190
00:20:12,710 --> 00:20:19,090
And the only reason we picked Perl at the time is because that's the only language we could iterate fast enough in to rewrite the entire damn thing.

191
00:20:21,770 --> 00:20:23,090
But yeah.

192
00:20:23,430 --> 00:20:29,750
Like I said, we also used MongoDB because NoSQL was the rage at the time.

193
00:20:30,070 --> 00:20:34,450
And out of all the NoSQL databases we tested, Mongo was the most performant.

194
00:20:35,010 --> 00:20:39,190
But then we had clients do things like throw 200 million hashes at hashtag.

195
00:20:39,950 --> 00:20:51,390
And that would cause some not only bottlenecks and blocking processes, but some really UI weirdness as well.

196
00:20:51,530 --> 00:21:01,350
They would go through the process of creating a job, uploading the hash list, and they would just sit there with the little spinner for a while until it timed out.

197
00:21:01,830 --> 00:21:04,650
So they'd go to create the job again because it timed out.

198
00:21:04,650 --> 00:21:10,590
And then they'd go back 10 minutes later and it was actually active and running.

199
00:21:10,590 --> 00:21:16,090
But because they went through the process four times trying to get it to start, now they have like four of the same process running on the cluster.

200
00:21:16,470 --> 00:21:21,930
So we threw Redis in front of Mongo as an in-memory caching layer.

201
00:21:22,290 --> 00:21:25,670
And that actually had a significant impact, like positive impact.

202
00:21:25,850 --> 00:21:29,230
And it enabled us to ingest hashes a lot faster.

203
00:21:30,930 --> 00:21:34,570
But still not a very great solution overall.

204
00:21:36,190 --> 00:21:41,310
The Web UI client was implemented as a single-page application in Angular.

205
00:21:41,750 --> 00:21:43,470
It was really clunky.

206
00:21:43,470 --> 00:21:44,670
It was really heavy.

207
00:21:44,670 --> 00:21:46,090
It was really slow.

208
00:21:46,110 --> 00:21:50,830
It constantly caused the browser to freeze if you left it up for more than 30 minutes.

209
00:21:50,930 --> 00:21:52,910
And it just had a ton of bugs.

210
00:21:53,390 --> 00:22:02,610
And I think probably the most annoying thing is if you're like a hashcat power user and you sit down and try to use the hashtag Web UI, it was just cumbersome as fuck.

211
00:22:03,010 --> 00:22:06,390
There's no way to move quickly in that UI.

212
00:22:06,550 --> 00:22:09,290
And it gets very repetitive and very tedious.

213
00:22:10,370 --> 00:22:14,950
So there was a lot of drawbacks to having that Angular Web UI.

214
00:22:15,250 --> 00:22:16,770
I absolutely hated it.

215
00:22:16,770 --> 00:22:26,910
And anyone who was really experienced with cracking and hashcat and stuff, they also hated it and begged us not to have that again.

216
00:22:27,570 --> 00:22:30,830
And we were just severely limited by what we could do in a browser.

217
00:22:31,410 --> 00:22:36,390
For non-hash formats at the time, we were scraping everything server-side.

218
00:22:36,390 --> 00:22:53,910
So you would actually upload a PDF or upload a .docx or whatever, and we would scrape the file server-side to get the necessary bits, like Office to hashcat and PDF to hashcat, shit like that.

219
00:22:54,530 --> 00:23:04,690
The problem then became, what do we do with things like TrueCrypt volumes or really large 7-zip volumes or RAR files?

220
00:23:04,690 --> 00:23:14,390
Are you really going to upload a 100-gigabyte disk image through the browser just so we can scrape off two kilobytes?

221
00:23:14,390 --> 00:23:17,010
No, that doesn't make any sense.

222
00:23:19,690 --> 00:23:21,710
So yeah, the browser was an issue.

223
00:23:21,710 --> 00:23:25,970
And then we implemented access controls, but there was nothing fine-grained.

224
00:23:25,970 --> 00:23:38,770
We assumed that because everyone operating the hashcat cluster was on the same team, that they all had equal rights and equal access to the cluster, and therefore all users are admins.

225
00:23:38,770 --> 00:23:42,010
We didn't have different roles or anything.

226
00:23:42,010 --> 00:23:48,410
It was just, if you have an account, then you obviously have a right to be there, so therefore you are an admin.

227
00:23:50,970 --> 00:23:59,730
And I think some of our clients didn't have a problem with that, because everyone on this team does have equal rights and access and privilege to the cluster, so this is totally fine.

228
00:23:59,730 --> 00:24:02,390
Other clients were like, this is absolutely unacceptable.

229
00:24:04,810 --> 00:24:10,970
And I eventually got to the point where the old version one code was completely unmanageable.

230
00:24:11,370 --> 00:24:18,210
Even just implementing what should have been a minor bug fix ended up requiring major refactoring.

231
00:24:19,890 --> 00:24:35,550
It just wasn't, because we threw it together in less than two months, so we didn't think of everything in advance that would come up, and try to implement new features, and try to work around some of the bugs in there.

232
00:24:35,550 --> 00:24:39,910
So we decided we needed an entire ground-up rewipe for Hashtag version two.

233
00:24:41,630 --> 00:24:49,650
And at some point, I got the idea to create a commercial fork of Hashcat with native distributed capabilities.

234
00:24:49,850 --> 00:24:52,690
And at the time, I was so fucking stoked on this idea.

235
00:24:52,690 --> 00:24:57,090
It seemed like the best idea in the world to me at the time.

236
00:24:58,050 --> 00:25:01,330
So we set out to do just that.

237
00:25:02,150 --> 00:25:13,890
Now this is right before Hashtag went open source, and then we started work on this literally the minute that Hashtag went open source.

238
00:25:14,450 --> 00:25:20,290
So we implemented in Golang in C with Postgres database.

239
00:25:20,510 --> 00:25:26,910
Now we chose Golang because of high concurrency with goroutines, right?

240
00:25:27,090 --> 00:25:31,150
So with Node.js, everything ran on a single CPU thread.

241
00:25:31,150 --> 00:25:34,490
And we wanted the opposite of that for version two, right?

242
00:25:34,490 --> 00:25:35,990
We're going to learn the lesson on that.

243
00:25:35,990 --> 00:25:44,910
So we wanted just lots and lots of goroutines to run everything so that we'd use all 24 threads plus on the box.

244
00:25:46,870 --> 00:25:51,130
And Golang actually also allows for pretty rapid development.

245
00:25:51,130 --> 00:25:53,010
So we thought it was a pretty good choice.

246
00:25:54,850 --> 00:25:58,690
But we still stuck with the traditional client server agent architecture.

247
00:25:58,730 --> 00:26:05,250
We have numerous clients and then one server and then however many agents.

248
00:26:06,350 --> 00:26:10,630
Meaning that that server is both a bottleneck and a single point of failure.

249
00:26:11,910 --> 00:26:15,430
So the first thing we had to do is we had to split Hashcat in two.

250
00:26:15,990 --> 00:26:27,610
We took everything that had to do with actually starting a cracking job and executing an OpenCL kernel on a GPU with the agent and nothing else.

251
00:26:27,610 --> 00:26:35,990
The agent was just barebones trimmed down Hashcat launch attack on GPU and that was it, nothing more.

252
00:26:36,410 --> 00:26:39,750
And everything else we placed into the server library.

253
00:26:40,290 --> 00:26:51,810
And then for any code that wasn't already in Hashcat such as the JSON API and the protobufs for agent server communication workload distribution

254
00:26:55,930 --> 00:26:59,390
fine -grained multi-role access control.

255
00:26:59,470 --> 00:27:03,650
All that was implemented in Go and then we just linked against the C libraries.

256
00:27:03,650 --> 00:27:12,150
And this effort actually laid the foundation for the Hashcat 4.0 refactor with libHashcat.

257
00:27:12,610 --> 00:27:31,390
And then because of how clunky our old web UI was we actually ditched it entirely for a cross-platform CLI which made our power users like really happy but our less experienced users were not very thrilled with that decision.

258
00:27:31,390 --> 00:27:36,830
They still wanted a GUI which, fair enough.

259
00:27:38,090 --> 00:27:44,590
So our plan initially was to maintain both Hashcat and Hashstack trees in parallel.

260
00:27:44,830 --> 00:27:55,510
And then if there was any features that we added to Hashstack or any new kernels that we wrote we would just backport those as we chose to Hashcat.

261
00:27:58,550 --> 00:28:00,790
This didn't really work out very well in practice though.

262
00:28:00,790 --> 00:28:07,370
So when we released version 2 it was in sync with Hashcat 3.5 but it didn't stay that way for very long.

263
00:28:08,410 --> 00:28:18,370
And of course we had to add a cross-platform GUI to the roadmap as well because we had significant feedback from our clients that they really wanted the goddamn GUI and they were not happy that we took away the web UI.

264
00:28:19,890 --> 00:28:24,450
It turns out this was the single dumbest motherfucking thing I've ever done in my life.

265
00:28:25,590 --> 00:28:31,370
I completely underestimated how much work was involved in creating a commercial fork of Hashcat.

266
00:28:32,010 --> 00:28:33,230
Like, genuinely

267
00:28:36,470 --> 00:28:36,970
I...

268
00:28:37,770 --> 00:28:42,310
Yeah, I really don't know how to convey how fucking stupid of an idea this was.

269
00:28:43,110 --> 00:28:51,650
We spent like 100% of our time just backporting things from Hashcat into Hashtag instead of actually adding new features or writing new kernels.

270
00:28:52,290 --> 00:28:58,130
Like, literally months of work you know, just backporting Hashcat commits.

271
00:28:58,870 --> 00:29:00,690
And it got really frustrating.

272
00:29:00,910 --> 00:29:03,030
You know, it's like, hey, are we working on these tickets?

273
00:29:03,030 --> 00:29:04,230
Are we working on these features?

274
00:29:04,230 --> 00:29:05,770
No, we're working on backports.

275
00:29:05,990 --> 00:29:12,230
You know, I got so sick of hearing working on backports, working on backports, like when are we ever going to be done with backports?

276
00:29:13,430 --> 00:29:20,990
And then when Adam refactored Hashcat for 4.0, that also meant that we had to refactor Hashtag as well.

277
00:29:21,150 --> 00:29:27,050
And we hadn't even finished implementing all the backports at the speed that Adam was, you know, committing things to Hashcat.

278
00:29:27,170 --> 00:29:32,130
So this was just like, absolutely unmaintainable, right?

279
00:29:32,130 --> 00:29:34,250
Like, fuck me.

280
00:29:35,550 --> 00:29:43,450
And then there was also this other little minor problem where we picked the wrong HTTP library for the API.

281
00:29:44,130 --> 00:29:49,770
And this basically limited the service capability to about 12 nodes.

282
00:29:49,790 --> 00:29:52,830
Now, 12 nodes might seem like a lot.

283
00:29:52,830 --> 00:29:54,510
That's 96 GPUs.

284
00:29:54,510 --> 00:29:57,670
You know, that's a pretty respectable size cluster.

285
00:29:58,890 --> 00:30:07,230
The problem is this particular client had ordered 320 GPUs, 40 nodes.

286
00:30:07,510 --> 00:30:18,070
And you can't really deliver a 40 node client to a cluster with a server that can only really handle 12 nodes.

287
00:30:22,080 --> 00:30:28,180
So this required not a complete rewrite, but a damn near complete rewrite time.

288
00:30:28,180 --> 00:30:33,140
Fuck , because we use that HTTP library extensively throughout the code .

289
00:30:33,880 --> 00:30:36,060
And there was no, like, drop in replacement.

290
00:30:36,060 --> 00:30:45,840
So basically anywhere where we were dealing with HTTP , we had to rip out the old library and then, you know, add in all the stuff for the new library.

291
00:30:45,960 --> 00:30:47,540
But they did resolve the issue.

292
00:30:48,320 --> 00:30:51,400
It was just really painful at the time to have to mess with that.

293
00:30:53,420 --> 00:30:59,720
But the server still continued to be a single point of failure and still proved to be a source of bottlenecks.

294
00:30:59,760 --> 00:31:11,800
As we, you know, tried to build larger and larger clusters , we kept running into problems with the server being a bottleneck and a single point of failure .

295
00:31:11,920 --> 00:31:23,540
And we also had clients coming to us asking us if they could order multiple cluster controllers and cluster them together, you know, either high availability or, like, load balance them or something.

296
00:31:24,560 --> 00:31:30,440
And we didn't have a solution for that, but I started thinking about a solution for that.

297
00:31:31,980 --> 00:31:35,920
And again, like, 40 % of our clients were just completely pissed that we had no GUI .

298
00:31:37,080 --> 00:31:49,160
So it got to the point where it's like, let's just rethink, like, the entire fucking thing and just, like, go as far away from these mistakes as we could possibly get.

299
00:31:49,220 --> 00:31:51,860
Like, we never, ever want to make any of these mistakes ever again.

300
00:31:51,860 --> 00:31:55,920
We've learned a lot of lessons from, you know, what we've done over the last seven years.

301
00:31:57,340 --> 00:32:00,020
So, but we need more than just a complete ground-up rewrite.

302
00:32:00,020 --> 00:32:01,540
Like , we need an entire paradigm shift.

303
00:32:01,540 --> 00:32:11,120
Like , we need to rethink this entire problem entirely and come up with something that's completely different from what's ever been done.

304
00:32:11,940 --> 00:32:15,180
So we need to eliminate the single point of failure of the server.

305
00:32:15,300 --> 00:32:25,180
We need to enable, like, actual infinite horizontal scaling because, again, we're, you know, we always try to strive for, like, warehouse as a computer , right?

306
00:32:25,180 --> 00:32:29,520
We're looking at warehouse scale computing as , like, the target that we're trying to hit.

307
00:32:30,000 --> 00:32:39,560
There should never be any, you know, limiting factors outside of budget for, like, how many nodes we can support in HashtagWrite .

308
00:32:40,220 --> 00:32:44,740
And then, of course, the other big requirement is we have to actually move at the pace of Hashtag development.

309
00:32:46,040 --> 00:32:56,260
We have to, you know, as Adam commits things to Hashtag, we need to have those not instantly available in Hashtag, but within a very reasonable time frame.

310
00:32:58,880 --> 00:33:07,460
So the very first idea that we came up with was let's not do a traditional client-server-agent model.

311
00:33:08,440 --> 00:33:11,400
And let's not even try to cluster the cluster controllers.

312
00:33:11,740 --> 00:33:19,860
Let's just eliminate the cluster controllers entirely and make everything peer-to-peer where all nodes are equal and there are no dedicated servers.

313
00:33:20,860 --> 00:33:25,760
And the biggest requirement to this is that work can be submitted to any node.

314
00:33:26,460 --> 00:33:32,000
So let's say you have, like, you know, 20 users and you have, you know, 10 boxes.

315
00:33:32,000 --> 00:33:34,900
Like, there shouldn't be just one node that all of them have to hit.

316
00:33:34,900 --> 00:33:40,600
They should be able to hit any node in the cluster and, you know, have the same view from any node.

317
00:33:41,880 --> 00:33:44,140
And it also needs to be more user-friendly than ever.

318
00:33:44,400 --> 00:33:46,340
And also needs to run on Windows.

319
00:33:48,100 --> 00:33:49,640
I'll tell you why in a minute.

320
00:33:52,240 --> 00:33:55,320
So set out to, like, figure out how we're going to implement this.

321
00:33:55,320 --> 00:33:57,540
Because we have the high-level requirements, right?

322
00:33:57,540 --> 00:33:59,380
Like, we know it needs to be peer-to-peer.

323
00:33:59,380 --> 00:34:01,060
We know all the nodes need to be equal.

324
00:34:01,240 --> 00:34:02,520
You know, all that shit.

325
00:34:03,040 --> 00:34:04,740
But how are we actually going to do this?

326
00:34:05,600 --> 00:34:09,880
So pretty obviously we're going to need some sort of distributed state machine.

327
00:34:10,040 --> 00:34:10,680
Right?

328
00:34:11,460 --> 00:34:14,520
And that's just straightforward and self-explanatory.

329
00:34:14,780 --> 00:34:21,500
But, you know, like, we just throw a message queue at it and some PubSub and, you know, Bob's your uncle.

330
00:34:22,160 --> 00:34:23,860
But what about disagreements?

331
00:34:24,060 --> 00:34:31,100
Or, you know, maybe even more than disagreements, what about, like, an actual, you know, rogue cluster member?

332
00:34:31,220 --> 00:34:42,700
Or, you know, a broken cluster node that's maybe not necessarily intentionally malicious, but it certainly, you know, has, you know, the appearance of malicious behavior.

333
00:34:42,700 --> 00:34:43,500
Right?

334
00:34:44,620 --> 00:34:48,060
So I started coming up with solutions for this.

335
00:34:48,680 --> 00:34:56,580
And I somehow got into my head, like, dude, what if computers could vote?

336
00:34:57,000 --> 00:34:57,840
Right?

337
00:34:57,840 --> 00:35:08,740
Like, you just make it to where, like, you know, if there's a dispute , then one of the nodes, like, requests a vote, right?

338
00:35:09,060 --> 00:35:11,060
And so they take a vote and majority wins.

339
00:35:11,060 --> 00:35:19,220
And if there's a tie, like we have an even number of nodes in the cluster, then, like, literally fucking flip a coin, you know, randomly, or play rock, paper, scissors.

340
00:35:19,360 --> 00:35:20,780
Like, why not?

341
00:35:22,100 --> 00:35:26,380
And if a peer doesn't accept the results, like, then the rest of the cluster just shuns that peer.

342
00:35:26,800 --> 00:35:29,380
And I started thinking about, like, is this actually, like, a thing?

343
00:35:29,380 --> 00:35:30,380
Like, can we actually do this?

344
00:35:30,380 --> 00:35:32,400
Am I, like , completely fucking stoned?

345
00:35:34,600 --> 00:35:38,120
It turns out that I really didn't have to put that much more thought into it because this already exists.

346
00:35:38,120 --> 00:35:42,800
Like, what I thought up is actually a thing called the wrapped consensus protocol .

347
00:35:43,620 --> 00:35:48,520
Now , it's cool that it already exists because that, you know , means we don't have to do a whole lot to implement it.

348
00:35:48,520 --> 00:35:54,260
But it's also cool because it kind of validated, you know, my ideas and theories on how to make this work.

349
00:35:55,080 --> 00:35:57,400
And told me that I wasn't just completely fucking insane.

350
00:35:58,380 --> 00:36:02,460
And we found several implementations of the wrapped consensus protocol.

351
00:36:03,640 --> 00:36:09,620
And the one that we ended up liking the most and settled on is Akka, which runs on the JVM.

352
00:36:09,620 --> 00:36:10,660
It's written in Scala.

353
00:36:11,140 --> 00:36:14,180
Now, I hate Java.

354
00:36:14,180 --> 00:36:18,500
I have shunned Java for damn near my entire life.

355
00:36:18,740 --> 00:36:23,700
You know, I completely condemn Oracle JVM.

356
00:36:26,660 --> 00:36:29,900
But, there is OpenJDK.

357
00:36:29,900 --> 00:36:31,360
We don't have to use Oracle.

358
00:36:31,580 --> 00:36:34,160
And we don't have to write in Java.

359
00:36:34,160 --> 00:36:35,860
We can use Scala and Kotlin.

360
00:36:36,780 --> 00:36:40,920
And since I've started to work on this, I actually really like Kotlin.

361
00:36:41,780 --> 00:36:44,860
I don't really have too many negative things to say about it.

362
00:36:44,860 --> 00:36:47,180
It's actually kind of fun to work with.

363
00:36:48,420 --> 00:36:57,160
And then Akka also has some plugins or modules that, you know, help us even further.

364
00:36:57,160 --> 00:36:58,880
Such as Akka persistence.

365
00:36:58,880 --> 00:37:06,120
What Akka persistence does, it takes our distributed state machine, which is normally resident in memory, and persists it to disk.

366
00:37:06,600 --> 00:37:11,960
And then Akka distributed data, of course, would be like what implements the distributed state machine.

367
00:37:12,420 --> 00:37:24,220
And then on top of all of that, we built a custom multicast -based protocol to enable cluster nodes to automatically discover each other on the network and then automatically join the cluster.

368
00:37:25,020 --> 00:37:31,400
Now, we made this opt-in because obviously that's kind of a security risk if anybody can just join the cluster at will.

369
00:37:33,880 --> 00:37:48,420
But for some scenarios like where all of your nodes are on a dedicated subnet that's properly firewalled off and you have proper layer 3 and layer 4 access controls, this would be a solution that would be really handy.

370
00:37:48,540 --> 00:38:07,500
So , again, thinking about warehouse -scale computing or data center as a computer , let's say you just have racks and racks and racks of compute nodes, you would just unbox one, literally throw it in the rack and power it on, and it would discover the cluster as soon as it's powered on and join the cluster and start doing work.

371
00:38:07,660 --> 00:38:09,840
That's kind of the vision that we have for this.

372
00:38:11,420 --> 00:38:17,280
And, of course, since it's all peer-to-peer, that enables it to actually be just literally that simple.

373
00:38:19,720 --> 00:38:25,040
But, of course, the default is you manually specify which peers you want to be part of the swarm.

374
00:38:29,110 --> 00:38:29,410
Okay.

375
00:38:29,410 --> 00:38:35,950
And then, in April this year, we had the unique opportunity to present itself to purchase loft crack.

376
00:38:37,210 --> 00:38:54,570
Dildog approached me and we entered negotiations and we purchased it earlier this year primarily for the GUI, which is written in C++ using QT as the windowing framework.

377
00:38:54,810 --> 00:39:02,470
And while it's not currently cross-platform, that's a really easy platform , or that's a really easy language and library to make cross-platform .

378
00:39:04,270 --> 00:39:12,590
So, instead of hashtag version 3 , we now have loft crack version 8.

379
00:39:13,970 --> 00:39:26,710
And while loft crack 7 uses John the Ripper as a backend , we've ripped all that out and we are in the process of integrating what was hashtag version 3 as the new backend for loft crack 8.

380
00:39:28,670 --> 00:39:35,310
And the final thing we're doing with this is we are dropping the TeraHash hardware requirement.

381
00:39:35,490 --> 00:39:42,830
Traditionally, you know, we do not let people purchase hashtag as a standalone product.

382
00:39:42,830 --> 00:39:50,870
We say, like, you know, the only way to get hashtag is to buy a TeraHash appliance and it comes preinstalled and that's the only way to get it.

383
00:39:51,430 --> 00:39:53,890
But loft crack has always been sold standalone.

384
00:39:54,350 --> 00:40:07,730
So we are changing the model now to where, you know, keeping with loft crack being sold standalone, we're going to be selling loft crack 8 as a standalone product without the TeraHash hardware requirement.

385
00:40:10,030 --> 00:40:14,270
So , yeah, that's where we are now with development.

386
00:40:14,270 --> 00:40:16,270
That's how the product has evolved over time.

387
00:40:16,270 --> 00:40:26,730
And that's some of the challenges that we have faced trying to obtain our goal of true warehouse scale computing.

388
00:40:26,830 --> 00:40:29,010
Thank you and hope you've enjoyed this.