start	end	text
0	6080	This is Jeremy Gosney. I'm coming to you live from Password Village HQ2 in sunny Austin, Texas.
6080	8400	Today's high is 104 degrees.
8880	15200	So today I'm going to be talking to you about what it's like cracking at Terrahash's scale.
16720	18920	I'm not going to make this a vendor pitch in any way.
18920	25900	I'm going to focus on the technical challenges that we encounter as we support clients
25900	28920	who have hundreds of GPUs in their clusters.
30620	35080	When I first started on this, I thought 100 GPUs was a really big deal.
35640	39660	Now we tend to view 100 GPUs as like, meh.
40660	43300	It's good, but it's kind of middle of the road.
47260	51880	There's a lot of distributed solutions for HashCat out there.
52360	58440	But as far as I know, Terrahash is the only one that aims to operate at warehouse scale.
58440	65340	What I mean by that is, more so with the most recent version, but with every version that passes,
65340	70020	we aim to embrace the concepts of data center as a computer.
70560	76820	While we currently don't have any clients who have an entire warehouse full of password cracking hardware,
76820	79300	that's the point where we're trying to go.
80880	87720	We try to enable the software to be at that level, even if our clients are not quite at that level.
90560	98400	I'm going to go through the history of how I got started in massive distributed cracking.
98520	106540	And I'm going to go through the technical challenges that we faced as we continued to evolve our hashtag distributed solution.
107580	109600	I'm going to start the slide deck.
117330	118810	All right.
121480	127300	So, in the beginning, I kind of got into this because I was messing around with distributed computing.
127300	136720	I was doing stuff with MPI, some other MOSIX software, rocks clusters, things along those natures.
137820	139840	But all that was all CPU-based.
139840	149900	And then when GPU cracking came about, we started looking for solutions that would enable us to distribute GPU workloads.
149940	154840	So, I found a piece of software at a Hebrew university called VirtualCL.
155160	164060	And I did a presentation on a 25 GPU proof of concept cluster that me and Bitweasel put together at passwords12 in Oslo.
164300	169740	And when I woke up the next morning after giving that presentation, we had gone viral.
169840	176060	On the front page of Slashdot, Gizmodo, Boing Boing, NBC News, The Register, we were everywhere.
177140	181480	And in hindsight, I really should have used a lot more than 25 GPUs.
181480	185300	I mean, we had more GPUs, that's just what we chose to dedicate to this proof of concept.
185300	192540	And had I known in advance in any way that the general public would react how they did to this proof of concept,
192540	195160	I would have used like three times the number of GPUs.
197060	202940	So just, you know, some of the headlines that came out the next morning that I woke up to,
202940	205980	which was pretty surreal because I didn't really see this as a big deal.
207360	215500	So, VirtualCL was cool at the time because it was the first solution that I knew of that enabled us to
216300	222820	sort of transparently distribute OpenCL jobs across the arbitrary cluster.
222820	227080	So what VirtualCL does is it provides a virtual OpenCL platform.
228620	234080	And the target software, in theory, doesn't need to be aware it's communicating with a virtual platform.
234080	240400	It just sees any, you know, it sees VirtualCL as any other, you know, OpenCL platform,
240400	243160	just like AMD, NVIDIA, Intel, what have you.
244560	250200	So the target software just simply links against libOpenCL, which is provided by VCL,
250200	255140	which then communicates to a broker daemon, and then that broker daemon distributes to the agent daemons,
255140	257400	and the agent daemons, which are installed on the compute nodes,
258100	264440	actually communicate with the real OpenCL library, be it AMD, NVIDIA, whatever.
266040	269680	So in theory, all this is transparent to the end application.
269680	272420	But in practice, it didn't quite work that way.
273380	278440	We had to have a special fork of OCLHashCat called VCLHashCat,
278440	282160	which had some workarounds for VCL quirks and such.
283280	287760	It also required InfiniBand, because it was really latency sensitive.
287980	293140	It ended up being a very chatty protocol, and it required essentially real-time communication.
293400	295000	Bandwidth wasn't so much of an issue.
295000	299780	It pretty much consistently pulled, you know, less than one gig of bandwidth,
299780	303600	but it was really sensitive to latency.
303960	308040	Even just a couple milliseconds was acceptable,
308040	310760	but as you start getting up into double digits of latency,
310760	313620	you had a noticeable impact on hash rate.
314480	316920	And of course, InfiniBand hardware is expensive.
316920	319300	It's way more expensive than Ethernet.
320240	323520	VCL was also closed source, and it required frequent updates.
323520	328120	So this is back in the time, this is 2012 through 2013,
328120	333300	where every GPU driver that was released broke some shit.
333300	336480	And then we had to go back in and implement workarounds and fixes,
336480	340540	whatever the fuck AMD or NVIDIA broke in their driver that day.
340540	342660	And VCL was no exception.
342660	348600	So with VCLHashCat, we had to have our own workarounds for VirtualCL,
348600	351780	and then the VCL team, or VirtualCL team,
351780	356560	had to turn around and implement any workarounds for any new driver releases
356560	358440	every time they were released.
358440	362280	Which is a pretty substantial job.
362560	365180	The problem is, VirtualCL was created by grad school students.
365180	368850	This was a grad school project, and they created this their senior year.
369360	373680	And when they graduated in spring 2013, that was it.
373680	375300	The project died.
375860	380720	So when the next version of FGLRX was released,
380720	384560	I think that was version 13.9 in September of 2013,
385600	390280	we go to install a VirtualCL cluster for our client.
390280	392800	This is actually a 64 GPU cluster.
393060	395740	We were just starting out, so this was actually a pretty big deal for us.
395740	398480	64 GPUs was pretty fucking cool at the time.
399780	404480	But we get the OS installed, we get the hardware built, all that shit,
404480	408980	and then we go to install VirtualCL and FGLRX,
409500	411440	and VirtualCL just shits the bed.
411440	414800	So I panic and email Hebrew University,
414800	416760	and I'm like, you guys really need to update VirtualCL
418240	422340	for Catalyst and FGLRX 13.9.
422780	426680	And they're like, well, we can't.
426680	429200	Everyone who was working on this project has gone.
430660	434660	And the intellectual property remains with Hebrew University and MOSIX,
434660	438520	so they can just have those people work on it outside of that.
438940	441380	I offered to take over the project for them,
442420	445040	because it was kind of essential to what we were doing,
445040	446420	and I didn't want to see the project die.
446420	447680	I thought it was really neat.
448940	452580	But the professor over there who was in charge of the project
452580	455180	was not amenable to that idea.
455180	457000	He actually had pretty grave concerns
457000	459460	when we first started using VirtualCL for password cracking,
459460	463140	because he envisioned us turning the world into a giant botnet.
463540	465860	And of course, that wasn't really on our agenda,
465860	467500	at least not at that time.
467500	472340	But yeah, he really didn't like the idea
472340	477660	of his VirtualCL shit ending up in our hands.
478280	482000	So we were left high and dry without a software solution,
482000	484640	and so we desperately needed something.
484700	487500	So we conceived the idea of Hashtag.
489700	492380	Initially, we thought about making our own VirtualCL clone,
492380	496680	since Hebrew University was not going to give us the VirtualCL source code.
496680	498180	And we're like, well, fuck it.
498180	501600	We'll start our own VirtualCL with hookers and blackjack.
502320	506380	But the more we actually sat down and thought about what that entailed,
506380	510700	we determined the level of effort for that was just way too high.
510760	514100	And the timeframe in which we needed it was way too short.
514500	517800	We literally had hardware in-house that we're building for clients
517800	521680	who now have no software solution because of VirtualCL.
525970	527950	So we decided to make a distributed wrapper.
527950	530470	And that's when we came up with the idea of Hashtag.
531190	533570	Like I said, we had to iterate really quickly on this.
533570	539190	So we went from a whiteboard session to production in less than two months.
539230	543330	And this was basically me and Tom Steele locking ourselves in the office
544850	547430	and ordering over $100 for the Taco Bell
547950	551310	and just banging this out as fast and furious as we could.
552630	556010	It had a traditional client-server-agent architecture.
556650	558570	And it was Hashtag focused,
558570	560690	but it actually had a generic plugin interface.
560990	565230	We had plugins for... and this was before there was just one Hashtag.
565230	573150	So we had Hashtag, OCL Hashtag Plus, OCL Hashtag Lite plugins.
573150	575090	And then we had John the Ripper plugin.
576110	578970	And then we had just a generic interface
578970	581650	to where any cracking tool could be made to work with Hashtag
581650	586890	as long as it adhered to a standard format that we had defined.
587350	590310	And then it also had the ability to run arbitrary commands when idle.
591010	593690	This was less for our clients and actually more for me
593690	596270	because we were running Hashtag in-house as well
596270	599410	for our cracking-as-a-service type stuff.
600470	604290	And this is back in a time when GPU mining was still profitable.
604370	606730	So when we weren't working on a password cracking job,
606730	609750	I wanted this thing to be generating coins in the background.
609750	614930	So the arbitrary idle exec command was literally just put in place
614930	619490	so that people could mine Litecoin or Namecoin or whatever
619490	622590	when their cluster was idle.
624090	626270	So, you know, totally different time.
626670	628170	But, I mean, it actually worked.
628170	632530	It did exactly what it was supposed to do and it was good.
633530	636270	And the whole goddamn thing was implemented
636270	638570	in this fancy new language called Node.js.
638570	644370	Again, this is early 2013, so Node.js was the new hotness
644370	646830	and everyone was starting to move that direction
646830	648610	with microservices and everything.
648850	652470	It's not on this slide, but we implemented Hashtag version 1
652470	654270	entirely as microservices.
655130	659110	And I shit you not, there was like 23 individual packages
659110	663210	that comprised Hashtag and almost all of those were microservices.
665350	667890	And we went that direction because that's kind of like
668570	669970	programming zeitgeist at the time, right?
669970	672930	Like everyone was pushing microservices and pushing asynchronous
673370	677170	and, you know, NoSQL and all that other horseshit, right?
677170	680190	And in the end, it created more headaches than not
680190	682810	if we just had a monolithic fucking program.
683490	686970	You know, maintaining like one server binary
686970	691570	and one agent binary is way simpler than like, you know,
691570	694270	maintaining 23 individual microservices and making sure
694270	695990	that they're all synced and communicating properly
695990	699230	and, you know, that the versions are correct, you know,
699230	700810	across all 23 packages.
700950	703310	And there's just a lot of unnecessary headache
703310	706270	that was introduced with that, that, you know,
706270	708510	we attempted to manage.
708870	711430	But even then, like our clients would still kind of mess it up.
711430	716010	You know, like they would upgrade like 10 of 20 packages
716010	718030	and then like the other ones would just be stuck at old versions
718030	721770	or, you know, one demon would die and another demon would know
721770	723470	that it's dead, so then it would shit the bed.
723470	729470	And anyway, so yeah, no JS, microservices,
729470	731270	no SQL, MongoDB, all that shit.
731370	733310	I tried to use all like the latest and greatest everything
733310	735950	because at the time, that seemed like the best idea.
737010	739710	And also, Hashtag was more than just workload distribution.
739710	741290	So a lot of our clients at the time thought
741290	745450	that Hashtag was just a GUI front-end for Hashcat.
745850	746990	And we're like, no, it's not.
746990	749530	It's not just a web UI for Hashcat.
749530	753370	It actually does like, you know, multi-user and, you know,
753470	755970	workload distribution with like a really complex
755970	757630	four-dimensional queuing mechanism.
757630	758830	Like you're trying to explain all this to the client
758830	760370	and they're just like, you know, eyes glazed over
760370	762310	and like, so it's a web UI for Hashcat.
762470	765110	Like fucking yes, okay, it's a web UI for Hashcat.
766090	767590	But it actually did a hell of a lot more than that.
767590	770290	In fact, when you installed Hashtag on your box,
770290	773070	it basically hijacked it and configured it
773070	774990	exactly the way that we needed it to be configured.
774990	776810	It would actually provision the entire server
776810	780570	from just installing like the initial meta package.
781410	783350	So it's a complete stack of packages.
783350	785430	They get stack, Hashtag, that's where it comes from.
785910	788190	So it handled driver installation,
788190	790170	driver configuration, driver updates.
790170	791630	And again, this is back at a time,
791630	793090	and even nowadays still kind of a challenge,
793090	798990	but this is back at a time when GPU drivers
798990	800130	were a pain in the ass to install.
800130	801210	And then once you got them installed,
801210	802570	it was a pain in the ass to update.
803150	804530	So the fact that we kind of handled this
804530	807230	automagically was a pretty big value add.
807970	809490	Also handled XOR configuration,
810870	812070	headless servers.
812350	813550	This is actually during a time
813550	815270	where when you had a headless server,
815270	816650	you had to have fucking dummy plugs
816650	818530	connected to each of the GPUs.
818530	820150	There's probably very few people watching this
820150	821470	who actually remember that.
822330	824250	But yeah, if you had a completely headless server,
824250	825410	each one of those GPUs had to have
825510	827190	a special dummy plug attached to it
827650	829330	that would connect the necessary pins
829330	830630	to trick it to think that there's actually
830770	831850	a monitor attached.
831990	833610	And we actually had to run XOR
833610	835710	because even though these are headless,
840570	841410	they don't have GPU clock rates
841830	843670	or set like power tune profiles
843670	844610	or any of that.
844610	846610	So even though this is a headless server,
846610	847770	we still had to have dummy plugs
847770	849090	to think there's a monitor attached,
849090	850130	still had to run XOR,
850130	853190	and still had to configure XOR properly
854010	856750	to enable us to control the fans
856750	857970	and control the clocks
857970	859630	and all the other shit that we needed to do.
860030	862110	And then along those lines,
862110	864290	Hashtag also did GPU tuning for you
864290	866090	and also automatic overclocking.
866090	873470	And this was kind of a double-edged sword.
873510	876710	So when the R9 290X came out,
876710	878490	at first it was like entirely unusable
879430	882190	because of the way that AMD aggressively
882190	885910	implemented power tune firmware,
885910	888570	I think it was 2.0 at the time.
890470	893490	But the GPUs would throttle so aggressively
893490	896070	that it was like way slower than the 7970.
897130	899050	And it was just essentially unusable.
899050	902310	So then I wrote od6config.
905310	906470	Sorry, did I say PowerTune?
906470	908670	PowerTune is fucking NVIDIA. Overdrive. I'm sorry.
909450	910770	Oh, no, no. PowerTune is AMD.
910810	912570	PowerMizer is NVIDIA. All right, whatever.
913870	916890	So od6config enabled us
916890	919970	to properly configure the R9 290X.
920930	922250	But it was good,
922250	923590	but it was still kind of mediocre.
924190	926010	But at that time, when I measured it,
926090	927910	it was only drawing 300 watts,
927910	928710	which is important
929450	931890	because our server platforms like the Brutalis
932450	935130	can only handle GPUs that drop to 300 watts.
935130	936970	You draw more than that, you're going to blow a fuse
936970	938730	on the PCI slots on the motherboard.
940550	944610	So we had a automatic overclock profile
944610	946730	for the R9 290X that we pushed out to our clients
947830	952410	that made the GPU draw just about 300 watts.
952410	956510	And then a little bit later down the road,
956510	958310	about a year, year and a half later,
958310	961210	AMD pushed out a new driver update
961210	963530	that actually caused the power consumption
963530	966730	to go up by a good 50 to 75 watts.
966730	969270	And we did not know this.
969270	971030	This kind of went undetected by us
971030	975450	until we started getting a rash of clients
976030	979530	who were seeing dead PCIe slots on their motherboards,
979530	981930	and all of them had R9 290X-based solutions.
982910	985230	And I finally busted out a kilowatt
985230	987190	and figured out what the fuck was going on.
988290	990510	But we actually had the ability,
990510	995290	through all these mechanisms that we had in Hashtag
995290	998670	at the time to enable the automatic overclocking
998670	1000490	and configuration and all that,
1000490	1003350	to push out a new underclock profile
1003350	1005870	to all of our clients to try to drop
1005870	1007450	the power consumption down to keep it
1007450	1010170	from blowing PCIe slots on the motherboards.
1011770	1012970	So it's both good and bad.
1012970	1015470	The fact that we're able to automatically overclock GPUs
1016150	1019530	to help get the most speed out of the GPUs is awesome.
1020530	1022590	But then in that one specific scenario,
1022590	1025310	it was bad because we started blowing PCIe slot fuses.
1025450	1026930	But then it was good that we had this functionality
1026930	1029330	because then we could correct it through software,
1029330	1032090	over the air, and automatically correct it
1032090	1034010	on all of our client systems all at the same time.
1035590	1036670	Which saved us a lot of money
1036670	1039110	because we were spending hundreds of thousands of dollars
1039110	1040390	on new motherboards.
1041790	1042430	So yeah.
1042430	1044210	And then, of course, Hashtag also had
1044210	1045830	cluster resource monitoring.
1048330	1050170	So there were some drawbacks.
1050170	1051830	Like I said, the entire Godend thing
1051830	1053050	was written in Node.js,
1053050	1055350	and while Node.js is asynchronous,
1055350	1058430	it also runs on a single CPU thread.
1058670	1060430	That may have changed at some point,
1060430	1061930	because we're talking like 2013,
1061930	1062910	that's seven years ago,
1062910	1065570	and I haven't even touched Node.js since.
1065730	1067710	But at the time, everything ran
1069110	1069430	on a single CPU thread,
1069430	1070010	which was stupid,
1070010	1071110	because our cluster controllers
1071110	1074570	had a minimum of 12 CPU cores,
1074570	1075450	24 threads,
1075450	1078230	and we're sitting here stuck in one thread
1078850	1080790	on the server daemon.
1081610	1082890	So yeah, any long-running methods
1082890	1084570	would just totally block
1084570	1085950	the entire application,
1085950	1089430	and we had massive scalability issues.
1090210	1093510	I think the very first alpha
1093510	1095010	that we released to clients
1095010	1097510	could only support four nodes.
1097510	1098750	The second one,
1098750	1101830	we were able to get up to nine or ten nodes.
1102290	1103670	Basically, any time a client ordered
1103830	1105230	a cluster larger than the previous
1105230	1106950	largest cluster we had ever built,
1106950	1108950	we ran into some kind of scalability issue,
1108950	1110130	identified some bottlenecks,
1110130	1111870	and had to work to remove those bottlenecks.
1112870	1115350	Unfortunately, 90% of the time,
1115350	1116370	the bottleneck was the fact
1116370	1118030	that JavaScript was too slow
1118030	1120110	for the task that we were trying to do.
1120290	1121650	So we had to reimplement
1122830	1124110	those bottleneck methods
1124110	1125210	in a faster language,
1125210	1127570	and then just shell out to them.
1128350	1129590	So keyspace calculation
1129590	1131070	was reimplemented in C.
1131070	1133230	Now, calculating keyspace for a mask attack
1133230	1134550	is really lightweight,
1134550	1136830	but when you talk about massive dictionaries
1136830	1139090	or rule files,
1139090	1140670	trying to count the number of lines
1140670	1142330	in a file with JavaScript
1142330	1144510	is painfully slow.
1145350	1146710	So that's largely why
1146710	1149270	that entire portion was reimplemented in C,
1149270	1151190	and we just shelled out to that binary
1151810	1153170	to eliminate that bottleneck.
1154310	1156430	We didn't dramatically reduce it.
1157850	1158590	Regex.
1158590	1160110	Anything Regex,
1160110	1161650	we actually implemented,
1161650	1162850	we had reimplemented it in Perl,
1162850	1165250	because Perl is amazing at Regex.
1165250	1168570	JavaScript is not very good at Regex.
1168950	1170610	The primary thing
1170610	1171810	that we were doing with Regex here
1171810	1174990	was validating the hash lists.
1174990	1177510	So a client uploads a hash list,
1177510	1179250	and some of our clients
1179250	1180210	wouldn't even bother throwing
1180210	1182010	200 million hashes at hashtag
1182010	1183570	in one job.
1183970	1185550	But then hashtag has to sit there
1185550	1190610	and receive the massively large hash file,
1190610	1193330	and then validate each individual hash,
1193330	1195710	and then stuff them in a database.
1196130	1198270	That was a massive bottleneck.
1198270	1199010	So we ended up reimplementing
1199590	1200730	all that in Perl
1200730	1202830	to try to get that down.
1202830	1204310	And then the agent,
1204310	1205710	the entire fucking agent,
1205710	1207410	we ended up just implementing
1207410	1208730	the whole thing in Perl.
1212710	1213930	And the only reason
1213930	1214790	we picked Perl at the time
1214790	1215930	is because that's the only language
1215930	1217250	we could iterate fast enough in
1217250	1219090	to rewrite the entire damn thing.
1221770	1223090	But yeah.
1223430	1224910	Like I said, we also used MongoDB
1226090	1229750	because NoSQL was the rage at the time.
1230070	1232590	And out of all the NoSQL databases we tested,
1232590	1234450	Mongo was the most performant.
1235010	1236790	But then we had clients do things
1236790	1239190	like throw 200 million hashes at hashtag.
1239950	1241490	And that would cause
1242970	1244330	some not only bottlenecks
1245290	1247110	and blocking processes,
1247110	1251390	but some really UI weirdness as well.
1251530	1253270	They would go through the process
1253270	1254490	of creating a job,
1254490	1255850	uploading the hash list,
1255850	1256890	and they would just sit there
1256890	1257790	with the little spinner
1258690	1261350	for a while until it timed out.
1261830	1263510	So they'd go to create the job again
1263510	1264650	because it timed out.
1264650	1266510	And then they'd go back 10 minutes later
1269190	1270590	and it was actually active and running.
1270590	1272430	But because they went through the process four times
1272430	1273390	trying to get it to start,
1273390	1274910	now they have like four of the same process
1274910	1276090	running on the cluster.
1276470	1279190	So we threw Redis in front of Mongo
1279190	1281930	as an in-memory caching layer.
1282290	1284330	And that actually had a significant impact,
1284330	1285670	like positive impact.
1285850	1287910	And it enabled us to ingest hashes
1288110	1289230	a lot faster.
1290930	1294570	But still not a very great solution overall.
1296190	1298510	The Web UI client was implemented
1298510	1301310	as a single-page application in Angular.
1301750	1303470	It was really clunky.
1303470	1304670	It was really heavy.
1304670	1306090	It was really slow.
1306110	1308350	It constantly caused the browser to freeze
1308350	1310830	if you left it up for more than 30 minutes.
1310930	1312910	And it just had a ton of bugs.
1313390	1314790	And I think probably the most annoying thing
1314790	1316370	is if you're like a hashcat power user
1316370	1317770	and you sit down and try to use
1317770	1319670	the hashtag Web UI,
1319670	1322610	it was just cumbersome as fuck.
1323010	1324550	There's no way to move quickly
1324570	1326390	in that UI.
1326550	1327810	And it gets very repetitive
1327810	1329290	and very tedious.
1330370	1332390	So there was a lot of drawbacks
1332390	1334950	to having that Angular Web UI.
1335250	1336770	I absolutely hated it.
1336770	1340370	And anyone who was really experienced
1340370	1342310	with cracking and hashcat and stuff,
1342310	1343550	they also hated it
1343550	1346910	and begged us not to have that again.
1347570	1349270	And we were just severely limited
1349270	1350830	by what we could do in a browser.
1351410	1354130	For non-hash formats at the time,
1354130	1356390	we were scraping everything server-side.
1356390	1358170	So you would actually upload a PDF
1358170	1359610	or upload a .docx
1359610	1361870	or whatever,
1361870	1365590	and we would scrape the file server-side
1365590	1369190	to get the necessary bits,
1369190	1371630	like Office to hashcat
1371630	1372870	and PDF to hashcat,
1372870	1373910	shit like that.
1374530	1376190	The problem then became,
1376190	1378310	what do we do with things like
1378310	1379510	TrueCrypt volumes
1379510	1382990	or really large 7-zip volumes
1382990	1384690	or RAR files?
1384690	1386110	Are you really going to upload
1386750	1389410	a 100-gigabyte disk image
1389410	1390370	through the browser
1390370	1392170	just so we can scrape off
1392850	1394390	two kilobytes?
1394390	1397010	No, that doesn't make any sense.
1399690	1401710	So yeah, the browser was an issue.
1401710	1404450	And then we implemented access controls,
1404450	1405970	but there was nothing fine-grained.
1405970	1408770	We assumed that because everyone
1408770	1410790	operating the hashcat cluster
1410790	1412210	was on the same team,
1412990	1415710	that they all had equal rights
1415710	1417190	and equal access to the cluster,
1417190	1418770	and therefore all users are admins.
1418770	1422010	We didn't have different roles or anything.
1422010	1424350	It was just, if you have an account,
1424350	1426650	then you obviously have a right to be there,
1426650	1428410	so therefore you are an admin.
1430970	1432590	And I think some of our clients
1432590	1433390	didn't have a problem with that,
1433390	1434970	because everyone on this team
1434970	1437290	does have equal rights and access
1437290	1438510	and privilege to the cluster,
1438510	1439730	so this is totally fine.
1439730	1441110	Other clients were like,
1441110	1442390	this is absolutely unacceptable.
1444810	1446210	And I eventually got to the point
1446210	1449010	where the old version one code
1449010	1450970	was completely unmanageable.
1451370	1452550	Even just implementing
1452550	1454350	what should have been a minor bug fix
1454350	1458210	ended up requiring major refactoring.
1459890	1460990	It just wasn't,
1460990	1461850	because we threw it together
1461850	1462870	in less than two months,
1462870	1465450	so we didn't think of everything in advance
1465450	1467130	that would come up,
1469730	1470770	and try to implement new features,
1470770	1472190	and try to work around
1472190	1475550	some of the bugs in there.
1475550	1476550	So we decided we needed
1476550	1477970	an entire ground-up rewipe
1477970	1479910	for Hashtag version two.
1481630	1483190	And at some point,
1483190	1483970	I got the idea
1484950	1487330	to create a commercial fork of Hashcat
1487330	1489650	with native distributed capabilities.
1489850	1490670	And at the time,
1490670	1492690	I was so fucking stoked on this idea.
1492690	1494610	It seemed like the best idea
1495170	1497090	in the world to me at the time.
1498050	1501330	So we set out to do just that.
1502150	1503730	Now this is right
1504530	1505210	before
1505890	1507810	Hashtag went open source,
1507810	1509930	and then we started work on this
1509930	1511510	literally the minute
1511510	1513890	that Hashtag went open source.
1514450	1516210	So we implemented in Golang
1516210	1517030	in C
1518010	1520290	with Postgres database.
1520510	1521730	Now we chose Golang
1521730	1523290	because of high concurrency
1523950	1526910	with goroutines, right?
1527090	1528970	So with Node.js,
1528970	1531150	everything ran on a single CPU thread.
1531150	1531930	And we wanted
1531930	1534490	the opposite of that for version two, right?
1534490	1535990	We're going to learn the lesson on that.
1535990	1537810	So we wanted
1538290	1539770	just lots and lots of goroutines
1539770	1540610	to run everything
1540610	1543450	so that we'd use all 24 threads
1543450	1544910	plus on the box.
1546870	1548510	And Golang actually also
1549490	1551130	allows for pretty rapid development.
1551130	1553010	So we thought it was a pretty good choice.
1554850	1555930	But we still stuck
1555930	1557370	with the traditional client server
1557370	1558690	agent architecture.
1558730	1561570	We have numerous clients
1561570	1563150	and then one server
1563150	1565250	and then however many agents.
1566350	1567570	Meaning that that server
1567570	1569230	is both a bottleneck
1569230	1570630	and a single point of failure.
1571910	1573550	So the first thing we had to do
1573550	1575430	is we had to split Hashcat in two.
1575990	1576930	We took everything
1576930	1578230	that had to do with
1578230	1581290	actually starting a cracking job
1581290	1583910	and executing an OpenCL kernel
1583910	1584550	on a GPU
1585930	1586150	with the agent
1586150	1587610	and nothing else.
1587610	1589830	The agent was just barebones
1589830	1591030	trimmed down Hashcat
1592830	1594210	launch attack on GPU
1594210	1595990	and that was it, nothing more.
1596410	1597490	And everything else
1597490	1599750	we placed into the server library.
1600290	1602350	And then for any code
1602350	1603910	that wasn't already in Hashcat
1603910	1605770	such as the JSON API
1605770	1607510	and the
1608510	1609110	protobufs
1609110	1610630	for agent server communication
1610630	1611810	workload distribution
1615930	1616730	fine-grained
1616730	1619390	multi-role access control.
1619470	1620850	All that was implemented in Go
1620850	1623650	and then we just linked against the C libraries.
1623650	1625270	And this effort actually
1625270	1627150	laid the foundation for
1628130	1630410	the Hashcat 4.0 refactor
1630410	1632150	with libHashcat.
1632610	1634910	And then because of how clunky
1634910	1636510	our old web UI was
1636510	1639130	we actually ditched it entirely
1639130	1640890	for a cross-platform CLI
1640890	1643410	which made our power users
1643410	1644590	like really happy
1645930	1648290	but our less experienced users
1648290	1650330	were not very thrilled
1650330	1651390	with that decision.
1651390	1654310	They still wanted a GUI
1654310	1656830	which, fair enough.
1658090	1660090	So our plan initially
1660090	1662350	was to maintain both Hashcat
1662350	1664590	and Hashstack trees in parallel.
1664830	1665330	And then
1666570	1667930	if there was any features
1667930	1669230	that we added to Hashstack
1669230	1670890	or any new kernels that we wrote
1670890	1672850	we would just backport those
1672850	1675510	as we chose to Hashcat.
1678550	1679770	This didn't really work out
1679770	1680790	very well in practice though.
1680790	1682430	So when we released version 2
1682430	1684750	it was in sync with Hashcat 3.5
1685230	1686370	but it didn't stay that way
1686370	1687370	for very long.
1688410	1689510	And of course we had to add
1689610	1690530	a cross-platform GUI
1690530	1691290	to the roadmap as well
1691290	1692790	because we had significant
1692790	1693870	feedback from our clients
1693870	1695390	that they really wanted
1695390	1696290	the goddamn GUI
1696290	1696990	and they were not happy
1696990	1698370	that we took away the web UI.
1699890	1700770	It turns out
1700770	1702130	this was the single dumbest
1702130	1703350	motherfucking thing I've ever
1703350	1704450	done in my life.
1705590	1707390	I completely underestimated
1707390	1708450	how much work was involved
1708450	1710410	in creating a commercial fork
1710410	1711370	of Hashcat.
1712010	1713230	Like, genuinely
1716470	1716970	I...
1717770	1719510	Yeah, I really don't know
1719510	1720910	how to convey how fucking stupid
1720910	1722310	of an idea this was.
1723110	1725570	We spent like 100%
1725570	1726650	of our time just backporting
1727210	1728930	things from Hashcat into Hashtag
1728930	1730110	instead of actually adding new
1730110	1731650	features or writing new kernels.
1732290	1734030	Like, literally months of work
1734550	1735610	you know, just backporting
1736930	1738130	Hashcat commits.
1738870	1740690	And it got really frustrating.
1740910	1741490	You know, it's like,
1741490	1742650	hey, are we working on these
1742650	1743690	tickets? Are we working on these
1743690	1744930	features? No, we're working on
1744930	1745770	backports.
1745990	1746950	You know, I got so sick of
1746950	1747930	hearing working on backports,
1747930	1749330	working on backports, like
1750290	1751230	when are we ever going to be
1751230	1752230	done with backports?
1753430	1755590	And then when Adam refactored
1755590	1757950	Hashcat for 4.0, that also
1757950	1759390	meant that we had to refactor
1759390	1760990	Hashtag as well.
1761150	1762550	And we hadn't even finished
1762550	1763750	implementing all the backports
1763750	1765410	at the speed that Adam was, you
1765410	1766250	know, committing things to
1766250	1767050	Hashcat.
1767170	1768710	So this was just like,
1768710	1769250	absolutely
1770510	1772130	unmaintainable, right?
1772130	1774250	Like, fuck me.
1775550	1776870	And then there was also this
1776870	1778010	other little minor problem
1778950	1780610	where we picked the wrong
1780610	1781570	HTTP library
1782230	1783450	for the API.
1784130	1785310	And this basically
1785310	1786750	limited the service capability
1786750	1789770	to about 12 nodes.
1789790	1791290	Now, 12 nodes
1791290	1792830	might seem like a lot.
1792830	1794510	That's 96 GPUs.
1794510	1795270	You know, that's a pretty
1795270	1797670	respectable size cluster.
1798890	1801310	The problem is this
1801310	1803050	particular client had
1803050	1805950	ordered 320 GPUs,
1805950	1807230	40 nodes.
1807510	1809190	And you can't really
1809190	1811250	deliver a 40 node client
1811250	1814290	to a cluster with a
1814290	1815970	server that can only
1815970	1817230	really handle
1817230	1822360	12 nodes.
1822360	1823300	So this required
1824680	1826280	not a complete rewrite,
1826440	1827840	but a damn near complete
1827840	1828320	rewrite time.
1828320	1829500	Fuck, because we use that
1829500	1830300	HTTP library
1831780	1832620	extensively throughout the
1832620	1834400	code.
1834400	1835440	And there was no, like, drop
1835440	1836140	in replacement.
1836140	1837200	So basically anywhere where
1837200	1838800	we were dealing with
1838800	1839880	HTTP,
1839880	1841220	we had to rip out the old
1841220	1842800	library and then,
1842800	1843200	you know,
1844060	1844960	add in all the stuff for
1844960	1845840	the new library.
1845960	1847000	But they did resolve the
1847000	1847540	issue.
1848320	1849500	It was just really painful
1849500	1850540	at the time to have to
1850540	1851400	mess with that.
1853420	1854600	But the server
1854600	1856740	still continued to be a
1856740	1857780	single point of failure
1857780	1858820	and still proved to be a
1858820	1860180	source of bottlenecks.
1860180	1861540	As we, you know,
1861540	1862150	tried to build
1863220	1864360	larger and larger
1865160	1866000	clusters,
1866580	1867000	we
1867000	1867640	kept running into
1867640	1869160	problems with the
1869640	1870060	server
1870200	1870660	being a bottleneck
1870760	1871380	and a single point of
1871380	1872340	failure.
1872340	1873080	And we also had clients
1873080	1874200	coming to us asking us
1874640	1875520	if they could order
1875520	1876400	multiple cluster
1876400	1877780	controllers and cluster
1877780	1878800	them together,
1878800	1879240	you know, either
1879240	1879960	high availability
1879960	1881860	or, like,
1881860	1882700	load balance them
1882700	1883540	or something.
1884560	1884980	And
1886680	1887660	we didn't have a
1887660	1888380	solution for that,
1888380	1889140	but I started thinking
1889280	1892300	about a solution for that.
1892300	1893220	And again, like,
1893220	1893980	40% of our clients
1893980	1894660	were just completely
1894660	1895620	pissed that we had no
1895620	1900280	GUI.
1900280	1900480	So
1900480	1901340	it got to the point
1901340	1902020	where it's like,
1901080	1902080	let's just
1902540	1904080	rethink, like,
1904080	1905000	the entire fucking thing
1905000	1906140	and just, like,
1906140	1907000	go as far away from
1907000	1908080	these mistakes as we
1908080	1909160	could possibly get.
1909220	1910080	Like, we never, ever
1910080	1910780	want to make any of
1910780	1911860	these mistakes ever again.
1911860	1912420	We've learned a lot of
1912420	1913960	lessons from, you know,
1913960	1914720	what we've done over
1914720	1915920	the last seven years.
1917340	1918640	So, but we need
1918640	1919360	more than just a
1919360	1920120	complete ground-up rewrite.
1920120	1921220	Like, we need an entire
1921220	1921640	paradigm shift.
1921640	1923100	Like, we need to
1923100	1924020	rethink this entire
1925480	1925880	problem
1926640	1927040	entirely
1927600	1928000	and
1928000	1928620	come up with
1928620	1929300	something that's
1929300	1929980	completely different
1929980	1930460	from what's ever been
1930460	1931120	done.
1931940	1932820	So we need to
1932820	1933740	eliminate the single
1933740	1934400	point of failure of
1934400	1935180	the server.
1935300	1936640	We need to enable,
1936640	1937600	like, actual infinite
1937600	1938440	horizontal scaling
1938440	1939440	because, again,
1939440	1940480	we're, you know,
1940480	1941740	we always try to
1941740	1943440	strive for, like,
1943440	1944460	warehouse as a
1944460	1945360	computer, right?
1945360	1946900	We're looking at
1946900	1947560	warehouse scale
1947560	1948300	computing as, like,
1948300	1948880	the target that we're
1948880	1951360	trying to hit.
1951360	1952660	There should never be
1952660	1954860	any, you know,
1954860	1955560	limiting factors
1955560	1956540	outside of budget
1956540	1958280	for, like, how many
1958280	1958860	nodes we can
1958860	1959280	support in
1959280	1959560	HashtagWrite.
1960220	1961160	And then, of course,
1961160	1962040	the other big requirement
1962040	1962700	is we have to actually
1962700	1963560	move at the pace of
1963560	1964740	Hashtag development.
1966040	1967620	We have to,
1967620	1968380	you know, as Adam
1968380	1969500	commits things to
1969500	1970420	Hashtag, we need to
1970420	1971040	have those
1971960	1973280	not instantly available
1973280	1974440	in Hashtag, but within
1974600	1975280	a very reasonable
1975280	1976260	time frame.
1978880	1979320	So
1980000	1981000	the very first
1981000	1981760	idea that we came up
1981760	1982620	with was let's not
1982620	1983540	do a traditional
1984540	1986000	client-server-agent
1986740	1987460	model.
1988440	1989540	And let's not even
1989540	1990340	try to cluster the
1990340	1991400	cluster controllers.
1991740	1992780	Let's just eliminate
1992780	1993620	the cluster controllers
1993620	1994800	entirely and make
1994800	1995580	everything peer-to-peer
1995580	1996740	where all nodes
1996740	1997160	are equal
1997860	1998620	and there are no
1998620	1999860	dedicated servers.
2000860	2001100	And
2001920	2002960	the biggest requirement
2002960	2003860	to this is that
2003860	2004660	work can be submitted
2004660	2005760	to any node.
2006460	2007540	So let's say you have,
2007540	2008620	like, you know, 20 users
2008620	2009780	and you have,
2009780	2011120	you know,
2011120	2012000	10 boxes.
2012000	2012780	Like, there shouldn't
2012780	2013720	be just one node that
2013720	2014900	all of them have to hit.
2014900	2015700	They should be able
2015700	2016420	to hit any node
2016420	2017160	in the cluster
2017160	2018400	and, you know, have
2018400	2019160	the same view
2019160	2020600	from any node.
2021880	2022780	And it also needs to
2022780	2023400	be more user-friendly
2023400	2024140	than ever.
2024400	2025400	And also needs to
2025400	2026340	run on Windows.
2028100	2028960	I'll tell you why
2028960	2029640	in a minute.
2032240	2033380	So set out to,
2033380	2034120	like, figure out
2034120	2034660	how we're going to
2034660	2035320	implement this.
2035320	2035820	Because we have the
2035820	2037280	high-level requirements,
2037280	2038040	right? Like, we know
2038040	2038760	it needs to be
2038760	2039380	peer-to-peer.
2039380	2040140	We know all the nodes
2040140	2041060	need to be equal.
2041240	2042520	You know, all that shit.
2043040	2043800	But how are we actually
2043800	2044740	going to do this?
2045600	2045960	So
2046840	2047560	pretty obviously
2047560	2048060	we're going to need
2048060	2048540	some sort of
2048540	2049880	distributed state machine.
2050040	2050680	Right?
2051460	2052400	And that's just
2052400	2053280	straightforward
2053280	2054520	and self-explanatory.
2054780	2056360	But, you know,
2056360	2057180	like, we just
2057180	2057600	throw a message
2057600	2058040	queue at it
2058040	2058520	and some PubSub
2058520	2060400	and, you know,
2060400	2061500	Bob's your uncle.
2062160	2062860	But what about
2062860	2063860	disagreements?
2064060	2065460	Or, you know,
2065460	2066060	maybe even more
2066060	2066760	than disagreements,
2066760	2067360	what about, like,
2067360	2068680	an actual, you know,
2068680	2069060	rogue
2069900	2071100	cluster member?
2071220	2073100	Or, you know,
2073980	2074380	a broken
2074820	2075520	cluster node
2075520	2076560	that's maybe not
2076560	2076980	necessarily
2076980	2077340	intentionally
2077340	2078180	malicious,
2078180	2079000	but it certainly,
2079000	2079300	you know,
2079300	2081180	has, you know,
2081180	2081820	the appearance of
2081820	2082700	malicious behavior.
2082700	2083500	Right?
2084620	2085600	So I started
2085600	2086320	coming up with
2086940	2087560	solutions for
2087560	2088060	this.
2088680	2089280	And
2089980	2090960	I somehow
2090960	2091740	got into my
2091740	2093220	head, like,
2093220	2094200	dude,
2094200	2094860	what if
2094860	2095600	computers could
2095600	2096580	vote?
2097000	2097840	Right?
2097840	2098980	Like,
2101520	2102360	you just
2102360	2102840	make it to
2102840	2103300	where, like,
2103300	2103580	you know,
2103580	2104420	if there's a
2104420	2105260	dispute,
2106100	2106440	then one of
2106440	2106920	the nodes,
2106920	2107520	like,
2107520	2108140	requests a vote,
2108140	2108740	right?
2109060	2110160	And so they take a vote
2110160	2111060	and majority wins.
2111060	2111820	And if there's a tie,
2111820	2112360	like we have an even
2112360	2113200	number of nodes in the
2113200	2114600	cluster, then,
2114600	2114700	like,
2114700	2115420	literally fucking
2115420	2116640	flip a coin,
2116640	2117140	you know,
2117140	2117640	randomly,
2117640	2118500	or play rock,
2118500	2118720	paper,
2118720	2119220	scissors.
2119360	2119980	Like,
2119980	2120780	why not?
2122100	2122960	And if a peer
2122960	2123660	doesn't accept the
2123660	2124440	results, like,
2124440	2124880	then the rest of
2124880	2125400	the cluster just
2125400	2126380	shuns that peer.
2126800	2127400	And I started
2127400	2127820	thinking about,
2127820	2127940	like,
2127940	2128920	is this actually,
2128920	2129040	like,
2129100	2129380	a thing?
2129380	2129500	Like,
2129500	2130080	can we actually do
2130080	2130380	this?
2130380	2130860	Am I,
2130860	2131160	like,
2131160	2131760	completely fucking
2131760	2132400	stoned?
2134600	2135420	It turns out that
2135420	2135780	I really didn't have
2135780	2136160	to put that much
2136160	2136640	more thought into
2136640	2137340	it because this
2137340	2138120	already exists.
2138120	2138320	Like,
2138320	2139180	what I thought up
2139660	2140820	is actually a
2140820	2141700	thing called the
2141700	2142480	wrapped consensus
2142480	2143940	protocol.
2143940	2145040	Now,
2145040	2145920	it's cool that it
2145920	2146420	already exists because
2146420	2146880	that,
2146880	2147220	you know,
2147220	2147460	means we don't
2147460	2147700	have to do a
2147700	2148080	whole lot to
2148080	2148520	implement it.
2148520	2149140	But it's also cool
2149140	2149980	because it kind of
2150480	2151200	validated,
2151200	2151940	you know,
2151940	2152860	my ideas and theories
2152860	2153560	on how to make this
2153560	2154260	work.
2155080	2155980	And told me that I
2155980	2156260	wasn't just
2156260	2156780	completely fucking
2156780	2157400	insane.
2158380	2159600	And we found
2159600	2160800	several implementations
2160800	2161420	of the wrapped
2161420	2162460	consensus protocol.
2163640	2164860	And the one that
2164860	2165900	we ended up liking
2165900	2166540	the most and
2166540	2167500	settled on is
2167500	2168100	Akka,
2168100	2168960	which runs on
2168960	2169620	the JVM.
2169620	2169980	It's written in
2169980	2170660	Scala.
2171140	2171920	Now,
2172580	2174180	I hate Java.
2174180	2175020	I have shunned
2175020	2176980	Java for damn
2176980	2177640	near my entire
2177640	2178500	life.
2178740	2179100	You know,
2179100	2180300	I completely
2180300	2181040	condemn
2182280	2183700	Oracle JVM.
2186660	2187700	But,
2187700	2188860	there is
2188860	2189900	OpenJDK.
2189900	2190420	We don't have
2190420	2191360	to use Oracle.
2191580	2193020	And we don't
2193020	2193400	have to write
2193400	2194160	in Java.
2194160	2194620	We can use
2194620	2195060	Scala and
2195060	2195860	Kotlin.
2196780	2197420	And since
2197420	2197840	I've started
2197840	2198200	to work on
2198200	2198480	this,
2198480	2199200	I actually
2199200	2200020	really like
2200020	2200920	Kotlin.
2201780	2202600	I don't
2202600	2203500	really have too
2203500	2203940	many negative
2203940	2204480	things to say
2204480	2204860	about it.
2204860	2205160	It's actually
2205160	2206040	kind of fun
2206040	2207180	to work with.
2208420	2208720	And then
2208720	2209620	Akka also
2209620	2210420	has some
2211400	2212220	plugins or
2212220	2214000	modules that,
2214780	2215760	you know,
2215760	2216600	help us even
2216600	2217160	further.
2217160	2217620	Such as
2217620	2217880	Akka
2217880	2218880	persistence.
2218880	2219260	What Akka
2219260	2220040	persistence does,
2220040	2220440	it takes our
2220440	2221080	distributed state
2221080	2222100	machine,
2222100	2223000	which is
2223000	2223200	normally
2223200	2223700	resident in
2223700	2224020	memory,
2224020	2224480	and persists
2224480	2225320	it to
2225320	2226120	disk.
2226600	2227040	And then
2227040	2228020	Akka
2228020	2228380	distributed
2228380	2228860	data, of
2228860	2229180	course, would
2229180	2229660	be like what
2229660	2230500	implements the
2230500	2231220	distributed state
2231220	2231960	machine.
2232420	2233400	And then on
2233400	2233880	top of all
2233880	2234840	of that,
2234840	2236120	we built a
2236120	2236700	custom
2236700	2237900	multicast-based
2237900	2239200	protocol to
2239200	2240040	enable cluster
2240040	2241000	nodes to
2241000	2241240	automatically
2241240	2241540	discover each
2241540	2241940	other on
2241940	2242700	the network
2242700	2243360	and then
2243360	2243480	automatically
2243480	2244220	join the
2245020	2245560	cluster.
2245560	2245980	Now, we
2245980	2246820	made this opt-in
2246820	2247080	because obviously
2247080	2247620	that's kind of
2256020	2257020	a security
2247620	2248120	risk if
2248120	2248800	anybody can
2248800	2249420	just join the
2249420	2250640	cluster at
2250640	2251400	will.
2253880	2254920	But for
2254920	2255780	some scenarios
2255780	2256320	like where
2256320	2256820	all of your
2256820	2257300	nodes are on
2257400	2257800	a dedicated
2258660	2259380	subnet that's
2259380	2260140	properly firewalled
2260140	2260600	off and you
2260600	2261120	have proper
2261120	2261780	layer 3 and
2261780	2262420	layer 4 access
2262420	2264440	controls,
2265760	2267060	this would be a
2267060	2267420	solution that would
2267420	2268540	be really handy.
2268540	2269720	So, again,
2269720	2270700	thinking about
2270700	2271440	warehouse-scale
2271440	2272620	computing or
2273200	2274080	data center as
2274080	2275280	a computer,
2275280	2275840	let's say you
2275840	2276160	just have racks
2276160	2276720	and racks and
2276720	2277460	racks of compute
2277460	2278180	nodes, you would
2278180	2280120	just unbox one,
2280120	2280740	literally throw it
2280740	2281180	in the rack and
2281180	2282560	power it on, and
2282560	2283220	it would discover
2283220	2284740	the cluster as
2284740	2285040	soon as it's
2285040	2285680	powered on and
2285680	2286220	join the cluster
2286220	2286840	and start doing
2286840	2287500	work.
2287660	2288380	That's kind of
2288380	2288860	the vision that we
2288860	2289840	have for this.
2291420	2292060	And, of course,
2292060	2292420	since it's all
2292420	2293580	peer-to-peer,
2293580	2295120	that enables it
2295120	2295900	to actually be
2295900	2296400	just literally
2296400	2297280	that simple.
2299720	2300580	But, of course,
2300580	2301760	the default is
2301760	2302260	you manually
2302260	2302980	specify which
2302980	2303800	peers you want
2303800	2304220	to be part of
2304220	2305040	the swarm.
2309110	2309410	Okay.
2309410	2311070	And then, in
2311070	2312050	April this year,
2312050	2312470	we had the
2312470	2313110	unique opportunity
2313110	2313410	to present
2313410	2314170	itself to
2314170	2314710	purchase
2314710	2315950	loft crack.
2317210	2318090	Dildog approached
2318090	2318610	me and we
2318610	2319430	entered negotiations
2319430	2321610	and we
2323050	2323630	purchased it
2323630	2323990	earlier this
2323990	2324610	year primarily
2326290	2327370	for the
2327370	2329310	GUI, which
2329310	2330330	is written
2330330	2330890	in C++
2330890	2332710	using QT
2332710	2333350	as the
2333350	2333690	windowing
2333690	2334570	framework.
2334810	2335170	And while
2335170	2335510	it's not
2335510	2335830	currently
2335830	2336670	cross-platform,
2336670	2337070	that's a
2337070	2337870	really easy
2337870	2339250	platform, or
2339370	2340470	that's a really
2340470	2340910	easy language
2340910	2341310	and library
2341310	2341910	to make
2341910	2348130	cross-platform.
2348130	2349310	So, instead
2349310	2349810	of
2349810	2350070	hashtag
2350070	2351230	version 3,
2351230	2351750	we now
2351750	2352590	have loft
2353970	2354410	crack version
2354410	2354930	8.
2354930	2356710	And while
2356710	2357110	loft crack
2357110	2357250	7 uses
2357250	2357810	John the
2357810	2358810	Ripper as
2358810	2359430	a backend,
2359430	2360470	we've ripped
2360470	2360690	all that out
2360690	2361630	and we are
2361630	2362790	in the process
2362830	2363830	of integrating
2363830	2364830	what was
2362790	2363390	hashtag version
2363390	2364770	3 as the
2364770	2365450	new backend
2365450	2366130	for loft crack
2366130	2366710	8.
2368670	2369710	And the
2369710	2371010	final thing
2371010	2371330	we're doing
2371330	2372170	with this is
2372170	2372870	we are dropping
2372870	2373590	the
2373590	2374070	TeraHash
2374070	2374330	hardware
2374330	2375310	requirement.
2375490	2376710	Traditionally,
2376710	2377550	you know, we
2377550	2379650	do not let
2379650	2380810	people purchase
2380810	2382270	hashtag as a
2382270	2382950	standalone product.
2382950	2383550	We say, like,
2383550	2384070	you know, the only
2384070	2384750	way to get
2384750	2385650	hashtag is to
2386010	2386410	buy a
2386410	2388670	TeraHash
2388670	2389310	appliance and
2389790	2390790	it comes
2390790	2391790	preinstalled
2389310	2389690	and that's the
2389690	2390050	only way to
2390050	2390870	get it.
2391430	2392230	But loft crack
2392230	2392810	has always been
2392810	2393890	sold standalone.
2394350	2396030	So we are changing
2396030	2396730	the model now
2397310	2398250	to where,
2399150	2399730	you know,
2399730	2400170	keeping with
2400170	2400570	loft crack
2400570	2401030	being sold
2401030	2401490	standalone,
2401490	2401710	we're going to
2401710	2402290	be selling
2403710	2404330	loft crack
2404330	2405070	8 as a
2405070	2405970	standalone product
2405970	2406590	without the
2406590	2410470	TeraHash
2410470	2410530	hardware
2410530	2410770	requirement.
2410770	2412010	So, yeah,
2412010	2413150	that's where
2413630	2414470	we are now
2414470	2415150	with development.
2415150	2415290	That's how the
2417310	2418310	product has
2415290	2415750	evolved over
2415750	2416270	time.
2416270	2416870	And that's some
2416870	2417410	of the
2417410	2418750	challenges that
2418750	2419490	we have faced
2419490	2421670	trying to
2422830	2423690	obtain our
2423690	2424390	goal of
2424390	2424890	true
2424890	2425750	warehouse scale
2425750	2426730	computing.
2426830	2427730	Thank you and
2427730	2428270	hope you've enjoyed
2428270	2429010	this.