This is Jeremy Gosney. I'm coming to you live from Password Village HQ2 in sunny Austin, Texas.
Today's high is 104 degrees.
So today I'm going to be talking to you about what it's like cracking at Terrahash's scale.
I'm not going to make this a vendor pitch in any way.
I'm going to focus on the technical challenges that we encounter as we support clients
who have hundreds of GPUs in their clusters.
When I first started on this, I thought 100 GPUs was a really big deal.
Now we tend to view 100 GPUs as like, meh.
It's good, but it's kind of middle of the road.
There's a lot of distributed solutions for HashCat out there.
But as far as I know, Terrahash is the only one that aims to operate at warehouse scale.
What I mean by that is, more so with the most recent version, but with every version that passes,
we aim to embrace the concepts of data center as a computer.
While we currently don't have any clients who have an entire warehouse full of password cracking hardware,
that's the point where we're trying to go.
We try to enable the software to be at that level, even if our clients are not quite at that level.
I'm going to go through the history of how I got started in massive distributed cracking.
And I'm going to go through the technical challenges that we faced as we continued to evolve our hashtag distributed solution.
I'm going to start the slide deck.
All right.
So, in the beginning, I kind of got into this because I was messing around with distributed computing.
I was doing stuff with MPI, some other MOSIX software, rocks clusters, things along those natures.
But all that was all CPU-based.
And then when GPU cracking came about, we started looking for solutions that would enable us to distribute GPU workloads.
So, I found a piece of software at a Hebrew university called VirtualCL.
And I did a presentation on a 25 GPU proof of concept cluster that me and Bitweasel put together at passwords12 in Oslo.
And when I woke up the next morning after giving that presentation, we had gone viral.
On the front page of Slashdot, Gizmodo, Boing Boing, NBC News, The Register, we were everywhere.
And in hindsight, I really should have used a lot more than 25 GPUs.
I mean, we had more GPUs, that's just what we chose to dedicate to this proof of concept.
And had I known in advance in any way that the general public would react how they did to this proof of concept,
I would have used like three times the number of GPUs.
So just, you know, some of the headlines that came out the next morning that I woke up to,
which was pretty surreal because I didn't really see this as a big deal.
So, VirtualCL was cool at the time because it was the first solution that I knew of that enabled us to
sort of transparently distribute OpenCL jobs across the arbitrary cluster.
So what VirtualCL does is it provides a virtual OpenCL platform.
And the target software, in theory, doesn't need to be aware it's communicating with a virtual platform.
It just sees any, you know, it sees VirtualCL as any other, you know, OpenCL platform,
just like AMD, NVIDIA, Intel, what have you.
So the target software just simply links against libOpenCL, which is provided by VCL,
which then communicates to a broker daemon, and then that broker daemon distributes to the agent daemons,
and the agent daemons, which are installed on the compute nodes,
actually communicate with the real OpenCL library, be it AMD, NVIDIA, whatever.
So in theory, all this is transparent to the end application.
But in practice, it didn't quite work that way.
We had to have a special fork of OCLHashCat called VCLHashCat,
which had some workarounds for VCL quirks and such.
It also required InfiniBand, because it was really latency sensitive.
It ended up being a very chatty protocol, and it required essentially real-time communication.
Bandwidth wasn't so much of an issue.
It pretty much consistently pulled, you know, less than one gig of bandwidth,
but it was really sensitive to latency.
Even just a couple milliseconds was acceptable,
but as you start getting up into double digits of latency,
you had a noticeable impact on hash rate.
And of course, InfiniBand hardware is expensive.
It's way more expensive than Ethernet.
VCL was also closed source, and it required frequent updates.
So this is back in the time, this is 2012 through 2013,
where every GPU driver that was released broke some shit.
And then we had to go back in and implement workarounds and fixes,
whatever the fuck AMD or NVIDIA broke in their driver that day.
And VCL was no exception.
So with VCLHashCat, we had to have our own workarounds for VirtualCL,
and then the VCL team, or VirtualCL team,
had to turn around and implement any workarounds for any new driver releases
every time they were released.
Which is a pretty substantial job.
The problem is, VirtualCL was created by grad school students.
This was a grad school project, and they created this their senior year.
And when they graduated in spring 2013, that was it.
The project died.
So when the next version of FGLRX was released,
I think that was version 13.9 in September of 2013,
we go to install a VirtualCL cluster for our client.
This is actually a 64 GPU cluster.
We were just starting out, so this was actually a pretty big deal for us.
64 GPUs was pretty fucking cool at the time.
But we get the OS installed, we get the hardware built, all that shit,
and then we go to install VirtualCL and FGLRX,
and VirtualCL just shits the bed.
So I panic and email Hebrew University,
and I'm like, you guys really need to update VirtualCL
for Catalyst and FGLRX 13.9.
And they're like, well, we can't.
Everyone who was working on this project has gone.
And the intellectual property remains with Hebrew University and MOSIX,
so they can just have those people work on it outside of that.
I offered to take over the project for them,
because it was kind of essential to what we were doing,
and I didn't want to see the project die.
I thought it was really neat.
But the professor over there who was in charge of the project
was not amenable to that idea.
He actually had pretty grave concerns
when we first started using VirtualCL for password cracking,
because he envisioned us turning the world into a giant botnet.
And of course, that wasn't really on our agenda,
at least not at that time.
But yeah, he really didn't like the idea
of his VirtualCL shit ending up in our hands.
So we were left high and dry without a software solution,
and so we desperately needed something.
So we conceived the idea of Hashtag.
Initially, we thought about making our own VirtualCL clone,
since Hebrew University was not going to give us the VirtualCL source code.
And we're like, well, fuck it.
We'll start our own VirtualCL with hookers and blackjack.
But the more we actually sat down and thought about what that entailed,
we determined the level of effort for that was just way too high.
And the timeframe in which we needed it was way too short.
We literally had hardware in-house that we're building for clients
who now have no software solution because of VirtualCL.
So we decided to make a distributed wrapper.
And that's when we came up with the idea of Hashtag.
Like I said, we had to iterate really quickly on this.
So we went from a whiteboard session to production in less than two months.
And this was basically me and Tom Steele locking ourselves in the office
and ordering over $100 for the Taco Bell
and just banging this out as fast and furious as we could.
It had a traditional client-server-agent architecture.
And it was Hashtag focused,
but it actually had a generic plugin interface.
We had plugins for... and this was before there was just one Hashtag.
So we had Hashtag, OCL Hashtag Plus, OCL Hashtag Lite plugins.
And then we had John the Ripper plugin.
And then we had just a generic interface
to where any cracking tool could be made to work with Hashtag
as long as it adhered to a standard format that we had defined.
And then it also had the ability to run arbitrary commands when idle.
This was less for our clients and actually more for me
because we were running Hashtag in-house as well
for our cracking-as-a-service type stuff.
And this is back in a time when GPU mining was still profitable.
So when we weren't working on a password cracking job,
I wanted this thing to be generating coins in the background.
So the arbitrary idle exec command was literally just put in place
so that people could mine Litecoin or Namecoin or whatever
when their cluster was idle.
So, you know, totally different time.
But, I mean, it actually worked.
It did exactly what it was supposed to do and it was good.
And the whole goddamn thing was implemented
in this fancy new language called Node.js.
Again, this is early 2013, so Node.js was the new hotness
and everyone was starting to move that direction
with microservices and everything.
It's not on this slide, but we implemented Hashtag version 1
entirely as microservices.
And I shit you not, there was like 23 individual packages
that comprised Hashtag and almost all of those were microservices.
And we went that direction because that's kind of like
programming zeitgeist at the time, right?
Like everyone was pushing microservices and pushing asynchronous
and, you know, NoSQL and all that other horseshit, right?
And in the end, it created more headaches than not
if we just had a monolithic fucking program.
You know, maintaining like one server binary
and one agent binary is way simpler than like, you know,
maintaining 23 individual microservices and making sure
that they're all synced and communicating properly
and, you know, that the versions are correct, you know,
across all 23 packages.
And there's just a lot of unnecessary headache
that was introduced with that, that, you know,
we attempted to manage.
But even then, like our clients would still kind of mess it up.
You know, like they would upgrade like 10 of 20 packages
and then like the other ones would just be stuck at old versions
or, you know, one demon would die and another demon would know
that it's dead, so then it would shit the bed.
And anyway, so yeah, no JS, microservices,
no SQL, MongoDB, all that shit.
I tried to use all like the latest and greatest everything
because at the time, that seemed like the best idea.
And also, Hashtag was more than just workload distribution.
So a lot of our clients at the time thought
that Hashtag was just a GUI front-end for Hashcat.
And we're like, no, it's not.
It's not just a web UI for Hashcat.
It actually does like, you know, multi-user and, you know,
workload distribution with like a really complex
four-dimensional queuing mechanism.
Like you're trying to explain all this to the client
and they're just like, you know, eyes glazed over
and like, so it's a web UI for Hashcat.
Like fucking yes, okay, it's a web UI for Hashcat.
But it actually did a hell of a lot more than that.
In fact, when you installed Hashtag on your box,
it basically hijacked it and configured it
exactly the way that we needed it to be configured.
It would actually provision the entire server
from just installing like the initial meta package.
So it's a complete stack of packages.
They get stack, Hashtag, that's where it comes from.
So it handled driver installation,
driver configuration, driver updates.
And again, this is back at a time,
and even nowadays still kind of a challenge,
but this is back at a time when GPU drivers
were a pain in the ass to install.
And then once you got them installed,
it was a pain in the ass to update.
So the fact that we kind of handled this
automagically was a pretty big value add.
Also handled XOR configuration,
headless servers.
This is actually during a time
where when you had a headless server,
you had to have fucking dummy plugs
connected to each of the GPUs.
There's probably very few people watching this
who actually remember that.
But yeah, if you had a completely headless server,
each one of those GPUs had to have
a special dummy plug attached to it
that would connect the necessary pins
to trick it to think that there's actually
a monitor attached.
And we actually had to run XOR
because even though these are headless,
they don't have GPU clock rates
or set like power tune profiles
or any of that.
So even though this is a headless server,
we still had to have dummy plugs
to think there's a monitor attached,
still had to run XOR,
and still had to configure XOR properly
to enable us to control the fans
and control the clocks
and all the other shit that we needed to do.
And then along those lines,
Hashtag also did GPU tuning for you
and also automatic overclocking.
And this was kind of a double-edged sword.
So when the R9 290X came out,
at first it was like entirely unusable
because of the way that AMD aggressively
implemented power tune firmware,
I think it was 2.0 at the time.
But the GPUs would throttle so aggressively
that it was like way slower than the 7970.
And it was just essentially unusable.
So then I wrote od6config.
Sorry, did I say PowerTune?
PowerTune is fucking NVIDIA. Overdrive. I'm sorry.
Oh, no, no. PowerTune is AMD.
PowerMizer is NVIDIA. All right, whatever.
So od6config enabled us
to properly configure the R9 290X.
But it was good,
but it was still kind of mediocre.
But at that time, when I measured it,
it was only drawing 300 watts,
which is important
because our server platforms like the Brutalis
can only handle GPUs that drop to 300 watts.
You draw more than that, you're going to blow a fuse
on the PCI slots on the motherboard.
So we had a automatic overclock profile
for the R9 290X that we pushed out to our clients
that made the GPU draw just about 300 watts.
And then a little bit later down the road,
about a year, year and a half later,
AMD pushed out a new driver update
that actually caused the power consumption
to go up by a good 50 to 75 watts.
And we did not know this.
This kind of went undetected by us
until we started getting a rash of clients
who were seeing dead PCIe slots on their motherboards,
and all of them had R9 290X-based solutions.
And I finally busted out a kilowatt
and figured out what the fuck was going on.
But we actually had the ability,
through all these mechanisms that we had in Hashtag
at the time to enable the automatic overclocking
and configuration and all that,
to push out a new underclock profile
to all of our clients to try to drop
the power consumption down to keep it
from blowing PCIe slots on the motherboards.
So it's both good and bad.
The fact that we're able to automatically overclock GPUs
to help get the most speed out of the GPUs is awesome.
But then in that one specific scenario,
it was bad because we started blowing PCIe slot fuses.
But then it was good that we had this functionality
because then we could correct it through software,
over the air, and automatically correct it
on all of our client systems all at the same time.
Which saved us a lot of money
because we were spending hundreds of thousands of dollars
on new motherboards.
So yeah.
And then, of course, Hashtag also had
cluster resource monitoring.
So there were some drawbacks.
Like I said, the entire Godend thing
was written in Node.js,
and while Node.js is asynchronous,
it also runs on a single CPU thread.
That may have changed at some point,
because we're talking like 2013,
that's seven years ago,
and I haven't even touched Node.js since.
But at the time, everything ran
on a single CPU thread,
which was stupid,
because our cluster controllers
had a minimum of 12 CPU cores,
24 threads,
and we're sitting here stuck in one thread
on the server daemon.
So yeah, any long-running methods
would just totally block
the entire application,
and we had massive scalability issues.
I think the very first alpha
that we released to clients
could only support four nodes.
The second one,
we were able to get up to nine or ten nodes.
Basically, any time a client ordered
a cluster larger than the previous
largest cluster we had ever built,
we ran into some kind of scalability issue,
identified some bottlenecks,
and had to work to remove those bottlenecks.
Unfortunately, 90% of the time,
the bottleneck was the fact
that JavaScript was too slow
for the task that we were trying to do.
So we had to reimplement
those bottleneck methods
in a faster language,
and then just shell out to them.
So keyspace calculation
was reimplemented in C.
Now, calculating keyspace for a mask attack
is really lightweight,
but when you talk about massive dictionaries
or rule files,
trying to count the number of lines
in a file with JavaScript
is painfully slow.
So that's largely why
that entire portion was reimplemented in C,
and we just shelled out to that binary
to eliminate that bottleneck.
We didn't dramatically reduce it.
Regex.
Anything Regex,
we actually implemented,
we had reimplemented it in Perl,
because Perl is amazing at Regex.
JavaScript is not very good at Regex.
The primary thing
that we were doing with Regex here
was validating the hash lists.
So a client uploads a hash list,
and some of our clients
wouldn't even bother throwing
200 million hashes at hashtag
in one job.
But then hashtag has to sit there
and receive the massively large hash file,
and then validate each individual hash,
and then stuff them in a database.
That was a massive bottleneck.
So we ended up reimplementing
all that in Perl
to try to get that down.
And then the agent,
the entire fucking agent,
we ended up just implementing
the whole thing in Perl.
And the only reason
we picked Perl at the time
is because that's the only language
we could iterate fast enough in
to rewrite the entire damn thing.
But yeah.
Like I said, we also used MongoDB
because NoSQL was the rage at the time.
And out of all the NoSQL databases we tested,
Mongo was the most performant.
But then we had clients do things
like throw 200 million hashes at hashtag.
And that would cause
some not only bottlenecks
and blocking processes,
but some really UI weirdness as well.
They would go through the process
of creating a job,
uploading the hash list,
and they would just sit there
with the little spinner
for a while until it timed out.
So they'd go to create the job again
because it timed out.
And then they'd go back 10 minutes later
and it was actually active and running.
But because they went through the process four times
trying to get it to start,
now they have like four of the same process
running on the cluster.
So we threw Redis in front of Mongo
as an in-memory caching layer.
And that actually had a significant impact,
like positive impact.
And it enabled us to ingest hashes
a lot faster.
But still not a very great solution overall.
The Web UI client was implemented
as a single-page application in Angular.
It was really clunky.
It was really heavy.
It was really slow.
It constantly caused the browser to freeze
if you left it up for more than 30 minutes.
And it just had a ton of bugs.
And I think probably the most annoying thing
is if you're like a hashcat power user
and you sit down and try to use
the hashtag Web UI,
it was just cumbersome as fuck.
There's no way to move quickly
in that UI.
And it gets very repetitive
and very tedious.
So there was a lot of drawbacks
to having that Angular Web UI.
I absolutely hated it.
And anyone who was really experienced
with cracking and hashcat and stuff,
they also hated it
and begged us not to have that again.
And we were just severely limited
by what we could do in a browser.
For non-hash formats at the time,
we were scraping everything server-side.
So you would actually upload a PDF
or upload a .docx
or whatever,
and we would scrape the file server-side
to get the necessary bits,
like Office to hashcat
and PDF to hashcat,
shit like that.
The problem then became,
what do we do with things like
TrueCrypt volumes
or really large 7-zip volumes
or RAR files?
Are you really going to upload
a 100-gigabyte disk image
through the browser
just so we can scrape off
two kilobytes?
No, that doesn't make any sense.
So yeah, the browser was an issue.
And then we implemented access controls,
but there was nothing fine-grained.
We assumed that because everyone
operating the hashcat cluster
was on the same team,
that they all had equal rights
and equal access to the cluster,
and therefore all users are admins.
We didn't have different roles or anything.
It was just, if you have an account,
then you obviously have a right to be there,
so therefore you are an admin.
And I think some of our clients
didn't have a problem with that,
because everyone on this team
does have equal rights and access
and privilege to the cluster,
so this is totally fine.
Other clients were like,
this is absolutely unacceptable.
And I eventually got to the point
where the old version one code
was completely unmanageable.
Even just implementing
what should have been a minor bug fix
ended up requiring major refactoring.
It just wasn't,
because we threw it together
in less than two months,
so we didn't think of everything in advance
that would come up,
and try to implement new features,
and try to work around
some of the bugs in there.
So we decided we needed
an entire ground-up rewipe
for Hashtag version two.
And at some point,
I got the idea
to create a commercial fork of Hashcat
with native distributed capabilities.
And at the time,
I was so fucking stoked on this idea.
It seemed like the best idea
in the world to me at the time.
So we set out to do just that.
Now this is right
before
Hashtag went open source,
and then we started work on this
literally the minute
that Hashtag went open source.
So we implemented in Golang
in C
with Postgres database.
Now we chose Golang
because of high concurrency
with goroutines, right?
So with Node.js,
everything ran on a single CPU thread.
And we wanted
the opposite of that for version two, right?
We're going to learn the lesson on that.
So we wanted
just lots and lots of goroutines
to run everything
so that we'd use all 24 threads
plus on the box.
And Golang actually also
allows for pretty rapid development.
So we thought it was a pretty good choice.
But we still stuck
with the traditional client server
agent architecture.
We have numerous clients
and then one server
and then however many agents.
Meaning that that server
is both a bottleneck
and a single point of failure.
So the first thing we had to do
is we had to split Hashcat in two.
We took everything
that had to do with
actually starting a cracking job
and executing an OpenCL kernel
on a GPU
with the agent
and nothing else.
The agent was just barebones
trimmed down Hashcat
launch attack on GPU
and that was it, nothing more.
And everything else
we placed into the server library.
And then for any code
that wasn't already in Hashcat
such as the JSON API
and the
protobufs
for agent server communication
workload distribution
fine-grained
multi-role access control.
All that was implemented in Go
and then we just linked against the C libraries.
And this effort actually
laid the foundation for
the Hashcat 4.0 refactor
with libHashcat.
And then because of how clunky
our old web UI was
we actually ditched it entirely
for a cross-platform CLI
which made our power users
like really happy
but our less experienced users
were not very thrilled
with that decision.
They still wanted a GUI
which, fair enough.
So our plan initially
was to maintain both Hashcat
and Hashstack trees in parallel.
And then
if there was any features
that we added to Hashstack
or any new kernels that we wrote
we would just backport those
as we chose to Hashcat.
This didn't really work out
very well in practice though.
So when we released version 2
it was in sync with Hashcat 3.5
but it didn't stay that way
for very long.
And of course we had to add
a cross-platform GUI
to the roadmap as well
because we had significant
feedback from our clients
that they really wanted
the goddamn GUI
and they were not happy
that we took away the web UI.
It turns out
this was the single dumbest
motherfucking thing I've ever
done in my life.
I completely underestimated
how much work was involved
in creating a commercial fork
of Hashcat.
Like, genuinely
I...
Yeah, I really don't know
how to convey how fucking stupid
of an idea this was.
We spent like 100%
of our time just backporting
things from Hashcat into Hashtag
instead of actually adding new
features or writing new kernels.
Like, literally months of work
you know, just backporting
Hashcat commits.
And it got really frustrating.
You know, it's like,
hey, are we working on these
tickets? Are we working on these
features? No, we're working on
backports.
You know, I got so sick of
hearing working on backports,
working on backports, like
when are we ever going to be
done with backports?
And then when Adam refactored
Hashcat for 4.0, that also
meant that we had to refactor
Hashtag as well.
And we hadn't even finished
implementing all the backports
at the speed that Adam was, you
know, committing things to
Hashcat.
So this was just like,
absolutely
unmaintainable, right?
Like, fuck me.
And then there was also this
other little minor problem
where we picked the wrong
HTTP library
for the API.
And this basically
limited the service capability
to about 12 nodes.
Now, 12 nodes
might seem like a lot.
That's 96 GPUs.
You know, that's a pretty
respectable size cluster.
The problem is this
particular client had
ordered 320 GPUs,
40 nodes.
And you can't really
deliver a 40 node client
to a cluster with a
server that can only
really handle
12 nodes.
So this required
not a complete rewrite,
but a damn near complete
rewrite time.
Fuck, because we use that
HTTP library
extensively throughout the
code.
And there was no, like, drop
in replacement.
So basically anywhere where
we were dealing with
HTTP,
we had to rip out the old
library and then,
you know,
add in all the stuff for
the new library.
But they did resolve the
issue.
It was just really painful
at the time to have to
mess with that.
But the server
still continued to be a
single point of failure
and still proved to be a
source of bottlenecks.
As we, you know,
tried to build
larger and larger
clusters,
we
kept running into
problems with the
server
being a bottleneck
and a single point of
failure.
And we also had clients
coming to us asking us
if they could order
multiple cluster
controllers and cluster
them together,
you know, either
high availability
or, like,
load balance them
or something.
And
we didn't have a
solution for that,
but I started thinking
about a solution for that.
And again, like,
40% of our clients
were just completely
pissed that we had no
GUI.
So
it got to the point
where it's like,
let's just
rethink, like,
the entire fucking thing
and just, like,
go as far away from
these mistakes as we
could possibly get.
Like, we never, ever
want to make any of
these mistakes ever again.
We've learned a lot of
lessons from, you know,
what we've done over
the last seven years.
So, but we need
more than just a
complete ground-up rewrite.
Like, we need an entire
paradigm shift.
Like, we need to
rethink this entire
problem
entirely
and
come up with
something that's
completely different
from what's ever been
done.
So we need to
eliminate the single
point of failure of
the server.
We need to enable,
like, actual infinite
horizontal scaling
because, again,
we're, you know,
we always try to
strive for, like,
warehouse as a
computer, right?
We're looking at
warehouse scale
computing as, like,
the target that we're
trying to hit.
There should never be
any, you know,
limiting factors
outside of budget
for, like, how many
nodes we can
support in
HashtagWrite.
And then, of course,
the other big requirement
is we have to actually
move at the pace of
Hashtag development.
We have to,
you know, as Adam
commits things to
Hashtag, we need to
have those
not instantly available
in Hashtag, but within
a very reasonable
time frame.
So
the very first
idea that we came up
with was let's not
do a traditional
client-server-agent
model.
And let's not even
try to cluster the
cluster controllers.
Let's just eliminate
the cluster controllers
entirely and make
everything peer-to-peer
where all nodes
are equal
and there are no
dedicated servers.
And
the biggest requirement
to this is that
work can be submitted
to any node.
So let's say you have,
like, you know, 20 users
and you have,
you know,
10 boxes.
Like, there shouldn't
be just one node that
all of them have to hit.
They should be able
to hit any node
in the cluster
and, you know, have
the same view
from any node.
And it also needs to
be more user-friendly
than ever.
And also needs to
run on Windows.
I'll tell you why
in a minute.
So set out to,
like, figure out
how we're going to
implement this.
Because we have the
high-level requirements,
right? Like, we know
it needs to be
peer-to-peer.
We know all the nodes
need to be equal.
You know, all that shit.
But how are we actually
going to do this?
So
pretty obviously
we're going to need
some sort of
distributed state machine.
Right?
And that's just
straightforward
and self-explanatory.
But, you know,
like, we just
throw a message
queue at it
and some PubSub
and, you know,
Bob's your uncle.
But what about
disagreements?
Or, you know,
maybe even more
than disagreements,
what about, like,
an actual, you know,
rogue
cluster member?
Or, you know,
a broken
cluster node
that's maybe not
necessarily
intentionally
malicious,
but it certainly,
you know,
has, you know,
the appearance of
malicious behavior.
Right?
So I started
coming up with
solutions for
this.
And
I somehow
got into my
head, like,
dude,
what if
computers could
vote?
Right?
Like,
you just
make it to
where, like,
you know,
if there's a
dispute,
then one of
the nodes,
like,
requests a vote,
right?
And so they take a vote
and majority wins.
And if there's a tie,
like we have an even
number of nodes in the
cluster, then,
like,
literally fucking
flip a coin,
you know,
randomly,
or play rock,
paper,
scissors.
Like,
why not?
And if a peer
doesn't accept the
results, like,
then the rest of
the cluster just
shuns that peer.
And I started
thinking about,
like,
is this actually,
like,
a thing?
Like,
can we actually do
this?
Am I,
like,
completely fucking
stoned?
It turns out that
I really didn't have
to put that much
more thought into
it because this
already exists.
Like,
what I thought up
is actually a
thing called the
wrapped consensus
protocol.
Now,
it's cool that it
already exists because
that,
you know,
means we don't
have to do a
whole lot to
implement it.
But it's also cool
because it kind of
validated,
you know,
my ideas and theories
on how to make this
work.
And told me that I
wasn't just
completely fucking
insane.
And we found
several implementations
of the wrapped
consensus protocol.
And the one that
we ended up liking
the most and
settled on is
Akka,
which runs on
the JVM.
It's written in
Scala.
Now,
I hate Java.
I have shunned
Java for damn
near my entire
life.
You know,
I completely
condemn
Oracle JVM.
But,
there is
OpenJDK.
We don't have
to use Oracle.
And we don't
have to write
in Java.
We can use
Scala and
Kotlin.
And since
I've started
to work on
this,
I actually
really like
Kotlin.
I don't
really have too
many negative
things to say
about it.
It's actually
kind of fun
to work with.
And then
Akka also
has some
plugins or
modules that,
you know,
help us even
further.
Such as
Akka
persistence.
What Akka
persistence does,
it takes our
distributed state
machine,
which is
normally
resident in
memory,
and persists
it to
disk.
And then
Akka
distributed
data, of
course, would
be like what
implements the
distributed state
machine.
And then on
top of all
of that,
we built a
custom
multicast-based
protocol to
enable cluster
nodes to
automatically
discover each
other on
the network
and then
automatically
join the
cluster.
Now, we
made this opt-in
because obviously
that's kind of
a security
risk if
anybody can
just join the
cluster at
will.
But for
some scenarios
like where
all of your
nodes are on
a dedicated
subnet that's
properly firewalled
off and you
have proper
layer 3 and
layer 4 access
controls,
this would be a
solution that would
be really handy.
So, again,
thinking about
warehouse-scale
computing or
data center as
a computer,
let's say you
just have racks
and racks and
racks of compute
nodes, you would
just unbox one,
literally throw it
in the rack and
power it on, and
it would discover
the cluster as
soon as it's
powered on and
join the cluster
and start doing
work.
That's kind of
the vision that we
have for this.
And, of course,
since it's all
peer-to-peer,
that enables it
to actually be
just literally
that simple.
But, of course,
the default is
you manually
specify which
peers you want
to be part of
the swarm.
Okay.
And then, in
April this year,
we had the
unique opportunity
to present
itself to
purchase
loft crack.
Dildog approached
me and we
entered negotiations
and we
purchased it
earlier this
year primarily
for the
GUI, which
is written
in C++
using QT
as the
windowing
framework.
And while
it's not
currently
cross-platform,
that's a
really easy
platform, or
that's a really
easy language
and library
to make
cross-platform.
So, instead
of
hashtag
version 3,
we now
have loft
crack version
8.
And while
loft crack
7 uses
John the
Ripper as
a backend,
we've ripped
all that out
and we are
in the process
of integrating
what was
hashtag version
3 as the
new backend
for loft crack
8.
And the
final thing
we're doing
with this is
we are dropping
the
TeraHash
hardware
requirement.
Traditionally,
you know, we
do not let
people purchase
hashtag as a
standalone product.
We say, like,
you know, the only
way to get
hashtag is to
buy a
TeraHash
appliance and
it comes
preinstalled
and that's the
only way to
get it.
But loft crack
has always been
sold standalone.
So we are changing
the model now
to where,
you know,
keeping with
loft crack
being sold
standalone,
we're going to
be selling
loft crack
8 as a
standalone product
without the
TeraHash
hardware
requirement.
So, yeah,
that's where
we are now
with development.
That's how the
product has
evolved over
time.
And that's some
of the
challenges that
we have faced
trying to
obtain our
goal of
true
warehouse scale
computing.
Thank you and
hope you've enjoyed
this.