Sounds like we're live. I wanted to start off today's presentation by thanking everybody in the
DEF CON community for welcoming me to present on the Storage Network in Tardigrade today.
This is my second time presenting at DEF CON and I'm really excited to be here.
Today what I'm going to do is I'm going to first give a quick update around the Storage Network,
which launched in March, talk about the recent wins from an adoption standpoint,
what tools are being built on the network, who are we working with.
Then I'm going to go into a quick workshop that will walk you through the process
of backing up Kubernetes configuration and object data to the decentralized cloud using an open
source tool called Valera, which is sponsored by VMware. And so with that, let's go ahead and get
started. And if anyone has questions, just post them in the chat and we can get to them at the
end right before the workshop. So as a company, our goal is to build the world's largest,
most secure, resilient, performant, and economical cloud storage service without owning or operating
a single data center. And what that really means is that rather than having the cloud
controlled by gatekeepers across a small number of data centers, we enable anybody to be the cloud
by monetizing their excess hardware capacity, their extra storage space, and their bandwidth.
And so we formed a decentralized cloud storage network that is highly competitive with, say,
an Amazon S3 or Azure Blob or Google Object Store just by incentivizing mass amounts of individuals,
data centers, crypto enthusiasts to offer their excess capacity to help build the network.
And so in terms of some stats, right, the original prototype for storage was first
built in 2014 by Shawn Wilkinson at a hackathon at Texas Bitcoin Hackathon.
It's gone through a lot of iterations since then. The key milestone was probably in 2019,
we began work on version three of the network after hiring Ben Golub, who is the former CEO of
Docker, you know, having built the largest open source community by Linux. And so we launched in
March. And in terms of stats, we're at 26 petabytes of capacity. We've done a number of
cool things, you know, outside of our early customers, we've actually backed up GitHub
through a tool called Git Backup. And that's a pretty cool open repository of data. And we're
helping other people share open data sets as well. We have about 8,000 storage nodes,
more than 8,000 target users, which are developers that are storing data on the platform,
and actually 72,000 unique wallet addresses of people that are participating in our token
ecosystem. We've today stored 50 million files across four satellites. And, you know,
are really proud to have been the number two project in terms of development activity
in the Ethereum ecosystem in 2019, as measured by sentiment insights and consensus media.
We have a really awesome partner ecosystem that's driven by a number of open source projects who
have integrated with Tardigrade directly, whether for backup tools, or for, you know,
second level kind of graphic user interfaces on the platform. One of those that's pretty notable,
we'll get into kind of deeper case studies in a second would be MongoDB. We recently launched a
native integration into their enterprise tool DevOps manager. So let's get into the fun part.
How does Tardigrade work? What's the security model? Why is this a better approach to storing
data than on traditional cloud data centers? So the way that Tardigrade works is that there
are three peer classes, or really three pieces of software that anybody can run that powers
the network. So first off, there's the uplink. The uplink uploads data to the network, encrypting it,
splitting it into pieces, and spreading it across nodes. The satellite talks to storage nodes,
which are sharing their capacity and monetizing it. The satellite builds reputation around the
storage nodes based off of their uptime, their response time, and their statistical
and correlation to each other, and then uses that reputational profile to determine where
data gets pushed from the uplink. And so the satellite acts as kind of the the key decision
maker. It's an open source peer class that anybody can run. We run a collection of them with
industry standard service level agreements through Tardigrade, but anyone
can run them and they decide basically where the data gets stored. So when you upload the data,
it gets broken up into pieces and uploaded in parallel directly to storage nodes without
touching the satellite. The satellite's just telling the uplink where to put the data. So let's talk
about the journey of a file, what happens when you upload a file to the decentralized cloud.
It's actually pretty exciting. So what first happens is the file gets encrypted. We'll get
into the standards in a bit. It's AES-GCM256. It then gets broken up into erasure-coded pieces,
and we'll get into what that means in just a second as well, and then pushed across statistically
uncorrelated endpoints across the world. And so what's really critical about Tardigrade is not
only do you have default encryption and default geo-redundancy, right, features that you might
pay extra for with alternative cloud storage providers, but you also have statistical
and correlation from an ownership of the hardware standpoint and network infrastructure. And so
your data, you know, there's not one single provider that could pull the plug on your data,
making for a much more resilient cloud storage system. We'll get into kind of how audit and
uptime is ensured as well in a second. So in terms of the security differentiators of Tardigrade,
there are a number of really interesting pieces that make Tardigrade more secure,
more reliant, more reliable than traditional cloud service providers. So first is client-side
encryption. So by default, all data is encrypted client-side. This prevents scenarios like the
leaky bucket scenario that Capital One faced, where an exposed S3 bucket was then downloaded
by everyone and exposed a bunch of customer data. So data is encrypted client-side rather
than server-side. It's encrypted at the edge, at the source. For API keys, we embed logic inside
of the API keys using a embedded logic structure called macaroons, which was originally formulated
by Google. What this means is that instead of, you know, querying an access control list,
the API key is actually the access control credential. And so inside the API key,
there's embedded logic that can restrict usage of a file from, say, read-write to only read
time-bound files to only be read for a certain period of time and is extensible. And so you can
actually embed third-party logic inside of it that enables you to do things like require
OAuth verification and pass an OAuth or SAML token. So there's some really cool constructs
with macaroons that we're playing around with. And today out of the box, we support kind of
standard file restriction capabilities. And all the developer tools on Tardigrade are simple.
And so it's a really easy, sweet software called libuplink that enables you to
connect your applications, object storage to the decentralized cloud.
So let's start with encryption. Encryption is really important.
Encryption is privacy by default on Tardigrade, and if it was only ratio-coded, it'd be easier
to take someone else's file. So in terms of specifications, all encryption, again,
occurs at the edge, at the absolute edge of the client. Using the AES-GCM-256 standard,
which is a military-grade standard, the encryption, if you went into libuplink library
and forked it, it's open source, you could plug in your own encryption if you wanted to as well.
It's hierarchically deterministic and it's path-based as well. And so you might have
heard of hierarchical deterministic encryption from BIP32, which is the Bitcoin Improvement
Protocol 32, which added hierarchical deterministic key derivation for Bitcoin.
So you can have one master wallet and sub-wallets. On storage, encryption for files works much the
same way. And similar to how on Ethereum or on Bitcoin, your keys are your money. On Tardigrade,
your keys in many ways are your data. And so protecting that key is really crucial
to the security of the network, and that can be held offline as well.
So erasure coding is kind of the next really important differentiator that we have
from a security standpoint. What that means is after you encrypt a file,
it's broken up into segments. Each segment is broken up into 80 pieces of which you only need
today 29 pieces of the 80 to reconstitute the file. And so from a resilience or security standpoint,
what that means is that you need 51 uncorrelated nodes with different owners and different geos
with different internet service providers to all go offline to put availability at risk
at a given point in time. If nodes are constantly measured for uptime, if any one of them goes down,
that data is repaired and new parity pieces are created. Erasure coding is also one of the core
reasons why we have a performance advantage. So the way that Tardigrade works is instead
of a single thread stream, data is streamed in parallel, somewhat like BitTorrent.
But erasure coding adds an extra benefit because when you call 80 pieces and you only need the
first 29 responders to reconstitute the file, you're always going to get the fastest 29 pieces.
Any what we call distributed systems, any long tail responses are basically smoothed out of the
distribution due to this cool erasure coding schema. And so this drives, again, both resilience
of the data and performance through parallel packet streaming and kind of native concurrency
methods in Go. The third piece, which we chatted about a second ago, is the access management piece
that's built around macaroons. So traditional cloud storage providers use an access control
list or an ACL that's stored in a cloud database to control who has permissions to each file.
We've seen a number of devastating attacks with the ACL model, mainly around super user privilege
escalation attacks, where someone climbs up the ACL privileges and then changes everyone else's
privilege and can create ransomware out of attacks like that. Tardigrade is natively designed to
prevent this. All access control happens at the edge and is associated with the macro and with
the API key. And so you can actually, again, embed logic into the API key, which determines identity
and access management. And so we think that edge-based identity access management alone
is a huge security advantage over traditional ACL cloud-based models because it is much more
resilient to privilege escalation attacks and to a number of other attacks as well.
And so Tardigrade is built by default from the ground up for a trustless environment, right?
The idea is that you can store your data on any endpoint, any uncorrelated to endpoint,
and it doesn't matter. It's built for zero trust, zero knowledge environment, making it from the
ground up much more secure than solutions that are built for trusted environments because,
as we know, trusted third parties often tend to be security holes. And so again, we have
client-side encryption, we have path-based encryption, all the content and metadata is encrypted before
it goes to the satellite, the data storage is distributed so there's not a single point of
failure, and all the endpoints are uncorrelated, and all that ties back to the encryption key
ownership. And so we think that this is a really exciting model to try and disrupt a lot of the
weaknesses that we see in standard object storage models. And so how do you get started with
Tardigrade? How do you upgrade your stack today? Well, here are a few ways that you can integrate
Tardigrade or the storage network into your stack. One of them that we've seen, one early
use case, is around database backups. And so we have tools today for MySQL, Postgres, InfluxDB,
which is a time series database, MongoDB, and other distributed databases like Flurry.
We have native backup tools that enable you to either dump or schedule and dedupe regular backups
of your database onto the decentralized cloud. And a number of companies are really interested
in this due to the uncorrelated nature of Tardigrade. So data is stored across 80
independent operators, there's no single point of failure from the storage side, and we think that
this creates a better security play for backups and for disaster recovery than storing your data
on a centralized provider that's going to be correlated with all the other providers.
In terms of, you know, the MongoDB integration, we're natively integrated into MongoDB's DevOps
Manager tool. And so that's something we're really excited about. We launched that integration
a couple of months ago. And if you go to MongoDB's website, if you search MongoDB Tardigrade,
there's a great blog post that walks you through the process of basically scheduling backups
from DevOps Manager to Tardigrade so that your backups are stored and secured on the
distributed cloud. Another great integration that we recently launched was the integration
with rClone. So rClone is a really popular, common command line program to sync files
and directories. It's often referred to as rSync for the cloud. And so any sysadmin here that's on
the call is going to be familiar with rSync. And rClone is a similar tool that enables you to kind
of map file systems, you know, basically map commands, whether they're dump commands or any
kind of commands to natively integrate with Tardigrade. And we have, again, a native integration
with rClone. So if you download the rClone command line suite and go to their website, you'll find
that you can just set up Tardigrade just by inputting your API key or your access directly
into rClone. Another interesting integration for users that might not be technical that we've seen
a lot of managed service providers and just non-technical end users use is our integration
with Duplicati. So Duplicati, again, is a GUI-based backup tool. So basically you click on your file
system and you choose what you want to back up and it backs it all up. And so it's great for, like,
just native desktop backups. And it has a lot of cool features around deduplication and compression,
which limit, you know, the actual amount of raw data that gets uploaded, which is more
economical than a non-compressed, non-deduped backup. Another really cool thing, which I'll
actually show as I stream the workshop, is that, you know, Tardigrade natively supports video
streaming today. And so from the command line, you can generate a link that allows you to stream
video. It's called Tardigrade Share. And so you can generate an access that's embedded inside of a URL
that embeds that macro and logic. You can see here it's a rather long URL string. So you might
need to use a shortener if you're going to share it publicly. But it embeds that logic inside of
the URL and lets people watch videos and streams it in parallel from the decentralized web. This is
something that's really cool and I think really powerful as well. I'm really excited to see where
we can start to do things like, you know, map link sharing outputs to decentralized DNS systems,
like Unstoppable Domains, Handshake, and some of the other ones that have been mentioned earlier
today. I think, you know, decentralized storage with decentralized DNS is going to be something
that's pretty cool. We also have great native integrations with a number of hardware providers.
And so QNAP, you may have heard of them. They're one of the larger hardware manufacturers out of
Taiwan, which there are many. QNAP creates network-attached storage arrays and, you know,
you can back those up directly to Tardigrade with a one-click app that also shares any of
the excess capacity to storage, monetizing both your storage and your bandwidth. And so if you
have a QNAP, a Synology, or Western Digital network-attached storage array, we have documentation
that shows how to natively and really easily install an application that both backs up your
device to the decentralized cloud and then shares excess capacity to help pay for it. And, you know,
that's also, I think, a super cool use case. Finally, we have a great integration, you know,
with Kubernetes, and we'll get into kind of a workshop around this in just a second.
And so we have a native integration with Valero, which is, again, a VMware-sponsored open-source
tool that enables you to back up your Kubernetes configuration object data as well as persistent
volumes. And so we'll get into why you might do that, right? You know, as a DevOps manager, you
know, for looking at disaster recovery, disaster prevention, it makes sense to have, you know,
backup of your cluster on the distributed cloud where it's globally distributed. What that means
is that it's really easy to rehydrate your cluster to any compute environment rapidly across
the globe with a single backup origin. And so we solve that problem that a lot of developers face
around data locality, which is, you know, where do I store my backups? Where do I store my data?
Do I store it in U.S. East? Do I store it in U.S. West? Do I replicate it to Asia and Europe?
With Tardigrade, data is replicated globally by default, and it's globally available by default,
which makes it great for disaster recovery when you need to rehydrate, you know, it could be
anywhere, right? Maybe you need to rehydrate to Asia, or maybe you need to rehydrate somewhere
in the United States with Tardigrade, you know, one backup solves kind of all those use cases.
And so again, right, this is a great platform that was built by developers, really for developers,
and really easy to use. All of the kind of the complexity around erasure coding, encryption,
audit, repair, and those kind of backend network functions are all abstracted. And so as a
developer, you're just looking at kind of a basic, you know, CRUD library, where you can create
buckets, put objects in buckets, stream them, delete them, very similar to, say, an S3, but at
half the price of S3, you know, with better security, resilience, and often better performance
as well, depending on the use case. And so again, super easy user interface, I won't get into how
to generate a key for this next demo, we're just gonna assume that that's already happened,
where you can create an account and generate an API key in just a couple of clicks.
And if you're interested, I recommend you do so. We also drop in S3 compatibility.
And so if you, and this is really a great demo that I often show, you can actually reconfigure
the AWS command line to talk directly to the decentralized cloud through a gateway hosted
process. And so this is a fork of Medio, which provides S3 compatibility for Tardigrade. And so
if your application today is using Amazon S3, and you want to migrate to the decentralized cloud to
cut your cloud storage bill in half, and have better resilience and performance, you can do that
by running the gateway and easily migrate over without having to rewrite any code, which is also
really powerful. And so that's the quick overview. Ajit, does it make sense to answer questions now
before I get into the hands-on workshop? Or should we wait till the end for questions?
Cool. Anyone have any questions? Well, we'll go ahead and get started. And if you have questions,
post them in the chat. So for the workshop today, we're going to walk through the process
of creating a decentralized storage backend for Kubernetes that uses Tardigrade and Valero
toolset to make that possible. And so let's go a little bit deeper in what Valero is. Valero
is a tool that enables you to back up and migrate Kubernetes clusters. And so Valero
natively uses the Kubernetes API to capture the state of the cluster resources, which makes it
easy to rehydrate at any target. And so again, Tardigrade is a great backup option for this tool
because it's globally distributed, because it's highly economical, and because it's highly
secure, encrypted, and resilient as well. And here's kind of a quick little diagram,
by backing up Tardigrade, you can restore your cluster to any compute environment,
whether it's cloud or self-hosted globally, really quickly. It's really easy to install.
There's a simple install command, and then you point it at your existing kubes environment,
and you can back it up. And I'll go ahead and show that off now. And so what I'll do is I'll
stream a demo from Ivan, one of our solution architects and engineers at Storage Labs,
and he'll walk through that process. But first, in terms of early use cases and wins,
we've had a number of customers who already started using Valera to backup their Kubernetes
databases, sorry, their Kubernetes instances today. And so what's really great, what we've
heard is that backups can capture a subset of the resources, so it's really flexible.
You can filter on namespace, or on label, or on resource type. So you can really granularly
define what gets backed up. Users also are excited that it backs up the underlying etcd database.
So often, other backup solutions don't backup the etcd database, but Valera does. And then also,
the resources are exposed to the API servers. And so even if the resources are in separate databases,
they can also be backed up. And so for Kubernetes applications that are looking to improve their
disaster prevention strategy, this is one piece of a larger puzzle, right? You need to
also backup all the state associated with your existing databases and persistent volumes.
But this is kind of a really great place to start in tinkering with the decentralized cloud to
improve the resiliency of your cluster. And so what I'm going to do is I'm actually
stream this demo directly from the decentralized cloud to show you how performant it is. If this
doesn't work, if the internet gods shame me, I can switch over. But let's go ahead
and get this started. In this video, I am going to show you how to use the Valera storage plugin.
Valera is a tool that allows you to backup Kubernetes configurations and persistent volume
snapshots. In order to use the storage Valera plugin, you have to create an account in Tably
server. You can go wherever Tably server, sign up, create a project, and after you have a project,
create an API key for that project. Once you have that, make sure that you write down or you
copy and paste that API key because it's the only time that you will see that API key and keep it.
So after that, install uplink-cli. That is the command line tool that Tably has for accessing
to the network from the command line. So with uplink installed, you call setup. After you select
the server where you sign up, in my case it's US Central, select a name or use the default one for the
access, grab your API key and set it, and just select a passphrase.
Once you did that, you need to grab the access grant that has been generated with this data.
The access grant is stored in the configuration file because I haven't specified any configuration
file. The uplink has created the configuration file in the default folder. The default folder
is in the case of a unit machine in my home directory under local, share, storage, uplink,
config file, config.yaml. So I want to grab that access. That is what I need for
configuring the developer storage plugin. I copy the access. I am going to export to
one environment variable that I am going to name it access, store access.
Because I will require that environment variable every time that I need this access.
And I will avoid to pollute this demo in every command with this long string.
Bear in mind that the API key is not something that you have to share. In my case, I created
this API key just for the purpose of this demo. Right after, I will delete it to avoid that
anybody can use it for using, obviously, my network. So see my files and create files on it.
So the first thing that I am going to do is create a bucket. This bucket is needed because
once you set up that, when you specify a backup in Galero, you have to specify a bucket.
And the bucket has to exist before doing the first backup. So let's make sure that I have
the bucket, uplink.tls. So right now I have this bucket Galero, but I am going to delete it
because I want to start it from scratch. I remove the bucket with whatever content it has.
Now I don't have any bucket, so I am going to create a bucket that is going to call,
I am going to call it Galero. Once I created a bucket, obviously, that bucket is empty.
But let's show it. Okay. Now I am going to install Galero in a Kubernetes cluster that I am running
in my log. At the same time that I am installing Galero, I am defining a bucket location.
The bucket location that I am defining is using the target grade plugin. Okay, so let's
install Galero with provider target grade, that is the name of our plugin,
and specify the image where we store that provider, that is storage third-party
slash Galero. And I am specifying this tag. This is a local image that I have
in my local only. This is not available online, but in Docker Hub we have the previous version.
We are still working on this plugin, so we are improving, and I am using this one because it's
much better. After, I have to specify a bucket. In this case, it's the bucket where I am going
to do the backups, that is the Galero bucket that I created right before, and specifically
tell to Galero that I am not going to backup as an option, persistent volume as an option.
This is because our plugin doesn't support it yet, but hopefully we will support it in the future.
And after, so I have to, for this configuration location that I am creating,
at the same time that I am installing Galero, I have to provide the access grant
that is the only configuration parameter that our plugin requires.
Oops, I mistyped the command, but
whatever. Oh yeah, and another thing that I have to do is specify no secret,
because Galero requires that you specify a secret, or if you don't need it, if you don't need one,
you have to specify that you don't want, that you cannot leave it empty. So in our case,
our plugin doesn't require it. So now Galero is installed, and I can see that in my cluster,
just getting deployment from components
components
equal Galero from, so from
namespace
and space Galero, that is
components namespace
deployment components
and deployments of components of Galero namespace of Galero,
that's what, this is component. Okay, that is what the installation has created.
What I'm going to do for the demo is creating an Nginx service.
And this is the service that we are going to do a backup, we are going to destroy it,
like simulating a disaster, and after we'll recover from the backup.
The Nginx service is something, is an example that it comes with the Galero installation.
So Galero has a folder inside of the tar file that has several examples.
This is the folder that I copied from the tar file. So let's deploy the
the Nginx service.
Once I did that, I can see in my Kubernetes cluster
that I have
they have a namespace that is called Nginx example and I have
and I have a service, the Nginx service.
So now let's do a backup. But before doing that, let's see if
backups, Galero backups get.
So this backup get doesn't show anything because we don't have any backup. So let's create one.
Galero backup
great Nginx backup. I call this backup Nginx backup and I only want to
backup the
the resources that have the level up Nginx, not everything.
So I create the backup. I can do backup get again and my backup is in progress.
And I wait a bit more and after it says that it's complete.
But okay, what we can do now is see what we have in our bucket.
We go away with the uplink ls and just list the, we can list the
the Galero bucket. Now you can see that it's not empty. There is a directory
that is called backups and with, well it's not a directory because we have keys but there is
this key inside of the kubernetes but it's kind of simulating a directory.
Nginx backup and we see that under that prefix we have all these files that Galero has created.
So great, we have our backup in our target network. So what we are going to do now is simulating this
this disaster. For doing that we are going to delete the entire namespace
of Nginx, call Nginx example.
Oh, a disaster is happening now and that namespace gets deleted.
We wait until we finish.
And now we are going to see how if we are getting the services
the services from the namespace
Nginx example. So basically there is no resources found. At the same time if we try to get
this namespace from the cluster, so basically it's not found. Okay, that's we are confirming
that disaster has happened. Now we are going to restore. Galero has a command that is restore
and you have to you can say what you are going to restore. So we are going to restore creating
from a backup that is called Nginx backup. That is the backup what we created previously.
With a restore. And now Galero restore get says that it's complete. So if we are going now to
list this so that is the namespace it should be there and it's there. And after we list the service
it should also be there and it's there. So we have recovered our Kubernetes configuration from
the delegate network. Okay, but now what we can do as well is at some point you don't want this
backup anymore. So we can delete the backup. Sorry, with Galero backup delete and you have to specify
the name Nginx backup. And after it asks for confirmation we confirm and after we do Galero
backup get we don't have any backup. So basically with this we unregister this backup so the backup
is not available anymore in Galero. But obviously we would like to know if those files are stored
has been deleted by Galero from our delegate network. Let's see that. So for that reason
I am going to list again
my backup, my Galero backup, and now it's empty. So basically everything works as expected.
I hope that you enjoyed the demo and I will encourage you to try our Galero plugin with
our great delegate object storage decentralized object storage network. See you next time.
Okay, thank you Ivan. And again that was streaming directly from the decentralized cloud
which I think is pretty neat and pretty telling of some of the capabilities of Tardigrade.
And so with that, that's the conclusion of my presentation and our Valero workshop for backing
up Kubernetes data to the decentralized cloud. Does anyone have any questions?
Looks like I see one popping up in the chat.
If anyone here is a developer, questions in Twitch. I don't know Ajit if you can rattle
those off, I'd be happy to answer them. But in the meantime, if anyone here is a developer,
you can get started with Tardigrade at documentation.tardigrade.io and we have a
number of walkthroughs on the website around just basically uploading data, sharing data,
streaming it on the internet, scheduling backups with databases, creating graphic user interfaces
around that data. You can find all of those walkthroughs and more on the documentation for
Tardigrade. No problem. So with that, that means the presentation went well. I answered everyone's
questions. And if anyone else has any other questions after the call, you can email me
directly kevin at storj storage.io. And we're always looking for partners and for developers
who want to build the next generation of cloud applications on the decentralized cloud.
We're happy to give you the white glove treatment and help you get building or migrate your
application to Tardigrade. So thanks again. And thanks again for Ajit for hosting me.
And I hope everyone has a great afternoon and happy Friday. Happy almost weekend.