start	end	text
0	7000	Well, thank you, Ajit. I want to welcome everybody to DEF CON and the Blockchain Village.
7000	11300	My name is Ron Stoner. I'm going to talk about securing the cosmos, and I appreciate that
11300	19600	wonderful introduction. I'm a repeat speaker here. I spoke last year when we were talking about the
19600	24200	cryptocurrency security standard with Michael Parkland, and it was an amazing experience,
24200	29680	so I'm happy to be back. I will preface this talk by saying I'm only on a few hours of sleep
29680	33860	in traditional DEF CON fashion, a little bit hungover, and I was working on my presentation
33860	38700	up until a few minutes ago. So if there's any inconsistencies in here, or you guys see
38700	43040	formatting issues, I'm more than happy to clean that up if you want to copy the presentation,
43040	47660	or if you want to discuss further just to clear up any ambiguity after the stream.
49000	56080	So who am I? I'm a senior security engineer for ShapeShift.com, one of the best digital
56080	60840	asset exchanges out there. And as a result, I get exposed to a lot of different technologies,
60840	64280	and I get to play with a lot of cool different things in the crypto space,
64280	70960	including Cosmos, the Atom token, and some cool new validation software that people are
70960	77560	releasing and trying out. I'm also the curator of the cryptocurrency security standard auditor exam,
77560	82800	so that's an exam regarding the cryptocurrency security standard that I referenced earlier.
82800	89340	I have my own consulting firm where I consult on security and blockchain technology. And then
89340	93600	if you guys want to reach me on Twitter, you can hit me up at forwardsecrecy.com.
93700	97580	The reason I'm giving this talk to you guys today is because I do have a lot of experience
97580	105780	on testnet and mainnet with deploying, configuring, administering, and securing validators
105780	114170	and validator infrastructure. I wanted to set up a goal for this talk. So you guys can read it there,
114170	119810	I'm not going to go through that. I do want to highlight my redundant joke here. Hopefully you
119810	126870	guys like that one. If not, oh well. But my goal for this talk is to have you guys walk away here
126870	132590	learning something. There's a lot of smart people here at DEF CON in the blockchain village.
133130	138030	But while we may be good at infrastructure, or we may be good at security, or we may be really
138030	143810	good at the blockchain aspects of it, nobody's perfect, right? Nobody's an expert in all areas.
143810	148970	Hopefully you walk away from this talk taking a tip or some piece of knowledge that you can
149490	152610	run with yourself or implement in your own system.
157100	166580	So what are validators? Everybody's kind of familiar with the concept of Bitcoin and SHA-256
166580	172920	and mining, right? So when you're mining, you're throwing a ton of resources at cryptographic
172920	179780	problems to solve them and write data to a blockchain. And mining was awesome. It was very
179780	184460	profitable for people for a long time. It helped secure the network. But there's a lot of downside
184460	191060	to miners that people have brought up over time, including energy consumption and administration
191060	198300	and things like I can't get the newest graphics card to play PUBG or Fortnite because there's
198300	203840	run on them, right? There are five, $600 now because everyone's buying them up and mining.
204480	209460	So some new coins and chains came out that implemented what was called proof of stake.
209540	214720	And unfortunately, it's not the stake that I enjoy every night. I would love a proof of actual
214720	220420	like a one type of stake. But this proof of stake is where we're actually staking and bonding coins
220420	227500	and doing our network through validators. So those validators actually provide a whole bunch
227500	232840	of energy savings, and they can achieve kind of the same end result. They're able to validate
232840	238960	transactions, validate blocks, provide signatures, create chain data and do it in a way where it's
238960	244740	more beneficial to the environment and it's lower cost and overhead. The way it works, as I said,
244740	251440	people end up bonding or locking or staking the coins with the validator. And then for this talk,
251440	255820	we're going to be focusing a little bit more on Cosmos. But with things like ETH2,
255820	260260	they have some different systems where they kind of bet on blocks and transactions and things for
260260	265100	the validators to process. But on the back end, what it's really doing is it's taking cryptographic
265100	276820	signatures and signing and broadcasting votes out to the chain. The incentive for the validator,
276820	283060	you can see right here is money, basically, digital value. So as a validator, you're validating all
283060	287360	these transactions, and you're running your infrastructure and you're putting in all the
287360	292380	time and effort. And what's the incentive to go and do this? So there's a few different pieces.
292380	298000	The first one's the security aspect. So you're providing back to the security of the chain,
298000	302720	the decentralization aspect. So you're participating in a decentralized system.
302720	309480	There's a governance aspect where as a validator, if you meet certain conditions based off of the
309480	314300	chain and governance that's been set, you can actually participate in governance votes and
314300	320400	help to modify and make changes to that chain and its different variables moving forward.
320760	325320	But there's also block rewards and transaction fees. And that's kind of where the money comes in
325320	331620	at. And this is just a very small example. So I think this is one day of rewards for a validator
331620	336560	that I was running on a new chain that was just released. And you can see here, I'm getting some
336560	343280	die and I'm getting some tick. And this validator, while it's using tick as its denomination and also
343280	348800	paying some commissions in die, this could be whatever coin or whatever chain that you're
348800	354240	running your validator for. So in the instance of Cosmos, that would be Atom, right? And today,
354240	359820	it's kind of set to a single chain. But the real value for validators is that you can run validators
359820	365100	on multiple different chains and tokens and provide the security and the governance and get the
365100	369400	kickbacks from that. But in the future, with some of the scaling efforts and new technology that's
369400	373940	coming out, those validators are going to be able to validate multiple types of coins.
373960	378380	So if I'm a validator that's running a whole bunch of different chains, and I'm validating on that,
378380	385180	my rewards for the day could be Bitcoin, Dogecoin, Litecoin, Die, Tick, whatever I choose to validate
385180	390840	on or stake my tokens with. And that's something that's going to be important that we're going to
390840	396040	talk about a little bit later. But from the validator aspect, like I said, the rewards,
396040	401060	transactions, governance, being able to participate in the system. And the other thing that's important
401060	406020	is the more stake, the more coins or tokens that have been bonded with that validator,
406020	411400	the more power and say that validator has within the system. So it's very important to make sure
411400	416260	that you're keeping up on that and that you're bonding those tokens and contributing back to
416260	424100	the ecosystem in order to be a good neighbor. The other reason validators run this, so while
424100	428900	you're getting block rewards and you're contributing, it's pretty cheap to do this. So this is an
428900	434120	unnamed cloud provider. And this is just some simulated costs. So as you can see for the month,
434120	438640	it's extremely cheap to run these instances or they can be virtualized instances, they could
438640	445360	be bare metal instances. There's a lot of benefits to running up in a cloud provider, which we're
445360	451260	going to jump into. But the overall goal of this is it's extremely cheap to do this from an
451260	457360	infrastructure side, not as necessarily as cheap from the management side and the personnel, the
457360	462640	human cost, but at least getting things spun up, running a validator. You can get this done in an
462640	467960	afternoon if you're the type of person that has some familiarity with cryptocurrency, has
467960	473500	familiarity with the Linux command line, and has some familiarity with best practices and security.
473560	479320	Because you are dealing with things that have a ton of value associated with them.
480460	486200	There's a lot of different networks that currently run validators and do these proof of stake type
486200	491420	of systems. Cosmos here in the middle is the big one that we're focusing on for this talk.
491420	495240	But a lot of the information presented here can be applied out to a lot of these different
495240	501480	networks that run validation systems. Microtip down here in the right is a brand new one that
501480	505540	just came out within the past couple of weeks, they launched their mainnet. And it's one that
505540	511540	I've practiced some validation on. And they're doing some interesting things with price discovery
511540	517340	and stuff of assets. And it's really interesting because you're seeing stuff use some of the Cosmos
517340	521540	SDK, where a lot of these things that we're talking about are going to be applicable.
521660	525860	Other chains are now forking those off, and kind of doing their own things with it,
525860	530760	creating great new products and great new services for end users that we can all jump in and play
530760	536220	with. So if anybody after this talk does have interest in validating, I would definitely go out
536220	540720	to some of these networks, look at their documentation, spin up some virtual instances,
540720	546420	or if you have some spare metal spare hardware, try jumping on their test net and seeing
546420	558820	how this all works out. With Cosmos specifically, we were talking about value. And we can see here
558820	565440	some stats from one of the block explorers for the Cosmos Hub. And right now the atom token,
565440	572940	I think, is at 415 at the time of this presentation. It might be 417 now, it might be 413 in a second
572940	578020	from now. They've got about 3 million blocks on their current chain. And this is really what's
578020	583660	important. So this is showing how many of the atoms are bonded or staked back into the system.
583660	589760	Right now it's at 71%. So 71% of all the atoms that are out there are being staked and held
589760	594720	securely for these validation services. Now, when we take a look at the current price and
594720	601640	we equate that out, that's about $760 million currently that's being bonded behind this network
601640	608080	trying to protect it and prop it up. Another important number here is inflation, which you'll
608080	612000	see at 7%. Remember that one because we're going to talk about it here in a couple slides.
616510	621530	So we now have validators set up in this system that are validating transactions.
621810	628730	And these validators could be run by you, they could be run by an entity, or some other third
628730	635230	party or trust. Anybody can run a validator. But the other thing you can do as a user is you can
635230	641410	stake your own atoms or tokens with a validator. And the reason you would want to do that is to
641410	646630	additionally profit. So this is kind of similar to the whole staking pool concept with mining,
646630	652790	where I don't want to run all the hardware and I don't want to take on the risk. But I do want to
652790	657790	contribute and benefit in some way. And this is a great way that anybody can go and participate
657790	662810	in these networks and contribute to these validators, and also potentially profit and
662810	669330	reward from it at the same time for contributing to that network. There is some things with
669330	674810	validators where they're going to get block rewards. And the reason you would want to risk
674810	680190	your money with a validator is that block reward, there's an inflation component here. And that was
680190	686670	the 7% that we were talking about. So there's an incentive for validators and holders to stake
686670	693390	their tokens over time. You can see here the average yield in this example is 9.46%. So if
693390	701550	you compare that to a 7% inflation rate, you're up to 2.4% at that point. So it still makes sense
701550	705730	to bond tokens into this. And as long as that average stays above the inflation rate on the
705730	712510	network, it's profitable for you to be bonding your tokens over time and getting those interest
713050	717670	tokens back, essentially. So you would think of this as kind of like a savings account or a high
718150	723810	account where you have to manually go back in and re-delegate your tokens to a validator,
723810	728390	pull out your rewards and re-delegate them. And then over time, your reward amount is going to
728390	734750	grow. And just some statistics here. So for 686 atoms monthly, you would be getting a return of
734750	743290	5.41. So at $4, you're looking at $20, $25 for basically just doing passive income at that point.
748020	754220	Now, when it comes to validators, another thing that incents them outside of the inflation
754220	758480	component is slashing. And when we're talking about this, we're not talking about the guy from
758480	765620	Guns N' Roses. We're actually talking about penalizing participants in the network for
765620	771320	violating certain conditions. And this can be in the network, this could be in the consensus layer.
771320	776160	But it's a huge financial incentive for validators to play nice because
776160	780760	anytime you introduce these types of systems, you're going to have hackers, you're going to
780760	785000	have bad actors, you're going to have people trying to gamify it and say, you know, how can
785000	790280	I increase my rewards? How can I knock you offline? How can I trick the system or the state of the
790280	797480	system to go in there and profit for me? What's the incentivization to not do that? And slashing
797480	803160	is a great component for that. And there's two ways that you can really get slashed when you're
803160	811020	talking about Cosmos and proof of stake coins currently. They are downtime and double sign.
811020	816920	Both start with D, pretty easy to remember. Downtime has some conditions where out of the
816920	823880	last 10,000 blocks, if you're down for more than 95% of that, you're risking a downtime slashing
823880	829460	event. You can see here in this block explorer, these blocks were being sent and everything was
829460	835560	great. And then at some point, something went offline and now we're missing them. So if that
835560	842820	continued for the next 9,500 blocks or whatever, and we are not able to get our validator troubleshot
842820	848500	or figure out what's going on or move to another system, we risk incurring a downtime event and
849360	854640	having a slashing event occur. The other big one is a double sign.
854780	859600	And this is an example of a double sign that happened on the Cosmos mainnet where what ended
859600	867160	up happening was the private key for the validator was used to sign two transactions,
867160	872760	sign the votes for them for the same block. And that was pushed out, that was detected.
872760	878440	And as a result, that validator was jailed because you can't double sign on your vote.
879260	884820	So once you're jailed for a double sign, unfortunately, you are in jail. There is
884820	888940	no getting out. Your validator will not be able to rejoin the system.
889660	892460	You've screwed up and you've screwed up in a major way.
895770	902270	When it comes to downtime, the amount of stake that can be slashed is very minimal. It's not
902410	907770	a huge amount, but depending on how many atoms are staked in your validator by yourself as the
907770	914570	validator, what you're doing is called self-staking. So a validator may have a million
914570	919130	atoms and stake their million atoms to say, I believe in my system, I believe in myself,
919130	922750	I'm going to stake all my atoms with it. But as I said, users can also stake.
922930	926930	So when these things, when these slashing events happen, not only is the self-stake affected,
926930	931850	but the user stake can be affected too. So as a validator, if you do have a slashing event and
931850	936830	you lose stake, you're losing money for your end users, tokens, and digital value for your
936830	942410	end users. They're not going to be very happy with you or want to contribute to you in the future.
943630	948330	The penalty for that, you miss your rewards while you're in a... and that goes for a 10 minute
948330	955390	period. And with these types of events, usually you only get hit for the first one. It's not
956250	962010	reoccurring. They put, I think it's called a tombstone. I know there's a tombstone here for
962010	966510	double signing, but I forget what it's called for downtime. I apologize. But it's basically kind of
966510	972630	like a buffer to say that we hit you for downtime. We're not going to hit you again if you're in a
972630	978210	certain period. So if you are being slashed for downtime because your server's down or you're
978210	984470	having some sort of redundancy issue, it's still within your best interest to get that up as soon
984470	989850	as possible because these things are going to start to get costly over time. But there is some
989850	993770	protection there for end users so that it's not just going to consistently stack over time unless
993770	1000310	certain conditions are met. With double signing, it's huge. It's 5%. So if you do have a million
1000310	1006270	dollars in Atom on your validator and taking a 5% slash stake, you guys can figure that out and see
1006270	1011410	how much money that is, right? You also hit an automatic unbonding penalty. If you double sign,
1011410	1017110	and it also incurs a three-week unbonding period where those Atoms cannot be touched while they're
1017110	1023450	being unbonded from the validator. I put the little tombstone here for the RIP because if
1023450	1027770	you double sign, you're done. You're not going to be able to move forward with that validator.
1027770	1034130	You're either going to need to re-delegate all of your stake to another validator or allow your
1034130	1041390	users to unbond all their stake and move forward with somebody else at that point. You're out.
1045660	1050060	Because of all this, you want to make sure you're doing your due diligence when you're either
1050980	1055900	staking with a validator or thinking about running one yourself. These are all points
1055900	1061040	you should be thinking about. So it's recommended from my experience that you want to diversify
1061040	1066740	at minimum probably three is a good bet. I don't want to say best practice. I usually like to stick
1066740	1071960	the three to five different validators for different tokens. The reason being is that
1071960	1076740	we're in a decentralized system and we should be decentralizing as much as possible. I'm also
1076740	1081820	mitigating my risk. So if one of those three or one of those five validators do incur a slashing
1081820	1086660	event or a downtime event for whatever reason, I've mitigated my risks where I'm not losing all
1086660	1093520	of my funds at the same time. You want to do some open source research, some open source intelligence
1093520	1099420	and look up the teams behind them. Look up the history of the validator. So what does their
1099420	1105260	history look like? Is it clean? Do they have a lot of downtime or maybe a misconfiguration where every
1105260	1111920	99 out of 100 blocks they're not signing for some reason? Well, why is nobody noticing that
1111920	1117460	or fixing that? Are they really paying attention to their system? Have they been jailed before? So
1117460	1123960	it is possible to go to jail and get out of jail. As we said, if you double sign, you're done.
1124120	1130540	But for minor events, you can get out of jail if you accept, if you levy the penalty at that point
1130540	1136820	and you wait for the time that you're jailed. So if they have a history of jail occurrences that
1136820	1140800	you're not comfortable with, you shouldn't be giving them your money because this is essentially
1140800	1145720	what you're doing is you're letting them hold digital value on your behalf, essentially.
1146620	1151520	And then the other thing is a lot of teams have gotten really good in the past year with
1152000	1156760	putting a presence out there. So there's people and there's corporations that are doing this as
1156880	1162200	a business, right, where they validate on all the chains I listed earlier and more,
1162200	1166520	because there's a ton of them out there doing their own little niche with different things
1166520	1171440	with proof of stake and their chains and their use case. But some of those teams are getting
1171440	1176340	really good with publishing a social presence. So they're listing their teams on web pages,
1176340	1179700	or they're showing you with chains, or they're actually showing you pictures of
1179700	1185440	here's what our setup looks like, our validator setup looks like. Be careful when you do that,
1185440	1190640	because it's very similar to the ICO craze of 2017, where some of the team pictures,
1190640	1196660	if you go and reverse Google search or Tanai reverse search those, you're going to find that
1196660	1201920	those are some LinkedIn picture for somebody that works in like, you know, aviation or something
1201920	1205960	like that, that has nothing to do, and they're just stealing your information to try to fleece
1205960	1212380	you. So do your due diligence, don't get scammed. The reason you want to do that.
1212380	1221560	So this is a personal example. I was validating on a testnet for a coin that was not released yet.
1221920	1228360	And because it was a testnet, and because I wasn't taking it as seriously as I should have,
1228360	1237800	I ended up using some spare hardware, and ended up experiencing like a 36 hour power outage.
1237800	1244480	This spare hardware was hooked up to a Comcast cable modem, no redundancy across the board.
1245040	1250320	And during the power outage, because I'm focusing on saving my food and a toddler,
1250320	1255160	and what are we going to do, supposed to be 95 degrees tomorrow, and all of those things that
1255160	1262500	you have to deal with when life hits you and kind of shit hits the fan. Those are the things that
1262500	1265680	you're not going to be thinking about your validator, right? You're not going to be thinking
1265680	1269840	about the thing that you've set up in the dressing room that just runs off the modem that
1269840	1278080	your wife doesn't like, or your kid resets all the time. So unfortunately, I'm a victim of this.
1278080	1283000	And thankfully, it was only on testnet, but I ended up losing 10,000 coins because I did not
1283000	1287820	have a redundant infrastructure set up. Now, if that was to occur on mainnet, and I would
1287820	1292620	say a validator, I would have just taken a big PR hit and lost my users funds.
1292620	1297120	In my instance, thankfully, it was only testnet, and it was a great lesson learned.
1298500	1304420	And this is actually the picture of my validator. So this was an Intel NUC that I had a spare
1304420	1309420	machine that I set up, and I said hooked it to a cable modem with a single Ethernet line,
1309420	1316720	single power line, right, single SSD in here somewhere. So no redundancy, no infrastructure,
1316720	1321200	not even really any status lights to tell me what's going on. So if you are thinking about
1321200	1327460	running a validator and you find that the people you're thinking about bonding your tokens with
1327460	1333280	are doing this, you should run away as fast as possible. Great for development and great
1333280	1338400	for testing, should never be done in production. That's what you want to do in production.
1338400	1343040	So we want to see the data center, we want to see the cool blinky lights. The reason you want
1343040	1350220	to do this is you're getting cooling here, you're getting heat and power and network and all the
1350220	1353920	good warm fuzzy feelings that go into your hosting bill when you have to pay that at the end of the
1353920	1360440	month to justify all the servers and things that are being filled in these racks. So you know,
1360440	1364840	stand on the shoulder of giants, use the experts, use the redundant infrastructures that have
1364840	1369520	already been built and hopefully been tested correctly to host these types of things and
1369520	1378810	do validation. We want to implement all like the standard best practices.
1379830	1383330	When it comes to redundancy, you want to hit all the...
1385290	1389290	Monitoring, we're going to jump into with a couple different solutions that can actually
1389290	1395890	alert you when you do have the 36-hour power outage. There's a lot of administration concerns
1395890	1402930	too with different configurations and things like Cosmos has published a security update for
1402930	1409510	its SDK and it's an emergency upgrade. How are we going to do that, right? Do you test the data
1409510	1414890	center tech with going in and upgrading that binary? Or do you have a subject matter expert
1414890	1420070	on staff that can handle that? Do they know the difference between go path and your environment
1420070	1424810	variable path and all of those types of things? Because all of those are considerations that
1424810	1429330	you're going to hit and speed bumps that are going to occur while you're working on these
1429330	1435110	types of systems. You need something to do some sort of repo monitoring, be able to check when
1435110	1439970	upgrades and new checkpoints and releases are released. And then deployment is another big
1439970	1449230	consideration for administration. There's a lot of different schools of thought on deployment.
1449450	1456050	And we'll jump into some of the use cases as to why some are better than the others.
1456050	1461450	But it's definitely a concern when you're architecting and administering these types of systems.
1461550	1464750	And then we want to jump into a lot of the security components with
1464750	1468890	controls and the different aspects on how you're mitigating your risk and how you're
1468890	1473330	delegating access and who knows what and why do they need to know it.
1477510	1482850	The keys that actually work on a validator when we're working on security, there's two keys specifically for
1482850	1487070	Cosmos that need to be protected and thought about with all those prior best practices
1487070	1492970	for redundancy and backups and things of that nature. And that's your tenderment key, which is
1492970	1498570	actually doing your consensus voting. And then the application key, which derives your validator
1498570	1503770	addresses and things. That signs your actual transactions. But both of those need to be
1503770	1509810	considered when you're building these systems on, one, where does it live? Two, how did we generate
1509810	1515810	those? Did we do it in a secure way? And I'll show you some ways that you guys can do that.
1515810	1523970	Three, where are we backing it up? So does it just exist on some EC2 instance somewhere? Or do I have
1523970	1531910	key material written down in a waterproof, fireproof safe that's access control? A lot of
1531910	1536030	that stuff needs to go into the thinking and mindset and paradigm of how you're protecting
1536030	1540550	the system and how you're doing it. And you really want to do it right kind of from the beginning,
1540550	1547090	because if you deploy a validator to determine that I didn't generate my key material securely,
1547090	1552210	you don't want to have to go back and redo all that stuff and re-delegate and try to make sure
1552210	1556110	that people are aware that, you know, we're validator two instead of validator one now,
1556110	1561170	those types of things. So when you're building this stuff, kind of encompass all this in your
1561170	1565770	thinking and do it right from the beginning. Because really what these keys are protecting,
1565770	1571290	I checked CoinCap IO right before this presentation, 1.3 billion market cap for
1571290	1578090	Cosmos Atoms right now. So for a coin that's still relatively brand new in the scheme of things,
1579090	1584170	we're already at a billion dollar market cap at current day prices. So if the bull market really,
1584170	1590410	really starts taking off, at that point, these keys are protecting crazy amounts of value. Are
1590410	1596570	you comfortable leaving that type of information sitting on a server that, you know, the data
1596570	1601510	center tech down the street that happened to work on the day that you called in and said,
1601510	1605990	my server's down, can you check the lights on my Dell system? You know, he's looking at console
1605990	1612550	and he's single usering and things. While your key material may be protected by local key stores
1612550	1617390	and things like that, what stops him from sniffing memory? What stops him from imaging the drive or
1617390	1622310	breaking RAID or things like that and then clearing logs? So it's considerations that you
1622310	1628450	want to think about when you're building those systems. For redundancy, it's all the great stuff
1628450	1635590	with power supplies and phasing and networking. And I've seen some validators where they run on
1635710	1641390	a single networking. So what happens if that network card fails or the cable goes bad,
1641710	1646670	a pair in that cable goes bad? You're going to be at risk of a potential downtime event.
1646670	1652990	When it comes to storage, there is a lot of data that goes into running a validator. A validator
1652990	1659870	essentially needs on itself a copy of the full node, the chain data, or it needs to link to
1659870	1666930	another server a century that has a copy of that full node chain data. So if you're building in
1666930	1672670	redundancy to say if this instance goes down, once you rebuild your new instance with your new
1672670	1678290	validator binaries, you need to sync all the chain data again. So if you're in that window,
1678290	1683430	that downtime window, that 95% of 10,000 blocks, which I think is like half a day,
1683430	1688170	and it's starting to get down to the wire, I've seen where we've started binaries and things like
1688170	1693150	that and my validators, you know, they still need two hours to catch up to sync chain data
1693150	1697610	because I didn't have a copy of it anyway. So I would definitely recommend looking at things like
1697610	1704650	RAID if you're using SANs, doing snapshots and things of that nature. The Cosmos team has
1704650	1710350	published snapshots over time of chain data where you can go and download those types of things to
1710350	1715610	sync it, but I'd be careful with what sources you're pulling that from because if a bad actor
1715610	1720250	wants to cede bad data to you, you could potentially risk another issue with your validator at that
1720250	1724810	point. And then there's other some some other gotchas that people don't necessarily think about
1724810	1729290	and like, there's some funny stories and some instances, you're standing in front of the
1729290	1736630	server in the data center, and you're going, I'm here. I have no way to log in, there's no keyboard,
1736630	1741190	right? There's no crash cart available in the data center. So run through some of your BCDR
1741190	1746650	type of policies with like, what do we need in the cage, right? Do we need spare hard drives? Do we
1746650	1752590	need installation software for OSs if we have to spin it up? Do we need config variables to point
1752590	1757450	to a PXE server or to point to a Kubernetes cluster? However you decide to architect your
1757450	1763650	solution that best works for your information system. And I note down here, any time that you're
1763650	1768330	running through this type of stuff and architecting it and running through these exercises, you always
1768330	1773430	want to think about the worst case scenario, which is double signing. And in order to achieve that
1773430	1779230	double sign, it means that those Cosmos keys need to be either duplicated somewhere or pulled into
1779370	1784610	a configuration or have two instances that are running that same material. So always think about
1784610	1789710	that. That's kind of the grenade in the mix that needs to always move forward and can never be
1789710	1799770	copied because if it is and it goes live, you're risking that 5%. I put this slide in here just
1800270	1807170	because I hate DNS. DNS is a big one. And you want to be careful with this because when you
1807170	1811970	start to build validator infrastructure and you start to do additional security controls and
1811970	1818550	different servers with copies of chain data, DNS is one that may get you that you don't even realize
1818550	1823790	because nobody ever thinks it's DNS, but it always is. So make sure your host names are correct.
1823790	1830050	Make sure you have redundancy upstream in your network stack, not just the cables, even the
1830050	1836870	routers in the rack. If you're using things like Netgear or Cisco equipment, do you have that in
1836990	1843070	a stack configuration? Do you have an AB failover? If your switch or your Cisco ASA license expires
1843070	1848510	and you start getting throttling or your feature stops off, what happens into your system? So those
1848510	1852750	are all things to simulate and think about when you're building that. And DNS is one of the ones
1852750	1862120	that I don't want people to forget about. So now we need to monitor all this stuff with our
1862660	1868600	validators. We've figured out how it works. We've got some coins. We've got a system set up.
1868600	1875640	Now, how do we avoid that 36-hour outage? So SimpliVC is a company that's out there that
1875640	1880400	does a ton of validation on chains and things like that. And they release this open source software
1880400	1885780	called Panic. And I'm a big, big fan of open source software. But what this will do is this
1885780	1891900	will actually monitor things from the tendermint layer and from the local server layer. And it'll
1891900	1899440	do email, telegram, through Twilio integrations, it's not responding. And it'll also do the repo
1899440	1904940	monitoring we talked about earlier. You know, a new version of Cosmos, okay, or Gaia-D or whatever
1904940	1909560	binary has been released, you need to be aware of that so that you can go check the release notes
1909560	1916040	and see if this is something I need to apply now or can I defer that to later? I love it. I use it
1916040	1921840	for all my validation stuff. This is my telegram integration. And you can see it doesn't just
1921900	1928700	do things like the services down. It actually tells things like my peers have increased or the
1928700	1934400	node was inaccessible. If people delegate to my validators, it gets updates on my voting power.
1934400	1938420	So I know that I've got more power in the system for consensus votes or that there's something
1938420	1942840	happening. And I need to be looking at my validator because people are delegating to me.
1942840	1947160	I want to know why. And then as I said, it gives you peering changes. So if other people are doing
1947900	1952560	peers dropping off the network, that may be indicative of a network event or a binary upgrade
1952560	1956340	or something else is going on with those other servers. They're doing some work to that stuff.
1956340	1960720	And it's good to be aware because we may need to take mitigation efforts.
1963450	1968170	Prometheus is another great one. This is all packaged with the current Gaia installation
1968170	1974490	with Cosmos. It's a very simple config change. And this is basically kind of like a logging port
1974490	1980630	that opens up a whole bunch of data that's going on in your validator that you can feed into Elk
1980630	1985910	stack or different dashboards or whatever you choose. But it has really good support for Grafana,
1985910	1992330	for example. And this brings it into the kind of DevOps infrastructure world of,
1992330	1996630	I want to build dashboards and I want cool graphs and stats and stuff like that.
1996850	2001710	So if you don't want to do that through necessarily a block explorer, if you want to run something
2001710	2006550	locally, these things are, it's great to turn Prometheus on. It's great to throw Grafana or
2006550	2015020	some sort of Elk stack monitoring on top. So we've got our monitoring taken care of and now we're...
2017180	2024680	So we talked about the data center tech at, let's use AWS, for example. So you're running
2024820	2029980	a validator in AWS and you've got this private key material on there that's validating transactions
2029980	2035480	and participating. What stops the data center tech from logging into your system and grabbing
2035480	2039540	your key material? What stops the hacker from getting on your server and grabbing that stuff
2039540	2044900	remotely if you haven't configured your security groups and your firewalls correctly? So that's
2044900	2050340	the problem and the solution is HSMs. And HSMs are great if you guys have never used them. They're a
2050340	2055080	little bit expensive, but it's very similar to the way that, you know, YubiKeys and things like that
2055080	2060940	where it stores the private key material actually on a hardware device and then signs messages
2060940	2066540	without actually having to expose or reveal that material. In theory, it's not supposed to allow
2066540	2071560	any private key extraction. Now there have been attacks and that's not to say that things can't
2071560	2077180	fail in the future, but currently it's not supposed to allow any private key extraction.
2077420	2082840	So what you would do is when you were generating key material to either set up your validator or
2082840	2088320	your wallet or functioning, you want to generate those keys in a secure way and then store that
2088320	2094140	private key material on a hardware device. The two I've used is the YubiHSM2 and the Ledger Nano S,
2094140	2099980	which both work very well with Cosmos. I know they work with MicroTik and they work with a few other
2099980	2105460	other chains that are out there. But this would actually physically sit into your bare metal
2105460	2112100	server in the data center, in my case, in the dressing room and provide the private key material
2112100	2117100	and signing functions without revealing it. So if somebody was to get into your server remotely,
2117100	2121500	or if the data center texting their single user in, they're not going to be able to extract this
2121500	2126940	without a whole bunch of different authentication information and, you know, secrets and things that
2126940	2132320	they don't have. And then the actual physical hardware security capability of the device too.
2133180	2140120	One thing I will note to be very careful with this stuff is that the YubiHSM in particular
2140120	2144700	has a very small profile and in order to firmware reset this device, I think it's like hold this
2144700	2149600	front side for 10 seconds or something. So if you're in a data center working and you're
2150560	2156300	failing over from server A to server B because you're doing maintenance and you pop your HSM out,
2156300	2159940	be careful how long you're holding it and where you're touching it on. Because if you wipe that
2159940	2164680	private key material, that's not going to be recoverable and you better hope to hell you have
2164780	2169580	a backup. The Ledger works really great with it too and it's got the little graphical display and
2170120	2174400	same thing, you know, there's always security vulnerabilities out for hardware wallets but
2174400	2179020	people are keeping up on those and they're auditing them and it's still a better layer
2179020	2190020	of defense than just leaving the key material on the server itself. Another thing you want to do
2190020	2196740	from a security aspect is you want to set up Sentry architecture. And a Sentry is basically a
2196740	2203260	copy, a full node of the chain that you're validating on. So you could run your validator
2203260	2208520	straight out to the internet and start broadcasting these things. But it's better
2208520	2215160	to put it behind another layer of obfuscation and network connectivity here. And what we're doing
2215160	2222760	is we've connected public internet that was gossiping a lot of information about our transaction
2222760	2228600	validator itself and we've moved it up a layer behind a Sentry layer. And when we do that,
2228600	2233380	this actually enables us to now load balance and have some redundancy at the Sentry layer
2233380	2237860	where we can deploy multiple Sentries here. So I've used two in this example, but it can really
2237860	2243480	scale. I don't know if there's any technical limits on that. That would be a great follow-up.
2243480	2247640	But as far as I'm aware, I haven't seen a technical limit on how many Sentries you can run.
2248140	2253020	But the whole point of this is that nobody's going to be able to get to your validator directly to
2253020	2257560	any public IPs or anything like that. You're going to have private communication between your
2257560	2264380	validator and your Sentry itself. And I've seen this done many different ways where this validator
2264380	2269740	could be inside its own VPC and you have IPsec tunnels talking out to your Sentries, which are
2269740	2275860	in its own VPC that consists of two or three instances. This could just be two or three EC2
2275860	2282860	instances that are talking to each other through private IP space. This validator may not even be
2282860	2287780	the same location or site. It could be bare metal talking up to cloud Sentries somewhere,
2287780	2293160	bare metal Sentries in another data center across the country. I wouldn't recommend that for time
2293160	2299800	to live and packet communication, but it is technically possible. This stops people from
2299800	2304860	denial of servicing your validator. It also gives you some redundancy here for failures with your
2304860	2309080	different Sentries and things like that. And as I said, it protects your communications between
2309080	2314260	your validator and your Sentry. So the validator would get the information from Sentry, sign
2314260	2320480	transactions, broadcast it out through the Sentry out to the public internet. So from an attacker
2320480	2324860	standpoint, they would only get as far as this layer. They wouldn't get to the stuff that actually
2324860	2336280	matters. Other considerations with your security architecture. We never want to double sign.
2337420	2343060	A lot of cloud providers offer auto scaling, and I think that's great for Sentries, but I would
2343060	2349560	avoid it for my validator infrastructure, because you start to get into instances of where does my
2349560	2355460	key material live and how do I migrate? And if server A still has it active and server B spins
2355460	2360360	up and I've copied my key material and that service starts, I've now double signed, right? And I'm in
2360360	2367460	it's game over. Any automation and orchestration pieces with the Sentry, you want to think about
2367460	2372720	the blockchain data and the state data. How are you moving that? How are you verifying that?
2372720	2376640	How are you loading that into new instances if you're doing auto scaling or automation?
2376760	2381520	Or do your new Sentries have to come up and sync the entire chain before they can be active?
2382180	2386560	You also need to worry about doing your config updates. So if I had to have a validator
2386560	2391300	infrastructure and I have two Sentries and add a third, well now I need to go back to all of my
2391300	2397280	existing systems and do some config updates so that I'm now listing that here in my config to say
2397280	2402500	talk to Sentry 3. Don't just talk to Sentry 1 and 2. So when you're building the automation,
2402500	2406440	if you have Sentries and things in that infrastructure, you need to kind of backfill
2406440	2411760	all of those changes to stuff that you've already implemented. And then Sentries are just like
2411760	2416260	validators because it's a full node. If there's security updates for the binary,
2416260	2420280	if there's issues with it where it goes down, you need to be doing all the same
2420280	2426060	all the same repo stuff, all the same binary upgrades when it comes to those Sentries. So
2426060	2431540	if you're scaling your Sentry architecture out horizontally to 40 Sentries, just realize you
2431540	2439010	need to do updates to all of those Sentries when things happen. Then there's some final
2439010	2446150	other considerations here. As I said earlier, there's a lot of different mindsets on should
2446150	2455410	you virtualize validators? And I've gone back and forth on this and I feel like validators should
2455410	2460670	be bare metal. And while it is technically possible to deploy like an EC2 instance or
2460670	2466690	Google's GCP instance and have it validate with key material on that validator, that's not going
2466690	2471670	to be the most secure, optimized, redundant solution that's out there. And if I was considering
2471670	2477170	and looking at two validators on where I wanted the funds, I would not pick that one. I would
2477170	2480670	go with the person that considered some of this stuff. And even if they were using one or two
2480670	2486210	bare metal servers in like a hot or cold fail, to me, I'm a little bit more comfortable with that
2486210	2491250	than somebody that's doing pods of validators that's moving key material back and forth,
2491250	2495550	because all that needs to happen is a weird edge case or anomaly where you're going to hit some
2495550	2501450	sort of double signing scenario, right? And unless you have your deploy scripts tweaked so good with
2501450	2507170	so many tests and error catching and then, yeah, it's scary when you look at the amount of money
2507170	2513810	with that 5% double slash. There's other things you want to consider with minimum config values.
2513810	2519590	So I've done some attacks and I've seen some defense on sentries and things with spam dust
2519590	2524870	transactions where people can try to tax your validator, your sentry resources. And the reason
2524870	2530150	they would do that is to try to edge because maybe they run their own validator or you were
2530150	2535930	jailed and they had money with you or tokens with you, some sort of value. So there's some minimum
2535930	2540990	config values that validators can tweak and sentries can tweak to say, ignore more of the spam
2540990	2546150	and only give me the good stuff. There's an art to it and you got to be careful
2546150	2550510	because you want to be a good system. But just know that you can get pretty granular in your
2550510	2554950	validator config with all the different values that are participating at the tendermint layer
2554950	2560610	and the consensus layer. And that goes for denial service too. And then this last one here that I
2560610	2565870	don't think a lot of people... and I've actually seen a lot of wallet software and validator
2565870	2572970	software getting better with this. People get in the habit of validating and starting to get rewards
2572970	2579510	and they'll pull out all of their rewards and say, re-delegate every single token I have,
2579510	2584510	all my rewards in my wallet, send it back to the validator because I want that compounding interest.
2585310	2589050	And what'll end up happening is it'll work. They'll still sign the transaction and send it
2589050	2593910	out to the validator. And then they literally have zero in their wallet. So you guys, you know,
2593910	2598730	you're familiar with ETH and other systems. If you send all your ETH out and you try to do an ERC20
2598730	2602290	transfer, what's it going to do? It's going to complain and say, you don't have enough ETH for
2602290	2606730	the gas for the transaction. So be careful with that because if you re-delegate all your rewards
2607350	2611690	thing that's in that wallet, you're not going to have enough gas to do any more transactions at
2611690	2615350	that point in the future. So you're going to have to go procure atoms from somebody else or wait
2615350	2619530	until you have enough rewards to get enough to be able to start to continue to do transactions.
2619530	2625110	So I've gotten in the habit of like leaving half an atom or an atom or whatever denomination or
2625110	2628870	whatever token I'm using, leaving one or two of those in the wallet, just so I know that I'm
2628870	2632790	always going to be covered for transactions in the future. And that stuff's getting way better
2632790	2638410	with doing calculations and fees and even notifying users. Cosmos station is a mobile
2638410	2642290	wallet that I love. And it'll come up and actually warn you and say like, hey, you shouldn't be
2642290	2647470	delegating max, like you should be delegating 90% of max or whatever you want to do.
2649410	2652750	So with all those things we talked about with redundancy,
2652750	2656270	I wanted to ask the question, should you be running a validator?
2658950	2666050	And in my opinion, no, not for the validator. If centuries and things like that, where it's
2666050	2672070	not actually signing transactions or dealing with value, that is great for scaling up and
2672070	2675850	auto scaling. And as long as you can take some of the data considerations and things I was talking
2675850	2682570	about earlier into that system, it's amazing for centuries. I fully recommend GCP, Amazon,
2682570	2688210	DigitalOcean. It takes the onus off you and it gives you all of those redundancy and good,
2688210	2691950	warm fuzzies. But for the validator, I feel like that's something that you should be controlling
2692150	2697750	a little bit more manually doing your own internal orchestration and things like that.
2698050	2702050	You want to be careful about where you're hosting that, who has access to it,
2702050	2708390	what is their security posture? What is their uptime in their SLA? If their SLA is more than a
2708390	2712450	day, that's not good for a validator because if you have a downtime event, you could be slashed.
2712570	2716470	All things to think about when you're negotiating with your service providers.
2717230	2722330	The last thing for best practices I wanted to talk about was the cryptocurrency standard. And this
2722330	2727690	was more in line with what I talked about last year here at the Blockchain Village. But it's
2727690	2733210	basically a standard that's amazing that was presented by C4 that dives into a lot of different
2733210	2739050	security controls and aspects that one can implement in a cryptocurrency information system.
2739990	2745270	It gives you the guidance, it gives you a control and different levels from level one to three to
2745270	2750690	say, here's the very basic control I can put into place versus level three. Here's the ultimate
2750690	2757170	paranoid control and kind of give you an example for that. One would be the key generation. So we
2757170	2762510	were talking about keys earlier with the keys and which ones were both for consensus and which ones
2762510	2767050	do transactions. And one of the things of the cryptocurrency security standard is how are those
2767050	2773210	generated? What was the device that you generated them on? What was the entropy device? Did you do
2773210	2778010	it on your network connected computer that you do work and Netflix and torrenting and all that stuff
2778010	2783850	on? Or did you do it on an air-gapped machine with a secure clean room type of setup? Did you back it
2783850	2789310	up correctly? So level one may say, I made a backup of the key, I'm good right now. It's on paper
2789310	2795810	somewhere. But level three would say, that backup needs to be sealed, it needs to be fireproof,
2795810	2800670	it needs to be waterproof, environmental proof, it needs to be access controlled. So as you kind
2800670	2804010	of go up the stack here, there's different requirements and different levels of security
2804010	2809070	and paranoia. And while you may not be able to use the cryptocurrency security standard
2809070	2816410	to certify your information system, depending on how it works or what you're doing, it's a great
2816410	2820990	framework to check your validator against for different levels of compliance. And it also gives
2820990	2827390	you some really good ideas for things like data sanitization policy. So if you are spinning
2827390	2832810	instances up and down, where are your logs going? Do you have a process for clearing all of that
2832810	2837790	stuff correctly and securely so that nobody can run forensics on it if they wanted to profile you
2837790	2842230	or get more information? So there's a lot of great things in here that I think definitely
2842230	2846350	apply to that information system, even if it's not every single one of them.
2848050	2854550	So what's next for the future? And this is really kind of exciting for me and validators and Cosmos.
2854690	2861350	IBC is right around the corner. And I just took place in Game of Zones, which was a attack,
2861350	2868970	defend scaling competition that the Interchain team and Zacky Mannion and Jack Sampling
2870870	2875910	threw. So we had a whole bunch of different validators and people that were taking place
2875910	2881030	in this. And as you can see in this graph here, these were all the participants. And they were
2881030	2886130	connected to a main hub, a main Cosmos hub. So this is very much indicative of the wheel and
2886130	2891690	spoke infrastructure that I think everybody's kind of used to. But this was to theoretically
2891690	2896170	prove that all of these people could connect into the main hub and get them prepared for the next
2896170	2901730	thing, which is the IBC protocol. And when we're talking about the IBC protocol, we're talking
2901730	2907930	about validating on chains that we're not running necessarily. All we're doing is syncing state with
2907930	2915010	them through light clients. So that if Alex chain, or I'm sorry, if Bob chain, for example, wanted to
2915010	2920470	hold Alice coin, Bob and Alice's two chains can now talk together through a light client through
2920470	2926130	IBC. So now we're breaking away from that hub and spoke model. And our validators can now choose who
2926130	2930530	they want to talk to and where they want to validate. So you're seeing here the very beginning
2930530	2935230	of this, where different zones are kind of breaking off into their own sovereign nations
2935630	2939570	and saying, I want to validate on this chain, or I want to participate on this chain in addition to
2939570	2944170	my own. And this is the magic where I was talking about earlier for validators, where when they go
2944170	2948470	to do the rewards, they could be validating on Bitcoin, they could be validating Litecoin,
2948470	2953410	they could be validating Ethereum and Adams, as well as Ron, Bob, Alice and Jim coin, because
2953410	2958750	they see some value there. And when you start to open up that ecosystem and that level of connectivity,
2958750	2962630	there's some really, really cool applications and cases and things that
2962630	2966010	people are talking about and playing with. And it gets me really excited.
2967590	2970810	So that's what I have for my talk. If you guys do have any questions,
2970810	2974630	feel free to reach out to me at my consulting company, Stoner Consulting.
2974990	2980010	I love talking about this stuff. I'd love to go into detail if anybody wants to jump more into it.
2980010	2984590	And I also wanted to throw a special thanks out to Sean Martin, one of the people that's worked
2984590	2991370	with me on validators and infrastructure and system. And he's really an infrastructure magician.
2991370	2995830	So if you guys ever have infrastructure questions or scaling issues or anything like that,
2995830	2998270	please let me know and I will get you in contact with Sean.
2998730	3001190	I'm Ron Stoner and thank you for attending my talk.
3050840	3052320	I don't know if he can, but I can.
3057110	3060970	Sorry, I was looking to see if there was any questions. So I did see one in there,
3060970	3066030	correction, yes, it is called a liveness check. That is correct. And that was in regards to the
3066930	3070630	violations, the consensus violation and the liveness violation. So downtime.
3070670	3073550	So that is the correct terminology for it. Thank you.
3080790	3084210	Cool. And I don't think I see any other questions unless anyone has any.
3115140	3118060	I would love to. Yeah. Thank you, Jay. Thank you, Nathan.
3118060	3122180	Thank you, everybody that put on this village and promoting blockchain. I love it.