[00:00.000 --> 00:07.000] Well, thank you, Ajit. I want to welcome everybody to DEF CON and the Blockchain Village. [00:07.000 --> 00:11.300] My name is Ron Stoner. I'm going to talk about securing the cosmos, and I appreciate that [00:11.300 --> 00:19.600] wonderful introduction. I'm a repeat speaker here. I spoke last year when we were talking about the [00:19.600 --> 00:24.200] cryptocurrency security standard with Michael Parkland, and it was an amazing experience, [00:24.200 --> 00:29.680] so I'm happy to be back. I will preface this talk by saying I'm only on a few hours of sleep [00:29.680 --> 00:33.860] in traditional DEF CON fashion, a little bit hungover, and I was working on my presentation [00:33.860 --> 00:38.700] up until a few minutes ago. So if there's any inconsistencies in here, or you guys see [00:38.700 --> 00:43.040] formatting issues, I'm more than happy to clean that up if you want to copy the presentation, [00:43.040 --> 00:47.660] or if you want to discuss further just to clear up any ambiguity after the stream. [00:49.000 --> 00:56.080] So who am I? I'm a senior security engineer for ShapeShift.com, one of the best digital [00:56.080 --> 01:00.840] asset exchanges out there. And as a result, I get exposed to a lot of different technologies, [01:00.840 --> 01:04.280] and I get to play with a lot of cool different things in the crypto space, [01:04.280 --> 01:10.960] including Cosmos, the Atom token, and some cool new validation software that people are [01:10.960 --> 01:17.560] releasing and trying out. I'm also the curator of the cryptocurrency security standard auditor exam, [01:17.560 --> 01:22.800] so that's an exam regarding the cryptocurrency security standard that I referenced earlier. [01:22.800 --> 01:29.340] I have my own consulting firm where I consult on security and blockchain technology. And then [01:29.340 --> 01:33.600] if you guys want to reach me on Twitter, you can hit me up at forwardsecrecy.com. [01:33.700 --> 01:37.580] The reason I'm giving this talk to you guys today is because I do have a lot of experience [01:37.580 --> 01:45.780] on testnet and mainnet with deploying, configuring, administering, and securing validators [01:45.780 --> 01:54.170] and validator infrastructure. I wanted to set up a goal for this talk. So you guys can read it there, [01:54.170 --> 01:59.810] I'm not going to go through that. I do want to highlight my redundant joke here. Hopefully you [01:59.810 --> 02:06.870] guys like that one. If not, oh well. But my goal for this talk is to have you guys walk away here [02:06.870 --> 02:12.590] learning something. There's a lot of smart people here at DEF CON in the blockchain village. [02:13.130 --> 02:18.030] But while we may be good at infrastructure, or we may be good at security, or we may be really [02:18.030 --> 02:23.810] good at the blockchain aspects of it, nobody's perfect, right? Nobody's an expert in all areas. [02:23.810 --> 02:28.970] Hopefully you walk away from this talk taking a tip or some piece of knowledge that you can [02:29.490 --> 02:32.610] run with yourself or implement in your own system. [02:37.100 --> 02:46.580] So what are validators? Everybody's kind of familiar with the concept of Bitcoin and SHA-256 [02:46.580 --> 02:52.920] and mining, right? So when you're mining, you're throwing a ton of resources at cryptographic [02:52.920 --> 02:59.780] problems to solve them and write data to a blockchain. And mining was awesome. It was very [02:59.780 --> 03:04.460] profitable for people for a long time. It helped secure the network. But there's a lot of downside [03:04.460 --> 03:11.060] to miners that people have brought up over time, including energy consumption and administration [03:11.060 --> 03:18.300] and things like I can't get the newest graphics card to play PUBG or Fortnite because there's [03:18.300 --> 03:23.840] run on them, right? There are five, $600 now because everyone's buying them up and mining. [03:24.480 --> 03:29.460] So some new coins and chains came out that implemented what was called proof of stake. [03:29.540 --> 03:34.720] And unfortunately, it's not the stake that I enjoy every night. I would love a proof of actual [03:34.720 --> 03:40.420] like a one type of stake. But this proof of stake is where we're actually staking and bonding coins [03:40.420 --> 03:47.500] and doing our network through validators. So those validators actually provide a whole bunch [03:47.500 --> 03:52.840] of energy savings, and they can achieve kind of the same end result. They're able to validate [03:52.840 --> 03:58.960] transactions, validate blocks, provide signatures, create chain data and do it in a way where it's [03:58.960 --> 04:04.740] more beneficial to the environment and it's lower cost and overhead. The way it works, as I said, [04:04.740 --> 04:11.440] people end up bonding or locking or staking the coins with the validator. And then for this talk, [04:11.440 --> 04:15.820] we're going to be focusing a little bit more on Cosmos. But with things like ETH2, [04:15.820 --> 04:20.260] they have some different systems where they kind of bet on blocks and transactions and things for [04:20.260 --> 04:25.100] the validators to process. But on the back end, what it's really doing is it's taking cryptographic [04:25.100 --> 04:36.820] signatures and signing and broadcasting votes out to the chain. The incentive for the validator, [04:36.820 --> 04:43.060] you can see right here is money, basically, digital value. So as a validator, you're validating all [04:43.060 --> 04:47.360] these transactions, and you're running your infrastructure and you're putting in all the [04:47.360 --> 04:52.380] time and effort. And what's the incentive to go and do this? So there's a few different pieces. [04:52.380 --> 04:58.000] The first one's the security aspect. So you're providing back to the security of the chain, [04:58.000 --> 05:02.720] the decentralization aspect. So you're participating in a decentralized system. [05:02.720 --> 05:09.480] There's a governance aspect where as a validator, if you meet certain conditions based off of the [05:09.480 --> 05:14.300] chain and governance that's been set, you can actually participate in governance votes and [05:14.300 --> 05:20.400] help to modify and make changes to that chain and its different variables moving forward. [05:20.760 --> 05:25.320] But there's also block rewards and transaction fees. And that's kind of where the money comes in [05:25.320 --> 05:31.620] at. And this is just a very small example. So I think this is one day of rewards for a validator [05:31.620 --> 05:36.560] that I was running on a new chain that was just released. And you can see here, I'm getting some [05:36.560 --> 05:43.280] die and I'm getting some tick. And this validator, while it's using tick as its denomination and also [05:43.280 --> 05:48.800] paying some commissions in die, this could be whatever coin or whatever chain that you're [05:48.800 --> 05:54.240] running your validator for. So in the instance of Cosmos, that would be Atom, right? And today, [05:54.240 --> 05:59.820] it's kind of set to a single chain. But the real value for validators is that you can run validators [05:59.820 --> 06:05.100] on multiple different chains and tokens and provide the security and the governance and get the [06:05.100 --> 06:09.400] kickbacks from that. But in the future, with some of the scaling efforts and new technology that's [06:09.400 --> 06:13.940] coming out, those validators are going to be able to validate multiple types of coins. [06:13.960 --> 06:18.380] So if I'm a validator that's running a whole bunch of different chains, and I'm validating on that, [06:18.380 --> 06:25.180] my rewards for the day could be Bitcoin, Dogecoin, Litecoin, Die, Tick, whatever I choose to validate [06:25.180 --> 06:30.840] on or stake my tokens with. And that's something that's going to be important that we're going to [06:30.840 --> 06:36.040] talk about a little bit later. But from the validator aspect, like I said, the rewards, [06:36.040 --> 06:41.060] transactions, governance, being able to participate in the system. And the other thing that's important [06:41.060 --> 06:46.020] is the more stake, the more coins or tokens that have been bonded with that validator, [06:46.020 --> 06:51.400] the more power and say that validator has within the system. So it's very important to make sure [06:51.400 --> 06:56.260] that you're keeping up on that and that you're bonding those tokens and contributing back to [06:56.260 --> 07:04.100] the ecosystem in order to be a good neighbor. The other reason validators run this, so while [07:04.100 --> 07:08.900] you're getting block rewards and you're contributing, it's pretty cheap to do this. So this is an [07:08.900 --> 07:14.120] unnamed cloud provider. And this is just some simulated costs. So as you can see for the month, [07:14.120 --> 07:18.640] it's extremely cheap to run these instances or they can be virtualized instances, they could [07:18.640 --> 07:25.360] be bare metal instances. There's a lot of benefits to running up in a cloud provider, which we're [07:25.360 --> 07:31.260] going to jump into. But the overall goal of this is it's extremely cheap to do this from an [07:31.260 --> 07:37.360] infrastructure side, not as necessarily as cheap from the management side and the personnel, the [07:37.360 --> 07:42.640] human cost, but at least getting things spun up, running a validator. You can get this done in an [07:42.640 --> 07:47.960] afternoon if you're the type of person that has some familiarity with cryptocurrency, has [07:47.960 --> 07:53.500] familiarity with the Linux command line, and has some familiarity with best practices and security. [07:53.560 --> 07:59.320] Because you are dealing with things that have a ton of value associated with them. [08:00.460 --> 08:06.200] There's a lot of different networks that currently run validators and do these proof of stake type [08:06.200 --> 08:11.420] of systems. Cosmos here in the middle is the big one that we're focusing on for this talk. [08:11.420 --> 08:15.240] But a lot of the information presented here can be applied out to a lot of these different [08:15.240 --> 08:21.480] networks that run validation systems. Microtip down here in the right is a brand new one that [08:21.480 --> 08:25.540] just came out within the past couple of weeks, they launched their mainnet. And it's one that [08:25.540 --> 08:31.540] I've practiced some validation on. And they're doing some interesting things with price discovery [08:31.540 --> 08:37.340] and stuff of assets. And it's really interesting because you're seeing stuff use some of the Cosmos [08:37.340 --> 08:41.540] SDK, where a lot of these things that we're talking about are going to be applicable. [08:41.660 --> 08:45.860] Other chains are now forking those off, and kind of doing their own things with it, [08:45.860 --> 08:50.760] creating great new products and great new services for end users that we can all jump in and play [08:50.760 --> 08:56.220] with. So if anybody after this talk does have interest in validating, I would definitely go out [08:56.220 --> 09:00.720] to some of these networks, look at their documentation, spin up some virtual instances, [09:00.720 --> 09:06.420] or if you have some spare metal spare hardware, try jumping on their test net and seeing [09:06.420 --> 09:18.820] how this all works out. With Cosmos specifically, we were talking about value. And we can see here [09:18.820 --> 09:25.440] some stats from one of the block explorers for the Cosmos Hub. And right now the atom token, [09:25.440 --> 09:32.940] I think, is at 415 at the time of this presentation. It might be 417 now, it might be 413 in a second [09:32.940 --> 09:38.020] from now. They've got about 3 million blocks on their current chain. And this is really what's [09:38.020 --> 09:43.660] important. So this is showing how many of the atoms are bonded or staked back into the system. [09:43.660 --> 09:49.760] Right now it's at 71%. So 71% of all the atoms that are out there are being staked and held [09:49.760 --> 09:54.720] securely for these validation services. Now, when we take a look at the current price and [09:54.720 --> 10:01.640] we equate that out, that's about $760 million currently that's being bonded behind this network [10:01.640 --> 10:08.080] trying to protect it and prop it up. Another important number here is inflation, which you'll [10:08.080 --> 10:12.000] see at 7%. Remember that one because we're going to talk about it here in a couple slides. [10:16.510 --> 10:21.530] So we now have validators set up in this system that are validating transactions. [10:21.810 --> 10:28.730] And these validators could be run by you, they could be run by an entity, or some other third [10:28.730 --> 10:35.230] party or trust. Anybody can run a validator. But the other thing you can do as a user is you can [10:35.230 --> 10:41.410] stake your own atoms or tokens with a validator. And the reason you would want to do that is to [10:41.410 --> 10:46.630] additionally profit. So this is kind of similar to the whole staking pool concept with mining, [10:46.630 --> 10:52.790] where I don't want to run all the hardware and I don't want to take on the risk. But I do want to [10:52.790 --> 10:57.790] contribute and benefit in some way. And this is a great way that anybody can go and participate [10:57.790 --> 11:02.810] in these networks and contribute to these validators, and also potentially profit and [11:02.810 --> 11:09.330] reward from it at the same time for contributing to that network. There is some things with [11:09.330 --> 11:14.810] validators where they're going to get block rewards. And the reason you would want to risk [11:14.810 --> 11:20.190] your money with a validator is that block reward, there's an inflation component here. And that was [11:20.190 --> 11:26.670] the 7% that we were talking about. So there's an incentive for validators and holders to stake [11:26.670 --> 11:33.390] their tokens over time. You can see here the average yield in this example is 9.46%. So if [11:33.390 --> 11:41.550] you compare that to a 7% inflation rate, you're up to 2.4% at that point. So it still makes sense [11:41.550 --> 11:45.730] to bond tokens into this. And as long as that average stays above the inflation rate on the [11:45.730 --> 11:52.510] network, it's profitable for you to be bonding your tokens over time and getting those interest [11:53.050 --> 11:57.670] tokens back, essentially. So you would think of this as kind of like a savings account or a high [11:58.150 --> 12:03.810] account where you have to manually go back in and re-delegate your tokens to a validator, [12:03.810 --> 12:08.390] pull out your rewards and re-delegate them. And then over time, your reward amount is going to [12:08.390 --> 12:14.750] grow. And just some statistics here. So for 686 atoms monthly, you would be getting a return of [12:14.750 --> 12:23.290] 5.41. So at $4, you're looking at $20, $25 for basically just doing passive income at that point. [12:28.020 --> 12:34.220] Now, when it comes to validators, another thing that incents them outside of the inflation [12:34.220 --> 12:38.480] component is slashing. And when we're talking about this, we're not talking about the guy from [12:38.480 --> 12:45.620] Guns N' Roses. We're actually talking about penalizing participants in the network for [12:45.620 --> 12:51.320] violating certain conditions. And this can be in the network, this could be in the consensus layer. [12:51.320 --> 12:56.160] But it's a huge financial incentive for validators to play nice because [12:56.160 --> 13:00.760] anytime you introduce these types of systems, you're going to have hackers, you're going to [13:00.760 --> 13:05.000] have bad actors, you're going to have people trying to gamify it and say, you know, how can [13:05.000 --> 13:10.280] I increase my rewards? How can I knock you offline? How can I trick the system or the state of the [13:10.280 --> 13:17.480] system to go in there and profit for me? What's the incentivization to not do that? And slashing [13:17.480 --> 13:23.160] is a great component for that. And there's two ways that you can really get slashed when you're [13:23.160 --> 13:31.020] talking about Cosmos and proof of stake coins currently. They are downtime and double sign. [13:31.020 --> 13:36.920] Both start with D, pretty easy to remember. Downtime has some conditions where out of the [13:36.920 --> 13:43.880] last 10,000 blocks, if you're down for more than 95% of that, you're risking a downtime slashing [13:43.880 --> 13:49.460] event. You can see here in this block explorer, these blocks were being sent and everything was [13:49.460 --> 13:55.560] great. And then at some point, something went offline and now we're missing them. So if that [13:55.560 --> 14:02.820] continued for the next 9,500 blocks or whatever, and we are not able to get our validator troubleshot [14:02.820 --> 14:08.500] or figure out what's going on or move to another system, we risk incurring a downtime event and [14:09.360 --> 14:14.640] having a slashing event occur. The other big one is a double sign. [14:14.780 --> 14:19.600] And this is an example of a double sign that happened on the Cosmos mainnet where what ended [14:19.600 --> 14:27.160] up happening was the private key for the validator was used to sign two transactions, [14:27.160 --> 14:32.760] sign the votes for them for the same block. And that was pushed out, that was detected. [14:32.760 --> 14:38.440] And as a result, that validator was jailed because you can't double sign on your vote. [14:39.260 --> 14:44.820] So once you're jailed for a double sign, unfortunately, you are in jail. There is [14:44.820 --> 14:48.940] no getting out. Your validator will not be able to rejoin the system. [14:49.660 --> 14:52.460] You've screwed up and you've screwed up in a major way. [14:55.770 --> 15:02.270] When it comes to downtime, the amount of stake that can be slashed is very minimal. It's not [15:02.410 --> 15:07.770] a huge amount, but depending on how many atoms are staked in your validator by yourself as the [15:07.770 --> 15:14.570] validator, what you're doing is called self-staking. So a validator may have a million [15:14.570 --> 15:19.130] atoms and stake their million atoms to say, I believe in my system, I believe in myself, [15:19.130 --> 15:22.750] I'm going to stake all my atoms with it. But as I said, users can also stake. [15:22.930 --> 15:26.930] So when these things, when these slashing events happen, not only is the self-stake affected, [15:26.930 --> 15:31.850] but the user stake can be affected too. So as a validator, if you do have a slashing event and [15:31.850 --> 15:36.830] you lose stake, you're losing money for your end users, tokens, and digital value for your [15:36.830 --> 15:42.410] end users. They're not going to be very happy with you or want to contribute to you in the future. [15:43.630 --> 15:48.330] The penalty for that, you miss your rewards while you're in a... and that goes for a 10 minute [15:48.330 --> 15:55.390] period. And with these types of events, usually you only get hit for the first one. It's not [15:56.250 --> 16:02.010] reoccurring. They put, I think it's called a tombstone. I know there's a tombstone here for [16:02.010 --> 16:06.510] double signing, but I forget what it's called for downtime. I apologize. But it's basically kind of [16:06.510 --> 16:12.630] like a buffer to say that we hit you for downtime. We're not going to hit you again if you're in a [16:12.630 --> 16:18.210] certain period. So if you are being slashed for downtime because your server's down or you're [16:18.210 --> 16:24.470] having some sort of redundancy issue, it's still within your best interest to get that up as soon [16:24.470 --> 16:29.850] as possible because these things are going to start to get costly over time. But there is some [16:29.850 --> 16:33.770] protection there for end users so that it's not just going to consistently stack over time unless [16:33.770 --> 16:40.310] certain conditions are met. With double signing, it's huge. It's 5%. So if you do have a million [16:40.310 --> 16:46.270] dollars in Atom on your validator and taking a 5% slash stake, you guys can figure that out and see [16:46.270 --> 16:51.410] how much money that is, right? You also hit an automatic unbonding penalty. If you double sign, [16:51.410 --> 16:57.110] and it also incurs a three-week unbonding period where those Atoms cannot be touched while they're [16:57.110 --> 17:03.450] being unbonded from the validator. I put the little tombstone here for the RIP because if [17:03.450 --> 17:07.770] you double sign, you're done. You're not going to be able to move forward with that validator. [17:07.770 --> 17:14.130] You're either going to need to re-delegate all of your stake to another validator or allow your [17:14.130 --> 17:21.390] users to unbond all their stake and move forward with somebody else at that point. You're out. [17:25.660 --> 17:30.060] Because of all this, you want to make sure you're doing your due diligence when you're either [17:30.980 --> 17:35.900] staking with a validator or thinking about running one yourself. These are all points [17:35.900 --> 17:41.040] you should be thinking about. So it's recommended from my experience that you want to diversify [17:41.040 --> 17:46.740] at minimum probably three is a good bet. I don't want to say best practice. I usually like to stick [17:46.740 --> 17:51.960] the three to five different validators for different tokens. The reason being is that [17:51.960 --> 17:56.740] we're in a decentralized system and we should be decentralizing as much as possible. I'm also [17:56.740 --> 18:01.820] mitigating my risk. So if one of those three or one of those five validators do incur a slashing [18:01.820 --> 18:06.660] event or a downtime event for whatever reason, I've mitigated my risks where I'm not losing all [18:06.660 --> 18:13.520] of my funds at the same time. You want to do some open source research, some open source intelligence [18:13.520 --> 18:19.420] and look up the teams behind them. Look up the history of the validator. So what does their [18:19.420 --> 18:25.260] history look like? Is it clean? Do they have a lot of downtime or maybe a misconfiguration where every [18:25.260 --> 18:31.920] 99 out of 100 blocks they're not signing for some reason? Well, why is nobody noticing that [18:31.920 --> 18:37.460] or fixing that? Are they really paying attention to their system? Have they been jailed before? So [18:37.460 --> 18:43.960] it is possible to go to jail and get out of jail. As we said, if you double sign, you're done. [18:44.120 --> 18:50.540] But for minor events, you can get out of jail if you accept, if you levy the penalty at that point [18:50.540 --> 18:56.820] and you wait for the time that you're jailed. So if they have a history of jail occurrences that [18:56.820 --> 19:00.800] you're not comfortable with, you shouldn't be giving them your money because this is essentially [19:00.800 --> 19:05.720] what you're doing is you're letting them hold digital value on your behalf, essentially. [19:06.620 --> 19:11.520] And then the other thing is a lot of teams have gotten really good in the past year with [19:12.000 --> 19:16.760] putting a presence out there. So there's people and there's corporations that are doing this as [19:16.880 --> 19:22.200] a business, right, where they validate on all the chains I listed earlier and more, [19:22.200 --> 19:26.520] because there's a ton of them out there doing their own little niche with different things [19:26.520 --> 19:31.440] with proof of stake and their chains and their use case. But some of those teams are getting [19:31.440 --> 19:36.340] really good with publishing a social presence. So they're listing their teams on web pages, [19:36.340 --> 19:39.700] or they're showing you with chains, or they're actually showing you pictures of [19:39.700 --> 19:45.440] here's what our setup looks like, our validator setup looks like. Be careful when you do that, [19:45.440 --> 19:50.640] because it's very similar to the ICO craze of 2017, where some of the team pictures, [19:50.640 --> 19:56.660] if you go and reverse Google search or Tanai reverse search those, you're going to find that [19:56.660 --> 20:01.920] those are some LinkedIn picture for somebody that works in like, you know, aviation or something [20:01.920 --> 20:05.960] like that, that has nothing to do, and they're just stealing your information to try to fleece [20:05.960 --> 20:12.380] you. So do your due diligence, don't get scammed. The reason you want to do that. [20:12.380 --> 20:21.560] So this is a personal example. I was validating on a testnet for a coin that was not released yet. [20:21.920 --> 20:28.360] And because it was a testnet, and because I wasn't taking it as seriously as I should have, [20:28.360 --> 20:37.800] I ended up using some spare hardware, and ended up experiencing like a 36 hour power outage. [20:37.800 --> 20:44.480] This spare hardware was hooked up to a Comcast cable modem, no redundancy across the board. [20:45.040 --> 20:50.320] And during the power outage, because I'm focusing on saving my food and a toddler, [20:50.320 --> 20:55.160] and what are we going to do, supposed to be 95 degrees tomorrow, and all of those things that [20:55.160 --> 21:02.500] you have to deal with when life hits you and kind of shit hits the fan. Those are the things that [21:02.500 --> 21:05.680] you're not going to be thinking about your validator, right? You're not going to be thinking [21:05.680 --> 21:09.840] about the thing that you've set up in the dressing room that just runs off the modem that [21:09.840 --> 21:18.080] your wife doesn't like, or your kid resets all the time. So unfortunately, I'm a victim of this. [21:18.080 --> 21:23.000] And thankfully, it was only on testnet, but I ended up losing 10,000 coins because I did not [21:23.000 --> 21:27.820] have a redundant infrastructure set up. Now, if that was to occur on mainnet, and I would [21:27.820 --> 21:32.620] say a validator, I would have just taken a big PR hit and lost my users funds. [21:32.620 --> 21:37.120] In my instance, thankfully, it was only testnet, and it was a great lesson learned. [21:38.500 --> 21:44.420] And this is actually the picture of my validator. So this was an Intel NUC that I had a spare [21:44.420 --> 21:49.420] machine that I set up, and I said hooked it to a cable modem with a single Ethernet line, [21:49.420 --> 21:56.720] single power line, right, single SSD in here somewhere. So no redundancy, no infrastructure, [21:56.720 --> 22:01.200] not even really any status lights to tell me what's going on. So if you are thinking about [22:01.200 --> 22:07.460] running a validator and you find that the people you're thinking about bonding your tokens with [22:07.460 --> 22:13.280] are doing this, you should run away as fast as possible. Great for development and great [22:13.280 --> 22:18.400] for testing, should never be done in production. That's what you want to do in production. [22:18.400 --> 22:23.040] So we want to see the data center, we want to see the cool blinky lights. The reason you want [22:23.040 --> 22:30.220] to do this is you're getting cooling here, you're getting heat and power and network and all the [22:30.220 --> 22:33.920] good warm fuzzy feelings that go into your hosting bill when you have to pay that at the end of the [22:33.920 --> 22:40.440] month to justify all the servers and things that are being filled in these racks. So you know, [22:40.440 --> 22:44.840] stand on the shoulder of giants, use the experts, use the redundant infrastructures that have [22:44.840 --> 22:49.520] already been built and hopefully been tested correctly to host these types of things and [22:49.520 --> 22:58.810] do validation. We want to implement all like the standard best practices. [22:59.830 --> 23:03.330] When it comes to redundancy, you want to hit all the... [23:05.290 --> 23:09.290] Monitoring, we're going to jump into with a couple different solutions that can actually [23:09.290 --> 23:15.890] alert you when you do have the 36-hour power outage. There's a lot of administration concerns [23:15.890 --> 23:22.930] too with different configurations and things like Cosmos has published a security update for [23:22.930 --> 23:29.510] its SDK and it's an emergency upgrade. How are we going to do that, right? Do you test the data [23:29.510 --> 23:34.890] center tech with going in and upgrading that binary? Or do you have a subject matter expert [23:34.890 --> 23:40.070] on staff that can handle that? Do they know the difference between go path and your environment [23:40.070 --> 23:44.810] variable path and all of those types of things? Because all of those are considerations that [23:44.810 --> 23:49.330] you're going to hit and speed bumps that are going to occur while you're working on these [23:49.330 --> 23:55.110] types of systems. You need something to do some sort of repo monitoring, be able to check when [23:55.110 --> 23:59.970] upgrades and new checkpoints and releases are released. And then deployment is another big [23:59.970 --> 24:09.230] consideration for administration. There's a lot of different schools of thought on deployment. [24:09.450 --> 24:16.050] And we'll jump into some of the use cases as to why some are better than the others. [24:16.050 --> 24:21.450] But it's definitely a concern when you're architecting and administering these types of systems. [24:21.550 --> 24:24.750] And then we want to jump into a lot of the security components with [24:24.750 --> 24:28.890] controls and the different aspects on how you're mitigating your risk and how you're [24:28.890 --> 24:33.330] delegating access and who knows what and why do they need to know it. [24:37.510 --> 24:42.850] The keys that actually work on a validator when we're working on security, there's two keys specifically for [24:42.850 --> 24:47.070] Cosmos that need to be protected and thought about with all those prior best practices [24:47.070 --> 24:52.970] for redundancy and backups and things of that nature. And that's your tenderment key, which is [24:52.970 --> 24:58.570] actually doing your consensus voting. And then the application key, which derives your validator [24:58.570 --> 25:03.770] addresses and things. That signs your actual transactions. But both of those need to be [25:03.770 --> 25:09.810] considered when you're building these systems on, one, where does it live? Two, how did we generate [25:09.810 --> 25:15.810] those? Did we do it in a secure way? And I'll show you some ways that you guys can do that. [25:15.810 --> 25:23.970] Three, where are we backing it up? So does it just exist on some EC2 instance somewhere? Or do I have [25:23.970 --> 25:31.910] key material written down in a waterproof, fireproof safe that's access control? A lot of [25:31.910 --> 25:36.030] that stuff needs to go into the thinking and mindset and paradigm of how you're protecting [25:36.030 --> 25:40.550] the system and how you're doing it. And you really want to do it right kind of from the beginning, [25:40.550 --> 25:47.090] because if you deploy a validator to determine that I didn't generate my key material securely, [25:47.090 --> 25:52.210] you don't want to have to go back and redo all that stuff and re-delegate and try to make sure [25:52.210 --> 25:56.110] that people are aware that, you know, we're validator two instead of validator one now, [25:56.110 --> 26:01.170] those types of things. So when you're building this stuff, kind of encompass all this in your [26:01.170 --> 26:05.770] thinking and do it right from the beginning. Because really what these keys are protecting, [26:05.770 --> 26:11.290] I checked CoinCap IO right before this presentation, 1.3 billion market cap for [26:11.290 --> 26:18.090] Cosmos Atoms right now. So for a coin that's still relatively brand new in the scheme of things, [26:19.090 --> 26:24.170] we're already at a billion dollar market cap at current day prices. So if the bull market really, [26:24.170 --> 26:30.410] really starts taking off, at that point, these keys are protecting crazy amounts of value. Are [26:30.410 --> 26:36.570] you comfortable leaving that type of information sitting on a server that, you know, the data [26:36.570 --> 26:41.510] center tech down the street that happened to work on the day that you called in and said, [26:41.510 --> 26:45.990] my server's down, can you check the lights on my Dell system? You know, he's looking at console [26:45.990 --> 26:52.550] and he's single usering and things. While your key material may be protected by local key stores [26:52.550 --> 26:57.390] and things like that, what stops him from sniffing memory? What stops him from imaging the drive or [26:57.390 --> 27:02.310] breaking RAID or things like that and then clearing logs? So it's considerations that you [27:02.310 --> 27:08.450] want to think about when you're building those systems. For redundancy, it's all the great stuff [27:08.450 --> 27:15.590] with power supplies and phasing and networking. And I've seen some validators where they run on [27:15.710 --> 27:21.390] a single networking. So what happens if that network card fails or the cable goes bad, [27:21.710 --> 27:26.670] a pair in that cable goes bad? You're going to be at risk of a potential downtime event. [27:26.670 --> 27:32.990] When it comes to storage, there is a lot of data that goes into running a validator. A validator [27:32.990 --> 27:39.870] essentially needs on itself a copy of the full node, the chain data, or it needs to link to [27:39.870 --> 27:46.930] another server a century that has a copy of that full node chain data. So if you're building in [27:46.930 --> 27:52.670] redundancy to say if this instance goes down, once you rebuild your new instance with your new [27:52.670 --> 27:58.290] validator binaries, you need to sync all the chain data again. So if you're in that window, [27:58.290 --> 28:03.430] that downtime window, that 95% of 10,000 blocks, which I think is like half a day, [28:03.430 --> 28:08.170] and it's starting to get down to the wire, I've seen where we've started binaries and things like [28:08.170 --> 28:13.150] that and my validators, you know, they still need two hours to catch up to sync chain data [28:13.150 --> 28:17.610] because I didn't have a copy of it anyway. So I would definitely recommend looking at things like [28:17.610 --> 28:24.650] RAID if you're using SANs, doing snapshots and things of that nature. The Cosmos team has [28:24.650 --> 28:30.350] published snapshots over time of chain data where you can go and download those types of things to [28:30.350 --> 28:35.610] sync it, but I'd be careful with what sources you're pulling that from because if a bad actor [28:35.610 --> 28:40.250] wants to cede bad data to you, you could potentially risk another issue with your validator at that [28:40.250 --> 28:44.810] point. And then there's other some some other gotchas that people don't necessarily think about [28:44.810 --> 28:49.290] and like, there's some funny stories and some instances, you're standing in front of the [28:49.290 --> 28:56.630] server in the data center, and you're going, I'm here. I have no way to log in, there's no keyboard, [28:56.630 --> 29:01.190] right? There's no crash cart available in the data center. So run through some of your BCDR [29:01.190 --> 29:06.650] type of policies with like, what do we need in the cage, right? Do we need spare hard drives? Do we [29:06.650 --> 29:12.590] need installation software for OSs if we have to spin it up? Do we need config variables to point [29:12.590 --> 29:17.450] to a PXE server or to point to a Kubernetes cluster? However you decide to architect your [29:17.450 --> 29:23.650] solution that best works for your information system. And I note down here, any time that you're [29:23.650 --> 29:28.330] running through this type of stuff and architecting it and running through these exercises, you always [29:28.330 --> 29:33.430] want to think about the worst case scenario, which is double signing. And in order to achieve that [29:33.430 --> 29:39.230] double sign, it means that those Cosmos keys need to be either duplicated somewhere or pulled into [29:39.370 --> 29:44.610] a configuration or have two instances that are running that same material. So always think about [29:44.610 --> 29:49.710] that. That's kind of the grenade in the mix that needs to always move forward and can never be [29:49.710 --> 29:59.770] copied because if it is and it goes live, you're risking that 5%. I put this slide in here just [30:00.270 --> 30:07.170] because I hate DNS. DNS is a big one. And you want to be careful with this because when you [30:07.170 --> 30:11.970] start to build validator infrastructure and you start to do additional security controls and [30:11.970 --> 30:18.550] different servers with copies of chain data, DNS is one that may get you that you don't even realize [30:18.550 --> 30:23.790] because nobody ever thinks it's DNS, but it always is. So make sure your host names are correct. [30:23.790 --> 30:30.050] Make sure you have redundancy upstream in your network stack, not just the cables, even the [30:30.050 --> 30:36.870] routers in the rack. If you're using things like Netgear or Cisco equipment, do you have that in [30:36.990 --> 30:43.070] a stack configuration? Do you have an AB failover? If your switch or your Cisco ASA license expires [30:43.070 --> 30:48.510] and you start getting throttling or your feature stops off, what happens into your system? So those [30:48.510 --> 30:52.750] are all things to simulate and think about when you're building that. And DNS is one of the ones [30:52.750 --> 31:02.120] that I don't want people to forget about. So now we need to monitor all this stuff with our [31:02.660 --> 31:08.600] validators. We've figured out how it works. We've got some coins. We've got a system set up. [31:08.600 --> 31:15.640] Now, how do we avoid that 36-hour outage? So SimpliVC is a company that's out there that [31:15.640 --> 31:20.400] does a ton of validation on chains and things like that. And they release this open source software [31:20.400 --> 31:25.780] called Panic. And I'm a big, big fan of open source software. But what this will do is this [31:25.780 --> 31:31.900] will actually monitor things from the tendermint layer and from the local server layer. And it'll [31:31.900 --> 31:39.440] do email, telegram, through Twilio integrations, it's not responding. And it'll also do the repo [31:39.440 --> 31:44.940] monitoring we talked about earlier. You know, a new version of Cosmos, okay, or Gaia-D or whatever [31:44.940 --> 31:49.560] binary has been released, you need to be aware of that so that you can go check the release notes [31:49.560 --> 31:56.040] and see if this is something I need to apply now or can I defer that to later? I love it. I use it [31:56.040 --> 32:01.840] for all my validation stuff. This is my telegram integration. And you can see it doesn't just [32:01.900 --> 32:08.700] do things like the services down. It actually tells things like my peers have increased or the [32:08.700 --> 32:14.400] node was inaccessible. If people delegate to my validators, it gets updates on my voting power. [32:14.400 --> 32:18.420] So I know that I've got more power in the system for consensus votes or that there's something [32:18.420 --> 32:22.840] happening. And I need to be looking at my validator because people are delegating to me. [32:22.840 --> 32:27.160] I want to know why. And then as I said, it gives you peering changes. So if other people are doing [32:27.900 --> 32:32.560] peers dropping off the network, that may be indicative of a network event or a binary upgrade [32:32.560 --> 32:36.340] or something else is going on with those other servers. They're doing some work to that stuff. [32:36.340 --> 32:40.720] And it's good to be aware because we may need to take mitigation efforts. [32:43.450 --> 32:48.170] Prometheus is another great one. This is all packaged with the current Gaia installation [32:48.170 --> 32:54.490] with Cosmos. It's a very simple config change. And this is basically kind of like a logging port [32:54.490 --> 33:00.630] that opens up a whole bunch of data that's going on in your validator that you can feed into Elk [33:00.630 --> 33:05.910] stack or different dashboards or whatever you choose. But it has really good support for Grafana, [33:05.910 --> 33:12.330] for example. And this brings it into the kind of DevOps infrastructure world of, [33:12.330 --> 33:16.630] I want to build dashboards and I want cool graphs and stats and stuff like that. [33:16.850 --> 33:21.710] So if you don't want to do that through necessarily a block explorer, if you want to run something [33:21.710 --> 33:26.550] locally, these things are, it's great to turn Prometheus on. It's great to throw Grafana or [33:26.550 --> 33:35.020] some sort of Elk stack monitoring on top. So we've got our monitoring taken care of and now we're... [33:37.180 --> 33:44.680] So we talked about the data center tech at, let's use AWS, for example. So you're running [33:44.820 --> 33:49.980] a validator in AWS and you've got this private key material on there that's validating transactions [33:49.980 --> 33:55.480] and participating. What stops the data center tech from logging into your system and grabbing [33:55.480 --> 33:59.540] your key material? What stops the hacker from getting on your server and grabbing that stuff [33:59.540 --> 34:04.900] remotely if you haven't configured your security groups and your firewalls correctly? So that's [34:04.900 --> 34:10.340] the problem and the solution is HSMs. And HSMs are great if you guys have never used them. They're a [34:10.340 --> 34:15.080] little bit expensive, but it's very similar to the way that, you know, YubiKeys and things like that [34:15.080 --> 34:20.940] where it stores the private key material actually on a hardware device and then signs messages [34:20.940 --> 34:26.540] without actually having to expose or reveal that material. In theory, it's not supposed to allow [34:26.540 --> 34:31.560] any private key extraction. Now there have been attacks and that's not to say that things can't [34:31.560 --> 34:37.180] fail in the future, but currently it's not supposed to allow any private key extraction. [34:37.420 --> 34:42.840] So what you would do is when you were generating key material to either set up your validator or [34:42.840 --> 34:48.320] your wallet or functioning, you want to generate those keys in a secure way and then store that [34:48.320 --> 34:54.140] private key material on a hardware device. The two I've used is the YubiHSM2 and the Ledger Nano S, [34:54.140 --> 34:59.980] which both work very well with Cosmos. I know they work with MicroTik and they work with a few other [34:59.980 --> 35:05.460] other chains that are out there. But this would actually physically sit into your bare metal [35:05.460 --> 35:12.100] server in the data center, in my case, in the dressing room and provide the private key material [35:12.100 --> 35:17.100] and signing functions without revealing it. So if somebody was to get into your server remotely, [35:17.100 --> 35:21.500] or if the data center texting their single user in, they're not going to be able to extract this [35:21.500 --> 35:26.940] without a whole bunch of different authentication information and, you know, secrets and things that [35:26.940 --> 35:32.320] they don't have. And then the actual physical hardware security capability of the device too. [35:33.180 --> 35:40.120] One thing I will note to be very careful with this stuff is that the YubiHSM in particular [35:40.120 --> 35:44.700] has a very small profile and in order to firmware reset this device, I think it's like hold this [35:44.700 --> 35:49.600] front side for 10 seconds or something. So if you're in a data center working and you're [35:50.560 --> 35:56.300] failing over from server A to server B because you're doing maintenance and you pop your HSM out, [35:56.300 --> 35:59.940] be careful how long you're holding it and where you're touching it on. Because if you wipe that [35:59.940 --> 36:04.680] private key material, that's not going to be recoverable and you better hope to hell you have [36:04.780 --> 36:09.580] a backup. The Ledger works really great with it too and it's got the little graphical display and [36:10.120 --> 36:14.400] same thing, you know, there's always security vulnerabilities out for hardware wallets but [36:14.400 --> 36:19.020] people are keeping up on those and they're auditing them and it's still a better layer [36:19.020 --> 36:30.020] of defense than just leaving the key material on the server itself. Another thing you want to do [36:30.020 --> 36:36.740] from a security aspect is you want to set up Sentry architecture. And a Sentry is basically a [36:36.740 --> 36:43.260] copy, a full node of the chain that you're validating on. So you could run your validator [36:43.260 --> 36:48.520] straight out to the internet and start broadcasting these things. But it's better [36:48.520 --> 36:55.160] to put it behind another layer of obfuscation and network connectivity here. And what we're doing [36:55.160 --> 37:02.760] is we've connected public internet that was gossiping a lot of information about our transaction [37:02.760 --> 37:08.600] validator itself and we've moved it up a layer behind a Sentry layer. And when we do that, [37:08.600 --> 37:13.380] this actually enables us to now load balance and have some redundancy at the Sentry layer [37:13.380 --> 37:17.860] where we can deploy multiple Sentries here. So I've used two in this example, but it can really [37:17.860 --> 37:23.480] scale. I don't know if there's any technical limits on that. That would be a great follow-up. [37:23.480 --> 37:27.640] But as far as I'm aware, I haven't seen a technical limit on how many Sentries you can run. [37:28.140 --> 37:33.020] But the whole point of this is that nobody's going to be able to get to your validator directly to [37:33.020 --> 37:37.560] any public IPs or anything like that. You're going to have private communication between your [37:37.560 --> 37:44.380] validator and your Sentry itself. And I've seen this done many different ways where this validator [37:44.380 --> 37:49.740] could be inside its own VPC and you have IPsec tunnels talking out to your Sentries, which are [37:49.740 --> 37:55.860] in its own VPC that consists of two or three instances. This could just be two or three EC2 [37:55.860 --> 38:02.860] instances that are talking to each other through private IP space. This validator may not even be [38:02.860 --> 38:07.780] the same location or site. It could be bare metal talking up to cloud Sentries somewhere, [38:07.780 --> 38:13.160] bare metal Sentries in another data center across the country. I wouldn't recommend that for time [38:13.160 --> 38:19.800] to live and packet communication, but it is technically possible. This stops people from [38:19.800 --> 38:24.860] denial of servicing your validator. It also gives you some redundancy here for failures with your [38:24.860 --> 38:29.080] different Sentries and things like that. And as I said, it protects your communications between [38:29.080 --> 38:34.260] your validator and your Sentry. So the validator would get the information from Sentry, sign [38:34.260 --> 38:40.480] transactions, broadcast it out through the Sentry out to the public internet. So from an attacker [38:40.480 --> 38:44.860] standpoint, they would only get as far as this layer. They wouldn't get to the stuff that actually [38:44.860 --> 38:56.280] matters. Other considerations with your security architecture. We never want to double sign. [38:57.420 --> 39:03.060] A lot of cloud providers offer auto scaling, and I think that's great for Sentries, but I would [39:03.060 --> 39:09.560] avoid it for my validator infrastructure, because you start to get into instances of where does my [39:09.560 --> 39:15.460] key material live and how do I migrate? And if server A still has it active and server B spins [39:15.460 --> 39:20.360] up and I've copied my key material and that service starts, I've now double signed, right? And I'm in [39:20.360 --> 39:27.460] it's game over. Any automation and orchestration pieces with the Sentry, you want to think about [39:27.460 --> 39:32.720] the blockchain data and the state data. How are you moving that? How are you verifying that? [39:32.720 --> 39:36.640] How are you loading that into new instances if you're doing auto scaling or automation? [39:36.760 --> 39:41.520] Or do your new Sentries have to come up and sync the entire chain before they can be active? [39:42.180 --> 39:46.560] You also need to worry about doing your config updates. So if I had to have a validator [39:46.560 --> 39:51.300] infrastructure and I have two Sentries and add a third, well now I need to go back to all of my [39:51.300 --> 39:57.280] existing systems and do some config updates so that I'm now listing that here in my config to say [39:57.280 --> 40:02.500] talk to Sentry 3. Don't just talk to Sentry 1 and 2. So when you're building the automation, [40:02.500 --> 40:06.440] if you have Sentries and things in that infrastructure, you need to kind of backfill [40:06.440 --> 40:11.760] all of those changes to stuff that you've already implemented. And then Sentries are just like [40:11.760 --> 40:16.260] validators because it's a full node. If there's security updates for the binary, [40:16.260 --> 40:20.280] if there's issues with it where it goes down, you need to be doing all the same [40:20.280 --> 40:26.060] all the same repo stuff, all the same binary upgrades when it comes to those Sentries. So [40:26.060 --> 40:31.540] if you're scaling your Sentry architecture out horizontally to 40 Sentries, just realize you [40:31.540 --> 40:39.010] need to do updates to all of those Sentries when things happen. Then there's some final [40:39.010 --> 40:46.150] other considerations here. As I said earlier, there's a lot of different mindsets on should [40:46.150 --> 40:55.410] you virtualize validators? And I've gone back and forth on this and I feel like validators should [40:55.410 --> 41:00.670] be bare metal. And while it is technically possible to deploy like an EC2 instance or [41:00.670 --> 41:06.690] Google's GCP instance and have it validate with key material on that validator, that's not going [41:06.690 --> 41:11.670] to be the most secure, optimized, redundant solution that's out there. And if I was considering [41:11.670 --> 41:17.170] and looking at two validators on where I wanted the funds, I would not pick that one. I would [41:17.170 --> 41:20.670] go with the person that considered some of this stuff. And even if they were using one or two [41:20.670 --> 41:26.210] bare metal servers in like a hot or cold fail, to me, I'm a little bit more comfortable with that [41:26.210 --> 41:31.250] than somebody that's doing pods of validators that's moving key material back and forth, [41:31.250 --> 41:35.550] because all that needs to happen is a weird edge case or anomaly where you're going to hit some [41:35.550 --> 41:41.450] sort of double signing scenario, right? And unless you have your deploy scripts tweaked so good with [41:41.450 --> 41:47.170] so many tests and error catching and then, yeah, it's scary when you look at the amount of money [41:47.170 --> 41:53.810] with that 5% double slash. There's other things you want to consider with minimum config values. [41:53.810 --> 41:59.590] So I've done some attacks and I've seen some defense on sentries and things with spam dust [41:59.590 --> 42:04.870] transactions where people can try to tax your validator, your sentry resources. And the reason [42:04.870 --> 42:10.150] they would do that is to try to edge because maybe they run their own validator or you were [42:10.150 --> 42:15.930] jailed and they had money with you or tokens with you, some sort of value. So there's some minimum [42:15.930 --> 42:20.990] config values that validators can tweak and sentries can tweak to say, ignore more of the spam [42:20.990 --> 42:26.150] and only give me the good stuff. There's an art to it and you got to be careful [42:26.150 --> 42:30.510] because you want to be a good system. But just know that you can get pretty granular in your [42:30.510 --> 42:34.950] validator config with all the different values that are participating at the tendermint layer [42:34.950 --> 42:40.610] and the consensus layer. And that goes for denial service too. And then this last one here that I [42:40.610 --> 42:45.870] don't think a lot of people... and I've actually seen a lot of wallet software and validator [42:45.870 --> 42:52.970] software getting better with this. People get in the habit of validating and starting to get rewards [42:52.970 --> 42:59.510] and they'll pull out all of their rewards and say, re-delegate every single token I have, [42:59.510 --> 43:04.510] all my rewards in my wallet, send it back to the validator because I want that compounding interest. [43:05.310 --> 43:09.050] And what'll end up happening is it'll work. They'll still sign the transaction and send it [43:09.050 --> 43:13.910] out to the validator. And then they literally have zero in their wallet. So you guys, you know, [43:13.910 --> 43:18.730] you're familiar with ETH and other systems. If you send all your ETH out and you try to do an ERC20 [43:18.730 --> 43:22.290] transfer, what's it going to do? It's going to complain and say, you don't have enough ETH for [43:22.290 --> 43:26.730] the gas for the transaction. So be careful with that because if you re-delegate all your rewards [43:27.350 --> 43:31.690] thing that's in that wallet, you're not going to have enough gas to do any more transactions at [43:31.690 --> 43:35.350] that point in the future. So you're going to have to go procure atoms from somebody else or wait [43:35.350 --> 43:39.530] until you have enough rewards to get enough to be able to start to continue to do transactions. [43:39.530 --> 43:45.110] So I've gotten in the habit of like leaving half an atom or an atom or whatever denomination or [43:45.110 --> 43:48.870] whatever token I'm using, leaving one or two of those in the wallet, just so I know that I'm [43:48.870 --> 43:52.790] always going to be covered for transactions in the future. And that stuff's getting way better [43:52.790 --> 43:58.410] with doing calculations and fees and even notifying users. Cosmos station is a mobile [43:58.410 --> 44:02.290] wallet that I love. And it'll come up and actually warn you and say like, hey, you shouldn't be [44:02.290 --> 44:07.470] delegating max, like you should be delegating 90% of max or whatever you want to do. [44:09.410 --> 44:12.750] So with all those things we talked about with redundancy, [44:12.750 --> 44:16.270] I wanted to ask the question, should you be running a validator? [44:18.950 --> 44:26.050] And in my opinion, no, not for the validator. If centuries and things like that, where it's [44:26.050 --> 44:32.070] not actually signing transactions or dealing with value, that is great for scaling up and [44:32.070 --> 44:35.850] auto scaling. And as long as you can take some of the data considerations and things I was talking [44:35.850 --> 44:42.570] about earlier into that system, it's amazing for centuries. I fully recommend GCP, Amazon, [44:42.570 --> 44:48.210] DigitalOcean. It takes the onus off you and it gives you all of those redundancy and good, [44:48.210 --> 44:51.950] warm fuzzies. But for the validator, I feel like that's something that you should be controlling [44:52.150 --> 44:57.750] a little bit more manually doing your own internal orchestration and things like that. [44:58.050 --> 45:02.050] You want to be careful about where you're hosting that, who has access to it, [45:02.050 --> 45:08.390] what is their security posture? What is their uptime in their SLA? If their SLA is more than a [45:08.390 --> 45:12.450] day, that's not good for a validator because if you have a downtime event, you could be slashed. [45:12.570 --> 45:16.470] All things to think about when you're negotiating with your service providers. [45:17.230 --> 45:22.330] The last thing for best practices I wanted to talk about was the cryptocurrency standard. And this [45:22.330 --> 45:27.690] was more in line with what I talked about last year here at the Blockchain Village. But it's [45:27.690 --> 45:33.210] basically a standard that's amazing that was presented by C4 that dives into a lot of different [45:33.210 --> 45:39.050] security controls and aspects that one can implement in a cryptocurrency information system. [45:39.990 --> 45:45.270] It gives you the guidance, it gives you a control and different levels from level one to three to [45:45.270 --> 45:50.690] say, here's the very basic control I can put into place versus level three. Here's the ultimate [45:50.690 --> 45:57.170] paranoid control and kind of give you an example for that. One would be the key generation. So we [45:57.170 --> 46:02.510] were talking about keys earlier with the keys and which ones were both for consensus and which ones [46:02.510 --> 46:07.050] do transactions. And one of the things of the cryptocurrency security standard is how are those [46:07.050 --> 46:13.210] generated? What was the device that you generated them on? What was the entropy device? Did you do [46:13.210 --> 46:18.010] it on your network connected computer that you do work and Netflix and torrenting and all that stuff [46:18.010 --> 46:23.850] on? Or did you do it on an air-gapped machine with a secure clean room type of setup? Did you back it [46:23.850 --> 46:29.310] up correctly? So level one may say, I made a backup of the key, I'm good right now. It's on paper [46:29.310 --> 46:35.810] somewhere. But level three would say, that backup needs to be sealed, it needs to be fireproof, [46:35.810 --> 46:40.670] it needs to be waterproof, environmental proof, it needs to be access controlled. So as you kind [46:40.670 --> 46:44.010] of go up the stack here, there's different requirements and different levels of security [46:44.010 --> 46:49.070] and paranoia. And while you may not be able to use the cryptocurrency security standard [46:49.070 --> 46:56.410] to certify your information system, depending on how it works or what you're doing, it's a great [46:56.410 --> 47:00.990] framework to check your validator against for different levels of compliance. And it also gives [47:00.990 --> 47:07.390] you some really good ideas for things like data sanitization policy. So if you are spinning [47:07.390 --> 47:12.810] instances up and down, where are your logs going? Do you have a process for clearing all of that [47:12.810 --> 47:17.790] stuff correctly and securely so that nobody can run forensics on it if they wanted to profile you [47:17.790 --> 47:22.230] or get more information? So there's a lot of great things in here that I think definitely [47:22.230 --> 47:26.350] apply to that information system, even if it's not every single one of them. [47:28.050 --> 47:34.550] So what's next for the future? And this is really kind of exciting for me and validators and Cosmos. [47:34.690 --> 47:41.350] IBC is right around the corner. And I just took place in Game of Zones, which was a attack, [47:41.350 --> 47:48.970] defend scaling competition that the Interchain team and Zacky Mannion and Jack Sampling [47:50.870 --> 47:55.910] threw. So we had a whole bunch of different validators and people that were taking place [47:55.910 --> 48:01.030] in this. And as you can see in this graph here, these were all the participants. And they were [48:01.030 --> 48:06.130] connected to a main hub, a main Cosmos hub. So this is very much indicative of the wheel and [48:06.130 --> 48:11.690] spoke infrastructure that I think everybody's kind of used to. But this was to theoretically [48:11.690 --> 48:16.170] prove that all of these people could connect into the main hub and get them prepared for the next [48:16.170 --> 48:21.730] thing, which is the IBC protocol. And when we're talking about the IBC protocol, we're talking [48:21.730 --> 48:27.930] about validating on chains that we're not running necessarily. All we're doing is syncing state with [48:27.930 --> 48:35.010] them through light clients. So that if Alex chain, or I'm sorry, if Bob chain, for example, wanted to [48:35.010 --> 48:40.470] hold Alice coin, Bob and Alice's two chains can now talk together through a light client through [48:40.470 --> 48:46.130] IBC. So now we're breaking away from that hub and spoke model. And our validators can now choose who [48:46.130 --> 48:50.530] they want to talk to and where they want to validate. So you're seeing here the very beginning [48:50.530 --> 48:55.230] of this, where different zones are kind of breaking off into their own sovereign nations [48:55.630 --> 48:59.570] and saying, I want to validate on this chain, or I want to participate on this chain in addition to [48:59.570 --> 49:04.170] my own. And this is the magic where I was talking about earlier for validators, where when they go [49:04.170 --> 49:08.470] to do the rewards, they could be validating on Bitcoin, they could be validating Litecoin, [49:08.470 --> 49:13.410] they could be validating Ethereum and Adams, as well as Ron, Bob, Alice and Jim coin, because [49:13.410 --> 49:18.750] they see some value there. And when you start to open up that ecosystem and that level of connectivity, [49:18.750 --> 49:22.630] there's some really, really cool applications and cases and things that [49:22.630 --> 49:26.010] people are talking about and playing with. And it gets me really excited. [49:27.590 --> 49:30.810] So that's what I have for my talk. If you guys do have any questions, [49:30.810 --> 49:34.630] feel free to reach out to me at my consulting company, Stoner Consulting. [49:34.990 --> 49:40.010] I love talking about this stuff. I'd love to go into detail if anybody wants to jump more into it. [49:40.010 --> 49:44.590] And I also wanted to throw a special thanks out to Sean Martin, one of the people that's worked [49:44.590 --> 49:51.370] with me on validators and infrastructure and system. And he's really an infrastructure magician. [49:51.370 --> 49:55.830] So if you guys ever have infrastructure questions or scaling issues or anything like that, [49:55.830 --> 49:58.270] please let me know and I will get you in contact with Sean. [49:58.730 --> 50:01.190] I'm Ron Stoner and thank you for attending my talk. [50:50.840 --> 50:52.320] I don't know if he can, but I can. [50:57.110 --> 51:00.970] Sorry, I was looking to see if there was any questions. So I did see one in there, [51:00.970 --> 51:06.030] correction, yes, it is called a liveness check. That is correct. And that was in regards to the [51:06.930 --> 51:10.630] violations, the consensus violation and the liveness violation. So downtime. [51:10.670 --> 51:13.550] So that is the correct terminology for it. Thank you. [51:20.790 --> 51:24.210] Cool. And I don't think I see any other questions unless anyone has any. [51:55.140 --> 51:58.060] I would love to. Yeah. Thank you, Jay. Thank you, Nathan. [51:58.060 --> 52:02.180] Thank you, everybody that put on this village and promoting blockchain. I love it.