WEBVTT 00:00.000 --> 00:01.720 Well, thank you, Ajit. 00:01.720 --> 00:07.000 I want to welcome everybody to DEF CON and the Blockchain Village. 00:07.000 --> 00:08.140 My name is Ron Stoner. 00:08.140 --> 00:12.980 I'm going to talk about securing the cosmos, and I appreciate that wonderful introduction. 00:13.040 --> 00:15.480 I'm a repeat speaker here. 00:15.500 --> 00:25.800 I spoke last year when we were talking about the cryptocurrency security standard with Michael Parkland, and it was an amazing experience, so I'm happy to be back. 00:26.100 --> 00:35.620 I will preface this talk by saying I'm only on a few hours of sleep in traditional DEF CON fashion, a little bit hungover, and I was working on my presentation up until a few minutes ago. 00:35.620 --> 00:47.660 So if there's any inconsistencies in here, or you guys see formatting issues, I'm more than happy to clean that up if you want to copy the presentation, or if you want to discuss further just to clear up any ambiguity after the stream. 00:49.000 --> 00:50.200 So who am I? 00:50.300 --> 00:57.840 I'm a senior security engineer for ShapeShift.com, one of the best digital asset exchanges out there. 00:57.840 --> 01:12.800 And as a result, I get exposed to a lot of different technologies, and I get to play with a lot of cool different things in the crypto space, including Cosmos, the Atom token, and some cool new validation software that people are releasing and trying out. 01:12.800 --> 01:22.800 I'm also the curator of the cryptocurrency security standard auditor exam, so that's an exam regarding the cryptocurrency security standard that I referenced earlier. 01:22.800 --> 01:28.960 I have my own consulting firm where I consult on security and blockchain technology. 01:29.160 --> 01:33.600 And then if you guys want to reach me on Twitter, you can hit me up at forwardsecrecy.com. 01:33.700 --> 01:47.440 The reason I'm giving this talk to you guys today is because I do have a lot of experience on testnet and mainnet with deploying, configuring, administering, and securing validators and validator infrastructure. 01:50.250 --> 01:52.390 I wanted to set up a goal for this talk. 01:52.390 --> 01:56.690 So you guys can read it there, I'm not going to go through that. 01:56.730 --> 01:59.430 I do want to highlight my redundant joke here. 01:59.430 --> 02:01.050 Hopefully you guys like that one. 02:01.530 --> 02:02.810 If not, oh well. 02:03.450 --> 02:08.130 But my goal for this talk is to have you guys walk away here learning something. 02:08.350 --> 02:12.590 There's a lot of smart people here at DEF CON in the blockchain village. 02:13.130 --> 02:21.750 But while we may be good at infrastructure, or we may be good at security, or we may be really good at the blockchain aspects of it, nobody's perfect, right? 02:21.750 --> 02:23.810 Nobody's an expert in all areas. 02:23.810 --> 02:32.610 Hopefully you walk away from this talk taking a tip or some piece of knowledge that you can run with yourself or implement in your own system. 02:37.100 --> 02:39.160 So what are validators? 02:40.320 --> 02:48.600 Everybody's kind of familiar with the concept of Bitcoin and SHA-256 and mining, right? 02:48.600 --> 02:57.720 So when you're mining, you're throwing a ton of resources at cryptographic problems to solve them and write data to a blockchain. 02:57.980 --> 02:59.480 And mining was awesome. 02:59.480 --> 03:01.680 It was very profitable for people for a long time. 03:01.680 --> 03:03.360 It helped secure the network. 03:03.360 --> 03:19.680 But there's a lot of downside to miners that people have brought up over time, including energy consumption and administration and things like I can't get the newest graphics card to play PUBG or Fortnite because there's run on them, right? 03:19.680 --> 03:23.840 There are five, $600 now because everyone's buying them up and mining. 03:24.480 --> 03:29.460 So some new coins and chains came out that implemented what was called proof of stake. 03:29.540 --> 03:33.140 And unfortunately, it's not the stake that I enjoy every night. 03:33.140 --> 03:36.560 I would love a proof of actual like a one type of stake. 03:36.740 --> 03:43.960 But this proof of stake is where we're actually staking and bonding coins and doing our network through validators. 03:44.240 --> 03:51.780 So those validators actually provide a whole bunch of energy savings, and they can achieve kind of the same end result. 03:51.780 --> 04:02.600 They're able to validate transactions, validate blocks, provide signatures, create chain data and do it in a way where it's more beneficial to the environment and it's lower cost and overhead. 04:03.060 --> 04:09.060 The way it works, as I said, people end up bonding or locking or staking the coins with the validator. 04:09.300 --> 04:14.500 And then for this talk, we're going to be focusing a little bit more on Cosmos. 04:14.500 --> 04:21.880 But with things like ETH2, they have some different systems where they kind of bet on blocks and transactions and things for the validators to process. 04:22.020 --> 04:28.960 But on the back end, what it's really doing is it's taking cryptographic signatures and signing and broadcasting votes out to the chain. 04:34.560 --> 04:40.860 The incentive for the validator, you can see right here is money, basically, digital value. 04:40.980 --> 04:48.600 So as a validator, you're validating all these transactions, and you're running your infrastructure and you're putting in all the time and effort. 04:48.680 --> 04:50.700 And what's the incentive to go and do this? 04:50.700 --> 04:52.380 So there's a few different pieces. 04:52.380 --> 04:54.740 The first one's the security aspect. 04:54.740 --> 04:59.960 So you're providing back to the security of the chain, the decentralization aspect. 04:59.960 --> 05:02.720 So you're participating in a decentralized system. 05:02.720 --> 05:20.400 There's a governance aspect where as a validator, if you meet certain conditions based off of the chain and governance that's been set, you can actually participate in governance votes and help to modify and make changes to that chain and its different variables moving forward. 05:20.760 --> 05:23.920 But there's also block rewards and transaction fees. 05:23.940 --> 05:25.960 And that's kind of where the money comes in at. 05:25.960 --> 05:28.420 And this is just a very small example. 05:28.420 --> 05:35.020 So I think this is one day of rewards for a validator that I was running on a new chain that was just released. 05:35.100 --> 05:38.500 And you can see here, I'm getting some die and I'm getting some tick. 05:38.520 --> 05:50.520 And this validator, while it's using tick as its denomination and also paying some commissions in die, this could be whatever coin or whatever chain that you're running your validator for. 05:50.520 --> 05:53.480 So in the instance of Cosmos, that would be Atom, right? 05:53.480 --> 05:56.340 And today, it's kind of set to a single chain. 05:56.340 --> 06:06.220 But the real value for validators is that you can run validators on multiple different chains and tokens and provide the security and the governance and get the kickbacks from that. 06:06.220 --> 06:13.940 But in the future, with some of the scaling efforts and new technology that's coming out, those validators are going to be able to validate multiple types of coins. 06:13.960 --> 06:28.200 So if I'm a validator that's running a whole bunch of different chains, and I'm validating on that, my rewards for the day could be Bitcoin, Dogecoin, Litecoin, Die, Tick, whatever I choose to validate on or stake my tokens with. 06:28.520 --> 06:32.460 And that's something that's going to be important that we're going to talk about a little bit later. 06:32.460 --> 06:39.480 But from the validator aspect, like I said, the rewards, transactions, governance, being able to participate in the system. 06:39.620 --> 06:49.880 And the other thing that's important is the more stake, the more coins or tokens that have been bonded with that validator, the more power and say that validator has within the system. 06:50.000 --> 07:00.000 So it's very important to make sure that you're keeping up on that and that you're bonding those tokens and contributing back to the ecosystem in order to be a good neighbor. 07:01.760 --> 07:08.060 The other reason validators run this, so while you're getting block rewards and you're contributing, it's pretty cheap to do this. 07:08.060 --> 07:10.380 So this is an unnamed cloud provider. 07:10.460 --> 07:12.760 And this is just some simulated costs. 07:12.860 --> 07:20.140 So as you can see for the month, it's extremely cheap to run these instances or they can be virtualized instances, they could be bare metal instances. 07:20.900 --> 07:26.580 There's a lot of benefits to running up in a cloud provider, which we're going to jump into. 07:26.660 --> 07:41.460 But the overall goal of this is it's extremely cheap to do this from an infrastructure side, not as necessarily as cheap from the management side and the personnel, the human cost, but at least getting things spun up, running a validator. 07:41.460 --> 07:53.500 You can get this done in an afternoon if you're the type of person that has some familiarity with cryptocurrency, has familiarity with the Linux command line, and has some familiarity with best practices and security. 07:53.560 --> 07:59.320 Because you are dealing with things that have a ton of value associated with them. 08:00.460 --> 08:07.220 There's a lot of different networks that currently run validators and do these proof of stake type of systems. 08:07.680 --> 08:11.420 Cosmos here in the middle is the big one that we're focusing on for this talk. 08:11.420 --> 08:18.080 But a lot of the information presented here can be applied out to a lot of these different networks that run validation systems. 08:19.180 --> 08:24.780 Microtip down here in the right is a brand new one that just came out within the past couple of weeks, they launched their mainnet. 08:24.780 --> 08:27.160 And it's one that I've practiced some validation on. 08:27.160 --> 08:33.240 And they're doing some interesting things with price discovery and stuff of assets. 08:33.240 --> 08:41.540 And it's really interesting because you're seeing stuff use some of the Cosmos SDK, where a lot of these things that we're talking about are going to be applicable. 08:41.660 --> 08:51.300 Other chains are now forking those off, and kind of doing their own things with it, creating great new products and great new services for end users that we can all jump in and play with. 08:51.300 --> 09:08.320 So if anybody after this talk does have interest in validating, I would definitely go out to some of these networks, look at their documentation, spin up some virtual instances, or if you have some spare metal spare hardware, try jumping on their test net and seeing how this all works out. 09:11.820 --> 09:16.000 With Cosmos specifically, we were talking about value. 09:16.300 --> 09:23.520 And we can see here some stats from one of the block explorers for the Cosmos Hub. 09:23.720 --> 09:29.580 And right now the atom token, I think, is at 415 at the time of this presentation. 09:29.780 --> 09:33.940 It might be 417 now, it might be 413 in a second from now. 09:33.960 --> 09:37.100 They've got about 3 million blocks on their current chain. 09:37.100 --> 09:38.740 And this is really what's important. 09:38.740 --> 09:43.660 So this is showing how many of the atoms are bonded or staked back into the system. 09:43.660 --> 09:45.620 Right now it's at 71%. 09:45.620 --> 09:52.240 So 71% of all the atoms that are out there are being staked and held securely for these validation services. 09:52.520 --> 10:04.160 Now, when we take a look at the current price and we equate that out, that's about $760 million currently that's being bonded behind this network trying to protect it and prop it up. 10:05.700 --> 10:08.940 Another important number here is inflation, which you'll see at 7%. 10:08.940 --> 10:12.000 Remember that one because we're going to talk about it here in a couple slides. 10:16.510 --> 10:21.530 So we now have validators set up in this system that are validating transactions. 10:21.810 --> 10:30.330 And these validators could be run by you, they could be run by an entity, or some other third party or trust. 10:30.330 --> 10:32.830 Anybody can run a validator. 10:32.830 --> 10:38.370 But the other thing you can do as a user is you can stake your own atoms or tokens with a validator. 10:38.630 --> 10:42.510 And the reason you would want to do that is to additionally profit. 10:42.510 --> 10:52.130 So this is kind of similar to the whole staking pool concept with mining, where I don't want to run all the hardware and I don't want to take on the risk. 10:52.130 --> 10:54.650 But I do want to contribute and benefit in some way. 10:54.650 --> 11:06.650 And this is a great way that anybody can go and participate in these networks and contribute to these validators, and also potentially profit and reward from it at the same time for contributing to that network. 11:07.710 --> 11:12.250 There is some things with validators where they're going to get block rewards. 11:12.250 --> 11:19.830 And the reason you would want to risk your money with a validator is that block reward, there's an inflation component here. 11:19.830 --> 11:22.770 And that was the 7% that we were talking about. 11:22.910 --> 11:28.590 So there's an incentive for validators and holders to stake their tokens over time. 11:28.750 --> 11:32.650 You can see here the average yield in this example is 9.46%. 11:32.650 --> 11:40.550 So if you compare that to a 7% inflation rate, you're up to 2.4% at that point. 11:40.550 --> 11:42.910 So it still makes sense to bond tokens into this. 11:42.910 --> 11:54.750 And as long as that average stays above the inflation rate on the network, it's profitable for you to be bonding your tokens over time and getting those interest tokens back, essentially. 11:54.750 --> 12:05.710 So you would think of this as kind of like a savings account or a high account where you have to manually go back in and re-delegate your tokens to a validator, pull out your rewards and re-delegate them. 12:05.710 --> 12:09.390 And then over time, your reward amount is going to grow. 12:09.710 --> 12:11.190 And just some statistics here. 12:11.190 --> 12:16.250 So for 686 atoms monthly, you would be getting a return of 5.41. 12:16.290 --> 12:23.290 So at $4, you're looking at $20, $25 for basically just doing passive income at that point. 12:28.020 --> 12:36.320 Now, when it comes to validators, another thing that incents them outside of the inflation component is slashing. 12:36.320 --> 12:39.560 And when we're talking about this, we're not talking about the guy from Guns N' Roses. 12:40.260 --> 12:48.000 We're actually talking about penalizing participants in the network for violating certain conditions. 12:48.000 --> 12:51.320 And this can be in the network, this could be in the consensus layer. 12:51.320 --> 13:06.360 But it's a huge financial incentive for validators to play nice because anytime you introduce these types of systems, you're going to have hackers, you're going to have bad actors, you're going to have people trying to gamify it and say, you know, how can I increase my rewards? 13:06.360 --> 13:07.880 How can I knock you offline? 13:07.880 --> 13:13.480 How can I trick the system or the state of the system to go in there and profit for me? 13:13.480 --> 13:16.860 What's the incentivization to not do that? 13:17.020 --> 13:19.520 And slashing is a great component for that. 13:19.600 --> 13:27.460 And there's two ways that you can really get slashed when you're talking about Cosmos and proof of stake coins currently. 13:28.220 --> 13:31.020 They are downtime and double sign. 13:31.020 --> 13:33.600 Both start with D, pretty easy to remember. 13:34.320 --> 13:44.720 Downtime has some conditions where out of the last 10,000 blocks, if you're down for more than 95% of that, you're risking a downtime slashing event. 13:45.000 --> 13:49.960 You can see here in this block explorer, these blocks were being sent and everything was great. 13:49.960 --> 13:54.760 And then at some point, something went offline and now we're missing them. 13:54.760 --> 14:11.400 So if that continued for the next 9,500 blocks or whatever, and we are not able to get our validator troubleshot or figure out what's going on or move to another system, we risk incurring a downtime event and having a slashing event occur. 14:11.660 --> 14:14.640 The other big one is a double sign. 14:14.780 --> 14:29.920 And this is an example of a double sign that happened on the Cosmos mainnet where what ended up happening was the private key for the validator was used to sign two transactions, sign the votes for them for the same block. 14:29.960 --> 14:32.760 And that was pushed out, that was detected. 14:32.760 --> 14:38.440 And as a result, that validator was jailed because you can't double sign on your vote. 14:39.260 --> 14:44.400 So once you're jailed for a double sign, unfortunately, you are in jail. 14:44.400 --> 14:46.100 There is no getting out. 14:46.100 --> 14:48.940 Your validator will not be able to rejoin the system. 14:49.660 --> 14:52.460 You've screwed up and you've screwed up in a major way. 14:55.770 --> 15:01.930 When it comes to downtime, the amount of stake that can be slashed is very minimal. 15:01.930 --> 15:12.870 It's not a huge amount, but depending on how many atoms are staked in your validator by yourself as the validator, what you're doing is called self-staking. 15:12.950 --> 15:20.670 So a validator may have a million atoms and stake their million atoms to say, I believe in my system, I believe in myself, I'm going to stake all my atoms with it. 15:20.670 --> 15:22.750 But as I said, users can also stake. 15:22.930 --> 15:29.030 So when these things, when these slashing events happen, not only is the self-stake affected, but the user stake can be affected too. 15:29.130 --> 15:37.590 So as a validator, if you do have a slashing event and you lose stake, you're losing money for your end users, tokens, and digital value for your end users. 15:37.590 --> 15:42.410 They're not going to be very happy with you or want to contribute to you in the future. 15:43.630 --> 15:49.130 The penalty for that, you miss your rewards while you're in a... and that goes for a 10 minute period. 15:49.650 --> 15:54.790 And with these types of events, usually you only get hit for the first one. 15:54.790 --> 15:57.130 It's not reoccurring. 15:57.390 --> 16:00.730 They put, I think it's called a tombstone. 16:00.730 --> 16:04.070 I know there's a tombstone here for double signing, but I forget what it's called for downtime. 16:04.070 --> 16:04.930 I apologize. 16:05.210 --> 16:10.310 But it's basically kind of like a buffer to say that we hit you for downtime. 16:10.310 --> 16:13.650 We're not going to hit you again if you're in a certain period. 16:13.790 --> 16:28.310 So if you are being slashed for downtime because your server's down or you're having some sort of redundancy issue, it's still within your best interest to get that up as soon as possible because these things are going to start to get costly over time. 16:28.570 --> 16:35.310 But there is some protection there for end users so that it's not just going to consistently stack over time unless certain conditions are met. 16:35.890 --> 16:37.710 With double signing, it's huge. 16:37.710 --> 16:38.450 It's 5%. 16:38.450 --> 16:48.310 So if you do have a million dollars in Atom on your validator and taking a 5% slash stake, you guys can figure that out and see how much money that is, right? 16:48.370 --> 16:50.690 You also hit an automatic unbonding penalty. 16:50.690 --> 16:58.990 If you double sign, and it also incurs a three-week unbonding period where those Atoms cannot be touched while they're being unbonded from the validator. 16:59.330 --> 17:05.630 I put the little tombstone here for the RIP because if you double sign, you're done. 17:05.630 --> 17:07.770 You're not going to be able to move forward with that validator. 17:07.770 --> 17:20.390 You're either going to need to re-delegate all of your stake to another validator or allow your users to unbond all their stake and move forward with somebody else at that point. 17:20.390 --> 17:21.390 You're out. 17:25.660 --> 17:35.120 Because of all this, you want to make sure you're doing your due diligence when you're either staking with a validator or thinking about running one yourself. 17:35.120 --> 17:37.380 These are all points you should be thinking about. 17:37.480 --> 17:44.300 So it's recommended from my experience that you want to diversify at minimum probably three is a good bet. 17:44.300 --> 17:45.800 I don't want to say best practice. 17:45.800 --> 17:50.620 I usually like to stick the three to five different validators for different tokens. 17:50.620 --> 17:56.240 The reason being is that we're in a decentralized system and we should be decentralizing as much as possible. 17:56.240 --> 17:58.060 I'm also mitigating my risk. 17:58.060 --> 18:08.520 So if one of those three or one of those five validators do incur a slashing event or a downtime event for whatever reason, I've mitigated my risks where I'm not losing all of my funds at the same time. 18:09.400 --> 18:15.860 You want to do some open source research, some open source intelligence and look up the teams behind them. 18:15.860 --> 18:17.840 Look up the history of the validator. 18:17.840 --> 18:20.180 So what does their history look like? 18:20.180 --> 18:21.280 Is it clean? 18:21.280 --> 18:30.600 Do they have a lot of downtime or maybe a misconfiguration where every 99 out of 100 blocks they're not signing for some reason? 18:30.600 --> 18:32.760 Well, why is nobody noticing that or fixing that? 18:32.760 --> 18:35.280 Are they really paying attention to their system? 18:35.540 --> 18:37.280 Have they been jailed before? 18:37.280 --> 18:40.160 So it is possible to go to jail and get out of jail. 18:41.180 --> 18:43.960 As we said, if you double sign, you're done. 18:44.120 --> 18:54.500 But for minor events, you can get out of jail if you accept, if you levy the penalty at that point and you wait for the time that you're jailed. 18:54.500 --> 19:05.720 So if they have a history of jail occurrences that you're not comfortable with, you shouldn't be giving them your money because this is essentially what you're doing is you're letting them hold digital value on your behalf, essentially. 19:06.620 --> 19:13.720 And then the other thing is a lot of teams have gotten really good in the past year with putting a presence out there. 19:13.720 --> 19:29.920 So there's people and there's corporations that are doing this as a business, right, where they validate on all the chains I listed earlier and more, because there's a ton of them out there doing their own little niche with different things with proof of stake and their chains and their use case. 19:30.060 --> 19:34.180 But some of those teams are getting really good with publishing a social presence. 19:34.180 --> 19:43.620 So they're listing their teams on web pages, or they're showing you with chains, or they're actually showing you pictures of here's what our setup looks like, our validator setup looks like. 19:43.780 --> 20:00.360 Be careful when you do that, because it's very similar to the ICO craze of 2017, where some of the team pictures, if you go and reverse Google search or Tanai reverse search those, you're going to find that those are some LinkedIn picture for somebody that works in like, 20:00.360 --> 20:06.440 you know, aviation or something like that, that has nothing to do, and they're just stealing your information to try to fleece you. 20:06.440 --> 20:08.840 So do your due diligence, don't get scammed. 20:10.460 --> 20:12.380 The reason you want to do that. 20:12.380 --> 20:14.720 So this is a personal example. 20:15.220 --> 20:21.560 I was validating on a testnet for a coin that was not released yet. 20:21.920 --> 20:37.800 And because it was a testnet, and because I wasn't taking it as seriously as I should have, I ended up using some spare hardware, and ended up experiencing like a 36 hour power outage. 20:37.800 --> 20:44.480 This spare hardware was hooked up to a Comcast cable modem, no redundancy across the board. 20:45.040 --> 20:59.480 And during the power outage, because I'm focusing on saving my food and a toddler, and what are we going to do, supposed to be 95 degrees tomorrow, and all of those things that you have to deal with when life hits you and kind of shit hits the fan. 21:01.140 --> 21:04.920 Those are the things that you're not going to be thinking about your validator, right? 21:04.920 --> 21:14.880 You're not going to be thinking about the thing that you've set up in the dressing room that just runs off the modem that your wife doesn't like, or your kid resets all the time. 21:15.060 --> 21:18.080 So unfortunately, I'm a victim of this. 21:18.080 --> 21:25.720 And thankfully, it was only on testnet, but I ended up losing 10,000 coins because I did not have a redundant infrastructure set up. 21:25.780 --> 21:32.620 Now, if that was to occur on mainnet, and I would say a validator, I would have just taken a big PR hit and lost my users funds. 21:32.620 --> 21:37.120 In my instance, thankfully, it was only testnet, and it was a great lesson learned. 21:38.500 --> 21:42.140 And this is actually the picture of my validator. 21:42.140 --> 21:53.540 So this was an Intel NUC that I had a spare machine that I set up, and I said hooked it to a cable modem with a single Ethernet line, single power line, right, single SSD in here somewhere. 21:53.720 --> 22:00.140 So no redundancy, no infrastructure, not even really any status lights to tell me what's going on. 22:00.140 --> 22:11.740 So if you are thinking about running a validator and you find that the people you're thinking about bonding your tokens with are doing this, you should run away as fast as possible. 22:11.880 --> 22:15.900 Great for development and great for testing, should never be done in production. 22:16.500 --> 22:18.400 That's what you want to do in production. 22:18.400 --> 22:22.300 So we want to see the data center, we want to see the cool blinky lights. 22:22.360 --> 22:38.640 The reason you want to do this is you're getting cooling here, you're getting heat and power and network and all the good warm fuzzy feelings that go into your hosting bill when you have to pay that at the end of the month to justify all the servers and things that are being filled in these racks. 22:38.920 --> 22:51.360 So you know, stand on the shoulder of giants, use the experts, use the redundant infrastructures that have already been built and hopefully been tested correctly to host these types of things and do validation. 22:54.770 --> 22:58.810 We want to implement all like the standard best practices. 22:59.830 --> 23:03.330 When it comes to redundancy, you want to hit all the... 23:05.290 --> 23:13.110 Monitoring, we're going to jump into with a couple different solutions that can actually alert you when you do have the 36-hour power outage. 23:13.550 --> 23:25.710 There's a lot of administration concerns too with different configurations and things like Cosmos has published a security update for its SDK and it's an emergency upgrade. 23:25.830 --> 23:27.690 How are we going to do that, right? 23:27.690 --> 23:32.690 Do you test the data center tech with going in and upgrading that binary? 23:32.690 --> 23:36.730 Or do you have a subject matter expert on staff that can handle that? 23:36.770 --> 23:42.350 Do they know the difference between go path and your environment variable path and all of those types of things? 23:42.350 --> 23:50.510 Because all of those are considerations that you're going to hit and speed bumps that are going to occur while you're working on these types of systems. 23:50.730 --> 23:58.650 You need something to do some sort of repo monitoring, be able to check when upgrades and new checkpoints and releases are released. 23:58.650 --> 24:02.030 And then deployment is another big consideration for administration. 24:04.950 --> 24:09.230 There's a lot of different schools of thought on deployment. 24:09.450 --> 24:16.050 And we'll jump into some of the use cases as to why some are better than the others. 24:16.050 --> 24:21.450 But it's definitely a concern when you're architecting and administering these types of systems. 24:21.550 --> 24:33.330 And then we want to jump into a lot of the security components with controls and the different aspects on how you're mitigating your risk and how you're delegating access and who knows what and why do they need to know it. 24:37.510 --> 24:50.830 The keys that actually work on a validator when we're working on security, there's two keys specifically for Cosmos that need to be protected and thought about with all those prior best practices for redundancy and backups and things of that nature. 24:51.050 --> 24:55.250 And that's your tenderment key, which is actually doing your consensus voting. 24:55.490 --> 25:00.370 And then the application key, which derives your validator addresses and things. 25:00.370 --> 25:02.590 That signs your actual transactions. 25:02.610 --> 25:08.190 But both of those need to be considered when you're building these systems on, one, where does it live? 25:08.230 --> 25:10.290 Two, how did we generate those? 25:10.290 --> 25:12.970 Did we do it in a secure way? 25:12.970 --> 25:15.810 And I'll show you some ways that you guys can do that. 25:15.810 --> 25:18.550 Three, where are we backing it up? 25:18.550 --> 25:22.430 So does it just exist on some EC2 instance somewhere? 25:22.430 --> 25:29.810 Or do I have key material written down in a waterproof, fireproof safe that's access control? 25:30.310 --> 25:37.790 A lot of that stuff needs to go into the thinking and mindset and paradigm of how you're protecting the system and how you're doing it. 25:37.790 --> 25:53.130 And you really want to do it right kind of from the beginning, because if you deploy a validator to determine that I didn't generate my key material securely, you don't want to have to go back and redo all that stuff and re-delegate and try to make sure that people are aware that, 25:53.130 --> 25:57.390 you know, we're validator two instead of validator one now, those types of things. 25:57.550 --> 26:02.890 So when you're building this stuff, kind of encompass all this in your thinking and do it right from the beginning. 26:02.890 --> 26:13.170 Because really what these keys are protecting, I checked CoinCap IO right before this presentation, 1.3 billion market cap for Cosmos Atoms right now. 26:13.270 --> 26:22.830 So for a coin that's still relatively brand new in the scheme of things, we're already at a billion dollar market cap at current day prices. 26:22.830 --> 26:30.250 So if the bull market really, really starts taking off, at that point, these keys are protecting crazy amounts of value. 26:30.250 --> 26:44.270 Are you comfortable leaving that type of information sitting on a server that, you know, the data center tech down the street that happened to work on the day that you called in and said, my server's down, can you check the lights on my Dell system? 26:44.490 --> 26:48.730 You know, he's looking at console and he's single usering and things. 26:48.970 --> 26:55.550 While your key material may be protected by local key stores and things like that, what stops him from sniffing memory? 26:55.550 --> 27:00.890 What stops him from imaging the drive or breaking RAID or things like that and then clearing logs? 27:00.890 --> 27:05.710 So it's considerations that you want to think about when you're building those systems. 27:06.250 --> 27:12.310 For redundancy, it's all the great stuff with power supplies and phasing and networking. 27:12.310 --> 27:16.510 And I've seen some validators where they run on a single networking. 27:17.170 --> 27:23.350 So what happens if that network card fails or the cable goes bad, a pair in that cable goes bad? 27:23.350 --> 27:26.670 You're going to be at risk of a potential downtime event. 27:26.670 --> 27:32.490 When it comes to storage, there is a lot of data that goes into running a validator. 27:32.490 --> 27:45.370 A validator essentially needs on itself a copy of the full node, the chain data, or it needs to link to another server a century that has a copy of that full node chain data. 27:45.810 --> 27:56.290 So if you're building in redundancy to say if this instance goes down, once you rebuild your new instance with your new validator binaries, you need to sync all the chain data again. 27:56.370 --> 28:15.050 So if you're in that window, that downtime window, that 95% of 10,000 blocks, which I think is like half a day, and it's starting to get down to the wire, I've seen where we've started binaries and things like that and my validators, you know, they still need two hours to catch up to sync chain data because I didn't have a copy of it anyway. 28:15.530 --> 28:23.450 So I would definitely recommend looking at things like RAID if you're using SANs, doing snapshots and things of that nature. 28:23.550 --> 28:40.930 The Cosmos team has published snapshots over time of chain data where you can go and download those types of things to sync it, but I'd be careful with what sources you're pulling that from because if a bad actor wants to cede bad data to you, you could potentially risk another issue with your validator at that point. 28:41.590 --> 28:51.970 And then there's other some some other gotchas that people don't necessarily think about and like, there's some funny stories and some instances, you're standing in front of the server in the data center, and you're going, I'm here. 28:54.330 --> 28:56.870 I have no way to log in, there's no keyboard, right? 28:56.870 --> 28:59.070 There's no crash cart available in the data center. 28:59.110 --> 29:05.050 So run through some of your BCDR type of policies with like, what do we need in the cage, right? 29:05.050 --> 29:06.390 Do we need spare hard drives? 29:06.390 --> 29:10.730 Do we need installation software for OSs if we have to spin it up? 29:10.730 --> 29:15.790 Do we need config variables to point to a PXE server or to point to a Kubernetes cluster? 29:15.790 --> 29:21.110 However you decide to architect your solution that best works for your information system. 29:21.630 --> 29:31.650 And I note down here, any time that you're running through this type of stuff and architecting it and running through these exercises, you always want to think about the worst case scenario, which is double signing. 29:31.650 --> 29:43.530 And in order to achieve that double sign, it means that those Cosmos keys need to be either duplicated somewhere or pulled into a configuration or have two instances that are running that same material. 29:43.630 --> 29:44.850 So always think about that. 29:44.850 --> 29:55.150 That's kind of the grenade in the mix that needs to always move forward and can never be copied because if it is and it goes live, you're risking that 5%. 29:55.150 --> 30:01.670 I put this slide in here just because I hate DNS. 30:02.150 --> 30:03.830 DNS is a big one. 30:03.830 --> 30:20.910 And you want to be careful with this because when you start to build validator infrastructure and you start to do additional security controls and different servers with copies of chain data, DNS is one that may get you that you don't even realize because nobody ever thinks it's DNS, 30:20.910 --> 30:22.270 but it always is. 30:22.270 --> 30:23.790 So make sure your host names are correct. 30:23.790 --> 30:31.170 Make sure you have redundancy upstream in your network stack, not just the cables, even the routers in the rack. 30:31.170 --> 30:38.170 If you're using things like Netgear or Cisco equipment, do you have that in a stack configuration? 30:38.170 --> 30:40.230 Do you have an AB failover? 30:40.350 --> 30:48.210 If your switch or your Cisco ASA license expires and you start getting throttling or your feature stops off, what happens into your system? 30:48.210 --> 30:51.590 So those are all things to simulate and think about when you're building that. 30:51.590 --> 30:54.570 And DNS is one of the ones that I don't want people to forget about. 30:58.680 --> 31:03.500 So now we need to monitor all this stuff with our validators. 31:04.120 --> 31:06.200 We've figured out how it works. 31:06.200 --> 31:07.180 We've got some coins. 31:07.180 --> 31:08.600 We've got a system set up. 31:08.600 --> 31:12.280 Now, how do we avoid that 36-hour outage? 31:12.680 --> 31:18.780 So SimpliVC is a company that's out there that does a ton of validation on chains and things like that. 31:18.780 --> 31:21.340 And they release this open source software called Panic. 31:21.340 --> 31:24.400 And I'm a big, big fan of open source software. 31:24.620 --> 31:31.260 But what this will do is this will actually monitor things from the tendermint layer and from the local server layer. 31:31.540 --> 31:38.240 And it'll do email, telegram, through Twilio integrations, it's not responding. 31:38.420 --> 31:41.100 And it'll also do the repo monitoring we talked about earlier. 31:41.180 --> 31:53.900 You know, a new version of Cosmos, okay, or Gaia-D or whatever binary has been released, you need to be aware of that so that you can go check the release notes and see if this is something I need to apply now or can I defer that to later? 31:54.440 --> 31:55.400 I love it. 31:55.400 --> 31:58.100 I use it for all my validation stuff. 31:58.240 --> 32:00.340 This is my telegram integration. 32:00.400 --> 32:05.380 And you can see it doesn't just do things like the services down. 32:05.460 --> 32:10.220 It actually tells things like my peers have increased or the node was inaccessible. 32:10.260 --> 32:14.400 If people delegate to my validators, it gets updates on my voting power. 32:14.400 --> 32:19.240 So I know that I've got more power in the system for consensus votes or that there's something happening. 32:19.320 --> 32:22.840 And I need to be looking at my validator because people are delegating to me. 32:22.840 --> 32:23.920 I want to know why. 32:24.300 --> 32:26.220 And then as I said, it gives you peering changes. 32:26.220 --> 32:34.740 So if other people are doing peers dropping off the network, that may be indicative of a network event or a binary upgrade or something else is going on with those other servers. 32:34.740 --> 32:36.340 They're doing some work to that stuff. 32:36.340 --> 32:40.720 And it's good to be aware because we may need to take mitigation efforts. 32:43.450 --> 32:45.430 Prometheus is another great one. 32:45.430 --> 32:49.670 This is all packaged with the current Gaia installation with Cosmos. 32:49.850 --> 32:52.070 It's a very simple config change. 32:52.070 --> 33:04.050 And this is basically kind of like a logging port that opens up a whole bunch of data that's going on in your validator that you can feed into Elk stack or different dashboards or whatever you choose. 33:04.050 --> 33:06.950 But it has really good support for Grafana, for example. 33:07.110 --> 33:16.630 And this brings it into the kind of DevOps infrastructure world of, I want to build dashboards and I want cool graphs and stats and stuff like that. 33:16.850 --> 33:25.410 So if you don't want to do that through necessarily a block explorer, if you want to run something locally, these things are, it's great to turn Prometheus on. 33:25.410 --> 33:28.870 It's great to throw Grafana or some sort of Elk stack monitoring on top. 33:32.560 --> 33:35.020 So we've got our monitoring taken care of and now we're... 33:37.180 --> 33:42.980 So we talked about the data center tech at, let's use AWS, for example. 33:43.840 --> 33:51.560 So you're running a validator in AWS and you've got this private key material on there that's validating transactions and participating. 33:52.260 --> 33:56.320 What stops the data center tech from logging into your system and grabbing your key material? 33:56.320 --> 34:03.440 What stops the hacker from getting on your server and grabbing that stuff remotely if you haven't configured your security groups and your firewalls correctly? 34:04.320 --> 34:07.020 So that's the problem and the solution is HSMs. 34:07.160 --> 34:09.920 And HSMs are great if you guys have never used them. 34:09.920 --> 34:24.180 They're a little bit expensive, but it's very similar to the way that, you know, YubiKeys and things like that where it stores the private key material actually on a hardware device and then signs messages without actually having to expose or reveal that material. 34:24.460 --> 34:28.260 In theory, it's not supposed to allow any private key extraction. 34:28.260 --> 34:37.180 Now there have been attacks and that's not to say that things can't fail in the future, but currently it's not supposed to allow any private key extraction. 34:37.420 --> 34:50.700 So what you would do is when you were generating key material to either set up your validator or your wallet or functioning, you want to generate those keys in a secure way and then store that private key material on a hardware device. 34:50.880 --> 34:56.460 The two I've used is the YubiHSM2 and the Ledger Nano S, which both work very well with Cosmos. 34:56.640 --> 35:01.860 I know they work with MicroTik and they work with a few other other chains that are out there. 35:02.620 --> 35:14.860 But this would actually physically sit into your bare metal server in the data center, in my case, in the dressing room and provide the private key material and signing functions without revealing it. 35:14.900 --> 35:27.840 So if somebody was to get into your server remotely, or if the data center texting their single user in, they're not going to be able to extract this without a whole bunch of different authentication information and, you know, secrets and things that they don't have. 35:27.840 --> 35:32.320 And then the actual physical hardware security capability of the device too. 35:33.180 --> 35:47.060 One thing I will note to be very careful with this stuff is that the YubiHSM in particular has a very small profile and in order to firmware reset this device, I think it's like hold this front side for 10 seconds or something. 35:47.160 --> 35:59.200 So if you're in a data center working and you're failing over from server A to server B because you're doing maintenance and you pop your HSM out, be careful how long you're holding it and where you're touching it on. 35:59.200 --> 36:05.460 Because if you wipe that private key material, that's not going to be recoverable and you better hope to hell you have a backup. 36:06.120 --> 36:22.800 The Ledger works really great with it too and it's got the little graphical display and same thing, you know, there's always security vulnerabilities out for hardware wallets but people are keeping up on those and they're auditing them and it's still a better layer of defense than just leaving the key material on the server itself. 36:28.540 --> 36:33.980 Another thing you want to do from a security aspect is you want to set up Sentry architecture. 36:34.660 --> 36:40.940 And a Sentry is basically a copy , a full node of the chain that you're validating on. 36:41.260 --> 36:47.740 So you could run your validator straight out to the internet and start broadcasting these things. 36:47.740 --> 36:53.500 But it's better to put it behind another layer of obfuscation and network connectivity here. 36:53.740 --> 37:07.280 And what we're doing is we've connected public internet that was gossiping a lot of information about our transaction validator itself and we've moved it up a layer behind a Sentry layer. 37:07.280 --> 37:15.520 And when we do that, this actually enables us to now load balance and have some redundancy at the Sentry layer where we can deploy multiple Sentries here. 37:15.780 --> 37:18.720 So I've used two in this example, but it can really scale. 37:19.920 --> 37:22.260 I don't know if there's any technical limits on that. 37:22.260 --> 37:23.480 That would be a great follow-up. 37:23.480 --> 37:27.640 But as far as I'm aware, I haven't seen a technical limit on how many Sentries you can run. 37:28.140 --> 37:35.440 But the whole point of this is that nobody's going to be able to get to your validator directly to any public IPs or anything like that. 37:35.440 --> 37:39.960 You're going to have private communication between your validator and your Sentry itself. 37:39.960 --> 37:53.420 And I've seen this done many different ways where this validator could be inside its own VPC and you have IPsec tunnels talking out to your Sentries, which are in its own VPC that consists of two or three instances. 37:53.420 --> 38:01.140 This could just be two or three EC2 instances that are talking to each other through private IP space. 38:01.140 --> 38:04.420 This validator may not even be the same location or site. 38:04.420 --> 38:11.160 It could be bare metal talking up to cloud Sentries somewhere, bare metal Sentries in another data center across the country. 38:11.240 --> 38:18.660 I wouldn't recommend that for time to live and packet communication, but it is technically possible. 38:18.860 --> 38:22.020 This stops people from denial of servicing your validator. 38:22.020 --> 38:26.940 It also gives you some redundancy here for failures with your different Sentries and things like that. 38:26.940 --> 38:30.800 And as I said, it protects your communications between your validator and your Sentry. 38:30.800 --> 38:39.520 So the validator would get the information from Sentry, sign transactions, broadcast it out through the Sentry out to the public internet. 38:39.560 --> 38:43.540 So from an attacker standpoint, they would only get as far as this layer. 38:43.540 --> 38:45.860 They wouldn't get to the stuff that actually matters. 38:49.680 --> 38:52.440 Other considerations with your security architecture. 38:53.700 --> 38:56.280 We never want to double sign. 38:57.420 --> 39:11.760 A lot of cloud providers offer auto scaling, and I think that's great for Sentries, but I would avoid it for my validator infrastructure, because you start to get into instances of where does my key material live and how do I migrate? 39:11.760 --> 39:20.040 And if server A still has it active and server B spins up and I've copied my key material and that service starts, I've now double signed, right? 39:20.040 --> 39:21.640 And I'm in it's game over. 39:22.280 --> 39:29.860 Any automation and orchestration pieces with the Sentry, you want to think about the blockchain data and the state data. 39:29.860 --> 39:31.120 How are you moving that? 39:31.120 --> 39:32.720 How are you verifying that? 39:32.720 --> 39:36.640 How are you loading that into new instances if you're doing auto scaling or automation? 39:36.760 --> 39:41.520 Or do your new Sentries have to come up and sync the entire chain before they can be active? 39:42.180 --> 39:45.220 You also need to worry about doing your config updates. 39:45.220 --> 39:58.880 So if I had to have a validator infrastructure and I have two Sentries and add a third, well now I need to go back to all of my existing systems and do some config updates so that I'm now listing that here in my config to say talk to Sentry 3. 39:58.880 --> 40:00.820 Don't just talk to Sentry 1 and 2. 40:00.980 --> 40:09.780 So when you're building the automation, if you have Sentries and things in that infrastructure, you need to kind of backfill all of those changes to stuff that you've already implemented. 40:10.240 --> 40:13.720 And then Sentries are just like validators because it's a full node. 40:13.720 --> 40:25.940 If there's security updates for the binary, if there's issues with it where it goes down, you need to be doing all the same all the same repo stuff, all the same binary upgrades when it comes to those Sentries. 40:25.940 --> 40:35.320 So if you're scaling your Sentry architecture out horizontally to 40 Sentries, just realize you need to do updates to all of those Sentries when things happen. 40:37.950 --> 40:40.970 Then there's some final other considerations here. 40:41.570 --> 40:48.230 As I said earlier, there's a lot of different mindsets on should you virtualize validators? 40:49.810 --> 40:57.010 And I've gone back and forth on this and I feel like validators should be bare metal. 40:57.010 --> 41:10.450 And while it is technically possible to deploy like an EC2 instance or Google's GCP instance and have it validate with key material on that validator, that's not going to be the most secure, optimized, redundant solution that's out there. 41:10.450 --> 41:16.810 And if I was considering and looking at two validators on where I wanted the funds, I would not pick that one. 41:16.810 --> 41:19.250 I would go with the person that considered some of this stuff. 41:19.250 --> 41:37.530 And even if they were using one or two bare metal servers in like a hot or cold fail, to me, I'm a little bit more comfortable with that than somebody that's doing pods of validators that's moving key material back and forth, because all that needs to happen is a weird edge case or anomaly where you're going to hit some sort of double signing scenario, 41:37.530 --> 41:37.830 right? 41:37.830 --> 41:49.430 And unless you have your deploy scripts tweaked so good with so many tests and error catching and then, yeah, it's scary when you look at the amount of money with that 5% double slash. 41:50.770 --> 41:53.810 There's other things you want to consider with minimum config values. 41:53.810 --> 42:04.290 So I've done some attacks and I've seen some defense on sentries and things with spam dust transactions where people can try to tax your validator, your sentry resources. 42:04.290 --> 42:13.850 And the reason they would do that is to try to edge because maybe they run their own validator or you were jailed and they had money with you or tokens with you, some sort of value. 42:14.790 --> 42:22.950 So there's some minimum config values that validators can tweak and sentries can tweak to say, ignore more of the spam and only give me the good stuff. 42:24.070 --> 42:28.350 There's an art to it and you got to be careful because you want to be a good system. 42:28.350 --> 42:36.450 But just know that you can get pretty granular in your validator config with all the different values that are participating at the tendermint layer and the consensus layer. 42:37.410 --> 42:39.410 And that goes for denial service too. 42:39.410 --> 42:47.850 And then this last one here that I don't think a lot of people... and I've actually seen a lot of wallet software and validator software getting better with this. 42:48.770 --> 43:04.510 People get in the habit of validating and starting to get rewards and they'll pull out all of their rewards and say, re-delegate every single token I have, all my rewards in my wallet, send it back to the validator because I want that compounding interest. 43:05.310 --> 43:07.470 And what'll end up happening is it'll work. 43:07.470 --> 43:10.070 They'll still sign the transaction and send it out to the validator. 43:10.130 --> 43:12.350 And then they literally have zero in their wallet. 43:12.690 --> 43:15.930 So you guys, you know, you're familiar with ETH and other systems. 43:16.030 --> 43:20.390 If you send all your ETH out and you try to do an ERC20 transfer, what's it going to do? 43:20.390 --> 43:23.590 It's going to complain and say, you don't have enough ETH for the gas for the transaction. 43:23.770 --> 43:32.710 So be careful with that because if you re-delegate all your rewards thing that's in that wallet, you're not going to have enough gas to do any more transactions at that point in the future. 43:32.710 --> 43:39.530 So you're going to have to go procure atoms from somebody else or wait until you have enough rewards to get enough to be able to start to continue to do transactions. 43:39.530 --> 43:51.410 So I've gotten in the habit of like leaving half an atom or an atom or whatever denomination or whatever token I'm using, leaving one or two of those in the wallet, just so I know that I'm always going to be covered for transactions in the future. 43:51.410 --> 43:56.930 And that stuff's getting way better with doing calculations and fees and even notifying users. 43:56.930 --> 43:59.670 Cosmos station is a mobile wallet that I love. 43:59.690 --> 44:07.470 And it'll come up and actually warn you and say like, hey, you shouldn't be delegating max, like you should be delegating 90% of max or whatever you want to do. 44:09.410 --> 44:16.270 So with all those things we talked about with redundancy, I wanted to ask the question, should you be running a validator? 44:18.950 --> 44:23.370 And in my opinion, no, not for the validator. 44:23.570 --> 44:32.910 If centuries and things like that, where it's not actually signing transactions or dealing with value, that is great for scaling up and auto scaling. 44:32.910 --> 44:40.030 And as long as you can take some of the data considerations and things I was talking about earlier into that system, it's amazing for centuries. 44:40.030 --> 44:43.490 I fully recommend GCP, Amazon, DigitalOcean. 44:43.490 --> 44:49.230 It takes the onus off you and it gives you all of those redundancy and good, warm fuzzies. 44:49.270 --> 44:57.750 But for the validator, I feel like that's something that you should be controlling a little bit more manually doing your own internal orchestration and things like that. 44:58.050 --> 45:03.730 You want to be careful about where you're hosting that, who has access to it, what is their security posture? 45:03.730 --> 45:06.050 What is their uptime in their SLA? 45:06.050 --> 45:12.450 If their SLA is more than a day , that's not good for a validator because if you have a downtime event, you could be slashed. 45:12.570 --> 45:16.470 All things to think about when you're negotiating with your service providers. 45:17.230 --> 45:21.850 The last thing for best practices I wanted to talk about was the cryptocurrency standard. 45:21.990 --> 45:27.410 And this was more in line with what I talked about last year here at the Blockchain Village. 45:27.410 --> 45:39.050 But it's basically a standard that's amazing that was presented by C4 that dives into a lot of different security controls and aspects that one can implement in a cryptocurrency information system. 45:39.990 --> 45:49.910 It gives you the guidance, it gives you a control and different levels from level one to three to say, here's the very basic control I can put into place versus level three. 45:49.910 --> 45:54.210 Here's the ultimate paranoid control and kind of give you an example for that. 45:54.230 --> 45:56.310 One would be the key generation. 45:56.910 --> 46:03.690 So we were talking about keys earlier with the keys and which ones were both for consensus and which ones do transactions. 46:03.930 --> 46:07.870 And one of the things of the cryptocurrency security standard is how are those generated? 46:08.310 --> 46:10.870 What was the device that you generated them on? 46:10.870 --> 46:12.670 What was the entropy device? 46:12.670 --> 46:18.410 Did you do it on your network connected computer that you do work and Netflix and torrenting and all that stuff on? 46:18.410 --> 46:23.090 Or did you do it on an air-gapped machine with a secure clean room type of setup? 46:23.210 --> 46:24.610 Did you back it up correctly? 46:24.610 --> 46:28.750 So level one may say, I made a backup of the key, I'm good right now. 46:28.750 --> 46:30.110 It's on paper somewhere. 46:30.330 --> 46:39.770 But level three would say, that backup needs to be sealed, it needs to be fireproof, it needs to be waterproof, environmental proof, it needs to be access controlled. 46:39.930 --> 46:45.070 So as you kind of go up the stack here, there's different requirements and different levels of security and paranoia. 46:45.190 --> 47:00.410 And while you may not be able to use the cryptocurrency security standard to certify your information system, depending on how it works or what you're doing, it's a great framework to check your validator against for different levels of compliance. 47:00.430 --> 47:05.770 And it also gives you some really good ideas for things like data sanitization policy. 47:05.770 --> 47:10.370 So if you are spinning instances up and down, where are your logs going? 47:10.370 --> 47:19.330 Do you have a process for clearing all of that stuff correctly and securely so that nobody can run forensics on it if they wanted to profile you or get more information? 47:19.870 --> 47:26.350 So there's a lot of great things in here that I think definitely apply to that information system, even if it's not every single one of them. 47:28.050 --> 47:29.910 So what's next for the future? 47:29.910 --> 47:34.550 And this is really kind of exciting for me and validators and Cosmos. 47:34.690 --> 47:37.350 IBC is right around the corner. 47:37.370 --> 47:51.890 And I just took place in Game of Zones, which was a attack, defend scaling competition that the Interchain team and Zacky Mannion and Jack Sampling threw. 47:51.890 --> 47:56.710 So we had a whole bunch of different validators and people that were taking place in this. 47:57.170 --> 48:00.490 And as you can see in this graph here, these were all the participants. 48:00.710 --> 48:04.190 And they were connected to a main hub, a main Cosmos hub. 48:04.190 --> 48:10.010 So this is very much indicative of the wheel and spoke infrastructure that I think everybody's kind of used to. 48:10.010 --> 48:18.830 But this was to theoretically prove that all of these people could connect into the main hub and get them prepared for the next thing, which is the IBC protocol. 48:19.110 --> 48:26.070 And when we're talking about the IBC protocol, we're talking about validating on chains that we're not running necessarily. 48:26.170 --> 48:29.490 All we're doing is syncing state with them through light clients. 48:29.730 --> 48:41.290 So that if Alex chain, or I'm sorry, if Bob chain, for example, wanted to hold Alice coin, Bob and Alice's two chains can now talk together through a light client through IBC. 48:41.290 --> 48:44.250 So now we're breaking away from that hub and spoke model. 48:44.250 --> 48:48.730 And our validators can now choose who they want to talk to and where they want to validate. 48:48.730 --> 49:00.430 So you're seeing here the very beginning of this, where different zones are kind of breaking off into their own sovereign nations and saying, I want to validate on this chain, or I want to participate on this chain in addition to my own. 49:00.610 --> 49:13.090 And this is the magic where I was talking about earlier for validators, where when they go to do the rewards, they could be validating on Bitcoin, they could be validating Litecoin, they could be validating Ethereum and Adams, as well as Ron, Bob, Alice and Jim coin, 49:13.090 --> 49:15.110 because they see some value there. 49:15.130 --> 49:24.350 And when you start to open up that ecosystem and that level of connectivity, there's some really, really cool applications and cases and things that people are talking about and playing with. 49:24.350 --> 49:26.010 And it gets me really excited. 49:27.590 --> 49:29.070 So that's what I have for my talk. 49:29.070 --> 49:34.630 If you guys do have any questions, feel free to reach out to me at my consulting company, Stoner Consulting. 49:34.990 --> 49:36.750 I love talking about this stuff. 49:36.750 --> 49:40.010 I'd love to go into detail if anybody wants to jump more into it. 49:40.010 --> 49:48.850 And I also wanted to throw a special thanks out to Sean Martin, one of the people that's worked with me on validators and infrastructure and system. 49:49.070 --> 49:51.370 And he's really an infrastructure magician. 49:51.370 --> 49:58.270 So if you guys ever have infrastructure questions or scaling issues or anything like that, please let me know and I will get you in contact with Sean. 49:58.730 --> 50:01.190 I'm Ron Stoner and thank you for attending my talk. 50:50.840 --> 50:52.320 I don't know if he can, but I can. 50:57.110 --> 50:59.330 Sorry, I was looking to see if there was any questions. 50:59.330 --> 51:02.970 So I did see one in there, correction, yes, it is called a liveness check. 51:02.970 --> 51:04.070 That is correct. 51:04.410 --> 51:09.730 And that was in regards to the violations, the consensus violation and the liveness violation. 51:09.730 --> 51:10.630 So downtime. 51:10.670 --> 51:12.470 So that is the correct terminology for it. 51:12.470 --> 51:13.550 Thank you. 51:20.790 --> 51:21.250 Cool. 51:21.250 --> 51:24.210 And I don't think I see any other questions unless anyone has any. 51:55.140 --> 51:56.240 I would love to. 51:56.240 --> 51:56.460 Yeah. 51:56.460 --> 51:57.120 Thank you, Jay. 51:57.120 --> 51:58.060 Thank you, Nathan. 51:58.060 --> 52:01.340 Thank you, everybody that put on this village and promoting blockchain. 52:01.340 --> 52:02.180 I love it.