WEBVTT 00:05.020 --> 00:18.560 Hello and welcome to Attacking and Defending Blockchain Nodes, where we're going to learn about a variety of risks and mitigations that we can implement to defend a very core component of the blockchain systems. 00:20.000 --> 00:33.080 We'll try to do a live discussion during the talk while we ask a question, so feel free to join DEF CON's Discord on BCV General Tech's channel, and I'll try to ask questions and respond live. 00:34.380 --> 00:37.400 In case we haven't met, my name is Peter Kaczynski. 00:37.620 --> 00:46.800 I'm a blockchain security engineer at Coinbase, where I spend most of my time breaking a variety of different blockchain systems and smart contracts. 00:46.940 --> 00:59.060 I'm also a writer for the Blockchain Threat Intelligence Newsletter, where I try to cover all the latest news, events, hacks, scams in the blockchain security ecosystem. 00:59.840 --> 01:06.140 Last year, you may have also participated in the Capture the Coin competition, which I helped organize. 01:06.840 --> 01:22.040 My background before deep diving into the cryptocurrency world was as a malware reverse engineer at FireEye, where I looked at a whole lot of APT malware and a penetration tester for the Federal Reserve System, where I was breaking Finance 1.0. 01:24.120 --> 01:29.700 So, just some idea, give you an idea of what we're going to talk about. 01:29.700 --> 01:31.660 I'm not going to drop any old days today. 01:31.980 --> 01:36.520 What my goal here is to educate, to share knowledge, to also learn from you. 01:37.180 --> 01:42.420 With the goal of just making, discussing, like, how can we secure node infrastructure? 01:43.080 --> 01:50.680 So we'll start with a simple exercise on how do we know, what is the state of truth that tells us how many coins we actually have? 01:50.940 --> 01:59.620 We'll go through a variety of examples of how nodes break, and we'll discuss a few of the ways that they could break in the future. 01:59.900 --> 02:09.820 Finally, I'll introduce a solid node security threat model for generic nodes, which you can apply later on for a specific node implementation . 02:09.980 --> 02:24.360 And most importantly, discuss a top 10 list of security defenses that you can implement today, whether you're a standalone operator, or you're working for an exchange, and share lessons learned from trying to secure nodes of my own. 02:25.200 --> 02:29.700 So with that, let's talk about how many coins you have. 02:29.700 --> 02:40.000 So what is, if you're running, if you wanted to, at any given moment, tell me how many coins of any given asset that you have, how do you go about that? 02:40.660 --> 02:52.620 So, well, you'll probably have a piece of software on your computer, on your phone, or a hardware wallet, maybe, which has some specific logic dedicated just for a specific asset. 02:52.620 --> 02:58.200 So if you have a Bitcoin wallet, or Ethereum, or Monero, or some other coin, sorry if I didn't list here. 02:59.840 --> 03:06.940 This particular wallet communicates with a wider network, somewhere to the cloud, and asks a question, hey, how much coins do I have? 03:07.040 --> 03:09.020 What do I actually own? 03:09.460 --> 03:14.000 In case of Bitcoins, you try to enumerate how many outputs that you have unspent. 03:14.440 --> 03:23.980 Ethereum is different in the sense that it's an account-based system, so you have to look up how much of the asset that I actually have on the network. 03:24.740 --> 03:27.640 The thing is, it gets a little bit more complicated. 03:27.640 --> 03:36.440 So if we start getting a little bit more granular, we're going to see that, in fact, we don't communicate with just some unknown entity, abstract entity. 03:36.440 --> 03:40.900 We actually communicate to specific nodes for that specific coin. 03:40.920 --> 03:47.600 So Bitcoin wallet communicates to Bitcoin nodes, Ethereum to Ethereum, Monero to Monero, and so on. 03:47.940 --> 03:56.580 This introduces some complications, because we realize that there are a lot of third-party components involved. 03:56.580 --> 04:05.360 So if we just want to manage our assets, we have to communicate to a whole bunch of software which is maintained by third parties. 04:06.600 --> 04:17.660 So just to give you like a few more very simplistic threat model here, let's talk about components of a friend's coin, just the imaginary coin from the Mario friend's land. 04:17.680 --> 04:28.060 So you have your wallet communicating to a node, communicating to the overall gossip network of some sort, trying to propagate blocks. 04:28.140 --> 04:43.360 You also have your developers who are contributing to the GitHub repository, so you have to rely on maintaining the repository, compiling the node maybe, or downloading binaries there, and so on. 04:43.360 --> 05:01.500 And this all works really well, and until we start thinking that, well, how can we be sure that all the software libraries, all the standalone software infrastructure does not act maliciously, does not drop, modify, or craft transactions to steal your funds. 05:02.640 --> 05:04.320 So let's see how this could happen. 05:04.320 --> 05:05.220 So we have this Mr. 05:05.220 --> 05:10.380 Robot character trying to attack a variety of different points of this threat model. 05:10.380 --> 05:19.540 So we'll start left to right, so we can see that we can attack the network connection between your node and the cloud. 05:19.540 --> 05:29.240 So someone could start dropping blocks, try to eclipse attack your node, so basically to isolate it to make sure it's not communicating to the rest of the network properly. 05:30.560 --> 05:39.680 The same character can also attack your node directly if there are some unknown weaknesses that they can exploit to do some kind of code execution or crash it. 05:39.680 --> 05:42.160 They can prevent you from accessing the network. 05:42.960 --> 05:46.140 At last, we also have to think about developers. 05:46.160 --> 05:51.320 The bad actors out there, they can go after developers like the toad over there. 05:51.320 --> 05:55.840 They can make them commit bad code. 05:56.020 --> 05:59.600 And I'll cover a few real-world examples where this does happen. 06:00.540 --> 06:10.520 So the idea is that it's an invitation to start thinking about the node security as a holistic ecosystem. 06:10.520 --> 06:13.100 So there are a variety of different components which are involved. 06:15.040 --> 06:22.080 So the examples that I just mentioned, they're not theoretical, they're practical, they're based on real-world incidents. 06:22.080 --> 06:31.600 So for example, in May 2014, someone attacked the BGP network used to eclipse attack Doge and Bitcoin miner pools. 06:32.440 --> 06:46.840 So this was, in case you're not familiar with BGP hijacks, it's a way to effectively isolate whole large swaths of network blocks in order to make them route traffic to unexpected directions. 06:46.840 --> 06:55.600 This is something that happened a decade ago, I believe with YouTube, where all the traffic was redirected to to Pakistan, I believe. 06:55.600 --> 06:57.600 And it keeps happening again and again. 06:59.180 --> 07:08.700 This is a threat that we have not observed within cryptocurrency world for a while, but it's something that we have to be aware of and build threat models around. 07:09.400 --> 07:15.100 On the EOS side, after the release of the Node software, they published a bug bounty. 07:15.300 --> 07:21.900 And oh boy, did they receive a lot of good finds there, because one, there was immediately a code execution flaw that was discovered. 07:22.160 --> 07:34.000 And for a while at least, they kept on getting one after another of high-value bounties submitted to them, which included denial of service, code execution, and similar. 07:34.580 --> 07:39.140 Finally, in 2019, Monero website was compromised. 07:39.360 --> 07:43.800 The binaries for the wallet were backdoored to steal clients' funds. 07:43.800 --> 07:47.900 So this is an example of how we have to look out for the developer. 07:47.900 --> 07:49.160 Where are you getting that software? 07:49.160 --> 07:51.040 How trustworthy is that resource? 07:51.040 --> 07:52.920 What can you do to verify it? 07:53.780 --> 07:59.680 So this was just over the few years, but I just want to instill in you that this is an ongoing problem. 07:59.680 --> 08:02.600 These are all the different things happening this year. 08:02.820 --> 08:12.280 So in March, we had on the testnet, but we still had a flaw discovered in the way Solana was processing transactions, where it was not doing sufficient validation. 08:12.280 --> 08:16.680 So it resulted in 500 million Solana getting stolen. 08:16.680 --> 08:28.880 Filecoin, again, had a nasty inflation bug, fortunately discovered on testnet, exploited also on testnet, but it resulted in 9 billion Filecoins minted, when only 2 billion could have existed in the first place. 08:28.880 --> 08:37.320 Inflation bugs are particularly nasty because these are the attacks not against individual users, but against the entire coin ecosystem. 08:37.820 --> 08:40.720 Tendermint is a library behind a variety of projects. 08:40.720 --> 08:46.080 It's a consensus mechanism used in projects like Cosmos, Oasis, and others. 08:46.720 --> 08:54.860 A denial of service vulnerability was discovered there when parsing specially crafted invalid blocks, which basically crashed the nodes. 08:54.860 --> 09:04.260 So if this was successfully executed in the wild, Cosmos was likely not vulnerable to this because they didn't upgrade just to that particular vulnerable version. 09:04.320 --> 09:10.920 But if that was the case and the vulnerability was maliciously discovered, then it would result in a network halt. 09:11.520 --> 09:21.160 The last example, and probably the most critical and interesting one, both from the impact and the way that it was handled, is Ravencoin. 09:21.160 --> 09:26.520 Ravencoin had an inflation bug discovered just last month in July. 09:27.440 --> 09:43.640 The inflation bug vulnerabilities basically allow someone to print money out of thin air, which is extremely dangerous because it allows an attacker just to profit and sell value to exchanges, and it basically impacts every single user of the coin. 09:43.640 --> 09:49.180 What's interesting about this particular vulnerability was that it was actually maliciously introduced. 09:49.180 --> 10:01.260 It was introduced back in January by an account which was seemingly making adjustments to the way the comments were made, on the debugging views were made. 10:02.360 --> 10:13.520 And the core developers unfortunately missed that until the point that someone reported them, a third party, which was building a blockchain explorer for Ravencoin, there's something weird going on in your network. 10:13.520 --> 10:21.220 We're observing someone minting a whole bunch of RVN and also, by the way, selling them on exchanges and profiting from it. 10:21.600 --> 10:36.260 So the Ravencoin team was able to issue an emergency patch, worked with the miners, but it's still 300 million RVNs were minted and exchanged for some other coins. 10:36.780 --> 10:40.200 Something that we have to learn from. 10:42.140 --> 10:53.660 So before we go into the section on how to secure the nodes, how to look out for those vulnerabilities, let's first think about all the different ways that we can break the nodes. 10:54.460 --> 11:02.400 So just to define the scope right away, we're not going to talk about the features of a node, which are documented and designs of the node. 11:02.400 --> 11:11.260 So if something implements proof of work, we're not going to deep dive into 51% attacks, reorgs, other protocol design issues. 11:11.260 --> 11:13.280 We'll accept them as is. 11:13.280 --> 11:21.560 What we're more concerned about is if the node promises to do something correctly, and it fails to do that, that's what we need to pay attention to. 11:22.340 --> 11:25.420 Key management is also outside of this. 11:25.420 --> 11:31.200 So you should probably be running nodes on a variety of tiers. 11:31.200 --> 11:37.780 So if you just want something which replicates the state of the network, you should probably keep your keys on a separate system. 11:38.000 --> 11:39.580 Incorrect usage. 11:39.580 --> 11:50.320 So I assume that if you're running a specific node that you read the documentation, you read everything that is known about it, so you don't necessarily make mistakes. 11:50.320 --> 12:06.920 Like a good example of that is Ripple's TF partial payment flag, where if you do not implement that particular flag correctly, then whenever someone sends you some payment with that flag set, it may appear that you actually receive those funds. 12:06.920 --> 12:14.500 But in reality, it's a IOU, essentially, that I will send you those things eventually, but not right now. 12:14.500 --> 12:17.420 So a lot of exchanges were exploited as a result of that. 12:17.840 --> 12:24.460 Finally, we're not going to focus on the layer two vulnerabilities such as smart contracts, DeFi, and so on. 12:25.080 --> 12:26.540 So what's in scope? 12:26.660 --> 12:29.160 What's in scope are the implementation flaws. 12:29.160 --> 12:36.540 So protocol, software flaws, attack resilience, can the node stand up to sustained attacks? 12:36.920 --> 12:37.660 Infrastructure. 12:37.660 --> 12:44.580 So we're not looking at nodes as just this one thing, and we're protecting just the node itself, and don't care about anything that it's built in software. 12:44.580 --> 12:50.580 We're worrying about underlying OS, we're worrying about the network stack, and anything that goes around it. 12:51.080 --> 12:52.340 Finally, management. 12:52.340 --> 12:57.620 Human factor is, always was, always will be the weakest link in the security chain. 12:57.700 --> 13:04.800 So we will discuss access management, we'll talk about configuration, source control, and all other ops type of things. 13:05.600 --> 13:14.320 So with that, just before we dive into the exercise, just wanted to establish some terminology so we're all on the same page. 13:14.840 --> 13:22.700 The threat model, and this is taken from OWASP's excellent threat modeling cheat sheet if you want to review it and build your own threat models. 13:23.780 --> 13:28.360 Threat model itself is a process to identify potential threats. 13:28.380 --> 13:35.200 So we want to enumerate all the different ways that things can go wrong, we can assess those threats, what is the risk coming from them? 13:35.200 --> 13:45.180 Are they basically going to result in me losing money, or is it just a denial of service, or some other maybe less critical impact? 13:45.880 --> 13:49.400 And also later on, we'll use that to prioritize mitigations. 13:49.400 --> 13:57.280 Why would we spend hours or weeks trying to address a risk which is not really that impactful? 13:58.320 --> 14:01.440 An important thing that we're going to do next, we'll define the attack surface. 14:01.440 --> 14:20.040 An attack surface, think of it as all the different ways that someone can access, can attack your node or node infrastructure, and the threat agents, which are the entities which have enough skill, opportunity, and interest to attack your nodes. 14:20.960 --> 14:32.920 So let's begin with just a really basic system, it's not interacting with any other nodes, it's not managed by anything, just like a really basic core software. 14:32.920 --> 14:52.320 So we have the node itself, which all it does is it's processing the packets, it's processing the transactions, blocks, does verification, it stores the blocks in some kind of storage system, it parses configuration files so it knows how to properly look at those things, 14:52.320 --> 15:05.600 and it may also include a VM interpreter process where it can interpret smart contracts or anything like full-blown, Turing-complete, Ethereum-style smart contracts, or something a little bit more locked down like Bitcoin script. 15:07.080 --> 15:10.840 Now, let's add some more components to that. 15:11.160 --> 15:27.400 In order for node to be useful, it needs to have some kind of management interface, and most nodes implement some kind of RPCE network client, so we have to think about the threats coming from that, who can access this thing, what is it they can do, and of course, 15:27.400 --> 15:37.780 node administrators themselves who are managing the machine, they can shut them down, they can bring it back online, they can reconfigure them and maybe introduce some flaws that I will cover in a moment. 15:38.740 --> 15:44.380 And also, these are just humans, can they be attacked on their own with malware or maybe any other social engineering? 15:45.640 --> 15:59.920 Finally, nodes need to communicate to other nodes to figure out what is the state of the network, and this is where a whole bunch of different things are coming into play, like network infrastructure, other nodes that we need to talk to, and miners. 15:59.920 --> 16:19.040 So network infrastructure includes basically underlying layers of communication for the nodes, so think of BGP and DNS client, DNS services, and just basic routing like TCP connections, UDP, all of that is in scope and needs to be secured as well. 16:19.720 --> 16:24.780 Finally, nodes are just not appearing on your machines out of nowhere, they must come from somewhere. 16:24.780 --> 16:33.000 Unless you are the developer, you probably are relying on third-party repositories, which also in turn rely on third-party dependencies. 16:33.380 --> 16:51.760 So this is something that we need to be aware of, because as I mentioned with Ravencoin, someone was able to backdoor the assets through a very sneaky code change and something that we also need to consider within our threat model. 16:53.540 --> 17:05.180 So with that in mind, join the BCV General Text on the DEF CON Discord channel and I'll ask you questions to just brainstorm together on how we can break nodes. 17:06.760 --> 17:08.960 So the first thing we're going to attack is the core. 17:08.960 --> 17:12.300 This is the node software itself. 17:12.300 --> 17:21.000 This is what processes the blocks, the transactions, and needs to verify them correctly. 17:21.420 --> 17:24.280 If you are this sneaky Mr. 17:24.280 --> 17:27.980 Robot character and you wanted to attack this, how would you do it? 17:27.980 --> 17:31.320 So I'll look in the channel for your responses. 17:39.860 --> 17:43.800 So I have one comment of software supply chain attack. 17:43.800 --> 17:51.780 So this is something that we're going to cover in the software repository attack, but I'm talking about the core itself. 17:51.780 --> 17:57.600 That's the piece of software that does the actual blockchain logic. 17:59.440 --> 18:02.060 BGP attacks to create an eclipse. 18:03.160 --> 18:10.300 So again, so this is something that we have to worry about on the network connectivity side. 18:16.190 --> 18:16.830 Let's 18:22.420 --> 18:22.740 see... 18:23.060 --> 18:30.280 Yeah, so a denial of service against the nodes to crash with the bad blocks, malformed block headers. 18:30.300 --> 18:32.280 So that's exactly that. 18:32.280 --> 18:46.540 So we can craft a packet, think of a tendermint flaw, which basically crashes a node or maybe executes some kind of really nasty code execution attack. 18:47.080 --> 18:51.740 So just to give you some background, this is how I created the model. 18:51.740 --> 18:54.640 So there's a title, protocol vulnerabilities, severity. 18:54.640 --> 18:59.560 So if you can crash or do code execution, know the severity is high. 18:59.760 --> 19:01.920 Probability, I would say medium. 19:01.920 --> 19:04.480 It's just harder to find, harder to exploit. 19:04.480 --> 19:08.220 It requires dedicated time to discover those things. 19:08.340 --> 19:20.300 But the impact is that anything that is not properly processing blocks, transactions, what is the correct fork, messing with the governance and so on, inflation vulnerabilities, this is all in scope for this. 19:20.380 --> 19:23.900 And it will have a severe impact on the node. 19:24.320 --> 19:29.560 However, I want you to consider it's less sexy of an attack. 19:29.560 --> 19:33.820 So we're not trying to find some kind of ode within blocks and transactions. 19:34.160 --> 19:38.640 And consider that nodes actually run all sorts of third-party software. 19:38.640 --> 19:47.500 So for example, Parity Nodes, they run or OpenTheorem now, they run web UIs to manage the thing. 19:47.940 --> 19:56.040 A bunch of node software relies on third-party database libraries and standalone databases to store blocks. 19:56.040 --> 19:58.980 So all of that are fair game to attack as well. 19:59.100 --> 19:59.960 So if you are Mr. 19:59.960 --> 20:10.260 Robot and you wanted to attack your node infrastructure, I'm not going to spend time trying to find old days for the way that you parse packets. 20:10.260 --> 20:23.220 It's a lot easier for me to find out what is the weakest link, what is the common off-the-shelf software that you're running, such as if you're running Nginx and for some reason it's a vulnerable version , I'll try to attack that as well. 20:23.220 --> 20:29.200 I'll try to attack that instead because it probably has more eyes on it and more vulnerabilities discovered so far. 20:29.440 --> 20:41.100 So consider those as your first line of things to lock down and the probability is definitely higher than finding a very directed attack to like in the way you parse blocks. 20:41.840 --> 20:45.580 So let's continue in the same style. 20:45.580 --> 20:47.080 So software repository attacks. 20:47.080 --> 20:51.360 So someone in the channel already mentioned that, you know, the supply chain attack. 20:51.400 --> 21:11.440 So whether you're attacking the software repository itself, so you can, you know, like a Ravencoin example or Monero example, if you can compromise the software repository itself, then you can introduce arbitrary changes, which are maybe different degrees of how hidden they are, 21:11.440 --> 21:14.180 depending how many eyes are looking at the repository. 21:15.460 --> 21:21.520 And again, this can result in fund theft or money printing type of vulnerabilities. 21:21.880 --> 21:30.820 However, what I want to invite you to also consider is not just the main software repository, but also software dependencies. 21:30.900 --> 21:41.380 So an example, not as actually a node vulnerability, but I believe it was Trinity Wallet, which had a, it was backdoored on the source code level. 21:41.880 --> 21:47.020 But the way that it was backdoored was not through the main repository, but through one of the dependencies. 21:47.020 --> 21:59.200 So someone figured out what is the weakest link in the supply chain to that particular piece of software, backdoored that, and then they were able to introduce code that was stealing keys. 21:59.780 --> 22:04.800 So again, probability is high on that as well, and severity is also extremely high. 22:07.240 --> 22:11.680 I mentioned this before, you know, human factor is always the weakest one. 22:11.680 --> 22:14.520 So node administrator, how can we attack that? 22:17.760 --> 22:23.580 Yep, phishing, send phishing email with some kind of malware. 22:24.660 --> 22:28.720 You know, the impact is very high. 22:28.860 --> 22:40.460 They can reconfigure the node, they can introduce a set of nodes, which they essentially creates an eclipse attack where you only communicate to something that is controlled by the attacker. 22:40.680 --> 22:51.080 Certain blockchains, they have a switch you can flip, which basically allows it to accept all blocks and transactions without doing any verifications, like a debugging mode almost. 22:51.120 --> 22:58.760 There's a variety of things that administrator can do and the variety of ways that administrator can get attacked through phishing, malware, social engineering, you name it. 23:00.440 --> 23:04.900 The last, you get the gist of how we're going through different components. 23:04.900 --> 23:08.060 The last one I want to cover is the network attack. 23:08.300 --> 23:12.340 So, correct, so BGP attacks to create an eclipse. 23:13.400 --> 23:25.760 So, exploiting the network infrastructure, you know, all you know, a lot of nodes, the way that they're built, is that they need to first boot up. 23:25.760 --> 23:40.880 When they first come online, they need to communicate to something hard-coded to figure out what is the network truth, like what are the basic trusted infrastructure that I can use to start getting an idea what is the current block state. 23:41.440 --> 23:50.620 A lot of them use protocols like DNS in order to find out what is the first set of nodes that I need to connect to. 23:51.460 --> 23:59.800 And if your underlying DNS infrastructure is compromised, they can, again, try to do some kind of eclipse against you. 24:00.000 --> 24:01.720 Same with denial of service. 24:01.720 --> 24:07.420 If your nodes can never communicate over those protocols, then they will never be able to connect to the network in the first place. 24:07.680 --> 24:15.740 BGP routing attacks, those are extremely damaging and something that needs to be considered. 24:15.920 --> 24:27.720 I'm putting the severity as high on these, but probability lower, just because it's a lot harder to do a targeted attack to figure out what is the exact node that is owned by your company or by you individual. 24:28.000 --> 24:37.200 So, it would most likely be an attack against the entire node network, but it's still probable. 24:39.900 --> 24:47.440 So, we can keep going through all of these individual components and try to go through like what are the main threats. 24:47.440 --> 24:57.280 Luckily, I went through this exercise over the last few months and I built a no threat model which applies to generic nodes out there. 24:57.280 --> 25:08.540 So, if you take a Bitcoin node versus, let's say, Ethereum nodes, they will have all these core issues, all these core threats, plus whatever makes them unique from each other. 25:08.540 --> 25:13.000 So, if you're like I mentioned that, for example, OpenEthereum has an extra web server in there. 25:13.000 --> 25:19.320 So, you'll need to slap in a web server into this model and consider like what are the threats there as well. 25:19.740 --> 25:25.860 Some more terminology so we can interpret the grid that I'm going to show you in a bit. 25:26.080 --> 25:30.300 So, stride is a common acronym used for threat modeling. 25:30.300 --> 25:39.320 It stands for simply just a variety of bad things you can do, which is spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege. 25:39.320 --> 25:43.780 It's just a guide to help you think better about like what is the impact there. 25:44.440 --> 25:46.740 It is also important to consider the likelihood. 25:47.220 --> 25:50.440 How likely is that the threat event is going to occur? 25:51.140 --> 25:59.220 How likely is that the threat actor not only has an opportunity but also the expertise to exploit something, the weakness. 26:00.220 --> 26:05.440 This will become useful later on as we talk about mitigations and how to prioritize them. 26:06.980 --> 26:14.420 So, overall in the threat model, we already went through the 11 components I showed you in a few slides ago. 26:14.420 --> 26:22.060 The total number of threats that I identified were 14 and the total number of critical threats for a generic node is 5. 26:22.060 --> 26:33.740 So, again, depending on how your ecosystem looks like, you know, if you're running a larger set of nodes and you have doctorized images and all that, your threat model will continue growing. 26:33.740 --> 26:37.820 This is just to get you started somewhere and provide you foundation. 26:38.980 --> 26:43.280 This is just a small glimpse into what the threat model looks like. 26:43.480 --> 27:00.700 We have the threat name, so node software generic vulnerabilities, the stride impact, so elevation of privilege, denial of service, severity, probability, description, most importantly, a variety of different mitigations that you can implement to address this threat. 27:04.720 --> 27:13.980 So, a key thing that we need to do when considering mitigations and the risk is to consider the cost. 27:14.740 --> 27:18.080 The cost, probability, and severity. 27:18.080 --> 27:23.720 So, the cost is how much time is it going to take you, how many resources you need to invest to address something. 27:23.720 --> 27:27.800 What is the severity and probability of this threat? 27:28.280 --> 27:45.760 So, in order to prioritize a variety of threats, I came up with this five-level, I guess, criticality grid where most critical flaws are the ones that are both have high impact, but also high probability of happening. 27:46.720 --> 27:55.420 Severe threats are the ones that have high impact, but maybe slightly lower probability of occurring. 27:55.420 --> 28:07.820 So, think of your generic web server getting exploited, which is running on your node versus some esoteric block parsing vulnerability that needs to be discovered to be exploited. 28:07.820 --> 28:16.180 And all the way to low, where these are the threats which we can address at one point, but only after we're done with the top two or top three, at least. 28:18.300 --> 28:26.580 So, of all these mitigations, in the model, I created about 41 mitigations that one could potentially implement. 28:26.580 --> 28:32.300 It could be probably more that we can think of, but I'll share the top 10 ones in a bit. 28:32.600 --> 28:36.560 Of those mitigations, they're addressing 22 critical threats. 28:37.040 --> 28:42.220 What's interesting is that half of those mitigations for critical threats are, in fact, very low cost. 28:42.220 --> 28:48.980 This is something that you can implement in a day and will significantly bolster the security stance of your nodes. 28:49.440 --> 28:59.560 So, yeah, you can start building fuzzers, you can start looking for every line of code of your nodes to make them secure, and they will make them very secure. 28:59.660 --> 29:15.940 But you can also do a bunch of things which are very low cost that will not make your nodes completely secure, but they will also, by the order of magnitude, make them resilient to some of the attacks that I described. 29:17.300 --> 29:30.700 So, this is just a different view that in the top bar of the critical vulnerabilities, the ones in light blue, you can see that half of them are basically low-cost solutions that you can implement. 29:30.760 --> 29:44.040 And then you have just, you know, if you really want to push it through this, like, it's 99.99% secure, you can continue on that trail, but that will be a little bit more costly. 29:46.420 --> 29:54.860 So, with that, let's talk about into node security, top 10 defenses that you can implement. 30:00.840 --> 30:12.860 So, the first one, and the most critical one, I think, is secure handling of where is the node software coming from, the repositories, the binary distributions, and so on. 30:13.340 --> 30:20.120 One key thing that we do at Coinbase, and something that you could consider as well, is repository pinning. 30:20.740 --> 30:37.740 Repositories can get compromised, but if you make sure that you pin to a specific non-Git version that you will work with and you will stick to until you go through some kind of process that, okay, I'm going to switch to the latest release instead of just downloading whatever is the latest one every time you deploy a node, 30:37.740 --> 30:44.100 that's something you could consider and could partially mitigate the risk of backdoored repos. 30:44.780 --> 30:47.020 Another thing is verification of signatures. 30:47.020 --> 31:04.260 So, if you're relying on either source code that you're downloading or binaries that you're downloading, make sure you verify all the signatures that, in the case of Monero, if the binaries were signed and the attackers did not have signed signatures correctly, 31:04.260 --> 31:06.220 then you would be able to detect that. 31:06.240 --> 31:18.720 In the case of source code, if you can at least verify, like, who is committing to the repositories, is it coming from, am I accessing a website, and it's, you know, the TLS is not breaking, no one is men in the middle of me trying to download the source code. 31:18.720 --> 31:23.900 Again, this is something that could make your nodes more secure. 31:26.160 --> 31:32.380 We continue on that train of thought, we can also make sure that we build all nodes from source code. 31:32.380 --> 31:45.760 I know this is a big ask, but this is something that you could consider, and the reason for this is, again, the Monero example, where only the binaries of Monero were backdoored, the source code remained intact. 31:46.580 --> 32:03.520 And the reason why, both could have been compromised, but if you consider how binaries for nodes are built, so there's a software repository with a lot of eyes, and then there could be a flow where one single developer with malware on their machine downloads that source code, 32:03.520 --> 32:09.700 builds the library, uploads the binary back on the website, there are a variety of things that could break along the way. 32:09.700 --> 32:14.620 Same goes with the CI automation of some sort, which builds binaries automatically. 32:14.960 --> 32:31.240 There are a lot more components which could be attacked and compromised, which is why, in case, again, lesson learned from that one incident, is that if you just stick to the source code, it may still be backdoored, but at least you have, like, one point of failure as opposed to a multitude of others, 32:31.240 --> 32:33.360 which are required when building binaries. 32:35.380 --> 32:37.020 Secure node configuration. 32:37.100 --> 32:44.880 So, every single node that I looked at has a whole variety of different levers that you can pull to make them secure. 32:44.960 --> 32:56.160 Some things that you should, generically, you could consider is make sure that you can diversify the number of connections that your node can make, and to which nodes it can make, it can connect to. 32:56.160 --> 33:00.320 So, an example would be things like Stellar, I believe. 33:00.320 --> 33:06.580 They need to be manually configured to distribute which nodes that you connect to. 33:06.760 --> 33:14.140 So, make sure that if you configure nodes like that, you select as wide of a node selection as possible. 33:14.140 --> 33:22.280 Others, they restrict the number of connections total, so you want to see, like, what is a reasonable number of connections you want to maintain throughout the network. 33:22.280 --> 33:27.340 This will help you with Eclipse attacks, making more resilient against forks, and so on. 33:28.260 --> 33:35.680 Another important key on configuring nodes securely is locking down your RPC interfaces. 33:35.680 --> 33:45.980 So, I believe last year, there was mass scanning of Ethereum nodes that were looking for open RPC interfaces to basically steal private keys. 33:46.100 --> 33:47.880 So, lock down your RPC interface. 33:47.880 --> 33:51.140 There's no reason why those should be exposed to the rest of the world. 33:51.140 --> 34:00.560 So, make sure you configure, you know, whatever IP or interface that RPC is listening on, make sure it's configured to something local to your network or just local host. 34:02.280 --> 34:03.980 Restrict node access. 34:04.740 --> 34:08.420 So, not everyone needs constant access to your nodes. 34:08.420 --> 34:12.580 Not everyone needs to be able to modify arbitrary things within your node. 34:12.580 --> 34:23.480 So, at Coinbase, what we do is we create a consensus mechanism with no configuration changes, no nodes, no one can bring up and down nodes just arbitrarily. 34:23.480 --> 34:25.200 It requires a consensus of people. 34:25.200 --> 34:29.780 The more critical your infrastructure is, the more people it requires a sign-off. 34:29.780 --> 34:39.240 So, you can implement it through a sign-off, through some kind of two-factor pings that you can send to people to approve any given action. 34:41.220 --> 34:43.340 Define and enforce administrator roles. 34:43.500 --> 34:51.140 Again, this is something to restrict to security nodes in larger environments, but you can apply this to individual nodes as well. 34:51.140 --> 35:02.280 Make sure that there are some specific roles set up for the node that, you know, you can either configure nodes and deploy them or shut them down, but not everything at the same time. 35:02.280 --> 35:08.000 So, you can get very granular there, but it depends, see how appropriate it is for your specific environment. 35:09.080 --> 35:15.900 Finally, restrict node access, as in the network traffic node access, both ingress and egress. 35:16.380 --> 35:29.100 So, on one side, it makes sense that you restrict your RPC interface already, so make sure those are firewalled off, that you cannot access nodes from anywhere, and then externally, you also have some kind of restriction set up as well. 35:29.220 --> 35:37.340 On the other side, nodes could be an excellent points of entry for the attacker who found some kind of vulnerability into the rest of your network. 35:37.340 --> 35:43.100 So, make sure that your nodes cannot communicate to the rest of your network as well, because they could be used as stepping stones. 35:44.540 --> 35:47.000 Monitor and log node activity. 35:47.160 --> 35:53.100 So, this one is probably the most interesting one, and there's a lot of things you can do there. 35:53.480 --> 35:59.500 Anytime your node loses internet connectivity, make sure you do some basic verification. 35:59.500 --> 36:04.360 Am I getting... is there like something with BGP, or am I getting... is my DNS going down? 36:04.360 --> 36:05.760 Am I under attack? 36:06.240 --> 36:07.740 Am I getting eclipsed? 36:07.740 --> 36:09.100 So, make sure you investigate those. 36:09.100 --> 36:10.300 Am I on the right fork? 36:10.300 --> 36:12.900 Am I not seeing sufficient traffic? 36:13.000 --> 36:14.900 All of that applies. 36:15.040 --> 36:23.480 You can do node-specific verifications too, not just generic network monitoring, such as periodically check, like, am I on the right fork? 36:23.480 --> 36:24.920 What is the block height? 36:24.920 --> 36:26.680 What is the current block hash? 36:26.680 --> 36:33.020 Do other nodes in my system, or externally, or on blockchain explorers, do they still see the same thing? 36:33.580 --> 36:45.460 So, in case of the recent ETC experiment incident, you would have been able to quickly diagnose, like, oh, okay, so there's something going on with nodes because there's some kind of forking happening. 36:46.420 --> 36:49.740 Finally, log all access, all network events. 36:49.740 --> 37:05.400 It may not be as essential during the normal operation of node, but when something goes wrong and you need to investigate and figure out what happened, who accessed it, what were the network events that led up to the attack, those logs will be highly instrumental for the investigation. 37:07.620 --> 37:08.780 Lockdown-based OS. 37:08.780 --> 37:14.980 So, definitely, you cannot build a secure node software if your underlying layer is insecure. 37:15.000 --> 37:35.100 If, again, if you want to go after the weakest link, and your node is very well configured and firewalled off, but your underlying OS is some outdated version of Linux, which has, like, exposed ports and they're easily attacked, the attacker is just going to go after that and going to skip all of the other mitigations you built. 37:35.100 --> 37:40.320 So, you could do things like doctorize your nodes, run SCLinux or similar toolkits. 37:40.340 --> 37:47.120 So, there are a variety of things you can do to lock down the OS, and there are guides for that, but something to consider as well. 37:48.820 --> 37:51.000 Hardened node's network connections. 37:51.300 --> 38:03.880 So, going off the stack a little bit, not just protecting what connections can be made, or what we're monitoring, make sure that if you configure DNS server for your node, make sure it's a secure one, uses DNSSEC. 38:03.880 --> 38:09.580 If you're using ISPs, make sure that whatever BGP connections that it uses are done in a secure fashion. 38:09.580 --> 38:17.480 If your node needs to communicate to some kind of HTTP resources, make sure you configure your CAs properly, so you're not getting met in the middle. 38:18.120 --> 38:20.620 Denial of service protection is critical. 38:20.780 --> 38:25.220 Again, it's potentially hard to determine. 38:25.380 --> 38:37.280 I want to attack just one particular node for one particular user or organization, but it's prudent to use something like Cloudflare to have a layer of denial of service protection, so you don't have to worry about that. 38:38.500 --> 38:40.880 Node-specific threats. 38:41.160 --> 38:42.980 Node and protocol-specific threats. 38:43.600 --> 38:44.860 This is a catch-all. 38:44.860 --> 38:50.120 Basically, be mindful that all nodes are unique in their own different ways. 38:50.120 --> 38:54.520 So, I mentioned before that Stellar, you have to manually configure what are the trusted nodes. 38:54.520 --> 38:59.540 You have to manually calculate, like, hey, what is my threshold for... it's a BFT network. 38:59.540 --> 39:02.240 What is my threshold that it can get attacked? 39:02.240 --> 39:05.980 So, make sure that I'm not going to be vulnerable to that. 39:05.980 --> 39:14.160 Other protocols like EOS, they implement debugging modes that you have to watch out for. 39:14.160 --> 39:16.280 So, read the config file completely. 39:16.280 --> 39:26.100 Make sure you understand what are the implications of all the variety of settings, and have a good secure configuration specific for that node. 39:28.260 --> 39:30.580 Verify node functionality before deployment. 39:30.580 --> 39:36.720 So, that's a much harder one, because, sure, you pinned the repository. 39:37.040 --> 39:45.840 You are pretty confident, okay, so it probably was not modified, but I'm not 100% sure that whatever I downloaded is working as it's supposed to. 39:45.840 --> 39:47.020 There are no backdoors there. 39:47.020 --> 39:50.500 How do you verify that it still works as intended? 39:51.180 --> 40:02.040 Well, ideally, you would run a set of tests, like, hey, if I transmit this transaction, it shows up on the network, or if I see a transaction coming from the network, my account gets properly credited. 40:02.260 --> 40:06.640 And that's a big ask to implement for every single node. 40:06.880 --> 40:18.500 So, one project which aims to solve this problem is Coinbase Rosetta project, which creates a kind of a middleware for a variety of different nodes to be able to speak the same API. 40:18.500 --> 40:42.880 So, instead of implementing a variety of different calls to transmit a transaction, get a current block hash and transactions in it, Rosetta creates a kind of a middleware for a variety of different nodes that you can just use a single API call, and it automatically translates to whatever is specific for that node. 40:43.240 --> 41:01.380 Now, what's interesting about the Rosetta project also includes a Rosetta validator, which exercises the node and makes sure that it passes a variety of very specific tests from transaction verification blocks and signing and so on to make sure that it works exactly as intended. 41:01.740 --> 41:14.720 Not every project supports Rosetta interface, but for the ones that do, you can implement it in the pipeline to make sure that the nodes that you deploy are acting as expected. 41:17.200 --> 41:20.840 And you notice that we're slowly ramping up the difficulty level. 41:20.840 --> 41:22.460 This is probably the hardest one. 41:22.460 --> 41:35.060 This requires you to examine the source code, examine the alerts that you're getting, and this is the basic security static analysis of node software. 41:35.440 --> 41:43.100 So, you may not need to go through the actual source code, but you can observe a variety of different alerts reported by tools like Salus. 41:43.100 --> 42:00.400 So, this is another Coinbase tool, which we use internally, and it's an open source tool you can use as well to quickly run through a variety of node source code packages, and it gives you a nice report of what are the high critical vulnerabilities within that. 42:00.400 --> 42:03.020 So, you can examine them, just get a general sense. 42:03.020 --> 42:16.100 It's hard to say which ones are false positives without actually looking at the source code, but you can see like, hey, if I suddenly see 100 or so, or like this really sneaky report, maybe I should investigate more or report or ask the original developers. 42:17.520 --> 42:27.080 In the Salus project, it supports a variety of different languages from Go to Ruby and so on. 42:27.740 --> 42:35.900 GoSec, like for if you're just running Ethereum or other Golang-based nodes, so GoSec is excellent. 42:35.900 --> 42:41.920 In fact, Salus is running a variety of open source projects, and it just creates a wrapper for them. 42:42.080 --> 42:43.480 So, GoSec is one of them. 42:43.880 --> 42:50.760 For all the other projects, you have something like Semgrep, so if you have SQL to analyze and so on, so you can use that as well. 42:52.580 --> 43:03.480 So, with that, just to summarize, what is our total view of all the different mitigations, top 10 things that you can do to secure your nodes. 43:03.600 --> 43:07.140 So, make sure you handle software repositories securely. 43:07.140 --> 43:17.160 Make sure that you pin them, that you understand that they can be compromised, and you need to just investigate a little bit more. 43:17.640 --> 43:19.480 Build all nodes from source code. 43:19.480 --> 43:25.480 So, again, it's just reducing your attack vectors just a little bit more. 43:25.480 --> 43:27.060 You just build everything from source. 43:27.060 --> 43:28.480 There are a lot of eyes in source code. 43:28.480 --> 43:30.360 Who knows what happens with those binaries? 43:31.080 --> 43:32.680 Securely configure nodes. 43:32.700 --> 43:37.360 So, make sure that you diversify what connections your nodes have. 43:37.360 --> 43:45.340 Make sure that they can't be just as easily eclipsed and people can access it and reconfigure them. 43:45.800 --> 43:47.280 Restrict node access. 43:47.660 --> 43:49.120 Monitor and log node activity. 43:49.120 --> 43:54.600 So, you can get, again, very creative with those, what you can do to judge what nodes help. 43:55.020 --> 43:56.940 Lock down your base OS. 43:56.940 --> 44:01.800 Make sure that you're not running, you're not building a castle on quicksand. 44:04.640 --> 44:07.200 Definitely consider node and protocol specific threats. 44:07.200 --> 44:09.980 All nodes are unique in a variety of different ways. 44:10.160 --> 44:12.000 So, you need to consider those as well. 44:12.000 --> 44:13.460 Verify node functionality. 44:13.460 --> 44:17.760 So , if your node supports Rosetta project, run Rosetta validator. 44:17.760 --> 44:26.200 It will give you this good feeling that, okay, at least for a variety of tests, this still works as expected. 44:26.460 --> 44:40.040 And then, if you have time and you have technical expertise, try to run basic static analysis tools just to periodically see, like, are there, like, major low-hanging fruit vulnerabilities that are getting introduced that I can catch, maybe report to original developers. 44:41.800 --> 44:44.980 So, with that, thank you so much for your time. 44:45.920 --> 44:47.580 I may have not mentioned this. 44:47.580 --> 44:50.480 I'm the blockchain threat intelligence newsletter. 44:50.480 --> 44:55.020 I list a lot of node incidents and vulnerabilities that are getting discovered. 44:55.080 --> 44:56.460 So, subscribe. 44:56.460 --> 45:01.520 See if you can catch the latest lessons learned from those vulnerabilities. 45:01.740 --> 45:04.100 And feel free to follow me on Twitter. 45:04.780 --> 45:09.200 And or feel free to contact me offline if you have any more questions. 45:09.200 --> 45:10.300 Thank you.