[00:05.020 --> 00:08.720] Hello and welcome to Attacking and Defending Blockchain Nodes, [00:08.720 --> 00:13.820] where we're going to learn about a variety of risks and mitigations that we can implement [00:13.820 --> 00:18.560] to defend a very core component of the blockchain systems. [00:20.000 --> 00:25.040] We'll try to do a live discussion during the talk while we ask a question, so feel free to join [00:25.900 --> 00:33.080] DEF CON's Discord on BCV General Tech's channel, and I'll try to ask questions and respond live. [00:34.380 --> 00:39.540] In case we haven't met, my name is Peter Kaczynski. I'm a blockchain security engineer at [00:39.540 --> 00:45.380] Coinbase, where I spend most of my time breaking a variety of different blockchain systems and [00:45.380 --> 00:51.420] smart contracts. I'm also a writer for the Blockchain Threat Intelligence Newsletter, [00:51.420 --> 00:59.060] where I try to cover all the latest news, events, hacks, scams in the blockchain security ecosystem. [00:59.840 --> 01:04.700] Last year, you may have also participated in the Capture the Coin competition, [01:04.700 --> 01:11.500] which I helped organize. My background before deep diving into the cryptocurrency world [01:11.500 --> 01:16.180] was as a malware reverse engineer at FireEye, where I looked at a whole lot of APT malware [01:16.180 --> 01:22.040] and a penetration tester for the Federal Reserve System, where I was breaking Finance 1.0. [01:24.120 --> 01:30.300] So, just some idea, give you an idea of what we're going to talk about. I'm not going to drop [01:30.300 --> 01:36.520] any old days today. What my goal here is to educate, to share knowledge, to also learn from you. [01:37.180 --> 01:42.420] With the goal of just making, discussing, like, how can we secure node infrastructure? [01:43.080 --> 01:48.700] So we'll start with a simple exercise on how do we know, what is the state of truth that tells us [01:48.700 --> 01:55.300] how many coins we actually have? We'll go through a variety of examples of how nodes break, and [01:55.300 --> 02:01.280] we'll discuss a few of the ways that they could break in the future. Finally, I'll introduce a [02:01.280 --> 02:09.260] solid node security threat model for generic nodes, which you can apply later on for a specific node [02:09.260 --> 02:15.300] implementation. And most importantly, discuss a top 10 list of security defenses that you can [02:15.300 --> 02:19.960] implement today, whether you're a standalone operator, or you're working for an exchange, [02:19.960 --> 02:28.280] and share lessons learned from trying to secure nodes of my own. So with that, let's talk about [02:28.280 --> 02:34.340] how many coins you have. So what is, if you're running, if you wanted to, at any given moment, [02:34.340 --> 02:40.000] tell me how many coins of any given asset that you have, how do you go about that? [02:40.660 --> 02:45.300] So, well, you'll probably have a piece of software on your computer, on your phone, [02:45.300 --> 02:51.840] or a hardware wallet, maybe, which has some specific logic dedicated just for a specific [02:51.840 --> 02:56.640] asset. So if you have a Bitcoin wallet, or Ethereum, or Monero, or some other coin, [02:56.640 --> 03:03.060] sorry if I didn't list here. This particular wallet communicates with a wider network, [03:03.060 --> 03:09.020] somewhere to the cloud, and asks a question, hey, how much coins do I have? What do I actually own? [03:09.460 --> 03:14.000] In case of Bitcoins, you try to enumerate how many outputs that you have unspent. [03:14.440 --> 03:19.000] Ethereum is different in the sense that it's an account-based system, so you have to look up [03:19.000 --> 03:26.880] how much of the asset that I actually have on the network. The thing is, it gets a little bit [03:26.880 --> 03:32.060] more complicated. So if we start getting a little bit more granular, we're going to see that, [03:32.060 --> 03:37.360] in fact, we don't communicate with just some unknown entity, abstract entity. We actually [03:37.360 --> 03:43.140] communicate to specific nodes for that specific coin. So Bitcoin wallet communicates to Bitcoin [03:43.140 --> 03:50.740] nodes, Ethereum to Ethereum, Monero to Monero, and so on. This introduces some complications, [03:50.740 --> 03:57.800] because we realize that there are a lot of third-party components involved. So if we just [03:57.800 --> 04:03.880] want to manage our assets, we have to communicate to a whole bunch of software which is maintained [04:03.880 --> 04:11.480] by third parties. So just to give you like a few more very simplistic threat model here, [04:11.480 --> 04:16.900] let's talk about components of a friend's coin, just the imaginary coin from the Mario friend's [04:16.900 --> 04:24.900] land. So you have your wallet communicating to a node, communicating to the overall gossip network [04:24.900 --> 04:33.000] of some sort, trying to propagate blocks. You also have your developers who are contributing to the [04:33.000 --> 04:40.140] GitHub repository, so you have to rely on maintaining the repository, compiling the node maybe, or [04:40.140 --> 04:47.640] downloading binaries there, and so on. And this all works really well, and until we start thinking [04:47.640 --> 04:54.420] that, well, how can we be sure that all the software libraries, all the standalone software [04:54.420 --> 05:00.940] infrastructure does not act maliciously, does not drop, modify, or craft transactions to steal your [05:00.940 --> 05:07.120] funds. So let's see how this could happen. So we have this Mr. Robot character trying to attack [05:08.060 --> 05:13.380] a variety of different points of this threat model. So we'll start left to right, [05:13.380 --> 05:19.040] so we can see that we can attack the network connection between your node and the [05:19.040 --> 05:25.160] cloud. So someone could start dropping blocks, try to eclipse attack your node, so basically to [05:25.160 --> 05:31.680] isolate it to make sure it's not communicating to the rest of the network properly. The same character [05:31.680 --> 05:36.620] can also attack your node directly if there are some unknown weaknesses that they can exploit to [05:36.620 --> 05:41.360] do some kind of code execution or crash it. They can prevent you from accessing the [05:41.360 --> 05:48.540] network. At last, we also have to think about developers. The bad actors out there, they can [05:48.540 --> 05:56.300] go after developers like the toad over there. They can make them commit bad code. And I'll [05:56.300 --> 06:02.200] cover a few real-world examples where this does happen. So the idea is that [06:03.980 --> 06:09.580] it's an invitation to start thinking about the node security as a holistic [06:09.580 --> 06:13.100] ecosystem. So there are a variety of different components which are involved. [06:15.040 --> 06:20.320] So the examples that I just mentioned, they're not theoretical, they're practical, they're based [06:20.320 --> 06:26.900] on real-world incidents. So for example, in May 2014, someone attacked the BGP network [06:26.900 --> 06:34.220] used to eclipse attack Doge and Bitcoin miner pools. So this was, in case you're not [06:34.220 --> 06:42.680] familiar with BGP hijacks, it's a way to effectively isolate whole large swaths of network blocks [06:42.680 --> 06:47.920] in order to make them route traffic to unexpected directions. This is something that happened [06:48.400 --> 06:53.060] a decade ago, I believe with YouTube, where all the traffic was redirected to [06:53.920 --> 06:57.600] to Pakistan, I believe. And it keeps happening again and again. [06:59.180 --> 07:04.220] This is a threat that we have not observed within cryptocurrency world for a while, but [07:04.220 --> 07:08.700] it's something that we have to be aware of and build threat models around. [07:09.400 --> 07:15.100] On the EOS side, after the release of the Node software, they published a bug bounty. [07:15.300 --> 07:19.860] And oh boy, did they receive a lot of good finds there, because one, there was immediately a code [07:19.860 --> 07:25.560] execution flaw that was discovered. And for a while at least, they kept on getting one [07:25.560 --> 07:32.560] after another of high-value bounties submitted to them, which included denial of service, code [07:32.560 --> 07:40.640] execution, and similar. Finally, in 2019, Monero website was compromised. The binaries for the [07:40.640 --> 07:46.460] wallet were backdoored to steal clients' funds. So this is an example of how we have to look out for [07:47.140 --> 07:51.040] the developer. Where are you getting that software? How trustworthy is that resource? [07:51.040 --> 07:57.920] What can you do to verify it? So this was just over the few years, but I just want to instill [07:57.920 --> 08:02.600] in you that this is an ongoing problem. These are all the different things happening this year. [08:02.820 --> 08:08.340] So in March, we had on the testnet, but we still had a flaw discovered in the way Solana was [08:08.340 --> 08:13.900] processing transactions, where it was not doing sufficient validation. So it resulted in 500 [08:13.900 --> 08:20.560] million Solana getting stolen. Filecoin, again, had a nasty inflation bug, fortunately discovered [08:20.560 --> 08:26.280] on testnet, exploited also on testnet, but it resulted in 9 billion Filecoins minted, [08:26.280 --> 08:30.820] when only 2 billion could have existed in the first place. Inflation bugs are particularly nasty [08:30.820 --> 08:36.160] because these are the attacks not against individual users, but against the entire coin [08:36.160 --> 08:43.280] ecosystem. Tendermint is a library behind a variety of projects. It's a consensus mechanism used in [08:43.280 --> 08:48.760] projects like Cosmos, Oasis, and others. A denial of service vulnerability was discovered there [08:48.760 --> 08:56.260] when parsing specially crafted invalid blocks, which basically crashed the nodes. So if this [08:56.260 --> 09:01.120] was successfully executed in the wild, Cosmos was likely not vulnerable to this because they [09:01.120 --> 09:06.260] didn't upgrade just to that particular vulnerable version. But if that was the case and the [09:06.260 --> 09:10.920] vulnerability was maliciously discovered, then it would result in a network halt. [09:11.520 --> 09:15.300] The last example, and probably the most critical and interesting one, [09:15.880 --> 09:22.580] both from the impact and the way that it was handled, is Ravencoin. Ravencoin had an inflation [09:22.580 --> 09:30.420] bug discovered just last month in July. The inflation bug vulnerabilities basically allow [09:30.420 --> 09:35.920] someone to print money out of thin air, which is extremely dangerous because it allows an attacker [09:35.920 --> 09:43.640] just to profit and sell value to exchanges, and it basically impacts every single user of the coin. [09:43.640 --> 09:47.840] What's interesting about this particular vulnerability was that it was actually [09:47.840 --> 09:54.980] maliciously introduced. It was introduced back in January by an account which was seemingly [09:54.980 --> 10:01.260] making adjustments to the way the comments were made, on the debugging views were made. [10:02.360 --> 10:07.880] And the core developers unfortunately missed that until the point that someone reported them, [10:08.320 --> 10:11.380] a third party, which was building a blockchain explorer for Ravencoin, [10:11.380 --> 10:14.400] there's something weird going on in your network. We're observing someone [10:15.060 --> 10:21.220] minting a whole bunch of RVN and also, by the way, selling them on exchanges and profiting from it. [10:21.600 --> 10:29.180] So the Ravencoin team was able to issue an emergency patch, worked with the miners, [10:29.180 --> 10:36.260] but it's still 300 million RVNs were minted and exchanged for some other coins. [10:36.780 --> 10:40.200] Something that we have to learn from. [10:42.140 --> 10:49.060] So before we go into the section on how to secure the nodes, how to look out for those [10:49.060 --> 10:53.660] vulnerabilities, let's first think about all the different ways that we can break the nodes. [10:54.460 --> 11:00.060] So just to define the scope right away, we're not going to talk about the features of a node, [11:00.060 --> 11:05.060] which are documented and designs of the node. So if something implements proof of work, [11:05.060 --> 11:12.020] we're not going to deep dive into 51% attacks, reorgs, other protocol design issues. We'll accept [11:12.020 --> 11:17.360] them as is. What we're more concerned about is if the node promises to do something correctly, [11:17.360 --> 11:21.560] and it fails to do that, that's what we need to pay attention to. [11:22.340 --> 11:30.640] Key management is also outside of this. So you should probably be running nodes on a variety [11:30.640 --> 11:34.620] of tiers. So if you just want something which replicates the state of the network, you should [11:34.620 --> 11:41.440] probably keep your keys on a separate system. Incorrect usage. So I assume that if you're [11:41.440 --> 11:45.240] running a specific node that you read the documentation, you read everything that is [11:45.240 --> 11:53.560] known about it, so you don't necessarily make mistakes. Like a good example of that is Ripple's [11:53.560 --> 12:00.380] TF partial payment flag, where if you do not implement that particular flag correctly, then [12:00.380 --> 12:06.080] whenever someone sends you some payment with that flag set, it may appear that you actually [12:06.080 --> 12:12.840] receive those funds. But in reality, it's a IOU, essentially, that I will send you those things [12:12.840 --> 12:17.420] eventually, but not right now. So a lot of exchanges were exploited as a result of that. [12:17.840 --> 12:22.740] Finally, we're not going to focus on the layer two vulnerabilities such as smart contracts, [12:22.740 --> 12:30.180] DeFi, and so on. So what's in scope? What's in scope are the implementation flaws. So protocol, [12:30.180 --> 12:37.740] software flaws, attack resilience, can the node stand up to sustained attacks? Infrastructure. So [12:37.740 --> 12:42.140] we're not looking at nodes as just this one thing, and we're protecting just the node itself, [12:42.140 --> 12:46.640] and don't care about anything that it's built in software. We're worrying about underlying OS, [12:46.640 --> 12:50.580] we're worrying about the network stack, and anything that goes around it. [12:51.080 --> 12:56.900] Finally, management. Human factor is, always was, always will be the weakest link in the security [12:56.900 --> 13:02.260] chain. So we will discuss access management, we'll talk about configuration, source control, [13:02.260 --> 13:11.240] and all other ops type of things. So with that, just before we dive into the exercise, just wanted [13:11.240 --> 13:16.660] to establish some terminology so we're all on the same page. The threat model, and this is taken [13:16.660 --> 13:21.840] from OWASP's excellent threat modeling cheat sheet if you want to review it and build your own threat [13:21.840 --> 13:30.100] models. Threat model itself is a process to identify potential threats. So we want to enumerate [13:30.100 --> 13:34.360] all the different ways that things can go wrong, we can assess those threats, what is the risk [13:34.360 --> 13:38.960] coming from them? Are they basically going to result in me losing money, or is it just [13:39.340 --> 13:47.500] a denial of service, or some other maybe less critical impact? And also later on, we'll use [13:47.500 --> 13:54.320] that to prioritize mitigations. Why would we spend hours or weeks trying to address a risk [13:54.320 --> 14:00.340] which is not really that impactful? An important thing that we're going to do next, we'll define [14:00.340 --> 14:06.020] the attack surface. An attack surface, think of it as all the different ways that someone can access, [14:06.020 --> 14:12.640] can attack your node or node infrastructure, and the threat agents, which are the entities [14:12.640 --> 14:20.040] which have enough skill, opportunity, and interest to attack your nodes. [14:20.960 --> 14:27.960] So let's begin with just a really basic system, it's not interacting with any other nodes, it's [14:27.960 --> 14:34.580] not managed by anything, just like a really basic core software. So we have the node itself, which [14:34.580 --> 14:43.720] all it does is it's processing the packets, it's processing the transactions, blocks, does [14:43.720 --> 14:49.080] verification, it stores the blocks in some kind of storage system, it parses configuration files [14:49.080 --> 14:55.360] so it knows how to properly look at those things, and it may also include a VM interpreter process [14:55.360 --> 15:01.020] where it can interpret smart contracts or anything like full-blown, Turing-complete, [15:01.020 --> 15:05.600] Ethereum-style smart contracts, or something a little bit more locked down like Bitcoin script. [15:07.080 --> 15:13.460] Now, let's add some more components to that. In order for node to be useful, it needs to [15:13.460 --> 15:18.060] have some kind of management interface, and most nodes implement some kind of RPCE [15:18.620 --> 15:24.100] network client, so we have to think about the threats coming from that, who can access this [15:24.100 --> 15:29.520] thing, what is it they can do, and of course, node administrators themselves who are managing [15:29.520 --> 15:33.840] the machine, they can shut them down, they can bring it back online, they can reconfigure them [15:33.840 --> 15:39.900] and maybe introduce some flaws that I will cover in a moment. And also, these are just humans, [15:39.900 --> 15:47.260] can they be attacked on their own with malware or maybe any other social engineering? Finally, [15:47.260 --> 15:51.040] nodes need to communicate to other nodes to figure out what is the state of the network, [15:51.040 --> 15:55.340] and this is where a whole bunch of different things are coming into play, like network [15:55.340 --> 16:00.760] infrastructure, other nodes that we need to talk to, and miners. So network infrastructure [16:00.760 --> 16:06.820] includes basically underlying layers of communication for the nodes, so think of BGP [16:08.220 --> 16:15.120] and DNS client, DNS services, and just basic routing like TCP connections, UDP, [16:15.120 --> 16:21.900] all of that is in scope and needs to be secured as well. Finally, nodes are just not appearing [16:21.900 --> 16:26.280] on your machines out of nowhere, they must come from somewhere. Unless you are the developer, [16:26.280 --> 16:31.460] you probably are relying on third-party repositories, which also in turn rely on [16:31.460 --> 16:39.660] third-party dependencies. So this is something that we need to be aware of, because as I mentioned [16:39.660 --> 16:46.440] with Ravencoin, someone was able to backdoor the assets through a very sneaky code change [16:46.440 --> 16:51.760] and something that we also need to consider within our threat model. [16:53.540 --> 16:58.540] So with that in mind, join the BCV General Text on the DEF CON Discord channel [16:59.600 --> 17:05.180] and I'll ask you questions to just brainstorm together on how we can break nodes. [17:06.760 --> 17:11.120] So the first thing we're going to attack is the core. This is the node software [17:11.120 --> 17:21.000] itself. This is what processes the blocks, the transactions, and needs to verify them correctly. [17:21.420 --> 17:26.840] If you are this sneaky Mr. Robot character and you wanted to attack this, [17:26.840 --> 17:31.320] how would you do it? So I'll look in the channel for your responses. [17:39.860 --> 17:46.900] So I have one comment of software supply chain attack. So this is something that we're going to [17:46.900 --> 17:52.300] cover in the software repository attack, but I'm talking about the core itself. That's the [17:52.300 --> 18:02.060] piece of software that does the actual blockchain logic. BGP attacks to create an eclipse. [18:03.160 --> 18:10.300] So again, so this is something that we have to worry about on the network connectivity side. [18:16.190 --> 18:22.740] Let's see... [18:23.060 --> 18:30.280] Yeah, so a denial of service against the nodes to crash with the bad blocks, malformed block headers. [18:30.300 --> 18:38.080] So that's exactly that. So we can craft a packet, think of a tendermint flaw, [18:38.080 --> 18:46.540] which basically crashes a node or maybe executes some kind of really nasty code execution attack. [18:47.080 --> 18:52.700] So just to give you some background, this is how I created the model. So there's a title, [18:52.700 --> 18:58.840] protocol vulnerabilities, severity. So if you can crash or do code execution, know the severity is [18:58.840 --> 19:05.380] high. Probability, I would say medium. It's just harder to find, harder to exploit. It requires [19:05.380 --> 19:11.860] dedicated time to discover those things. But the impact is that anything that is not properly [19:11.860 --> 19:17.380] processing blocks, transactions, what is the correct fork, messing with the governance and so [19:17.380 --> 19:21.860] on, inflation vulnerabilities, this is all in scope for this. And it will have a severe impact [19:22.600 --> 19:30.560] on the node. However, I want you to consider it's less sexy of an attack. So we're not trying to [19:30.560 --> 19:35.700] find some kind of ode within blocks and transactions. And consider that nodes actually [19:35.700 --> 19:42.260] run all sorts of third-party software. So for example, Parity Nodes, they run or OpenTheorem [19:42.260 --> 19:50.640] now, they run web UIs to manage the thing. A bunch of node software relies on third-party [19:51.220 --> 19:57.720] database libraries and standalone databases to store blocks. So all of that are fair game to [19:57.720 --> 20:04.440] attack as well. So if you are Mr. Robot and you wanted to attack your node infrastructure, [20:04.440 --> 20:10.740] I'm not going to spend time trying to find old days for the way that you parse packets. It's a [20:10.740 --> 20:16.260] lot easier for me to find out what is the weakest link, what is the common off-the-shelf software [20:16.260 --> 20:21.300] that you're running, such as if you're running Nginx and for some reason it's a vulnerable [20:21.300 --> 20:26.500] version, I'll try to attack that as well. I'll try to attack that instead because it probably has [20:26.500 --> 20:32.200] more eyes on it and more vulnerabilities discovered so far. So consider those as your first line [20:32.200 --> 20:37.420] of things to lock down and the probability is definitely higher than finding a very [20:37.420 --> 20:43.460] directed attack to like in the way you parse blocks. So let's continue in the same [20:44.840 --> 20:49.560] style. So software repository attacks. So someone in the channel already mentioned that, [20:49.560 --> 20:55.480] you know, the supply chain attack. So whether you're attacking the software repository itself, [20:55.480 --> 21:01.140] so you can, you know, like a Ravencoin example or Monero example, if you can compromise the [21:01.140 --> 21:09.660] software repository itself, then you can introduce arbitrary changes, which are maybe [21:09.660 --> 21:14.180] different degrees of how hidden they are, depending how many eyes are looking at the repository. [21:15.460 --> 21:22.760] And again, this can result in fund theft or money printing type of vulnerabilities. However, [21:22.760 --> 21:28.980] what I want to invite you to also consider is not just the main software repository, but also [21:28.980 --> 21:34.560] software dependencies. So an example, not as actually a node vulnerability, but I believe it [21:34.560 --> 21:43.100] was Trinity Wallet, which had a, it was backdoored on the source code level. But the way that it was [21:43.100 --> 21:47.500] backdoored was not through the main repository, but through one of the dependencies. So someone [21:47.500 --> 21:53.040] figured out what is the weakest link in the supply chain to that particular piece of software, [21:53.040 --> 21:59.200] backdoored that, and then they were able to introduce code that was stealing keys. [21:59.780 --> 22:04.800] So again, probability is high on that as well, and severity is also extremely high. [22:07.240 --> 22:10.640] I mentioned this before, you know, human factor is always the weakest [22:10.640 --> 22:14.520] one. So node administrator, how can we attack that? [22:17.760 --> 22:28.720] Yep, phishing, send phishing email with some kind of malware. You know, the impact is very high. [22:28.860 --> 22:36.440] They can reconfigure the node, they can introduce a set of nodes, which they essentially creates [22:36.440 --> 22:40.460] an eclipse attack where you only communicate to something that is controlled by the attacker. [22:40.680 --> 22:45.300] Certain blockchains, they have a switch you can flip, which basically allows it to accept all [22:45.820 --> 22:51.080] blocks and transactions without doing any verifications, like a debugging mode almost. [22:51.120 --> 22:54.840] There's a variety of things that administrator can do and the variety of ways that administrator [22:54.840 --> 22:58.760] can get attacked through phishing, malware, social engineering, you name it. [23:00.440 --> 23:05.680] The last, you get the gist of how we're going through different components. The last one I [23:05.680 --> 23:12.340] want to cover is the network attack. So, correct, so BGP attacks to create an eclipse. [23:13.400 --> 23:23.020] So, exploiting the network infrastructure, you know, all you know, a lot of nodes, [23:23.020 --> 23:26.800] the way that they're built, is that they need to first boot up. When they first come online, [23:26.800 --> 23:32.520] they need to communicate to something hard-coded to figure out what is the network truth, like [23:32.520 --> 23:39.460] what are the basic trusted infrastructure that I can use to start getting an idea what is the [23:39.460 --> 23:48.720] current block state. A lot of them use protocols like DNS in order to find out what is the first [23:48.720 --> 23:55.400] set of nodes that I need to connect to. And if your underlying DNS infrastructure is compromised, [23:55.400 --> 24:01.860] they can, again, try to do some kind of eclipse against you. Same with denial of service. If [24:01.860 --> 24:05.460] your nodes can never communicate over those protocols, then they will never be able to [24:05.460 --> 24:11.100] connect to the network in the first place. BGP routing attacks, those are extremely damaging [24:12.980 --> 24:17.920] and something that needs to be considered. I'm putting the severity as high on these, [24:17.920 --> 24:22.400] but probability lower, just because it's a lot harder to do a targeted attack to figure out what [24:22.400 --> 24:29.320] is the exact node that is owned by your company or by you individual. So, it would most likely be [24:29.320 --> 24:37.200] an attack against the entire node network, but it's still probable. [24:39.900 --> 24:45.540] So, we can keep going through all of these individual components and try to go through [24:45.540 --> 24:50.160] like what are the main threats. Luckily, I went through this exercise over the last few months [24:50.160 --> 24:58.460] and I built a no threat model which applies to generic nodes out there. So, if you take a [24:58.460 --> 25:04.580] Bitcoin node versus, let's say, Ethereum nodes, they will have all these core issues, all these [25:04.580 --> 25:09.780] core threats, plus whatever makes them unique from each other. So, if you're like I mentioned [25:09.780 --> 25:14.540] that, for example, OpenEthereum has an extra web server in there. So, you'll need to slap in a web [25:14.540 --> 25:20.660] server into this model and consider like what are the threats there as well. Some more terminology [25:20.660 --> 25:28.060] so we can interpret the grid that I'm going to show you in a bit. So, stride is a common [25:28.060 --> 25:33.240] acronym used for threat modeling. It stands for simply just a variety of bad things you can do, [25:33.240 --> 25:38.200] which is spoofing, tampering, repudiation, information disclosure, denial of service, [25:38.200 --> 25:43.060] elevation of privilege. It's just a guide to help you think better about like what is the impact [25:43.060 --> 25:49.060] there. It is also important to consider the likelihood. How likely is that the threat event [25:49.060 --> 25:55.060] is going to occur? How likely is that the threat actor not only has an opportunity but also the [25:55.060 --> 26:03.080] expertise to exploit something, the weakness. This will become useful later on as we talk about [26:03.080 --> 26:10.640] mitigations and how to prioritize them. So, overall in the threat model, we already went through the [26:10.640 --> 26:16.020] 11 components I showed you in a few slides ago. The total number of threats that I identified [26:16.020 --> 26:23.340] were 14 and the total number of critical threats for a generic node is 5. So, again, depending on [26:23.340 --> 26:29.600] how your ecosystem looks like, you know, if you're running a larger set of nodes and you have [26:29.600 --> 26:34.960] doctorized images and all that, your threat model will continue growing. This is just to get you [26:34.960 --> 26:42.420] started somewhere and provide you foundation. This is just a small glimpse into what the threat [26:42.420 --> 26:49.140] model looks like. We have the threat name, so node software generic vulnerabilities, the stride [26:49.140 --> 26:55.240] impact, so elevation of privilege, denial of service, severity, probability, description, [26:55.240 --> 27:00.700] most importantly, a variety of different mitigations that you can implement to address this threat. [27:04.720 --> 27:12.760] So, a key thing that we need to do when considering mitigations and the risk is to [27:12.760 --> 27:19.580] consider the cost. The cost, probability, and severity. So, the cost is how much time is it [27:19.580 --> 27:24.720] going to take you, how many resources you need to invest to address something. What is the severity [27:24.720 --> 27:32.660] and probability of this threat? So, in order to prioritize a variety of threats, I came up with [27:32.660 --> 27:41.460] this five-level, I guess, criticality grid where most critical flaws are the ones that are both [27:41.460 --> 27:50.480] have high impact, but also high probability of happening. Severe threats are the ones that have [27:50.480 --> 27:57.900] high impact, but maybe slightly lower probability of occurring. So, think of your generic web [27:57.900 --> 28:03.800] server getting exploited, which is running on your node versus some esoteric block [28:04.820 --> 28:09.180] parsing vulnerability that needs to be discovered to be exploited. And all the way to low, where [28:09.180 --> 28:13.880] these are the threats which we can address at one point, but only after we're done with [28:13.880 --> 28:21.860] the top two or top three, at least. So, of all these mitigations, [28:21.860 --> 28:26.800] in the model, I created about 41 mitigations that one could potentially implement. It could [28:26.800 --> 28:33.240] be probably more that we can think of, but I'll share the top 10 ones in a bit. Of those [28:33.240 --> 28:38.920] mitigations, they're addressing 22 critical threats. What's interesting is that half of those [28:38.920 --> 28:43.100] mitigations for critical threats are, in fact, very low cost. This is something that you can [28:43.100 --> 28:48.980] implement in a day and will significantly bolster the security stance of your nodes. [28:49.440 --> 28:55.780] So, yeah, you can start building fuzzers, you can start looking for every line of code of your nodes [28:55.780 --> 29:02.880] to make them secure, and they will make them very secure. But you can also do a bunch of things which [29:02.880 --> 29:09.120] are very low cost that will not make your nodes completely secure, [29:09.120 --> 29:15.060] but they will also, by the order of magnitude, make them resilient to some of the attacks that I [29:15.060 --> 29:22.380] described. So, this is just a different view that in the top bar of the critical vulnerabilities, [29:22.380 --> 29:29.500] the ones in light blue, you can see that half of them are basically low-cost solutions that [29:29.500 --> 29:34.560] you can implement. And then you have just, you know, if you really want to push it through this, [29:34.560 --> 29:44.040] like, it's 99.99% secure, you can continue on that trail, but that will be a little bit more costly. [29:46.420 --> 29:54.860] So, with that, let's talk about into node security, top 10 defenses that you can implement. [30:00.840 --> 30:07.660] So, the first one, and the most critical one, I think, is secure handling of where is the [30:07.660 --> 30:14.280] node software coming from, the repositories, the binary distributions, and so on. One key thing [30:14.280 --> 30:20.120] that we do at Coinbase, and something that you could consider as well, is repository pinning. [30:20.740 --> 30:25.820] Repositories can get compromised, but if you make sure that you pin to a specific non-Git version [30:26.350 --> 30:31.840] that you will work with and you will stick to until you go through some kind of process that, [30:31.840 --> 30:35.720] okay, I'm going to switch to the latest release instead of just downloading whatever is the [30:35.720 --> 30:41.360] latest one every time you deploy a node, that's something you could consider and could partially [30:41.360 --> 30:47.360] mitigate the risk of backdoored repos. Another thing is verification of signatures. So, if you're [30:47.360 --> 30:52.820] relying on either source code that you're downloading or binaries that you're downloading, [30:52.820 --> 30:56.240] make sure you verify all the signatures that, in the case of Monero, [30:57.820 --> 31:04.260] if the binaries were signed and the attackers did not have signed signatures correctly, [31:04.260 --> 31:08.860] then you would be able to detect that. In the case of source code, if you can at least verify, [31:08.860 --> 31:13.220] like, who is committing to the repositories, is it coming from, am I accessing a website, [31:13.220 --> 31:17.220] and it's, you know, the TLS is not breaking, no one is men in the middle of me trying to [31:17.220 --> 31:23.900] download the source code. Again, this is something that could make your nodes more secure. [31:26.160 --> 31:31.780] We continue on that train of thought, we can also make sure that we build all nodes from source [31:31.780 --> 31:37.160] code. I know this is a big ask, but this is something that you could consider, and the [31:37.160 --> 31:43.860] reason for this is, again, the Monero example, where only the binaries of Monero were backdoored, [31:43.860 --> 31:50.220] the source code remained intact. And the reason why, both could have been compromised, but if you [31:50.220 --> 31:55.940] consider how binaries for nodes are built, so there's a software repository with a lot of eyes, [31:55.940 --> 32:02.300] and then there could be a flow where one single developer with malware on their machine downloads [32:02.300 --> 32:07.880] that source code, builds the library, uploads the binary back on the website, there are a variety [32:07.880 --> 32:12.800] of things that could break along the way. Same goes with the CI automation of some sort, which [32:12.800 --> 32:17.360] builds binaries automatically. There are a lot more components which could be attacked and [32:17.360 --> 32:24.320] compromised, which is why, in case, again, lesson learned from that one incident, is that if you just [32:24.320 --> 32:29.080] stick to the source code, it may still be backdoored, but at least you have, like, one point of failure [32:29.080 --> 32:36.080] as opposed to a multitude of others, which are required when building binaries. Secure node [32:36.080 --> 32:42.700] configuration. So, every single node that I looked at has a whole variety of different levers that you [32:42.700 --> 32:48.300] can pull to make them secure. Some things that you should, generically, you could consider is make [32:48.300 --> 32:54.120] sure that you can diversify the number of connections that your node can make, and to which [32:54.120 --> 33:01.080] nodes it can make, it can connect to. So, an example would be things like Stellar, I believe. They need [33:01.080 --> 33:07.760] to be manually configured to distribute which nodes that you connect to. So, make sure that if [33:07.760 --> 33:14.920] you configure nodes like that, you select as wide of a node selection as possible. Others, they [33:14.920 --> 33:19.460] restrict the number of connections total, so you want to see, like, what is a reasonable number [33:19.460 --> 33:24.360] of connections you want to maintain throughout the network. This will help you with Eclipse attacks, [33:24.360 --> 33:32.240] making more resilient against forks, and so on. Another important key on configuring nodes [33:32.240 --> 33:38.560] securely is locking down your RPC interfaces. So, I believe last year, there was mass scanning of [33:38.560 --> 33:46.540] Ethereum nodes that were looking for open RPC interfaces to basically steal private keys. So, [33:46.540 --> 33:51.140] lock down your RPC interface. There's no reason why those should be exposed to the rest of the world. [33:51.140 --> 33:56.840] So, make sure you configure, you know, whatever IP or interface that RPC is listening on, [33:56.840 --> 34:00.560] make sure it's configured to something local to your network or just local host. [34:02.280 --> 34:08.760] Restrict node access. So, not everyone needs constant access to your nodes. Not everyone [34:08.760 --> 34:14.200] needs to be able to modify arbitrary things within your node. So, at Coinbase, what we do [34:14.200 --> 34:21.240] is we create a consensus mechanism with no configuration changes, no nodes, no one can bring [34:21.240 --> 34:26.120] up and down nodes just arbitrarily. It requires a consensus of people. The more critical your [34:26.120 --> 34:32.220] infrastructure is, the more people it requires a sign-off. So, you can implement it through a [34:32.220 --> 34:38.340] sign-off, through some kind of two-factor pings that you can send to people to approve any given [34:38.340 --> 34:46.440] action. Define and enforce administrator roles. Again, this is something to restrict to security [34:46.440 --> 34:51.740] nodes in larger environments, but you can apply this to individual nodes as well. Make sure that [34:52.300 --> 34:59.120] there are some specific roles set up for the node that, you know, you can either configure nodes and [34:59.120 --> 35:03.720] deploy them or shut them down, but not everything at the same time. So, you can get very granular [35:03.720 --> 35:10.120] there, but it depends, see how appropriate it is for your specific environment. Finally, restrict [35:10.120 --> 35:18.220] node access, as in the network traffic node access, both ingress and egress. So, on one side, [35:18.220 --> 35:22.780] it makes sense that you restrict your RPC interface already, so make sure those are firewalled off, [35:22.780 --> 35:27.480] that you cannot access nodes from anywhere, and then externally, you also have some kind [35:27.480 --> 35:33.380] of restriction set up as well. On the other side, nodes could be an excellent points of entry for [35:33.380 --> 35:37.920] the attacker who found some kind of vulnerability into the rest of your network. So, make sure that [35:37.920 --> 35:42.020] your nodes cannot communicate to the rest of your network as well, because they could be used as [35:42.020 --> 35:50.460] stepping stones. Monitor and log node activity. So, this one is probably the most interesting one, [35:50.460 --> 35:56.380] and there's a lot of things you can do there. Anytime your node loses internet connectivity, [35:57.480 --> 36:02.160] make sure you do some basic verification. Am I getting... is there like something with BGP, [36:02.160 --> 36:07.980] or am I getting... is my DNS going down? Am I under attack? Am I getting eclipsed? So, [36:07.980 --> 36:12.900] make sure you investigate those. Am I on the right fork? Am I not seeing sufficient traffic? [36:13.000 --> 36:18.820] All of that applies. You can do node-specific verifications too, not just generic network [36:19.780 --> 36:24.920] monitoring, such as periodically check, like, am I on the right fork? What is the block height? [36:24.920 --> 36:30.700] What is the current block hash? Do other nodes in my system, or externally, or on blockchain [36:30.700 --> 36:37.440] explorers, do they still see the same thing? So, in case of the recent ETC experiment incident, [36:37.440 --> 36:41.320] you would have been able to quickly diagnose, like, oh, okay, so there's something going [36:41.320 --> 36:48.480] on with nodes because there's some kind of forking happening. Finally, log all access, [36:48.480 --> 36:54.720] all network events. It may not be as essential during the normal operation of node, but when [36:54.920 --> 36:58.900] something goes wrong and you need to investigate and figure out what happened, who accessed it, [36:58.900 --> 37:02.520] what were the network events that led up to the attack, those logs will be [37:02.520 --> 37:05.400] highly instrumental for the investigation. [37:07.620 --> 37:13.520] Lockdown-based OS. So, definitely, you cannot build a secure node software if your underlying [37:13.520 --> 37:20.340] layer is insecure. If, again, if you want to go after the weakest link, and your node is [37:20.340 --> 37:26.640] very well configured and firewalled off, but your underlying OS is some outdated version of Linux, [37:26.640 --> 37:31.700] which has, like, exposed ports and they're easily attacked, the attacker is just going to go [37:31.700 --> 37:36.500] after that and going to skip all of the other mitigations you built. So, you could do things [37:36.500 --> 37:41.700] like doctorize your nodes, run SCLinux or similar toolkits. So, there are a variety of things you [37:41.700 --> 37:47.120] can do to lock down the OS, and there are guides for that, but something to consider as well. [37:48.820 --> 37:54.880] Hardened node's network connections. So, going off the stack a little bit, not just protecting [37:54.880 --> 37:59.880] what connections can be made, or what we're monitoring, make sure that if you configure DNS [37:59.880 --> 38:05.980] server for your node, make sure it's a secure one, uses DNSSEC. If you're using ISPs, make sure that [38:05.980 --> 38:10.360] whatever BGP connections that it uses are done in a secure fashion. If your node needs to [38:10.360 --> 38:15.720] communicate to some kind of HTTP resources, make sure you configure your CAs properly, [38:15.720 --> 38:21.520] so you're not getting met in the middle. Denial of service protection is critical. Again, it's [38:23.360 --> 38:27.860] potentially hard to determine. I want to attack just one particular node for one particular [38:28.460 --> 38:32.580] user or organization, but it's prudent to use something like Cloudflare to [38:32.580 --> 38:37.280] have a layer of denial of service protection, so you don't have to worry about that. [38:38.500 --> 38:45.500] Node-specific threats. Node and protocol-specific threats. This is a catch-all. Basically, [38:45.940 --> 38:51.180] be mindful that all nodes are unique in their own different ways. So, I mentioned before that [38:51.180 --> 38:55.740] Stellar, you have to manually configure what are the trusted nodes. You have to manually calculate, [38:55.740 --> 39:01.000] like, hey, what is my threshold for... it's a BFT network. What is my threshold that [39:01.000 --> 39:05.980] it can get attacked? So, make sure that I'm not going to be vulnerable to that. [39:05.980 --> 39:14.160] Other protocols like EOS, they implement debugging modes that you have to watch out for. [39:14.160 --> 39:19.120] So, read the config file completely. Make sure you understand what are the implications of all the [39:19.120 --> 39:26.100] variety of settings, and have a good secure configuration specific for that node. [39:28.260 --> 39:32.600] Verify node functionality before deployment. So, that's a much harder one, [39:32.600 --> 39:39.860] because, sure, you pinned the repository. You are pretty confident, okay, so it probably was [39:39.860 --> 39:45.340] not modified, but I'm not 100% sure that whatever I downloaded is working as it's [39:45.340 --> 39:50.500] supposed to. There are no backdoors there. How do you verify that it still works as intended? [39:51.180 --> 39:55.120] Well, ideally, you would run a set of tests, like, hey, if I transmit this transaction, [39:55.640 --> 39:59.800] it shows up on the network, or if I see a transaction coming from the network, [39:59.800 --> 40:06.640] my account gets properly credited. And that's a big ask to implement for every single node. [40:06.880 --> 40:14.060] So, one project which aims to solve this problem is Coinbase Rosetta project, which creates a kind [40:14.060 --> 40:19.020] of a middleware for a variety of different nodes to be able to speak the same API. So, [40:19.020 --> 40:26.820] instead of implementing a variety of different calls to transmit a transaction, get a current [40:26.820 --> 40:33.800] block hash and transactions in it, Rosetta creates a kind of a middleware for a variety [40:33.800 --> 40:40.120] of different nodes that you can just use a single API call, and it automatically translates to [40:40.120 --> 40:45.840] whatever is specific for that node. Now, what's interesting about the Rosetta project also [40:45.840 --> 40:52.160] includes a Rosetta validator, which exercises the node and makes sure that it passes a variety of [40:52.160 --> 40:59.380] very specific tests from transaction verification blocks and signing and so on to make sure that it [40:59.380 --> 41:07.040] works exactly as intended. Not every project supports Rosetta interface, but for the ones [41:07.040 --> 41:13.140] that do, you can implement it in the pipeline to make sure that the nodes that you deploy are acting [41:13.140 --> 41:21.320] as expected. And you notice that we're slowly ramping up the difficulty level. This is probably [41:21.320 --> 41:28.340] the hardest one. This requires you to examine the source code, examine the alerts that you're [41:28.340 --> 41:36.880] getting, and this is the basic security static analysis of node software. So, [41:36.880 --> 41:40.880] you may not need to go through the actual source code, but you can observe a variety of different [41:40.880 --> 41:46.320] alerts reported by tools like Salus. So, this is another Coinbase tool, which we use internally, [41:46.320 --> 41:53.400] and it's an open source tool you can use as well to quickly run through a variety of node [41:53.400 --> 41:59.360] source code packages, and it gives you a nice report of what are the high critical vulnerabilities [41:59.360 --> 42:04.560] within that. So, you can examine them, just get a general sense. It's hard to say which ones are [42:04.560 --> 42:07.900] false positives without actually looking at the source code, but you can see like, hey, if I [42:07.900 --> 42:13.860] suddenly see 100 or so, or like this really sneaky report, maybe I should investigate more or report [42:13.860 --> 42:21.000] or ask the original developers. In the Salus project, it supports a variety of different [42:22.600 --> 42:31.440] languages from Go to Ruby and so on. GoSec, like for if you're just running Ethereum or other [42:31.440 --> 42:39.260] Golang-based nodes, so GoSec is excellent. In fact, Salus is running a variety of open source [42:39.260 --> 42:44.460] projects, and it just creates a wrapper for them. So, GoSec is one of them. For all the [42:44.460 --> 42:48.760] other projects, you have something like Semgrep, so if you have SQL to analyze and so on, [42:48.760 --> 42:56.460] so you can use that as well. So, with that, just to summarize, what is our total view of [42:57.460 --> 43:04.000] all the different mitigations, top 10 things that you can do to secure your nodes. So, [43:04.000 --> 43:08.860] make sure you handle software repositories securely. Make sure that you pin them, [43:08.860 --> 43:16.320] that you understand that they can be compromised, and you need to just investigate a little bit [43:16.320 --> 43:25.040] more. Build all nodes from source code. So, again, it's just reducing your attack vectors just a [43:25.040 --> 43:28.600] little bit more. You just build everything from source. There are a lot of eyes in source code. [43:28.600 --> 43:33.940] Who knows what happens with those binaries? Securely configure nodes. So, make sure that [43:33.940 --> 43:40.200] you diversify what connections your nodes have. Make sure that they can't be just as easily [43:40.200 --> 43:48.600] eclipsed and people can access it and reconfigure them. Restrict node access. Monitor and log node [43:48.600 --> 43:53.760] activity. So, you can get, again, very creative with those, what you can do to judge what nodes [43:53.760 --> 44:00.580] help. Lock down your base OS. Make sure that you're not running, you're not building a castle [44:00.580 --> 44:08.060] on quicksand. Definitely consider node and protocol specific threats. All nodes are unique [44:08.060 --> 44:13.460] in a variety of different ways. So, you need to consider those as well. Verify node functionality. [44:13.460 --> 44:20.500] So, if your node supports Rosetta project, run Rosetta validator. It will give you this good [44:20.500 --> 44:26.920] feeling that, okay, at least for a variety of tests, this still works as expected. And then, [44:26.920 --> 44:31.780] if you have time and you have technical expertise, try to run basic static analysis tools [44:31.780 --> 44:34.720] just to periodically see, like, are there, like, major [44:34.720 --> 44:38.220] low-hanging fruit vulnerabilities that are getting introduced that I can catch, [44:38.220 --> 44:44.980] maybe report to original developers. So, with that, thank you so much for your time. [44:45.920 --> 44:51.480] I may have not mentioned this. I'm the blockchain threat intelligence newsletter. I list a lot of [44:51.920 --> 44:57.020] node incidents and vulnerabilities that are getting discovered. So, subscribe. See if you [44:57.020 --> 45:03.200] can catch the latest lessons learned from those vulnerabilities. And feel free to follow me on [45:03.200 --> 45:10.300] Twitter. And or feel free to contact me offline if you have any more questions. Thank you.