1
00:00:05,020 --> 00:00:18,560
Hello and welcome to Attacking and Defending Blockchain Nodes, where we're going to learn about a variety of risks and mitigations that we can implement to defend a very core component of the blockchain systems.

2
00:00:20,000 --> 00:00:33,080
We'll try to do a live discussion during the talk while we ask a question, so feel free to join DEF CON's Discord on BCV General Tech's channel, and I'll try to ask questions and respond live.

3
00:00:34,380 --> 00:00:37,400
In case we haven't met, my name is Peter Kaczynski.

4
00:00:37,620 --> 00:00:46,800
I'm a blockchain security engineer at Coinbase, where I spend most of my time breaking a variety of different blockchain systems and smart contracts.

5
00:00:46,940 --> 00:00:59,060
I'm also a writer for the Blockchain Threat Intelligence Newsletter, where I try to cover all the latest news, events, hacks, scams in the blockchain security ecosystem.

6
00:00:59,840 --> 00:01:06,140
Last year, you may have also participated in the Capture the Coin competition, which I helped organize.

7
00:01:06,840 --> 00:01:22,040
My background before deep diving into the cryptocurrency world was as a malware reverse engineer at FireEye, where I looked at a whole lot of APT malware and a penetration tester for the Federal Reserve System, where I was breaking Finance 1.0.

8
00:01:24,120 --> 00:01:29,700
So, just some idea, give you an idea of what we're going to talk about.

9
00:01:29,700 --> 00:01:31,660
I'm not going to drop any old days today.

10
00:01:31,980 --> 00:01:36,520
What my goal here is to educate, to share knowledge, to also learn from you.

11
00:01:37,180 --> 00:01:42,420
With the goal of just making, discussing, like, how can we secure node infrastructure?

12
00:01:43,080 --> 00:01:50,680
So we'll start with a simple exercise on how do we know, what is the state of truth that tells us how many coins we actually have?

13
00:01:50,940 --> 00:01:59,620
We'll go through a variety of examples of how nodes break, and we'll discuss a few of the ways that they could break in the future.

14
00:01:59,900 --> 00:02:09,820
Finally, I'll introduce a solid node security threat model for generic nodes, which you can apply later on for a specific node implementation .

15
00:02:09,980 --> 00:02:24,360
And most importantly, discuss a top 10 list of security defenses that you can implement today, whether you're a standalone operator, or you're working for an exchange, and share lessons learned from trying to secure nodes of my own.

16
00:02:25,200 --> 00:02:29,700
So with that, let's talk about how many coins you have.

17
00:02:29,700 --> 00:02:40,000
So what is, if you're running, if you wanted to, at any given moment, tell me how many coins of any given asset that you have, how do you go about that?

18
00:02:40,660 --> 00:02:52,620
So, well, you'll probably have a piece of software on your computer, on your phone, or a hardware wallet, maybe, which has some specific logic dedicated just for a specific asset.

19
00:02:52,620 --> 00:02:58,200
So if you have a Bitcoin wallet, or Ethereum, or Monero, or some other coin, sorry if I didn't list here.

20
00:02:59,840 --> 00:03:06,940
This particular wallet communicates with a wider network, somewhere to the cloud, and asks a question, hey, how much coins do I have?

21
00:03:07,040 --> 00:03:09,020
What do I actually own?

22
00:03:09,460 --> 00:03:14,000
In case of Bitcoins, you try to enumerate how many outputs that you have unspent.

23
00:03:14,440 --> 00:03:23,980
Ethereum is different in the sense that it's an account-based system, so you have to look up how much of the asset that I actually have on the network.

24
00:03:24,740 --> 00:03:27,640
The thing is, it gets a little bit more complicated.

25
00:03:27,640 --> 00:03:36,440
So if we start getting a little bit more granular, we're going to see that, in fact, we don't communicate with just some unknown entity, abstract entity.

26
00:03:36,440 --> 00:03:40,900
We actually communicate to specific nodes for that specific coin.

27
00:03:40,920 --> 00:03:47,600
So Bitcoin wallet communicates to Bitcoin nodes, Ethereum to Ethereum, Monero to Monero, and so on.

28
00:03:47,940 --> 00:03:56,580
This introduces some complications, because we realize that there are a lot of third-party components involved.

29
00:03:56,580 --> 00:04:05,360
So if we just want to manage our assets, we have to communicate to a whole bunch of software which is maintained by third parties.

30
00:04:06,600 --> 00:04:17,660
So just to give you like a few more very simplistic threat model here, let's talk about components of a friend's coin, just the imaginary coin from the Mario friend's land.

31
00:04:17,680 --> 00:04:28,060
So you have your wallet communicating to a node, communicating to the overall gossip network of some sort, trying to propagate blocks.

32
00:04:28,140 --> 00:04:43,360
You also have your developers who are contributing to the GitHub repository, so you have to rely on maintaining the repository, compiling the node maybe, or downloading binaries there, and so on.

33
00:04:43,360 --> 00:05:01,500
And this all works really well, and until we start thinking that, well, how can we be sure that all the software libraries, all the standalone software infrastructure does not act maliciously, does not drop, modify, or craft transactions to steal your funds.

34
00:05:02,640 --> 00:05:04,320
So let's see how this could happen.

35
00:05:04,320 --> 00:05:05,220
So we have this Mr.

36
00:05:05,220 --> 00:05:10,380
Robot character trying to attack a variety of different points of this threat model.

37
00:05:10,380 --> 00:05:19,540
So we'll start left to right, so we can see that we can attack the network connection between your node and the cloud.

38
00:05:19,540 --> 00:05:29,240
So someone could start dropping blocks, try to eclipse attack your node, so basically to isolate it to make sure it's not communicating to the rest of the network properly.

39
00:05:30,560 --> 00:05:39,680
The same character can also attack your node directly if there are some unknown weaknesses that they can exploit to do some kind of code execution or crash it.

40
00:05:39,680 --> 00:05:42,160
They can prevent you from accessing the network.

41
00:05:42,960 --> 00:05:46,140
At last, we also have to think about developers.

42
00:05:46,160 --> 00:05:51,320
The bad actors out there, they can go after developers like the toad over there.

43
00:05:51,320 --> 00:05:55,840
They can make them commit bad code.

44
00:05:56,020 --> 00:05:59,600
And I'll cover a few real-world examples where this does happen.

45
00:06:00,540 --> 00:06:10,520
So the idea is that it's an invitation to start thinking about the node security as a holistic ecosystem.

46
00:06:10,520 --> 00:06:13,100
So there are a variety of different components which are involved.

47
00:06:15,040 --> 00:06:22,080
So the examples that I just mentioned, they're not theoretical, they're practical, they're based on real-world incidents.

48
00:06:22,080 --> 00:06:31,600
So for example, in May 2014, someone attacked the BGP network used to eclipse attack Doge and Bitcoin miner pools.

49
00:06:32,440 --> 00:06:46,840
So this was, in case you're not familiar with BGP hijacks, it's a way to effectively isolate whole large swaths of network blocks in order to make them route traffic to unexpected directions.

50
00:06:46,840 --> 00:06:55,600
This is something that happened a decade ago, I believe with YouTube, where all the traffic was redirected to to Pakistan, I believe.

51
00:06:55,600 --> 00:06:57,600
And it keeps happening again and again.

52
00:06:59,180 --> 00:07:08,700
This is a threat that we have not observed within cryptocurrency world for a while, but it's something that we have to be aware of and build threat models around.

53
00:07:09,400 --> 00:07:15,100
On the EOS side, after the release of the Node software, they published a bug bounty.

54
00:07:15,300 --> 00:07:21,900
And oh boy, did they receive a lot of good finds there, because one, there was immediately a code execution flaw that was discovered.

55
00:07:22,160 --> 00:07:34,000
And for a while at least, they kept on getting one after another of high-value bounties submitted to them, which included denial of service, code execution, and similar.

56
00:07:34,580 --> 00:07:39,140
Finally, in 2019, Monero website was compromised.

57
00:07:39,360 --> 00:07:43,800
The binaries for the wallet were backdoored to steal clients' funds.

58
00:07:43,800 --> 00:07:47,900
So this is an example of how we have to look out for the developer.

59
00:07:47,900 --> 00:07:49,160
Where are you getting that software?

60
00:07:49,160 --> 00:07:51,040
How trustworthy is that resource?

61
00:07:51,040 --> 00:07:52,920
What can you do to verify it?

62
00:07:53,780 --> 00:07:59,680
So this was just over the few years, but I just want to instill in you that this is an ongoing problem.

63
00:07:59,680 --> 00:08:02,600
These are all the different things happening this year.

64
00:08:02,820 --> 00:08:12,280
So in March, we had on the testnet, but we still had a flaw discovered in the way Solana was processing transactions, where it was not doing sufficient validation.

65
00:08:12,280 --> 00:08:16,680
So it resulted in 500 million Solana getting stolen.

66
00:08:16,680 --> 00:08:28,880
Filecoin, again, had a nasty inflation bug, fortunately discovered on testnet, exploited also on testnet, but it resulted in 9 billion Filecoins minted, when only 2 billion could have existed in the first place.

67
00:08:28,880 --> 00:08:37,320
Inflation bugs are particularly nasty because these are the attacks not against individual users, but against the entire coin ecosystem.

68
00:08:37,820 --> 00:08:40,720
Tendermint is a library behind a variety of projects.

69
00:08:40,720 --> 00:08:46,080
It's a consensus mechanism used in projects like Cosmos, Oasis, and others.

70
00:08:46,720 --> 00:08:54,860
A denial of service vulnerability was discovered there when parsing specially crafted invalid blocks, which basically crashed the nodes.

71
00:08:54,860 --> 00:09:04,260
So if this was successfully executed in the wild, Cosmos was likely not vulnerable to this because they didn't upgrade just to that particular vulnerable version.

72
00:09:04,320 --> 00:09:10,920
But if that was the case and the vulnerability was maliciously discovered, then it would result in a network halt.

73
00:09:11,520 --> 00:09:21,160
The last example, and probably the most critical and interesting one, both from the impact and the way that it was handled, is Ravencoin.

74
00:09:21,160 --> 00:09:26,520
Ravencoin had an inflation bug discovered just last month in July.

75
00:09:27,440 --> 00:09:43,640
The inflation bug vulnerabilities basically allow someone to print money out of thin air, which is extremely dangerous because it allows an attacker just to profit and sell value to exchanges, and it basically impacts every single user of the coin.

76
00:09:43,640 --> 00:09:49,180
What's interesting about this particular vulnerability was that it was actually maliciously introduced.

77
00:09:49,180 --> 00:10:01,260
It was introduced back in January by an account which was seemingly making adjustments to the way the comments were made, on the debugging views were made.

78
00:10:02,360 --> 00:10:13,520
And the core developers unfortunately missed that until the point that someone reported them, a third party, which was building a blockchain explorer for Ravencoin, there's something weird going on in your network.

79
00:10:13,520 --> 00:10:21,220
We're observing someone minting a whole bunch of RVN and also, by the way, selling them on exchanges and profiting from it.

80
00:10:21,600 --> 00:10:36,260
So the Ravencoin team was able to issue an emergency patch, worked with the miners, but it's still 300 million RVNs were minted and exchanged for some other coins.

81
00:10:36,780 --> 00:10:40,200
Something that we have to learn from.

82
00:10:42,140 --> 00:10:53,660
So before we go into the section on how to secure the nodes, how to look out for those vulnerabilities, let's first think about all the different ways that we can break the nodes.

83
00:10:54,460 --> 00:11:02,400
So just to define the scope right away, we're not going to talk about the features of a node, which are documented and designs of the node.

84
00:11:02,400 --> 00:11:11,260
So if something implements proof of work, we're not going to deep dive into 51% attacks, reorgs, other protocol design issues.

85
00:11:11,260 --> 00:11:13,280
We'll accept them as is.

86
00:11:13,280 --> 00:11:21,560
What we're more concerned about is if the node promises to do something correctly, and it fails to do that, that's what we need to pay attention to.

87
00:11:22,340 --> 00:11:25,420
Key management is also outside of this.

88
00:11:25,420 --> 00:11:31,200
So you should probably be running nodes on a variety of tiers.

89
00:11:31,200 --> 00:11:37,780
So if you just want something which replicates the state of the network, you should probably keep your keys on a separate system.

90
00:11:38,000 --> 00:11:39,580
Incorrect usage.

91
00:11:39,580 --> 00:11:50,320
So I assume that if you're running a specific node that you read the documentation, you read everything that is known about it, so you don't necessarily make mistakes.

92
00:11:50,320 --> 00:12:06,920
Like a good example of that is Ripple's TF partial payment flag, where if you do not implement that particular flag correctly, then whenever someone sends you some payment with that flag set, it may appear that you actually receive those funds.

93
00:12:06,920 --> 00:12:14,500
But in reality, it's a IOU, essentially, that I will send you those things eventually, but not right now.

94
00:12:14,500 --> 00:12:17,420
So a lot of exchanges were exploited as a result of that.

95
00:12:17,840 --> 00:12:24,460
Finally, we're not going to focus on the layer two vulnerabilities such as smart contracts, DeFi, and so on.

96
00:12:25,080 --> 00:12:26,540
So what's in scope?

97
00:12:26,660 --> 00:12:29,160
What's in scope are the implementation flaws.

98
00:12:29,160 --> 00:12:36,540
So protocol, software flaws, attack resilience, can the node stand up to sustained attacks?

99
00:12:36,920 --> 00:12:37,660
Infrastructure.

100
00:12:37,660 --> 00:12:44,580
So we're not looking at nodes as just this one thing, and we're protecting just the node itself, and don't care about anything that it's built in software.

101
00:12:44,580 --> 00:12:50,580
We're worrying about underlying OS, we're worrying about the network stack, and anything that goes around it.

102
00:12:51,080 --> 00:12:52,340
Finally, management.

103
00:12:52,340 --> 00:12:57,620
Human factor is, always was, always will be the weakest link in the security chain.

104
00:12:57,700 --> 00:13:04,800
So we will discuss access management, we'll talk about configuration, source control, and all other ops type of things.

105
00:13:05,600 --> 00:13:14,320
So with that, just before we dive into the exercise, just wanted to establish some terminology so we're all on the same page.

106
00:13:14,840 --> 00:13:22,700
The threat model, and this is taken from OWASP's excellent threat modeling cheat sheet if you want to review it and build your own threat models.

107
00:13:23,780 --> 00:13:28,360
Threat model itself is a process to identify potential threats.

108
00:13:28,380 --> 00:13:35,200
So we want to enumerate all the different ways that things can go wrong, we can assess those threats, what is the risk coming from them?

109
00:13:35,200 --> 00:13:45,180
Are they basically going to result in me losing money, or is it just a denial of service, or some other maybe less critical impact?

110
00:13:45,880 --> 00:13:49,400
And also later on, we'll use that to prioritize mitigations.

111
00:13:49,400 --> 00:13:57,280
Why would we spend hours or weeks trying to address a risk which is not really that impactful?

112
00:13:58,320 --> 00:14:01,440
An important thing that we're going to do next, we'll define the attack surface.

113
00:14:01,440 --> 00:14:20,040
An attack surface, think of it as all the different ways that someone can access, can attack your node or node infrastructure, and the threat agents, which are the entities which have enough skill, opportunity, and interest to attack your nodes.

114
00:14:20,960 --> 00:14:32,920
So let's begin with just a really basic system, it's not interacting with any other nodes, it's not managed by anything, just like a really basic core software.

115
00:14:32,920 --> 00:14:52,320
So we have the node itself, which all it does is it's processing the packets, it's processing the transactions, blocks, does verification, it stores the blocks in some kind of storage system, it parses configuration files so it knows how to properly look at those things,

116
00:14:52,320 --> 00:15:05,600
and it may also include a VM interpreter process where it can interpret smart contracts or anything like full-blown, Turing-complete, Ethereum-style smart contracts, or something a little bit more locked down like Bitcoin script.

117
00:15:07,080 --> 00:15:10,840
Now, let's add some more components to that.

118
00:15:11,160 --> 00:15:27,400
In order for node to be useful, it needs to have some kind of management interface, and most nodes implement some kind of RPCE network client, so we have to think about the threats coming from that, who can access this thing, what is it they can do, and of course,

119
00:15:27,400 --> 00:15:37,780
node administrators themselves who are managing the machine, they can shut them down, they can bring it back online, they can reconfigure them and maybe introduce some flaws that I will cover in a moment.

120
00:15:38,740 --> 00:15:44,380
And also, these are just humans, can they be attacked on their own with malware or maybe any other social engineering?

121
00:15:45,640 --> 00:15:59,920
Finally, nodes need to communicate to other nodes to figure out what is the state of the network, and this is where a whole bunch of different things are coming into play, like network infrastructure, other nodes that we need to talk to, and miners.

122
00:15:59,920 --> 00:16:19,040
So network infrastructure includes basically underlying layers of communication for the nodes, so think of BGP and DNS client, DNS services, and just basic routing like TCP connections, UDP, all of that is in scope and needs to be secured as well.

123
00:16:19,720 --> 00:16:24,780
Finally, nodes are just not appearing on your machines out of nowhere, they must come from somewhere.

124
00:16:24,780 --> 00:16:33,000
Unless you are the developer, you probably are relying on third-party repositories, which also in turn rely on third-party dependencies.

125
00:16:33,380 --> 00:16:51,760
So this is something that we need to be aware of, because as I mentioned with Ravencoin, someone was able to backdoor the assets through a very sneaky code change and something that we also need to consider within our threat model.

126
00:16:53,540 --> 00:17:05,180
So with that in mind, join the BCV General Text on the DEF CON Discord channel and I'll ask you questions to just brainstorm together on how we can break nodes.

127
00:17:06,760 --> 00:17:08,960
So the first thing we're going to attack is the core.

128
00:17:08,960 --> 00:17:12,300
This is the node software itself.

129
00:17:12,300 --> 00:17:21,000
This is what processes the blocks, the transactions, and needs to verify them correctly.

130
00:17:21,420 --> 00:17:24,280
If you are this sneaky Mr.

131
00:17:24,280 --> 00:17:27,980
Robot character and you wanted to attack this, how would you do it?

132
00:17:27,980 --> 00:17:31,320
So I'll look in the channel for your responses.

133
00:17:39,860 --> 00:17:43,800
So I have one comment of software supply chain attack.

134
00:17:43,800 --> 00:17:51,780
So this is something that we're going to cover in the software repository attack, but I'm talking about the core itself.

135
00:17:51,780 --> 00:17:57,600
That's the piece of software that does the actual blockchain logic.

136
00:17:59,440 --> 00:18:02,060
BGP attacks to create an eclipse.

137
00:18:03,160 --> 00:18:10,300
So again, so this is something that we have to worry about on the network connectivity side.

138
00:18:16,190 --> 00:18:16,830
Let's

139
00:18:22,420 --> 00:18:22,740
see...

140
00:18:23,060 --> 00:18:30,280
Yeah, so a denial of service against the nodes to crash with the bad blocks, malformed block headers.

141
00:18:30,300 --> 00:18:32,280
So that's exactly that.

142
00:18:32,280 --> 00:18:46,540
So we can craft a packet, think of a tendermint flaw, which basically crashes a node or maybe executes some kind of really nasty code execution attack.

143
00:18:47,080 --> 00:18:51,740
So just to give you some background, this is how I created the model.

144
00:18:51,740 --> 00:18:54,640
So there's a title, protocol vulnerabilities, severity.

145
00:18:54,640 --> 00:18:59,560
So if you can crash or do code execution, know the severity is high.

146
00:18:59,760 --> 00:19:01,920
Probability, I would say medium.

147
00:19:01,920 --> 00:19:04,480
It's just harder to find, harder to exploit.

148
00:19:04,480 --> 00:19:08,220
It requires dedicated time to discover those things.

149
00:19:08,340 --> 00:19:20,300
But the impact is that anything that is not properly processing blocks, transactions, what is the correct fork, messing with the governance and so on, inflation vulnerabilities, this is all in scope for this.

150
00:19:20,380 --> 00:19:23,900
And it will have a severe impact on the node.

151
00:19:24,320 --> 00:19:29,560
However, I want you to consider it's less sexy of an attack.

152
00:19:29,560 --> 00:19:33,820
So we're not trying to find some kind of ode within blocks and transactions.

153
00:19:34,160 --> 00:19:38,640
And consider that nodes actually run all sorts of third-party software.

154
00:19:38,640 --> 00:19:47,500
So for example, Parity Nodes, they run or OpenTheorem now, they run web UIs to manage the thing.

155
00:19:47,940 --> 00:19:56,040
A bunch of node software relies on third-party database libraries and standalone databases to store blocks.

156
00:19:56,040 --> 00:19:58,980
So all of that are fair game to attack as well.

157
00:19:59,100 --> 00:19:59,960
So if you are Mr.

158
00:19:59,960 --> 00:20:10,260
Robot and you wanted to attack your node infrastructure, I'm not going to spend time trying to find old days for the way that you parse packets.

159
00:20:10,260 --> 00:20:23,220
It's a lot easier for me to find out what is the weakest link, what is the common off-the-shelf software that you're running, such as if you're running Nginx and for some reason it's a vulnerable version , I'll try to attack that as well.

160
00:20:23,220 --> 00:20:29,200
I'll try to attack that instead because it probably has more eyes on it and more vulnerabilities discovered so far.

161
00:20:29,440 --> 00:20:41,100
So consider those as your first line of things to lock down and the probability is definitely higher than finding a very directed attack to like in the way you parse blocks.

162
00:20:41,840 --> 00:20:45,580
So let's continue in the same style.

163
00:20:45,580 --> 00:20:47,080
So software repository attacks.

164
00:20:47,080 --> 00:20:51,360
So someone in the channel already mentioned that, you know, the supply chain attack.

165
00:20:51,400 --> 00:21:11,440
So whether you're attacking the software repository itself, so you can, you know, like a Ravencoin example or Monero example, if you can compromise the software repository itself, then you can introduce arbitrary changes, which are maybe different degrees of how hidden they are,

166
00:21:11,440 --> 00:21:14,180
depending how many eyes are looking at the repository.

167
00:21:15,460 --> 00:21:21,520
And again, this can result in fund theft or money printing type of vulnerabilities.

168
00:21:21,880 --> 00:21:30,820
However, what I want to invite you to also consider is not just the main software repository, but also software dependencies.

169
00:21:30,900 --> 00:21:41,380
So an example, not as actually a node vulnerability, but I believe it was Trinity Wallet, which had a, it was backdoored on the source code level.

170
00:21:41,880 --> 00:21:47,020
But the way that it was backdoored was not through the main repository, but through one of the dependencies.

171
00:21:47,020 --> 00:21:59,200
So someone figured out what is the weakest link in the supply chain to that particular piece of software, backdoored that, and then they were able to introduce code that was stealing keys.

172
00:21:59,780 --> 00:22:04,800
So again, probability is high on that as well, and severity is also extremely high.

173
00:22:07,240 --> 00:22:11,680
I mentioned this before, you know, human factor is always the weakest one.

174
00:22:11,680 --> 00:22:14,520
So node administrator, how can we attack that?

175
00:22:17,760 --> 00:22:23,580
Yep, phishing, send phishing email with some kind of malware.

176
00:22:24,660 --> 00:22:28,720
You know, the impact is very high.

177
00:22:28,860 --> 00:22:40,460
They can reconfigure the node, they can introduce a set of nodes, which they essentially creates an eclipse attack where you only communicate to something that is controlled by the attacker.

178
00:22:40,680 --> 00:22:51,080
Certain blockchains, they have a switch you can flip, which basically allows it to accept all blocks and transactions without doing any verifications, like a debugging mode almost.

179
00:22:51,120 --> 00:22:58,760
There's a variety of things that administrator can do and the variety of ways that administrator can get attacked through phishing, malware, social engineering, you name it.

180
00:23:00,440 --> 00:23:04,900
The last, you get the gist of how we're going through different components.

181
00:23:04,900 --> 00:23:08,060
The last one I want to cover is the network attack.

182
00:23:08,300 --> 00:23:12,340
So, correct, so BGP attacks to create an eclipse.

183
00:23:13,400 --> 00:23:25,760
So, exploiting the network infrastructure, you know, all you know, a lot of nodes, the way that they're built, is that they need to first boot up.

184
00:23:25,760 --> 00:23:40,880
When they first come online, they need to communicate to something hard-coded to figure out what is the network truth, like what are the basic trusted infrastructure that I can use to start getting an idea what is the current block state.

185
00:23:41,440 --> 00:23:50,620
A lot of them use protocols like DNS in order to find out what is the first set of nodes that I need to connect to.

186
00:23:51,460 --> 00:23:59,800
And if your underlying DNS infrastructure is compromised, they can, again, try to do some kind of eclipse against you.

187
00:24:00,000 --> 00:24:01,720
Same with denial of service.

188
00:24:01,720 --> 00:24:07,420
If your nodes can never communicate over those protocols, then they will never be able to connect to the network in the first place.

189
00:24:07,680 --> 00:24:15,740
BGP routing attacks, those are extremely damaging and something that needs to be considered.

190
00:24:15,920 --> 00:24:27,720
I'm putting the severity as high on these, but probability lower, just because it's a lot harder to do a targeted attack to figure out what is the exact node that is owned by your company or by you individual.

191
00:24:28,000 --> 00:24:37,200
So, it would most likely be an attack against the entire node network, but it's still probable.

192
00:24:39,900 --> 00:24:47,440
So, we can keep going through all of these individual components and try to go through like what are the main threats.

193
00:24:47,440 --> 00:24:57,280
Luckily, I went through this exercise over the last few months and I built a no threat model which applies to generic nodes out there.

194
00:24:57,280 --> 00:25:08,540
So, if you take a Bitcoin node versus, let's say, Ethereum nodes, they will have all these core issues, all these core threats, plus whatever makes them unique from each other.

195
00:25:08,540 --> 00:25:13,000
So, if you're like I mentioned that, for example, OpenEthereum has an extra web server in there.

196
00:25:13,000 --> 00:25:19,320
So, you'll need to slap in a web server into this model and consider like what are the threats there as well.

197
00:25:19,740 --> 00:25:25,860
Some more terminology so we can interpret the grid that I'm going to show you in a bit.

198
00:25:26,080 --> 00:25:30,300
So, stride is a common acronym used for threat modeling.

199
00:25:30,300 --> 00:25:39,320
It stands for simply just a variety of bad things you can do, which is spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege.

200
00:25:39,320 --> 00:25:43,780
It's just a guide to help you think better about like what is the impact there.

201
00:25:44,440 --> 00:25:46,740
It is also important to consider the likelihood.

202
00:25:47,220 --> 00:25:50,440
How likely is that the threat event is going to occur?

203
00:25:51,140 --> 00:25:59,220
How likely is that the threat actor not only has an opportunity but also the expertise to exploit something, the weakness.

204
00:26:00,220 --> 00:26:05,440
This will become useful later on as we talk about mitigations and how to prioritize them.

205
00:26:06,980 --> 00:26:14,420
So, overall in the threat model, we already went through the 11 components I showed you in a few slides ago.

206
00:26:14,420 --> 00:26:22,060
The total number of threats that I identified were 14 and the total number of critical threats for a generic node is 5.

207
00:26:22,060 --> 00:26:33,740
So, again, depending on how your ecosystem looks like, you know, if you're running a larger set of nodes and you have doctorized images and all that, your threat model will continue growing.

208
00:26:33,740 --> 00:26:37,820
This is just to get you started somewhere and provide you foundation.

209
00:26:38,980 --> 00:26:43,280
This is just a small glimpse into what the threat model looks like.

210
00:26:43,480 --> 00:27:00,700
We have the threat name, so node software generic vulnerabilities, the stride impact, so elevation of privilege, denial of service, severity, probability, description, most importantly, a variety of different mitigations that you can implement to address this threat.

211
00:27:04,720 --> 00:27:13,980
So, a key thing that we need to do when considering mitigations and the risk is to consider the cost.

212
00:27:14,740 --> 00:27:18,080
The cost, probability, and severity.

213
00:27:18,080 --> 00:27:23,720
So, the cost is how much time is it going to take you, how many resources you need to invest to address something.

214
00:27:23,720 --> 00:27:27,800
What is the severity and probability of this threat?

215
00:27:28,280 --> 00:27:45,760
So, in order to prioritize a variety of threats, I came up with this five-level, I guess, criticality grid where most critical flaws are the ones that are both have high impact, but also high probability of happening.

216
00:27:46,720 --> 00:27:55,420
Severe threats are the ones that have high impact, but maybe slightly lower probability of occurring.

217
00:27:55,420 --> 00:28:07,820
So, think of your generic web server getting exploited, which is running on your node versus some esoteric block parsing vulnerability that needs to be discovered to be exploited.

218
00:28:07,820 --> 00:28:16,180
And all the way to low, where these are the threats which we can address at one point, but only after we're done with the top two or top three, at least.

219
00:28:18,300 --> 00:28:26,580
So, of all these mitigations, in the model, I created about 41 mitigations that one could potentially implement.

220
00:28:26,580 --> 00:28:32,300
It could be probably more that we can think of, but I'll share the top 10 ones in a bit.

221
00:28:32,600 --> 00:28:36,560
Of those mitigations, they're addressing 22 critical threats.

222
00:28:37,040 --> 00:28:42,220
What's interesting is that half of those mitigations for critical threats are, in fact, very low cost.

223
00:28:42,220 --> 00:28:48,980
This is something that you can implement in a day and will significantly bolster the security stance of your nodes.

224
00:28:49,440 --> 00:28:59,560
So, yeah, you can start building fuzzers, you can start looking for every line of code of your nodes to make them secure, and they will make them very secure.

225
00:28:59,660 --> 00:29:15,940
But you can also do a bunch of things which are very low cost that will not make your nodes completely secure, but they will also, by the order of magnitude, make them resilient to some of the attacks that I described.

226
00:29:17,300 --> 00:29:30,700
So, this is just a different view that in the top bar of the critical vulnerabilities, the ones in light blue, you can see that half of them are basically low-cost solutions that you can implement.

227
00:29:30,760 --> 00:29:44,040
And then you have just, you know, if you really want to push it through this, like, it's 99.99% secure, you can continue on that trail, but that will be a little bit more costly.

228
00:29:46,420 --> 00:29:54,860
So, with that, let's talk about into node security, top 10 defenses that you can implement.

229
00:30:00,840 --> 00:30:12,860
So, the first one, and the most critical one, I think, is secure handling of where is the node software coming from, the repositories, the binary distributions, and so on.

230
00:30:13,340 --> 00:30:20,120
One key thing that we do at Coinbase, and something that you could consider as well, is repository pinning.

231
00:30:20,740 --> 00:30:37,740
Repositories can get compromised, but if you make sure that you pin to a specific non-Git version that you will work with and you will stick to until you go through some kind of process that, okay, I'm going to switch to the latest release instead of just downloading whatever is the latest one every time you deploy a node,

232
00:30:37,740 --> 00:30:44,100
that's something you could consider and could partially mitigate the risk of backdoored repos.

233
00:30:44,780 --> 00:30:47,020
Another thing is verification of signatures.

234
00:30:47,020 --> 00:31:04,260
So, if you're relying on either source code that you're downloading or binaries that you're downloading, make sure you verify all the signatures that, in the case of Monero, if the binaries were signed and the attackers did not have signed signatures correctly,

235
00:31:04,260 --> 00:31:06,220
then you would be able to detect that.

236
00:31:06,240 --> 00:31:18,720
In the case of source code, if you can at least verify, like, who is committing to the repositories, is it coming from, am I accessing a website, and it's, you know, the TLS is not breaking, no one is men in the middle of me trying to download the source code.

237
00:31:18,720 --> 00:31:23,900
Again, this is something that could make your nodes more secure.

238
00:31:26,160 --> 00:31:32,380
We continue on that train of thought, we can also make sure that we build all nodes from source code.

239
00:31:32,380 --> 00:31:45,760
I know this is a big ask, but this is something that you could consider, and the reason for this is, again, the Monero example, where only the binaries of Monero were backdoored, the source code remained intact.

240
00:31:46,580 --> 00:32:03,520
And the reason why, both could have been compromised, but if you consider how binaries for nodes are built, so there's a software repository with a lot of eyes, and then there could be a flow where one single developer with malware on their machine downloads that source code,

241
00:32:03,520 --> 00:32:09,700
builds the library, uploads the binary back on the website, there are a variety of things that could break along the way.

242
00:32:09,700 --> 00:32:14,620
Same goes with the CI automation of some sort, which builds binaries automatically.

243
00:32:14,960 --> 00:32:31,240
There are a lot more components which could be attacked and compromised, which is why, in case, again, lesson learned from that one incident, is that if you just stick to the source code, it may still be backdoored, but at least you have, like, one point of failure as opposed to a multitude of others,

244
00:32:31,240 --> 00:32:33,360
which are required when building binaries.

245
00:32:35,380 --> 00:32:37,020
Secure node configuration.

246
00:32:37,100 --> 00:32:44,880
So, every single node that I looked at has a whole variety of different levers that you can pull to make them secure.

247
00:32:44,960 --> 00:32:56,160
Some things that you should, generically, you could consider is make sure that you can diversify the number of connections that your node can make, and to which nodes it can make, it can connect to.

248
00:32:56,160 --> 00:33:00,320
So, an example would be things like Stellar, I believe.

249
00:33:00,320 --> 00:33:06,580
They need to be manually configured to distribute which nodes that you connect to.

250
00:33:06,760 --> 00:33:14,140
So, make sure that if you configure nodes like that, you select as wide of a node selection as possible.

251
00:33:14,140 --> 00:33:22,280
Others, they restrict the number of connections total, so you want to see, like, what is a reasonable number of connections you want to maintain throughout the network.

252
00:33:22,280 --> 00:33:27,340
This will help you with Eclipse attacks, making more resilient against forks, and so on.

253
00:33:28,260 --> 00:33:35,680
Another important key on configuring nodes securely is locking down your RPC interfaces.

254
00:33:35,680 --> 00:33:45,980
So, I believe last year, there was mass scanning of Ethereum nodes that were looking for open RPC interfaces to basically steal private keys.

255
00:33:46,100 --> 00:33:47,880
So, lock down your RPC interface.

256
00:33:47,880 --> 00:33:51,140
There's no reason why those should be exposed to the rest of the world.

257
00:33:51,140 --> 00:34:00,560
So, make sure you configure, you know, whatever IP or interface that RPC is listening on, make sure it's configured to something local to your network or just local host.

258
00:34:02,280 --> 00:34:03,980
Restrict node access.

259
00:34:04,740 --> 00:34:08,420
So, not everyone needs constant access to your nodes.

260
00:34:08,420 --> 00:34:12,580
Not everyone needs to be able to modify arbitrary things within your node.

261
00:34:12,580 --> 00:34:23,480
So, at Coinbase, what we do is we create a consensus mechanism with no configuration changes, no nodes, no one can bring up and down nodes just arbitrarily.

262
00:34:23,480 --> 00:34:25,200
It requires a consensus of people.

263
00:34:25,200 --> 00:34:29,780
The more critical your infrastructure is, the more people it requires a sign-off.

264
00:34:29,780 --> 00:34:39,240
So, you can implement it through a sign-off, through some kind of two-factor pings that you can send to people to approve any given action.

265
00:34:41,220 --> 00:34:43,340
Define and enforce administrator roles.

266
00:34:43,500 --> 00:34:51,140
Again, this is something to restrict to security nodes in larger environments, but you can apply this to individual nodes as well.

267
00:34:51,140 --> 00:35:02,280
Make sure that there are some specific roles set up for the node that, you know, you can either configure nodes and deploy them or shut them down, but not everything at the same time.

268
00:35:02,280 --> 00:35:08,000
So, you can get very granular there, but it depends, see how appropriate it is for your specific environment.

269
00:35:09,080 --> 00:35:15,900
Finally, restrict node access, as in the network traffic node access, both ingress and egress.

270
00:35:16,380 --> 00:35:29,100
So, on one side, it makes sense that you restrict your RPC interface already, so make sure those are firewalled off, that you cannot access nodes from anywhere, and then externally, you also have some kind of restriction set up as well.

271
00:35:29,220 --> 00:35:37,340
On the other side, nodes could be an excellent points of entry for the attacker who found some kind of vulnerability into the rest of your network.

272
00:35:37,340 --> 00:35:43,100
So, make sure that your nodes cannot communicate to the rest of your network as well, because they could be used as stepping stones.

273
00:35:44,540 --> 00:35:47,000
Monitor and log node activity.

274
00:35:47,160 --> 00:35:53,100
So, this one is probably the most interesting one, and there's a lot of things you can do there.

275
00:35:53,480 --> 00:35:59,500
Anytime your node loses internet connectivity, make sure you do some basic verification.

276
00:35:59,500 --> 00:36:04,360
Am I getting... is there like something with BGP, or am I getting... is my DNS going down?

277
00:36:04,360 --> 00:36:05,760
Am I under attack?

278
00:36:06,240 --> 00:36:07,740
Am I getting eclipsed?

279
00:36:07,740 --> 00:36:09,100
So, make sure you investigate those.

280
00:36:09,100 --> 00:36:10,300
Am I on the right fork?

281
00:36:10,300 --> 00:36:12,900
Am I not seeing sufficient traffic?

282
00:36:13,000 --> 00:36:14,900
All of that applies.

283
00:36:15,040 --> 00:36:23,480
You can do node-specific verifications too, not just generic network monitoring, such as periodically check, like, am I on the right fork?

284
00:36:23,480 --> 00:36:24,920
What is the block height?

285
00:36:24,920 --> 00:36:26,680
What is the current block hash?

286
00:36:26,680 --> 00:36:33,020
Do other nodes in my system, or externally, or on blockchain explorers, do they still see the same thing?

287
00:36:33,580 --> 00:36:45,460
So, in case of the recent ETC experiment incident, you would have been able to quickly diagnose, like, oh, okay, so there's something going on with nodes because there's some kind of forking happening.

288
00:36:46,420 --> 00:36:49,740
Finally, log all access, all network events.

289
00:36:49,740 --> 00:37:05,400
It may not be as essential during the normal operation of node, but when something goes wrong and you need to investigate and figure out what happened, who accessed it, what were the network events that led up to the attack, those logs will be highly instrumental for the investigation.

290
00:37:07,620 --> 00:37:08,780
Lockdown-based OS.

291
00:37:08,780 --> 00:37:14,980
So, definitely, you cannot build a secure node software if your underlying layer is insecure.

292
00:37:15,000 --> 00:37:35,100
If, again, if you want to go after the weakest link, and your node is very well configured and firewalled off, but your underlying OS is some outdated version of Linux, which has, like, exposed ports and they're easily attacked, the attacker is just going to go after that and going to skip all of the other mitigations you built.

293
00:37:35,100 --> 00:37:40,320
So, you could do things like doctorize your nodes, run SCLinux or similar toolkits.

294
00:37:40,340 --> 00:37:47,120
So, there are a variety of things you can do to lock down the OS, and there are guides for that, but something to consider as well.

295
00:37:48,820 --> 00:37:51,000
Hardened node's network connections.

296
00:37:51,300 --> 00:38:03,880
So, going off the stack a little bit, not just protecting what connections can be made, or what we're monitoring, make sure that if you configure DNS server for your node, make sure it's a secure one, uses DNSSEC.

297
00:38:03,880 --> 00:38:09,580
If you're using ISPs, make sure that whatever BGP connections that it uses are done in a secure fashion.

298
00:38:09,580 --> 00:38:17,480
If your node needs to communicate to some kind of HTTP resources, make sure you configure your CAs properly, so you're not getting met in the middle.

299
00:38:18,120 --> 00:38:20,620
Denial of service protection is critical.

300
00:38:20,780 --> 00:38:25,220
Again, it's potentially hard to determine.

301
00:38:25,380 --> 00:38:37,280
I want to attack just one particular node for one particular user or organization, but it's prudent to use something like Cloudflare to have a layer of denial of service protection, so you don't have to worry about that.

302
00:38:38,500 --> 00:38:40,880
Node-specific threats.

303
00:38:41,160 --> 00:38:42,980
Node and protocol-specific threats.

304
00:38:43,600 --> 00:38:44,860
This is a catch-all.

305
00:38:44,860 --> 00:38:50,120
Basically, be mindful that all nodes are unique in their own different ways.

306
00:38:50,120 --> 00:38:54,520
So, I mentioned before that Stellar, you have to manually configure what are the trusted nodes.

307
00:38:54,520 --> 00:38:59,540
You have to manually calculate, like, hey, what is my threshold for... it's a BFT network.

308
00:38:59,540 --> 00:39:02,240
What is my threshold that it can get attacked?

309
00:39:02,240 --> 00:39:05,980
So, make sure that I'm not going to be vulnerable to that.

310
00:39:05,980 --> 00:39:14,160
Other protocols like EOS, they implement debugging modes that you have to watch out for.

311
00:39:14,160 --> 00:39:16,280
So, read the config file completely.

312
00:39:16,280 --> 00:39:26,100
Make sure you understand what are the implications of all the variety of settings, and have a good secure configuration specific for that node.

313
00:39:28,260 --> 00:39:30,580
Verify node functionality before deployment.

314
00:39:30,580 --> 00:39:36,720
So, that's a much harder one, because, sure, you pinned the repository.

315
00:39:37,040 --> 00:39:45,840
You are pretty confident, okay, so it probably was not modified, but I'm not 100% sure that whatever I downloaded is working as it's supposed to.

316
00:39:45,840 --> 00:39:47,020
There are no backdoors there.

317
00:39:47,020 --> 00:39:50,500
How do you verify that it still works as intended?

318
00:39:51,180 --> 00:40:02,040
Well, ideally, you would run a set of tests, like, hey, if I transmit this transaction, it shows up on the network, or if I see a transaction coming from the network, my account gets properly credited.

319
00:40:02,260 --> 00:40:06,640
And that's a big ask to implement for every single node.

320
00:40:06,880 --> 00:40:18,500
So, one project which aims to solve this problem is Coinbase Rosetta project, which creates a kind of a middleware for a variety of different nodes to be able to speak the same API.

321
00:40:18,500 --> 00:40:42,880
So, instead of implementing a variety of different calls to transmit a transaction, get a current block hash and transactions in it, Rosetta creates a kind of a middleware for a variety of different nodes that you can just use a single API call, and it automatically translates to whatever is specific for that node.

322
00:40:43,240 --> 00:41:01,380
Now, what's interesting about the Rosetta project also includes a Rosetta validator, which exercises the node and makes sure that it passes a variety of very specific tests from transaction verification blocks and signing and so on to make sure that it works exactly as intended.

323
00:41:01,740 --> 00:41:14,720
Not every project supports Rosetta interface, but for the ones that do, you can implement it in the pipeline to make sure that the nodes that you deploy are acting as expected.

324
00:41:17,200 --> 00:41:20,840
And you notice that we're slowly ramping up the difficulty level.

325
00:41:20,840 --> 00:41:22,460
This is probably the hardest one.

326
00:41:22,460 --> 00:41:35,060
This requires you to examine the source code, examine the alerts that you're getting, and this is the basic security static analysis of node software.

327
00:41:35,440 --> 00:41:43,100
So, you may not need to go through the actual source code, but you can observe a variety of different alerts reported by tools like Salus.

328
00:41:43,100 --> 00:42:00,400
So, this is another Coinbase tool, which we use internally, and it's an open source tool you can use as well to quickly run through a variety of node source code packages, and it gives you a nice report of what are the high critical vulnerabilities within that.

329
00:42:00,400 --> 00:42:03,020
So, you can examine them, just get a general sense.

330
00:42:03,020 --> 00:42:16,100
It's hard to say which ones are false positives without actually looking at the source code, but you can see like, hey, if I suddenly see 100 or so, or like this really sneaky report, maybe I should investigate more or report or ask the original developers.

331
00:42:17,520 --> 00:42:27,080
In the Salus project, it supports a variety of different languages from Go to Ruby and so on.

332
00:42:27,740 --> 00:42:35,900
GoSec, like for if you're just running Ethereum or other Golang-based nodes, so GoSec is excellent.

333
00:42:35,900 --> 00:42:41,920
In fact, Salus is running a variety of open source projects, and it just creates a wrapper for them.

334
00:42:42,080 --> 00:42:43,480
So, GoSec is one of them.

335
00:42:43,880 --> 00:42:50,760
For all the other projects, you have something like Semgrep, so if you have SQL to analyze and so on, so you can use that as well.

336
00:42:52,580 --> 00:43:03,480
So, with that, just to summarize, what is our total view of all the different mitigations, top 10 things that you can do to secure your nodes.

337
00:43:03,600 --> 00:43:07,140
So, make sure you handle software repositories securely.

338
00:43:07,140 --> 00:43:17,160
Make sure that you pin them, that you understand that they can be compromised, and you need to just investigate a little bit more.

339
00:43:17,640 --> 00:43:19,480
Build all nodes from source code.

340
00:43:19,480 --> 00:43:25,480
So, again, it's just reducing your attack vectors just a little bit more.

341
00:43:25,480 --> 00:43:27,060
You just build everything from source.

342
00:43:27,060 --> 00:43:28,480
There are a lot of eyes in source code.

343
00:43:28,480 --> 00:43:30,360
Who knows what happens with those binaries?

344
00:43:31,080 --> 00:43:32,680
Securely configure nodes.

345
00:43:32,700 --> 00:43:37,360
So, make sure that you diversify what connections your nodes have.

346
00:43:37,360 --> 00:43:45,340
Make sure that they can't be just as easily eclipsed and people can access it and reconfigure them.

347
00:43:45,800 --> 00:43:47,280
Restrict node access.

348
00:43:47,660 --> 00:43:49,120
Monitor and log node activity.

349
00:43:49,120 --> 00:43:54,600
So, you can get, again, very creative with those, what you can do to judge what nodes help.

350
00:43:55,020 --> 00:43:56,940
Lock down your base OS.

351
00:43:56,940 --> 00:44:01,800
Make sure that you're not running, you're not building a castle on quicksand.

352
00:44:04,640 --> 00:44:07,200
Definitely consider node and protocol specific threats.

353
00:44:07,200 --> 00:44:09,980
All nodes are unique in a variety of different ways.

354
00:44:10,160 --> 00:44:12,000
So, you need to consider those as well.

355
00:44:12,000 --> 00:44:13,460
Verify node functionality.

356
00:44:13,460 --> 00:44:17,760
So , if your node supports Rosetta project, run Rosetta validator.

357
00:44:17,760 --> 00:44:26,200
It will give you this good feeling that, okay, at least for a variety of tests, this still works as expected.

358
00:44:26,460 --> 00:44:40,040
And then, if you have time and you have technical expertise, try to run basic static analysis tools just to periodically see, like, are there, like, major low-hanging fruit vulnerabilities that are getting introduced that I can catch, maybe report to original developers.

359
00:44:41,800 --> 00:44:44,980
So, with that, thank you so much for your time.

360
00:44:45,920 --> 00:44:47,580
I may have not mentioned this.

361
00:44:47,580 --> 00:44:50,480
I'm the blockchain threat intelligence newsletter.

362
00:44:50,480 --> 00:44:55,020
I list a lot of node incidents and vulnerabilities that are getting discovered.

363
00:44:55,080 --> 00:44:56,460
So, subscribe.

364
00:44:56,460 --> 00:45:01,520
See if you can catch the latest lessons learned from those vulnerabilities.

365
00:45:01,740 --> 00:45:04,100
And feel free to follow me on Twitter.

366
00:45:04,780 --> 00:45:09,200
And or feel free to contact me offline if you have any more questions.

367
00:45:09,200 --> 00:45:10,300
Thank you.