start	end	text
5020	8720	Hello and welcome to Attacking and Defending Blockchain Nodes,
8720	13820	where we're going to learn about a variety of risks and mitigations that we can implement
13820	18560	to defend a very core component of the blockchain systems.
20000	25040	We'll try to do a live discussion during the talk while we ask a question, so feel free to join
25900	33080	DEF CON's Discord on BCV General Tech's channel, and I'll try to ask questions and respond live.
34380	39540	In case we haven't met, my name is Peter Kaczynski. I'm a blockchain security engineer at
39540	45380	Coinbase, where I spend most of my time breaking a variety of different blockchain systems and
45380	51420	smart contracts. I'm also a writer for the Blockchain Threat Intelligence Newsletter,
51420	59060	where I try to cover all the latest news, events, hacks, scams in the blockchain security ecosystem.
59840	64700	Last year, you may have also participated in the Capture the Coin competition,
64700	71500	which I helped organize. My background before deep diving into the cryptocurrency world
71500	76180	was as a malware reverse engineer at FireEye, where I looked at a whole lot of APT malware
76180	82040	and a penetration tester for the Federal Reserve System, where I was breaking Finance 1.0.
84120	90300	So, just some idea, give you an idea of what we're going to talk about. I'm not going to drop
90300	96520	any old days today. What my goal here is to educate, to share knowledge, to also learn from you.
97180	102420	With the goal of just making, discussing, like, how can we secure node infrastructure?
103080	108700	So we'll start with a simple exercise on how do we know, what is the state of truth that tells us
108700	115300	how many coins we actually have? We'll go through a variety of examples of how nodes break, and
115300	121280	we'll discuss a few of the ways that they could break in the future. Finally, I'll introduce a
121280	129260	solid node security threat model for generic nodes, which you can apply later on for a specific node
129260	135300	implementation. And most importantly, discuss a top 10 list of security defenses that you can
135300	139960	implement today, whether you're a standalone operator, or you're working for an exchange,
139960	148280	and share lessons learned from trying to secure nodes of my own. So with that, let's talk about
148280	154340	how many coins you have. So what is, if you're running, if you wanted to, at any given moment,
154340	160000	tell me how many coins of any given asset that you have, how do you go about that?
160660	165300	So, well, you'll probably have a piece of software on your computer, on your phone,
165300	171840	or a hardware wallet, maybe, which has some specific logic dedicated just for a specific
171840	176640	asset. So if you have a Bitcoin wallet, or Ethereum, or Monero, or some other coin,
176640	183060	sorry if I didn't list here. This particular wallet communicates with a wider network,
183060	189020	somewhere to the cloud, and asks a question, hey, how much coins do I have? What do I actually own?
189460	194000	In case of Bitcoins, you try to enumerate how many outputs that you have unspent.
194440	199000	Ethereum is different in the sense that it's an account-based system, so you have to look up
199000	206880	how much of the asset that I actually have on the network. The thing is, it gets a little bit
206880	212060	more complicated. So if we start getting a little bit more granular, we're going to see that,
212060	217360	in fact, we don't communicate with just some unknown entity, abstract entity. We actually
217360	223140	communicate to specific nodes for that specific coin. So Bitcoin wallet communicates to Bitcoin
223140	230740	nodes, Ethereum to Ethereum, Monero to Monero, and so on. This introduces some complications,
230740	237800	because we realize that there are a lot of third-party components involved. So if we just
237800	243880	want to manage our assets, we have to communicate to a whole bunch of software which is maintained
243880	251480	by third parties. So just to give you like a few more very simplistic threat model here,
251480	256900	let's talk about components of a friend's coin, just the imaginary coin from the Mario friend's
256900	264900	land. So you have your wallet communicating to a node, communicating to the overall gossip network
264900	273000	of some sort, trying to propagate blocks. You also have your developers who are contributing to the
273000	280140	GitHub repository, so you have to rely on maintaining the repository, compiling the node maybe, or
280140	287640	downloading binaries there, and so on. And this all works really well, and until we start thinking
287640	294420	that, well, how can we be sure that all the software libraries, all the standalone software
294420	300940	infrastructure does not act maliciously, does not drop, modify, or craft transactions to steal your
300940	307120	funds. So let's see how this could happen. So we have this Mr. Robot character trying to attack
308060	313380	a variety of different points of this threat model. So we'll start left to right,
313380	319040	so we can see that we can attack the network connection between your node and the
319040	325160	cloud. So someone could start dropping blocks, try to eclipse attack your node, so basically to
325160	331680	isolate it to make sure it's not communicating to the rest of the network properly. The same character
331680	336620	can also attack your node directly if there are some unknown weaknesses that they can exploit to
336620	341360	do some kind of code execution or crash it. They can prevent you from accessing the
341360	348540	network. At last, we also have to think about developers. The bad actors out there, they can
348540	356300	go after developers like the toad over there. They can make them commit bad code. And I'll
356300	362200	cover a few real-world examples where this does happen. So the idea is that
363980	369580	it's an invitation to start thinking about the node security as a holistic
369580	373100	ecosystem. So there are a variety of different components which are involved.
375040	380320	So the examples that I just mentioned, they're not theoretical, they're practical, they're based
380320	386900	on real-world incidents. So for example, in May 2014, someone attacked the BGP network
386900	394220	used to eclipse attack Doge and Bitcoin miner pools. So this was, in case you're not
394220	402680	familiar with BGP hijacks, it's a way to effectively isolate whole large swaths of network blocks
402680	407920	in order to make them route traffic to unexpected directions. This is something that happened
408400	413060	a decade ago, I believe with YouTube, where all the traffic was redirected to
413920	417600	to Pakistan, I believe. And it keeps happening again and again.
419180	424220	This is a threat that we have not observed within cryptocurrency world for a while, but
424220	428700	it's something that we have to be aware of and build threat models around.
429400	435100	On the EOS side, after the release of the Node software, they published a bug bounty.
435300	439860	And oh boy, did they receive a lot of good finds there, because one, there was immediately a code
439860	445560	execution flaw that was discovered. And for a while at least, they kept on getting one
445560	452560	after another of high-value bounties submitted to them, which included denial of service, code
452560	460640	execution, and similar. Finally, in 2019, Monero website was compromised. The binaries for the
460640	466460	wallet were backdoored to steal clients' funds. So this is an example of how we have to look out for
467140	471040	the developer. Where are you getting that software? How trustworthy is that resource?
471040	477920	What can you do to verify it? So this was just over the few years, but I just want to instill
477920	482600	in you that this is an ongoing problem. These are all the different things happening this year.
482820	488340	So in March, we had on the testnet, but we still had a flaw discovered in the way Solana was
488340	493900	processing transactions, where it was not doing sufficient validation. So it resulted in 500
493900	500560	million Solana getting stolen. Filecoin, again, had a nasty inflation bug, fortunately discovered
500560	506280	on testnet, exploited also on testnet, but it resulted in 9 billion Filecoins minted,
506280	510820	when only 2 billion could have existed in the first place. Inflation bugs are particularly nasty
510820	516160	because these are the attacks not against individual users, but against the entire coin
516160	523280	ecosystem. Tendermint is a library behind a variety of projects. It's a consensus mechanism used in
523280	528760	projects like Cosmos, Oasis, and others. A denial of service vulnerability was discovered there
528760	536260	when parsing specially crafted invalid blocks, which basically crashed the nodes. So if this
536260	541120	was successfully executed in the wild, Cosmos was likely not vulnerable to this because they
541120	546260	didn't upgrade just to that particular vulnerable version. But if that was the case and the
546260	550920	vulnerability was maliciously discovered, then it would result in a network halt.
551520	555300	The last example, and probably the most critical and interesting one,
555880	562580	both from the impact and the way that it was handled, is Ravencoin. Ravencoin had an inflation
562580	570420	bug discovered just last month in July. The inflation bug vulnerabilities basically allow
570420	575920	someone to print money out of thin air, which is extremely dangerous because it allows an attacker
575920	583640	just to profit and sell value to exchanges, and it basically impacts every single user of the coin.
583640	587840	What's interesting about this particular vulnerability was that it was actually
587840	594980	maliciously introduced. It was introduced back in January by an account which was seemingly
594980	601260	making adjustments to the way the comments were made, on the debugging views were made.
602360	607880	And the core developers unfortunately missed that until the point that someone reported them,
608320	611380	a third party, which was building a blockchain explorer for Ravencoin,
611380	614400	there's something weird going on in your network. We're observing someone
615060	621220	minting a whole bunch of RVN and also, by the way, selling them on exchanges and profiting from it.
621600	629180	So the Ravencoin team was able to issue an emergency patch, worked with the miners,
629180	636260	but it's still 300 million RVNs were minted and exchanged for some other coins.
636780	640200	Something that we have to learn from.
642140	649060	So before we go into the section on how to secure the nodes, how to look out for those
649060	653660	vulnerabilities, let's first think about all the different ways that we can break the nodes.
654460	660060	So just to define the scope right away, we're not going to talk about the features of a node,
660060	665060	which are documented and designs of the node. So if something implements proof of work,
665060	672020	we're not going to deep dive into 51% attacks, reorgs, other protocol design issues. We'll accept
672020	677360	them as is. What we're more concerned about is if the node promises to do something correctly,
677360	681560	and it fails to do that, that's what we need to pay attention to.
682340	690640	Key management is also outside of this. So you should probably be running nodes on a variety
690640	694620	of tiers. So if you just want something which replicates the state of the network, you should
694620	701440	probably keep your keys on a separate system. Incorrect usage. So I assume that if you're
701440	705240	running a specific node that you read the documentation, you read everything that is
705240	713560	known about it, so you don't necessarily make mistakes. Like a good example of that is Ripple's
713560	720380	TF partial payment flag, where if you do not implement that particular flag correctly, then
720380	726080	whenever someone sends you some payment with that flag set, it may appear that you actually
726080	732840	receive those funds. But in reality, it's a IOU, essentially, that I will send you those things
732840	737420	eventually, but not right now. So a lot of exchanges were exploited as a result of that.
737840	742740	Finally, we're not going to focus on the layer two vulnerabilities such as smart contracts,
742740	750180	DeFi, and so on. So what's in scope? What's in scope are the implementation flaws. So protocol,
750180	757740	software flaws, attack resilience, can the node stand up to sustained attacks? Infrastructure. So
757740	762140	we're not looking at nodes as just this one thing, and we're protecting just the node itself,
762140	766640	and don't care about anything that it's built in software. We're worrying about underlying OS,
766640	770580	we're worrying about the network stack, and anything that goes around it.
771080	776900	Finally, management. Human factor is, always was, always will be the weakest link in the security
776900	782260	chain. So we will discuss access management, we'll talk about configuration, source control,
782260	791240	and all other ops type of things. So with that, just before we dive into the exercise, just wanted
791240	796660	to establish some terminology so we're all on the same page. The threat model, and this is taken
796660	801840	from OWASP's excellent threat modeling cheat sheet if you want to review it and build your own threat
801840	810100	models. Threat model itself is a process to identify potential threats. So we want to enumerate
810100	814360	all the different ways that things can go wrong, we can assess those threats, what is the risk
814360	818960	coming from them? Are they basically going to result in me losing money, or is it just
819340	827500	a denial of service, or some other maybe less critical impact? And also later on, we'll use
827500	834320	that to prioritize mitigations. Why would we spend hours or weeks trying to address a risk
834320	840340	which is not really that impactful? An important thing that we're going to do next, we'll define
840340	846020	the attack surface. An attack surface, think of it as all the different ways that someone can access,
846020	852640	can attack your node or node infrastructure, and the threat agents, which are the entities
852640	860040	which have enough skill, opportunity, and interest to attack your nodes.
860960	867960	So let's begin with just a really basic system, it's not interacting with any other nodes, it's
867960	874580	not managed by anything, just like a really basic core software. So we have the node itself, which
874580	883720	all it does is it's processing the packets, it's processing the transactions, blocks, does
883720	889080	verification, it stores the blocks in some kind of storage system, it parses configuration files
889080	895360	so it knows how to properly look at those things, and it may also include a VM interpreter process
895360	901020	where it can interpret smart contracts or anything like full-blown, Turing-complete,
901020	905600	Ethereum-style smart contracts, or something a little bit more locked down like Bitcoin script.
907080	913460	Now, let's add some more components to that. In order for node to be useful, it needs to
913460	918060	have some kind of management interface, and most nodes implement some kind of RPCE
918620	924100	network client, so we have to think about the threats coming from that, who can access this
924100	929520	thing, what is it they can do, and of course, node administrators themselves who are managing
929520	933840	the machine, they can shut them down, they can bring it back online, they can reconfigure them
933840	939900	and maybe introduce some flaws that I will cover in a moment. And also, these are just humans,
939900	947260	can they be attacked on their own with malware or maybe any other social engineering? Finally,
947260	951040	nodes need to communicate to other nodes to figure out what is the state of the network,
951040	955340	and this is where a whole bunch of different things are coming into play, like network
955340	960760	infrastructure, other nodes that we need to talk to, and miners. So network infrastructure
960760	966820	includes basically underlying layers of communication for the nodes, so think of BGP
968220	975120	and DNS client, DNS services, and just basic routing like TCP connections, UDP,
975120	981900	all of that is in scope and needs to be secured as well. Finally, nodes are just not appearing
981900	986280	on your machines out of nowhere, they must come from somewhere. Unless you are the developer,
986280	991460	you probably are relying on third-party repositories, which also in turn rely on
991460	999660	third-party dependencies. So this is something that we need to be aware of, because as I mentioned
999660	1006440	with Ravencoin, someone was able to backdoor the assets through a very sneaky code change
1006440	1011760	and something that we also need to consider within our threat model.
1013540	1018540	So with that in mind, join the BCV General Text on the DEF CON Discord channel
1019600	1025180	and I'll ask you questions to just brainstorm together on how we can break nodes.
1026760	1031120	So the first thing we're going to attack is the core. This is the node software
1031120	1041000	itself. This is what processes the blocks, the transactions, and needs to verify them correctly.
1041420	1046840	If you are this sneaky Mr. Robot character and you wanted to attack this,
1046840	1051320	how would you do it? So I'll look in the channel for your responses.
1059860	1066900	So I have one comment of software supply chain attack. So this is something that we're going to
1066900	1072300	cover in the software repository attack, but I'm talking about the core itself. That's the
1072300	1082060	piece of software that does the actual blockchain logic. BGP attacks to create an eclipse.
1083160	1090300	So again, so this is something that we have to worry about on the network connectivity side.
1096190	1102740	Let's see...
1103060	1110280	Yeah, so a denial of service against the nodes to crash with the bad blocks, malformed block headers.
1110300	1118080	So that's exactly that. So we can craft a packet, think of a tendermint flaw,
1118080	1126540	which basically crashes a node or maybe executes some kind of really nasty code execution attack.
1127080	1132700	So just to give you some background, this is how I created the model. So there's a title,
1132700	1138840	protocol vulnerabilities, severity. So if you can crash or do code execution, know the severity is
1138840	1145380	high. Probability, I would say medium. It's just harder to find, harder to exploit. It requires
1145380	1151860	dedicated time to discover those things. But the impact is that anything that is not properly
1151860	1157380	processing blocks, transactions, what is the correct fork, messing with the governance and so
1157380	1161860	on, inflation vulnerabilities, this is all in scope for this. And it will have a severe impact
1162600	1170560	on the node. However, I want you to consider it's less sexy of an attack. So we're not trying to
1170560	1175700	find some kind of ode within blocks and transactions. And consider that nodes actually
1175700	1182260	run all sorts of third-party software. So for example, Parity Nodes, they run or OpenTheorem
1182260	1190640	now, they run web UIs to manage the thing. A bunch of node software relies on third-party
1191220	1197720	database libraries and standalone databases to store blocks. So all of that are fair game to
1197720	1204440	attack as well. So if you are Mr. Robot and you wanted to attack your node infrastructure,
1204440	1210740	I'm not going to spend time trying to find old days for the way that you parse packets. It's a
1210740	1216260	lot easier for me to find out what is the weakest link, what is the common off-the-shelf software
1216260	1221300	that you're running, such as if you're running Nginx and for some reason it's a vulnerable
1221300	1226500	version, I'll try to attack that as well. I'll try to attack that instead because it probably has
1226500	1232200	more eyes on it and more vulnerabilities discovered so far. So consider those as your first line
1232200	1237420	of things to lock down and the probability is definitely higher than finding a very
1237420	1243460	directed attack to like in the way you parse blocks. So let's continue in the same
1244840	1249560	style. So software repository attacks. So someone in the channel already mentioned that,
1249560	1255480	you know, the supply chain attack. So whether you're attacking the software repository itself,
1255480	1261140	so you can, you know, like a Ravencoin example or Monero example, if you can compromise the
1261140	1269660	software repository itself, then you can introduce arbitrary changes, which are maybe
1269660	1274180	different degrees of how hidden they are, depending how many eyes are looking at the repository.
1275460	1282760	And again, this can result in fund theft or money printing type of vulnerabilities. However,
1282760	1288980	what I want to invite you to also consider is not just the main software repository, but also
1288980	1294560	software dependencies. So an example, not as actually a node vulnerability, but I believe it
1294560	1303100	was Trinity Wallet, which had a, it was backdoored on the source code level. But the way that it was
1303100	1307500	backdoored was not through the main repository, but through one of the dependencies. So someone
1307500	1313040	figured out what is the weakest link in the supply chain to that particular piece of software,
1313040	1319200	backdoored that, and then they were able to introduce code that was stealing keys.
1319780	1324800	So again, probability is high on that as well, and severity is also extremely high.
1327240	1330640	I mentioned this before, you know, human factor is always the weakest
1330640	1334520	one. So node administrator, how can we attack that?
1337760	1348720	Yep, phishing, send phishing email with some kind of malware. You know, the impact is very high.
1348860	1356440	They can reconfigure the node, they can introduce a set of nodes, which they essentially creates
1356440	1360460	an eclipse attack where you only communicate to something that is controlled by the attacker.
1360680	1365300	Certain blockchains, they have a switch you can flip, which basically allows it to accept all
1365820	1371080	blocks and transactions without doing any verifications, like a debugging mode almost.
1371120	1374840	There's a variety of things that administrator can do and the variety of ways that administrator
1374840	1378760	can get attacked through phishing, malware, social engineering, you name it.
1380440	1385680	The last, you get the gist of how we're going through different components. The last one I
1385680	1392340	want to cover is the network attack. So, correct, so BGP attacks to create an eclipse.
1393400	1403020	So, exploiting the network infrastructure, you know, all you know, a lot of nodes,
1403020	1406800	the way that they're built, is that they need to first boot up. When they first come online,
1406800	1412520	they need to communicate to something hard-coded to figure out what is the network truth, like
1412520	1419460	what are the basic trusted infrastructure that I can use to start getting an idea what is the
1419460	1428720	current block state. A lot of them use protocols like DNS in order to find out what is the first
1428720	1435400	set of nodes that I need to connect to. And if your underlying DNS infrastructure is compromised,
1435400	1441860	they can, again, try to do some kind of eclipse against you. Same with denial of service. If
1441860	1445460	your nodes can never communicate over those protocols, then they will never be able to
1445460	1451100	connect to the network in the first place. BGP routing attacks, those are extremely damaging
1452980	1457920	and something that needs to be considered. I'm putting the severity as high on these,
1457920	1462400	but probability lower, just because it's a lot harder to do a targeted attack to figure out what
1462400	1469320	is the exact node that is owned by your company or by you individual. So, it would most likely be
1469320	1477200	an attack against the entire node network, but it's still probable.
1479900	1485540	So, we can keep going through all of these individual components and try to go through
1485540	1490160	like what are the main threats. Luckily, I went through this exercise over the last few months
1490160	1498460	and I built a no threat model which applies to generic nodes out there. So, if you take a
1498460	1504580	Bitcoin node versus, let's say, Ethereum nodes, they will have all these core issues, all these
1504580	1509780	core threats, plus whatever makes them unique from each other. So, if you're like I mentioned
1509780	1514540	that, for example, OpenEthereum has an extra web server in there. So, you'll need to slap in a web
1514540	1520660	server into this model and consider like what are the threats there as well. Some more terminology
1520660	1528060	so we can interpret the grid that I'm going to show you in a bit. So, stride is a common
1528060	1533240	acronym used for threat modeling. It stands for simply just a variety of bad things you can do,
1533240	1538200	which is spoofing, tampering, repudiation, information disclosure, denial of service,
1538200	1543060	elevation of privilege. It's just a guide to help you think better about like what is the impact
1543060	1549060	there. It is also important to consider the likelihood. How likely is that the threat event
1549060	1555060	is going to occur? How likely is that the threat actor not only has an opportunity but also the
1555060	1563080	expertise to exploit something, the weakness. This will become useful later on as we talk about
1563080	1570640	mitigations and how to prioritize them. So, overall in the threat model, we already went through the
1570640	1576020	11 components I showed you in a few slides ago. The total number of threats that I identified
1576020	1583340	were 14 and the total number of critical threats for a generic node is 5. So, again, depending on
1583340	1589600	how your ecosystem looks like, you know, if you're running a larger set of nodes and you have
1589600	1594960	doctorized images and all that, your threat model will continue growing. This is just to get you
1594960	1602420	started somewhere and provide you foundation. This is just a small glimpse into what the threat
1602420	1609140	model looks like. We have the threat name, so node software generic vulnerabilities, the stride
1609140	1615240	impact, so elevation of privilege, denial of service, severity, probability, description,
1615240	1620700	most importantly, a variety of different mitigations that you can implement to address this threat.
1624720	1632760	So, a key thing that we need to do when considering mitigations and the risk is to
1632760	1639580	consider the cost. The cost, probability, and severity. So, the cost is how much time is it
1639580	1644720	going to take you, how many resources you need to invest to address something. What is the severity
1644720	1652660	and probability of this threat? So, in order to prioritize a variety of threats, I came up with
1652660	1661460	this five-level, I guess, criticality grid where most critical flaws are the ones that are both
1661460	1670480	have high impact, but also high probability of happening. Severe threats are the ones that have
1670480	1677900	high impact, but maybe slightly lower probability of occurring. So, think of your generic web
1677900	1683800	server getting exploited, which is running on your node versus some esoteric block
1684820	1689180	parsing vulnerability that needs to be discovered to be exploited. And all the way to low, where
1689180	1693880	these are the threats which we can address at one point, but only after we're done with
1693880	1701860	the top two or top three, at least. So, of all these mitigations,
1701860	1706800	in the model, I created about 41 mitigations that one could potentially implement. It could
1706800	1713240	be probably more that we can think of, but I'll share the top 10 ones in a bit. Of those
1713240	1718920	mitigations, they're addressing 22 critical threats. What's interesting is that half of those
1718920	1723100	mitigations for critical threats are, in fact, very low cost. This is something that you can
1723100	1728980	implement in a day and will significantly bolster the security stance of your nodes.
1729440	1735780	So, yeah, you can start building fuzzers, you can start looking for every line of code of your nodes
1735780	1742880	to make them secure, and they will make them very secure. But you can also do a bunch of things which
1742880	1749120	are very low cost that will not make your nodes completely secure,
1749120	1755060	but they will also, by the order of magnitude, make them resilient to some of the attacks that I
1755060	1762380	described. So, this is just a different view that in the top bar of the critical vulnerabilities,
1762380	1769500	the ones in light blue, you can see that half of them are basically low-cost solutions that
1769500	1774560	you can implement. And then you have just, you know, if you really want to push it through this,
1774560	1784040	like, it's 99.99% secure, you can continue on that trail, but that will be a little bit more costly.
1786420	1794860	So, with that, let's talk about into node security, top 10 defenses that you can implement.
1800840	1807660	So, the first one, and the most critical one, I think, is secure handling of where is the
1807660	1814280	node software coming from, the repositories, the binary distributions, and so on. One key thing
1814280	1820120	that we do at Coinbase, and something that you could consider as well, is repository pinning.
1820740	1825820	Repositories can get compromised, but if you make sure that you pin to a specific non-Git version
1826350	1831840	that you will work with and you will stick to until you go through some kind of process that,
1831840	1835720	okay, I'm going to switch to the latest release instead of just downloading whatever is the
1835720	1841360	latest one every time you deploy a node, that's something you could consider and could partially
1841360	1847360	mitigate the risk of backdoored repos. Another thing is verification of signatures. So, if you're
1847360	1852820	relying on either source code that you're downloading or binaries that you're downloading,
1852820	1856240	make sure you verify all the signatures that, in the case of Monero,
1857820	1864260	if the binaries were signed and the attackers did not have signed signatures correctly,
1864260	1868860	then you would be able to detect that. In the case of source code, if you can at least verify,
1868860	1873220	like, who is committing to the repositories, is it coming from, am I accessing a website,
1873220	1877220	and it's, you know, the TLS is not breaking, no one is men in the middle of me trying to
1877220	1883900	download the source code. Again, this is something that could make your nodes more secure.
1886160	1891780	We continue on that train of thought, we can also make sure that we build all nodes from source
1891780	1897160	code. I know this is a big ask, but this is something that you could consider, and the
1897160	1903860	reason for this is, again, the Monero example, where only the binaries of Monero were backdoored,
1903860	1910220	the source code remained intact. And the reason why, both could have been compromised, but if you
1910220	1915940	consider how binaries for nodes are built, so there's a software repository with a lot of eyes,
1915940	1922300	and then there could be a flow where one single developer with malware on their machine downloads
1922300	1927880	that source code, builds the library, uploads the binary back on the website, there are a variety
1927880	1932800	of things that could break along the way. Same goes with the CI automation of some sort, which
1932800	1937360	builds binaries automatically. There are a lot more components which could be attacked and
1937360	1944320	compromised, which is why, in case, again, lesson learned from that one incident, is that if you just
1944320	1949080	stick to the source code, it may still be backdoored, but at least you have, like, one point of failure
1949080	1956080	as opposed to a multitude of others, which are required when building binaries. Secure node
1956080	1962700	configuration. So, every single node that I looked at has a whole variety of different levers that you
1962700	1968300	can pull to make them secure. Some things that you should, generically, you could consider is make
1968300	1974120	sure that you can diversify the number of connections that your node can make, and to which
1974120	1981080	nodes it can make, it can connect to. So, an example would be things like Stellar, I believe. They need
1981080	1987760	to be manually configured to distribute which nodes that you connect to. So, make sure that if
1987760	1994920	you configure nodes like that, you select as wide of a node selection as possible. Others, they
1994920	1999460	restrict the number of connections total, so you want to see, like, what is a reasonable number
1999460	2004360	of connections you want to maintain throughout the network. This will help you with Eclipse attacks,
2004360	2012240	making more resilient against forks, and so on. Another important key on configuring nodes
2012240	2018560	securely is locking down your RPC interfaces. So, I believe last year, there was mass scanning of
2018560	2026540	Ethereum nodes that were looking for open RPC interfaces to basically steal private keys. So,
2026540	2031140	lock down your RPC interface. There's no reason why those should be exposed to the rest of the world.
2031140	2036840	So, make sure you configure, you know, whatever IP or interface that RPC is listening on,
2036840	2040560	make sure it's configured to something local to your network or just local host.
2042280	2048760	Restrict node access. So, not everyone needs constant access to your nodes. Not everyone
2048760	2054200	needs to be able to modify arbitrary things within your node. So, at Coinbase, what we do
2054200	2061240	is we create a consensus mechanism with no configuration changes, no nodes, no one can bring
2061240	2066120	up and down nodes just arbitrarily. It requires a consensus of people. The more critical your
2066120	2072220	infrastructure is, the more people it requires a sign-off. So, you can implement it through a
2072220	2078340	sign-off, through some kind of two-factor pings that you can send to people to approve any given
2078340	2086440	action. Define and enforce administrator roles. Again, this is something to restrict to security
2086440	2091740	nodes in larger environments, but you can apply this to individual nodes as well. Make sure that
2092300	2099120	there are some specific roles set up for the node that, you know, you can either configure nodes and
2099120	2103720	deploy them or shut them down, but not everything at the same time. So, you can get very granular
2103720	2110120	there, but it depends, see how appropriate it is for your specific environment. Finally, restrict
2110120	2118220	node access, as in the network traffic node access, both ingress and egress. So, on one side,
2118220	2122780	it makes sense that you restrict your RPC interface already, so make sure those are firewalled off,
2122780	2127480	that you cannot access nodes from anywhere, and then externally, you also have some kind
2127480	2133380	of restriction set up as well. On the other side, nodes could be an excellent points of entry for
2133380	2137920	the attacker who found some kind of vulnerability into the rest of your network. So, make sure that
2137920	2142020	your nodes cannot communicate to the rest of your network as well, because they could be used as
2142020	2150460	stepping stones. Monitor and log node activity. So, this one is probably the most interesting one,
2150460	2156380	and there's a lot of things you can do there. Anytime your node loses internet connectivity,
2157480	2162160	make sure you do some basic verification. Am I getting... is there like something with BGP,
2162160	2167980	or am I getting... is my DNS going down? Am I under attack? Am I getting eclipsed? So,
2167980	2172900	make sure you investigate those. Am I on the right fork? Am I not seeing sufficient traffic?
2173000	2178820	All of that applies. You can do node-specific verifications too, not just generic network
2179780	2184920	monitoring, such as periodically check, like, am I on the right fork? What is the block height?
2184920	2190700	What is the current block hash? Do other nodes in my system, or externally, or on blockchain
2190700	2197440	explorers, do they still see the same thing? So, in case of the recent ETC experiment incident,
2197440	2201320	you would have been able to quickly diagnose, like, oh, okay, so there's something going
2201320	2208480	on with nodes because there's some kind of forking happening. Finally, log all access,
2208480	2214720	all network events. It may not be as essential during the normal operation of node, but when
2214920	2218900	something goes wrong and you need to investigate and figure out what happened, who accessed it,
2218900	2222520	what were the network events that led up to the attack, those logs will be
2222520	2225400	highly instrumental for the investigation.
2227620	2233520	Lockdown-based OS. So, definitely, you cannot build a secure node software if your underlying
2233520	2240340	layer is insecure. If, again, if you want to go after the weakest link, and your node is
2240340	2246640	very well configured and firewalled off, but your underlying OS is some outdated version of Linux,
2246640	2251700	which has, like, exposed ports and they're easily attacked, the attacker is just going to go
2251700	2256500	after that and going to skip all of the other mitigations you built. So, you could do things
2256500	2261700	like doctorize your nodes, run SCLinux or similar toolkits. So, there are a variety of things you
2261700	2267120	can do to lock down the OS, and there are guides for that, but something to consider as well.
2268820	2274880	Hardened node's network connections. So, going off the stack a little bit, not just protecting
2274880	2279880	what connections can be made, or what we're monitoring, make sure that if you configure DNS
2279880	2285980	server for your node, make sure it's a secure one, uses DNSSEC. If you're using ISPs, make sure that
2285980	2290360	whatever BGP connections that it uses are done in a secure fashion. If your node needs to
2290360	2295720	communicate to some kind of HTTP resources, make sure you configure your CAs properly,
2295720	2301520	so you're not getting met in the middle. Denial of service protection is critical. Again, it's
2303360	2307860	potentially hard to determine. I want to attack just one particular node for one particular
2308460	2312580	user or organization, but it's prudent to use something like Cloudflare to
2312580	2317280	have a layer of denial of service protection, so you don't have to worry about that.
2318500	2325500	Node-specific threats. Node and protocol-specific threats. This is a catch-all. Basically,
2325940	2331180	be mindful that all nodes are unique in their own different ways. So, I mentioned before that
2331180	2335740	Stellar, you have to manually configure what are the trusted nodes. You have to manually calculate,
2335740	2341000	like, hey, what is my threshold for... it's a BFT network. What is my threshold that
2341000	2345980	it can get attacked? So, make sure that I'm not going to be vulnerable to that.
2345980	2354160	Other protocols like EOS, they implement debugging modes that you have to watch out for.
2354160	2359120	So, read the config file completely. Make sure you understand what are the implications of all the
2359120	2366100	variety of settings, and have a good secure configuration specific for that node.
2368260	2372600	Verify node functionality before deployment. So, that's a much harder one,
2372600	2379860	because, sure, you pinned the repository. You are pretty confident, okay, so it probably was
2379860	2385340	not modified, but I'm not 100% sure that whatever I downloaded is working as it's
2385340	2390500	supposed to. There are no backdoors there. How do you verify that it still works as intended?
2391180	2395120	Well, ideally, you would run a set of tests, like, hey, if I transmit this transaction,
2395640	2399800	it shows up on the network, or if I see a transaction coming from the network,
2399800	2406640	my account gets properly credited. And that's a big ask to implement for every single node.
2406880	2414060	So, one project which aims to solve this problem is Coinbase Rosetta project, which creates a kind
2414060	2419020	of a middleware for a variety of different nodes to be able to speak the same API. So,
2419020	2426820	instead of implementing a variety of different calls to transmit a transaction, get a current
2426820	2433800	block hash and transactions in it, Rosetta creates a kind of a middleware for a variety
2433800	2440120	of different nodes that you can just use a single API call, and it automatically translates to
2440120	2445840	whatever is specific for that node. Now, what's interesting about the Rosetta project also
2445840	2452160	includes a Rosetta validator, which exercises the node and makes sure that it passes a variety of
2452160	2459380	very specific tests from transaction verification blocks and signing and so on to make sure that it
2459380	2467040	works exactly as intended. Not every project supports Rosetta interface, but for the ones
2467040	2473140	that do, you can implement it in the pipeline to make sure that the nodes that you deploy are acting
2473140	2481320	as expected. And you notice that we're slowly ramping up the difficulty level. This is probably
2481320	2488340	the hardest one. This requires you to examine the source code, examine the alerts that you're
2488340	2496880	getting, and this is the basic security static analysis of node software. So,
2496880	2500880	you may not need to go through the actual source code, but you can observe a variety of different
2500880	2506320	alerts reported by tools like Salus. So, this is another Coinbase tool, which we use internally,
2506320	2513400	and it's an open source tool you can use as well to quickly run through a variety of node
2513400	2519360	source code packages, and it gives you a nice report of what are the high critical vulnerabilities
2519360	2524560	within that. So, you can examine them, just get a general sense. It's hard to say which ones are
2524560	2527900	false positives without actually looking at the source code, but you can see like, hey, if I
2527900	2533860	suddenly see 100 or so, or like this really sneaky report, maybe I should investigate more or report
2533860	2541000	or ask the original developers. In the Salus project, it supports a variety of different
2542600	2551440	languages from Go to Ruby and so on. GoSec, like for if you're just running Ethereum or other
2551440	2559260	Golang-based nodes, so GoSec is excellent. In fact, Salus is running a variety of open source
2559260	2564460	projects, and it just creates a wrapper for them. So, GoSec is one of them. For all the
2564460	2568760	other projects, you have something like Semgrep, so if you have SQL to analyze and so on,
2568760	2576460	so you can use that as well. So, with that, just to summarize, what is our total view of
2577460	2584000	all the different mitigations, top 10 things that you can do to secure your nodes. So,
2584000	2588860	make sure you handle software repositories securely. Make sure that you pin them,
2588860	2596320	that you understand that they can be compromised, and you need to just investigate a little bit
2596320	2605040	more. Build all nodes from source code. So, again, it's just reducing your attack vectors just a
2605040	2608600	little bit more. You just build everything from source. There are a lot of eyes in source code.
2608600	2613940	Who knows what happens with those binaries? Securely configure nodes. So, make sure that
2613940	2620200	you diversify what connections your nodes have. Make sure that they can't be just as easily
2620200	2628600	eclipsed and people can access it and reconfigure them. Restrict node access. Monitor and log node
2628600	2633760	activity. So, you can get, again, very creative with those, what you can do to judge what nodes
2633760	2640580	help. Lock down your base OS. Make sure that you're not running, you're not building a castle
2640580	2648060	on quicksand. Definitely consider node and protocol specific threats. All nodes are unique
2648060	2653460	in a variety of different ways. So, you need to consider those as well. Verify node functionality.
2653460	2660500	So, if your node supports Rosetta project, run Rosetta validator. It will give you this good
2660500	2666920	feeling that, okay, at least for a variety of tests, this still works as expected. And then,
2666920	2671780	if you have time and you have technical expertise, try to run basic static analysis tools
2671780	2674720	just to periodically see, like, are there, like, major
2674720	2678220	low-hanging fruit vulnerabilities that are getting introduced that I can catch,
2678220	2684980	maybe report to original developers. So, with that, thank you so much for your time.
2685920	2691480	I may have not mentioned this. I'm the blockchain threat intelligence newsletter. I list a lot of
2691920	2697020	node incidents and vulnerabilities that are getting discovered. So, subscribe. See if you
2697020	2703200	can catch the latest lessons learned from those vulnerabilities. And feel free to follow me on
2703200	2710300	Twitter. And or feel free to contact me offline if you have any more questions. Thank you.