start	end	text
3900	10000	The problem that I think we're all seeing, and certainly that we've kind of been growing
10000	20080	increasingly concerned about for some time, is that it's not enough anymore that you can
21000	28520	participate in a peer-to-peer network and nobody can tell what you are doing. Increasingly
28520	34340	it's the case in this country and elsewhere that simply the act of participating in a
34340	42940	peer-to-peer network is in itself a pretty risky thing to do. As a result, we believe
42940	50440	that future anonymous peer-to-peer networks will need to limit who your peer connects
50440	57320	to to a group of trusted friends, people that you have decided you don't mind that they
57320	64500	know that you're a part of this peer-to-peer network. The big question is, is it possible?
64500	73880	And if it's possible, will it be useful? Just for those, quickly to clarify the term peer-to-peer,
73880	78340	because if you ask 10 people what peer-to-peer means, you'll get 10 different answers.
79160	86920	From our point of view, a peer-to-peer network is designed to help people find information.
87320	92720	Specifically, it's designed to help people find information that is widely distributed
92720	100620	across a large number of computers which are connected through some network, in practice
100620	106760	typically the internet. Users of this system want to find information. They need to know
108880	113220	where the information is so that they can go out and retrieve it.
113220	120700	There are different approaches to this. The first peer-to-peer network in many ways was a piece of
120700	125920	software called Napster, and its approach was pretty simple. Every peer in the network simply
125920	134380	reported to a central server or eventually one of several central servers what data was being
134380	139780	stored on that computer. And so if you wanted to find some data, you'd go to the central server
139780	143500	and say, you know, where's this thing, and it would give you a list of computers where
143500	148820	you could find it, and you would then connect to those computers directly and retrieve it.
151120	155960	Subsequent to that, when Napster and similar systems were being shut down, because of course
156140	162120	a centralized system is very easy to shut down, people started working on what we call
162120	167760	semi-centralized peer-to-peer networks, where it's essentially the same idea but instead of
167760	177620	one centrally administered index of data, you have a large number of decentralized indexes. So
177620	182640	when you connect to the network, you're allocated one or more of these indexes, and you use that
182640	189960	pretty much like a mini-Napster. Kazaz is probably the archetypal example of that approach.
191140	197320	Well, other peer-to-peer networks strive to be completely decentralized, such that in essence
197320	203060	each peer in the network does pretty much the same job and has loosely the same degree of
203060	208620	responsibility as every other peer in the network, and Freenet is one of several examples of
208620	218090	peer-to-peer networks like that. Another way that you can classify peer-to-peer networks, and it's
218090	224810	particularly useful for the purposes of our talk, is into light and dark. Examples of light
224810	230410	peer-to-peer networks, or you could also call them promiscuous peer-to-peer networks, include
230410	235670	Nutella, Freenet, distributed hash tables, indeed the vast majority of peer-to-peer networks that
235670	244310	you've probably heard of. In essence, a light peer-to-peer network is defined by the fact that
244890	250650	your computer will willingly connect to strangers, which means that strangers, people you don't know
250650	259410	or trust, can discover that you are running the software. The advantage of this is that if you do
259410	267550	it right, it can be globally scalable, which essentially means whether you have 100,000 users
267550	275250	or 100 million users, you can still find information pretty efficiently. The disadvantage is it's
275250	282830	vulnerable to harvesting. If, for example, the Chinese government wanted to get a nice, convenient
282830	289050	list of all of the people in China that were running Freenet, they could do that. It wouldn't
289050	296090	necessarily be easy, but it would certainly be possible. When we first came up with the idea,
296090	303170	we didn't really see this as... well, this wasn't really on our radar, but over the past few years
303170	308730	we've realized that it's not only in increasing threatened countries like China, but also in this
308730	316310	country. I think most people here are probably familiar with recent court decisions which have
316310	324790	further rolled back the freedom of people to share information in the ways that they might want to do.
325310	331990	The alternative to this are dark, or you can also call them friend-to-friend, peer-to-peer networks.
331990	338530	These networks are characterized by the fact that peers in this network only talk to other peers
338530	344450	directly, only establish a direct line of communication with other peers that they trust,
344450	350510	that their user trusts. So the user is saying, I don't mind that these people know that I'm part of
350510	358510	this network. An example of this is a piece of software called Waste. The advantage of this,
358510	363810	as I said, is that only your friends know that you're part of the network. The disadvantage is
363810	370830	that until now this type of network really doesn't scale. Typically you might have 10...
370830	377890	you have isolated pockets of perhaps 5 to 10 people. And while some people have found that
377890	382690	to be a useful thing, clearly there's a big difference between that and being part of a
382690	390710	global network with millions of users. Shifting gears a little bit,
391250	398610	networks like Freenet and completely decentralized networks rely on something called the small world
398610	404490	phenomenon in order to find information in this completely decentralized way.
404990	409970	Some of you, I'm sure most of you are familiar with the small world concept,
409970	414810	or you may also be familiar with the game of Kevin Bacon where you try to
414810	422690	get from Kevin Bacon to any one actor just by going through movies they've been in.
425290	434130	So a guy called Stanley Milgram in the 1960s did a very interesting sociological experiment
434650	440410	where he wrote the names of two people, of a number of people actually in Cambridge,
440410	445430	Massachusetts on some letters and gave them just to random people spread throughout the United
445430	451810	States with the instructions that they were to try to get these letters to the intended recipients
451810	457890	but only by giving them to someone they personally knew who themselves would give it to someone else
457890	464830	they personally knew. And very interestingly he found that when those letters arrived
464830	471070	they did so in perhaps they went through just five or six people. So that's five or six people
471070	480170	in a country of 270 million odd people. Clearly a very scalable way to get something from where
480170	486650	it is to where you want it to be and interestingly also completely decentralized. These people were
486650	492590	not relying on any central phone directory in order to figure out where they should route this
492590	499870	information. They were doing it purely on the basis of their friends. So what characterizes
500690	505830	a small world network? Well typically in small world networks
508030	517650	in small in small world networks what you find is is that you have short paths exist between any
517650	523070	two people. You can, if you know how, you can get from any one person to any other person
523070	529170	in a short number of steps even if there are millions of people in the network. The challenge
529170	536410	is that these short paths aren't necessarily easy to find. In the in the case of
536410	545090	in the case of Milgram's experiment people were able to use information that they had
545090	551670	about their friends in effect as signposts as to which of their friends they should send
551670	557730	the letter to next. They knew that if one of their friends perhaps lived near Massachusetts
557730	562870	that they would be a good choice to forward the letter to if having reached that person
562870	568310	that person knew somebody who was close to the street or perhaps went to the same university
568850	575430	as the intended recipient they could send it on. So in in social networks people are able to use
575430	582110	their knowledge of their friends as signposts for route it for finding these short paths.
582730	588430	Well the reason this works is that there is people have a concept of similarity
589150	595790	between other people. You know that if two guys live on the same street they're likely to be more
595790	601970	similar to each other and therefore more likely to to be connected to each other than someone living
601970	611580	out in Zimbabwe. So so the the algorithm is simple once you've got this concept of similarity
611580	617040	you simply route to the person that you know of that is closest to where you want to be
617040	624400	at each step. We call this greedy routing. Freenet and distributed hash tables rely on
624400	635520	this principle to find data in a scalable decentralized way. So so Ian just started
635520	639960	by talking about the these peer-to-peer networks and essentially the problem the dark peer-to-peer
639960	643600	networks that we want to address and then he talks about the small worlds and social
643600	649040	networks forming small worlds. So what we want to talk about here today and what we what we're
649040	655700	interested in is how can we apply this knowledge about small world and about these social networks
655700	659780	to deal with these dark peer-to-peer networks and try to make them more useful.
660280	666080	And essentially this starts with the realization that the darknet is just a social network.
666080	670580	This darknet the peer-to-peer network that's formed when people connect only to people they trust
670580	677700	will be just this big encrypted network of people's relationships just like the social
677700	683620	networks that sociologists and people work with. So the environment that we're trying to deal with
683620	689520	in our peer-to-peer networks is just this social network that Milgram's experiment took place in
689520	695140	and that we want to find a way to deal with. And we're interested of course always when we're
695140	699800	trying to search and being able to find a way from one point to another in the network. We have
699800	704960	this network that's large network connections going all over the place and we need to find a
704960	709940	way from one point to another. Well we know that in the social network and the darknet is like a
709940	715840	social network, is a social network, we people can find a way from go from one place to another.
715840	721940	Milgram's experiment showed people can root in social networks and they can do it well. He took
721940	726240	only five or six steps to go from one side of this country to another to a complete stranger
726810	732680	and people can do that. So if people can do it then well so should computers. I mean computers
732680	742240	are much smarter than people right? So to try to do this somehow, make computers do this, we need
742240	747700	some sort of mathematical understanding for what's going on in the small world network.
747740	752820	And this mathematical understanding has is something rather recent but a Cornell professor
752820	758920	named John Kleinberg provided a model about five years ago that explains how these networks can
758920	764580	look in order to be navigable. In order for it to be possible as Ian said to find signposts in the
764580	769700	network that will take you where you want to go, it needs to have a certain configuration.
771160	777520	And this configuration is that the possibility of being able to find routes efficiently from
777520	783620	one point to another rests on the fact that you have people who have similarity to one another
783620	789240	and as Ian said are more likely to be connected and then you have a certain proportion of connections
789780	796120	of different lengths with respect to some sort of position. And in the simplest model position can
796120	800760	just be where people live. And you say that people are more likely to be connected if they live close
800760	807880	to one another. And as we get further away we need this to be the probability that people know
807880	812540	each other that they are connected needs to decrease. And when this happens that's when you
812540	818880	can find an easy way to route in the network. So for our talk we're actually going to allow people's
818880	823860	positions to just be in a ring. And that doesn't mean that we think that people exist on a ring
823860	829000	but this is the very simplest model. We allow each person to just get a position on the ring
829000	835680	and then we see people knowing each other as these chords in the circle that skip over the people.
835680	840740	So here I have two examples. We've placed these positions in a ring in the mathematical idealized
840740	845080	model and we say that there are two green people who know each other and two red people who know
845080	851060	each other. But if you can see the red one covers more of the circle. It's a longer link therefore
851060	855920	it needs to be less common. And according to Kleinberg's model it needs to be these long
855920	860340	links which this one covers double as many steps along the edge here so it needs to be
860340	868620	half as common as the green one does. And if this is the case, if you have these connections,
868620	875220	if we can put people into this base model with a circle or just a... and we have connections
875220	880380	that agree with Kleinberg's model. We have some... every once in a while we run into a long connection.
880460	885540	Somebody happens to know somebody who lives way on the other side of the world but that's rather
885540	890340	rare. More commonly people know lots of people who live close to them who are close to them who
890340	896180	are similar to them. If this is the case and you have this model then we can do this greedy routing
896180	901660	that Ian describes which just says I'll always go to the... if I'm going to some destination I always
901660	907780	just step to the neighbor who's closest to my destination. And if I can do that in... if the
907780	912280	configuration of these friends is correct then the greedy routing can be mathematically proved
912280	917840	to perform in something on the order of log squared n steps. And when something is log that's
917840	922960	good. If it's not log it's bad. But we have log here so that's going to help us. That means that
922960	927920	we have an efficient way of finding steps from one way to another which is exactly what we want
927920	934980	if we want to be able to search and route in our dark peer-to-peer network. So this is an example,
935040	939440	a simulated example of how such a network would look. Now I've imagined again I've just placed
939440	944440	people's positions on a circle like this. All around there are people placed on the circle.
944440	948760	We have two red people who are circled at the end. And if you want to do a greedy route from
948760	954100	one to the other you simply follow these cords which represent the friendships and you try to
954100	958860	get from one place to another by always choosing the friend who is closest to the destination.
959100	963660	And so a route would look something like that. In the beginning, in the very first step we can
963660	968360	take a very long step because that takes us a lot closer to our destination. As we get closer
968360	973400	it becomes harder and harder to find steps that take us a long way. But eventually you do reach
973400	980660	the destination and you do so in a bounded number of steps. But this is greedy routing and this is
980660	985580	on a circle and I placed everyone on a circle in this imaginary world where it is extremely easy
985580	990500	to find to say if you're closer to somebody than somebody else because you have a circle
990500	995800	and you can say how many steps is it to him. But if we're trying to do greedy routing in a real
995800	1000560	social network, in the kind of social network that our darknet is going to form, we don't have
1000560	1005600	an easy way of saying is one person closer to another. We end up having to answer questions
1005600	1011240	like this. Is Alice closer to Harry than Bob? I'm trying to route in a social network. I'm this
1011240	1015840	person in the Milgram experiment with the letter and the letter is going to Harry.
1016880	1022480	And I have to ask myself should I send it to Alice or Bob? The letter has to go to Harry.
1022480	1027480	I have to make a decision. Should I send it to Alice or should I send it to Bob? Well, as a person
1027480	1033500	I make some sort of a judgment and in Milgram's experiment people have to make this judgment
1033500	1039040	based on some idea of who of closeness between people. The most basic form would then be where
1039040	1044520	do they live. If Alice happens to be Harry's neighbor then she is probably closer.
1044520	1049720	There's a greater chance that they know each other than that Bob would know Harry. But you could also
1049720	1054340	go on, for example, what their occupations are. People who are all hackers are more likely to have
1054340	1062220	met at Defcon than somebody who's an ice dancer or something. But you could go, you know,
1062220	1068580	jobs, interests. As a person, we do these kind of judgments all the time and it's not that
1068580	1073980	hard for us to say what would be closer to one person than another. But now what we want to do
1073980	1078340	is we have this dark net which is this social network. We want to find a way to route and we
1078340	1084460	want a computer to do it. And in practice we can't really ask the computer to make these decisions
1084460	1090740	based on what people live, what they do, etc. I mean actually this isn't as insane as it may sound
1090740	1096420	because people have actually tried this and one can try to put into the computer as much
1096420	1101460	personal information as the people as possible. Then you try to do some sort of closeness algorithm
1101460	1105860	based on people's personal information trying to match it so that you can make these routing
1105860	1112560	decisions. But the fact is that what it has shown, even if you try to use a large
1112560	1117180	number of criteria, computers are not good at making these sort of value judgments. And even
1117180	1121800	more than that what we're trying to build are anonymous file sharing networks and they will
1121800	1126020	not be a big hit if the first thing we do is ask everyone to put in all their personal information
1126020	1131580	so that we can route. It just doesn't go over well with our target audience. So we have to find a
1131580	1137500	different way to answer these things with Alice, Harry, and Bob. Who is closer to who?
1137620	1143320	And so the thing is, and this is what's extremely important, is the information is there in the
1143320	1149700	network. We can let the network itself tell us who is closer to who and we don't need to know
1149700	1155340	anything about them except the network itself. We don't need to know where they live, we don't
1155340	1161320	need to know what they do, all we need to have is this network and we work based on that.
1162460	1168200	So how would we do this? Well, we have this model of John Kleinberg that tells us what should the
1168200	1173420	world look like for it to be possible to greet a route efficiently. Well, what the world should
1173420	1178260	look like is that there should be few long connections and many short ones. Most people
1178260	1183620	should be... most friends that you have should be close to one another, whereas you should once in
1183720	1189860	a while have a long connection out to somebody who doesn't. And then, knowing this and having this
1189860	1196600	graph, all we need to do is basically give people positions in a way so that this becomes true.
1196780	1202380	We just say, forget about the world, forget about where you live, forget about what you do,
1202380	1205760	I don't care any about that, we're going to make believe, we're going to make positions for you
1205760	1212540	based on the network and make it so that the properties that we are looking for are fulfilled.
1212820	1218180	So what you can say is in a way, rather than taking the positions and trying to root on people's real
1218180	1223560	positions in this sort of social space, we reverse engineer the positions of the people
1223560	1231260	based on how the social network looks. And that is what allows us to give people new positions
1231260	1236520	that we then can use to root. And once we have these positions, we can then do the simple greedy
1236520	1242740	rooting of always trying to go to the closest point and we get what we want. And the method
1242740	1247780	to do this is actually surprisingly simple. There's some deep mathematical magic involved,
1247780	1252940	but it actually turns out to be extremely simple. When people join the network, they simply choose
1253080	1257800	a position that would be a position on this circle, and they do so in a random fashion.
1257800	1263140	Everyone just decides, okay, do a random number generator, give yourself a position.
1263640	1266980	And when you do that, of course, this is not going to be true, because two people who could be
1266980	1272420	very close in some sense in the social network, of course, could have chosen positions at opposite
1272420	1276660	ends, because it's completely random. In fact, they will choose positions at opposite ends.
1276660	1282740	But then we have an algorithm where people get together, and then they swap their positions
1282740	1290060	with one another, so as to minimize the product of the distances to their friends. And this is
1290060	1294420	extremely important, and it has to do with the math. But they get together, they contact each
1294420	1298060	other over the network, and they say, I have positioned this, you have positioned that,
1298060	1303720	maybe we'll just switch and we'll be better off. And this is an example of that sort of switch.
1303720	1308280	These two people have chosen their positions randomly, and so have their friends. And you
1308280	1312840	have a green person who's got three green friends marked by these cords in the circle,
1312840	1317760	and you have a red person who's got three red friends. But as it has been, since they chose
1317760	1322960	their positions randomly, they happen to choose them on opposite sides of the network from their
1322960	1326980	friends. And this is not what we're looking for. We're looking for something that fulfills this
1326980	1332800	model, that people who have lots of friends on one side, they should be on one side. Well,
1332800	1336480	what if red and green were to swap positions with one another? Then we get something that
1336480	1341220	looks like this. And this is much more in line with what we are looking for in our model. We
1341220	1346220	have one or two long connections, we need some long connections, but mostly we have these short
1346220	1350560	local connections, so the people are close to the people that they are connected to.
1351880	1357500	And so this is essentially the algorithm as it goes. I mean, there's some math involved,
1357500	1361760	and some formulas and stuff that I can give to the interested people. But essentially,
1361760	1366920	choose positions randomly, and then they switch when it looks like switching will make this
1366920	1373620	network more like what we're looking for. And some things to note is that this switching step
1373620	1379520	is essential. We couldn't say, for instance, let's just have everyone choose a position,
1379520	1384200	and just take a position that's close to their neighbors. Because what happens if you do that,
1384200	1388140	is everyone ends up choosing the same position in the end. Once you've done it enough times,
1388140	1390860	everyone's like, oh, I want to be closer to my neighbors, and they want to be closer,
1390860	1394380	and it just converges on the same point. You get a bunch of people who are all saying that
1394380	1397760	they're at the same point, and I promise you, you can't root based on that.
1397920	1402060	So you need this random choice, and then the switching back and forth.
1403820	1408280	And one problem with this technique, of course, is that these identities that we use to root
1408280	1412280	are going to be changing as people switch. And that's something that has to be dealt with at
1412280	1417220	the application level, because people do not have a permanent identity. But there are ways of going
1417220	1421140	around this. As long as we have one way of finding path from one path to another,
1421140	1425460	one place to another, we can deal with the fact that people's identities may change.
1426220	1432140	Okay, so now for the simulations, always a necessity to show that we've done some simulator
1432140	1438620	data, to show that this can actually work. And so what we have is three different modes here,
1438620	1443500	which are essentially two controls and one good, and one sort of where we have applied our
1443500	1448160	algorithm. The first control is the random route. If you start with a network, we have
1448160	1453540	no positions whatsoever, and we're just looking to find a way of looking for data in this network.
1453540	1457980	With no positions, no knowledge about anything, we can't do any form of greedy routing. All we
1457980	1462940	can do is stumble around randomly and hope that we find our destination. So this is sort of the
1462940	1468620	control of where you start out, the random network, the random search. That's all you can do.
1469660	1475000	And the other control is going to be the completely idealized situation, where we have created a
1475000	1480840	network that agrees with this model according to some position scheme, and we have this good
1481380	1486880	network that's known to perform in log squared time. This is where everything is good, but that's
1486880	1490960	not the situation that we actually have. The situation that we actually have is something
1490960	1495480	where we have a network just like this good one, but we don't know anything about the positions,
1495480	1500180	and we have to use our algorithm to try to restore the positions. And that's what I have
1500180	1504860	as the third data set will be the restored data. That will be our algorithm applied.
1505620	1509360	So the first thing I'm going to do is show the success rate within these log n squared
1509360	1514600	number of steps, where n is the network size. You can see the size grows logarithmically
1514600	1519320	there at the bottom, up to about 256,000, I think, for this data set.
1521220	1526780	And you can see that the random one falls off very fast. Random routing simply cannot perform
1526780	1533380	in this number of steps. It fails to find data as the network grows, whereas the good line
1533380	1538200	will stay at the top. It always succeeds. The math tells us that it must, so it would only be a bug
1538200	1545120	in my simulator if it wasn't. But this... and we can see that restoring, applying our algorithm,
1545120	1550520	doing these switches, actually brings the success rate all the way up from the red line
1550520	1555840	almost to the green line. And only at the very largest size do we see some significant deviation.
1555880	1561080	But even at that level, we're talking about 96% success rate for the restored network.
1562400	1565960	And the other thing one's interested in, of course, is the number of steps that it takes
1565960	1570620	on average. And now we can see that even with the falling success rate, the successful queries
1570620	1576460	in the random walk will increase as the size of the network grows dramatically. The good ones
1576460	1582220	will stay low, just like the math tells us, and the restored one can bring us almost all the way down
1582220	1590800	to an acceptable number of steps. But of course, that's simulated data, and we started with the
1590800	1596460	model that we're applying, and that's always a little suspect. I'm basically doing, making,
1596460	1601380	giving myself an idealized situation. So a more interesting question is, how do we do this if we
1601380	1606820	have real-world stuff? Well, so to get some real-world stuff, rather than asking for permission,
1606820	1613180	we simply borrowed some, and we went and grabbed 2,200 people from orkut.com, which is this social
1613180	1619640	networking site that Google started, I think, about a year and a half ago. Everyone ran there
1619640	1624060	because it was a fad or something, and put in all their friends, and then no one's done anything
1624060	1630360	with it. So I think, because as far as I know, it's rather useless, but it does provide a very good
1630360	1636220	data on social networks and how they look. So we grabbed some data from this, and we spidered a
1636220	1643940	set of people, starting with our friend Ian here, and going and taking people so as to create a
1643940	1652040	rather dense social network. And as it happened, this was done rather randomly, but as it happened,
1652040	1658740	I guess that's not too surprisingly, given maybe Ian's... starting with Ian, but most of the people
1658740	1665620	were programmers, techies, people in the technology community, and mostly Americans, very few non-Americans
1665620	1672240	in the set. And I think that I would imagine that, say, maybe some of the people who are in this set
1672240	1678320	are actually here today. Not just Ian, but somebody who was in it. And just as a note, we got no
1678320	1682800	Brazilians. If someone has logged in to Orkut recently, you know that it's been overrun with
1682800	1691720	Brazilians, because apparently in Brazil it's useful for something. So there are like 70 or 80%
1691720	1696320	Brazilians there, but this set, apparently that social network is rather disconnected from this
1696320	1701340	one. So our set contains programmers, Americans. There weren't a lot of these Brazilians who do
1701340	1706840	whatever they do on Orkut. And just as a note, because when people talk about small worlds,
1706840	1712040	they often talk about degree distributions and power laws. And we did have a power law degree
1712040	1717040	distribution here, in the sense that there were a couple of people. When you talk about a degree
1717040	1722340	in a network, you mean the number of people that somebody knows. So the most popular person here,
1722340	1727240	and I think the most popular person in the set was Orkut Bukhokan himself, the guy who founded
1727240	1733660	the site. He had 289 friends in the set. So out of 2000, he knew a hell of a lot of those people.
1733660	1740080	And whereas most people had only around like 18-20 neighbors there, as you can see.
1740200	1745660	So this is the sort of power law degree distribution that can affect the network.
1746300	1750380	So dealing with this data set now, we want to apply this switching algorithm,
1750380	1755060	and hope that this will do something good. Something that can beat just a random search.
1755240	1759200	And a random search performs rather well on this data. And the reason that a random search
1759200	1763960	performs well is that it will... is because of this degree distribution. Because you run into
1763960	1769340	people who know so many people, it's easy to run into the person that you're looking for.
1769340	1774500	So with power laws, it tends to perform rather well. So in this log n squared steps, and
1775440	1780560	you should all be able to do that in your head, but 2000 log n squared, that's about 120 something,
1780560	1786800	right? So in about 120 steps, the success rate was 72. And of the successful ones, it was about
1786800	1791820	43. So for randomly searching a network of 2000, this is rather good.
1791900	1795720	So the question of course is, if we run the switching, try to apply this. This is a real
1795720	1802060	social network. Can we just get somewhere with this greedy routing by assigning identities? And
1802060	1808580	the answer is yes. So with our algorithm, we can search with a 97% success rate in this data,
1808580	1814100	and on the mean number of steps is only about 7.7. So we are rather happy with that result,
1814100	1819940	because it really shows that we can massacre the random search by performing this ordering,
1819940	1824040	and we can get something that actually seems, it actually looks like the social network
1824040	1830120	lives up to the model and what we were looking for. And one thing one can see is that this
1830120	1833800	degree distribution, the fact that there are people who knows many people, can affect the
1833800	1838520	results. So if we clip the degree distribution, meaning I don't allow anybody to have more than
1838520	1843680	40 friends, I just throw away everybody's friends after 40, we can look, was this what was actually
1843680	1848360	going on? And in the random search, you can see that yes, in this case, success rate falls
1848360	1853200	dramatically, because you don't have these people who know tons of people. But in the case of our
1853200	1857600	thing, it adds a couple of steps, but the success rate is still good, and it's still dramatically
1857600	1863520	better than the random search. So we can say that our algorithm can take advantage of this
1864620	1869340	degree distribution, but it was not use it. And it seems like it actually does work. Starting
1869340	1875580	with a social network, you can reverse engineer positions, so as to make it possible for a
1875580	1880940	computer to do what people will do when they're trying to pass a letter along, based purely on
1880940	1889560	this stuff. Are we going to go with the demo? So to demonstrate this, we were going to show
1889560	1899320	routing in this, between some people in this data set from Orkut. So one thing to start with,
1902740	1908880	is starting from Ian himself, who was of course the start of our set, and going to for instance
1908880	1913560	Sergey Brin, who I think we all know. Ian wants to know how many steps does Sergey bring, because
1913560	1919440	I want to borrow some money or something. And they will actually search the set, and you can see that
1919440	1926700	it skips from friend to friend, and finds a path from Ian to James in four steps. And what needs
1926700	1931640	to be noted about this, is that when this does the search, it does it in a decentralized fashion.
1931640	1937020	It never looks at anything at any step, except who your friends are, and it decides which of Ian's
1937020	1941760	friends was closest to Sergey Brin, which of Sean Parker's friends was closest to Sergey Brin, and
1941760	1946640	then when it gets to James Joachim, it finds that he knows Sergey Brin. So the search is completed.
1948080	1956660	Another example, we could do person in the set number 1708, which is Fyodor,
1956660	1961980	who I think is at the conference somewhere. And we could go from him, to now somebody else who's
1961980	1975020	in the set, like Lawrence Lessig. And that one also succeeds, and you can see how we are honing in
1975020	1980560	in these positions in the circle, always choosing the friend of Fyodor who was closest, and then
1980560	1986460	the closest, and it actually has managed to assign these positions in a way that it can find this
1986460	1996940	path going in. So one other interesting thing, just while we have this tool that we discovered,
1997340	2003180	is that there are certain people in this data set who aren't... well, they are real people,
2003180	2008440	but it's kind of an abuse of the system. So for example, you get celebrities or politicians,
2008440	2014760	one example being John Kerry, who someone signed up as John Kerry, and then people will connect
2014760	2021540	to that person, I guess to show that they're going to vote for him or something. But the
2021540	2031160	interesting thing about people like John Kerry in this data set, is that they don't fit into this
2032640	2038760	small world model. If you've just got random people connecting to you, then it doesn't
2038760	2046240	fulfill this criteria for a small world, whereby if you know somebody and you know somebody else,
2046240	2054420	then they're more likely to know each other. So let's say we try to route from Sergey Brin to
2056100	2065040	the fake John Kerry, the results are kind of interesting, and what you actually find is that
2065040	2074540	this will keep going for 200 steps and never actually succeed in finding John Kerry. And again,
2074540	2081840	the reason for that is very simply that this John Kerry user does not fit into the social network
2081840	2091320	model that we assume. Yeah, because he doesn't actually know the people that are connected to
2091320	2106820	him. Indeed, they don't know him either. So that's kind of the theory, and I hope we've
2106820	2114800	demonstrated through simulation, including a simulation on real world data, that this idea
2114800	2124040	can actually work. So just real quick, I should talk about some of the practical concerns. We're
2124040	2130840	actually implementing this right now. We're about three weeks into coding, and we hope to have a
2130840	2143890	prototype of it in the next couple of weeks. Hang on, this has gone mad. Okay. So theory works.
2143890	2149610	What are the concerns in practice? Well, preventing malicious behavior, obviously.
2149810	2155030	Freenet is the type of peer-to-peer software that expects to be attacked,
2155030	2163770	and therefore it's essential that one person cannot easily take down the network or otherwise
2164550	2172430	make themselves a nuisance. It's essential that it's easy to use. This is a problem that Freenet
2172430	2179610	has certainly experienced. But right back for the past two decades of security software,
2179610	2185550	including, for example, PGP, people have always found that when you create a really cool piece
2185550	2193530	of software that does something really cool in terms of anonymity, privacy, or other kind of
2193530	2199570	crypto stuff, people just don't use it, or at least it doesn't achieve wide adoption because
2199570	2207370	it's just too difficult to use. With this next iteration of Freenet, we really want to tackle
2207370	2216050	that head on. So we need to think about how do we actually make this thing usable? Additionally,
2216050	2221570	how is data going to be stored in this architecture? And perhaps maybe it can do
2221570	2227490	other things other than just storing and retrieving data as Freenet currently does.
2228790	2236450	So, preventing malicious behavior. What are the threats? Well, somebody could potentially select
2236450	2244650	an identity to attract certain types of data or otherwise act maliciously. They could potentially,
2244650	2250750	through a similar means, manipulate the identities of other nodes for some nefarious purpose.
2251370	2256390	How do we ensure ease of use? Well, one requirement is that peers will need to be,
2256390	2261090	or at least most peers will need to be always on. That's not so much of an issue now that
2262610	2270550	broadband is better deployed and is being deployed so quickly. How do we introduce peers?
2271270	2278170	Can we do it by email? Kind of a PGP style thing where you have a block of data that you can cut
2278170	2283670	and paste into Freenet and that can facilitate introduction. Maybe you can do it by phone,
2283670	2288270	verbally phoning up your friend and saying, hey, let's do this. Here's the kind of code number
2289090	2294010	to allow us to connect to each other securely. Or maybe you're willing, maybe you've got a lower
2295330	2301670	threshold and you're willing to trust some third party to negotiate this introduction.
2301770	2306350	What about NATs and firewalls? One of the big problems that peer-to-peer has experienced over
2306350	2311930	the past few years is that with peer-to-peer, it's essential that people can make incoming
2311930	2319030	connections to your peer and NATs and firewalls make that a lot more difficult. Fortunately,
2319030	2324110	people have developed techniques to get around that, but that adds challenges in terms of peer
2324110	2329470	introduction. So we can use UDP hole punching as kind of an increasingly standardized way to
2329470	2336390	achieve that. It's used by Digger, another one of my projects, and Skype. But that would require
2336510	2345350	a third party for negotiation. So in conclusion, real quick, so we have time for a few questions.
2345770	2351530	We believe that this is not only possible, we're going to do it and we are doing it. There's still
2351710	2357810	a lot of work to do on the theory. In other words, can other models work better? In particular,
2357810	2365630	right now, when nodes choose each other to decide whether swapping would be advantageous,
2365630	2372070	they do so at random. That's kind of nice for some of the mathematics that we haven't really
2372070	2380250	discussed for lack of time, but it may mean that it doesn't converge to a good
2380250	2385610	network as quickly as it could. So we need to explore perhaps a more directed search for peers
2385610	2393790	to swap with. And it needs, like everything, it needs to be tested on more data.
2394410	2399550	We've learned the hard way that practice is more difficult than theory, particularly with
2399550	2406850	Freenet. Security issues clearly are critical. That's one of our our major focuses. And
2406850	2411250	deployment of the network, you know, we could come up with the coolest thing in the world that's
2411250	2417570	ultra secure. If it's not easy to use, we'll get 10 users and it won't be useful to anyone.
2418440	2424550	So if you're interested in participating, helping Freenet, if you don't know, it's an open source
2424550	2432210	project where we try to be very open to people who are willing and able to make a contribution.
2432210	2441750	So if this interests you, our website is freenetproject.org. And Oscar insists that I
2441750	2451270	show you this slide. So I guess we have about five minutes for some questions.
2451770	2474650	So the question was, what is the incentive for people to acquire a lot of neighbors? Well,
2474650	2480510	the incentive is actually quite simple. The more neighbors you have, the better connected to the
2480510	2486090	network you are, and therefore you'll be able to find and retrieve data more quickly. If you,
2486090	2491490	let's say you've only got one connection to the network, chances are they're on a ADSL broadband
2491490	2497750	line. They've got perhaps 10 or 20 kilobytes per second upstream. If you want to download
2497750	2502690	something and you're just relying on one person, that's going to place a serious limit on the
2502690	2506990	speed with which you can do that. So do you want to comment? No, I just wanted to say it is an
2506990	2511210	extremely important question that needs to be addressed because you really do need people to
2511210	2515430	have a lot of friends. If everyone just connects to one person, you get a network that's one long
2515430	2520090	string and you're not going to get anywhere. So you're right, and that has to do with this when
2520090	2524430	we're talking about ease of use, that how we deploy this, how we get people to install it,
2524430	2528870	and what they do right afterwards is really going to affect whether it can work or not.
2528870	2534090	So it's something that one needs to be very careful with. And this is the insane part about
2534090	2539830	trying to write these networks which rely on these emergent effects, that some of it is out of our
2539830	2544710	hands. It's not like deploying normal software. We just write software that works. We actually
2544710	2561700	need people to use it in the right way. But, you know, I don't think we've gotten there.
2563760	2575620	Yeah, you basically, you would just disconnect from them at that point. I mean, you can,
2575620	2583020	as you can start them up. And also, I mean, there are lots of these trust issues. Like,
2583020	2587580	for instance, you could want to deal with, people who are less paranoid might want to
2587580	2592520	allow for a friend of a friend connection so as to speed the network up and stuff like that.
2636650	2644850	So if I can kind of address your core point, because no one else can hear you. So
2644850	2651290	I think your core question was, we make the assumption that shorter paths are almost always
2651290	2656470	better. And in this, we've ignored the fact that some people are on modems, some people are on
2656470	2661670	fast ADSL connections, etc, etc. And that's a good point. We haven't discussed that. But we
2661670	2669790	have actually considered that. The approach that we're considering in order to address that issue
2670210	2679230	is, in effect, you split up the friends that you're connected to into several groups. Let's
2679230	2683850	say it's two groups for simplicity. You have your fast friends and your slow friends. So
2683850	2702430	the top 50% who's faster, the bottom 50% who's slower. Yeah, yeah. So you don't slot them into
2702430	2709290	fixed categories. You can adapt that as time goes on. And then when you start out routing,
2709710	2717890	you want to try and find it from your fast friends first. So when you initiate a request,
2717890	2724970	you will send it to your faster friends. And if that fails to find it, then you resort to
2724970	2729830	your slower friends. I'm sorry, I don't really have time to give a more in-depth explanation.
2746130	2752410	Well, well, I mean, that is one of the dangers. People trying to manipulate the network with
2752410	2758070	fraudulent swap requests. And there are a variety of ways that you can mitigate. There are a variety
2758070	2763670	of ways you can mitigate it. One example would be, well, I don't really have time to go into it,
2763670	2771690	but we have thought about a technique that's a variation on how you can do secure group random
2771690	2777870	number generation. So basically get someone to cryptographically commit to a certain rate. Well,
2777870	2781950	I can't really explain it in the time available, but we'll talk later.
2786490	2808270	We're just done. One more question. Right. Well, I mean, that is in effect up to
2808270	2814950	users' self-interest. It is in users' self-interest to trust the people that they're connecting to
2814950	2825610	and as such, our hypothesis is that they will tend to form a social network. It so happens that the
2825610	2836770	criteria for a structured network to work here are actually pretty robust. It doesn't demand that you
2836770	2843670	have exactly the one over distance distribution that Kleinberg suggested. It's actually very,
2844230	2851310	and beyond that, addressing the issue while you're saying the John Kerry problem, in many ways,
2851310	2857870	that's kind of a usability issue, steering people towards a certain model of usage of the software,
2857870	2862490	just in terms of how it interacts with users. I'm afraid we're going to have to end it there.
2862490	2863210	Thank you.
2871400	2872600	We'll talk about that.