start	end	text
1820	6280	I'm Raven, and it's nice to see a bunch of familiar folks, and...
7040	8340	Is that better?
8840	11140	Alright, I will eat the microphone.
14120	16140	Not like that.
17400	18420	Alright.
21500	25880	I'm here to talk about penetration testing for background networks,
25880	29320	and I have these really interesting, great slides,
29320	33560	and then, Wednesday, some guy came and revolutionized my field, so...
33560	38060	I haven't gone to any parties this year, and I've done a lot of late-night rewriting.
38280	40740	So, thank you, Michael Lin, and...
40740	42620	The reason we were waiting...
43420	45720	is that my lawyer isn't here yet.
46820	48620	Oh, she's here. Fantastic.
49740	53260	So, the first thing that you really need to address
53260	55960	when you're looking at penetration testing for background networks
55960	63120	is that there's a pretty severe culture gap between ISP folks and the hacker folks,
63120	69520	and there's not really similar views on disclosure or vulnerabilities or anything like that,
69520	71460	and that can be kind of problematic.
72500	76300	So, how many of you do work or have worked for an ISP?
76300	77320	Show of hands?
77440	80300	Okay, you guys have probably run into this.
80300	84480	How many of you were in the routing department rather than the security department?
84480	85260	Show of hands?
86000	87200	Okay.
88900	90880	I used to work for ISPs.
90880	96340	I spent a bunch of years doing senior ISP background engineering and things like that,
96340	100260	and I eventually left for a more security-focused role
100260	104700	because I got really frustrated with just the differences of opinion.
104880	108180	A lot of ISP engineers, when they look at vulnerabilities,
108480	109960	they tend to have a very,
109960	113820	oh, well, no one's going to do that sort of view of things.
113820	117860	And their views on actual attacks, if they don't see it in the wild,
118060	122280	a lot of times they're very unwilling to actually treat something as a real problem.
122560	127520	And so it takes something like, well, Wednesday to make people sit up and take notice.
128360	133020	ISP engineers, in general, tend to go for a limited tiered disclosure method,
133020	135660	so when something is found, you know,
135660	138720	and people like that hear first,
138720	143260	and then Cisco's next favorite children hear second, and so on and so on.
143260	146740	And so if you have their gear, which are a small network near the edge,
146740	149100	you will hear late, if at all.
149380	152340	And most ISP engineers think that this is good.
152360	155940	They believe that, you know, catching the core is the most important thing,
155940	161000	and then moving outwards from there will release things in a more slow fashion
161000	163300	and minimize the window of vulnerability.
163300	167040	I have a feeling that most of you don't think this is a good idea.
168900	172060	So when you're trying to do pen testing on a backbone,
172060	174880	you're going to have to work with these ISP engineers,
174880	178840	and you have to sort of be a little bit gentle with them,
178840	181300	because otherwise they'll immediately say,
181300	183140	evil hacksaw, and stop listening.
183480	186320	So for maximum efficiency and security,
186320	189320	generally, when you're looking at testing a backbone,
189320	193360	I generally recommend doing it in a white box fashion rather than a black box,
193360	197340	because you can do all sorts of recon and data gathering and such
197340	200220	when you're actually doing it in a black box fashion,
200220	203180	but so many of the vulnerabilities that there are in backbone gear
203180	205120	are denial of service that people get a little pissed
205120	206620	when you take down their network.
207220	210620	And even if you've explained this to them,
210620	212080	you've signed the contracts beforehand,
212080	213660	you've made sure that you're legally covered, and say,
213660	215740	no, look, I could knock over your router,
215740	219300	they don't know what that means a lot of the time.
219300	222220	And so then they get ticked off that their backup didn't go through,
222220	224940	or suddenly they can't get to their router that you knocked over
224940	226520	because it's offline.
226760	229820	And so you want to be really careful when you're setting up your gig
229820	231540	if you're going to pentest a backbone,
231540	234780	because a lot of people don't really understand their ramifications.
235160	240120	So when I do this, I generally go with a white box sort of testing methodology.
242060	245120	So, when you're doing it, it's just like any other pentest.
245120	246780	You do your initial block of recon,
246780	250840	you look at the registrars for the address blocks and the autonomous systems that they have,
250840	252600	and any other data that's associated.
252600	255580	You can often get contact information for social engineering,
255580	260640	you can pull all sorts of information about the network that you are intending to test.
261040	264020	By doing a lot of these standard pentest techniques,
264020	266060	you can identify hosts on the edge,
266060	268620	but looking at the other IP addresses in their net blocks
268620	272020	is more likely to give you addresses that they're using on their backbone.
272020	277080	Now, a lot of people are, or should be, filtering information on the backbone
277080	279280	from things past the edge of their network.
279300	282820	In a perfect world, you shouldn't be able to hit the routers themselves
282820	287480	from outside of the network, unless you are on a management station or something like that.
287480	290620	It's just traffic should go through, and you shouldn't be able to target.
291060	294260	This is really, really not a perfect world.
294860	297400	A lot of ISPs fail at that level of filtering,
297400	300800	and we'll look at some stuff from the routing protocol security group a bit later on
300800	302800	that will show you some data on that.
303580	306760	It is worthwhile. There are many public route servers,
306760	309360	and it's worthwhile to look on those and internet maps
309360	312760	to try and determine who is connected to whom and where the peering is.
312760	315060	A lot of peering happens at public exchange points.
315060	316420	A lot of it happens elsewhere.
316420	319160	You have private interconnects between different ISPs,
319160	322400	and those will be crucial points when you're actually mapping your network.
323240	327680	You can often also, on route servers, get information about their BGP routing policy
327680	331800	and try and figure out what they accept and what they send to where.
331840	334100	Community strings will often be viewable.
334780	338720	Important information about BGP and which routes are preferred.
338740	340540	And there are many public route servers over there
340540	343160	that you can look at things from their perspective.
343280	348320	So if the network that you're analyzing is not one that is directly connected to a route server,
348320	350780	looking through two or three will give you several points of view
350780	353140	into how to look at it on the net.
353720	355780	Whatever you get, dump it into Google.
355780	359340	And if you get good results in that, dump them right back into Google.
359340	362340	And that will really help you build a profile of the network.
362620	365940	I believe it's really important not to just take the network maps
365940	368040	that their admins give you in Visio or what have you,
368040	370820	but to also try and build your own map non-destructively.
370900	372680	Their engineers aren't aware of everything.
372680	374520	Often they're not aware of much.
376320	380900	And that can be really important when you're looking at things from a pentester's point of view.
380980	383840	If they don't tell you about a link and you don't look and you don't find it,
383840	385400	you're not going to test that.
386020	391020	Also, a lot of ISP engineers aren't terribly security aware.
391320	393440	And so they'll post a mailing list and say,
393440	396320	here's my config, gee, this isn't working guys, what do you think?
396320	399520	You can get all sorts of information about what they're running,
399520	401540	how they're running it, what kind of gear they have,
401540	405480	and all of that can be dumped into your favorite friendly neighborhood vulnerability database
405480	410120	to build a likely profile of attacks and exploits that you could use against their network.
412200	416940	Now, a lot of the backbone is built on gear from a very few vendors.
417160	421080	And these vendors are generally not fans of full disclosure.
421660	427920	Cisco and Juniper both have on their networks things that are behind the front end of their webpage.
427920	432860	You have to have a login, in many cases you have to have a contract in order to see details.
432860	437840	Cisco does release some vulnerability information publicly, but there's a lot that they don't.
437840	442780	Juniper releases almost nothing publicly, I think I've seen like six advisories from them ever
442780	446720	that didn't require some sort of back end acknowledgement.
447000	452540	And so they believe that by doing this they are helping to keep it out of the hands of script kitties.
452680	456800	If you have a contract with them, or if you have a login to their site or something like that,
456800	460360	you will have a much greater database of information that's available to you
460360	463000	when you're trying to determine what to test for.
463700	468940	We're going to talk a lot about vendor negotiation and dealing with your vendors
468940	472360	in order to secure the network as best as possible.
473840	478700	If you are working for a company, they may allow you to use their CCO or Juniper logins
478700	482060	to access the bug databases and look for vulnerability information.
482640	485840	Also, don't forget that the backbone is more than just routers.
485840	488820	There are switches, there's lots of stuff that gets done on Layer 2,
488820	490760	and a lot of people just kind of ignore that.
490760	492940	So test through your switch gear also.
497380	502060	Cisco in particular has really complex code trains.
502200	506180	iOS has a zillion different branches which support different features
506180	508640	and for different hardware platforms and such,
508640	513320	and you will need to be aware of what you are running in the network you're testing.
513340	518240	If they've left CDP on or something stupid like that, then you can get them to tell you.
518240	521600	If their engineers are friendly, you can get them to tell you.
521600	524300	You can log into all the routers if they're giving you that level of access
524300	527620	and do a chauffeur and pull details on that.
527620	532560	But something that is vulnerable in one train may or may not be implemented in other trains.
533100	536600	It's worth looking with a broad brush to start off with.
536800	539280	Something might be vulnerable and then going,
539280	541940	oh, okay, it's not a vulnerable version is much better than saying,
541940	545600	ah, well, it's only this one version, let's not test that.
546140	550600	It's also worthwhile in looking at code trains and where they're derived from.
550600	555400	Code is forked all the time, and sometimes you'll come up with an error that was in one version
555400	559600	and years and years down the road, oh, look, it's still there.
559600	565020	There was a BSD vulnerability in Telnet a while back that whacked half the stuff on the Internet,
565020	569800	almost regardless of operating system, because so much was derived from that old code base.
569800	574060	And I have a link here to the vulnerability in Cisco's CatOS boxes.
574060	579660	We use that same bit of code and we're vulnerable to that same bit of exploit.
580180	583380	There are some scanners that will incorporate these checks for you.
583380	589260	I know Nessus has some Cisco vulnerability information in it, but a lot of it isn't automated yet.
589500	592380	And because of the great potential for denial of service,
592380	596660	you can see why a lot of players got real thrilled to have that run against them
596660	601040	and why a lot of people haven't been too eager to put it in the exploit tools.
601040	606080	So this is something that, I think, pen testing like 5 or 10 years ago,
606080	610420	you're going to have to do a lot of this by hand in order to make sure that you've got everything.
612660	617280	As with any other pen test, failure paths and trust relationships are important.
617340	620540	When you look at their architecture, look for redundancy,
620540	622940	look for network robustness, and look for security.
623620	627320	One thing that I think is really important when you're pen testing a backbone
627320	630900	is to make sure that availability is a factor.
630900	633660	I mean, denial of service is such a big deal for them,
633660	638200	and yet so many people build networks with one path where a link goes down and you're screwed.
638200	641420	Or they have two lines terminating into one card on one router,
641420	643440	and a hardware failure will cause problems.
643500	647100	I think, as security analysts, it's our job to point these things out and go,
647100	649700	oh, you've got a single point of failure if someone knocks this out.
649700	653200	If there's an exploit, if you have a hardware failure, you have an issue.
653240	657300	I personally try and include things like that when I present a report to the client.
657920	662780	Also, check for the authentication means and see whether those have redundancy.
662780	667840	Just as with some of the SSL versions, you know, if you can force it back to the weakest point,
667840	669760	then you can exploit that.
669760	674280	There was a vulnerability on a RADIUS server in Cisco's implementation of RADIUS
674280	678840	where you could give it several ways of authenticating and none was an option.
678840	681540	So if you can just say, oh, well, I don't want to authenticate, no.
681540	683620	You know, oh, okay then.
684120	686040	Things like that are kind of bad.
689500	693360	So with things like that, check for what the weakest way is
693360	698100	because you know your attacker is going to take the weakest way and if Pentester is still awake.
698980	701760	Once you're authenticated, is trust transitive?
701760	705400	You know, if I get a login to that RADIUS server, what else does that give me?
705480	707620	It's important to look at that.
707620	711640	And so as soon as you get every new level of access, every new point,
711640	715500	if you try and map the network again from that point of view, you may find extra stuff.
718900	723980	One of the things you'll see a lot and one of the ways I've had most success in testing backbone networks
723980	726060	is authentication servers.
726100	730740	If you have a centralized authentication server, that's a really good point to start at.
730740	734720	If you look for a login to that, you can sniff traffic and get someone's login
734720	736700	if they are foolish enough not to encrypt it.
736700	739760	If you can own one of the hosts that they use for management stations,
739760	744380	you can then grab it before it's encrypted and sent on the wire with a keystroke logger or something like it.
744460	747240	And that will often give you the keys to the kingdom.
747240	751560	Very many places don't think to harden their authentication servers,
751560	755160	and so that's a really common point of failure.
755620	760320	Another thing that I have successfully done is if they have fallback mechanisms
760320	766500	where they have a local login in case their RADIUS or their CACX or their whatever server is down,
766500	770480	well, if you can knock that server offline and then brute force the local password,
770480	772120	that will also get you in.
772340	776120	And once you brute force the local password, depending on how they're storing it,
776120	781680	like for a Cisco box, they use a 7 encoding, which it's not encryption, it's encoding.
781680	785240	It's symmetric, it's easily reversible, alcrypto.co.uk.
785240	790260	Cisco has an online decryptor, although you may wonder where that's going.
790480	794620	So they also have programs you can download and run locally.
794640	798180	But a lot of the passwords that are saved in the configs,
798180	801040	you can just throw them in the internet, get the password out,
801040	802920	and they may be using that in multiple places.
802920	806320	But a lot of places do not do password management well,
806320	809860	and so a local login at one place will get you a local login at another place.
809860	813780	And as long as that central authentication server is down, there you go.
814600	819980	Obviously, once you've logged into the boxes, if you have enabled, you can change that.
820620	824220	But that will let them know you're there. You know, please be nice.
827700	833880	Also, man-in-the-middling against the authentication server is usually a fairly successful technique,
833880	836400	if you can get access to do that.
839760	843600	It shouldn't need mentioning, but physical security is really important.
843600	847480	A lot of places forget this. Everywhere you have a data center,
847480	852060	and especially everywhere you're peering, please have good physical security.
852280	855420	Please audit and make sure you have good physical security.
855420	860380	Please don't let me walk into Macy's with a switch under my arm and a laptop under the other one and plug in.
860380	861780	Yeah, bad.
863180	869100	A lot of the time, if you have local access, local physical access, that will get you a lot.
869340	876820	And many of the places, they have a very clear profile of what they're looking for.
876980	880680	So, you know, if I was 40 and had a beard and came in,
880680	886400	they might be a little more suspicious of me than if I'm 29 and got my eyelashes and go,
886400	888740	gee, I don't know, I'm just supposed to go look at this router.
889160	894220	They have poorly placed trust relationships in terms of local access,
894220	898600	and if you want to take a social engineering approach to that, then that can work to your advantage.
899240	902040	If you don't have physical security, you're screwed.
904650	908490	One of the things that I've seen fairly often that's kind of appalling
908490	914330	is how many people will leak data about the processes of the core of their network to the edge.
914570	918290	So they'll have EIGRP as their internal routing protocol,
918290	922410	and they won't turn it off for interfaces that face out towards the edge of their network.
922410	928190	They just sort of send those multicast, hey, hey, who wants to be my neighbor, out to anyone.
928210	932550	And, well, you shouldn't let people at the edge of your network establish a neighbor relationship
932550	936270	and feed you a bunch of route data, which may be bad.
937370	942810	Also, CDP. People will leave it on by default, they won't turn it off,
942810	946990	and it will happily send you all kinds of information about the connected device.
946990	950370	It will tell you what it is, what version of the operating system it's running,
950370	954190	what sort of hardware it has, device ID, all kinds of stuff.
954550	959530	And Cisco has designed this so that it's very useful when you're troubleshooting.
959630	963610	I have had cases where I wanted to see what was on the other end of the link,
963610	969930	and we weren't sure that the person at the peering site or the central office had us plugged into the right ports.
970010	976290	So the other side turns on CDP, and you go, whoa, I'm not supposed to be connected to a 7206 VXR.
976290	977670	Yeah, that's wrong.
977670	979830	So when troubleshooting, it's a good tool.
979830	983070	But when you're not troubleshooting, turn it off.
984010	988950	Switching data. Also, switch management protocols, things like that, will be happily sent out.
988950	992150	Hey, welcome to my spanning tree, who wants to be my root bridge?
992150	996170	These things should not be seen by the end users, and they often are.
996170	998470	So that's something for you pen testers to look for.
1000010	1003150	If you can see the data, you can send things back.
1003150	1009450	You can spoof packets, you can send real data, you can feed them information on who's supposed to be bridging to what,
1009450	1011210	or who's supposed to be routing to whom.
1011210	1017270	If you can get a more preferred route and inject it, then that will, depending on the implementation, of course,
1017270	1022570	it may be able to take over the route that's actually on the legitimate internet.
1023690	1028670	A lot of this is... when you're speaking internal routing protocols rather than BGP,
1028670	1031130	it's a lot easier to deal with on the local net.
1031130	1034990	BGP has its own separate set of rules because of how it's handled between providers
1034990	1037730	and how you set up neighbor and peering relationships.
1037910	1041570	But on the local network, it's often not that hard.
1041570	1048010	If you want to do a really sort of not high-skilled but very effective DOS attack,
1048010	1050030	you can tear down links with spoof packets.
1050030	1053010	A lot of the time, they don't have authentication on there,
1053010	1059790	and so many routing protocols support the capacity for even something as simple as an mb5 hash.
1059870	1064450	It's often called BGP password, although many other protocols also support the same thing.
1064770	1070410	I know you guys have heard about mb5 fairly recently, but, you know, it's still a whole lot better than nothing.
1075260	1079220	Similar to the theory of rogue access points, if you have physical access to the site,
1079220	1080760	you can just walk right in with a router and say,
1080760	1084840	yeah, I'm supposed to install this, and some places aren't very good about saying,
1084840	1088300	who the hell are you, why didn't I know you were coming, you know, go home.
1088620	1091840	And you can plug in a router, you can put it into the peering switch,
1091840	1094280	you can just, you know, start snipping traffic.
1094280	1098320	If you can manage to get the switch ports set promiscuous or something like that,
1098320	1101100	then you can see what they're speaking and start to speak it too.
1101440	1103280	This also is obviously bad.
1104860	1110040	Depending on what your contract is with the client, you can even go in there and say,
1110040	1113960	oh, yeah, you know, I'm from this company, I'm supposed to work on their router,
1113960	1117000	yank out theirs, put in yours with the configuration of whatever you want,
1117000	1118920	and, hey, as long as you pass it through to them,
1118920	1121900	they may not even notice except for the ten minutes of reboot time.
1124410	1128330	A lot of places do not have good passwords.
1128370	1130570	It's better than it used to be because of spammers.
1130570	1133630	There have been cases of spammers taking over Cisco routers
1133630	1138890	and then using the telnet function to send spam out of the router itself one character at a time.
1139030	1141570	So, yeah, bad.
1142050	1148310	But a lot of places have gotten better about not having, at least pathetically, default passwords.
1148690	1151510	However, if I get called in and someone's locked out of their router,
1151510	1154970	I always give Cisco Cisco a try, you know, maybe one time in four it works,
1154970	1157490	and just, it shouldn't.
1158490	1161710	You can redirect traffic through a GRE tunnel.
1161710	1166910	I've seen that in the wild, and people were a little bit confused about what was going on there,
1166910	1170470	and said, I don't remember setting that up. Bob, did you set that up?
1170470	1177350	And meanwhile, it's terminating in, you know, Moldavia or somewhere, and there go all your packets.
1178990	1184610	Another serious concern is that, especially when you're speaking BGP,
1184610	1188350	you may be able to advertise additional net blocks out to the net,
1188350	1193790	and if your route is preferred to other people's routes, you've just hijacked a net block.
1194170	1196750	And this happened.
1196850	1201850	There was a bunch of attackers that set up a site through an ISP that wasn't doing good security checks
1201850	1203790	about who actually owned that IP space.
1203790	1208490	They announced this block out to the world, and their route was preferred in most places
1208490	1211950	because of the way that they did it and how their provider was set up,
1211950	1215970	and they happily spent about a week sending out spam and sending out worms
1215970	1219090	instead of a botnet, and were owning boxes left and right.
1219090	1223510	Meanwhile, the legitimate users of that net block can't even get online because no one's routing to them.
1223510	1226370	Everyone's routing to the own site.
1226450	1231050	And so, when that was finally resolved at the ISP level,
1231050	1234710	then the original owners are really left holding the bag there.
1234710	1237030	Hi, you're on every blacklist on the planet,
1237030	1240590	you have a whole bunch of angry people pounding down the door at your abuse desk,
1240590	1242530	and you were offline for a week.
1242530	1246910	And because some ISPs do not do good checking when they're hooking up a new client
1246910	1249790	of do you really own this block, is this legitimate,
1249790	1252110	that can happen, and it shouldn't.
1252110	1254630	There are some really good people working ISP security
1254630	1258490	that are cooperating between ISPs to try and prevent that.
1258610	1261430	And, you know, more power to them. We need more of them.
1264290	1269210	A great place to start when looking at the actual configurations of the backbone devices
1269210	1272810	is to look at the secure BGP template,
1272810	1275650	iOS template, Junos template, and Junos BGP template
1275650	1277830	that the Cymru people do.
1277830	1280690	These are very, very well done documents.
1280690	1283930	They have bogon lists so that you cannot accept routes
1283930	1287490	from IP addresses that haven't been allocated or don't exist.
1287490	1290290	They do strong filtering so that you don't have, like, you know,
1290290	1293130	RFC 1918 addresses speaking to you,
1293130	1295250	you shouldn't allow people to advertise, you know,
1295250	1298230	your own net blocks to you, for example, things like that.
1298230	1302310	They have good checking for the strongest possible authentication methods.
1302310	1305910	They try and turn off things like, oh, hey, let's speak finger,
1305910	1309650	you know, ancient insecure daemons that you have no business really needing
1309650	1311730	to run on your major backbone routers.
1311730	1314610	I can't recommend these enough. Those guys do really good work,
1314610	1318650	and it's also handy for a beginning place to start with an engagement
1318650	1321330	if you're doing a white box design analysis.
1323010	1325590	Peering security is particularly important.
1325910	1327670	When you look at how someone is peering,
1327670	1330070	look at the routes they're willing to accept and look at the routes
1330070	1332050	that they're willing to send out.
1332410	1335970	There was a case several years ago where someone actually advertised
1335970	1340870	10.0.0.0 to the whole internet and claimed that they owned it.
1340870	1343910	And that got a lot of people a little bit irate.
1344130	1347090	It broke a lot of people's private network implementations,
1347090	1349630	and it broke a lot of VPN stuff.
1350530	1352190	So that's worth checking.
1352310	1354390	Again, route servers are your friend.
1354390	1358250	And when you do peering changes, make sure you're doing good authentication.
1358250	1360530	You don't want just anyone to call you up and say,
1360530	1365870	Hi, I'm Raven from Uunet, and yeah, I want you to change this config.
1365870	1369190	Can you take out this net block of this guy who, you know,
1369190	1370930	stole my girlfriend last week?
1371430	1374850	You want actual passwords, some sort of validation method
1374850	1377630	to make sure the person who you're talking to to make these major
1377630	1381450	affecting changes actually is the person that they claim to be.
1382030	1385670	Also, make sure that the machines that you use for your network policy
1385670	1387710	and your network admin are well secured.
1387730	1391150	I have worked at a job where we had a couple of management stations,
1391150	1394310	and they were Windows boxes, and wow, they got owned.
1395090	1396750	That's really dumb.
1396750	1399510	If you have only a few machines that are allowed to access
1399510	1402490	these crucial backbone devices, make sure they're hardened.
1402490	1403490	Make sure they're patched.
1403490	1406450	Make sure that people can't get onto them and, you know...
1408370	1410970	In that case, I think what happened was
1410970	1413530	these machines that we did backbone stuff from
1413530	1416170	were the same machines that we did abuse tests from,
1416170	1418330	and so you're investigating an abuse case,
1418330	1420930	and oh, our user's surfing porn, go to the porn site,
1420930	1425010	porn site's full of malware, mandated corporate IE, boom.
1425390	1426890	Don't let that happen.
1429030	1431710	And filtering is also very important.
1431710	1434930	Not just what you allow to your router for management traffic,
1434930	1437390	but also what you allow as far as routes.
1437710	1440370	You want to make sure that you do not allow
1440370	1442790	internal routes that should stay internal to your network
1442790	1443890	outside to the world.
1443890	1447750	Yes, aggregation happens, but more precise routes are often preferred,
1447750	1448690	and that can be a problem.
1448690	1450830	It bloats the routing tables, it uses up memory,
1450830	1455370	and it can be revealing more details about the interior of your network
1455370	1456510	than you wanted.
1457090	1461150	The RPSec Working Group did a small survey of people in the field,
1461150	1464210	and they surveyed people who were peering at an exchange point.
1464210	1469070	So these are some of the more aware, more security conscious people out there.
1469170	1472090	And they still weren't using everything that was available.
1472710	1477390	This may be because, for lack of a better term, ISP culture.
1478090	1482150	They view change management in the time that it takes to do good password control
1482150	1487950	and to change all these policies as more of a threat than the security implications.
1488110	1490810	And so you've got to keep track of who's allowed to do what,
1490810	1492270	and who's filtering where,
1492270	1494670	and, you know, oh, we screwed up our peering list,
1494670	1497630	and, you know, okay, now there's downtime because I wasn't allowing your route,
1497630	1499250	or I had too many prefixes.
1499290	1502090	And so, you know, they think that the hassle of management
1502090	1505070	is often not worth the additional security.
1505390	1508750	And from an ISP point of view, I can see why they take that approach.
1508750	1510110	When you're driven by the bottom line,
1510110	1513110	and you're a network engineer with too much work and too little time,
1513110	1515490	it can be difficult to deal with these sorts of things.
1515630	1518410	Nevertheless, I think this is going to become increasingly important
1518410	1522410	as more attention focuses on the backbone as, you know, started this week.
1524050	1525790	The RPSEC site security.
1526630	1531550	Many people are actually doing an incoming prefix filter for each BGP customer,
1531550	1532110	which is good.
1532110	1535070	You should only allow people to advertise their own netblocks to you.
1535070	1537130	You don't want them to start advertising your netblocks,
1537130	1538490	or, you know, someone else's netblocks,
1538490	1540470	or that guy who stole your girlfriend's netblocks.
1540470	1543030	That can get you in legal hot water.
1543030	1547810	A lot of them also are checking for incoming autonomous system paths.
1547890	1551750	A lot of people use community strings in order to control this sort of thing.
1551830	1555490	However, very few people are using an outgoing AS path filter,
1555490	1559450	and in that case, it really depends on who you are.
1559470	1562250	If you are a small ISP near the edge of the network,
1562250	1566290	you have your AS, and you have a few others that are your customers,
1566290	1568110	that may actually be a good idea.
1568130	1572410	If you are a tier 1, and you have every AS in the world flowing through you,
1572410	1574910	that can be a real change management challenge,
1574910	1577070	and probably operationally unwise.
1578910	1580850	Outgoing updates towards your peers.
1580890	1583670	A lot of people use communities on that, but not everyone.
1583730	1587550	And there is the URL here for the actual survey,
1587550	1590050	if you want to look at the results from the RPSEC group.
1590050	1591250	I didn't put them all up.
1592450	1595550	So, when you're looking at actually doing a pentest,
1595550	1597430	where do you get your backbone vulnerabilities?
1597890	1599490	They're not all in the scanning tools.
1599490	1600790	We've established that earlier.
1600790	1604950	And Cisco and Juniper may have you log into their site to actually look at it.
1605070	1607730	Mailing lists will have public vulnerability announcements.
1607790	1611990	I've often found that it is a good idea to check the mailing list against the vendor site,
1611990	1615010	because, hey, people change their webpages all the time.
1615030	1616950	Mailing list archives are usually more static.
1616950	1619750	You can at least see what the first release was of something.
1619810	1624570	We saw this happen this week, when Cisco announced their IPv6 vulnerability.
1624790	1627310	And if you look on their website now, it's 1.1.
1627310	1629690	The original one that went full disclosure is 1.0.
1629690	1631230	There have been some changes made.
1631830	1635110	And it is valuable to Pentester to be able to see both versions.
1635370	1642990	There's been, in the last five years, increasing interest in this field by hackers who don't come from an ISP background.
1643570	1647930	FX and Paul Watson and Mike Lynn have all made valuable and important contributions
1647930	1651350	that have really caused a bit of uproar in the ISP community.
1651570	1655770	And after this week's uproar, I think there will be a whole lot more people looking for it.
1659140	1664700	So, FX has done a lot of stuff in this field that is really useful.
1665040	1669520	There is a EIGRP Spoof Packet Neighbor Saturation bug,
1669520	1675500	where essentially, if you send it more than 256 neighbors, then it just falls over.
1676540	1679140	EIGRP works by sending out multitasks.
1679140	1681240	Hey, here I am! Who wants to be my neighbor?
1681360	1684320	And so if you say, me, and me, and me, and me, and me,
1684320	1688640	you can essentially overflow the amount of memory that it allocates for that,
1688640	1692700	and, you know, it's at least a denial of service.
1692740	1694000	It could be worse.
1694520	1697680	Similarly, for the CBP neighbor announcement,
1697680	1702880	if you send Cisco Discovery Protocol packets with device IDs that are really, really long,
1702880	1706980	you can fill up the memory tables and possibly do evil after that.
1707100	1711200	There's been GRE attacks, a TFTP server advisory, and a bunch of other stuff.
1711200	1715660	If you look at FX's site on Phenolate, there's tons of stuff.
1715660	1720540	There's actually a tool for pen testing that was released specifically to address these kinds of things.
1720540	1727240	It only addresses the vulnerabilities that FX actually found, but it is well worth looking at.
1727860	1733460	There's also been a really interesting paper about Cisco memory mapping,
1733460	1740200	and I'm sure that any of you that were at Black Hat on Wednesday saw similar work in Mike Lynn's talk.
1740200	1743880	The memory structures are still the same, the magic numbers are still the same,
1743880	1748820	they're all online, and it is very useful if you want to dig into the assembler.
1751090	1755890	Then about a year ago, any of you that were at CanSec probably heard the storm and fury
1755890	1758570	surrounding Paul Watson and the TCP window.
1758850	1762210	Essentially, TCP stacks on background equipment and many other things
1762210	1766310	were not correctly looking at the window size and anything that hit spot on.
1766310	1768890	Just anything in the general neighborhood was good enough.
1768890	1771790	So if you send a reset that's in the general neighborhood,
1771790	1776030	and you're able to correctly guess the IP addresses and port numbers on either side,
1776030	1778310	you can knock down the session with a spoof packet.
1778830	1786510	And this allowed unauthenticated BGP sessions, if you could guess it all correctly, to be knocked down.
1786910	1791530	And this was really interesting when you look at disclosure and how the whole thing went out,
1791530	1797230	because essentially what happened was when Cisco started notifying people of this,
1797230	1801810	because Watson worked with Cisco and said, hey, look, we need to fix this.
1802110	1807130	Many people on the backbone and in NANOG and similar ISP communities
1807130	1810050	had known about this for years. They just didn't think it was very possible.
1810150	1814570	And again, we see that attitude of, well, I'm really busy unless I see something in the wild.
1814570	1816490	It's not a problem. It's not going to happen.
1816950	1823890	And so Cisco said, okay, everyone, you need to put md5 authentication passwords on your BGP sessions now.
1823890	1827690	Okay, you need to upgrade now. They didn't tell people why.
1828090	1831170	And so they told their favorite people first, and then their second favorite people,
1831170	1833410	and their third favorite people, and so on out.
1833450	1839150	And so it took, I think, the better part of a month for it to actually get heard by most people.
1839150	1841230	And then Watson gave his CanSec talk.
1842050	1845590	Once the ISP operators found out what the actual flaw was,
1845590	1849650	there was a very strong cultural backlash saying, oh, I knew all about that.
1849650	1852430	You know, why are you making such a big deal?
1852430	1858550	And I was really depressed by going to NANOG this year, where a lot of operators were like, oh, yeah, you know,
1858550	1861490	we thought that was important, but wow, we really freaked out.
1861490	1864650	And I've rolled back. I don't have any authentication on my sessions anymore.
1864650	1866030	You know, stupid hackers.
1866630	1869430	And that was upsetting.
1869490	1877170	Because I really saw the community response to the Watson vulnerability as a strong positive step in securing the backbone.
1877430	1881430	And then seeing everyone go, wow, we freaked out over nothing, did not help.
1881430	1887430	I think Cisco's approach of we're not telling you why, just patch it because we say so, no really, just trust us,
1887970	1891850	I think they hurt themselves because people trust them less than they used to.
1894550	1897890	And then we come to Wednesday and what made me rewrite my slides.
1899070	1903210	If you were at Black Hat and saw Michael in session on Wednesday morning,
1903210	1908470	he essentially demonstrated what I view as a remote exploit in Cisco IOS.
1908470	1913230	Cisco has announced in at least one of their vulnerability announcements that it was just a local.
1914010	1917570	I think we have different meanings for the same word there.
1920510	1928730	And again, the disclosure process here was important because when Mike gave his talk,
1928730	1932970	he essentially talked about the internals of how it worked.
1932970	1938770	That stack overflows are hard to find in Cisco IOS, that the code base looked strongly audited,
1938770	1943310	and that you had to work with heap overflows, but that Cisco has a check-heaps process,
1943310	1946890	which will go through and validate, make sure the previous and next values are the same,
1946890	1951750	so that if you actually do corrupt the heap, then it will say, oh, this isn't okay,
1951750	1954450	all right, I'm crashing the router because this is screwed up.
1955710	1959830	He did not give details of what his entry vector was.
1959830	1966030	He was very careful to basically try and not give enough information for people to recreate it.
1966330	1970970	Now, even though the vulnerability has been patched and the old versions have been yanked off of Cisco's website,
1970970	1973470	there are still vulnerable versions out there in the wild.
1974470	1980090	Cisco, really shockingly to me, yesterday morning released an advisory to full disclosure that said,
1980090	1985550	yeah, for the first time ever, we have a denial of service and possible remote code execution
1986370	1988790	in our handling of IPv6 packets.
1990330	1993750	Lynn didn't tell us that. Cisco told us that.
1994210	1999250	In the latest version of their advisory, they say that it was disclosed at Black Hat,
1999250	2002950	so that really does nail it down to, yes, this was the vulnerability.
2003970	2007470	And this is really revolutionary for the field, because for the first time,
2007470	2009890	it looks like you can remotely own a Cisco box.
2009890	2015810	You can get it to shovel you back a shell that will allow you to have full enable access.
2016090	2018810	And this is a scary thing if you're an ISP operator.
2020090	2025170	I think that Lynn was pretty sensible in saying that people need to treat these like normal computers,
2025170	2027170	they need to patch, they need to be aware of security,
2027170	2030370	and they need to be diligent about making sure that they patch through the versions,
2030370	2033930	because this is a real threat, people. We've now seen the remote route.
2035210	2040030	Now, when you look at the Cisco advisory, what they appear to mean by local,
2040030	2043150	to the best of my understanding, and, you know, disclaimer,
2043150	2047090	I don't and never have worked for Cisco, I don't and never have worked for ISS,
2047090	2054310	I don't speak for anyone but me, but Cisco claims that you have to be connected to the same local subnet
2054310	2057610	in order for that to work, and that makes it a local exploit, right?
2059450	2060330	Yeah.
2061670	2067590	No, if you have to be on the same subnet from one remote box to another remote box, that's a remote route.
2070560	2072380	And then came the cover-up.
2073580	2077920	The talk's been ripped out of the books, the presentation's not on the CD,
2077920	2080900	everyone got modified CDs, the slides aren't available,
2080900	2083700	oh my god, there's suing, and there's suing, and there's the FBI,
2083700	2087460	and there's restraining orders, and rumors, and news stories, and all sorts of stuff,
2087460	2090140	and Cisco, you're really screwing up here.
2090840	2109730	You know, to give you an idea of the attitude of the security community,
2109730	2112650	there are people walking around wearing these.
2119820	2126100	I think that gives you an idea of the mentality of what happens when you try and cover something up.
2126320	2128500	Okay, so you're not really happy.
2128680	2129640	Big pardon?
2130880	2132740	It said, fuck Cisco.
2134420	2136260	Let the record reflect.
2141360	2146360	And I really think that Mike Lynn was right on when he talked about why he gave the speech,
2146360	2149060	and why he released as much information as he did.
2149060	2154700	I mean, okay, the reason he got onto it in the first place, again, from the best of my understanding,
2154700	2157940	I don't know the guy, I don't work for anyone, yada yada.
2158380	2161720	He said that he found some Mandarin Chinese documents,
2161720	2165900	which when translated were talking about how to hook debugger hooks in iOS,
2165900	2169060	and how to make those speak to your disassembler and your debuggers.
2169440	2171300	This means people are working on it.
2171300	2176860	Their source code has been stolen, and there's now a storm of interest.
2176860	2181820	I know that there have been several people who have suddenly been diving into their assembler
2181820	2184420	to attempt to figure out what's going on.
2184960	2187640	Hiding your head in the sand is not going to help.
2188080	2191620	If you have people who are attacking you that are actively looking for this,
2191620	2195760	going, no, no, it doesn't happen, quick, shut everyone up, is not going to help.
2195860	2200720	And even if you destroy all the slides, and you aren't going to let things get released,
2200720	2203520	well, his slides are out there on the internet.
2203520	2206960	Suing researchers is not going to achieve security.
2206960	2208840	I mean, only speaking for myself.
2209300	2210920	I was in his talk.
2211340	2212680	I speak assembler.
2212680	2214140	I know Cisco iOS.
2214920	2218200	With enough work, I could probably recreate his exploit.
2218540	2220840	I'm sure I'm not the only one.
2221160	2224280	Trying to pretend it won't happen will not fix things.
2224380	2226960	And alienating the security community,
2226960	2231180	and making everyone think that you're going to be sued into oblivion if you ever speak up,
2231180	2235760	that's not going to encourage people to come to you and say, hey, we've got a problem, let's work with you.
2235920	2239660	When I was reading the reports of extant backbone vulnerabilities,
2239660	2243040	it sounded like they treated FX a lot nicer than they treated Mike Lynn.
2243040	2245900	FX actually thanks people on the Fino Elite Advisories,
2245900	2249660	and says, hey, thanks for working with me, Cisco P-Cert, you guys were really great.
2250440	2252400	They don't sound really great right now.
2253420	2257660	They filed a temporary restraining order to prevent him from speaking about it.
2257660	2260480	So, he can't talk about it, but I can.
2273960	2277400	Well, his slides are easily findable on the internet,
2277400	2282040	and not even the version that he presented at the talk.
2282040	2285300	The original slides from before he got fired by his employer,
2285300	2290480	I think like yesterday morning they were up on CryptoMay and several other sites,
2290480	2293480	and this has just been a ridiculous media circus.
2294020	2296440	They were put up on Infowarrior.org,
2296440	2299100	and they got pulled down with a cease and desist by Cisco,
2299100	2303020	and guys, just help us fix this.
2303560	2306260	The CD for that was from ISS, apparently.
2306760	2310920	So, he had to return his computer, he's not allowed to talk anymore,
2310920	2313320	and this is really dumb.
2314540	2317160	Dear Cisco and ISS,
2318020	2320000	Responsible disclosure is good.
2320400	2323440	Working with security researchers is good.
2323440	2325700	Trying to cover up things that are known problems
2325700	2327420	that have been released into the public domain
2327420	2332600	that are on millions and millions of machines all over the internet now is good.
2332820	2334640	If you try and shut people up,
2334640	2337940	you're only going to alienate the very people that you need to help you.
2339780	2340660	Duh.
2345140	2348220	Alright, so, what Mike Lynn actually said.
2349180	2351320	Heap overflows are way more common in iOS.
2351320	2353620	He said a bunch of nice things about their code base,
2353620	2358800	but he said that the watchdog process of check heaps will check for heap corruption,
2358800	2360600	and if it finds it, it will crash it.
2360840	2363660	And most of you that have worked with Cisco routers I'm sure will see
2363660	2367100	that if you have a Cisco router and it has a fault,
2367260	2369900	a lot of times, rather than actually trying to recover,
2369900	2371600	it will just crash itself.
2371840	2375560	And the really important thing that I think came out of Lynn's research
2375560	2378620	was that the crash process itself is interruptible.
2378620	2380680	So, you can interrupt it when it says,
2380680	2385400	Hey, this bad thing has happened. My heap has gone badly. Am I crashing?
2386080	2387960	And you can tell it,
2387960	2389200	Why, yes I am.
2389200	2391880	Okay, we're already crashing. I don't need to do anything then.
2391880	2395100	Please continue executing whatever the heck it is you were executing there.
2395780	2397800	So, you can buy yourself some time that way.
2397800	2401800	He did mention that spreading memory corruption on the heap was going to be an issue,
2401800	2407220	and that you might have two to five minutes before it exploded all over your memory, basically,
2407220	2409440	unless you actually handle that in your shellcode,
2409440	2412400	and anyone who's written shellcode knows that that can be tricky.
2413200	2415140	But once you've actually gained execution,
2415140	2417300	there are hardware interrupts that will try and interrupt you,
2417300	2418540	but you can disable those.
2418540	2421840	Once the CPU is yours, it's yours until the router crashes.
2423240	2425040	And through methods like that,
2425040	2428620	you can get the remote box to shovel an enabled shell back to your listener,
2428620	2430340	which is what Lynn did in his demo.
2431280	2434100	Enable is the Cisco equivalent of root. Game over.
2435260	2440340	He indicated that though he didn't give us the specific vector, and Cisco did,
2440340	2444200	that there were other vulnerabilities that you could do this for.
2444200	2447720	And here's some of them that might be fruitful grounds for exploration.
2449420	2453720	Anything that talks about memory corruption that leads to a denial of service
2453720	2457340	is probably a likely factor for this sort of thing,
2457340	2460640	simply because since the Cisco routers tend to crash,
2460640	2463080	rather than actually trying to recover,
2463080	2465500	if you screw up the memory enough to induce a crash,
2465500	2469660	it may just be, okay, your offset was wrong, or something like that.
2469800	2473980	I really strongly encourage Cisco to look at their code
2473980	2477860	and look at the way that he was able to interrupt the CheckHeaps process
2477860	2482660	and avert crashing, because that's the underlying problem.
2482660	2487820	The front-end vector, whether it's IPv6 or CDP or OSPF or BGP,
2487820	2490240	that's not really going to be your issue.
2490880	2494860	The fact that you can defeat CheckHeaps, that's your problem.
2496620	2501860	So, yeah, here's a link to the advisory about vulnerabilities in IPv6 processing
2501860	2503400	that Cisco put up.
2503540	2506860	And just in bold, because this is the first time I've seen it
2506860	2510000	and I've been waiting for years to see when this was going to occur,
2510000	2513880	because I believe it was possible, but Mike Lynn's shellcode is better than mine.
2515920	2519340	Cisco Internetwork Operating System is vulnerable to a denial of service
2519340	2522800	and potentially an arbitrary code execution attack.
2522800	2524740	So, Cisco did admit it.
2528380	2532980	Again, just to hammer home the point about local segment versus remote segment,
2532980	2538520	local network segment does not mean that it is not a remote group.
2538940	2540020	Thank you.
2541340	2544300	So, look at this for yourself. Don't just take my word.
2544300	2548980	Don't take FX's, don't take Mike Lynn's, don't take Paul Watson's.
2548980	2552740	Go and look at the source of these sorts of things.
2552740	2555060	Read the advisories, read the vulnerabilities,
2555560	2559140	and check what Cisco says now versus what Cisco said on full disclosure
2559140	2560980	versus what Cisco said before.
2561520	2565260	Read FX's paper about how Cisco handles memory management.
2566120	2570820	And there is a link to where these Lynn slides are,
2570820	2572180	and I believe they were still up.
2572180	2575980	They were still up when I talked about an hour or so before I got up here.
2577160	2578720	They're up. All right.
2580840	2583100	So, you network defender people.
2584560	2587220	Start treating your backbone devices like computers.
2587220	2588980	Realize that they will have vulnerabilities,
2588980	2591220	that you need to allocate for security,
2591220	2592700	that you need to work for this,
2592700	2597380	that you can't just sort of set it up like an ancient BSD box
2597380	2599220	and, oh yeah, no one's ever going to hack that.
2599220	2601960	It's fine. Routing is hard. No one understands that.
2602000	2604420	Well, I'd like to point out that of the people that I know
2604420	2605940	that are working in this field,
2605940	2609300	I think I'm the only one that actually has an ISP background.
2609660	2614260	You can't just depend on it's hard and security through obscurity.
2615980	2618660	So, yes, you need to treat them like computers,
2618660	2622000	but when you're testing, because denial of service is so prevalent,
2622000	2623900	treat them like really important computers.
2623900	2628180	Don't knock them over without your client's explicit consent.
2628280	2629520	Maybe packet log.
2629520	2632500	When I'm pentesting, I run a packet log
2632500	2634940	so that if they come and say, you know, you broke XYZ,
2634940	2635980	I can go, really, where?
2635980	2638940	Here's my timestamps, here's my packets, here's exactly what I did.
2639960	2643720	So that may be a contract issue for those of you who are doing that.
2645020	2647380	Attacks only get better. They don't get worse.
2647860	2652020	In time, these test tools are going to become a regular part of pentesters' toolkits.
2652040	2656840	I would not be shocked if next year I suddenly saw Cisco attacks in Metasploit
2656840	2661040	or something like that, or more Cisco's vulnerabilities in Nessus.
2661040	2664280	We're still kind of in the dark ages of pentesting through the backbone,
2664280	2666720	but keep in mind that this is where things are going.
2666720	2669580	Enough people are interested in working on it that it's going to happen.
2670600	2674260	And please don't take me off the internet, or slap me with a gag order,
2674260	2676040	or try to sue me, or anything like that.
2676040	2680120	There are mirrors. Everyone, please go out and mirror my stuff, put it on BitTorrent.
2680120	2681560	Don't let them shut me up.
2698020	2704380	Just a few last comments, and then I believe we have a brief time for questions.
2704380	2710980	You can reach me at raven at nmrc.org with feedback, additional ideas, comments, lawsuits.
2711700	2717480	I would like to thank Jericho and the OSVDB people for promptly putting all these vulnerabilities up
2717480	2721220	in a public space where everyone can see them and have equal access for research.
2721220	2726060	I would like to thank people who are mirroring these for me, and also possibly staring down the eyes of lawsuits.
2726180	2730280	And I'd like to thank the EFF for sitting in the front row. I love you, new best friends.
2741000	2743080	Alright, we have three minutes. Any questions?
2747080	2748240	Beg pardon?
2756120	2763400	My updated PowerPoint will be up momentarily. I think people are going to run out after this and mirror it.
2763400	2771340	I will put a site online where there can be a list of the extant mirrors, and if that gets knocked down, it gets knocked down.
2771820	2781780	Also, there is a legal defense fund for Mike Lynn. If you PayPal to abaddon, A-B-A-D-D-O-N, at io.com, you may need it.
2783560	2785980	Oh, apparently you will need it.
2787740	2788680	Yes.
2797170	2806150	The question was, given the negative reaction to the Paul Watson TCP issue, how confident am I that ISP engineers will actually pay attention to this?
2806410	2814310	Well, despite working in this field, I'm perpetually optimistic, but I think the fact that we have enough people continually saying,
2814310	2824150	Hey, look. Hey, look. Hey, look. I am cheered that even though I was not going to announce the details of what I was talking about beforehand,
2824150	2830930	because I didn't want to be shut up, a lot of my ISP friends contacted me over the last two days and went,
2830930	2835590	Oh, holy shit, what do we need to do? So, there are at least some people who care.
2835590	2841650	There is an ISP community that's dedicated to security. There are both public and private mailing lists that are working on this.
2841650	2846290	The problem is not that there aren't some people who care. The problem is that most of the people don't care.
2846290	2848410	So, truthful people, be loud.
2851990	2860400	Alright, one more question. Yes, in the back. That's you. I can't hear you.
2874400	2882840	The question was, since I've talked about ISP infrastructure and such, can the ISP protect its hosts against zombies and things like that?
2883180	2886940	Yes, the ISPs can to some degree. The question is whether they are willing to.
2886940	2890500	A lot of times to do that level of filtering takes additional resources.
2890500	2894580	You need stronger CPUs. You need more memory in the routers.
2894580	2897880	And a lot of places are running pretty close to bare bones on that.
2897880	2901200	So, they don't have a lot of resources to spare and they don't think it's a good allocation.
2901380	2904280	What they need to be able to do that is to hear from people and say,
2904280	2907460	No, really, I want you to do this. I want you to filter me.
2907460	2912180	There is client demand for it because if they don't see operational demand, they won't do it.
2927510	2933690	The comment was, don't you have to have an interest in keeping spam and zombies and such from happening in the first place?
2933690	2940990	Alluding to the poor record of many ISPs who benefit economically from allowing such things to happen across their network.
2941290	2947390	And, you know, I could go on for an hour about that, but suffice to say, market demand.
2948910	2952810	They are following where the money is. If you inform them where you want the money to be,
2952810	2955570	the more people that inform them, the more likely they will be to listen.
2960110	2962230	Something that we can do to fix what quickly?
2964110	2970930	It's already fixed. If you go and you update your IOS to the most recent version of firmware in your code train, you'll be safe.
2972150	2974670	All right, I think we're out of time. Thank you very much.