[00:01.820 --> 00:06.280]  I'm Raven, and it's nice to see a bunch of familiar folks, and...
[00:07.040 --> 00:08.340]  Is that better?
[00:08.840 --> 00:11.140]  Alright, I will eat the microphone.
[00:14.120 --> 00:16.140]  Not like that.
[00:17.400 --> 00:18.420]  Alright.
[00:21.500 --> 00:25.880]  I'm here to talk about penetration testing for background networks,
[00:25.880 --> 00:29.320]  and I have these really interesting, great slides,
[00:29.320 --> 00:33.560]  and then, Wednesday, some guy came and revolutionized my field, so...
[00:33.560 --> 00:38.060]  I haven't gone to any parties this year, and I've done a lot of late-night rewriting.
[00:38.280 --> 00:40.740]  So, thank you, Michael Lin, and...
[00:40.740 --> 00:42.620]  The reason we were waiting...
[00:43.420 --> 00:45.720]  is that my lawyer isn't here yet.
[00:46.820 --> 00:48.620]  Oh, she's here. Fantastic.
[00:49.740 --> 00:53.260]  So, the first thing that you really need to address
[00:53.260 --> 00:55.960]  when you're looking at penetration testing for background networks
[00:55.960 --> 01:03.120]  is that there's a pretty severe culture gap between ISP folks and the hacker folks,
[01:03.120 --> 01:09.520]  and there's not really similar views on disclosure or vulnerabilities or anything like that,
[01:09.520 --> 01:11.460]  and that can be kind of problematic.
[01:12.500 --> 01:16.300]  So, how many of you do work or have worked for an ISP?
[01:16.300 --> 01:17.320]  Show of hands?
[01:17.440 --> 01:20.300]  Okay, you guys have probably run into this.
[01:20.300 --> 01:24.480]  How many of you were in the routing department rather than the security department?
[01:24.480 --> 01:25.260]  Show of hands?
[01:26.000 --> 01:27.200]  Okay.
[01:28.900 --> 01:30.880]  I used to work for ISPs.
[01:30.880 --> 01:36.340]  I spent a bunch of years doing senior ISP background engineering and things like that,
[01:36.340 --> 01:40.260]  and I eventually left for a more security-focused role
[01:40.260 --> 01:44.700]  because I got really frustrated with just the differences of opinion.
[01:44.880 --> 01:48.180]  A lot of ISP engineers, when they look at vulnerabilities,
[01:48.480 --> 01:49.960]  they tend to have a very,
[01:49.960 --> 01:53.820]  oh, well, no one's going to do that sort of view of things.
[01:53.820 --> 01:57.860]  And their views on actual attacks, if they don't see it in the wild,
[01:58.060 --> 02:02.280]  a lot of times they're very unwilling to actually treat something as a real problem.
[02:02.560 --> 02:07.520]  And so it takes something like, well, Wednesday to make people sit up and take notice.
[02:08.360 --> 02:13.020]  ISP engineers, in general, tend to go for a limited tiered disclosure method,
[02:13.020 --> 02:15.660]  so when something is found, you know,
[02:15.660 --> 02:18.720]  and people like that hear first,
[02:18.720 --> 02:23.260]  and then Cisco's next favorite children hear second, and so on and so on.
[02:23.260 --> 02:26.740]  And so if you have their gear, which are a small network near the edge,
[02:26.740 --> 02:29.100]  you will hear late, if at all.
[02:29.380 --> 02:32.340]  And most ISP engineers think that this is good.
[02:32.360 --> 02:35.940]  They believe that, you know, catching the core is the most important thing,
[02:35.940 --> 02:41.000]  and then moving outwards from there will release things in a more slow fashion
[02:41.000 --> 02:43.300]  and minimize the window of vulnerability.
[02:43.300 --> 02:47.040]  I have a feeling that most of you don't think this is a good idea.
[02:48.900 --> 02:52.060]  So when you're trying to do pen testing on a backbone,
[02:52.060 --> 02:54.880]  you're going to have to work with these ISP engineers,
[02:54.880 --> 02:58.840]  and you have to sort of be a little bit gentle with them,
[02:58.840 --> 03:01.300]  because otherwise they'll immediately say,
[03:01.300 --> 03:03.140]  evil hacksaw, and stop listening.
[03:03.480 --> 03:06.320]  So for maximum efficiency and security,
[03:06.320 --> 03:09.320]  generally, when you're looking at testing a backbone,
[03:09.320 --> 03:13.360]  I generally recommend doing it in a white box fashion rather than a black box,
[03:13.360 --> 03:17.340]  because you can do all sorts of recon and data gathering and such
[03:17.340 --> 03:20.220]  when you're actually doing it in a black box fashion,
[03:20.220 --> 03:23.180]  but so many of the vulnerabilities that there are in backbone gear
[03:23.180 --> 03:25.120]  are denial of service that people get a little pissed
[03:25.120 --> 03:26.620]  when you take down their network.
[03:27.220 --> 03:30.620]  And even if you've explained this to them,
[03:30.620 --> 03:32.080]  you've signed the contracts beforehand,
[03:32.080 --> 03:33.660]  you've made sure that you're legally covered, and say,
[03:33.660 --> 03:35.740]  no, look, I could knock over your router,
[03:35.740 --> 03:39.300]  they don't know what that means a lot of the time.
[03:39.300 --> 03:42.220]  And so then they get ticked off that their backup didn't go through,
[03:42.220 --> 03:44.940]  or suddenly they can't get to their router that you knocked over
[03:44.940 --> 03:46.520]  because it's offline.
[03:46.760 --> 03:49.820]  And so you want to be really careful when you're setting up your gig
[03:49.820 --> 03:51.540]  if you're going to pentest a backbone,
[03:51.540 --> 03:54.780]  because a lot of people don't really understand their ramifications.
[03:55.160 --> 04:00.120]  So when I do this, I generally go with a white box sort of testing methodology.
[04:02.060 --> 04:05.120]  So, when you're doing it, it's just like any other pentest.
[04:05.120 --> 04:06.780]  You do your initial block of recon,
[04:06.780 --> 04:10.840]  you look at the registrars for the address blocks and the autonomous systems that they have,
[04:10.840 --> 04:12.600]  and any other data that's associated.
[04:12.600 --> 04:15.580]  You can often get contact information for social engineering,
[04:15.580 --> 04:20.640]  you can pull all sorts of information about the network that you are intending to test.
[04:21.040 --> 04:24.020]  By doing a lot of these standard pentest techniques,
[04:24.020 --> 04:26.060]  you can identify hosts on the edge,
[04:26.060 --> 04:28.620]  but looking at the other IP addresses in their net blocks
[04:28.620 --> 04:32.020]  is more likely to give you addresses that they're using on their backbone.
[04:32.020 --> 04:37.080]  Now, a lot of people are, or should be, filtering information on the backbone
[04:37.080 --> 04:39.280]  from things past the edge of their network.
[04:39.300 --> 04:42.820]  In a perfect world, you shouldn't be able to hit the routers themselves
[04:42.820 --> 04:47.480]  from outside of the network, unless you are on a management station or something like that.
[04:47.480 --> 04:50.620]  It's just traffic should go through, and you shouldn't be able to target.
[04:51.060 --> 04:54.260]  This is really, really not a perfect world.
[04:54.860 --> 04:57.400]  A lot of ISPs fail at that level of filtering,
[04:57.400 --> 05:00.800]  and we'll look at some stuff from the routing protocol security group a bit later on
[05:00.800 --> 05:02.800]  that will show you some data on that.
[05:03.580 --> 05:06.760]  It is worthwhile. There are many public route servers,
[05:06.760 --> 05:09.360]  and it's worthwhile to look on those and internet maps
[05:09.360 --> 05:12.760]  to try and determine who is connected to whom and where the peering is.
[05:12.760 --> 05:15.060]  A lot of peering happens at public exchange points.
[05:15.060 --> 05:16.420]  A lot of it happens elsewhere.
[05:16.420 --> 05:19.160]  You have private interconnects between different ISPs,
[05:19.160 --> 05:22.400]  and those will be crucial points when you're actually mapping your network.
[05:23.240 --> 05:27.680]  You can often also, on route servers, get information about their BGP routing policy
[05:27.680 --> 05:31.800]  and try and figure out what they accept and what they send to where.
[05:31.840 --> 05:34.100]  Community strings will often be viewable.
[05:34.780 --> 05:38.720]  Important information about BGP and which routes are preferred.
[05:38.740 --> 05:40.540]  And there are many public route servers over there
[05:40.540 --> 05:43.160]  that you can look at things from their perspective.
[05:43.280 --> 05:48.320]  So if the network that you're analyzing is not one that is directly connected to a route server,
[05:48.320 --> 05:50.780]  looking through two or three will give you several points of view
[05:50.780 --> 05:53.140]  into how to look at it on the net.
[05:53.720 --> 05:55.780]  Whatever you get, dump it into Google.
[05:55.780 --> 05:59.340]  And if you get good results in that, dump them right back into Google.
[05:59.340 --> 06:02.340]  And that will really help you build a profile of the network.
[06:02.620 --> 06:05.940]  I believe it's really important not to just take the network maps
[06:05.940 --> 06:08.040]  that their admins give you in Visio or what have you,
[06:08.040 --> 06:10.820]  but to also try and build your own map non-destructively.
[06:10.900 --> 06:12.680]  Their engineers aren't aware of everything.
[06:12.680 --> 06:14.520]  Often they're not aware of much.
[06:16.320 --> 06:20.900]  And that can be really important when you're looking at things from a pentester's point of view.
[06:20.980 --> 06:23.840]  If they don't tell you about a link and you don't look and you don't find it,
[06:23.840 --> 06:25.400]  you're not going to test that.
[06:26.020 --> 06:31.020]  Also, a lot of ISP engineers aren't terribly security aware.
[06:31.320 --> 06:33.440]  And so they'll post a mailing list and say,
[06:33.440 --> 06:36.320]  here's my config, gee, this isn't working guys, what do you think?
[06:36.320 --> 06:39.520]  You can get all sorts of information about what they're running,
[06:39.520 --> 06:41.540]  how they're running it, what kind of gear they have,
[06:41.540 --> 06:45.480]  and all of that can be dumped into your favorite friendly neighborhood vulnerability database
[06:45.480 --> 06:50.120]  to build a likely profile of attacks and exploits that you could use against their network.
[06:52.200 --> 06:56.940]  Now, a lot of the backbone is built on gear from a very few vendors.
[06:57.160 --> 07:01.080]  And these vendors are generally not fans of full disclosure.
[07:01.660 --> 07:07.920]  Cisco and Juniper both have on their networks things that are behind the front end of their webpage.
[07:07.920 --> 07:12.860]  You have to have a login, in many cases you have to have a contract in order to see details.
[07:12.860 --> 07:17.840]  Cisco does release some vulnerability information publicly, but there's a lot that they don't.
[07:17.840 --> 07:22.780]  Juniper releases almost nothing publicly, I think I've seen like six advisories from them ever
[07:22.780 --> 07:26.720]  that didn't require some sort of back end acknowledgement.
[07:27.000 --> 07:32.540]  And so they believe that by doing this they are helping to keep it out of the hands of script kitties.
[07:32.680 --> 07:36.800]  If you have a contract with them, or if you have a login to their site or something like that,
[07:36.800 --> 07:40.360]  you will have a much greater database of information that's available to you
[07:40.360 --> 07:43.000]  when you're trying to determine what to test for.
[07:43.700 --> 07:48.940]  We're going to talk a lot about vendor negotiation and dealing with your vendors
[07:48.940 --> 07:52.360]  in order to secure the network as best as possible.
[07:53.840 --> 07:58.700]  If you are working for a company, they may allow you to use their CCO or Juniper logins
[07:58.700 --> 08:02.060]  to access the bug databases and look for vulnerability information.
[08:02.640 --> 08:05.840]  Also, don't forget that the backbone is more than just routers.
[08:05.840 --> 08:08.820]  There are switches, there's lots of stuff that gets done on Layer 2,
[08:08.820 --> 08:10.760]  and a lot of people just kind of ignore that.
[08:10.760 --> 08:12.940]  So test through your switch gear also.
[08:17.380 --> 08:22.060]  Cisco in particular has really complex code trains.
[08:22.200 --> 08:26.180]  iOS has a zillion different branches which support different features
[08:26.180 --> 08:28.640]  and for different hardware platforms and such,
[08:28.640 --> 08:33.320]  and you will need to be aware of what you are running in the network you're testing.
[08:33.340 --> 08:38.240]  If they've left CDP on or something stupid like that, then you can get them to tell you.
[08:38.240 --> 08:41.600]  If their engineers are friendly, you can get them to tell you.
[08:41.600 --> 08:44.300]  You can log into all the routers if they're giving you that level of access
[08:44.300 --> 08:47.620]  and do a chauffeur and pull details on that.
[08:47.620 --> 08:52.560]  But something that is vulnerable in one train may or may not be implemented in other trains.
[08:53.100 --> 08:56.600]  It's worth looking with a broad brush to start off with.
[08:56.800 --> 08:59.280]  Something might be vulnerable and then going,
[08:59.280 --> 09:01.940]  oh, okay, it's not a vulnerable version is much better than saying,
[09:01.940 --> 09:05.600]  ah, well, it's only this one version, let's not test that.
[09:06.140 --> 09:10.600]  It's also worthwhile in looking at code trains and where they're derived from.
[09:10.600 --> 09:15.400]  Code is forked all the time, and sometimes you'll come up with an error that was in one version
[09:15.400 --> 09:19.600]  and years and years down the road, oh, look, it's still there.
[09:19.600 --> 09:25.020]  There was a BSD vulnerability in Telnet a while back that whacked half the stuff on the Internet,
[09:25.020 --> 09:29.800]  almost regardless of operating system, because so much was derived from that old code base.
[09:29.800 --> 09:34.060]  And I have a link here to the vulnerability in Cisco's CatOS boxes.
[09:34.060 --> 09:39.660]  We use that same bit of code and we're vulnerable to that same bit of exploit.
[09:40.180 --> 09:43.380]  There are some scanners that will incorporate these checks for you.
[09:43.380 --> 09:49.260]  I know Nessus has some Cisco vulnerability information in it, but a lot of it isn't automated yet.
[09:49.500 --> 09:52.380]  And because of the great potential for denial of service,
[09:52.380 --> 09:56.660]  you can see why a lot of players got real thrilled to have that run against them
[09:56.660 --> 10:01.040]  and why a lot of people haven't been too eager to put it in the exploit tools.
[10:01.040 --> 10:06.080]  So this is something that, I think, pen testing like 5 or 10 years ago,
[10:06.080 --> 10:10.420]  you're going to have to do a lot of this by hand in order to make sure that you've got everything.
[10:12.660 --> 10:17.280]  As with any other pen test, failure paths and trust relationships are important.
[10:17.340 --> 10:20.540]  When you look at their architecture, look for redundancy,
[10:20.540 --> 10:22.940]  look for network robustness, and look for security.
[10:23.620 --> 10:27.320]  One thing that I think is really important when you're pen testing a backbone
[10:27.320 --> 10:30.900]  is to make sure that availability is a factor.
[10:30.900 --> 10:33.660]  I mean, denial of service is such a big deal for them,
[10:33.660 --> 10:38.200]  and yet so many people build networks with one path where a link goes down and you're screwed.
[10:38.200 --> 10:41.420]  Or they have two lines terminating into one card on one router,
[10:41.420 --> 10:43.440]  and a hardware failure will cause problems.
[10:43.500 --> 10:47.100]  I think, as security analysts, it's our job to point these things out and go,
[10:47.100 --> 10:49.700]  oh, you've got a single point of failure if someone knocks this out.
[10:49.700 --> 10:53.200]  If there's an exploit, if you have a hardware failure, you have an issue.
[10:53.240 --> 10:57.300]  I personally try and include things like that when I present a report to the client.
[10:57.920 --> 11:02.780]  Also, check for the authentication means and see whether those have redundancy.
[11:02.780 --> 11:07.840]  Just as with some of the SSL versions, you know, if you can force it back to the weakest point,
[11:07.840 --> 11:09.760]  then you can exploit that.
[11:09.760 --> 11:14.280]  There was a vulnerability on a RADIUS server in Cisco's implementation of RADIUS
[11:14.280 --> 11:18.840]  where you could give it several ways of authenticating and none was an option.
[11:18.840 --> 11:21.540]  So if you can just say, oh, well, I don't want to authenticate, no.
[11:21.540 --> 11:23.620]  You know, oh, okay then.
[11:24.120 --> 11:26.040]  Things like that are kind of bad.
[11:29.500 --> 11:33.360]  So with things like that, check for what the weakest way is
[11:33.360 --> 11:38.100]  because you know your attacker is going to take the weakest way and if Pentester is still awake.
[11:38.980 --> 11:41.760]  Once you're authenticated, is trust transitive?
[11:41.760 --> 11:45.400]  You know, if I get a login to that RADIUS server, what else does that give me?
[11:45.480 --> 11:47.620]  It's important to look at that.
[11:47.620 --> 11:51.640]  And so as soon as you get every new level of access, every new point,
[11:51.640 --> 11:55.500]  if you try and map the network again from that point of view, you may find extra stuff.
[11:58.900 --> 12:03.980]  One of the things you'll see a lot and one of the ways I've had most success in testing backbone networks
[12:03.980 --> 12:06.060]  is authentication servers.
[12:06.100 --> 12:10.740]  If you have a centralized authentication server, that's a really good point to start at.
[12:10.740 --> 12:14.720]  If you look for a login to that, you can sniff traffic and get someone's login
[12:14.720 --> 12:16.700]  if they are foolish enough not to encrypt it.
[12:16.700 --> 12:19.760]  If you can own one of the hosts that they use for management stations,
[12:19.760 --> 12:24.380]  you can then grab it before it's encrypted and sent on the wire with a keystroke logger or something like it.
[12:24.460 --> 12:27.240]  And that will often give you the keys to the kingdom.
[12:27.240 --> 12:31.560]  Very many places don't think to harden their authentication servers,
[12:31.560 --> 12:35.160]  and so that's a really common point of failure.
[12:35.620 --> 12:40.320]  Another thing that I have successfully done is if they have fallback mechanisms
[12:40.320 --> 12:46.500]  where they have a local login in case their RADIUS or their CACX or their whatever server is down,
[12:46.500 --> 12:50.480]  well, if you can knock that server offline and then brute force the local password,
[12:50.480 --> 12:52.120]  that will also get you in.
[12:52.340 --> 12:56.120]  And once you brute force the local password, depending on how they're storing it,
[12:56.120 --> 13:01.680]  like for a Cisco box, they use a 7 encoding, which it's not encryption, it's encoding.
[13:01.680 --> 13:05.240]  It's symmetric, it's easily reversible, alcrypto.co.uk.
[13:05.240 --> 13:10.260]  Cisco has an online decryptor, although you may wonder where that's going.
[13:10.480 --> 13:14.620]  So they also have programs you can download and run locally.
[13:14.640 --> 13:18.180]  But a lot of the passwords that are saved in the configs,
[13:18.180 --> 13:21.040]  you can just throw them in the internet, get the password out,
[13:21.040 --> 13:22.920]  and they may be using that in multiple places.
[13:22.920 --> 13:26.320]  But a lot of places do not do password management well,
[13:26.320 --> 13:29.860]  and so a local login at one place will get you a local login at another place.
[13:29.860 --> 13:33.780]  And as long as that central authentication server is down, there you go.
[13:34.600 --> 13:39.980]  Obviously, once you've logged into the boxes, if you have enabled, you can change that.
[13:40.620 --> 13:44.220]  But that will let them know you're there. You know, please be nice.
[13:47.700 --> 13:53.880]  Also, man-in-the-middling against the authentication server is usually a fairly successful technique,
[13:53.880 --> 13:56.400]  if you can get access to do that.
[13:59.760 --> 14:03.600]  It shouldn't need mentioning, but physical security is really important.
[14:03.600 --> 14:07.480]  A lot of places forget this. Everywhere you have a data center,
[14:07.480 --> 14:12.060]  and especially everywhere you're peering, please have good physical security.
[14:12.280 --> 14:15.420]  Please audit and make sure you have good physical security.
[14:15.420 --> 14:20.380]  Please don't let me walk into Macy's with a switch under my arm and a laptop under the other one and plug in.
[14:20.380 --> 14:21.780]  Yeah, bad.
[14:23.180 --> 14:29.100]  A lot of the time, if you have local access, local physical access, that will get you a lot.
[14:29.340 --> 14:36.820]  And many of the places, they have a very clear profile of what they're looking for.
[14:36.980 --> 14:40.680]  So, you know, if I was 40 and had a beard and came in,
[14:40.680 --> 14:46.400]  they might be a little more suspicious of me than if I'm 29 and got my eyelashes and go,
[14:46.400 --> 14:48.740]  gee, I don't know, I'm just supposed to go look at this router.
[14:49.160 --> 14:54.220]  They have poorly placed trust relationships in terms of local access,
[14:54.220 --> 14:58.600]  and if you want to take a social engineering approach to that, then that can work to your advantage.
[14:59.240 --> 15:02.040]  If you don't have physical security, you're screwed.
[15:04.650 --> 15:08.490]  One of the things that I've seen fairly often that's kind of appalling
[15:08.490 --> 15:14.330]  is how many people will leak data about the processes of the core of their network to the edge.
[15:14.570 --> 15:18.290]  So they'll have EIGRP as their internal routing protocol,
[15:18.290 --> 15:22.410]  and they won't turn it off for interfaces that face out towards the edge of their network.
[15:22.410 --> 15:28.190]  They just sort of send those multicast, hey, hey, who wants to be my neighbor, out to anyone.
[15:28.210 --> 15:32.550]  And, well, you shouldn't let people at the edge of your network establish a neighbor relationship
[15:32.550 --> 15:36.270]  and feed you a bunch of route data, which may be bad.
[15:37.370 --> 15:42.810]  Also, CDP. People will leave it on by default, they won't turn it off,
[15:42.810 --> 15:46.990]  and it will happily send you all kinds of information about the connected device.
[15:46.990 --> 15:50.370]  It will tell you what it is, what version of the operating system it's running,
[15:50.370 --> 15:54.190]  what sort of hardware it has, device ID, all kinds of stuff.
[15:54.550 --> 15:59.530]  And Cisco has designed this so that it's very useful when you're troubleshooting.
[15:59.630 --> 16:03.610]  I have had cases where I wanted to see what was on the other end of the link,
[16:03.610 --> 16:09.930]  and we weren't sure that the person at the peering site or the central office had us plugged into the right ports.
[16:10.010 --> 16:16.290]  So the other side turns on CDP, and you go, whoa, I'm not supposed to be connected to a 7206 VXR.
[16:16.290 --> 16:17.670]  Yeah, that's wrong.
[16:17.670 --> 16:19.830]  So when troubleshooting, it's a good tool.
[16:19.830 --> 16:23.070]  But when you're not troubleshooting, turn it off.
[16:24.010 --> 16:28.950]  Switching data. Also, switch management protocols, things like that, will be happily sent out.
[16:28.950 --> 16:32.150]  Hey, welcome to my spanning tree, who wants to be my root bridge?
[16:32.150 --> 16:36.170]  These things should not be seen by the end users, and they often are.
[16:36.170 --> 16:38.470]  So that's something for you pen testers to look for.
[16:40.010 --> 16:43.150]  If you can see the data, you can send things back.
[16:43.150 --> 16:49.450]  You can spoof packets, you can send real data, you can feed them information on who's supposed to be bridging to what,
[16:49.450 --> 16:51.210]  or who's supposed to be routing to whom.
[16:51.210 --> 16:57.270]  If you can get a more preferred route and inject it, then that will, depending on the implementation, of course,
[16:57.270 --> 17:02.570]  it may be able to take over the route that's actually on the legitimate internet.
[17:03.690 --> 17:08.670]  A lot of this is... when you're speaking internal routing protocols rather than BGP,
[17:08.670 --> 17:11.130]  it's a lot easier to deal with on the local net.
[17:11.130 --> 17:14.990]  BGP has its own separate set of rules because of how it's handled between providers
[17:14.990 --> 17:17.730]  and how you set up neighbor and peering relationships.
[17:17.910 --> 17:21.570]  But on the local network, it's often not that hard.
[17:21.570 --> 17:28.010]  If you want to do a really sort of not high-skilled but very effective DOS attack,
[17:28.010 --> 17:30.030]  you can tear down links with spoof packets.
[17:30.030 --> 17:33.010]  A lot of the time, they don't have authentication on there,
[17:33.010 --> 17:39.790]  and so many routing protocols support the capacity for even something as simple as an mb5 hash.
[17:39.870 --> 17:44.450]  It's often called BGP password, although many other protocols also support the same thing.
[17:44.770 --> 17:50.410]  I know you guys have heard about mb5 fairly recently, but, you know, it's still a whole lot better than nothing.
[17:55.260 --> 17:59.220]  Similar to the theory of rogue access points, if you have physical access to the site,
[17:59.220 --> 18:00.760]  you can just walk right in with a router and say,
[18:00.760 --> 18:04.840]  yeah, I'm supposed to install this, and some places aren't very good about saying,
[18:04.840 --> 18:08.300]  who the hell are you, why didn't I know you were coming, you know, go home.
[18:08.620 --> 18:11.840]  And you can plug in a router, you can put it into the peering switch,
[18:11.840 --> 18:14.280]  you can just, you know, start snipping traffic.
[18:14.280 --> 18:18.320]  If you can manage to get the switch ports set promiscuous or something like that,
[18:18.320 --> 18:21.100]  then you can see what they're speaking and start to speak it too.
[18:21.440 --> 18:23.280]  This also is obviously bad.
[18:24.860 --> 18:30.040]  Depending on what your contract is with the client, you can even go in there and say,
[18:30.040 --> 18:33.960]  oh, yeah, you know, I'm from this company, I'm supposed to work on their router,
[18:33.960 --> 18:37.000]  yank out theirs, put in yours with the configuration of whatever you want,
[18:37.000 --> 18:38.920]  and, hey, as long as you pass it through to them,
[18:38.920 --> 18:41.900]  they may not even notice except for the ten minutes of reboot time.
[18:44.410 --> 18:48.330]  A lot of places do not have good passwords.
[18:48.370 --> 18:50.570]  It's better than it used to be because of spammers.
[18:50.570 --> 18:53.630]  There have been cases of spammers taking over Cisco routers
[18:53.630 --> 18:58.890]  and then using the telnet function to send spam out of the router itself one character at a time.
[18:59.030 --> 19:01.570]  So, yeah, bad.
[19:02.050 --> 19:08.310]  But a lot of places have gotten better about not having, at least pathetically, default passwords.
[19:08.690 --> 19:11.510]  However, if I get called in and someone's locked out of their router,
[19:11.510 --> 19:14.970]  I always give Cisco Cisco a try, you know, maybe one time in four it works,
[19:14.970 --> 19:17.490]  and just, it shouldn't.
[19:18.490 --> 19:21.710]  You can redirect traffic through a GRE tunnel.
[19:21.710 --> 19:26.910]  I've seen that in the wild, and people were a little bit confused about what was going on there,
[19:26.910 --> 19:30.470]  and said, I don't remember setting that up. Bob, did you set that up?
[19:30.470 --> 19:37.350]  And meanwhile, it's terminating in, you know, Moldavia or somewhere, and there go all your packets.
[19:38.990 --> 19:44.610]  Another serious concern is that, especially when you're speaking BGP,
[19:44.610 --> 19:48.350]  you may be able to advertise additional net blocks out to the net,
[19:48.350 --> 19:53.790]  and if your route is preferred to other people's routes, you've just hijacked a net block.
[19:54.170 --> 19:56.750]  And this happened.
[19:56.850 --> 20:01.850]  There was a bunch of attackers that set up a site through an ISP that wasn't doing good security checks
[20:01.850 --> 20:03.790]  about who actually owned that IP space.
[20:03.790 --> 20:08.490]  They announced this block out to the world, and their route was preferred in most places
[20:08.490 --> 20:11.950]  because of the way that they did it and how their provider was set up,
[20:11.950 --> 20:15.970]  and they happily spent about a week sending out spam and sending out worms
[20:15.970 --> 20:19.090]  instead of a botnet, and were owning boxes left and right.
[20:19.090 --> 20:23.510]  Meanwhile, the legitimate users of that net block can't even get online because no one's routing to them.
[20:23.510 --> 20:26.370]  Everyone's routing to the own site.
[20:26.450 --> 20:31.050]  And so, when that was finally resolved at the ISP level,
[20:31.050 --> 20:34.710]  then the original owners are really left holding the bag there.
[20:34.710 --> 20:37.030]  Hi, you're on every blacklist on the planet,
[20:37.030 --> 20:40.590]  you have a whole bunch of angry people pounding down the door at your abuse desk,
[20:40.590 --> 20:42.530]  and you were offline for a week.
[20:42.530 --> 20:46.910]  And because some ISPs do not do good checking when they're hooking up a new client
[20:46.910 --> 20:49.790]  of do you really own this block, is this legitimate,
[20:49.790 --> 20:52.110]  that can happen, and it shouldn't.
[20:52.110 --> 20:54.630]  There are some really good people working ISP security
[20:54.630 --> 20:58.490]  that are cooperating between ISPs to try and prevent that.
[20:58.610 --> 21:01.430]  And, you know, more power to them. We need more of them.
[21:04.290 --> 21:09.210]  A great place to start when looking at the actual configurations of the backbone devices
[21:09.210 --> 21:12.810]  is to look at the secure BGP template,
[21:12.810 --> 21:15.650]  iOS template, Junos template, and Junos BGP template
[21:15.650 --> 21:17.830]  that the Cymru people do.
[21:17.830 --> 21:20.690]  These are very, very well done documents.
[21:20.690 --> 21:23.930]  They have bogon lists so that you cannot accept routes
[21:23.930 --> 21:27.490]  from IP addresses that haven't been allocated or don't exist.
[21:27.490 --> 21:30.290]  They do strong filtering so that you don't have, like, you know,
[21:30.290 --> 21:33.130]  RFC 1918 addresses speaking to you,
[21:33.130 --> 21:35.250]  you shouldn't allow people to advertise, you know,
[21:35.250 --> 21:38.230]  your own net blocks to you, for example, things like that.
[21:38.230 --> 21:42.310]  They have good checking for the strongest possible authentication methods.
[21:42.310 --> 21:45.910]  They try and turn off things like, oh, hey, let's speak finger,
[21:45.910 --> 21:49.650]  you know, ancient insecure daemons that you have no business really needing
[21:49.650 --> 21:51.730]  to run on your major backbone routers.
[21:51.730 --> 21:54.610]  I can't recommend these enough. Those guys do really good work,
[21:54.610 --> 21:58.650]  and it's also handy for a beginning place to start with an engagement
[21:58.650 --> 22:01.330]  if you're doing a white box design analysis.
[22:03.010 --> 22:05.590]  Peering security is particularly important.
[22:05.910 --> 22:07.670]  When you look at how someone is peering,
[22:07.670 --> 22:10.070]  look at the routes they're willing to accept and look at the routes
[22:10.070 --> 22:12.050]  that they're willing to send out.
[22:12.410 --> 22:15.970]  There was a case several years ago where someone actually advertised
[22:15.970 --> 22:20.870]  10.0.0.0 to the whole internet and claimed that they owned it.
[22:20.870 --> 22:23.910]  And that got a lot of people a little bit irate.
[22:24.130 --> 22:27.090]  It broke a lot of people's private network implementations,
[22:27.090 --> 22:29.630]  and it broke a lot of VPN stuff.
[22:30.530 --> 22:32.190]  So that's worth checking.
[22:32.310 --> 22:34.390]  Again, route servers are your friend.
[22:34.390 --> 22:38.250]  And when you do peering changes, make sure you're doing good authentication.
[22:38.250 --> 22:40.530]  You don't want just anyone to call you up and say,
[22:40.530 --> 22:45.870]  Hi, I'm Raven from Uunet, and yeah, I want you to change this config.
[22:45.870 --> 22:49.190]  Can you take out this net block of this guy who, you know,
[22:49.190 --> 22:50.930]  stole my girlfriend last week?
[22:51.430 --> 22:54.850]  You want actual passwords, some sort of validation method
[22:54.850 --> 22:57.630]  to make sure the person who you're talking to to make these major
[22:57.630 --> 23:01.450]  affecting changes actually is the person that they claim to be.
[23:02.030 --> 23:05.670]  Also, make sure that the machines that you use for your network policy
[23:05.670 --> 23:07.710]  and your network admin are well secured.
[23:07.730 --> 23:11.150]  I have worked at a job where we had a couple of management stations,
[23:11.150 --> 23:14.310]  and they were Windows boxes, and wow, they got owned.
[23:15.090 --> 23:16.750]  That's really dumb.
[23:16.750 --> 23:19.510]  If you have only a few machines that are allowed to access
[23:19.510 --> 23:22.490]  these crucial backbone devices, make sure they're hardened.
[23:22.490 --> 23:23.490]  Make sure they're patched.
[23:23.490 --> 23:26.450]  Make sure that people can't get onto them and, you know...
[23:28.370 --> 23:30.970]  In that case, I think what happened was
[23:30.970 --> 23:33.530]  these machines that we did backbone stuff from
[23:33.530 --> 23:36.170]  were the same machines that we did abuse tests from,
[23:36.170 --> 23:38.330]  and so you're investigating an abuse case,
[23:38.330 --> 23:40.930]  and oh, our user's surfing porn, go to the porn site,
[23:40.930 --> 23:45.010]  porn site's full of malware, mandated corporate IE, boom.
[23:45.390 --> 23:46.890]  Don't let that happen.
[23:49.030 --> 23:51.710]  And filtering is also very important.
[23:51.710 --> 23:54.930]  Not just what you allow to your router for management traffic,
[23:54.930 --> 23:57.390]  but also what you allow as far as routes.
[23:57.710 --> 24:00.370]  You want to make sure that you do not allow
[24:00.370 --> 24:02.790]  internal routes that should stay internal to your network
[24:02.790 --> 24:03.890]  outside to the world.
[24:03.890 --> 24:07.750]  Yes, aggregation happens, but more precise routes are often preferred,
[24:07.750 --> 24:08.690]  and that can be a problem.
[24:08.690 --> 24:10.830]  It bloats the routing tables, it uses up memory,
[24:10.830 --> 24:15.370]  and it can be revealing more details about the interior of your network
[24:15.370 --> 24:16.510]  than you wanted.
[24:17.090 --> 24:21.150]  The RPSec Working Group did a small survey of people in the field,
[24:21.150 --> 24:24.210]  and they surveyed people who were peering at an exchange point.
[24:24.210 --> 24:29.070]  So these are some of the more aware, more security conscious people out there.
[24:29.170 --> 24:32.090]  And they still weren't using everything that was available.
[24:32.710 --> 24:37.390]  This may be because, for lack of a better term, ISP culture.
[24:38.090 --> 24:42.150]  They view change management in the time that it takes to do good password control
[24:42.150 --> 24:47.950]  and to change all these policies as more of a threat than the security implications.
[24:48.110 --> 24:50.810]  And so you've got to keep track of who's allowed to do what,
[24:50.810 --> 24:52.270]  and who's filtering where,
[24:52.270 --> 24:54.670]  and, you know, oh, we screwed up our peering list,
[24:54.670 --> 24:57.630]  and, you know, okay, now there's downtime because I wasn't allowing your route,
[24:57.630 --> 24:59.250]  or I had too many prefixes.
[24:59.290 --> 25:02.090]  And so, you know, they think that the hassle of management
[25:02.090 --> 25:05.070]  is often not worth the additional security.
[25:05.390 --> 25:08.750]  And from an ISP point of view, I can see why they take that approach.
[25:08.750 --> 25:10.110]  When you're driven by the bottom line,
[25:10.110 --> 25:13.110]  and you're a network engineer with too much work and too little time,
[25:13.110 --> 25:15.490]  it can be difficult to deal with these sorts of things.
[25:15.630 --> 25:18.410]  Nevertheless, I think this is going to become increasingly important
[25:18.410 --> 25:22.410]  as more attention focuses on the backbone as, you know, started this week.
[25:24.050 --> 25:25.790]  The RPSEC site security.
[25:26.630 --> 25:31.550]  Many people are actually doing an incoming prefix filter for each BGP customer,
[25:31.550 --> 25:32.110]  which is good.
[25:32.110 --> 25:35.070]  You should only allow people to advertise their own netblocks to you.
[25:35.070 --> 25:37.130]  You don't want them to start advertising your netblocks,
[25:37.130 --> 25:38.490]  or, you know, someone else's netblocks,
[25:38.490 --> 25:40.470]  or that guy who stole your girlfriend's netblocks.
[25:40.470 --> 25:43.030]  That can get you in legal hot water.
[25:43.030 --> 25:47.810]  A lot of them also are checking for incoming autonomous system paths.
[25:47.890 --> 25:51.750]  A lot of people use community strings in order to control this sort of thing.
[25:51.830 --> 25:55.490]  However, very few people are using an outgoing AS path filter,
[25:55.490 --> 25:59.450]  and in that case, it really depends on who you are.
[25:59.470 --> 26:02.250]  If you are a small ISP near the edge of the network,
[26:02.250 --> 26:06.290]  you have your AS, and you have a few others that are your customers,
[26:06.290 --> 26:08.110]  that may actually be a good idea.
[26:08.130 --> 26:12.410]  If you are a tier 1, and you have every AS in the world flowing through you,
[26:12.410 --> 26:14.910]  that can be a real change management challenge,
[26:14.910 --> 26:17.070]  and probably operationally unwise.
[26:18.910 --> 26:20.850]  Outgoing updates towards your peers.
[26:20.890 --> 26:23.670]  A lot of people use communities on that, but not everyone.
[26:23.730 --> 26:27.550]  And there is the URL here for the actual survey,
[26:27.550 --> 26:30.050]  if you want to look at the results from the RPSEC group.
[26:30.050 --> 26:31.250]  I didn't put them all up.
[26:32.450 --> 26:35.550]  So, when you're looking at actually doing a pentest,
[26:35.550 --> 26:37.430]  where do you get your backbone vulnerabilities?
[26:37.890 --> 26:39.490]  They're not all in the scanning tools.
[26:39.490 --> 26:40.790]  We've established that earlier.
[26:40.790 --> 26:44.950]  And Cisco and Juniper may have you log into their site to actually look at it.
[26:45.070 --> 26:47.730]  Mailing lists will have public vulnerability announcements.
[26:47.790 --> 26:51.990]  I've often found that it is a good idea to check the mailing list against the vendor site,
[26:51.990 --> 26:55.010]  because, hey, people change their webpages all the time.
[26:55.030 --> 26:56.950]  Mailing list archives are usually more static.
[26:56.950 --> 26:59.750]  You can at least see what the first release was of something.
[26:59.810 --> 27:04.570]  We saw this happen this week, when Cisco announced their IPv6 vulnerability.
[27:04.790 --> 27:07.310]  And if you look on their website now, it's 1.1.
[27:07.310 --> 27:09.690]  The original one that went full disclosure is 1.0.
[27:09.690 --> 27:11.230]  There have been some changes made.
[27:11.830 --> 27:15.110]  And it is valuable to Pentester to be able to see both versions.
[27:15.370 --> 27:22.990]  There's been, in the last five years, increasing interest in this field by hackers who don't come from an ISP background.
[27:23.570 --> 27:27.930]  FX and Paul Watson and Mike Lynn have all made valuable and important contributions
[27:27.930 --> 27:31.350]  that have really caused a bit of uproar in the ISP community.
[27:31.570 --> 27:35.770]  And after this week's uproar, I think there will be a whole lot more people looking for it.
[27:39.140 --> 27:44.700]  So, FX has done a lot of stuff in this field that is really useful.
[27:45.040 --> 27:49.520]  There is a EIGRP Spoof Packet Neighbor Saturation bug,
[27:49.520 --> 27:55.500]  where essentially, if you send it more than 256 neighbors, then it just falls over.
[27:56.540 --> 27:59.140]  EIGRP works by sending out multitasks.
[27:59.140 --> 28:01.240]  Hey, here I am! Who wants to be my neighbor?
[28:01.360 --> 28:04.320]  And so if you say, me, and me, and me, and me, and me,
[28:04.320 --> 28:08.640]  you can essentially overflow the amount of memory that it allocates for that,
[28:08.640 --> 28:12.700]  and, you know, it's at least a denial of service.
[28:12.740 --> 28:14.000]  It could be worse.
[28:14.520 --> 28:17.680]  Similarly, for the CBP neighbor announcement,
[28:17.680 --> 28:22.880]  if you send Cisco Discovery Protocol packets with device IDs that are really, really long,
[28:22.880 --> 28:26.980]  you can fill up the memory tables and possibly do evil after that.
[28:27.100 --> 28:31.200]  There's been GRE attacks, a TFTP server advisory, and a bunch of other stuff.
[28:31.200 --> 28:35.660]  If you look at FX's site on Phenolate, there's tons of stuff.
[28:35.660 --> 28:40.540]  There's actually a tool for pen testing that was released specifically to address these kinds of things.
[28:40.540 --> 28:47.240]  It only addresses the vulnerabilities that FX actually found, but it is well worth looking at.
[28:47.860 --> 28:53.460]  There's also been a really interesting paper about Cisco memory mapping,
[28:53.460 --> 29:00.200]  and I'm sure that any of you that were at Black Hat on Wednesday saw similar work in Mike Lynn's talk.
[29:00.200 --> 29:03.880]  The memory structures are still the same, the magic numbers are still the same,
[29:03.880 --> 29:08.820]  they're all online, and it is very useful if you want to dig into the assembler.
[29:11.090 --> 29:15.890]  Then about a year ago, any of you that were at CanSec probably heard the storm and fury
[29:15.890 --> 29:18.570]  surrounding Paul Watson and the TCP window.
[29:18.850 --> 29:22.210]  Essentially, TCP stacks on background equipment and many other things
[29:22.210 --> 29:26.310]  were not correctly looking at the window size and anything that hit spot on.
[29:26.310 --> 29:28.890]  Just anything in the general neighborhood was good enough.
[29:28.890 --> 29:31.790]  So if you send a reset that's in the general neighborhood,
[29:31.790 --> 29:36.030]  and you're able to correctly guess the IP addresses and port numbers on either side,
[29:36.030 --> 29:38.310]  you can knock down the session with a spoof packet.
[29:38.830 --> 29:46.510]  And this allowed unauthenticated BGP sessions, if you could guess it all correctly, to be knocked down.
[29:46.910 --> 29:51.530]  And this was really interesting when you look at disclosure and how the whole thing went out,
[29:51.530 --> 29:57.230]  because essentially what happened was when Cisco started notifying people of this,
[29:57.230 --> 30:01.810]  because Watson worked with Cisco and said, hey, look, we need to fix this.
[30:02.110 --> 30:07.130]  Many people on the backbone and in NANOG and similar ISP communities
[30:07.130 --> 30:10.050]  had known about this for years. They just didn't think it was very possible.
[30:10.150 --> 30:14.570]  And again, we see that attitude of, well, I'm really busy unless I see something in the wild.
[30:14.570 --> 30:16.490]  It's not a problem. It's not going to happen.
[30:16.950 --> 30:23.890]  And so Cisco said, okay, everyone, you need to put md5 authentication passwords on your BGP sessions now.
[30:23.890 --> 30:27.690]  Okay, you need to upgrade now. They didn't tell people why.
[30:28.090 --> 30:31.170]  And so they told their favorite people first, and then their second favorite people,
[30:31.170 --> 30:33.410]  and their third favorite people, and so on out.
[30:33.450 --> 30:39.150]  And so it took, I think, the better part of a month for it to actually get heard by most people.
[30:39.150 --> 30:41.230]  And then Watson gave his CanSec talk.
[30:42.050 --> 30:45.590]  Once the ISP operators found out what the actual flaw was,
[30:45.590 --> 30:49.650]  there was a very strong cultural backlash saying, oh, I knew all about that.
[30:49.650 --> 30:52.430]  You know, why are you making such a big deal?
[30:52.430 --> 30:58.550]  And I was really depressed by going to NANOG this year, where a lot of operators were like, oh, yeah, you know,
[30:58.550 --> 31:01.490]  we thought that was important, but wow, we really freaked out.
[31:01.490 --> 31:04.650]  And I've rolled back. I don't have any authentication on my sessions anymore.
[31:04.650 --> 31:06.030]  You know, stupid hackers.
[31:06.630 --> 31:09.430]  And that was upsetting.
[31:09.490 --> 31:17.170]  Because I really saw the community response to the Watson vulnerability as a strong positive step in securing the backbone.
[31:17.430 --> 31:21.430]  And then seeing everyone go, wow, we freaked out over nothing, did not help.
[31:21.430 --> 31:27.430]  I think Cisco's approach of we're not telling you why, just patch it because we say so, no really, just trust us,
[31:27.970 --> 31:31.850]  I think they hurt themselves because people trust them less than they used to.
[31:34.550 --> 31:37.890]  And then we come to Wednesday and what made me rewrite my slides.
[31:39.070 --> 31:43.210]  If you were at Black Hat and saw Michael in session on Wednesday morning,
[31:43.210 --> 31:48.470]  he essentially demonstrated what I view as a remote exploit in Cisco IOS.
[31:48.470 --> 31:53.230]  Cisco has announced in at least one of their vulnerability announcements that it was just a local.
[31:54.010 --> 31:57.570]  I think we have different meanings for the same word there.
[32:00.510 --> 32:08.730]  And again, the disclosure process here was important because when Mike gave his talk,
[32:08.730 --> 32:12.970]  he essentially talked about the internals of how it worked.
[32:12.970 --> 32:18.770]  That stack overflows are hard to find in Cisco IOS, that the code base looked strongly audited,
[32:18.770 --> 32:23.310]  and that you had to work with heap overflows, but that Cisco has a check-heaps process,
[32:23.310 --> 32:26.890]  which will go through and validate, make sure the previous and next values are the same,
[32:26.890 --> 32:31.750]  so that if you actually do corrupt the heap, then it will say, oh, this isn't okay,
[32:31.750 --> 32:34.450]  all right, I'm crashing the router because this is screwed up.
[32:35.710 --> 32:39.830]  He did not give details of what his entry vector was.
[32:39.830 --> 32:46.030]  He was very careful to basically try and not give enough information for people to recreate it.
[32:46.330 --> 32:50.970]  Now, even though the vulnerability has been patched and the old versions have been yanked off of Cisco's website,
[32:50.970 --> 32:53.470]  there are still vulnerable versions out there in the wild.
[32:54.470 --> 33:00.090]  Cisco, really shockingly to me, yesterday morning released an advisory to full disclosure that said,
[33:00.090 --> 33:05.550]  yeah, for the first time ever, we have a denial of service and possible remote code execution
[33:06.370 --> 33:08.790]  in our handling of IPv6 packets.
[33:10.330 --> 33:13.750]  Lynn didn't tell us that. Cisco told us that.
[33:14.210 --> 33:19.250]  In the latest version of their advisory, they say that it was disclosed at Black Hat,
[33:19.250 --> 33:22.950]  so that really does nail it down to, yes, this was the vulnerability.
[33:23.970 --> 33:27.470]  And this is really revolutionary for the field, because for the first time,
[33:27.470 --> 33:29.890]  it looks like you can remotely own a Cisco box.
[33:29.890 --> 33:35.810]  You can get it to shovel you back a shell that will allow you to have full enable access.
[33:36.090 --> 33:38.810]  And this is a scary thing if you're an ISP operator.
[33:40.090 --> 33:45.170]  I think that Lynn was pretty sensible in saying that people need to treat these like normal computers,
[33:45.170 --> 33:47.170]  they need to patch, they need to be aware of security,
[33:47.170 --> 33:50.370]  and they need to be diligent about making sure that they patch through the versions,
[33:50.370 --> 33:53.930]  because this is a real threat, people. We've now seen the remote route.
[33:55.210 --> 34:00.030]  Now, when you look at the Cisco advisory, what they appear to mean by local,
[34:00.030 --> 34:03.150]  to the best of my understanding, and, you know, disclaimer,
[34:03.150 --> 34:07.090]  I don't and never have worked for Cisco, I don't and never have worked for ISS,
[34:07.090 --> 34:14.310]  I don't speak for anyone but me, but Cisco claims that you have to be connected to the same local subnet
[34:14.310 --> 34:17.610]  in order for that to work, and that makes it a local exploit, right?
[34:19.450 --> 34:20.330]  Yeah.
[34:21.670 --> 34:27.590]  No, if you have to be on the same subnet from one remote box to another remote box, that's a remote route.
[34:30.560 --> 34:32.380]  And then came the cover-up.
[34:33.580 --> 34:37.920]  The talk's been ripped out of the books, the presentation's not on the CD,
[34:37.920 --> 34:40.900]  everyone got modified CDs, the slides aren't available,
[34:40.900 --> 34:43.700]  oh my god, there's suing, and there's suing, and there's the FBI,
[34:43.700 --> 34:47.460]  and there's restraining orders, and rumors, and news stories, and all sorts of stuff,
[34:47.460 --> 34:50.140]  and Cisco, you're really screwing up here.
[34:50.840 --> 35:09.730]  You know, to give you an idea of the attitude of the security community,
[35:09.730 --> 35:12.650]  there are people walking around wearing these.
[35:19.820 --> 35:26.100]  I think that gives you an idea of the mentality of what happens when you try and cover something up.
[35:26.320 --> 35:28.500]  Okay, so you're not really happy.
[35:28.680 --> 35:29.640]  Big pardon?
[35:30.880 --> 35:32.740]  It said, fuck Cisco.
[35:34.420 --> 35:36.260]  Let the record reflect.
[35:41.360 --> 35:46.360]  And I really think that Mike Lynn was right on when he talked about why he gave the speech,
[35:46.360 --> 35:49.060]  and why he released as much information as he did.
[35:49.060 --> 35:54.700]  I mean, okay, the reason he got onto it in the first place, again, from the best of my understanding,
[35:54.700 --> 35:57.940]  I don't know the guy, I don't work for anyone, yada yada.
[35:58.380 --> 36:01.720]  He said that he found some Mandarin Chinese documents,
[36:01.720 --> 36:05.900]  which when translated were talking about how to hook debugger hooks in iOS,
[36:05.900 --> 36:09.060]  and how to make those speak to your disassembler and your debuggers.
[36:09.440 --> 36:11.300]  This means people are working on it.
[36:11.300 --> 36:16.860]  Their source code has been stolen, and there's now a storm of interest.
[36:16.860 --> 36:21.820]  I know that there have been several people who have suddenly been diving into their assembler
[36:21.820 --> 36:24.420]  to attempt to figure out what's going on.
[36:24.960 --> 36:27.640]  Hiding your head in the sand is not going to help.
[36:28.080 --> 36:31.620]  If you have people who are attacking you that are actively looking for this,
[36:31.620 --> 36:35.760]  going, no, no, it doesn't happen, quick, shut everyone up, is not going to help.
[36:35.860 --> 36:40.720]  And even if you destroy all the slides, and you aren't going to let things get released,
[36:40.720 --> 36:43.520]  well, his slides are out there on the internet.
[36:43.520 --> 36:46.960]  Suing researchers is not going to achieve security.
[36:46.960 --> 36:48.840]  I mean, only speaking for myself.
[36:49.300 --> 36:50.920]  I was in his talk.
[36:51.340 --> 36:52.680]  I speak assembler.
[36:52.680 --> 36:54.140]  I know Cisco iOS.
[36:54.920 --> 36:58.200]  With enough work, I could probably recreate his exploit.
[36:58.540 --> 37:00.840]  I'm sure I'm not the only one.
[37:01.160 --> 37:04.280]  Trying to pretend it won't happen will not fix things.
[37:04.380 --> 37:06.960]  And alienating the security community,
[37:06.960 --> 37:11.180]  and making everyone think that you're going to be sued into oblivion if you ever speak up,
[37:11.180 --> 37:15.760]  that's not going to encourage people to come to you and say, hey, we've got a problem, let's work with you.
[37:15.920 --> 37:19.660]  When I was reading the reports of extant backbone vulnerabilities,
[37:19.660 --> 37:23.040]  it sounded like they treated FX a lot nicer than they treated Mike Lynn.
[37:23.040 --> 37:25.900]  FX actually thanks people on the Fino Elite Advisories,
[37:25.900 --> 37:29.660]  and says, hey, thanks for working with me, Cisco P-Cert, you guys were really great.
[37:30.440 --> 37:32.400]  They don't sound really great right now.
[37:33.420 --> 37:37.660]  They filed a temporary restraining order to prevent him from speaking about it.
[37:37.660 --> 37:40.480]  So, he can't talk about it, but I can.
[37:53.960 --> 37:57.400]  Well, his slides are easily findable on the internet,
[37:57.400 --> 38:02.040]  and not even the version that he presented at the talk.
[38:02.040 --> 38:05.300]  The original slides from before he got fired by his employer,
[38:05.300 --> 38:10.480]  I think like yesterday morning they were up on CryptoMay and several other sites,
[38:10.480 --> 38:13.480]  and this has just been a ridiculous media circus.
[38:14.020 --> 38:16.440]  They were put up on Infowarrior.org,
[38:16.440 --> 38:19.100]  and they got pulled down with a cease and desist by Cisco,
[38:19.100 --> 38:23.020]  and guys, just help us fix this.
[38:23.560 --> 38:26.260]  The CD for that was from ISS, apparently.
[38:26.760 --> 38:30.920]  So, he had to return his computer, he's not allowed to talk anymore,
[38:30.920 --> 38:33.320]  and this is really dumb.
[38:34.540 --> 38:37.160]  Dear Cisco and ISS,
[38:38.020 --> 38:40.000]  Responsible disclosure is good.
[38:40.400 --> 38:43.440]  Working with security researchers is good.
[38:43.440 --> 38:45.700]  Trying to cover up things that are known problems
[38:45.700 --> 38:47.420]  that have been released into the public domain
[38:47.420 --> 38:52.600]  that are on millions and millions of machines all over the internet now is good.
[38:52.820 --> 38:54.640]  If you try and shut people up,
[38:54.640 --> 38:57.940]  you're only going to alienate the very people that you need to help you.
[38:59.780 --> 39:00.660]  Duh.
[39:05.140 --> 39:08.220]  Alright, so, what Mike Lynn actually said.
[39:09.180 --> 39:11.320]  Heap overflows are way more common in iOS.
[39:11.320 --> 39:13.620]  He said a bunch of nice things about their code base,
[39:13.620 --> 39:18.800]  but he said that the watchdog process of check heaps will check for heap corruption,
[39:18.800 --> 39:20.600]  and if it finds it, it will crash it.
[39:20.840 --> 39:23.660]  And most of you that have worked with Cisco routers I'm sure will see
[39:23.660 --> 39:27.100]  that if you have a Cisco router and it has a fault,
[39:27.260 --> 39:29.900]  a lot of times, rather than actually trying to recover,
[39:29.900 --> 39:31.600]  it will just crash itself.
[39:31.840 --> 39:35.560]  And the really important thing that I think came out of Lynn's research
[39:35.560 --> 39:38.620]  was that the crash process itself is interruptible.
[39:38.620 --> 39:40.680]  So, you can interrupt it when it says,
[39:40.680 --> 39:45.400]  Hey, this bad thing has happened. My heap has gone badly. Am I crashing?
[39:46.080 --> 39:47.960]  And you can tell it,
[39:47.960 --> 39:49.200]  Why, yes I am.
[39:49.200 --> 39:51.880]  Okay, we're already crashing. I don't need to do anything then.
[39:51.880 --> 39:55.100]  Please continue executing whatever the heck it is you were executing there.
[39:55.780 --> 39:57.800]  So, you can buy yourself some time that way.
[39:57.800 --> 40:01.800]  He did mention that spreading memory corruption on the heap was going to be an issue,
[40:01.800 --> 40:07.220]  and that you might have two to five minutes before it exploded all over your memory, basically,
[40:07.220 --> 40:09.440]  unless you actually handle that in your shellcode,
[40:09.440 --> 40:12.400]  and anyone who's written shellcode knows that that can be tricky.
[40:13.200 --> 40:15.140]  But once you've actually gained execution,
[40:15.140 --> 40:17.300]  there are hardware interrupts that will try and interrupt you,
[40:17.300 --> 40:18.540]  but you can disable those.
[40:18.540 --> 40:21.840]  Once the CPU is yours, it's yours until the router crashes.
[40:23.240 --> 40:25.040]  And through methods like that,
[40:25.040 --> 40:28.620]  you can get the remote box to shovel an enabled shell back to your listener,
[40:28.620 --> 40:30.340]  which is what Lynn did in his demo.
[40:31.280 --> 40:34.100]  Enable is the Cisco equivalent of root. Game over.
[40:35.260 --> 40:40.340]  He indicated that though he didn't give us the specific vector, and Cisco did,
[40:40.340 --> 40:44.200]  that there were other vulnerabilities that you could do this for.
[40:44.200 --> 40:47.720]  And here's some of them that might be fruitful grounds for exploration.
[40:49.420 --> 40:53.720]  Anything that talks about memory corruption that leads to a denial of service
[40:53.720 --> 40:57.340]  is probably a likely factor for this sort of thing,
[40:57.340 --> 41:00.640]  simply because since the Cisco routers tend to crash,
[41:00.640 --> 41:03.080]  rather than actually trying to recover,
[41:03.080 --> 41:05.500]  if you screw up the memory enough to induce a crash,
[41:05.500 --> 41:09.660]  it may just be, okay, your offset was wrong, or something like that.
[41:09.800 --> 41:13.980]  I really strongly encourage Cisco to look at their code
[41:13.980 --> 41:17.860]  and look at the way that he was able to interrupt the CheckHeaps process
[41:17.860 --> 41:22.660]  and avert crashing, because that's the underlying problem.
[41:22.660 --> 41:27.820]  The front-end vector, whether it's IPv6 or CDP or OSPF or BGP,
[41:27.820 --> 41:30.240]  that's not really going to be your issue.
[41:30.880 --> 41:34.860]  The fact that you can defeat CheckHeaps, that's your problem.
[41:36.620 --> 41:41.860]  So, yeah, here's a link to the advisory about vulnerabilities in IPv6 processing
[41:41.860 --> 41:43.400]  that Cisco put up.
[41:43.540 --> 41:46.860]  And just in bold, because this is the first time I've seen it
[41:46.860 --> 41:50.000]  and I've been waiting for years to see when this was going to occur,
[41:50.000 --> 41:53.880]  because I believe it was possible, but Mike Lynn's shellcode is better than mine.
[41:55.920 --> 41:59.340]  Cisco Internetwork Operating System is vulnerable to a denial of service
[41:59.340 --> 42:02.800]  and potentially an arbitrary code execution attack.
[42:02.800 --> 42:04.740]  So, Cisco did admit it.
[42:08.380 --> 42:12.980]  Again, just to hammer home the point about local segment versus remote segment,
[42:12.980 --> 42:18.520]  local network segment does not mean that it is not a remote group.
[42:18.940 --> 42:20.020]  Thank you.
[42:21.340 --> 42:24.300]  So, look at this for yourself. Don't just take my word.
[42:24.300 --> 42:28.980]  Don't take FX's, don't take Mike Lynn's, don't take Paul Watson's.
[42:28.980 --> 42:32.740]  Go and look at the source of these sorts of things.
[42:32.740 --> 42:35.060]  Read the advisories, read the vulnerabilities,
[42:35.560 --> 42:39.140]  and check what Cisco says now versus what Cisco said on full disclosure
[42:39.140 --> 42:40.980]  versus what Cisco said before.
[42:41.520 --> 42:45.260]  Read FX's paper about how Cisco handles memory management.
[42:46.120 --> 42:50.820]  And there is a link to where these Lynn slides are,
[42:50.820 --> 42:52.180]  and I believe they were still up.
[42:52.180 --> 42:55.980]  They were still up when I talked about an hour or so before I got up here.
[42:57.160 --> 42:58.720]  They're up. All right.
[43:00.840 --> 43:03.100]  So, you network defender people.
[43:04.560 --> 43:07.220]  Start treating your backbone devices like computers.
[43:07.220 --> 43:08.980]  Realize that they will have vulnerabilities,
[43:08.980 --> 43:11.220]  that you need to allocate for security,
[43:11.220 --> 43:12.700]  that you need to work for this,
[43:12.700 --> 43:17.380]  that you can't just sort of set it up like an ancient BSD box
[43:17.380 --> 43:19.220]  and, oh yeah, no one's ever going to hack that.
[43:19.220 --> 43:21.960]  It's fine. Routing is hard. No one understands that.
[43:22.000 --> 43:24.420]  Well, I'd like to point out that of the people that I know
[43:24.420 --> 43:25.940]  that are working in this field,
[43:25.940 --> 43:29.300]  I think I'm the only one that actually has an ISP background.
[43:29.660 --> 43:34.260]  You can't just depend on it's hard and security through obscurity.
[43:35.980 --> 43:38.660]  So, yes, you need to treat them like computers,
[43:38.660 --> 43:42.000]  but when you're testing, because denial of service is so prevalent,
[43:42.000 --> 43:43.900]  treat them like really important computers.
[43:43.900 --> 43:48.180]  Don't knock them over without your client's explicit consent.
[43:48.280 --> 43:49.520]  Maybe packet log.
[43:49.520 --> 43:52.500]  When I'm pentesting, I run a packet log
[43:52.500 --> 43:54.940]  so that if they come and say, you know, you broke XYZ,
[43:54.940 --> 43:55.980]  I can go, really, where?
[43:55.980 --> 43:58.940]  Here's my timestamps, here's my packets, here's exactly what I did.
[43:59.960 --> 44:03.720]  So that may be a contract issue for those of you who are doing that.
[44:05.020 --> 44:07.380]  Attacks only get better. They don't get worse.
[44:07.860 --> 44:12.020]  In time, these test tools are going to become a regular part of pentesters' toolkits.
[44:12.040 --> 44:16.840]  I would not be shocked if next year I suddenly saw Cisco attacks in Metasploit
[44:16.840 --> 44:21.040]  or something like that, or more Cisco's vulnerabilities in Nessus.
[44:21.040 --> 44:24.280]  We're still kind of in the dark ages of pentesting through the backbone,
[44:24.280 --> 44:26.720]  but keep in mind that this is where things are going.
[44:26.720 --> 44:29.580]  Enough people are interested in working on it that it's going to happen.
[44:30.600 --> 44:34.260]  And please don't take me off the internet, or slap me with a gag order,
[44:34.260 --> 44:36.040]  or try to sue me, or anything like that.
[44:36.040 --> 44:40.120]  There are mirrors. Everyone, please go out and mirror my stuff, put it on BitTorrent.
[44:40.120 --> 44:41.560]  Don't let them shut me up.
[44:58.020 --> 45:04.380]  Just a few last comments, and then I believe we have a brief time for questions.
[45:04.380 --> 45:10.980]  You can reach me at raven at nmrc.org with feedback, additional ideas, comments, lawsuits.
[45:11.700 --> 45:17.480]  I would like to thank Jericho and the OSVDB people for promptly putting all these vulnerabilities up
[45:17.480 --> 45:21.220]  in a public space where everyone can see them and have equal access for research.
[45:21.220 --> 45:26.060]  I would like to thank people who are mirroring these for me, and also possibly staring down the eyes of lawsuits.
[45:26.180 --> 45:30.280]  And I'd like to thank the EFF for sitting in the front row. I love you, new best friends.
[45:41.000 --> 45:43.080]  Alright, we have three minutes. Any questions?
[45:47.080 --> 45:48.240]  Beg pardon?
[45:56.120 --> 46:03.400]  My updated PowerPoint will be up momentarily. I think people are going to run out after this and mirror it.
[46:03.400 --> 46:11.340]  I will put a site online where there can be a list of the extant mirrors, and if that gets knocked down, it gets knocked down.
[46:11.820 --> 46:21.780]  Also, there is a legal defense fund for Mike Lynn. If you PayPal to abaddon, A-B-A-D-D-O-N, at io.com, you may need it.
[46:23.560 --> 46:25.980]  Oh, apparently you will need it.
[46:27.740 --> 46:28.680]  Yes.
[46:37.170 --> 46:46.150]  The question was, given the negative reaction to the Paul Watson TCP issue, how confident am I that ISP engineers will actually pay attention to this?
[46:46.410 --> 46:54.310]  Well, despite working in this field, I'm perpetually optimistic, but I think the fact that we have enough people continually saying,
[46:54.310 --> 47:04.150]  Hey, look. Hey, look. Hey, look. I am cheered that even though I was not going to announce the details of what I was talking about beforehand,
[47:04.150 --> 47:10.930]  because I didn't want to be shut up, a lot of my ISP friends contacted me over the last two days and went,
[47:10.930 --> 47:15.590]  Oh, holy shit, what do we need to do? So, there are at least some people who care.
[47:15.590 --> 47:21.650]  There is an ISP community that's dedicated to security. There are both public and private mailing lists that are working on this.
[47:21.650 --> 47:26.290]  The problem is not that there aren't some people who care. The problem is that most of the people don't care.
[47:26.290 --> 47:28.410]  So, truthful people, be loud.
[47:31.990 --> 47:40.400]  Alright, one more question. Yes, in the back. That's you. I can't hear you.
[47:54.400 --> 48:02.840]  The question was, since I've talked about ISP infrastructure and such, can the ISP protect its hosts against zombies and things like that?
[48:03.180 --> 48:06.940]  Yes, the ISPs can to some degree. The question is whether they are willing to.
[48:06.940 --> 48:10.500]  A lot of times to do that level of filtering takes additional resources.
[48:10.500 --> 48:14.580]  You need stronger CPUs. You need more memory in the routers.
[48:14.580 --> 48:17.880]  And a lot of places are running pretty close to bare bones on that.
[48:17.880 --> 48:21.200]  So, they don't have a lot of resources to spare and they don't think it's a good allocation.
[48:21.380 --> 48:24.280]  What they need to be able to do that is to hear from people and say,
[48:24.280 --> 48:27.460]  No, really, I want you to do this. I want you to filter me.
[48:27.460 --> 48:32.180]  There is client demand for it because if they don't see operational demand, they won't do it.
[48:47.510 --> 48:53.690]  The comment was, don't you have to have an interest in keeping spam and zombies and such from happening in the first place?
[48:53.690 --> 49:00.990]  Alluding to the poor record of many ISPs who benefit economically from allowing such things to happen across their network.
[49:01.290 --> 49:07.390]  And, you know, I could go on for an hour about that, but suffice to say, market demand.
[49:08.910 --> 49:12.810]  They are following where the money is. If you inform them where you want the money to be,
[49:12.810 --> 49:15.570]  the more people that inform them, the more likely they will be to listen.
[49:20.110 --> 49:22.230]  Something that we can do to fix what quickly?
[49:24.110 --> 49:30.930]  It's already fixed. If you go and you update your IOS to the most recent version of firmware in your code train, you'll be safe.
[49:32.150 --> 49:34.670]  All right, I think we're out of time. Thank you very much.