I'm Raven, and it's nice to see a bunch of familiar folks, and...
Is that better?
Alright, I will eat the microphone.
Not like that.
Alright.
I'm here to talk about penetration testing for background networks,
and I have these really interesting, great slides,
and then, Wednesday, some guy came and revolutionized my field, so...
I haven't gone to any parties this year, and I've done a lot of late-night rewriting.
So, thank you, Michael Lin, and...
The reason we were waiting...
is that my lawyer isn't here yet.
Oh, she's here. Fantastic.
So, the first thing that you really need to address
when you're looking at penetration testing for background networks
is that there's a pretty severe culture gap between ISP folks and the hacker folks,
and there's not really similar views on disclosure or vulnerabilities or anything like that,
and that can be kind of problematic.
So, how many of you do work or have worked for an ISP?
Show of hands?
Okay, you guys have probably run into this.
How many of you were in the routing department rather than the security department?
Show of hands?
Okay.
I used to work for ISPs.
I spent a bunch of years doing senior ISP background engineering and things like that,
and I eventually left for a more security-focused role
because I got really frustrated with just the differences of opinion.
A lot of ISP engineers, when they look at vulnerabilities,
they tend to have a very,
oh, well, no one's going to do that sort of view of things.
And their views on actual attacks, if they don't see it in the wild,
a lot of times they're very unwilling to actually treat something as a real problem.
And so it takes something like, well, Wednesday to make people sit up and take notice.
ISP engineers, in general, tend to go for a limited tiered disclosure method,
so when something is found, you know,
and people like that hear first,
and then Cisco's next favorite children hear second, and so on and so on.
And so if you have their gear, which are a small network near the edge,
you will hear late, if at all.
And most ISP engineers think that this is good.
They believe that, you know, catching the core is the most important thing,
and then moving outwards from there will release things in a more slow fashion
and minimize the window of vulnerability.
I have a feeling that most of you don't think this is a good idea.
So when you're trying to do pen testing on a backbone,
you're going to have to work with these ISP engineers,
and you have to sort of be a little bit gentle with them,
because otherwise they'll immediately say,
evil hacksaw, and stop listening.
So for maximum efficiency and security,
generally, when you're looking at testing a backbone,
I generally recommend doing it in a white box fashion rather than a black box,
because you can do all sorts of recon and data gathering and such
when you're actually doing it in a black box fashion,
but so many of the vulnerabilities that there are in backbone gear
are denial of service that people get a little pissed
when you take down their network.
And even if you've explained this to them,
you've signed the contracts beforehand,
you've made sure that you're legally covered, and say,
no, look, I could knock over your router,
they don't know what that means a lot of the time.
And so then they get ticked off that their backup didn't go through,
or suddenly they can't get to their router that you knocked over
because it's offline.
And so you want to be really careful when you're setting up your gig
if you're going to pentest a backbone,
because a lot of people don't really understand their ramifications.
So when I do this, I generally go with a white box sort of testing methodology.
So, when you're doing it, it's just like any other pentest.
You do your initial block of recon,
you look at the registrars for the address blocks and the autonomous systems that they have,
and any other data that's associated.
You can often get contact information for social engineering,
you can pull all sorts of information about the network that you are intending to test.
By doing a lot of these standard pentest techniques,
you can identify hosts on the edge,
but looking at the other IP addresses in their net blocks
is more likely to give you addresses that they're using on their backbone.
Now, a lot of people are, or should be, filtering information on the backbone
from things past the edge of their network.
In a perfect world, you shouldn't be able to hit the routers themselves
from outside of the network, unless you are on a management station or something like that.
It's just traffic should go through, and you shouldn't be able to target.
This is really, really not a perfect world.
A lot of ISPs fail at that level of filtering,
and we'll look at some stuff from the routing protocol security group a bit later on
that will show you some data on that.
It is worthwhile. There are many public route servers,
and it's worthwhile to look on those and internet maps
to try and determine who is connected to whom and where the peering is.
A lot of peering happens at public exchange points.
A lot of it happens elsewhere.
You have private interconnects between different ISPs,
and those will be crucial points when you're actually mapping your network.
You can often also, on route servers, get information about their BGP routing policy
and try and figure out what they accept and what they send to where.
Community strings will often be viewable.
Important information about BGP and which routes are preferred.
And there are many public route servers over there
that you can look at things from their perspective.
So if the network that you're analyzing is not one that is directly connected to a route server,
looking through two or three will give you several points of view
into how to look at it on the net.
Whatever you get, dump it into Google.
And if you get good results in that, dump them right back into Google.
And that will really help you build a profile of the network.
I believe it's really important not to just take the network maps
that their admins give you in Visio or what have you,
but to also try and build your own map non-destructively.
Their engineers aren't aware of everything.
Often they're not aware of much.
And that can be really important when you're looking at things from a pentester's point of view.
If they don't tell you about a link and you don't look and you don't find it,
you're not going to test that.
Also, a lot of ISP engineers aren't terribly security aware.
And so they'll post a mailing list and say,
here's my config, gee, this isn't working guys, what do you think?
You can get all sorts of information about what they're running,
how they're running it, what kind of gear they have,
and all of that can be dumped into your favorite friendly neighborhood vulnerability database
to build a likely profile of attacks and exploits that you could use against their network.
Now, a lot of the backbone is built on gear from a very few vendors.
And these vendors are generally not fans of full disclosure.
Cisco and Juniper both have on their networks things that are behind the front end of their webpage.
You have to have a login, in many cases you have to have a contract in order to see details.
Cisco does release some vulnerability information publicly, but there's a lot that they don't.
Juniper releases almost nothing publicly, I think I've seen like six advisories from them ever
that didn't require some sort of back end acknowledgement.
And so they believe that by doing this they are helping to keep it out of the hands of script kitties.
If you have a contract with them, or if you have a login to their site or something like that,
you will have a much greater database of information that's available to you
when you're trying to determine what to test for.
We're going to talk a lot about vendor negotiation and dealing with your vendors
in order to secure the network as best as possible.
If you are working for a company, they may allow you to use their CCO or Juniper logins
to access the bug databases and look for vulnerability information.
Also, don't forget that the backbone is more than just routers.
There are switches, there's lots of stuff that gets done on Layer 2,
and a lot of people just kind of ignore that.
So test through your switch gear also.
Cisco in particular has really complex code trains.
iOS has a zillion different branches which support different features
and for different hardware platforms and such,
and you will need to be aware of what you are running in the network you're testing.
If they've left CDP on or something stupid like that, then you can get them to tell you.
If their engineers are friendly, you can get them to tell you.
You can log into all the routers if they're giving you that level of access
and do a chauffeur and pull details on that.
But something that is vulnerable in one train may or may not be implemented in other trains.
It's worth looking with a broad brush to start off with.
Something might be vulnerable and then going,
oh, okay, it's not a vulnerable version is much better than saying,
ah, well, it's only this one version, let's not test that.
It's also worthwhile in looking at code trains and where they're derived from.
Code is forked all the time, and sometimes you'll come up with an error that was in one version
and years and years down the road, oh, look, it's still there.
There was a BSD vulnerability in Telnet a while back that whacked half the stuff on the Internet,
almost regardless of operating system, because so much was derived from that old code base.
And I have a link here to the vulnerability in Cisco's CatOS boxes.
We use that same bit of code and we're vulnerable to that same bit of exploit.
There are some scanners that will incorporate these checks for you.
I know Nessus has some Cisco vulnerability information in it, but a lot of it isn't automated yet.
And because of the great potential for denial of service,
you can see why a lot of players got real thrilled to have that run against them
and why a lot of people haven't been too eager to put it in the exploit tools.
So this is something that, I think, pen testing like 5 or 10 years ago,
you're going to have to do a lot of this by hand in order to make sure that you've got everything.
As with any other pen test, failure paths and trust relationships are important.
When you look at their architecture, look for redundancy,
look for network robustness, and look for security.
One thing that I think is really important when you're pen testing a backbone
is to make sure that availability is a factor.
I mean, denial of service is such a big deal for them,
and yet so many people build networks with one path where a link goes down and you're screwed.
Or they have two lines terminating into one card on one router,
and a hardware failure will cause problems.
I think, as security analysts, it's our job to point these things out and go,
oh, you've got a single point of failure if someone knocks this out.
If there's an exploit, if you have a hardware failure, you have an issue.
I personally try and include things like that when I present a report to the client.
Also, check for the authentication means and see whether those have redundancy.
Just as with some of the SSL versions, you know, if you can force it back to the weakest point,
then you can exploit that.
There was a vulnerability on a RADIUS server in Cisco's implementation of RADIUS
where you could give it several ways of authenticating and none was an option.
So if you can just say, oh, well, I don't want to authenticate, no.
You know, oh, okay then.
Things like that are kind of bad.
So with things like that, check for what the weakest way is
because you know your attacker is going to take the weakest way and if Pentester is still awake.
Once you're authenticated, is trust transitive?
You know, if I get a login to that RADIUS server, what else does that give me?
It's important to look at that.
And so as soon as you get every new level of access, every new point,
if you try and map the network again from that point of view, you may find extra stuff.
One of the things you'll see a lot and one of the ways I've had most success in testing backbone networks
is authentication servers.
If you have a centralized authentication server, that's a really good point to start at.
If you look for a login to that, you can sniff traffic and get someone's login
if they are foolish enough not to encrypt it.
If you can own one of the hosts that they use for management stations,
you can then grab it before it's encrypted and sent on the wire with a keystroke logger or something like it.
And that will often give you the keys to the kingdom.
Very many places don't think to harden their authentication servers,
and so that's a really common point of failure.
Another thing that I have successfully done is if they have fallback mechanisms
where they have a local login in case their RADIUS or their CACX or their whatever server is down,
well, if you can knock that server offline and then brute force the local password,
that will also get you in.
And once you brute force the local password, depending on how they're storing it,
like for a Cisco box, they use a 7 encoding, which it's not encryption, it's encoding.
It's symmetric, it's easily reversible, alcrypto.co.uk.
Cisco has an online decryptor, although you may wonder where that's going.
So they also have programs you can download and run locally.
But a lot of the passwords that are saved in the configs,
you can just throw them in the internet, get the password out,
and they may be using that in multiple places.
But a lot of places do not do password management well,
and so a local login at one place will get you a local login at another place.
And as long as that central authentication server is down, there you go.
Obviously, once you've logged into the boxes, if you have enabled, you can change that.
But that will let them know you're there. You know, please be nice.
Also, man-in-the-middling against the authentication server is usually a fairly successful technique,
if you can get access to do that.
It shouldn't need mentioning, but physical security is really important.
A lot of places forget this. Everywhere you have a data center,
and especially everywhere you're peering, please have good physical security.
Please audit and make sure you have good physical security.
Please don't let me walk into Macy's with a switch under my arm and a laptop under the other one and plug in.
Yeah, bad.
A lot of the time, if you have local access, local physical access, that will get you a lot.
And many of the places, they have a very clear profile of what they're looking for.
So, you know, if I was 40 and had a beard and came in,
they might be a little more suspicious of me than if I'm 29 and got my eyelashes and go,
gee, I don't know, I'm just supposed to go look at this router.
They have poorly placed trust relationships in terms of local access,
and if you want to take a social engineering approach to that, then that can work to your advantage.
If you don't have physical security, you're screwed.
One of the things that I've seen fairly often that's kind of appalling
is how many people will leak data about the processes of the core of their network to the edge.
So they'll have EIGRP as their internal routing protocol,
and they won't turn it off for interfaces that face out towards the edge of their network.
They just sort of send those multicast, hey, hey, who wants to be my neighbor, out to anyone.
And, well, you shouldn't let people at the edge of your network establish a neighbor relationship
and feed you a bunch of route data, which may be bad.
Also, CDP. People will leave it on by default, they won't turn it off,
and it will happily send you all kinds of information about the connected device.
It will tell you what it is, what version of the operating system it's running,
what sort of hardware it has, device ID, all kinds of stuff.
And Cisco has designed this so that it's very useful when you're troubleshooting.
I have had cases where I wanted to see what was on the other end of the link,
and we weren't sure that the person at the peering site or the central office had us plugged into the right ports.
So the other side turns on CDP, and you go, whoa, I'm not supposed to be connected to a 7206 VXR.
Yeah, that's wrong.
So when troubleshooting, it's a good tool.
But when you're not troubleshooting, turn it off.
Switching data. Also, switch management protocols, things like that, will be happily sent out.
Hey, welcome to my spanning tree, who wants to be my root bridge?
These things should not be seen by the end users, and they often are.
So that's something for you pen testers to look for.
If you can see the data, you can send things back.
You can spoof packets, you can send real data, you can feed them information on who's supposed to be bridging to what,
or who's supposed to be routing to whom.
If you can get a more preferred route and inject it, then that will, depending on the implementation, of course,
it may be able to take over the route that's actually on the legitimate internet.
A lot of this is... when you're speaking internal routing protocols rather than BGP,
it's a lot easier to deal with on the local net.
BGP has its own separate set of rules because of how it's handled between providers
and how you set up neighbor and peering relationships.
But on the local network, it's often not that hard.
If you want to do a really sort of not high-skilled but very effective DOS attack,
you can tear down links with spoof packets.
A lot of the time, they don't have authentication on there,
and so many routing protocols support the capacity for even something as simple as an mb5 hash.
It's often called BGP password, although many other protocols also support the same thing.
I know you guys have heard about mb5 fairly recently, but, you know, it's still a whole lot better than nothing.
Similar to the theory of rogue access points, if you have physical access to the site,
you can just walk right in with a router and say,
yeah, I'm supposed to install this, and some places aren't very good about saying,
who the hell are you, why didn't I know you were coming, you know, go home.
And you can plug in a router, you can put it into the peering switch,
you can just, you know, start snipping traffic.
If you can manage to get the switch ports set promiscuous or something like that,
then you can see what they're speaking and start to speak it too.
This also is obviously bad.
Depending on what your contract is with the client, you can even go in there and say,
oh, yeah, you know, I'm from this company, I'm supposed to work on their router,
yank out theirs, put in yours with the configuration of whatever you want,
and, hey, as long as you pass it through to them,
they may not even notice except for the ten minutes of reboot time.
A lot of places do not have good passwords.
It's better than it used to be because of spammers.
There have been cases of spammers taking over Cisco routers
and then using the telnet function to send spam out of the router itself one character at a time.
So, yeah, bad.
But a lot of places have gotten better about not having, at least pathetically, default passwords.
However, if I get called in and someone's locked out of their router,
I always give Cisco Cisco a try, you know, maybe one time in four it works,
and just, it shouldn't.
You can redirect traffic through a GRE tunnel.
I've seen that in the wild, and people were a little bit confused about what was going on there,
and said, I don't remember setting that up. Bob, did you set that up?
And meanwhile, it's terminating in, you know, Moldavia or somewhere, and there go all your packets.
Another serious concern is that, especially when you're speaking BGP,
you may be able to advertise additional net blocks out to the net,
and if your route is preferred to other people's routes, you've just hijacked a net block.
And this happened.
There was a bunch of attackers that set up a site through an ISP that wasn't doing good security checks
about who actually owned that IP space.
They announced this block out to the world, and their route was preferred in most places
because of the way that they did it and how their provider was set up,
and they happily spent about a week sending out spam and sending out worms
instead of a botnet, and were owning boxes left and right.
Meanwhile, the legitimate users of that net block can't even get online because no one's routing to them.
Everyone's routing to the own site.
And so, when that was finally resolved at the ISP level,
then the original owners are really left holding the bag there.
Hi, you're on every blacklist on the planet,
you have a whole bunch of angry people pounding down the door at your abuse desk,
and you were offline for a week.
And because some ISPs do not do good checking when they're hooking up a new client
of do you really own this block, is this legitimate,
that can happen, and it shouldn't.
There are some really good people working ISP security
that are cooperating between ISPs to try and prevent that.
And, you know, more power to them. We need more of them.
A great place to start when looking at the actual configurations of the backbone devices
is to look at the secure BGP template,
iOS template, Junos template, and Junos BGP template
that the Cymru people do.
These are very, very well done documents.
They have bogon lists so that you cannot accept routes
from IP addresses that haven't been allocated or don't exist.
They do strong filtering so that you don't have, like, you know,
RFC 1918 addresses speaking to you,
you shouldn't allow people to advertise, you know,
your own net blocks to you, for example, things like that.
They have good checking for the strongest possible authentication methods.
They try and turn off things like, oh, hey, let's speak finger,
you know, ancient insecure daemons that you have no business really needing
to run on your major backbone routers.
I can't recommend these enough. Those guys do really good work,
and it's also handy for a beginning place to start with an engagement
if you're doing a white box design analysis.
Peering security is particularly important.
When you look at how someone is peering,
look at the routes they're willing to accept and look at the routes
that they're willing to send out.
There was a case several years ago where someone actually advertised
10.0.0.0 to the whole internet and claimed that they owned it.
And that got a lot of people a little bit irate.
It broke a lot of people's private network implementations,
and it broke a lot of VPN stuff.
So that's worth checking.
Again, route servers are your friend.
And when you do peering changes, make sure you're doing good authentication.
You don't want just anyone to call you up and say,
Hi, I'm Raven from Uunet, and yeah, I want you to change this config.
Can you take out this net block of this guy who, you know,
stole my girlfriend last week?
You want actual passwords, some sort of validation method
to make sure the person who you're talking to to make these major
affecting changes actually is the person that they claim to be.
Also, make sure that the machines that you use for your network policy
and your network admin are well secured.
I have worked at a job where we had a couple of management stations,
and they were Windows boxes, and wow, they got owned.
That's really dumb.
If you have only a few machines that are allowed to access
these crucial backbone devices, make sure they're hardened.
Make sure they're patched.
Make sure that people can't get onto them and, you know...
In that case, I think what happened was
these machines that we did backbone stuff from
were the same machines that we did abuse tests from,
and so you're investigating an abuse case,
and oh, our user's surfing porn, go to the porn site,
porn site's full of malware, mandated corporate IE, boom.
Don't let that happen.
And filtering is also very important.
Not just what you allow to your router for management traffic,
but also what you allow as far as routes.
You want to make sure that you do not allow
internal routes that should stay internal to your network
outside to the world.
Yes, aggregation happens, but more precise routes are often preferred,
and that can be a problem.
It bloats the routing tables, it uses up memory,
and it can be revealing more details about the interior of your network
than you wanted.
The RPSec Working Group did a small survey of people in the field,
and they surveyed people who were peering at an exchange point.
So these are some of the more aware, more security conscious people out there.
And they still weren't using everything that was available.
This may be because, for lack of a better term, ISP culture.
They view change management in the time that it takes to do good password control
and to change all these policies as more of a threat than the security implications.
And so you've got to keep track of who's allowed to do what,
and who's filtering where,
and, you know, oh, we screwed up our peering list,
and, you know, okay, now there's downtime because I wasn't allowing your route,
or I had too many prefixes.
And so, you know, they think that the hassle of management
is often not worth the additional security.
And from an ISP point of view, I can see why they take that approach.
When you're driven by the bottom line,
and you're a network engineer with too much work and too little time,
it can be difficult to deal with these sorts of things.
Nevertheless, I think this is going to become increasingly important
as more attention focuses on the backbone as, you know, started this week.
The RPSEC site security.
Many people are actually doing an incoming prefix filter for each BGP customer,
which is good.
You should only allow people to advertise their own netblocks to you.
You don't want them to start advertising your netblocks,
or, you know, someone else's netblocks,
or that guy who stole your girlfriend's netblocks.
That can get you in legal hot water.
A lot of them also are checking for incoming autonomous system paths.
A lot of people use community strings in order to control this sort of thing.
However, very few people are using an outgoing AS path filter,
and in that case, it really depends on who you are.
If you are a small ISP near the edge of the network,
you have your AS, and you have a few others that are your customers,
that may actually be a good idea.
If you are a tier 1, and you have every AS in the world flowing through you,
that can be a real change management challenge,
and probably operationally unwise.
Outgoing updates towards your peers.
A lot of people use communities on that, but not everyone.
And there is the URL here for the actual survey,
if you want to look at the results from the RPSEC group.
I didn't put them all up.
So, when you're looking at actually doing a pentest,
where do you get your backbone vulnerabilities?
They're not all in the scanning tools.
We've established that earlier.
And Cisco and Juniper may have you log into their site to actually look at it.
Mailing lists will have public vulnerability announcements.
I've often found that it is a good idea to check the mailing list against the vendor site,
because, hey, people change their webpages all the time.
Mailing list archives are usually more static.
You can at least see what the first release was of something.
We saw this happen this week, when Cisco announced their IPv6 vulnerability.
And if you look on their website now, it's 1.1.
The original one that went full disclosure is 1.0.
There have been some changes made.
And it is valuable to Pentester to be able to see both versions.
There's been, in the last five years, increasing interest in this field by hackers who don't come from an ISP background.
FX and Paul Watson and Mike Lynn have all made valuable and important contributions
that have really caused a bit of uproar in the ISP community.
And after this week's uproar, I think there will be a whole lot more people looking for it.
So, FX has done a lot of stuff in this field that is really useful.
There is a EIGRP Spoof Packet Neighbor Saturation bug,
where essentially, if you send it more than 256 neighbors, then it just falls over.
EIGRP works by sending out multitasks.
Hey, here I am! Who wants to be my neighbor?
And so if you say, me, and me, and me, and me, and me,
you can essentially overflow the amount of memory that it allocates for that,
and, you know, it's at least a denial of service.
It could be worse.
Similarly, for the CBP neighbor announcement,
if you send Cisco Discovery Protocol packets with device IDs that are really, really long,
you can fill up the memory tables and possibly do evil after that.
There's been GRE attacks, a TFTP server advisory, and a bunch of other stuff.
If you look at FX's site on Phenolate, there's tons of stuff.
There's actually a tool for pen testing that was released specifically to address these kinds of things.
It only addresses the vulnerabilities that FX actually found, but it is well worth looking at.
There's also been a really interesting paper about Cisco memory mapping,
and I'm sure that any of you that were at Black Hat on Wednesday saw similar work in Mike Lynn's talk.
The memory structures are still the same, the magic numbers are still the same,
they're all online, and it is very useful if you want to dig into the assembler.
Then about a year ago, any of you that were at CanSec probably heard the storm and fury
surrounding Paul Watson and the TCP window.
Essentially, TCP stacks on background equipment and many other things
were not correctly looking at the window size and anything that hit spot on.
Just anything in the general neighborhood was good enough.
So if you send a reset that's in the general neighborhood,
and you're able to correctly guess the IP addresses and port numbers on either side,
you can knock down the session with a spoof packet.
And this allowed unauthenticated BGP sessions, if you could guess it all correctly, to be knocked down.
And this was really interesting when you look at disclosure and how the whole thing went out,
because essentially what happened was when Cisco started notifying people of this,
because Watson worked with Cisco and said, hey, look, we need to fix this.
Many people on the backbone and in NANOG and similar ISP communities
had known about this for years. They just didn't think it was very possible.
And again, we see that attitude of, well, I'm really busy unless I see something in the wild.
It's not a problem. It's not going to happen.
And so Cisco said, okay, everyone, you need to put md5 authentication passwords on your BGP sessions now.
Okay, you need to upgrade now. They didn't tell people why.
And so they told their favorite people first, and then their second favorite people,
and their third favorite people, and so on out.
And so it took, I think, the better part of a month for it to actually get heard by most people.
And then Watson gave his CanSec talk.
Once the ISP operators found out what the actual flaw was,
there was a very strong cultural backlash saying, oh, I knew all about that.
You know, why are you making such a big deal?
And I was really depressed by going to NANOG this year, where a lot of operators were like, oh, yeah, you know,
we thought that was important, but wow, we really freaked out.
And I've rolled back. I don't have any authentication on my sessions anymore.
You know, stupid hackers.
And that was upsetting.
Because I really saw the community response to the Watson vulnerability as a strong positive step in securing the backbone.
And then seeing everyone go, wow, we freaked out over nothing, did not help.
I think Cisco's approach of we're not telling you why, just patch it because we say so, no really, just trust us,
I think they hurt themselves because people trust them less than they used to.
And then we come to Wednesday and what made me rewrite my slides.
If you were at Black Hat and saw Michael in session on Wednesday morning,
he essentially demonstrated what I view as a remote exploit in Cisco IOS.
Cisco has announced in at least one of their vulnerability announcements that it was just a local.
I think we have different meanings for the same word there.
And again, the disclosure process here was important because when Mike gave his talk,
he essentially talked about the internals of how it worked.
That stack overflows are hard to find in Cisco IOS, that the code base looked strongly audited,
and that you had to work with heap overflows, but that Cisco has a check-heaps process,
which will go through and validate, make sure the previous and next values are the same,
so that if you actually do corrupt the heap, then it will say, oh, this isn't okay,
all right, I'm crashing the router because this is screwed up.
He did not give details of what his entry vector was.
He was very careful to basically try and not give enough information for people to recreate it.
Now, even though the vulnerability has been patched and the old versions have been yanked off of Cisco's website,
there are still vulnerable versions out there in the wild.
Cisco, really shockingly to me, yesterday morning released an advisory to full disclosure that said,
yeah, for the first time ever, we have a denial of service and possible remote code execution
in our handling of IPv6 packets.
Lynn didn't tell us that. Cisco told us that.
In the latest version of their advisory, they say that it was disclosed at Black Hat,
so that really does nail it down to, yes, this was the vulnerability.
And this is really revolutionary for the field, because for the first time,
it looks like you can remotely own a Cisco box.
You can get it to shovel you back a shell that will allow you to have full enable access.
And this is a scary thing if you're an ISP operator.
I think that Lynn was pretty sensible in saying that people need to treat these like normal computers,
they need to patch, they need to be aware of security,
and they need to be diligent about making sure that they patch through the versions,
because this is a real threat, people. We've now seen the remote route.
Now, when you look at the Cisco advisory, what they appear to mean by local,
to the best of my understanding, and, you know, disclaimer,
I don't and never have worked for Cisco, I don't and never have worked for ISS,
I don't speak for anyone but me, but Cisco claims that you have to be connected to the same local subnet
in order for that to work, and that makes it a local exploit, right?
Yeah.
No, if you have to be on the same subnet from one remote box to another remote box, that's a remote route.
And then came the cover-up.
The talk's been ripped out of the books, the presentation's not on the CD,
everyone got modified CDs, the slides aren't available,
oh my god, there's suing, and there's suing, and there's the FBI,
and there's restraining orders, and rumors, and news stories, and all sorts of stuff,
and Cisco, you're really screwing up here.
You know, to give you an idea of the attitude of the security community,
there are people walking around wearing these.
I think that gives you an idea of the mentality of what happens when you try and cover something up.
Okay, so you're not really happy.
Big pardon?
It said, fuck Cisco.
Let the record reflect.
And I really think that Mike Lynn was right on when he talked about why he gave the speech,
and why he released as much information as he did.
I mean, okay, the reason he got onto it in the first place, again, from the best of my understanding,
I don't know the guy, I don't work for anyone, yada yada.
He said that he found some Mandarin Chinese documents,
which when translated were talking about how to hook debugger hooks in iOS,
and how to make those speak to your disassembler and your debuggers.
This means people are working on it.
Their source code has been stolen, and there's now a storm of interest.
I know that there have been several people who have suddenly been diving into their assembler
to attempt to figure out what's going on.
Hiding your head in the sand is not going to help.
If you have people who are attacking you that are actively looking for this,
going, no, no, it doesn't happen, quick, shut everyone up, is not going to help.
And even if you destroy all the slides, and you aren't going to let things get released,
well, his slides are out there on the internet.
Suing researchers is not going to achieve security.
I mean, only speaking for myself.
I was in his talk.
I speak assembler.
I know Cisco iOS.
With enough work, I could probably recreate his exploit.
I'm sure I'm not the only one.
Trying to pretend it won't happen will not fix things.
And alienating the security community,
and making everyone think that you're going to be sued into oblivion if you ever speak up,
that's not going to encourage people to come to you and say, hey, we've got a problem, let's work with you.
When I was reading the reports of extant backbone vulnerabilities,
it sounded like they treated FX a lot nicer than they treated Mike Lynn.
FX actually thanks people on the Fino Elite Advisories,
and says, hey, thanks for working with me, Cisco P-Cert, you guys were really great.
They don't sound really great right now.
They filed a temporary restraining order to prevent him from speaking about it.
So, he can't talk about it, but I can.
Well, his slides are easily findable on the internet,
and not even the version that he presented at the talk.
The original slides from before he got fired by his employer,
I think like yesterday morning they were up on CryptoMay and several other sites,
and this has just been a ridiculous media circus.
They were put up on Infowarrior.org,
and they got pulled down with a cease and desist by Cisco,
and guys, just help us fix this.
The CD for that was from ISS, apparently.
So, he had to return his computer, he's not allowed to talk anymore,
and this is really dumb.
Dear Cisco and ISS,
Responsible disclosure is good.
Working with security researchers is good.
Trying to cover up things that are known problems
that have been released into the public domain
that are on millions and millions of machines all over the internet now is good.
If you try and shut people up,
you're only going to alienate the very people that you need to help you.
Duh.
Alright, so, what Mike Lynn actually said.
Heap overflows are way more common in iOS.
He said a bunch of nice things about their code base,
but he said that the watchdog process of check heaps will check for heap corruption,
and if it finds it, it will crash it.
And most of you that have worked with Cisco routers I'm sure will see
that if you have a Cisco router and it has a fault,
a lot of times, rather than actually trying to recover,
it will just crash itself.
And the really important thing that I think came out of Lynn's research
was that the crash process itself is interruptible.
So, you can interrupt it when it says,
Hey, this bad thing has happened. My heap has gone badly. Am I crashing?
And you can tell it,
Why, yes I am.
Okay, we're already crashing. I don't need to do anything then.
Please continue executing whatever the heck it is you were executing there.
So, you can buy yourself some time that way.
He did mention that spreading memory corruption on the heap was going to be an issue,
and that you might have two to five minutes before it exploded all over your memory, basically,
unless you actually handle that in your shellcode,
and anyone who's written shellcode knows that that can be tricky.
But once you've actually gained execution,
there are hardware interrupts that will try and interrupt you,
but you can disable those.
Once the CPU is yours, it's yours until the router crashes.
And through methods like that,
you can get the remote box to shovel an enabled shell back to your listener,
which is what Lynn did in his demo.
Enable is the Cisco equivalent of root. Game over.
He indicated that though he didn't give us the specific vector, and Cisco did,
that there were other vulnerabilities that you could do this for.
And here's some of them that might be fruitful grounds for exploration.
Anything that talks about memory corruption that leads to a denial of service
is probably a likely factor for this sort of thing,
simply because since the Cisco routers tend to crash,
rather than actually trying to recover,
if you screw up the memory enough to induce a crash,
it may just be, okay, your offset was wrong, or something like that.
I really strongly encourage Cisco to look at their code
and look at the way that he was able to interrupt the CheckHeaps process
and avert crashing, because that's the underlying problem.
The front-end vector, whether it's IPv6 or CDP or OSPF or BGP,
that's not really going to be your issue.
The fact that you can defeat CheckHeaps, that's your problem.
So, yeah, here's a link to the advisory about vulnerabilities in IPv6 processing
that Cisco put up.
And just in bold, because this is the first time I've seen it
and I've been waiting for years to see when this was going to occur,
because I believe it was possible, but Mike Lynn's shellcode is better than mine.
Cisco Internetwork Operating System is vulnerable to a denial of service
and potentially an arbitrary code execution attack.
So, Cisco did admit it.
Again, just to hammer home the point about local segment versus remote segment,
local network segment does not mean that it is not a remote group.
Thank you.
So, look at this for yourself. Don't just take my word.
Don't take FX's, don't take Mike Lynn's, don't take Paul Watson's.
Go and look at the source of these sorts of things.
Read the advisories, read the vulnerabilities,
and check what Cisco says now versus what Cisco said on full disclosure
versus what Cisco said before.
Read FX's paper about how Cisco handles memory management.
And there is a link to where these Lynn slides are,
and I believe they were still up.
They were still up when I talked about an hour or so before I got up here.
They're up. All right.
So, you network defender people.
Start treating your backbone devices like computers.
Realize that they will have vulnerabilities,
that you need to allocate for security,
that you need to work for this,
that you can't just sort of set it up like an ancient BSD box
and, oh yeah, no one's ever going to hack that.
It's fine. Routing is hard. No one understands that.
Well, I'd like to point out that of the people that I know
that are working in this field,
I think I'm the only one that actually has an ISP background.
You can't just depend on it's hard and security through obscurity.
So, yes, you need to treat them like computers,
but when you're testing, because denial of service is so prevalent,
treat them like really important computers.
Don't knock them over without your client's explicit consent.
Maybe packet log.
When I'm pentesting, I run a packet log
so that if they come and say, you know, you broke XYZ,
I can go, really, where?
Here's my timestamps, here's my packets, here's exactly what I did.
So that may be a contract issue for those of you who are doing that.
Attacks only get better. They don't get worse.
In time, these test tools are going to become a regular part of pentesters' toolkits.
I would not be shocked if next year I suddenly saw Cisco attacks in Metasploit
or something like that, or more Cisco's vulnerabilities in Nessus.
We're still kind of in the dark ages of pentesting through the backbone,
but keep in mind that this is where things are going.
Enough people are interested in working on it that it's going to happen.
And please don't take me off the internet, or slap me with a gag order,
or try to sue me, or anything like that.
There are mirrors. Everyone, please go out and mirror my stuff, put it on BitTorrent.
Don't let them shut me up.
Just a few last comments, and then I believe we have a brief time for questions.
You can reach me at raven at nmrc.org with feedback, additional ideas, comments, lawsuits.
I would like to thank Jericho and the OSVDB people for promptly putting all these vulnerabilities up
in a public space where everyone can see them and have equal access for research.
I would like to thank people who are mirroring these for me, and also possibly staring down the eyes of lawsuits.
And I'd like to thank the EFF for sitting in the front row. I love you, new best friends.
Alright, we have three minutes. Any questions?
Beg pardon?
My updated PowerPoint will be up momentarily. I think people are going to run out after this and mirror it.
I will put a site online where there can be a list of the extant mirrors, and if that gets knocked down, it gets knocked down.
Also, there is a legal defense fund for Mike Lynn. If you PayPal to abaddon, A-B-A-D-D-O-N, at io.com, you may need it.
Oh, apparently you will need it.
Yes.
The question was, given the negative reaction to the Paul Watson TCP issue, how confident am I that ISP engineers will actually pay attention to this?
Well, despite working in this field, I'm perpetually optimistic, but I think the fact that we have enough people continually saying,
Hey, look. Hey, look. Hey, look. I am cheered that even though I was not going to announce the details of what I was talking about beforehand,
because I didn't want to be shut up, a lot of my ISP friends contacted me over the last two days and went,
Oh, holy shit, what do we need to do? So, there are at least some people who care.
There is an ISP community that's dedicated to security. There are both public and private mailing lists that are working on this.
The problem is not that there aren't some people who care. The problem is that most of the people don't care.
So, truthful people, be loud.
Alright, one more question. Yes, in the back. That's you. I can't hear you.
The question was, since I've talked about ISP infrastructure and such, can the ISP protect its hosts against zombies and things like that?
Yes, the ISPs can to some degree. The question is whether they are willing to.
A lot of times to do that level of filtering takes additional resources.
You need stronger CPUs. You need more memory in the routers.
And a lot of places are running pretty close to bare bones on that.
So, they don't have a lot of resources to spare and they don't think it's a good allocation.
What they need to be able to do that is to hear from people and say,
No, really, I want you to do this. I want you to filter me.
There is client demand for it because if they don't see operational demand, they won't do it.
The comment was, don't you have to have an interest in keeping spam and zombies and such from happening in the first place?
Alluding to the poor record of many ISPs who benefit economically from allowing such things to happen across their network.
And, you know, I could go on for an hour about that, but suffice to say, market demand.
They are following where the money is. If you inform them where you want the money to be,
the more people that inform them, the more likely they will be to listen.
Something that we can do to fix what quickly?
It's already fixed. If you go and you update your IOS to the most recent version of firmware in your code train, you'll be safe.
All right, I think we're out of time. Thank you very much.