So our next talk is called Hacking ISPs with 0.2 Pwn Protocol over Ethernet, P-P-P-P-O-E, with Gal.
It's the second time at DEF CON. I guess we didn't scare him the first time.
But please give him a big cheer, and let's start. All right?
Thank you.
Thank you.
All right. So, hi, everybody, and welcome to my talk on Hacking ISP with P-P-P-O-E, or as the title says, 0.2 Pwn Protocol over Ethernet.
And first, I'd like to thank DEF CON for having me here again this year, and this time in person, so I'm excited.
And before I begin, let me share a quick story with you.
So on this research, I've been working on and off for the last two years.
And as of course, it was during the pandemic, and like everybody else, I had to work from home.
However, this research involved this device on my kitchen table for almost a year.
And let me give you just a quick example how it sounds like.
I'm also going to pray the audio gods that it's going to work.
All right.
No audio.
Oh, damn.
Wait, so I got a fallback for that.
And I'm just going to play it from my phone.
So this is how the device sounds like.
Yeah.
So, yeah, so before I begin, I would especially like to thank my wife, Ortal, that sits here with us today,
for sharing a single bedroom.
And I'm going to play the music for you.
Embedded Room Apartment, with me and that device.
Yeah.
Thank you, honey.
Yeah.
Yeah.
Tough year, the pandemic.
All right.
Okay.
So enough about that.
So, okay.
So let's begin.
My name is Gal, and I've been causing ruckus on embedded device for quite some time.
And as I said, this is two years long research.
And I actually started this research in the early 2000s.
And I started working at it when I used to work at Aleph Research.
And I ended it in CyberArk Labs, where I work today as a research manager.
And today, we're going to talk about three subjects.
First, I'll introduce the idea and motivation behind hacking internet service providers,
ISPs.
After that, we'll do a crash course in layer two communication protocols.
And I present some cool vulnerabilities I found.
And then, we'll talk about the process.
And lastly, I'll share how I was able to research this kind of equipment.
And hopefully, I might convince you that at the end of the day, big ISP network equipment,
it's not that different from small home router research.
Okay.
So let's begin with pwning internet service providers.
So we all know how a classic remote attack is executed.
Usually a attacker looks for a victim.
And the target is a victim on the internet.
Maps the attack surface by discovering what protocol the target uses.
And then try to use Oday or Nday and hopefully get RC on the target.
Ironically, many of these classic attacks are nowadays taking place on home routers.
So the idea is that ISP network equipment, it's not that different.
Instead of the internet, we got our ISP network accessibility.
Well, from our local modem.
And instead of the internet server, we got the ISP equipment that provides us services.
So all we have to do is to map the attack surface, find an Oday, and hopefully get an
RC on the ISP network equipment.
But why attacking ISP?
Well, if I was the proud pwner of Evil.com LTD, I might be able to do some of the following
stuff.
For a start, the obvious denial of service and other ransom activities like shutting
down the network and asking for money to stop doing it.
But I can also execute DNS hijack for the entire network, aka become man in the middle
for all subscribers by redirecting their DNS to my own evil DNS server.
And another interesting idea is to target a specific subscriber by their actual identity,
aka the actual name and other information.
The ISP server.
I also might be able to might be connected to the internet backbone for IP allocation.
So I might be able to execute this crazy scale distributed denial of service.
And lastly, I can execute all sorts of attacks by abusing common protocol and technologies
used by the ISP.
Okay.
So to understand what we are about to attack.
Let's understand how basic ISP is.
First of all, the ISP network operates.
Keep in mind that this is a simplified example for DSL-based network.
But other tunneling protocols such as PPPoE, L2TP, GPON, they all follow more or less
the same concept.
So when you get your router from your ISP, it has two roles.
One is to provide network services such as Ethernet and Wi-Fi to your home premise.
And the other role is DSL modem functionality.
The DSL modem basically uses the good old telephone line to connect you to your ISP
and then to the internet.
Your modem is hooked up to a phone line and the telephone company connect the other end
of the line to something called DSLAM.
DSLAM is basically a multiplexer that extracts digital information from analog signals by
sends from multiple modems over the telephone lines.
Okay.
There.
DSLAM is then connected to a broadband remote access server or BROS, which is basically
a big router that routes traffic to and from the DSLAM to the ISP network.
So we can abstractly think that our modem is connected with a layer two cable directly
into the BROS itself.
And we can also bridge the modem traffic to our own evil machine.
So now we can think abstractly.
That our malicious machine is connected with a very long ethernet cable directly to the
BROS.
So if the BROS is the ISP's frontline router and we are connected to it with a network
connectivity, then hacking this BROS is the equivalent of hacking the ISP's home router.
All right.
And this is where home router takes the power back.
Uh.
They can attack the BROS in the same way hackers can attack them from the internet.
Once we control the BROS, attackers can execute some of the attacks I mentioned before or
attack other ISP network equipment.
All right.
So for this research, I decided to target Redback network smart edge equipment.
Back in the 2000s, Redback wore a big player in ISP equipment.
They were actually involved in defining some of the protocols I'll show you today.
In 2007, they were acquired by Ericsson that continues to manufacture new products under
the Redback brand.
Some of these devices are pretty big and can support up to half a million subscribers.
For example, the one that you see at the bottom right.
And they all use custom NetBSD operation system called SEOS, smart edge.
And they also use a PowerPC architecture.
Okay.
So now that we have a target and we see it's worth hacking to it, let's understand what
kind of protocol can be used to attack ISP's.
All right.
So let's start with point to point protocol over ethernet or PPPoE, which is a layer
two encapsulation protocol.
This.
This is a common protocol to connect to ISP's.
And in our case, the PPPoE server is the ISP's brass.
And the PPP client is our DSL modem or malicious machine.
And remember that we just saw that abstractly the client is connected with an ethernet cable
to the brass.
So as the name of the protocol suggests, by using PPPoE, we can encapsulate PPP sessions
over ethernet frame.
Okay.
Great.
But what is PPP?
All right.
So PPP stands for point to point protocol.
It is also a layer two protocol, which is mainly used to tunnel ISP packets to and from
the modem.
And by doing so, enabling the internet connectivity.
So PPP negotiation is where the clients authenticate to the server and get its configuration parameters.
And if all goes well.
So smoothly, a PPP tunneling interface is created on the client side.
At this point, the client received DNS configuration and usually an internet facing IP address.
So at the end of negotiation, our router or machine should end up with an interface
similar to this one.
All right.
Also at any time, both the client or the server can terminate the session.
Okay.
Yeah.
And that's it.
Now that we understand the general flow of the protocol, I'd like to focus on PPP and
especially on the session negotiation part of PPP.
So PPP is a layered protocol that has three components.
First, there's an encapsulation component that is used to transmit datagrams over a
specific physical layer.
I'll soon go over on the specific format and how it looks like.
The encapsulation is used to transmit something called link control protocol, LCP.
This protocol establish, configure, and test link as well as negotiate settings, options,
and use of features.
And lastly, after the connection is established, different protocol can be used to negotiate
and facilitate a layer-free network layer.
In our case, to tunnel IPv4, and we use for that the IP control protocol or IPCP.
Since PPP is a relatively big and have many protocol layers, I decided to focus on LCP,
and this is why I did that.
Well, this is the first protocol used in PPP.
It's used to set and receive different configurations, and no authentication is needed.
Okay, so to really understand PPP, let's understand the encapsulation format.
So first we got an Ethernet frame with a PPPoE payload.
The PPP encapsulate a PPP payload.
Okay.
Okay.
For example, the encapsulated PPP packet is LCP type of packet.
Now let's understand how LCP packet looks like.
LCP packet contains an option payload to pass different parameters.
For example, authentication protocol, magic number, maximum received units, et cetera.
And each option has a code number, length, and data.
For example, here we see that LCP packet that contains parameter for maximum received units,
authentication protocol, and a magic number.
And, yeah, this is the different option values for all these fields.
All right.
And there are all sorts of other parameters defined in the protocol, but the one I found most interesting was actually the last one.
Okay.
Code number 19, endpoint discrimination option.
I don't have enough time to get in what this option does
and actually soon you'll see it doesn't really matter.
But yeah.
But I discovered that although the RFC defines
that the parameter length should not exceed 20,
in the smart edge devices,
it actually handles packet with bigger length.
And it also has no validation on the data field.
Also, I noticed that when the PPP log is enabled
in the smart edge device,
I get log entries similar to these.
Here we see the different LCP parameters written to the log.
For example, this is the authentication protocol used
and this is the magic number
and this is the AAA,
as part of the endpoint discrimination field.
So, I'm writing a log entry with any data I wish
and with a bigger length than accepted.
Yeah, I think you all know what's coming next.
Yeah, vanilla stack overflow in the smart edge log entry.
And this is a great time for demo time.
Okay, so for this demo I will be using four terms.
The bottom blue terminal is only a monitor terminal
connected to the smart edge device.
Yeah, this is the monitor.
And the red and the green terminals on the left
will act as the STD in and STD out listeners
for the reverse shell.
And on the black terminal, I will execute my attack
connected as a DSL subscriber.
Okay, so now I'm running netcat to listen to STD in
and STD out on two ports.
Yeah, here I'm using the first one and the second.
All right, and now I will execute my attack
by piping binSH to two telnet clients.
So here I'm piping the STD in
to port
1337 and then pipe binSH and STD out to the other port.
Okay.
Fire away.
So now I'm sending the first two PPPoE session,
creating a PPPoE session by sending two packets.
And this is the LCP payload, malicious payload.
And you can see on the monitor screen that I got a core dump,
meaning that I managed to smash the stack.
And now I'm using my STD in netcat to send commands.
Here I'm using LS.
And of course, I'm gonna echo my user
and see that I am root.
And yeah, let's do it again just to, thank you.
Thank you.
I can also grab some other configuration like the DNS,
and I can write and read the DNS.
DNS configuration and the admin for the device itself.
But yeah, once I got root, it pretty much ends.
Okay, so now that we've seen a full working RCE,
let's talk about how to research this kind of equipment.
So the most interesting conclusion I got from this research
is that ISP network equipment is not that different
from home routers when it comes to vulnerability research.
Let's talk about the differences.
So home routers are, of course, cheaper,
usually around 300 bucks,
while brass entry level is a bit higher.
Both of them usually never get updates from the ISP,
which is great for hackers,
but really bad for everybody else.
And as you already heard, brass are way noisier.
So if you plan to store them at, on your kitchen table,
you should expect to annoy people around you.
And probably the hardest part is that setup
and configuration part.
Since ISP equipment usually takes a specific expertise
to install.
So if you got enough, so if you got some extra dollars,
and you're stubborn enough, you could also
pawn an ISP network equipment.
So based on my experience with this research,
I'm thrilled to present my seven easy steps
to research and pawn ISP equipment.
First, firmware emulation.
Then setting up debug and development environment.
Jailbreak if needed.
Get or buy an actual device.
Search and hopefully find vulnerabilities.
Write an exploit.
And finally, celebrate with your favorite beverage.
Okay, so first step for every embedded device research
is usually getting the firmware.
I was lucky enough to find one online,
and after a quick bin walk, I realized Smart Edge
uses a NetBSD OS and a PowerPC architecture.
Luckily, up until 2006, Apple, which is the biggest company in the world,
were using PowerPC processors with their BSD-based kernel.
And this nice fella, called Kurenai,
posted a wonderful Reddit blog on how to emulate
a very similar system to the one Smart Edge are using.
And that way, I was able to emulate the user space
of the Smart Edge firmware.
Right, now for debug and development.
I realized that cross-compilation to a database is not a good idea.
A different OS and a different architecture
is pretty much a nightmare.
So I decided to use my QEMU machine
to just compile statically tools.
I also used this SSH TCP dump trick
to sniff packets from my emulated device
into my Wireshark host.
Well, that was super useful to understand
the different protocols I'm presenting.
But I also had an issue.
An issue with debugging.
My GDB multi-arch on my Ubuntu machine
refuses to connect to a PowerPC net BSD system.
So instead of spending time solving this issue,
I just decided to run the GDB client
from my emulated environment.
And yeah, this actually was very useful later on
when I remotely debug an actual device.
So it just saved me some time.
Okay, so next step was to understand
if a jailbreak is needed for debugging an actual device.
So as you can expect,
SmartEdge are using this exec underscore CLI binary
to handle all console commands.
So logging from Telnet, SSH, serial ports,
they all end up with this jail terminal.
But exec CLI must run some kind of other binaries.
For example, when you use the Telnet command in that shell,
it executes this SE underscore Telnet binary.
By the way, this is the same Telnet client
I was using for my exploitation earlier at the POC.
Luckily, I found that this Telnet client had, you know,
it has an internal Telnet command
for just popping up a jail-free sub-shell.
So yeah, so another thing that,
please note that this is a Telnet client.
So to run the Telnet internal command
to pop the jail-free shell,
I use this trick where I do Telnet to my local host.
And here you can see, they invoke sub-shell.
And here you can see, they invoke sub-shell command.
So all I have to do is just use the exclamation point command
and pop myself a jail-free CLI.
Now that I know that if I'll get my hands
on an actual device, I can execute any code I desire.
For example, this GDB server that Redback
was kind enough to leave in their firmware.
Right, so next I decided to buy an actual device.
Theoretically, I could continue my research
on emulated device and maybe even find vulnerabilities.
But I was missing the actual device configuration.
I also had to do some serious LD pre-load voodoo magic
to get the emulation working,
which made it limited to specific binaries
and it was pretty unstable in general.
And also, if I have an actual device,
I can run it on my own.
And I could develop a full working exploit
like the one I demonstrated.
Right, so I went to eBay and bought the cheapest device
I could find, which is the Smart Edge 100.
It cost around two grand.
And Aleph Research was kind enough to fund this purchase.
By the way, at that point, I left Aleph Research
and moved to CyberArk Labs.
So I finally got an actual device.
But I soon realized that I got little knowledge
in setting up ISP equipment.
And I had to acquire some kind of ISP technician skills.
So I started by reading this 100 page of basic hardware guide
to understand how to physically connect stuff.
I then had to learn how to connect stuff.
I had to read this 360 page of basic configuration
to understand what configuration I need to do
in order to create an ISP-alike setup.
And lastly, to work with the CLI,
I had to use this 900 page manual.
And this is how I pretty much felt
from all this useless information
that I'm gonna use only once.
Yeah, okay.
So to apply all this information,
Let's first go over the device Sashi.
Most of the ports on the device
are either ethernet or optical ports.
To these ports, the ISP connect other network equipment
such as switches, routers, and DSLAM.
And most importantly,
the subscribers are connected through these ports.
So my attack should be executed from one of these ports.
The other ports are used for managing the device.
So we got two ethernet management ports
and a single serial port.
So there's two ways to configure and manage the device.
The most convenient is of course the ethernet ports
by using Telnet or SSH,
but these ports can also be used for other monitoring
such as
log monitoring, packet monitoring,
and even remote debugging with GDB.
But the serial port is still very useful
since it's foolproof.
Meaning that even if I completely messed up
the device configuration,
I could always reverse it with the serial port.
And believe me, I have messed the device configuration
like a million times.
All right, so finally I was able to configure the device
with around 200 hours.
200 configuration commands.
And I was ready for a full ISP-like installation.
My setup included an Ubuntu machine,
three network cards, and a single serial port.
So I started by connecting a serial port
to the foolproof console.
I then used one network device to connect
to the management device port.
And then connected another network interface
to simulate an actual subscriber connected to the brass.
This is where I executed my vulnerability through this port.
And lastly, I installed the device in our server room
and connected my Ubuntu machine to the internet
so I won't have to work from my kitchen table no more.
Okay, so now that I own a smart edge device,
and I understand how to configure it,
and I got myself a server rack,
I finally became the proud owner of Evil Communication LTD.
So you guys are more than welcome to become my victim's
head, to become my clients, of course, free of charge.
All right, but beside becoming an ISP,
I finally reached the research phase.
So now I had three useful tools
to help me with the research.
The smart edge log, a GDB server,
and a Wireshark monitoring.
And now let's talk about the daemon itself
that handles PPPD.
Smart edge is running a daemon called PPPD
that handles PPP sessions.
With some reversing, I realized that incoming PPP packets
are being handled by a function called packet receive.
Packet receive calls a function called vmux input packet.
For every packet it receives.
And this function is a log stream multiplexer
for input function.
Basically, it means it uses function pointers
for different log, depends on
what sub PPP protocol is being used.
Here we see three logs protocol functions.
For example, the LCP, the IPCP, and the NLCP.
And as you guys already know,
I was interested in the LCP log.
And this is where I discovered that
the endpoint discriminator log uses this unsafe mem copy
to the stack.
So if we go back to the LCP options payload example,
this is where my mem copy copies the data I control
with invalid length to 35 bytes of stack array.
All right, and finally, it's exploit time.
So, exploit development was very straightforward.
There were zero stack mitigation, and I chose to use a single
structural orb gadget to exec VE.
But when I was using the reverse IP shell
in my demo, I was making a naive assumption.
In the demo, I assumed that the brass has some kind
of IP connectivity and was able to create a reverse shell
to the remote server.
When the attacker can control the brass by entering a single add end node,
And then the attacker can control the brass
by connecting to that remote server.
But what if the brass has no IP connectivity
or it blocks by a firewall?
How can I get a shell on the device using only Layer 2?
So I decided to try and develop a Layer 2 only shell,
meaning the entire shell will be executed over Ethernet frame
without the need for IP protocol,
meaning the subscriber can control the device directly.
So the first problem was that my exploit
was limited to command shells only, like we saw on the POC.
So first, I had to execute my exploit to deploy a bridgehead.
Bridgehead is a term for running small exploits
to update a bigger exploit.
So if I was looking.
Sorry, I was looking for a way to execute a shell code that
will allow me to upload files using only Layer 2
communication.
Luckily, I had tcpdump installed on the device,
and I found this very cool trick to upload a file
from a specific MAC address.
Here we see that the TCP writes the content of 106 frames to a file called,
L2 shell.
And I also used a filter to only pick up frames with a specific
magic MAC source address.
So now I am no longer limited to a shell command.
I can upload any binary I want and execute it by running my exploit again.
But since I'm dealing with Layer 2 connection, I needed some sort of
raw socket functionality.
Unfortunately, Smart Edge implemented their own undocumented raw socket,
so I couldn't just use it easily.
But fortunately, slash dev slash BPF was installed.
And ironically, the only binary using slash dev slash BPF
was the tcpdump I just abused.
So I had to configure BBF with these two flags.
The first is telling BPF which interface it's using.
And the other flag is just receive the packets immediately.
And now BPF listen to alert to traffic.
And if it detects a frame with a specific magic,
it extracts and runs its codes, the specific commands.
And now I'm truly done with exploiting the device.
But wait.
There's more.
I want to show you something.
I'd like to share another interesting vulnerability discovered by Omar Tsarfati,
one of our researchers at CyberArk Labs.
For this, let me explain how PPPoE session works.
So every PPPoE packet has a code field.
This field sets what type of message is being used in the PPPoE negotiation part.
So the first phrase is not mandatory.
And it's mainly needed for the client to discover the PPPoE server MAC address.
It does that by sending an initiation packet and receive an offer from the server.
And the next step is mandatory.
And in this phase, the client requests the session number to start a session.
So the client sends PADAR.
AKA request.
And receive PADAS.
AKA session from the server.
Also it's very important to notice that both sides can terminate the session at any time
by using PATI.
AKA termination.
So Omar discovered that when the client sends PADAR request with a broadcast as a source
address.
The SmartEdge PPPoE server.
sends a session packet with a broadcast as destination server destination
address since no client expects an FF FF as destination address it sends it
again and again and if the server is configured with session timeout it sends
a termination packet but this termination packet has a broadcast as
destination MAC address meaning the server asks all the clients to
terminate their session
thank you there is a bot here there is a there is a bot here that this attack is
it's it still depends on the switch policy and whether the DSL endpoints
determine if
if the ADSL endpoints terminate the session if they receive such kind of a
request I mean a broadcast termination with no session number but yeah but
anyway it's a it's a bug for sure and a really cool idea for an attack yeah and
now that we managed to pawn our smart edge devices all it left is to drink rum
with the fellas here at DEF CON by the way this is my noob shop
from 2020 all right and as for fingerprinting so I managed to detect
around 500 devices in 55 different organization from 20 different countries
using that were using the smart edge devices but remember folks these devices
are not publicly did not supposed to be publicly facing so I'm sure there are
many more
out there and I reach out to Ericsson with my finding and we work together to
reproduce and understand the issues and their severity Ericsson suggested to
handle the CV assignment and they provided a CV number for the stack overflow with a
critical CVSS score and another CV for the denial of service bug but I haven't updated slides yet
with a medium CVSS score.
Ericsson also announced that smart edge devices
has reached end of life,
meaning these vulnerabilities are infinity day.
And we also brainstormed together on possible mitigation,
and they communicated the issues to their customers.
And as for mitigations,
well I strongly recommend to disable PPP logs
on the smart edge devices.
As for Omer Dinalov service attack,
I believe it can be blocked with switch configuration
to block this kind of messages.
And also smart edge, well they are really old devices,
they are from the back 2000s,
and it's highly recommend just get rid of them
and replace them.
All right, so to conclude,
I hope I convinced you that ISP network devices
are not that different from home routers
when it comes to vulnerability research.
That old school is still cool,
and a 50 years old network device
can still cause a big ruckus.
And as always, ISP usually don't pay attention to updates
when it comes to both their IT and their network.
They support theiravaitingsendpoint
and their back-end equipment.
Right, looking forward,
a blog post will be published soon,
and I might also look into other attack surfaces
I've discovered while researching the smart edge devices.
And lastly, I might have a look at other vendors
and see if some of those technique works,
I mean the denial of service works.
Alright, thank you for your time, feel free to reach out to me on Twitter and go see the
latest season of Solo Opposites, it's awesome, and if you've got any questions feel free
to ask.