Let's give Jay Legorio a big welcome. First time speaker at DEF CON. Come on!
Man after my own heart, we're digging into reversing here, breaking apart some
encryption. So let's go. Good evening DEF CON and happy 30th. And I'm sure this has been
asked like a dozen times today, who's first DEF CON? Oh, that's awesome. Okay, that's great.
So welcome. Glad you're enjoying it. There's a lot more for you to see tonight. Hacker
Jeopardy is here after this. I think it's in a little bit, but that's the next thing. So let's
get this started. This talk is called Tear Down the Xywall, Breaking Open the Xycell
Encrypted Firmware. So we'll talk about how I got into this, into the different analysis that
went into it. There's a hundred different ways to do it. There's a hundred different
hardware, software, some firmware stuff going on, where that gets us later, and then what's
left for other people to do after they take up this mantle. So who am I? My name's Jay. I went
to school at UMBC in Maryland, and then I went to school again in California. And I have a
CISSP and some various SAN certs that I'm sure have expired, but they were fun. And when I think
about myself, I think, like, I'm a Windows developer, so you're going to see a lot of Windows
stuff in here, even though this is very much the same thing. So I'm going to talk a little bit about
this. I'm a security consultant, and I'm also a licensed private investigator in D.C. And
although I'm a first-time DEF CON speaker on the main stage, I've spoken at Hackfest in Quebec.
That was super fun. I got to talk about making artificial pancreases at DEF CON 27 in the
biohacking village, which, by the way, is not a good village. It is an excellent village. So if
you can get over there, you definitely should. And I got to do sort of a re-up on that talk with
some updates at Colonel Con 2, which was in March of 2020, and very suddenly virtual. But, boy, did
they pull that off in, like, two weeks' notice. It was good. And, yeah, I'm an unrepentant nerd.
So it was December 2020, and we all just got through the first year of the plague. And things were
looking up. Vaccines were just over the horizon, and things were looking a little hopeful. But then,
like usual, it all got ruined.
Because I got an e-mail. And so the text here is kind of small, but what it says is, hey,
we're ZYSL, we make your firewall, and we know that you diligently updated and patched for the
last quarter, just like we asked you to. And we know it's only been two weeks since you installed
that patch, but we've got this thing called an undocumented user account, and this is the fix
for it. And so when I think undocumented user account, I think, hey, we've got this thing called
ZYFWP, and they just fucking backdoored all my firewalls. You've got to be kidding me.
And so but maybe it's not that bad, right? So you go look up the CVE, and you go look at the research,
and, yeah, yeah, that's exactly what they did. So that was super cool. If you've got the username
ZYFWP and the password was out there, you could log into any ZYSL firewall of this series, and that
kind of seems terrible.
So all this is bad, right? But I'm nosy, and I want to see the bug.
And even though I've not really gone into reversing the ZYSL firmware before, I've always kind of
thought about it, so I figure, why not? This is a great opportunity, and it's not like I'm doing much
else. So even though it didn't affect me directly because, and I'll quote from the article, the
account seemed to work on both the SSH and web interface, I don't expose those to the Internet.
But what's to stop someone from getting into the network?
So I'm going to go back to the network a different way, and then turning around and enabling more
access via my backdoored firewall. So, again, all this seems pretty bad, and what makes it more
interesting is that at the time, there was 100,000 to 300,000 potentially affected devices.
So with all this, let's take a look. And I assume it will be easy, because everything that I go into
starting with seems like it's going to be easy, and then it definitely isn't. But what do we see here
right off the bat?
It's an IPK. That's a really good sign for us, because that means it's a zip file. So great news.
Let's open it in 7-zip. But 7-zip says there's a password, which is weird, because I've worked with
these firewalls for a while and never had to know a password. So we've looked at this for about
five minutes. What do we know so far? We know it's a zip file. We know the file's password
protected. We can see the files inside. That's how Zip Crypto works. And some of those files are
available online, we think, maybe. And the update process doesn't require a password. And the other
interesting thing is that 7-zip says there's a trailer. And I don't mean a trailer like, oh, five or
eight bytes got added on to the end because of some weird download thing. That's 124K. That's not
unintentional. So what do we do first? Default passwords. ZYSL has been raked over the coals for
ZYSL123. It was the first thing I tried. It was the first thing I tried. It was the first thing I
tried. It was the first thing not to work. Tried a few others that other people suggested, and none
of that worked. So we can see the files inside the archive. Maybe we can do something with that.
And what I tried next was called the Zip Plain Text Attack. And this is not novel. It's not new.
It's from 1994. And what you can do is if you have one file from outside of the archive, you can
derive the key stream for the rest of the files in the archive with, like, math and magic. I
don't know. It's not my thing. But I tried it. And, yeah, it did not work. And one reason that it
didn't work, oh, no. There's nothing at the FTP site. It's just gone. My go-to for years for files
from ZYSL is just gone. And what happened was between when I submitted this paper and got
accepted, they took that service offline. So I don't have that to show you because it's gone.
And it's replaced with a thing where you need to log in and give them a serial number and register.
So all of those files are unavailable. But maybe somebody else has a copy. So we go to archive.org.
And it's not, I mean, granted, great, great folks. It's not the most updated thing. This is from
2014 in the June timeframe. But let's at least give it a try, right? It's something to go on, and
it's better than what I got. So we can use something called BK Crack. It's available on the
Github. And it didn't work. And I'm pretty sure that what I had was just the wrong files because
they have to be, you know, byte by byte identical in order for this to work. So I have one more
strategy to try. You all might be familiar with it. Our very last resort, password cracking.
And I tried lost password. If you've used it, it's very pointy, clicky, easy to use. It's great.
And John the Ripper, which is less easy to use, but really fast and really good if you can get the
hang of it. And they were both touch-and-go. So I'm going to show you a couple of examples.
I'm not going to be flashy about the trailer, but it gave me a couple of days to brainstorm.
And when I was done brainstorming, I just felt depressed. So this doesn't feel great. I don't
really know where to go here until I realized what I have was a chicken and an egg problem.
So I've got encrypted firmware and a device that knows how to decrypt it. So somewhere the
firmware is decrypted, and it just so happens that I'm a hoarder. And so I have the previous
generation of this device on the shelf, so maybe we can bench it and pull something out of it.
So now we'll move on to the hardware. This is the original device in the rack. Not going to win any
cable porn awards or anything, but there it is. It does its job for the most part when it's not
backdoored. And so this is a typical MIPS box. You can see a serial port up at the top. You can see
the Ethernet ports down at the bottom. And where I've highlighted in red is a separate board
where the storage goes. And this is the top view. It's your standard Samsung NAND flash chip.
And, yeah, we're just going to bend that back. And I cut the hell out of those pins. I hoped
they weren't needed later. And what we've got is a 10-pin connector. We've got this
Phison PS2251, which might look familiar to some of you. And, yeah, whatever the hell this is.
But it turns out that the 10-pin connector is the same thing that you'd see on your regular desktop
motherboard to connect external USB pins. And the Phison chip is your typical NAND to USB storage
chip that you see these on thumb drives all the time. And so I looked for a USB hub that I really
didn't care about because this was going to be a bet. This was going to be fraught. And so you just
shove some pins into the header and solder it in. And then you just plug it in. And then you just
solder them on to a USB cable that I just cut in half and hope the magic smoke doesn't come out
when you plug it in. But the drive mounted and I had a disk, which honestly surprised the hell out of me.
So you can use, like, Win32 disk imager to image the entire drive. You get all of the partitions and you
throw it into binwalk and then out pops the firmware in plain text, which is great. And so we're not
done yet, though. Because when I kind of get to the end of this, I'm going to go back to the
software. I want to set out for this, I didn't want the firmware from this one device that's now in the
trash. I wanted to decrypt these arbitrary firmware files whenever I got them off of the ZYSL website.
But now we have a starting point. So let's look at the software because now we have access to it, right?
The device runs the Apache web server. Pretty typical. You submit the firmware to firmware upload
CGI and CGI.bin and the first thing you do with that is run strings. And it's got a really interesting one.
ZLD FS extract. Looks pretty promising. So we'll throw it into Google and we get one solitary result.
And it's ten years old. So it could mean nothing. But it says USG firmwares are zipped with a very strong
password. Okay. That's, yeah, I guess so. I mean, the password cracking sure didn't work. And it says
the ZLD FS extract command line verifies the zip and unzips it. Yeah, okay. It also says watch the
console output during the update and I didn't do that because, again, I just cut this thing off the board.
So that wasn't really an option. But it looks like we're on the right path. Now, again, I took the old device
that I had on a shelf, right? The devices have the same processor architecture. The firmware formats are
similar. Encrypted zip.
Trailing bytes. And so there's nothing on the Internet to say how to break into this firmware.
So, like, I don't know. From a vendor's perspective, if it ain't broke, don't fix it. So what's to say
that it ended up changing for the new model, which is what I was actually interested in?
Now, the first thing I wanted to run on this binary ZLD FS extract is file. Just give me something to go on.
And the thing that I really hated was that it's statically linked and stripped. And that means that I
can't find things like memcopy, stircopy, the normal library functions that you would want to see
because it's going to help you reverse engineer the rest of the thing. And maybe that doesn't hurt us so bad,
but it didn't feel great. And the next thing that I ran was strings. And I got the best one that you could
ever hope for, a usage string. This is great. It tells me where to start when I want to submit a file to this thing.
And the second best news is that this thing calls 7ZA, which is 7-zip. So, again, feeling good. I'm on the right path.
Now, let's look at our other friend here that's going to help us with our software reverse engineering.
And that's Ghidra. Now, if you've been living under a rock, it is a free disassembler and decompiler from the
National Security Agency. You may have heard of them. And we can see a function that is building a command line.
And part of the command line is C.
7ZA. Okay, great. That's where we want to start. And the other thing that it adds to the command line is
dash P. And that's a really good sign because after dash P, that's where it's going to drop the password.
So this is the function that calls the function that generates the password.
Now, I didn't want to try and emulate the entire device just because that seemed hard. And I'm lazy.
I don't know. And so QEMU user mode emulation was great. It jumped right into action.
So the only little trick that I had to do was I had to grab a library that you see that dot SO off the device
and just link it to somewhere else that was in my Windows subsystem for Linux build so that it could run.
And then when you run it, you add S trace to it so that you can see the different syscalls that it makes
and maybe try to suss out some information.
So speaking of information, my first run at this, you can see I added dash I because part of the usage seemed to indicate
if I do that, it'll just tell me some information. It won't try to extract anything.
And I thought that was a good starting point.
But just when I was going to go look for where it ran 7-zip, I saw this here where right off the bat,
we open the file for read and seek to 124K from the end of the file.
Now, I'm not saying I don't believe in coincidences, but that should look familiar.
So I think we're on the right path here.
Now, again, I don't believe in coincidences.
I tried to run dash I because I thought it was going to build this command line and have it be 7-zip the whole thing.
But what I actually found was that I control that parameter entirely.
So if you look at this exactly call here, it tries to run dash I.
And that means I can run whatever I want in there.
And then after dash I, you have dash O, dash Q, and dash P and that really long string.
So we could plug a tool in here that spits out the password.
Since we have direct control of this parameter.
And earmuffs, goons, earmuffs.
I didn't tell the conference that I was going to release a tool today.
So I'm going to put it on the screen and it will be our little secret.
This is the most complicated tool I've ever released.
Put the camera down.
Please.
This is hard work.
All right.
We're going to print out the fourth argument.
And it works.
So you can feed the extractor any first argument.
This is a firmware file.
And quickly output the password for easy extraction with 7zip.
And I think we got there.
So that's success.
And you can see the password is there multiple times because instead of, I don't know, being efficient,
it runs 7zip for each file it wants to extract.
It doesn't just dump the entire archive.
And those dots there on the left side, if you see that, that's what would otherwise be just progress dots, indicators.
And, yeah.
So that's really all you had to do.
And there's the command line to do it.
You can pass any xysl.bin file there, including the one that was released, I think it was like a month ago.
Or maybe a couple months ago.
So can we see the bug yet?
No!
It turns out the backup of my firmware got corrupted, which was awesome.
Because this bug exists in one but only one version of the firmware now.
Not the one before it and not the one after it.
And it's just gone.
Xysl really effectively memory holed this backdoored firmware from the internet.
I feel really good about my Google foo.
And it is just gone.
So I kind of sat around and said, you know, what the hell do I do now?
And I thought I'd DM the researcher.
His name's Neil to sync on Twitter.
And he sent over his copy.
For the USG 40.
And even though this wasn't the model that I had, that's fine.
I didn't want to load this onto a real device or anything.
And so I extracted it like this.
And since we already know the IOCs, we can just grep for them.
And it's pretty easy.
So you can see we grep for ZYFWP.
And the first two files are winners.
Shadow.basic and password.basic.
And remember, this is a firmware for a device that maybe has never booted before.
So these will become your ETSI Shadow, ETSI password files.
And that hash right there in Shadow.basic, that's a good hash.
You know, we're not looking at MD5 over here.
And the other thing that kind of struck me was imagine Capwap serve, which is an executable.
And that seemed a little weird.
So throw that into Geiger2.
And what you get is a function that loads up this.
This structure called FWinfo.
And it plugs the username in, ZYFWP.
And then it populates the password field.
And this password is otherwise really strong.
So nice job, guys.
But, you know, maybe don't backdoor all your firmwares with it.
I don't know.
That's just me.
So continuing on, this is for someone else who might want to pick up that mantle.
Now the first thing I would do if it were me.
I would make a Python script.
And here is where the deobfuscation function is.
And here's the function that calls it.
I am bad at MIPS.
Boy, I cannot read that stuff.
So I leave that as an exercise to somebody who's way better than that.
But I would turn the deobfuscation algorithm into a Python script so you can just point it at the firmware file and get the password.
And I do hope somebody does this because it would be way better than this Rube Goldberg machine nonsense that I've kind of cobbled together here.
And I did also mention that I'm not releasing a tool.
But if I'm giving you some offsets and saying go forth, how are you going to do that if you don't have anything to start with?
Because remember, I cut this thing open.
Well, it turns out it was all way easier than I made it.
So the firmware files, the firmware releases include the firmware files, the release notes, and the .RI file, which is a recovery image.
And you can blast that over a serial port if you've got your firewall in a bad state.
And it's unencrypted, so you can just throw it into binwalk.
And it's got FS extract, which is great.
Now you all have FS extract, and I'm not responsible for that.
And I want to thank HN Security for pointing this out.
It should have been obvious, but, you know, I don't claim to be good at obvious things.
And so if you want to go that route, no firewall.
The files need to be harmed in the process of your research.
So here's their screenshot of the binwalk.RI file.
You can see ZLD FS extract is right there.
And so, yeah, now we thank our firewalls for their sacrifice and protecting me from just looking at what was literally right there.
And the other thing that I would do is I would run the user land image through EMBA.
EMBA is a really great firmware analysis tool.
It's pretty new, and it's gotten really good, even in the short time it's been alive.
So it'll generate this really slick report at the end that shows everything that it found, and it can find a lot.
So it started out as this text output only tool, and it would generate, like, 15 different files of all kinds of different categories of findings, which was good, but, I mean, it was dense.
And so now it's got this really sweet web-based output that you can browse locally, pointy, clicky, look at everything.
And from our process above,
it can do everything after the unzip process, including the automated binwalk, the check against known CVEs, and even emulation of the target binaries for the whole spectrum of static and dynamic analysis.
So let's look at what things it found in this older firmware.
Now, the only thing I love more than star copy is mem copy.
And the only thing I love more than that is a binary that'll call system and has one or two of those.
So I think libnet snmpmibs.so looks pretty interesting if I'm looking at the Venn diagram of what calls those things.
And the other thing that it can look at is file permissions and just bad decisions related to those.
So I see a lot of files that can suid to just whatever permissions they want.
And some of them make sense, but others definitely don't.
So I'll give you suexec. Makes sense.
xisudo.suit.
Yep, on board. That makes sense.
WebAuthError.cgi. That's a weird one.
Dynamicscript.cgi. That sounds terrifying.
FacebookWifiAuth.cgi. Why does that need to be root?
And the two that I don't really have any questions over is ping and traceroute.
I don't know why ping and traceroute need suid permissions.
Maybe, again, I'm not a Unix person. Maybe somebody can help me with that.
But that's it.
That seems like you're just asking for trouble.
So the other thing, this is the last screen that I'll show that I really like,
is it'll catalog your CDEs and give you a little more information about those.
So, like, it shows these three. They're known exploited.
It'll give you a whole bunch of 10 out of 10s if you're looking for those,
and it gives you the CDE number there.
But my favorite part is if there is a POC out there, it'll point you right at it.
So what this helps you in other products with is if you are otherwise running
the latest firmware, but it's using old libraries that are exploitable,
this is the dead giveaway that you might have something there to look at.
And so it turns out EMBA on WSL might not be a long way off,
because I was working with the developer, Michael Messner, on how this can be done.
And the older version of EMBA was really, really powerful and very good,
but the newer version has a dependency on Docker.
And WSL on Windows 10 has some manner of incompatibility.
So it's not on them, but the immediate answer was to load up an Ubuntu VM on Windows 10.
And that felt maybe like a heavier lift than it should have been.
But they added some improvements, not only for that,
but also based on the decrypted user mode images that otherwise would have been inaccessible
to them without running through all this.
And the next thing, or I guess the last thing that I'd pay attention to,
is when XISL starts changing things.
So I love this diagram in a prior talk, and I couldn't find it again,
so if you know who made it, please tag me on Twitter or whatever so I can cite them,
because I really love this kind of paradigm here.
Because in theory, we're over here on the left where we can get the plain text version of the format.
And then there'll be a transition, and then it'll be encrypted again,
and we won't be able to see anything.
But to get here, in the middle, the firmware they've published
will have to understand the format changes going forward.
And it will itself be vulnerable to this process here.
And there'll be dead giveaway language to indicate which version has the new stuff.
You can only install version X before you can install version Y and beyond in future updates,
because firmware before version X won't know how to handle the new update mechanism.
Because it won't conform to the old scheme.
And so once you trace the upgrade flow here in this transitional firmware,
you'll know how to decrypt the new protection scheme,
and for whenever that happens, I leave that as an exercise for the listener too.
Alright, so this went by as I was making these slides,
and I think Stack Smashing is presenting this year somewhere here.
But I thought it good to point out,
and I guess I've seen some bad firmware encryption,
like static XORs, like one-byte XORs, or maybe even four-byte XORs,
because they're super clever, or rolling XORs,
none of which is encryption, but you get my point.
But I guess what we have here isn't the worst,
but man, if you're shipping backdoor accounts,
I don't know, maybe Dave's not wrong.
And it's just kind of window dressing.
So with that I say, happy bug hunting in this firmware.
Format and others.
And just to kind of recap what we went over,
I talked the tiniest bit about myself,
and how I got into all this stuff.
We covered a few different methods of analysis to figure all this out,
and what that got us,
and what continuing forward for you could get you in the future.
And before I take any questions,
I want to thank my friends and family,
and especially the DEF CON CFP review team,
because, I mean, without them,
I just wouldn't be here.
And Neil Stasink for responding to a message from an internet rando,
literally out of nowhere.
And Michael Messner for EMBA,
the author of every other tool mentioned,
because I kind of, you know, burned through a lot.
And I'll call them out separately too,
the Geiger Project,
because decompilers are free now,
unless you're an American,
because you damn well paid for that shit.
So, and with that, I'll take any questions,
but I thank you for your time tonight.
Thank you.