NAND flash chips, a little different than standard flash chips,
be a little more problematic. So we're going to talk about that and kind of step through with
a number of demos. So great, let's go ahead and get started. So thanks everyone this evening or
this afternoon based on where you're at in the United States. Presentation is NAND chips,
introduction to NAND chip extraction and data reconstruction. This is an intro primer.
We're going to cover a number of different topics and we have like three different demos.
I think you'll get a lot out of this. So my name is Daryl Hyland. I'm the Principal Security
Researcher for IoT at Rapid7. So section one, we're going to just give a basic introduction
and look at structure. So what is a NAND flash memory? Simple, you can read it. It's non-volatile
storage technology that doesn't require power to retain the data. Pretty straightforward.
See those references there? If you're interested in learning more in the nitty-gritty,
take a screenshot of that real quick because that is great references. I go back to these
quite regularly to kind of refresh myself because some of this stuff can get pretty deep
as you're trying to reconstruct data in a number of cases. So go ahead and get a screenshot of
that. I'm getting ready to jump off of it now. So a couple of things. When we start thinking
about NAND flash memory chips, you need to go to the data sheet. You really need to understand how
things are laid out. This will play a part in recovery a number of times. So this is the memory
array structure that lays out the page, the blocks, and the planes. In this case, the page is
2K in size or 2048 bytes. And then there's what's referred to as an OOB area, a 64-byte area at the
end. And this is critical. So what is OOB? It's out of band. It's a spare area. It's used for
containing error correction code information, bad block information, and it exists in every page.
So what does that mean? Generally, what it means is you have all your data jammed full of
potential garbage when you're trying to parse it out and rebuild the data. And this could be
impactful and can cause problems in some cases. So we're going to talk about how we're going to
fix that, and we're going to show that in a couple examples. The OOB area, the out-of-band area,
is also laid out in two formats. One is known as adjacent, and one is known as separate.
If it's laid out, the OOB is laid out adjacent. So a 2048-byte page would be laid out with a 16K
or 16-byte of OOB after every 512 bytes. This would be an adjacent layout.
The most common I see is what's known as separate. A separate area is the 64 bytes of data exist at
the end of the 2048-byte page. This is the most common. But often you need to verify that
as you're working through this and trying to remove the data or correct the problem.
Another thing that comes in handy, and this is in reference to a number of the tools that
we're going to use today, your chip ID code. An example, anytime you use a flash memory reader,
so you pull a chip out, drop it into a memory chip reader, often it will, when you set it up
for that chip, it associates that with a ID, and it'll read the chip, compare the ID on the chip
to make sure it matches before it tries to read the data. Now, a lot of applications out there have
a use for this type of chip ID data. This is out of the datasheet also. And in this example here,
it's showing the datasheet for a S34MS02G1 chip, which is a 512-megabyte NAND flash.
And you can see at the bottom, it shows what that chip ID is for that chip. And you'll see that
we'll use this later on, and we'll probably come back and look at it in the datasheet again when
we get ready to do that. So, at this point, are there any questions? So, Jonathan, has anyone
posted any questions or anyone not have any questions about some of these basic concepts?
Let's see here. So far, it looks like no questions posted here. One quick question that is actually
coming up here that I'm seeing. Whenever you're, I guess, pulling firmware from NAND flash,
do you have like a consistent philosophy that you follow, such as like always first see if you can,
you know, first pull, maybe if you can like log into it via like serial port through UART?
Or do you typically go straight for attempting to dump the NAND flash content?
Yeah, usually the first way if I want to get the data, the most effective way is get the data
when it's in a constructed fashion. That would mean if I can get a console on the device with
the entire operating system running, and the entire partitions for all the data laid out,
that's easier because then I could easily DD the partitions off the chip in their constructed
fashion. And that would avoid me almost always having to do what we're going to talk about here.
What we're going to talk about here is in that case where you actually have to read the data
raw, right off the flash chip, either chip off where you remove the chip and drop it into a
reader, or some other methods where you may be able to clip on or hook onto the device and read
the raw data out of the chip. And that's when you get this structure that we're going to be dealing
with today, and how to clean it up and a few methods of recovering that. So I shout out to
everyone, if you're interested in interacting or asking questions during this, join the Zoom chat,
the Zoom channel that we have set up for the webcast. Once you're into there, you'll be able
to post questions up during the presentation, and hopefully we'll have time and we'll be able
to answer them. So please do that if you can. It makes this way much more interactive. Instead of
just being a data dump, we have a chance to actually interact with the audience and the people
and learn more in this process. So I'll go ahead and move on. So section two is a general
chip readers. Since I talked about chip off from the question Jonathan gave us,
we get into chip readers. So when you're interacting with a NAND flash chip, here are
the signals you're typically going to need to interact with. And they go everything from the
8-bit data channel, the bi-directional channel, all the way down to input signal, output signal
structure, and some other various pieces. So if you end up buying an off-the-shelf chip reader,
this kind of solves all these problems. But there was some work done out there
by this gentleman here. I would recommend taking a look at this write-up. If you're interested in
building your own chip reader, this one actually works against a TSOP48 NAND flash chip. And he
uses a technique known as bit banging using this FTDI board that he has here. He's able to bit bang
the data off of the actual chip. So I would look at looking at what bit banging is, because I'm
not going to explain it here, and look at how he's set this thing up. If you're the hacker mentality
and you like the idea of building things your own, I would check this out. It's kind of cool.
If you're like me and you just like to throw money at it, these chip readers are reasonably
inexpensive. The TI-866 Plus, you get all the sockets with it. It's like $130. This will read
a number of NAND chips. Pretty good. And it uses a TSOP48. Right below it is a socket that doesn't
come with the set that you need to buy. This socket is for from TSOP32s to 48s. It's known as
NAND08. This is for larger memory chips. So anytime you encounter larger NAND flash memory
chips in the TSOP48, the one that comes with the kit will not work and only work with smaller
memory blocks. So you need to get the socket for that. The ProMan isn't made, this model
wasn't made anymore, but they do make a similar version. So if you search for ProMan, you'll find
a similar version out there. The socket below that is designed for 1.8 volts. The ProMan is
typically a 3.3 volt device. And we're starting to see way more chips in the 1.8 volt arena. So
it's good to have that add-on. Most of these add-ons are $8, $9, $10 on both of these chip
readers. So they're fairly inexpensive. The thing I like about the ProMan is if I can't identify a
TSOP48, I can't find the data sheets or not, well, let's say I do have the data sheets and I can't
find any information on none of my chip readers. They don't identify it, they can't detect it.
This will let you hard set page OB block sizes into the actual tool that comes with the application
that comes with this, so that you can attempt to read it using that kind of raw information.
And it's been successful several times with me, making it possible for me to read a NAND flash chip
where I had the data sheet to identify that structure, but none of my chip readers would
recognize the chip. And the last one's RT809. And that one's a little more pricey if you buy it
Amazon, but I think you can track that one down on AliExpress cheaper. I like this one. It's not a
bad chip reader, and it has, you can buy a lot of different sockets for it. Most of the sockets
that come with the TL866+, the one on the left, will work on this, all but the TSOP48 sockets.
Because the TSOP48 sockets have a circuitry built into them, so they won't work on this reader. You
have to have a straight pin through. And the one I have below shows that TSOP48 socket that's a
straight pin through. I also recently had to pull a NAND flash that was a BGA ball girder array,
63 balls on the bottom of it. And I was able to buy this off AliExpress for the RT809 for about
40 bucks, which isn't bad. If you get into the high-end chip readers that run, start off at like
$1,600, $1,700 and go up for them, the sockets for them start off at $100, $200 and go up into the
thousands for BGA sockets. So, these things are very cost effective. The sockets are not high
quality, so you have to take care of them. But they're overall, they're pretty good products.
And I use them quite regularly in my lab without issues working with NAND chips.
So, any questions in reference to the chip readers and chip reading process?
One question here. What are the most challenging NAND packages or pitches to work with that you've
encountered? Is there any package or pitch you would not attempt messing with?
I would never say there's nothing I wouldn't attempt messing with. I would literally mess
with everything. Probably some of the biggest problem is, obviously, if you're using a chip
reader, try to read this. The BGAs or ball grid array devices are the more problematic,
because there may not be a socket available for some of these devices. In cases like that,
if you have the data sheets, you can always go to the process of dead bugging. That is turning the
ball grid array chip on its backside and soldering small 40-gauge wire to the actual balls, and then
feed that into a connector or a chip reader or a device for doing bit banging to actually be able
to do that. I have been successful doing that a couple times, but it's a pain in the butt. It's
way much easier to pull the chip with hot air or IR if it's a BGA, clean it up, drop it in some
device, close the lid, and have it give you all the data. But typically, if I need the data off
it, and I can't get it any other way other than chip off, I will chip off from the board. And if
necessarily, I will dead bug one. I mean, yeah, me and you, Jonathan, we worked together at one time
where we dead bugged that one weird device. We had a 0.25 millimeter pitch BGA, and we had to
solder 40 wires to it or 20 wires to it to get it hooked up. So yeah, we're going to try anything. I'm
not going to pass up any potential target if it has data on it that I want.
Makes sense. And you're definitely bringing back some crazy memories with that. And actually, you
partially answered the next question as well. The next question was, what hot air tools do you
recommend for BGA removal?
Actually, I have both hot air and I have a IR oven. I like the IR oven. Once I got it, it's a small
it's a small device, it'll take a board that's probably eight by nine, eight by 10 inches or
something like that. To me, that's the sweetest thing in the world. I usually just stick it in
the oven, run it through the cycle that I brings it up to the temperature. And I take a suction
tool, I pull the chip off of it, and then I let the whole board cool down. Because, you know,
it heats up the whole board, you have to watch out for things like plastic components that are
on the board. So you may have to remove those prior to putting them in the oven. But you can
usually pick up one of these IR ovens. If I remember what was the name of the one I got,
it's spelled P-U-H-U-I. It's a Chinese made product or Asian made product oven that I have.
I think it's a model 2962 is what it is. Small oven. I think I only paid like $130, $140 for it,
somewhere around there. And there's a number of products out there like that. But an infrared is
way much better, in my opinion, than hot air. So... Another question. Sorry, Daryl.
Elvis Cousin asks, is it hard to reball BGA without killing it to re-solder?
Yes. It's not that hard. It's just a process of learning how to do it.
Not to drag this all out, but I had a project where I wanted to learn how to do this. Because
I wanted to get root access on a device. It had 153 ball BGA embedded multimedia controller on
device. So I used hot air. I pulled it off. And I had a couple of devices because I wanted to,
in case I killed one of them. The first one, when I put the balls back on, there's a couple
ways of doing it. You can use screens with paste, or you can use just the balls. In my case,
I just used the balls. I floated on there correctly, but I overheated the chip and it
looked like a chocolate brownie when I was done. That's when I learned that you don't want to go
on these BGA chips. You don't want to go over about 210. Well, melting temperature is about
183 degrees Celsius, if I remember right. And you don't want to go over more than 25,
27 on the temperature in Celsius. If you hit 215 or higher, it starts causing the
fiberglass substrate of the chips to start breaking down. But if you keep it within those
range and you don't shock it by overheating it too quick or cooling it too fast, it'll work.
And I was successful. The first one I burned up and I failed. The second one was perfect.
Since then, I've never had a problem ever working with BGAs, either pulling them or putting them
back on without damage. And once I learned those temperature parameters. Okay, so the next section
we want to move into is going to be demo. So, we're going to get off into the demo area and see
how that works out. Hopefully, we won't have any technical problems. So, what we're going to do here
is we have a flash memory that's been pulled. This has been pulled from... let's go ahead and look at
a couple of them here. It's pulled from an Arlo device, which is a doorbell, if I remember rightly.
It's actually a kind of like a ring doorbell type product. Its memory chip was a W25N01GV.
So, typically when you pull this thing, what's the first thing you want to do is binwalk.
And again, binwalk works many times, but sometimes it does not. So, we'll give this a try
and see what happens here. Hopefully, we won't run into any real technical problems.
And this is kind of the common flow. We see we hit a squash file system. We should have at least
two squash file systems on there if it's coming off a standard embedded device. And there's a
second one. And then it kind of hangs right there. If we sit here, this thing would sit here in idle
for hours and not produce anything. So, we're going to stop that. So, if we change over to
the extraction from binwalk, we can see that it opened up a bunch of stuff, but then there's a
squash file system. So, as we notice, there's nothing in there. And truthfully, because it
went past the squash file system file, it should have put something in there. So, as we can see,
literally nothing in there. So, for the most part, this didn't work. This is actually a fresh rebuild
of binwalk. And also, I have a binwalk pro account. The results are pretty much the same
in this particular case because there's some things that have to be dealt with here. So,
let's go ahead and remove this structure just so I don't start running into space issues by
blowing all this stuff out of here. So, what do we want to do? Well, as we talked earlier,
we talked about the actual OOB area. So, we want to be able to remove that area. So, we have a tool
in here called NandDump. And at the end of this presentation, I'm going to have links for
everything, which will cover all this stuff. So, here we can actually do input files, output files.
Remember that ID I mentioned to you, chip ID? We can use chip ID, or we can specify actual page
size, OOBs, and the layout. Remember, adjacent, separate. So, in this particular case, we are
going to... well, before we go ahead and separate the OOB out of here, let's go ahead and look at
the data sheet. So, as we look at the data sheet, we see that it is 2048 bytes by 64. So, it is in
a separate format. That means the 64 bytes is at the end of the page. That's a good sign. That
tells us the structure. You always want to go to the data sheet. But typically, I always like to
dig a little deeper and make sure that the chip hasn't been kind of messed with. And why I say
that? Because just because the data sheet says this is how the chip's going to be laid out,
the manufacturer that actually used the chip, I've seen them do some strange things, to say the least,
to the data. Meaning that, you know, you go and remove the OOB area, thinking you know where it's
at, and literally have it not be laid out the same way. So, we're going to look at this
with HexEdit. So, we're going to look at the raw data of the actual device so we can see what it
looks like. There we go. So, at this point, we should be able to go to 2048, which would
be 800 in hex, and it should be the beginning of the second block. So, we can see that there's
something here. It looks a little different, but it may not be. We're not quite sure. It doesn't
really tell us a lot. So, we could go to the next block. So, for example, if hex is 800,
and we want to, of course, we want to go, remember, is 20, gosh, we get it right here.
So, if we go 2048, we'll get it cleared here in a second. Try it again. 2048,
which would be 800, and we want to go up to the next beginning, which should be 2,112 bytes.
Remember, 2048 plus the 64, and we look at this, it should be at 1040. So, if we go to 1040,
we still see some interesting things, and this may tell us that this is correct,
but I've seen a lot of devices where the actual OOB area was nothing but all Fs and wasn't
actually configured. So, what we're going to do is let's look a little deeper so we can see some
patterns, because I'm always a big fan of spotting patterns. So, we come over here,
and get it going, and we're going to search for strings so we can get some data. So,
here we are. So, we're in an area where there's a lot of strings,
and let's scroll down until we see some patterns.
And quickly, we see a pattern right here. So, got a pattern right there. So, if that is...
So, we go, hold up, 2112, and go. So, it'd be 840. So, we go 840 plus E880. E880 equals F0C0. So,
the next pattern should be an F0C0. And we can see that we're starting to see that pattern there
also. So, as we can see, more than likely, we are witnessing the 64 bytes at the end of the 2048
as the OOB area is showing up. And again, if we look over in the string area, there's no common
words, because I would not expect that to be in the area. So, let's scroll on down to the next
one. There's another thing I like to do sometimes, especially if I'm having problems positively
identifying this location. So, this will involve a little bit of math. So, I like to use a
calculator for that. So, if we come over here, and we put this in here, like that, and we turn it to
decimal. And then, because the programming calculator does not round off or have decimal
points, we want to do this basic. So, what do we know about the overall structure? The first OOB
area or OOB area showed up after 2048. So, we had the first 2048. And then, to get to the next
leading ones, we had 2112. So, we go ahead and subtract. And hopefully, my math works here and
doesn't mess it up. 2048. And then, divide it by 2112. And hopefully, we'll get an even number. And
we did. If you do not get an even number, then this is not mathematically correct for an OOB area.
So, what we're seeing here is 0500F900 is actually the 39,749th page of data in this particular
device. Okay. So, now that I punished you with some math, we're going to go ahead and strip that out
of it. So, we have this command called NAND Dump PI. We're going to use .I Arlo for the input. The
layout's separate. Page size happens to be 2048. The OOB size is 64. And then, the output will be
Arlo No OOB. Hopefully, this runs. And it finished up. So, there, once we finished up, we have a file
that's a little smaller, as you can see, where the OOB's been stripped out of it. And let's go ahead
and run this and see if this works for us. The first squash system. You see, the pattern's
different now. And then, we also see that there's an OOB structure also starting to show up.
Often, you'll see inside an operating system on embedded device, you'll see squash file systems,
which are typically read-only. OOB file systems allow read and write. So, it may be an area that's
used dynamically that's being created that can use to read and write data into the device. Okay.
So, that one actually finished. So, let's go ahead and jump over here and see if it worked for us.
A little different there. We see we got three folders. We have an OOB root system.
So, we go to squash root. And there you go. We were actually able to extract this data
by effectively removing the out-of-band data within the NAND flash dump. So, do we have any questions?
Yeah, we got a couple of questions here, T-Dub. So, Elvis Cousin asks,
I have two BGAs on a daughterboard. Is it likely I can read each individually,
or will I need to reassemble them somehow to try to get a squash FS?
And they're also both Intel-based flash.
Oh, you're saying you have two flash memory chips on the device.
Really comes down to how they lay those out.
I would not expect them to stripe across separate chips like that. I've never seen that.
Often, when I've seen two chips done up like that, each one will contain a different partition,
or they may be redundant of each other. But typically, they'll be partitioned
separately on embedded. That's the way I've seen it.
And I've seen everything from where one chip actually contained the boot,
U-boot, and the kernel, and the other one contained the file system,
as an example. So, the only way is to pull them off, examine the data,
and make a determination from there to see if you can do it.
Of course, the best solution is if you can get a root console on the device somehow,
then obviously, you could figure that out more quickly and DD the partitions off that way.
Which is also the sweetest way of doing it. But if you have to chip off,
yeah, you still may have to do some analysis of them.
Yeah, you kind of pivoted off that, said, both are connected to a common connector,
but I don't have a pinout to try to recover a cross connector by soldering something.
Yeah, it's kind of hard to say. If you have some kind of pictures or images of the data,
or not the data, the chips and the layout and the board, shoot them to me sometime,
and maybe I can take a look at it and help you come up with a plan of attack.
Like I said, anyone's feel free to reach out to me anytime.
I may or may not have an answer for you, but I'd be more than glad to give you a hand if I can.
One final quick question here that I'm seeing. So, in this particular case,
it looks like you're able to find the data sheet. Have you ever encountered a NAND chip
that you wanted to dump but couldn't identify it or could not locate a data sheet for? If so,
how did you approach that? Yes, guessing.
Often, if I can't find a data sheet actually for that, I've actually gone out and looked at
other chips that may have data sheets that are in that same series or produced by that manufacturer
and figure out how they do their layout. Even though I may not be able to
quickly identify the chip ID, there are ways to actually enumerate those, and it
usually typically need a data sheet to do that. But guessing does work, and crazy as can be,
I've actually dumped a lot of chips by going, hey, here's a manufacturer, and it seems like
most of their chips, or a large portion of them within this structure range, are laid out this
way. They have this page structure, this size. Often the naming conventions, like here on the
screen, W25N01GV, that's one gigabit chip. And often in the name, you can figure out what the
chip is a lot of times. So, that would be eight bits, 128 kbytes or whatever. So,
it's doable just looking at the name sometimes to pull some typical information out of it,
and then guessing on some of the other stuff. But again, I've also smoked chips
by doing weird things to them that obviously didn't work. So, got to take some caution
and take some risk. But amazingly enough, it can be successful. One of the things to do also is
kind of look at, if you can get a UART console onto the main processor, let's say the main
processor has a console, even though you can't get root access to it, capture all the debug data
from the boot up. That can also be very revealing sometimes. Okay, time to move on.
So, we've kind of done that. We're able to get the data off that. I'm going to go ahead and
clean this up. Because we have a couple exercises to go to and I don't want to run out of time.
So, okay. So, the next one we want to look at... I'm also going to RM the article. So, the next
one we want to look at is the Nest. This actually came from a friend of mine who had his Nest
thermostat crash out. It just died. He threw it my way, said, hey, can you get any interesting
data off of it? So, and I struggled with getting data off of this, but it was eventually
successful. We're going to show some of this. There was no confidential data actually extracted
from the device. So, there's no problem with us using it here. But so, what we want to do
is what we're going to use is a process where we're going to create a NAND sim.
So, what does that mean? We're actually going to simulate a NAND chip in memory. We're going to
use like ModProbe. If you haven't used ModProbe, ModProbe will let you load kernel modules onto
the system. So, we're going to build a block device using ModProbe kernel modules. We're
going to load those up and then we're going to write the data out to the chip. We're actually
going to tell the data with NAND write to use the OOB data to reconstruct any error corrections
and that type of stuff. And then we'll mount it up. So, this particular one, I actually ran
Benwalk on this and it never recovered anything. I actually shot it out to good friends at
that do the Benwalk Pro. And unfortunately, I didn't get back to the system for a couple
weeks and it churned out there. So, I apologize to them again for eating up several terabytes
of data. So, this thing was constructually a pain in the butt and running Benwalk was
potentially catastrophic. It would eat up all your memory and produce nothing. And I'm not
going to run it here, but if you run it, every page or every header for a GIFs to file system
or every header for one of those, it would create a folder for it, put nothing in it,
and literally eat up terabytes of data in a couple days quickly.
So, let's go ahead and get going here. So, we're going to load ModProbe MTD,
ModProbe Block. This will help us set up a block device. And then we're going to go ahead and
we're going to load this up here. So, the way this works, ModProbe NANDSim, BCH is
error correction functionality. This was the chip, it uses a 8-bit structure. There's a BCH16
for a 16-bit structure, if that's actually being used. And this here happens to be the
chip ID. So, if we go over to this particular chip, that is not the one I want. So, this is the
actual chip. And I believe we go to page 25. I have this memorized. Okay. And this would be the
read ID, the ID of the actual chip. We know this chip is a 2-gig chip. It's organized in 8-bit
structure. It is a 1.8-volt bit chip. So, 01 is the first one, AA, 90, 15. 44 isn't used in this case,
but it deals with error corrections and multi-plane information. Multi-plane information,
just to make note here, if you have a chip that has multi-planes on it, and you'll see it in the
datasheet, it may or may not stripe across the two planes, meaning it puts one block on one plane
and one block on the next plane versus putting them contiguously on the same plane. I have not
encountered this, but I have read about it. So, it's something to be noted. So, at this point,
we're going to go ahead and load this up. So, now we've loaded up that particular module, kernel
module, and now let's go ahead and just look at some information, mtdinfo, and this will tell us
a little bit about what we just created. So, we created a simulated. It's how many bytes?
256-megabyte chip, 2048, subpage size, 64 bytes, pretty straightforward. And then we go ahead,
and this one may show some errors. Sometimes it does, sometimes it doesn't. And we're going to
look at this within DMessage. And if we come down here, we can see that it took the manufacturer ID,
chip ID, and information, and actually have identified the actual chip. That is the chip ID
that does that and helps lay this out. So, the next thing is we want to be able to write into it.
So, we're going to use NanWrite, which is another tool.
Hope you don't mind me not typing. It saves you guys a lot of headaches. So,
let's cancel out of that. NanWrite. I want to see the command. So, here's all the functions. So,
what we're going to do is we're going to input contains OOB data. So, we're telling it to
utilize the OOB data. And we're going to go ahead and let this run. And hopefully,
it'll run through clean. Boom! There we wrote all of the blocks out onto the device.
So, the next step is... we'll come over here so you see this. We go ls.
And we can see there's nothing there. And then what we're going to do is cut and paste. So,
I don't have to type. We're going to mount this up on block zero. Change directory to gifs.
And voila, we have data. So, if we look through here, it starts looking pretty good. Hey,
we've recovered all this data. The sad thing is most of this stuff is system logs.
That we recovered. But if we get up here toward the top, we find some weird things.
Here's bin. It's a file. etc is a file. Well, truthfully, this is bullcrap.
We know that's not a file. So, if we cat etc, wow, look what we get. Now, I wouldn't expect
to see that in etc. So, we recovered a lot of it. But some of it's corrupted in a weird fashion.
So, how did I recover from this? I experimented. And we're going to go over that a minute. But
before we go forward, I actually want to see if there are any questions. Jonathan, any questions?
Yep, taking a look now. No updates here. I'm checking Twitch real quick.
Yes, here's a question. In IoT devices you've examined, do you ever come across
NAND storing non-file system data? If so, do you just dump and then use binwalk
and other usual tools and methods for poking around?
Yeah, I've had them storing non-file system data on NAND flash. So, yeah, typically I get into that
and when I get those and there's no structure to them, I start off with throwing things like
strings to the thing. With strings, can we identify any known structure in there?
I will run binwalk just to see what it'll do. Sometimes it'll point out some structure headers
that I can more quickly find. So, I think that plays a big role. But yeah, sometimes you'll have
them storing things like blocks of configuration data. And I've seen them just blocks of encrypted
data, which may be a file system structure, but they're not framed as such. So, it's just
encrypted blocks of useless information. So, yeah, that's not all that in common. Well,
most of the time when I get a NAND chip, it's a file system, but you will encounter those,
to be honest. So, right now, before we move on to the next section, I want to remove the
kernel modules so they don't conflict with what we're doing. Okay, and we are down to...
we still got some time, doing pretty good. So, the next section I want to get into,
we are going to use MTD RAM to actually do this. So, we're going to take that same system,
that same dump that we just did, where we mounted up the file system,
it did not come out exactly the way we want it. And we're going to mess with that. So,
we're going to make a make directory, and we're actually going to create our own block structure.
And then we'll go, and then we're going to make a node.
If I learn how to spell it, I should just copy this over, make node dev slash MTD block.
And we're going to create zero. We're going to create it as a block device.
We're going to give it a master number, a primary number, and a minor number of zero.
So, we're going to create that device in there. And then what we're going to do is we're going
use modprobe, and we're going to bring in a kernel module called MTDRAM. So, this kernel
module's MTDRAM. We're setting the total size to 274432. That is bigger than a 256 megabyte RAM area.
256 megabyte RAM area would be 262144, and that would not contain OOB data.
In this case here, we're bringing in the OOB data also. Because if I bring in without the OOB data,
I noticed that we get the same results as the other one. So, experimenting, and I encourage
you to experiment when you're dealing with these NAND chips. It can be quite fascinating what you
can find and figure out and make work to get you data out of this system. I'm currently working
on one. It has me slightly befuddled, but I will master it eventually, where the actual entire data
on the chip is shifted 10 bytes. They even shifted all the OOB data 10 bytes, and it took
looking at it manually to figure this all out and doing that math and looking at the data
in the structure to determine that they had also shifted that. Because I was getting block errors
saying that block is 10 bytes longer than it's supposed to be type thing. That's when we found
out that everything was shifted and was able to confirm that the OOB was shifted too. But I digress.
So, what we're going to do is we're going to load this up, and it's supposed to contain the entire
thing. And then we want to load one more module. MTD block for kernel module. And then we're going
to dd. Excuse me. And dd is a bit copy. I use this all the time. Matter of fact, if you get access
to almost every embedded IoT device, you'll find dd is actually installed on it. Because most of
the time, dd is actually used for doing firmware upgrades. You'll have duplicate root file structures,
duplicate kernels on the device. And what happens is one is primary and running. So, what will happen
is when the firmware comes down, it'll dd it over the secondary, and then it'll change the U-boot
or secure boot arguments to make the new one primary and the old one secondary. That way, if
you have a failed firmware update, it does not break the device. It just goes back to the original
one, which was not overwritten. So, again, input file here is the nest file. And then output is to
our block device that we created. And if I did everything right, this should start writing data.
Should be fairly quick. And as you see, we're able to write out 277 megabytes of information
fairly quick, because it's all internal to the system. So, from here, we're going to go ahead and
mount this up. Same way we did on the other one. And we're going to mount MTD block zero up.
Now, if someone wants to explain to me why this worked, because I really not sure why.
As I said, often dealing with NAND chips, little quirky things have worked for me. So, I'm not going to
speak bad of them. And now we can see etc and bin are actual files. We do have some
corruptions in here. So, there's a few files actually corrupted within this. But for the
most part, a large portion of them were actually able to be recovered properly.
And so, these are methods that I use quite regularly for recovering. When you start
working with UBI file systems, if I ever get some good methods, methodologies hammered out
for working with some of the strange UBI file systems, like the one with the 10 bytes that's
shifted. I did one on an engagement here earlier this year, where again, nothing are large portions
of it were not laid out contiguously and trying to build good solid methodologies around UBI.
I'll probably come back around and put together a training session presentation for that down
the road. That's what I'm currently working on, trying to work out those kinks and bugs and build
that knowledge level to build to work with us. Now, probably half the chips that are UBI file
systems, I'm successful recovering them using various techniques and UBI tools that are available
out there. But typically, GIFs, squash file systems, things like that can easily be quickly
recovered from NAND flash chips using any one of these three methods that we covered today
in this presentation. So, hopefully there's some questions out there.
Yeah, I got a couple here. So, one question here asks, have you encountered an IoT device
that was encrypting the data it was storing in the NAND flash?
No, I haven't, which is kind of amazing to me. I mean, the only time I've ever seen IoT
technology using any kind of encryption, if it was running an Android operating system.
And by nature, Android can support the data section being completely encrypted.
So, that's quite common. I do have some other tech that I consider IoT, but it's more enterprise
IoT and not necessarily the normal consumer grade, which actually is very much encrypted
and most of the, not all the file system, but most of the file system that contains user data,
similar to the Android file system, is also encrypted. So, typically, if I see it's an
Android, it's usually 50-50. If it is a standard phone-style Android operating system dumped on
there, then most always it has encrypted. But yet, if it's an older Android style that's put on
there, there was no requirement or typically that's not always encrypted. So, it's a mix and
match, but it's pretty rare to see encrypted, at least from a lot of the IoT I've looked at,
unless you get into more enterprise level technology.
Anonymous attendee asks, Hi, I am new to this. Sorry,
you already have the binary file. How did you extract it from the hardware device?
In the cases of these, both of them were chip off. So, early in the presentation,
and so you can rewatch this presentation, or in the presentation earlier today,
where we talked about building an IoT lab and we showed a number of devices,
I have a number of chip readers. So, I actually use those. Other ways are
often if the chip that's being used has some kind of debug function, you can put it in debug
and pull the data off of it. Also, you may be able to check, connect in with JTAG or
SerWire debug and actually read it off the device. But if those fail, I often rely on just
desoldering the chip and dropping it into a reader and extract the data from there.
If I need to put the device back into functionality, then I'll go through either
resoldering the chip back on or reballing the chip and reflowing it back on an IR oven,
which are fairly cumbersome, at least the reballing BGAs, but it's still doable.
And it's not all that complex. I did a presentation not last year, but the year before last
at Nuit de Hacq. I did it there. I also did it, oh gosh, I did it in Perth, Australia,
and I also did it at DerbyCon. And it was, I'm trying to remember the name of it. Basically,
it was how to get root access the hard way. And it talks about slashing, smashing,
cutting the device apart and rebuilding it to get root access or something to that effect.
So you can look at that, and it goes through some of the concepts
in my learning process on learning how to manually reball BGA chips.
Elvis Cousin asks, these are all variants of Linux. On a Windows CE device,
might it still use SquashFS or some other Windows thing?
Ooh, probably some other Windows thing. I have not looked at a Windows CE device.
I have been looking at some Chrome devices. In those particular cases,
this has an SD drive, solid state drive, where you're able to get the data off of it and just
mount it up on your system. I'm not sure if they have NAND chips. Is it going to be
a Linux file system? Probably not. It's probably going to be some variation of Windows. But I have
not tore into any Windows or Windows CE type devices to be able to give you a solid answer
on that. Let me know if you find out. Any other questions? Nope, all caught up.
All right. Well, I think we're... what time we start? Six o'clock. So we're probably right on
time. So let me jump over here. Let's do a final stop to share. So let's go ahead and
ask question one more time. Anyone have any questions, any requests for questions or
information? If not, you'll probably catch me on Discord either here in a little bit after I've
eaten or late tonight or tomorrow. Feel free to hit me up for any questions. Also, oh gosh,
yeah, I was promising that. Hold on before you all drop off there. Let's get this up.
So I had promised... slideshow. So I want to be able to get some of these things captured in the
video. So here's the NAND sim example that I use. The mod probe commands, the NAND write commands,
and the mount to be able to mount this particular GIF file system up.
And here is the commands that I use for mtd-dram example. Quick note on some of the commands we
use. There is the link for NAND dump tool, concept of mod probe, make node, NAND write, DD. And then
here is the reference. So that gets captured in here. All of this is kind of cool material.
There's a lot of great work that's been done out there on NAND chips. Like I said, I'm constantly
reviewing and reading and consuming this material on a regular basis because every time I encounter
NAND chip that wants to fight with me, I have to go out and start rethinking,
okay, what am I missing? What do I need to learn to expand my knowledge? How do we kind of move
forward from here? And there's been a lot of work done out there. And of course, we shared
much of the information that these people provided in this demonstration today. Also,
if you're interested in connecting up with me on Twitter, my handles are %X, so it would be
P-E-R-C-E-N-T underscore X. And feel free to connect with me out there or reach out to me
and ask questions out there. Awesome, Daryl. Thank you so much.
Oh, man. Sure enough, Sam. Had a blast. Yeah. And I mean, honestly, if there are any last
questions, we still have a little bit of time. So was there anything else that you saw there,
Jonathan? Yeah, another quick question just popped up. It says, this is some very epic stuff,
extracting flash from NAND chips. I have to ask, did you find anything epic while performing
post-exploitation on the extracted firmware? Yeah, yeah. I mean, any remote shells? No,
unfortunately. But again, it's information that we could feed back to manufacturers to help
improve processes. You know, they get into hard-coded stuff. Mainly, a lot of this lately
was purely an effort to go, hey, what happens when IoT is thrown in the trash type thing?
You know, is there data is like the guy with the Nest thermostat? What happens when something
dies if you throw it away? Does it contain PII information? So this was kind of a lot of it was
driven out of that and some paid assessments. And also, you know, we tested a lot of devices from,
hey, if you go ahead and flush the thing back to factory, does data stay on it? We kind of
started checking for things like that. It's all about a privacy risk and issue, because we've
gotten really big on the whole idea that, hey, we don't want to throw our hard drives away.
But in this modern world, these chips are our hard drives. And we need to probably start thinking
about the potential risks there. So that was kind of the main purpose of a lot of this,
and the fact that I had been working on some engagements that involved band chips that were
problematic. In those cases, we were working on things like reconstructing keys to gain access.
A lot of this stuff ties into engagements also, where we're looking for data since we're testing
an entire ecosystem. That isn't just the hardware, but it may be mobile apps, cloud services,
and other devices. What can we get off this chip that could allow us to attack everything else
or attack somebody else using the same product because of key reuse and key session keys and
password reuse and things like that. So a lot of it's within that direction.
Another question here. This individual asks, does it seem like it is possible to be able to create
a module so that a chip reader would automatically be able to detect those out of the OOB areas of
the flash memory? Oh, a number of chip readers will let you identify. Some of them will allow
you to extract the chip, the data, without the OOB. So yeah, it's not uncommon. You can
often manipulate data from the chip reader and include or not include OOB on some chip readers.
Obviously, you get more control as the price of the chip reader goes up.
And I don't necessarily have a $2,500 chip reader in here. But I think one of these actually had
some OOB settings. But for some reason, I didn't necessarily trust it because I told it not to
include OOB. And then when I visually looked at the data, the OOB data was there. So I kind of lost
trust. I have a tendency to also like to map it out, not let the chip reader make the determination
for me that this data is structured a certain way. Because like I said, it's not uncommon with
vendors to futz with it, so that it may be incorrect, or that the OOB shifted or is not
used in the same place. And they've just written a new controller to read the data the way they
made it. This way allows me to go, hey, this is correct or isn't correct. And then I can easily
manually remove it or modify code or scripts that are out there to take it out the way I
wanted it removed. So that's kind of my take on it. That makes sense. I think that we're at the bottom
of the list here for the questions. Okay, well, I think that kind of concludes it. If anyone has
any questions, catch up with me later, or reach out through me on Twitter. And again, I'll be on
Discord probably tomorrow and maybe later tonight. So I'll catch everyone later. Have a good night.