start	end	text
640	6460	NAND flash chips, a little different than standard flash chips,
6460	11120	be a little more problematic. So we're going to talk about that and kind of step through with
11740	18320	a number of demos. So great, let's go ahead and get started. So thanks everyone this evening or
18320	23380	this afternoon based on where you're at in the United States. Presentation is NAND chips,
23380	29760	introduction to NAND chip extraction and data reconstruction. This is an intro primer.
30000	34800	We're going to cover a number of different topics and we have like three different demos.
35020	39720	I think you'll get a lot out of this. So my name is Daryl Hyland. I'm the Principal Security
39720	48500	Researcher for IoT at Rapid7. So section one, we're going to just give a basic introduction
49080	58120	and look at structure. So what is a NAND flash memory? Simple, you can read it. It's non-volatile
58120	64340	storage technology that doesn't require power to retain the data. Pretty straightforward.
64700	69860	See those references there? If you're interested in learning more in the nitty-gritty,
69860	77000	take a screenshot of that real quick because that is great references. I go back to these
77000	82280	quite regularly to kind of refresh myself because some of this stuff can get pretty deep
82920	89400	as you're trying to reconstruct data in a number of cases. So go ahead and get a screenshot of
89400	95300	that. I'm getting ready to jump off of it now. So a couple of things. When we start thinking
95300	103780	about NAND flash memory chips, you need to go to the data sheet. You really need to understand how
103780	110780	things are laid out. This will play a part in recovery a number of times. So this is the memory
110780	119980	array structure that lays out the page, the blocks, and the planes. In this case, the page is
120740	128820	2K in size or 2048 bytes. And then there's what's referred to as an OOB area, a 64-byte area at the
128820	135780	end. And this is critical. So what is OOB? It's out of band. It's a spare area. It's used for
135780	144500	containing error correction code information, bad block information, and it exists in every page.
144500	151100	So what does that mean? Generally, what it means is you have all your data jammed full of
151100	156660	potential garbage when you're trying to parse it out and rebuild the data. And this could be
156660	163100	impactful and can cause problems in some cases. So we're going to talk about how we're going to
163100	169220	fix that, and we're going to show that in a couple examples. The OOB area, the out-of-band area,
169220	177040	is also laid out in two formats. One is known as adjacent, and one is known as separate.
177160	186800	If it's laid out, the OOB is laid out adjacent. So a 2048-byte page would be laid out with a 16K
186800	194200	or 16-byte of OOB after every 512 bytes. This would be an adjacent layout.
194920	202720	The most common I see is what's known as separate. A separate area is the 64 bytes of data exist at
202720	210640	the end of the 2048-byte page. This is the most common. But often you need to verify that
211220	216080	as you're working through this and trying to remove the data or correct the problem.
216080	221540	Another thing that comes in handy, and this is in reference to a number of the tools that
221540	230340	we're going to use today, your chip ID code. An example, anytime you use a flash memory reader,
230340	237360	so you pull a chip out, drop it into a memory chip reader, often it will, when you set it up
237360	244720	for that chip, it associates that with a ID, and it'll read the chip, compare the ID on the chip
244720	250820	to make sure it matches before it tries to read the data. Now, a lot of applications out there have
253080	259660	a use for this type of chip ID data. This is out of the datasheet also. And in this example here,
259660	268260	it's showing the datasheet for a S34MS02G1 chip, which is a 512-megabyte NAND flash.
268260	273800	And you can see at the bottom, it shows what that chip ID is for that chip. And you'll see that
273800	278760	we'll use this later on, and we'll probably come back and look at it in the datasheet again when
278760	286860	we get ready to do that. So, at this point, are there any questions? So, Jonathan, has anyone
286860	292340	posted any questions or anyone not have any questions about some of these basic concepts?
293560	300500	Let's see here. So far, it looks like no questions posted here. One quick question that is actually
300500	308340	coming up here that I'm seeing. Whenever you're, I guess, pulling firmware from NAND flash,
308340	314080	do you have like a consistent philosophy that you follow, such as like always first see if you can,
314080	319780	you know, first pull, maybe if you can like log into it via like serial port through UART?
319780	324280	Or do you typically go straight for attempting to dump the NAND flash content?
324280	331140	Yeah, usually the first way if I want to get the data, the most effective way is get the data
331140	337700	when it's in a constructed fashion. That would mean if I can get a console on the device with
337700	344720	the entire operating system running, and the entire partitions for all the data laid out,
344720	351560	that's easier because then I could easily DD the partitions off the chip in their constructed
351560	356680	fashion. And that would avoid me almost always having to do what we're going to talk about here.
356780	361040	What we're going to talk about here is in that case where you actually have to read the data
361040	367160	raw, right off the flash chip, either chip off where you remove the chip and drop it into a
367160	372720	reader, or some other methods where you may be able to clip on or hook onto the device and read
372720	378500	the raw data out of the chip. And that's when you get this structure that we're going to be dealing
378500	387590	with today, and how to clean it up and a few methods of recovering that. So I shout out to
387590	395390	everyone, if you're interested in interacting or asking questions during this, join the Zoom chat,
395390	400570	the Zoom channel that we have set up for the webcast. Once you're into there, you'll be able
400570	405670	to post questions up during the presentation, and hopefully we'll have time and we'll be able
405670	411430	to answer them. So please do that if you can. It makes this way much more interactive. Instead of
411430	416010	just being a data dump, we have a chance to actually interact with the audience and the people
416550	424030	and learn more in this process. So I'll go ahead and move on. So section two is a general
424510	431050	chip readers. Since I talked about chip off from the question Jonathan gave us,
431050	437530	we get into chip readers. So when you're interacting with a NAND flash chip, here are
437530	443550	the signals you're typically going to need to interact with. And they go everything from the
443550	449530	8-bit data channel, the bi-directional channel, all the way down to input signal, output signal
449530	456070	structure, and some other various pieces. So if you end up buying an off-the-shelf chip reader,
456070	460070	this kind of solves all these problems. But there was some work done out there
460070	467030	by this gentleman here. I would recommend taking a look at this write-up. If you're interested in
467030	474490	building your own chip reader, this one actually works against a TSOP48 NAND flash chip. And he
474490	483050	uses a technique known as bit banging using this FTDI board that he has here. He's able to bit bang
483050	490510	the data off of the actual chip. So I would look at looking at what bit banging is, because I'm
490510	497410	not going to explain it here, and look at how he's set this thing up. If you're the hacker mentality
497410	501950	and you like the idea of building things your own, I would check this out. It's kind of cool.
502950	508610	If you're like me and you just like to throw money at it, these chip readers are reasonably
508610	517770	inexpensive. The TI-866 Plus, you get all the sockets with it. It's like $130. This will read
517910	526410	a number of NAND chips. Pretty good. And it uses a TSOP48. Right below it is a socket that doesn't
526410	533890	come with the set that you need to buy. This socket is for from TSOP32s to 48s. It's known as
533890	541110	NAND08. This is for larger memory chips. So anytime you encounter larger NAND flash memory
541110	546710	chips in the TSOP48, the one that comes with the kit will not work and only work with smaller
546710	553110	memory blocks. So you need to get the socket for that. The ProMan isn't made, this model
553110	561890	wasn't made anymore, but they do make a similar version. So if you search for ProMan, you'll find
562130	570870	a similar version out there. The socket below that is designed for 1.8 volts. The ProMan is
570870	579190	typically a 3.3 volt device. And we're starting to see way more chips in the 1.8 volt arena. So
579190	584530	it's good to have that add-on. Most of these add-ons are $8, $9, $10 on both of these chip
584530	592970	readers. So they're fairly inexpensive. The thing I like about the ProMan is if I can't identify a
592970	600530	TSOP48, I can't find the data sheets or not, well, let's say I do have the data sheets and I can't
600530	605110	find any information on none of my chip readers. They don't identify it, they can't detect it.
605110	612290	This will let you hard set page OB block sizes into the actual tool that comes with the application
612290	618330	that comes with this, so that you can attempt to read it using that kind of raw information.
618430	624890	And it's been successful several times with me, making it possible for me to read a NAND flash chip
624890	629650	where I had the data sheet to identify that structure, but none of my chip readers would
629650	637070	recognize the chip. And the last one's RT809. And that one's a little more pricey if you buy it
637070	644870	Amazon, but I think you can track that one down on AliExpress cheaper. I like this one. It's not a
644870	652070	bad chip reader, and it has, you can buy a lot of different sockets for it. Most of the sockets
652070	660830	that come with the TL866+, the one on the left, will work on this, all but the TSOP48 sockets.
660830	666390	Because the TSOP48 sockets have a circuitry built into them, so they won't work on this reader. You
666390	674010	have to have a straight pin through. And the one I have below shows that TSOP48 socket that's a
674010	681590	straight pin through. I also recently had to pull a NAND flash that was a BGA ball girder array,
681590	690690	63 balls on the bottom of it. And I was able to buy this off AliExpress for the RT809 for about
690690	698410	40 bucks, which isn't bad. If you get into the high-end chip readers that run, start off at like
698410	706410	$1,600, $1,700 and go up for them, the sockets for them start off at $100, $200 and go up into the
706410	713570	thousands for BGA sockets. So, these things are very cost effective. The sockets are not high
713570	718910	quality, so you have to take care of them. But they're overall, they're pretty good products.
718910	725670	And I use them quite regularly in my lab without issues working with NAND chips.
726070	734530	So, any questions in reference to the chip readers and chip reading process?
736550	743070	One question here. What are the most challenging NAND packages or pitches to work with that you've
743070	748770	encountered? Is there any package or pitch you would not attempt messing with?
748850	754910	I would never say there's nothing I wouldn't attempt messing with. I would literally mess
754910	761430	with everything. Probably some of the biggest problem is, obviously, if you're using a chip
761430	768420	reader, try to read this. The BGAs or ball grid array devices are the more problematic,
768980	777900	because there may not be a socket available for some of these devices. In cases like that,
778380	784720	if you have the data sheets, you can always go to the process of dead bugging. That is turning the
784720	793520	ball grid array chip on its backside and soldering small 40-gauge wire to the actual balls, and then
793520	802180	feed that into a connector or a chip reader or a device for doing bit banging to actually be able
802180	807420	to do that. I have been successful doing that a couple times, but it's a pain in the butt. It's
807420	814260	way much easier to pull the chip with hot air or IR if it's a BGA, clean it up, drop it in some
814260	820680	device, close the lid, and have it give you all the data. But typically, if I need the data off
820680	829180	it, and I can't get it any other way other than chip off, I will chip off from the board. And if
829180	837460	necessarily, I will dead bug one. I mean, yeah, me and you, Jonathan, we worked together at one time
837460	846560	where we dead bugged that one weird device. We had a 0.25 millimeter pitch BGA, and we had to
846560	855320	solder 40 wires to it or 20 wires to it to get it hooked up. So yeah, we're going to try anything. I'm
855320	860160	not going to pass up any potential target if it has data on it that I want.
861420	865760	Makes sense. And you're definitely bringing back some crazy memories with that. And actually, you
865760	870760	partially answered the next question as well. The next question was, what hot air tools do you
870760	872980	recommend for BGA removal?
874120	884760	Actually, I have both hot air and I have a IR oven. I like the IR oven. Once I got it, it's a small
884760	891580	it's a small device, it'll take a board that's probably eight by nine, eight by 10 inches or
891580	897080	something like that. To me, that's the sweetest thing in the world. I usually just stick it in
897080	902120	the oven, run it through the cycle that I brings it up to the temperature. And I take a suction
902120	908500	tool, I pull the chip off of it, and then I let the whole board cool down. Because, you know,
908500	912580	it heats up the whole board, you have to watch out for things like plastic components that are
912580	918240	on the board. So you may have to remove those prior to putting them in the oven. But you can
918240	923080	usually pick up one of these IR ovens. If I remember what was the name of the one I got,
923080	931520	it's spelled P-U-H-U-I. It's a Chinese made product or Asian made product oven that I have.
931520	942000	I think it's a model 2962 is what it is. Small oven. I think I only paid like $130, $140 for it,
942000	947280	somewhere around there. And there's a number of products out there like that. But an infrared is
947280	956240	way much better, in my opinion, than hot air. So... Another question. Sorry, Daryl.
956780	962140	Elvis Cousin asks, is it hard to reball BGA without killing it to re-solder?
963040	971920	Yes. It's not that hard. It's just a process of learning how to do it.
972260	978540	Not to drag this all out, but I had a project where I wanted to learn how to do this. Because
978540	985800	I wanted to get root access on a device. It had 153 ball BGA embedded multimedia controller on
985800	992880	device. So I used hot air. I pulled it off. And I had a couple of devices because I wanted to,
992880	997700	in case I killed one of them. The first one, when I put the balls back on, there's a couple
997700	1004060	ways of doing it. You can use screens with paste, or you can use just the balls. In my case,
1004060	1009260	I just used the balls. I floated on there correctly, but I overheated the chip and it
1009260	1013760	looked like a chocolate brownie when I was done. That's when I learned that you don't want to go
1014360	1021380	on these BGA chips. You don't want to go over about 210. Well, melting temperature is about
1021380	1027240	183 degrees Celsius, if I remember right. And you don't want to go over more than 25,
1028560	1034540	27 on the temperature in Celsius. If you hit 215 or higher, it starts causing the
1035140	1040260	fiberglass substrate of the chips to start breaking down. But if you keep it within those
1040260	1046740	range and you don't shock it by overheating it too quick or cooling it too fast, it'll work.
1046740	1052060	And I was successful. The first one I burned up and I failed. The second one was perfect.
1052060	1058500	Since then, I've never had a problem ever working with BGAs, either pulling them or putting them
1058500	1067970	back on without damage. And once I learned those temperature parameters. Okay, so the next section
1067970	1074810	we want to move into is going to be demo. So, we're going to get off into the demo area and see
1074810	1083250	how that works out. Hopefully, we won't have any technical problems. So, what we're going to do here
1083250	1092450	is we have a flash memory that's been pulled. This has been pulled from... let's go ahead and look at
1092590	1099950	a couple of them here. It's pulled from an Arlo device, which is a doorbell, if I remember rightly.
1099990	1108730	It's actually a kind of like a ring doorbell type product. Its memory chip was a W25N01GV.
1108750	1113550	So, typically when you pull this thing, what's the first thing you want to do is binwalk.
1113550	1120670	And again, binwalk works many times, but sometimes it does not. So, we'll give this a try
1121370	1126470	and see what happens here. Hopefully, we won't run into any real technical problems.
1126590	1132890	And this is kind of the common flow. We see we hit a squash file system. We should have at least
1132890	1138670	two squash file systems on there if it's coming off a standard embedded device. And there's a
1138670	1145490	second one. And then it kind of hangs right there. If we sit here, this thing would sit here in idle
1145490	1152390	for hours and not produce anything. So, we're going to stop that. So, if we change over to
1152390	1159250	the extraction from binwalk, we can see that it opened up a bunch of stuff, but then there's a
1159250	1172610	squash file system. So, as we notice, there's nothing in there. And truthfully, because it
1172610	1181680	went past the squash file system file, it should have put something in there. So, as we can see,
1181680	1190720	literally nothing in there. So, for the most part, this didn't work. This is actually a fresh rebuild
1190720	1202400	of binwalk. And also, I have a binwalk pro account. The results are pretty much the same
1202400	1207240	in this particular case because there's some things that have to be dealt with here. So,
1207240	1215300	let's go ahead and remove this structure just so I don't start running into space issues by
1215300	1223380	blowing all this stuff out of here. So, what do we want to do? Well, as we talked earlier,
1223380	1231180	we talked about the actual OOB area. So, we want to be able to remove that area. So, we have a tool
1231180	1237660	in here called NandDump. And at the end of this presentation, I'm going to have links for
1237660	1244200	everything, which will cover all this stuff. So, here we can actually do input files, output files.
1244200	1250540	Remember that ID I mentioned to you, chip ID? We can use chip ID, or we can specify actual page
1250540	1257720	size, OOBs, and the layout. Remember, adjacent, separate. So, in this particular case, we are
1257720	1267840	going to... well, before we go ahead and separate the OOB out of here, let's go ahead and look at
1267840	1278000	the data sheet. So, as we look at the data sheet, we see that it is 2048 bytes by 64. So, it is in
1278260	1284520	a separate format. That means the 64 bytes is at the end of the page. That's a good sign. That
1284520	1290580	tells us the structure. You always want to go to the data sheet. But typically, I always like to
1290580	1297760	dig a little deeper and make sure that the chip hasn't been kind of messed with. And why I say
1297760	1302580	that? Because just because the data sheet says this is how the chip's going to be laid out,
1302580	1308360	the manufacturer that actually used the chip, I've seen them do some strange things, to say the least,
1308360	1314200	to the data. Meaning that, you know, you go and remove the OOB area, thinking you know where it's
1314200	1320520	at, and literally have it not be laid out the same way. So, we're going to look at this
1321440	1327600	with HexEdit. So, we're going to look at the raw data of the actual device so we can see what it
1327600	1353970	looks like. There we go. So, at this point, we should be able to go to 2048, which would
1353970	1361990	be 800 in hex, and it should be the beginning of the second block. So, we can see that there's
1361990	1370150	something here. It looks a little different, but it may not be. We're not quite sure. It doesn't
1370150	1378230	really tell us a lot. So, we could go to the next block. So, for example, if hex is 800,
1380330	1387370	and we want to, of course, we want to go, remember, is 20, gosh, we get it right here.
1387470	1393910	So, if we go 2048, we'll get it cleared here in a second. Try it again. 2048,
1395230	1403950	which would be 800, and we want to go up to the next beginning, which should be 2,112 bytes.
1403950	1414870	Remember, 2048 plus the 64, and we look at this, it should be at 1040. So, if we go to 1040,
1420570	1425490	we still see some interesting things, and this may tell us that this is correct,
1425490	1432430	but I've seen a lot of devices where the actual OOB area was nothing but all Fs and wasn't
1432430	1438610	actually configured. So, what we're going to do is let's look a little deeper so we can see some
1438610	1446730	patterns, because I'm always a big fan of spotting patterns. So, we come over here,
1446730	1453110	and get it going, and we're going to search for strings so we can get some data. So,
1453110	1456150	here we are. So, we're in an area where there's a lot of strings,
1456150	1459050	and let's scroll down until we see some patterns.
1461370	1468450	And quickly, we see a pattern right here. So, got a pattern right there. So, if that is...
1471670	1488150	So, we go, hold up, 2112, and go. So, it'd be 840. So, we go 840 plus E880. E880 equals F0C0. So,
1488150	1496590	the next pattern should be an F0C0. And we can see that we're starting to see that pattern there
1496590	1505390	also. So, as we can see, more than likely, we are witnessing the 64 bytes at the end of the 2048
1505390	1512230	as the OOB area is showing up. And again, if we look over in the string area, there's no common
1512230	1517310	words, because I would not expect that to be in the area. So, let's scroll on down to the next
1517310	1523330	one. There's another thing I like to do sometimes, especially if I'm having problems positively
1523330	1531230	identifying this location. So, this will involve a little bit of math. So, I like to use a
1531230	1542690	calculator for that. So, if we come over here, and we put this in here, like that, and we turn it to
1542690	1550730	decimal. And then, because the programming calculator does not round off or have decimal
1550730	1557830	points, we want to do this basic. So, what do we know about the overall structure? The first OOB
1557830	1566290	area or OOB area showed up after 2048. So, we had the first 2048. And then, to get to the next
1566290	1573230	leading ones, we had 2112. So, we go ahead and subtract. And hopefully, my math works here and
1573230	1583890	doesn't mess it up. 2048. And then, divide it by 2112. And hopefully, we'll get an even number. And
1583890	1591550	we did. If you do not get an even number, then this is not mathematically correct for an OOB area.
1591550	1604630	So, what we're seeing here is 0500F900 is actually the 39,749th page of data in this particular
1604630	1616460	device. Okay. So, now that I punished you with some math, we're going to go ahead and strip that out
1616460	1629840	of it. So, we have this command called NAND Dump PI. We're going to use .I Arlo for the input. The
1629840	1638420	layout's separate. Page size happens to be 2048. The OOB size is 64. And then, the output will be
1638420	1648460	Arlo No OOB. Hopefully, this runs. And it finished up. So, there, once we finished up, we have a file
1648460	1658810	that's a little smaller, as you can see, where the OOB's been stripped out of it. And let's go ahead
1658810	1675850	and run this and see if this works for us. The first squash system. You see, the pattern's
1675850	1683650	different now. And then, we also see that there's an OOB structure also starting to show up.
1685050	1690950	Often, you'll see inside an operating system on embedded device, you'll see squash file systems,
1690950	1699130	which are typically read-only. OOB file systems allow read and write. So, it may be an area that's
1699130	1705870	used dynamically that's being created that can use to read and write data into the device. Okay.
1705870	1713610	So, that one actually finished. So, let's go ahead and jump over here and see if it worked for us.
1714210	1718590	A little different there. We see we got three folders. We have an OOB root system.
1718690	1728210	So, we go to squash root. And there you go. We were actually able to extract this data
1728210	1737690	by effectively removing the out-of-band data within the NAND flash dump. So, do we have any questions?
1739110	1743650	Yeah, we got a couple of questions here, T-Dub. So, Elvis Cousin asks,
1743650	1748370	I have two BGAs on a daughterboard. Is it likely I can read each individually,
1748370	1753190	or will I need to reassemble them somehow to try to get a squash FS?
1753190	1755990	And they're also both Intel-based flash.
1756990	1762690	Oh, you're saying you have two flash memory chips on the device.
1763270	1766670	Really comes down to how they lay those out.
1767610	1774550	I would not expect them to stripe across separate chips like that. I've never seen that.
1774550	1779710	Often, when I've seen two chips done up like that, each one will contain a different partition,
1780130	1785770	or they may be redundant of each other. But typically, they'll be partitioned
1786370	1789570	separately on embedded. That's the way I've seen it.
1789570	1797550	And I've seen everything from where one chip actually contained the boot,
1797550	1800790	U-boot, and the kernel, and the other one contained the file system,
1802150	1806810	as an example. So, the only way is to pull them off, examine the data,
1806810	1810630	and make a determination from there to see if you can do it.
1810630	1814750	Of course, the best solution is if you can get a root console on the device somehow,
1814750	1819550	then obviously, you could figure that out more quickly and DD the partitions off that way.
1819550	1825830	Which is also the sweetest way of doing it. But if you have to chip off,
1825830	1828410	yeah, you still may have to do some analysis of them.
1829610	1834590	Yeah, you kind of pivoted off that, said, both are connected to a common connector,
1834590	1839390	but I don't have a pinout to try to recover a cross connector by soldering something.
1840730	1846570	Yeah, it's kind of hard to say. If you have some kind of pictures or images of the data,
1846570	1850930	or not the data, the chips and the layout and the board, shoot them to me sometime,
1850930	1857090	and maybe I can take a look at it and help you come up with a plan of attack.
1857550	1860810	Like I said, anyone's feel free to reach out to me anytime.
1860810	1865850	I may or may not have an answer for you, but I'd be more than glad to give you a hand if I can.
1866850	1870950	One final quick question here that I'm seeing. So, in this particular case,
1870950	1875110	it looks like you're able to find the data sheet. Have you ever encountered a NAND chip
1875110	1880690	that you wanted to dump but couldn't identify it or could not locate a data sheet for? If so,
1880690	1886070	how did you approach that? Yes, guessing.
1887830	1896350	Often, if I can't find a data sheet actually for that, I've actually gone out and looked at
1896350	1902110	other chips that may have data sheets that are in that same series or produced by that manufacturer
1902110	1908030	and figure out how they do their layout. Even though I may not be able to
1909830	1915410	quickly identify the chip ID, there are ways to actually enumerate those, and it
1915410	1924310	usually typically need a data sheet to do that. But guessing does work, and crazy as can be,
1924310	1931050	I've actually dumped a lot of chips by going, hey, here's a manufacturer, and it seems like
1932110	1938370	most of their chips, or a large portion of them within this structure range, are laid out this
1938370	1945630	way. They have this page structure, this size. Often the naming conventions, like here on the
1945630	1955370	screen, W25N01GV, that's one gigabit chip. And often in the name, you can figure out what the
1955370	1966130	chip is a lot of times. So, that would be eight bits, 128 kbytes or whatever. So,
1966790	1972470	it's doable just looking at the name sometimes to pull some typical information out of it,
1972470	1977030	and then guessing on some of the other stuff. But again, I've also smoked chips
1977670	1982650	by doing weird things to them that obviously didn't work. So, got to take some caution
1982650	1990950	and take some risk. But amazingly enough, it can be successful. One of the things to do also is
1990950	1997030	kind of look at, if you can get a UART console onto the main processor, let's say the main
1997030	2003870	processor has a console, even though you can't get root access to it, capture all the debug data
2003870	2014030	from the boot up. That can also be very revealing sometimes. Okay, time to move on.
2014350	2019870	So, we've kind of done that. We're able to get the data off that. I'm going to go ahead and
2019870	2027130	clean this up. Because we have a couple exercises to go to and I don't want to run out of time.
2027130	2044890	So, okay. So, the next one we want to look at... I'm also going to RM the article. So, the next
2044890	2051830	one we want to look at is the Nest. This actually came from a friend of mine who had his Nest
2052310	2059110	thermostat crash out. It just died. He threw it my way, said, hey, can you get any interesting
2059110	2063850	data off of it? So, and I struggled with getting data off of this, but it was eventually
2063850	2072790	successful. We're going to show some of this. There was no confidential data actually extracted
2072790	2078650	from the device. So, there's no problem with us using it here. But so, what we want to do
2078650	2084150	is what we're going to use is a process where we're going to create a NAND sim.
2084570	2091910	So, what does that mean? We're actually going to simulate a NAND chip in memory. We're going to
2091910	2100990	use like ModProbe. If you haven't used ModProbe, ModProbe will let you load kernel modules onto
2100990	2109730	the system. So, we're going to build a block device using ModProbe kernel modules. We're
2109730	2114770	going to load those up and then we're going to write the data out to the chip. We're actually
2114770	2124390	going to tell the data with NAND write to use the OOB data to reconstruct any error corrections
2124390	2132630	and that type of stuff. And then we'll mount it up. So, this particular one, I actually ran
2132630	2141770	Benwalk on this and it never recovered anything. I actually shot it out to good friends at
2145970	2152310	that do the Benwalk Pro. And unfortunately, I didn't get back to the system for a couple
2152310	2157750	weeks and it churned out there. So, I apologize to them again for eating up several terabytes
2157750	2164250	of data. So, this thing was constructually a pain in the butt and running Benwalk was
2164250	2170430	potentially catastrophic. It would eat up all your memory and produce nothing. And I'm not
2170430	2179390	going to run it here, but if you run it, every page or every header for a GIFs to file system
2179390	2185410	or every header for one of those, it would create a folder for it, put nothing in it,
2186170	2190430	and literally eat up terabytes of data in a couple days quickly.
2190470	2196190	So, let's go ahead and get going here. So, we're going to load ModProbe MTD,
2196190	2203630	ModProbe Block. This will help us set up a block device. And then we're going to go ahead and
2208700	2214900	we're going to load this up here. So, the way this works, ModProbe NANDSim, BCH is
2215460	2223820	error correction functionality. This was the chip, it uses a 8-bit structure. There's a BCH16
2223820	2229480	for a 16-bit structure, if that's actually being used. And this here happens to be the
2229480	2241160	chip ID. So, if we go over to this particular chip, that is not the one I want. So, this is the
2241160	2253220	actual chip. And I believe we go to page 25. I have this memorized. Okay. And this would be the
2253220	2260620	read ID, the ID of the actual chip. We know this chip is a 2-gig chip. It's organized in 8-bit
2260620	2272080	structure. It is a 1.8-volt bit chip. So, 01 is the first one, AA, 90, 15. 44 isn't used in this case,
2272080	2278760	but it deals with error corrections and multi-plane information. Multi-plane information,
2278760	2285040	just to make note here, if you have a chip that has multi-planes on it, and you'll see it in the
2285040	2292920	datasheet, it may or may not stripe across the two planes, meaning it puts one block on one plane
2292920	2298500	and one block on the next plane versus putting them contiguously on the same plane. I have not
2298500	2303560	encountered this, but I have read about it. So, it's something to be noted. So, at this point,
2303560	2311520	we're going to go ahead and load this up. So, now we've loaded up that particular module, kernel
2311520	2319960	module, and now let's go ahead and just look at some information, mtdinfo, and this will tell us
2320080	2328380	a little bit about what we just created. So, we created a simulated. It's how many bytes?
2329860	2338340	256-megabyte chip, 2048, subpage size, 64 bytes, pretty straightforward. And then we go ahead,
2338340	2344700	and this one may show some errors. Sometimes it does, sometimes it doesn't. And we're going to
2344700	2353500	look at this within DMessage. And if we come down here, we can see that it took the manufacturer ID,
2353500	2359440	chip ID, and information, and actually have identified the actual chip. That is the chip ID
2360100	2370650	that does that and helps lay this out. So, the next thing is we want to be able to write into it.
2370670	2376350	So, we're going to use NanWrite, which is another tool.
2378370	2383890	Hope you don't mind me not typing. It saves you guys a lot of headaches. So,
2384830	2392090	let's cancel out of that. NanWrite. I want to see the command. So, here's all the functions. So,
2392090	2397550	what we're going to do is we're going to input contains OOB data. So, we're telling it to
2397550	2404250	utilize the OOB data. And we're going to go ahead and let this run. And hopefully,
2404250	2411070	it'll run through clean. Boom! There we wrote all of the blocks out onto the device.
2411110	2416370	So, the next step is... we'll come over here so you see this. We go ls.
2422670	2428890	And we can see there's nothing there. And then what we're going to do is cut and paste. So,
2428890	2441880	I don't have to type. We're going to mount this up on block zero. Change directory to gifs.
2443200	2450500	And voila, we have data. So, if we look through here, it starts looking pretty good. Hey,
2450500	2458720	we've recovered all this data. The sad thing is most of this stuff is system logs.
2461700	2468480	That we recovered. But if we get up here toward the top, we find some weird things.
2468540	2478720	Here's bin. It's a file. etc is a file. Well, truthfully, this is bullcrap.
2479560	2487200	We know that's not a file. So, if we cat etc, wow, look what we get. Now, I wouldn't expect
2487200	2494400	to see that in etc. So, we recovered a lot of it. But some of it's corrupted in a weird fashion.
2494740	2500020	So, how did I recover from this? I experimented. And we're going to go over that a minute. But
2500020	2511250	before we go forward, I actually want to see if there are any questions. Jonathan, any questions?
2511910	2517430	Yep, taking a look now. No updates here. I'm checking Twitch real quick.
2519050	2523650	Yes, here's a question. In IoT devices you've examined, do you ever come across
2523650	2529170	NAND storing non-file system data? If so, do you just dump and then use binwalk
2529170	2532810	and other usual tools and methods for poking around?
2533530	2544830	Yeah, I've had them storing non-file system data on NAND flash. So, yeah, typically I get into that
2544830	2550910	and when I get those and there's no structure to them, I start off with throwing things like
2550910	2561370	strings to the thing. With strings, can we identify any known structure in there?
2561930	2567610	I will run binwalk just to see what it'll do. Sometimes it'll point out some structure headers
2568330	2574470	that I can more quickly find. So, I think that plays a big role. But yeah, sometimes you'll have
2574470	2582910	them storing things like blocks of configuration data. And I've seen them just blocks of encrypted
2582910	2590250	data, which may be a file system structure, but they're not framed as such. So, it's just
2590250	2597710	encrypted blocks of useless information. So, yeah, that's not all that in common. Well,
2600290	2605070	most of the time when I get a NAND chip, it's a file system, but you will encounter those,
2605070	2612650	to be honest. So, right now, before we move on to the next section, I want to remove the
2614830	2626840	kernel modules so they don't conflict with what we're doing. Okay, and we are down to...
2628160	2633960	we still got some time, doing pretty good. So, the next section I want to get into,
2633960	2643060	we are going to use MTD RAM to actually do this. So, we're going to take that same system,
2643060	2651900	that same dump that we just did, where we mounted up the file system,
2651900	2657400	it did not come out exactly the way we want it. And we're going to mess with that. So,
2657400	2663640	we're going to make a make directory, and we're actually going to create our own block structure.
2669350	2673270	And then we'll go, and then we're going to make a node.
2675170	2684150	If I learn how to spell it, I should just copy this over, make node dev slash MTD block.
2684710	2689130	And we're going to create zero. We're going to create it as a block device.
2689130	2695350	We're going to give it a master number, a primary number, and a minor number of zero.
2695350	2699850	So, we're going to create that device in there. And then what we're going to do is we're going
2699850	2708570	use modprobe, and we're going to bring in a kernel module called MTDRAM. So, this kernel
2708570	2720970	module's MTDRAM. We're setting the total size to 274432. That is bigger than a 256 megabyte RAM area.
2722090	2729370	256 megabyte RAM area would be 262144, and that would not contain OOB data.
2729370	2736910	In this case here, we're bringing in the OOB data also. Because if I bring in without the OOB data,
2736910	2743090	I noticed that we get the same results as the other one. So, experimenting, and I encourage
2743090	2749070	you to experiment when you're dealing with these NAND chips. It can be quite fascinating what you
2749070	2756370	can find and figure out and make work to get you data out of this system. I'm currently working
2756370	2764270	on one. It has me slightly befuddled, but I will master it eventually, where the actual entire data
2764270	2771750	on the chip is shifted 10 bytes. They even shifted all the OOB data 10 bytes, and it took
2773610	2779390	looking at it manually to figure this all out and doing that math and looking at the data
2779390	2785410	in the structure to determine that they had also shifted that. Because I was getting block errors
2785410	2791130	saying that block is 10 bytes longer than it's supposed to be type thing. That's when we found
2791130	2796450	out that everything was shifted and was able to confirm that the OOB was shifted too. But I digress.
2796450	2801310	So, what we're going to do is we're going to load this up, and it's supposed to contain the entire
2801310	2813430	thing. And then we want to load one more module. MTD block for kernel module. And then we're going
2813430	2826480	to dd. Excuse me. And dd is a bit copy. I use this all the time. Matter of fact, if you get access
2826480	2832060	to almost every embedded IoT device, you'll find dd is actually installed on it. Because most of
2832060	2839600	the time, dd is actually used for doing firmware upgrades. You'll have duplicate root file structures,
2839600	2845480	duplicate kernels on the device. And what happens is one is primary and running. So, what will happen
2845480	2852760	is when the firmware comes down, it'll dd it over the secondary, and then it'll change the U-boot
2852760	2859780	or secure boot arguments to make the new one primary and the old one secondary. That way, if
2859780	2866360	you have a failed firmware update, it does not break the device. It just goes back to the original
2866360	2875580	one, which was not overwritten. So, again, input file here is the nest file. And then output is to
2875580	2882080	our block device that we created. And if I did everything right, this should start writing data.
2882080	2891160	Should be fairly quick. And as you see, we're able to write out 277 megabytes of information
2891160	2899320	fairly quick, because it's all internal to the system. So, from here, we're going to go ahead and
2899320	2908100	mount this up. Same way we did on the other one. And we're going to mount MTD block zero up.
2912780	2920360	Now, if someone wants to explain to me why this worked, because I really not sure why.
2920360	2927280	As I said, often dealing with NAND chips, little quirky things have worked for me. So, I'm not going to
2927280	2934700	speak bad of them. And now we can see etc and bin are actual files. We do have some
2934700	2942000	corruptions in here. So, there's a few files actually corrupted within this. But for the
2942000	2949060	most part, a large portion of them were actually able to be recovered properly.
2957990	2965410	And so, these are methods that I use quite regularly for recovering. When you start
2965410	2974070	working with UBI file systems, if I ever get some good methods, methodologies hammered out
2974070	2979150	for working with some of the strange UBI file systems, like the one with the 10 bytes that's
2979150	2986870	shifted. I did one on an engagement here earlier this year, where again, nothing are large portions
2986870	2993730	of it were not laid out contiguously and trying to build good solid methodologies around UBI.
2993730	2999570	I'll probably come back around and put together a training session presentation for that down
2999570	3004070	the road. That's what I'm currently working on, trying to work out those kinks and bugs and build
3004070	3010430	that knowledge level to build to work with us. Now, probably half the chips that are UBI file
3010430	3017530	systems, I'm successful recovering them using various techniques and UBI tools that are available
3017530	3026210	out there. But typically, GIFs, squash file systems, things like that can easily be quickly
3026210	3033570	recovered from NAND flash chips using any one of these three methods that we covered today
3033570	3039250	in this presentation. So, hopefully there's some questions out there.
3040550	3048610	Yeah, I got a couple here. So, one question here asks, have you encountered an IoT device
3048610	3052850	that was encrypting the data it was storing in the NAND flash?
3054330	3062010	No, I haven't, which is kind of amazing to me. I mean, the only time I've ever seen IoT
3062010	3068670	technology using any kind of encryption, if it was running an Android operating system.
3068670	3076890	And by nature, Android can support the data section being completely encrypted.
3076890	3088030	So, that's quite common. I do have some other tech that I consider IoT, but it's more enterprise
3088030	3094890	IoT and not necessarily the normal consumer grade, which actually is very much encrypted
3094890	3101390	and most of the, not all the file system, but most of the file system that contains user data,
3101390	3109670	similar to the Android file system, is also encrypted. So, typically, if I see it's an
3109670	3116970	Android, it's usually 50-50. If it is a standard phone-style Android operating system dumped on
3116970	3123830	there, then most always it has encrypted. But yet, if it's an older Android style that's put on
3123830	3129390	there, there was no requirement or typically that's not always encrypted. So, it's a mix and
3129390	3135530	match, but it's pretty rare to see encrypted, at least from a lot of the IoT I've looked at,
3135530	3141350	unless you get into more enterprise level technology.
3143690	3147170	Anonymous attendee asks, Hi, I am new to this. Sorry,
3147170	3152190	you already have the binary file. How did you extract it from the hardware device?
3153830	3161410	In the cases of these, both of them were chip off. So, early in the presentation,
3162110	3167630	and so you can rewatch this presentation, or in the presentation earlier today,
3167630	3172750	where we talked about building an IoT lab and we showed a number of devices,
3173110	3179210	I have a number of chip readers. So, I actually use those. Other ways are
3180650	3187190	often if the chip that's being used has some kind of debug function, you can put it in debug
3188030	3193490	and pull the data off of it. Also, you may be able to check, connect in with JTAG or
3193490	3200290	SerWire debug and actually read it off the device. But if those fail, I often rely on just
3200290	3205990	desoldering the chip and dropping it into a reader and extract the data from there.
3205990	3211410	If I need to put the device back into functionality, then I'll go through either
3211410	3218430	resoldering the chip back on or reballing the chip and reflowing it back on an IR oven,
3218430	3224450	which are fairly cumbersome, at least the reballing BGAs, but it's still doable.
3224450	3231730	And it's not all that complex. I did a presentation not last year, but the year before last
3232370	3239070	at Nuit de Hacq. I did it there. I also did it, oh gosh, I did it in Perth, Australia,
3239070	3248450	and I also did it at DerbyCon. And it was, I'm trying to remember the name of it. Basically,
3248450	3254830	it was how to get root access the hard way. And it talks about slashing, smashing,
3254830	3260390	cutting the device apart and rebuilding it to get root access or something to that effect.
3260390	3263310	So you can look at that, and it goes through some of the concepts
3263750	3269270	in my learning process on learning how to manually reball BGA chips.
3271050	3275990	Elvis Cousin asks, these are all variants of Linux. On a Windows CE device,
3275990	3280150	might it still use SquashFS or some other Windows thing?
3280310	3289390	Ooh, probably some other Windows thing. I have not looked at a Windows CE device.
3291230	3301090	I have been looking at some Chrome devices. In those particular cases,
3303290	3309890	this has an SD drive, solid state drive, where you're able to get the data off of it and just
3309890	3315410	mount it up on your system. I'm not sure if they have NAND chips. Is it going to be
3317570	3322890	a Linux file system? Probably not. It's probably going to be some variation of Windows. But I have
3322890	3329130	not tore into any Windows or Windows CE type devices to be able to give you a solid answer
3329130	3339570	on that. Let me know if you find out. Any other questions? Nope, all caught up.
3340410	3345650	All right. Well, I think we're... what time we start? Six o'clock. So we're probably right on
3345650	3358010	time. So let me jump over here. Let's do a final stop to share. So let's go ahead and
3359270	3365250	ask question one more time. Anyone have any questions, any requests for questions or
3365250	3372550	information? If not, you'll probably catch me on Discord either here in a little bit after I've
3372550	3381750	eaten or late tonight or tomorrow. Feel free to hit me up for any questions. Also, oh gosh,
3381750	3385950	yeah, I was promising that. Hold on before you all drop off there. Let's get this up.
3392020	3409120	So I had promised... slideshow. So I want to be able to get some of these things captured in the
3409120	3415720	video. So here's the NAND sim example that I use. The mod probe commands, the NAND write commands,
3415720	3420220	and the mount to be able to mount this particular GIF file system up.
3420840	3432440	And here is the commands that I use for mtd-dram example. Quick note on some of the commands we
3432440	3442000	use. There is the link for NAND dump tool, concept of mod probe, make node, NAND write, DD. And then
3442000	3448600	here is the reference. So that gets captured in here. All of this is kind of cool material.
3448620	3453500	There's a lot of great work that's been done out there on NAND chips. Like I said, I'm constantly
3453500	3461000	reviewing and reading and consuming this material on a regular basis because every time I encounter
3461720	3465660	NAND chip that wants to fight with me, I have to go out and start rethinking,
3465660	3470700	okay, what am I missing? What do I need to learn to expand my knowledge? How do we kind of move
3470700	3476420	forward from here? And there's been a lot of work done out there. And of course, we shared
3477300	3484980	much of the information that these people provided in this demonstration today. Also,
3484980	3493140	if you're interested in connecting up with me on Twitter, my handles are %X, so it would be
3493140	3502180	P-E-R-C-E-N-T underscore X. And feel free to connect with me out there or reach out to me
3502180	3511980	and ask questions out there. Awesome, Daryl. Thank you so much.
3512220	3518720	Oh, man. Sure enough, Sam. Had a blast. Yeah. And I mean, honestly, if there are any last
3518720	3522780	questions, we still have a little bit of time. So was there anything else that you saw there,
3522780	3531880	Jonathan? Yeah, another quick question just popped up. It says, this is some very epic stuff,
3531880	3537820	extracting flash from NAND chips. I have to ask, did you find anything epic while performing
3537820	3550300	post-exploitation on the extracted firmware? Yeah, yeah. I mean, any remote shells? No,
3550300	3558420	unfortunately. But again, it's information that we could feed back to manufacturers to help
3558420	3565440	improve processes. You know, they get into hard-coded stuff. Mainly, a lot of this lately
3565440	3571960	was purely an effort to go, hey, what happens when IoT is thrown in the trash type thing?
3572020	3580180	You know, is there data is like the guy with the Nest thermostat? What happens when something
3580180	3586860	dies if you throw it away? Does it contain PII information? So this was kind of a lot of it was
3586860	3594940	driven out of that and some paid assessments. And also, you know, we tested a lot of devices from,
3594940	3601840	hey, if you go ahead and flush the thing back to factory, does data stay on it? We kind of
3601840	3607540	started checking for things like that. It's all about a privacy risk and issue, because we've
3607540	3613080	gotten really big on the whole idea that, hey, we don't want to throw our hard drives away.
3613080	3619100	But in this modern world, these chips are our hard drives. And we need to probably start thinking
3619100	3625580	about the potential risks there. So that was kind of the main purpose of a lot of this,
3625580	3631040	and the fact that I had been working on some engagements that involved band chips that were
3631040	3638480	problematic. In those cases, we were working on things like reconstructing keys to gain access.
3638480	3644460	A lot of this stuff ties into engagements also, where we're looking for data since we're testing
3644460	3650180	an entire ecosystem. That isn't just the hardware, but it may be mobile apps, cloud services,
3650180	3656640	and other devices. What can we get off this chip that could allow us to attack everything else
3656640	3663100	or attack somebody else using the same product because of key reuse and key session keys and
3663100	3668120	password reuse and things like that. So a lot of it's within that direction.
3670540	3676980	Another question here. This individual asks, does it seem like it is possible to be able to create
3677180	3684420	a module so that a chip reader would automatically be able to detect those out of the OOB areas of
3684420	3690140	the flash memory? Oh, a number of chip readers will let you identify. Some of them will allow
3690140	3696440	you to extract the chip, the data, without the OOB. So yeah, it's not uncommon. You can
3696440	3706360	often manipulate data from the chip reader and include or not include OOB on some chip readers.
3706360	3710360	Obviously, you get more control as the price of the chip reader goes up.
3710780	3718460	And I don't necessarily have a $2,500 chip reader in here. But I think one of these actually had
3718460	3726160	some OOB settings. But for some reason, I didn't necessarily trust it because I told it not to
3726160	3732120	include OOB. And then when I visually looked at the data, the OOB data was there. So I kind of lost
3732120	3739360	trust. I have a tendency to also like to map it out, not let the chip reader make the determination
3739360	3745680	for me that this data is structured a certain way. Because like I said, it's not uncommon with
3745680	3752660	vendors to futz with it, so that it may be incorrect, or that the OOB shifted or is not
3752660	3758440	used in the same place. And they've just written a new controller to read the data the way they
3758440	3764920	made it. This way allows me to go, hey, this is correct or isn't correct. And then I can easily
3764920	3770740	manually remove it or modify code or scripts that are out there to take it out the way I
3770740	3778320	wanted it removed. So that's kind of my take on it. That makes sense. I think that we're at the bottom
3778320	3786040	of the list here for the questions. Okay, well, I think that kind of concludes it. If anyone has
3786040	3793020	any questions, catch up with me later, or reach out through me on Twitter. And again, I'll be on
3793020	3800440	Discord probably tomorrow and maybe later tonight. So I'll catch everyone later. Have a good night.