start	end	text
17820	24700	All right, and welcome back, everybody. We have another great speaker here today. We've got
24700	29400	Eyal Iqin here to talk about light bulb security. How's it going, Eyal?
30120	31260	Fine, how are you?
31260	36700	I'm doing okay. We also have my co-goon here, Siget. Hey, Siget, how's it going?
40740	47300	So, Eyal, can you kind of just take a minute and maybe give like a brief introduction of
47300	51880	yourself and the presentation that you gave that people can go watch on DEF CON?
53620	62580	Yes. So, hi, I'm Eyal. I'm a vulnerability researcher. I usually focus on embedded devices
62580	69380	and network protocols. And this is a classic research in which I combine them both together
69380	77000	to hack some, in this case, smart device. Essentially, in our research, we continued
77850	83950	the previous research by two great researchers, Eyal Ronen and Colin O'Flynn,
84260	91780	and they showed how they can take over a smart light bulb and propagate from one light bulb to
91780	99160	the other. And our goal in this research was to take it one step further. We wanted to take
99160	105080	over a light bulb and then from the light bulb take over the smart controller. And the controller
105080	112200	is connected to both the radio network, the ZigBee network, and to the regular computer network. So,
112200	119120	we could manage our smart light bulbs from our mobile app, for example. This basically means
119120	127880	that using a malicious light bulb or an antenna, we could take over the controller and then
127880	133360	infiltrate the network and attack computers inside it. So, we only need a radio antenna and
133360	141560	we can infiltrate your home or your office network. So, that was the goal and this is what we
141560	150180	demonstrated. That's excellent. All right. So, I know in your presentation, you kind of mentioned
150400	158140	a whole bunch of the people's research that you based yours off of. What got you interested in
158140	163620	this kind of thing? What made you want to look into trying to take things one step further
163620	171640	from what the other research that previous people did? Okay. So, it was quite simple. One of
171640	177360	the researchers, Eyal Ronen, is an old colleague of mine. And when he started working on his initial
177360	183800	research roughly four years ago, it was the first time I actually heard of smart light bulbs.
184000	189720	And I really didn't know why people need smart light bulbs because I really only need the light
189720	195500	when I'm physically in the room and I'm physically capable of toggling on and off the switch. So,
195500	199800	I don't really know why I need an app in order to control the light in the room that I'm currently
199800	208020	in. And I figured out that smart light bulbs won't be a real thing. People won't buy it and
208020	214280	forgot about it and continued on with my life. But it turns out that people actually buy
214280	219600	smart lighting solutions. It turns out a lot of people specifically in the UK
220440	229780	use it. It's not so popular here in Israel. So, Eyal Ronen talked with me again and he said,
229800	237860	we have an opportunity to continue on our research because we had an additional task that we
237860	243300	couldn't complete in our research. And since it turns out that people actually use smart light
243300	250040	bulbs and in previous DEF CON talks, we infiltrated the network using a fax machine, it turns out that
250040	256740	it could be interesting to do that over the radio as well, over ZigBee. So, this was the goal.
256740	263220	We collaborated with the previous research team to get their equipment and knowledge from their
263220	270320	results. And then we started on our way. Awesome. That's all great stuff. So,
270320	274000	again, do we have any questions coming in yet or did you have any?
276710	282370	Not sure if he's hearing me at all over there. Okay. Go for it.
283070	287990	No, there's a comment that says, I hope you got a bounty for showing that the
287990	300210	weren't completely implemented. The second part of the patch that you tested, right?
300210	306910	Yeah. We didn't get any bounty. We usually don't get any bounty.
307310	314610	I never worked on an embedded research which we received any bounty on any vulnerability
314610	324330	whatsoever. Usually even when we report the vulnerabilities, they are only partially fixed
324330	332350	or only the crucial vulnerabilities are actually patched. So, for example, our vulnerability
332910	339810	was that we were able to show that in previous versions there were arrays which weren't parsed
339810	345130	correctly. And in newer versions, they migrated the strings. But behind the scenes, we're still
345130	353250	using them as arrays with the same vulnerability. So, the patch was to check the sizes of the
353250	359890	arrays and handle them correctly instead of treating them as strings because strings are
359890	366290	handled correctly and we only can send strings over the radio. So, there's no reason to treat
366290	371930	them internally as arrays just because someone did it three years ago and that's the code. But
373090	381650	it's harder to fix it on a big design issue than just adding an if in the code and that's it.
381790	388130	So, one of the questions that we see coming in from looks like Jiska. Did you get any further
388130	392990	with the ZigBee modem itself? Like, were you able to extract its firmware?
395690	401270	It's a good question. The problem we had with the ZigBee modem is that the firmware update is
401270	408690	encrypted. And in a previous research, what they did was they had a power analysis attack and they
408690	416770	extracted a key and decrypted a firmware update and signed their own firmware update. But it was
416770	422210	only applicable for the firmware to the light bulbs themselves. It turns out that the modem
423030	431150	uses a bit different cryptographic scheme. So, we had to, if we wished to get the firmware of the
431150	438810	modem, we had to reiterate all of the research that the previous research team did. And we
438810	445210	didn't have the resources for that at the time. So, we tried to live with it as a black box and
445790	452490	we managed. It could be interesting to look on it because it's more advanced than the light bulb
452490	460750	itself. It does more functionality. But no, we don't have the firmware, sadly.
461030	467910	Okay. Yeah, I think in your presentation, you mentioned that previously, before your research,
467910	472850	it was kind of thought the worst things that you could do was turn all the light bulbs on or turn
472850	477870	all the light bulbs off or flicker them or anything like that. So now, based on the research that you
477870	483190	did, what do you see as kind of a worst case scenario if somebody wanted to do bad things
483190	490630	with your research? The classic scenario we thought about is either a neighbor attacking
491390	498230	your network or someone parking a large van across the street and deploying an attack. And then
499650	507550	we can easily hijack a given light bulb and convince it to connect to our network. And then,
507550	513030	as you saw in the demo, if we turn off the light bulb, you think that it's broken and you just
513030	518350	toss it away. And instead, we change the color so you'll know that something is wrong with your
518350	524950	light bulb and try to reset it. But there's no reset button. There's no buttons at all to the
524950	531210	light bulb. So the only way to reset the light bulb is to delete it from the app and then search
531210	535530	for new light bulbs. And when you search for new light bulbs, we can actually attack your
535530	541630	controller and infiltrate the network. So once we take over the controller, we are inside your
542120	550490	home network or your office network. We can attack whichever computer we wish. We can sniff the
550490	559090	traffic. And if inside your own network you use HTTP instead of HTTPS, we can sniff all of your
559090	566330	traffic. And essentially, no one will notice an intruder on the bridge. You don't have an
566330	572290	antivirus on your bridge. You don't have any security solution on it. So we can have a persistent
572290	579490	foothold inside your network and attack additional computers. And just from being near your network
579490	586810	with an antenna for five minutes. Wow. So knowing what you know now from this exploit, what would
586810	595790	you suggest to the makers of IoT devices? Specifically in this case, there were some
595790	603690	security mitigations in place, which is sadly quite rare. Usually we get none. And still,
603690	608890	although there were some security mitigations, we're still able to exploit the vulnerability
609630	616790	because there weren't enough mitigations. And actually, after our research,
617450	627050	we looked again on the heap implementation that was used. And it was really weird for us that
627050	632750	there were some security mitigations, but because the vendors didn't compile their own binary
632750	638730	correctly, we were able to bypass the mitigations. Because it turns out that if you don't have
638730	645670	security on by default, the vendor won't compile correctly their firmware. And they won't get
645670	653430	anything because on their compiler, it's off by default. So in our team, we designed a new
653430	663190	security mitigation called Safe Linking. And we integrated that into both glibc and uclibc-ng so
663190	670930	that it will be a new security mitigation in popular malloc implementations. And it's on by
671330	680190	default. So just this week, glibc introduced a new version that includes Safe Linking. And uclibc
680190	687490	already have versions with Safe Linking. And this security mitigation would have blocked our attack.
687490	693170	It would have demanded us to have an additional infleek vulnerability, which we couldn't find.
693830	702750	So nowadays, vendors just need to use the most updated versions of libraries that they use.
702750	711110	The most updated Linux version, glibc or uclibc version, and preferably even a good compiler
711110	718130	with a good toolchain to actually compile with security mitigation. But just updating your libc
718130	725690	implementation will be a huge step forward. Excellent. Segev, are you seeing any other
725690	731370	questions for him coming through the Discord channel at track one live QA?
731810	736350	So we have one. So your discovery was a modern firmware security problem,
736350	738510	or just a ZigBee protocol issue?
740710	748750	It turns out that ZigBee is complex, which wasn't a big surprise. But the vulnerability itself
748750	756370	was an implementation vulnerability specific to that product. So Philips, now Signify,
756370	762850	had an implementation vulnerability over the years. It changed a bit over the firmware versions.
762850	767910	But essentially, it was an implementation vulnerability on their side. It wasn't a
767910	777310	vulnerability on ZigBee. That said, ZigBee defines data types with variable lengths
777310	783750	that should be received over the network. And usually, embedded devices and embedded developers
783750	792850	are not that good with handling variable-sized objects. Although there was no vulnerability
792850	800850	in ZigBee, but only the implementation, I can guess that if five developers will have to
800850	808850	implement this model, at least two of them will get it wrong. So it still should be done correctly.
809170	816010	So have you checked the updated bridge firmware, like if they used more mitigations?
817410	822130	We checked the updated firmware version for the patch, so we would know how did
822130	827230	they actually fix it. And to check that they fixed it correctly, so we could continue on
827230	835090	with the publication. We only saw a small patch, as I said earlier, just an additional lift check.
835090	840890	We didn't notice any change in the compilation or anything of this sort.
841570	849490	If they are hearing this switch, I really urge them to update their libc version and compile
850170	856810	to use dynamic addresses instead of static addresses. This will be a huge step forward
856810	862650	in the security of their device. That's great. Let's see, what else do we have coming through
862650	870530	over there? Anything else, Sighad? No, I'll ask a question. I noticed in your talk,
870530	877330	you spoke a lot about your sort of false starts, right? And you tried this, you tried that,
877850	883570	and these didn't work out, and those didn't work out. And then you also kind of mentioned hitting
883770	891650	a wall on the erase issue before you were able to find the second implementation that still
891650	899230	contained the vulnerability. So, two questions. One was just, you know, what about those false
899230	904470	starts did you think were important enough to include in the talk? And secondly, you know,
904470	912670	what got you to the thinking that got you past that wall? Okay. So, I started the first one.
913210	919790	Without the false starts, I wouldn't have been able to actually find the vulnerability that
919790	929890	eventually I found, and to reliably exploit it. Because even as in the slides, I didn't even have
929890	935890	enough time in the blog post, which was just uploaded, you can see that there was an additional
935890	946050	false vulnerability, but there is a vulnerability, but it isn't exploitable. And even if I wouldn't
946050	955190	have encountered this code module, and the treatment of ZigBee addresses and the ZigBee
955190	961070	phone book, which initially had a vulnerability, but it wasn't exploitable, I wouldn't have been
961070	967930	able to understand where to store the shellcode and how to construct my shellcode and how to use
967930	974270	it. So, every part in this research, even if I didn't eventually find a vulnerability in the
974270	981730	specific model that I tried to, I learned a few things about the device and about the memory
981730	987490	layout. And every one of them was important in the overall attack at the end. So, that was the
987490	993770	main reason I decided to talk about all of the steps in the research, even if there were some
993770	1001230	dead ends, because it gives additional information that we later on use during the exploit. So,
1001230	1009870	it was important about hitting the wall. Then, when you start a research with
1010510	1017030	an interesting attack vector, and you're already into it, you got the firmware, it's loaded into
1017030	1023730	your disassembler, you bought a kit with the hardware, you have everything, you really want
1023730	1031130	to find vulnerabilities. And in most of the projects that I worked on, there are critical
1031130	1038730	vulnerabilities. We just need to find them. So, you check one module, it's a dead end. You go to
1038730	1045990	the other one, it's a dead end. You go to the third one, it's a dead end. And then you go back
1045990	1054290	and you check your initial assumptions. Maybe there is some assumption that we thought about,
1054290	1062070	but it's wrong. The device might behave differently. Maybe we skipped a module because it looked
1062810	1067790	well written, but we didn't check it deeply enough. So, we can go back to that module
1068310	1073990	and test it again. So, essentially, in a security research, we attack the device
1073990	1081910	in layers. We want to find the path of least resistance. And it's really common to reiterate
1081910	1089290	over the same module again and again, because now we want to go deep into that module after we had
1089610	1096640	a shadow look on all of the modules we initially marked. So, we have a few iterations.
1097270	1106540	We most of the times we find critical vulnerabilities. It's not easy to continue on
1106540	1115720	when you get to a dead end and when you get to multiple dead ends. But you're already into it
1115720	1124320	and you don't want to raise a white flag and surrender and to the research team in the office
1124320	1129280	to laugh about you. So, you want to continue on. You want to prove that you actually can
1129280	1135020	break this device. And it's really important not to never to give up.
1136700	1141200	That sounds like there might be a little bit of an extra story there about being
1141200	1146180	laughed at in the office, but certainly not one that we can need to talk about today.
1148240	1154280	Were you ever able to find a way to overcome the 60 second limit in the exploit?
1158100	1168280	Sadly, no. Technically, there is a grace period that once the user requested a bridge to
1168280	1175880	search for new light bulbs, we have roughly 60 seconds. In some experiments,
1175880	1183400	it wasn't exactly 60 seconds. It looked more like two minutes. But essentially, there's a
1183400	1193140	timer. Most probably some thread on the main CPU initiates a timer. And when the timer expires,
1193220	1200400	a message is sent to the modem and the modem shuts down the communication. So, unless we starve
1201440	1209560	the thread that is responsible for the timer, the timer will expire and we will be blocked out.
1209560	1215620	And it is a good security design. We can't send messages to the bridge
1217360	1222960	unless the user requested a bridge to look for new light bulbs. So, it is a good design.
1222960	1229580	It blocks most of the attack vector. However, there's no reset button for the light bulbs.
1229580	1236080	And there's no indication of please just pair this light bulb again, according to the addresses
1236080	1242940	of the light bulb. So, essentially, the design started quite good, but broke somewhere in the
1242940	1250900	middle. We did find out that if we bombed the controller with multiple messages, we can starve
1250900	1258180	the rest of the threads. But it has some disadvantages as well. Because if we starve
1258180	1264700	the rest of the threads, then the watchdog will restart the bridge. And we mostly don't want that
1264700	1274400	to happen. So, the 60 seconds is... it is a hard limit. We can't really do a lot about it.
1275360	1279560	Excellent. So, yeah, do we have anything else currently coming through the
1280420	1285820	Discord chat? Yes, we have a question. Did you start working on IoT exploitation at Checkpoint,
1285820	1288680	or is this an area you've been working on beforehand?
1289640	1298780	That's a good question. It really depends how you define IoT. If IoT is anything that is not
1298780	1308020	Windows, Linux, or mobile, then I started working on embedded devices practically from my first
1308020	1322680	year on work. If we rule out network equipment, for example, then actual embedded devices
1323680	1333740	were from after three or four years of work. So, it wasn't the first device that I encountered.
1333740	1340360	But I did research embedded devices way before I started working on Checkpoint.
1341680	1348610	Excellent. And did you ever think of attempting to brick the other bulbs or possibly the controller?
1351620	1358340	There is a sad story about that. Because if you break the device,
1358960	1365320	you also lose your own device. And then you need to go and purchase a new device and convince some
1365320	1370600	manager to once again buy it. And then you need to tell the manager, yeah, we need to buy it again,
1370600	1380180	because we bricked the earlier one. And you can't do that often. We had that on the research in the
1380180	1388240	previous year. We presented a DEF CON ransomware on DSLR cameras, and we had to test the patch from
1388240	1396100	the vendor. But if the patch will work, we'll get locked out from the device, and we won't be able
1396100	1404640	to demonstrate the attack, which we wanted to demonstrate on a live demo. So, once we noticed
1404640	1411820	that it will be some issue about it, we went to buy an additional camera. So, if we lose one of
1411820	1420420	them, we still had a backup. So, it is quite easy to break the devices. And most of the times,
1420420	1425580	you don't want to do that, because you lose your own research device and you need to buy a new one.
1425580	1433600	A few years ago, I actually bricked my device on a different research project. And if you break
1433600	1441040	your device, then people really laugh. Because, yeah, he bricked his device, we need to buy a new
1441040	1446080	one, and you don't want to be that researcher. So, you really don't want to break your own device.
1446700	1453640	Sounds like quite the environment at work that you have there. So, it seems like a lot of times,
1453720	1459760	a lot of years at DEF CON, you know, the big story comes out about this big thing is insecure,
1459760	1464620	there's going to be these problems. And you're talking about how that people could get into a
1464620	1470660	home network, or maybe even a business network through a light bulb. So, what do you think about
1470660	1476740	overall security concerns based on your research? Is this something that people at large should be
1476740	1482020	really concerned about? Or is it being handled really well by the manufacturers?
1483500	1489720	I wouldn't trust the manufacturers at all, specifically in IoT devices. It doesn't look
1489720	1498540	like security is a top priority in their devices. If we look specifically on the light bulbs case,
1498540	1504600	you would be more worried about a neighbor hijacking all of my light bulbs and turning them
1504600	1512660	to whichever color they want. Because they could have done that years ago, even from the conclusions
1512660	1521240	of the previous researchers. And it's a flaw in ZigBee, you can't fix that. So, specifically in
1521240	1527680	light bulbs, anyone can hijack your light bulb and steal them to control them from their own
1527680	1534700	network. And there's nothing really you can do about that. If you're worried about someone
1534700	1543880	attacking your bridge and infiltrating your network, most probably it won't happen a lot.
1544560	1549560	We are not releasing our full exploit, and we work quite hard on the exploit,
1549560	1559680	it will be reliable and will work correctly. If you're a real target and some high
1561880	1569160	authority wants to hijack your network, they could do that. Will someone do that to your
1569160	1578220	neighbor? Most probably not. That said, on the light bulbs, two years ago, we showed you we can
1579020	1586500	infiltrate the network from a fax machine. And then the attack was easier, more reliable,
1586500	1596900	and I can be in China and still attack your all-in-one printer. So, that one is more concerning.
1598060	1604120	And I would worry more about using a fax than using smart light bulbs.
1604120	1611100	Excellent. Sighed, how are we doing with that Discord channel there? Coming up with anything
1611100	1618220	else, or should we kind of start giving him some wrap-up questions? So, we had one question about,
1618220	1623820	did this research ever give you the kind of insight you started out with on why people
1623820	1632020	want connected light bulbs? It actually did. It turns out that smart light bulbs are actually
1632020	1640060	quite cool. I read about a few people in the north which use the light bulbs and calibrate them
1640480	1648700	to match the season and the clock. So, they will still get sunrise and sunset, even if they don't
1648700	1656760	actually have that way in the north because of the season. You have quite a lot of control over
1656760	1663720	the light bulb, both the intensity and the color itself, and you can sync multiple light bulbs to
1663720	1673780	get a scene. And it is really cool. It isn't very practical because, for example, if there's a power
1673780	1682960	shutdown, then all of your light bulbs go reset, and then you lose all of your scheme, all of the
1682960	1689860	colors, and all of the intensity, and you get all of the light bulbs go white, go to the maximum
1690700	1700000	lighting, and it happens quite a lot. The power has some sort of glitch, and you're not supposed
1700000	1707820	to connect your light bulbs to a backup system or to a UPS or anything of this sort. So, if you're
1707820	1716360	into the colors of the light bulbs, you need to have pretty good electricity in your city or in
1716360	1723800	your state. Otherwise, you need to recalibrate your light bulbs again and again. That's quite annoying.
1724160	1732540	That's great. So, what would be your one main takeaway from your research or call to action?
1732540	1738910	To happen after this? What should people take away from your research and your presentation?
1739640	1749140	So, if I have a message to manufacturers, then please compile with security mitigations and use
1749140	1756460	the latest versions of each library. It really adds a large amount of security. If it's for
1756460	1766220	researchers, then it turns out that practically any device that you can think of using any
1766220	1773860	protocol will most probably still be vulnerable. On ZigBee, we have a maximal transmission unit of
1773860	1783020	127 bytes, and we still reliably exploited the vulnerability. So, even with really harsh
1783020	1789760	constraints and really esoteric protocols, we can still consistently, year over year,
1789760	1796940	present in DEF CON that we broke an additional device. So, everything practically could be
1796940	1803580	broken. And even if you have harsh constraints, you most probably will still be able to exploit
1803580	1811260	the vulnerability once you find it. So, if you pick a target, don't give up. Most probably,
1811260	1817140	you could get a win out of your research. So, just try hard enough.
1817640	1821820	Let's slide one last question in there for you, from Brief Halo. What recommendations
1821820	1827620	would you give to someone interested in studying IoT exploitation formally, like at a university?
1830260	1837000	There are some good courses in several universities, but usually reading write-ups
1837000	1845040	from researchers that published their research, and you read about it, and it was interesting.
1845040	1851340	Reading the write-ups usually gets you a lot of tips. So, if you read the publications from
1851340	1856740	Colin O'Flynn, you get a lot of new knowledge about hardware, and how UART works, and how
1856740	1863220	UBIT works, and you can learn a lot from practical write-ups of researchers. It will be more
1863220	1869780	practical than most of the courses in many universities. That's great. All right. Thank
1869780	1875000	you so much for doing this. This was a great talk with you. I really enjoyed your presentation as
1875000	1880060	well. Thank you so much for doing this. Thank you. All right. Have a good night,
1880060	1884040	and we will be back in about 30 minutes with another speaker.