start	end	text
1120	7220	Hopefully everything will be working. Check the audio real quick and start into the questions.
13630	22250	Alright. Looks like we are on Twitch. Can you verify that we've got audio available?
22990	28270	Yeah, let me verify that. Standby. We're going to click over here and I have to turn myself on.
28330	29130	Yeah, let me verify that.
29130	29970	Yeah, we're in.
29970	30990	Alright, great.
30990	38090	Well, I'd like to welcome everybody to the Q&A on the Hacking Supply Chain presentation by
38870	43570	Sholbi Oberman, Moshir Khal, and Ariel Song.
43810	47530	And I hope I came halfway close to getting those names right.
47670	56230	And I have the distinct pleasure of congratulating three brand new DEF CON speakers.
56230	61850	And so we have a position with DEF CON where we share a shot with the new speakers.
61850	71170	And so, although I'd love to be handing you a bottle right now, I'll ask you to take your favorite drink and join me in a quick shot.
72970	75510	Cheers on your talk.
76310	77110	Cheers.
82970	85770	Okay, so that concludes the Q&A session.
87450	89950	It's been good. Good questions.
90650	93690	Well, yeah, the jokes don't get any better, guys. I'm sorry.
94390	98650	No, we do actually have some questions already queued up on the chat.
98650	105390	So the first one is, other than the one UPS you guys demoed, what other sort of devices did you successfully exploit?
108250	116110	Yeah, so for the remote code execution, we exploited two different vulnerabilities on two different devices.
116270	120370	They appear in our white papers with full details of the exploits.
120370	125410	One is the UPS, and then the other one is called a DigiConnect module.
125410	129050	It's a system or module for Ethernet connections.
129090	132290	And then there's a few different devices that embed that module.
132290	137650	The exploit would work the same on any device embedding that system or module.
137670	139650	But that's just those two.
139790	144710	And then for information leak and just denial of service and stuff like that,
144710	150230	we exploited on quite a few different devices just to make sure they're vulnerable.
152290	154170	Great. So we have another one.
154970	158790	You have the track identified script that you provided,
158790	162310	but we were wondering if there were more active indicators found,
162310	165670	and if there will be an update to the track identification script.
170390	177110	So we're staying with the same identification script that we've used to date.
177110	184650	We have seen some false positives, and we're trying to investigate that and see why we get those false positives.
184650	187870	There's something interesting going on.
187870	190810	The reason we're getting false positives, and we're not quite sure what it is,
190810	192930	so there might be updates on that front.
193530	200750	And we have released, together with McAfee, we've released IPS-IDS signatures.
200750	203910	So I'm not sure if that's the question or regarding detection.
203970	209790	But for detection, we're using the same scripts, which anyone can just ask for,
209790	211350	and we'll send you by mail.
211350	221750	And there's some false positives with those scripts, unfortunately.
224730	227070	Great. So we have another question about,
227070	234970	is there a way to safety-release consumer projects based upon ESP32 or ESP8266?
235050	238710	If no, what hardware development kit would you recommend to start with?
239770	241670	Is that relevant to your talk?
243430	245890	I think it's a slight off topic.
251360	256140	Okay. Moving on.
256580	263000	So what research methodology was your use? Did you manually fuzz, review, or something else?
263000	265560	I know from your talk, you talked about static analysis,
265560	268760	and I was actually interested in hearing more about how you did that.
270000	276020	Yeah, so we mainly reverse-engineered, and we were starting with a track demo
276020	281360	they were offering on their website, and now they removed it.
281360	287220	And then we purchased this DigiConnect device, and we found some headers online.
287680	293960	And we had luck, because this DigiConnect device has symbols,
293960	301160	so we can compile custom firmware, and we have debugging capabilities.
301680	310220	So we reversed the firmware, and that's basically how we found most of the vulnerabilities.
310900	315560	And along the way, with the research, we purchased more devices,
315560	319060	and found some more variants of the vulnerabilities.
320440	322380	So, yeah.
322820	329340	However, on the UPS device, that's relevant regarding our methodology of exploitation,
329980	332940	we saw a slightly different version of Trek.
333200	340080	And we also had... well, we didn't invest much in creating debugging capabilities,
340080	345280	so we didn't have a JTAG connector, or a PDB interface, or nothing of that sort.
346460	352400	So we relied mainly on laborious static analysis.
354460	364520	And also some relatively informative crash dumps, small stack traces, stuff like that.
365880	366980	So, yeah.
368940	377860	So researching heap corruptions using static analysis, I think, may be not traditional.
379560	382520	So had you had better tools, would you have used them?
382520	386680	Or do you think you got the results because you went the way you did?
388100	392280	Well, if we had better tools, we would definitely use them,
392280	396960	but we didn't see fit to invest in creating these tools.
397760	401500	Just because the exploitation primitive is relatively strong.
401500	404460	It's a controlled overflow of the heap.
404460	409340	And the heap on this device is very simple, rather deterministic.
409380	413680	So we decided to go for the low-hanging fruit first, and paid off.
414500	423360	For the actual vulnerability research, I think the manual static analysis was really the way to go.
423360	427960	Just because there were so many different architectures and different versions,
427960	429880	and different types of devices.
429880	437260	So there wasn't one tool that could be used to emulate all of these,
437260	439540	or fuzz all of these, or something of that sort.
440380	443860	And there were a lot of special cases of different types,
443860	447100	even architectures that weren't supported by IDA, and stuff like that.
447100	452260	So, yeah, I think this was a case for manual analysis.
454100	456660	Well, great. So we have another question.
456660	461760	Your second whitepaper, is it independent of the first one, or did it build on the first one?
463280	464540	Completely independent.
465040	467840	I mean, they're related because they're the same vulnerable.
467840	472780	They're the same Ripple20, but different vulnerability, different device, different network.
477780	481740	So for the DNS vulnerability, does the shellcode have to be alphanumeric?
481740	486120	If so, is it possible to get an RCE on other architectures?
490500	497100	Well, on the UPS device, as we said earlier, we saw a slightly new version of Trek.
497100	499440	You can also see this in our presentation.
499740	505580	That on some versions, alphanumeric characters and DNS domain names are enforced.
505960	509600	And that's the overflow, you have to use alphanumeric characters.
509600	512720	But on some versions, it is not enforced.
513340	516300	So on the UPS device, it was enforced.
516480	520680	And luckily, the processor was also x86 based.
521980	523960	That's a solved problem.
523960	527320	I mean, alphanumeric shellcode and x86 processor.
527720	537400	So, I mean, it could probably be done in theory in a different way.
537400	543560	For example, you can have a location primitive that allocates non-alphanumeric data with the heap.
543640	549460	Such as just, you know, the DNS packet data, for example, is not alphanumeric, it's in the heap.
549640	552580	So in theory, you could do something different.
553720	555440	But we just chose the easy path.
555440	558500	So yeah, it is exploitable on other architectures as well.
559540	560500	Great.
560780	564640	Yeah, I think it depends on the heap implementation also.
566220	579140	So some strike versions use heap which is not validated and asserted as the UPS heap.
579240	587220	So in these circumstances, it's probably more easy.
589300	592400	Great. So another question came in.
592400	596940	Are there similar TCPIP stacks that might be interesting to look at?
600440	603920	That's a really interesting question.
605720	615220	So with one of the vulnerabilities, we're not going to say which, but we looked for reference in other TCPIP stacks.
615220	624720	And then we found that about half the TCPIP stacks that we looked at have the same variants of the thread vulnerabilities.
624720	628040	And that's research we're going to be releasing in the future.
630020	635100	So different TCPIP stacks seem to make the same types of mistakes.
635100	640440	So that's a good way to do research into TCPIP stacks.
641420	645820	And then as far as which TCPIP stacks to look at, there's a bunch.
645820	647660	Just open Google.
648880	660080	Depending on if you want proprietary or open source, just type in your search and you'll find that nobody's hiding the available TCPIP stacks.
660080	665440	There's a bunch of proprietary or open source stacks out there.
665440	671500	There seems to be a follow-up question that expands out a bit beyond just the TCPIP stack,
671500	676740	since that was just an example of a library at the root of a huge supply chain.
676740	684480	And did you see other libraries or other stacks that you would consider evaluating next?
685620	687100	Yeah, we did.
687460	694880	But we're still deciding what to evaluate next, so we're keeping those to ourselves at this point.
694880	698760	Did you see other super interesting stacks with a really long supply chain?
700440	704660	I mean, that's how IoT is built, just like Lego blocks, right?
704660	710800	So just take any device, break it apart, when you get to the smallest piece, that's your research target.
713600	716200	Fantastic. So another question came in.
716200	721180	How do you explain how you got to, quote, hundreds of millions of affected devices, unquote?
725460	731600	So there's quite a lot of devices of different types, and we sort of tried to sum them up.
731600	737880	The majority of the numbers actually come from Intel devices and HP devices.
737960	743060	And then Digi also adds quite a big number of devices.
743720	751120	But we just sort of tried to assess, given open information, how many of each of these types of devices is sold per year.
752000	754080	To reach hundreds of millions.
754600	760160	If you sort of count all the devices that contain the code, you'd probably reach billions.
760160	768060	If you count all the actually affected and turned it on, it will be between tens of millions and hundreds of millions.
768820	771140	So it sort of depends how you count.
771280	776480	But yeah, most of them are printers and Intel AMT.
777760	778800	Great.
780720	791520	So one question I had was, when I look at your website, the company in your presentation, it looks like there's quite a big team that was involved in pulling this together.
792240	801620	I mean, it sounds like the core of you all were at JSOF, but how did you rope in the collaborators that you used on this?
803760	817020	So the vast majority of the work was done at JSOF, with Moshe doing most of the work from the beginning, finding the vulnerabilities, etc.
817060	821600	And Ariel doing the exploitation work.
821860	830680	And then for very brief periods, we used some outside help just when we were under timeline or something like that.
831640	837500	Most of the people that you see there, most of the names actually work for JSOF.
839120	841440	You'll see a few names that don't work for JSOF.
841440	848480	One is the kid of one of the employees who we just mentioned because his mother was away.
848800	852180	And then somebody else that helped along the way.
852180	855540	But most of the work, 99% of the work was done in-house.
855800	856720	Very cool.
857360	860000	I have a really bad question. Are you ready for it?
860000	861840	How'd you pick Liani?
863760	871600	I mean, you talked about in your presentation why you gave the 20, but why Ripple?
874220	878940	Okay, so we'll give you the real answer about why 20 that nobody knows.
878940	888460	And the real reason why 20 is because if we called it Ripple 19, then people would associate it with the current affair, current situation.
888460	890220	So that's why we went for 20.
890220	896760	I mean, there's other reasons, but that's the behind the scenes just between you and me and all our viewers.
897340	899800	The Ripple is because of the Ripple effect, right?
899800	905120	This one vulnerability that sort of expands and the impact expands and expands and expands.
905120	908900	It's how we experienced it.
908900	923120	But it's also how over the years Threadstack sort of went from device to device and got wider and wider effect and wider and wider impact and embedded in more and more devices.
923360	929940	So we felt like that really captures the supply chain Ripple effect.
930340	932960	That's cool. I like what you're going with on that.
932960	935380	So how big this one got?
935380	937300	Obviously, this got a lot of attention.
937300	939100	There were a lot of people talking about this.
939100	944840	How has the added attention to your work changed the way that you do your work?
944840	946380	Has it changed your daily life?
946380	950880	Have you found more people are asking you questions about more stuff?
953840	956400	So it's been quite busy.
957400	963120	We've seen all kinds of responses, all kinds of questions, all kinds of reach out.
965120	976860	Mostly to date, most of the stuff has been sort of reactionary network operators, you know, want to figure out what to do about Ripple 20.
977200	986260	And then, of course, some researchers interested in the work, reaching out, asking questions, companies wanting to collaborate on helping mitigate and fix these issues.
987820	994000	We're hoping for sort of more long term, more companies thinking about the next, we're calling it Ripple 21.
994000	995780	Like, how do you prevent the next time?
995780	999680	How do vendors use security hygiene, exploit mitigations?
999680	1002400	You know, how they look into their supply chain?
1002400	1005940	Because I think that's like, this is not the last time this is happening, right?
1005940	1008240	Somebody asked about good research targets.
1008240	1012680	So there's already people in the audience looking for the next supply chain vulnerability.
1013720	1017320	And, you know, I think by this time next year, we'll have another one for sure.
1017920	1021460	Probably not us going to be publishing it, but somebody will.
1021620	1029160	So I think that's the interesting talk about how do we go forward from Ripple 20.
1031740	1033680	Great. So we have a question.
1033680	1043680	How do you evaluate the patchability of such high level supply chain vulnerabilities down the line, especially with regard to less frequently updated IoT?
1047990	1057710	So we sort of know, it's called common knowledge or something that we say that some devices are unpatchable and some of the companies we know ceased operations.
1057990	1061470	Some of the companies are still investigating whether they are vulnerable.
1063050	1068410	So it's going to be difficult. Many of the devices are not going to be patched for years just for those reasons.
1068410	1070630	Some of them it's difficult to patch.
1070630	1079470	We have seen companies reach out and saying, we can't patch these devices for a while because they're too critical.
1080570	1085390	And they serve a lot of customers and we can't afford any downtime.
1085390	1091750	And they're sort of trying all kinds of virtual patching, network level mitigations.
1092910	1096290	I'm not sure how effective that is. I guess it could be effective if done right.
1096290	1102590	But we definitely assume many of the devices are not going to be patched for years.
1102590	1105830	That's my personal assumption.
1107750	1114910	So I guess a follow-up question to that would be, if you're a developer of IoT, what's the lesson to take from this?
1118400	1119760	Hey, Gabe.
1121800	1127380	So I think one lesson is look into your supply chain and understand their security.
1127940	1130000	Ask them the right questions.
1130100	1136300	And another question is prepare for vulnerabilities using good exploit mitigations and security hygiene.
1136620	1143800	So some of the vendors affected were actually able to release a statement saying, yes, we're affected, but it's not that bad.
1143800	1154300	Like Greenhills and Intel to some extent with some of their devices, because they use different types of exploit mitigations and security hygiene.
1154300	1160000	And then even though they didn't know about the Ripple 20 vulnerabilities, the impact was lesser.
1160960	1176760	And then even with the Schneider UPS devices, which of course have been patched since, the heap was a stronger version of the heap than we've seen in other devices,
1176760	1187960	and therefore was slightly more difficult to exploit than, say, the DigiConnect device that used an older allocator, which was less safe.
1187960	1193560	So third-party risk supply chain and exploit mitigations are the top two.
1196300	1201120	Great, thank you. So another question from the audience, more of a general research question.
1201180	1204000	When starting a project like this, how does the timeline work?
1204160	1212480	You say, like, let's give it a month and then you give up, or you allocate three people until they figure out something, or is it some other approach?
1215980	1222640	So it's always difficult with research, but we've been doing this for a while.
1222640	1228560	Our timeline worked more or less like this.
1228560	1236340	We took the DEFCON and also Black Hat call for papers finish date.
1237080	1240280	We took as much time as we thought we might need back.
1240280	1254960	And then we also scheduled time for the disclosure process, the coordination process, etc.
1254960	1257660	And that's when we started the specific research.
1258300	1262520	And then towards the end, towards the disclosure date, we were scrambling with the exploit.
1262520	1267120	And we actually pulled the exploit together in about three weeks from start to finish.
1269100	1274900	But we sort of just dedicated a specific amount of time to hit DEFCON and Black Hat.
1275740	1279900	Oftentimes, it's very, very hard to schedule this type of research.
1279900	1282440	You never know what you're going to find, how long it's going to take.
1285480	1291520	Great. So I guess a follow-up question would be, I mean, did you feel confident you were going to find something when you started?
1291520	1294620	Or was this mostly a hunting trip?
1297540	1304140	There's always something. I don't think we've ever encountered a target where we haven't found anything.
1307640	1309740	So, yeah, there's always something.
1313140	1313980	Great.
1316400	1329920	So I think you answered this a minute ago, but one of the questions that came up out of band was, did the UPSs devices that you demonstrated, did those have patches available?
1330220	1332220	Did I hear you say yes now?
1333780	1347120	They do have patches available. Of course, you have to patch your device using the patches, but they are available. You have to install them.
1350580	1358180	Great. So we've reached that point where my abilities to do radio completely fail me because I've run out of questions.
1358940	1367120	So folks, please keep the questions coming in the last eight minutes we've got here.
1367120	1378060	I will offer you guys a chance to elaborate on anything that you wanted to add to your talks or any additional points you wanted to make while we're looking for more questions.
1378060	1388280	So there was an additional question we started with. You had several things you talked about at the very beginning of your presentation.
1388880	1394980	These are a number of vulnerabilities, yet you showed just one.
1394980	1404100	Would you talk a little more maybe about why just the one and any other things we can look forward to seeing in the future?
1404100	1409020	I know you talked about it a little bit, but there are some other questions about what are the work you're doing?
1413240	1423080	We chose this vulnerability because we think it's sort of the most interesting vulnerability, and we had a cool exploit on a demo to show that it's real.
1424060	1433760	We think it's cool because it will bypass NAT, right? The DNS request will exit your corporate network potentially, depends how it's configured.
1434700	1447500	I think we're mostly done with Drupal 20 because we're looking forward to our next research project in the research lab.
1447940	1454000	There are some offshoots, some other stuff that we found out that we're still working on and we're going to be releasing shortly.
1454280	1458180	We do have some new research that we're going to be releasing in a few months.
1458180	1468580	And then, of course, we do a lot of work for our customers. Most of our work is just for customers. We do penetration testing and research and stuff like that.
1469260	1485200	From the research lab's side, a little bit more about Drupal 20, some new stuff we found out. Something that we sort of released only on Twitter, but we never called it documented on a white paper.
1486100	1510100	And the listeners might find interesting is that the track stack actually supports encapsulation of IP and IP, and it will forward packets to itself if the destination address is the right one, which is really cool because that can help you bypass firewalls and other network equipment that might stop exploitation.
1510100	1519300	And we found that out when we wanted to test the exploit routes over the internet, not this exploit, but the one for the DigiConnect.
1520960	1529620	And just our company firewall was blocking it, so we encapsulated the packets and they bypassed the firewall.
1529620	1545320	So that's something new, but mostly we're working towards our next research, and I think in a couple of months we should be able to release that. It's currently under disclosure, a coordinated disclosure process.
1546200	1553420	Great. Yeah, unfortunately your future research becomes DEF CON talks, you won't get to drink at those.
1555280	1556720	That's only the first time.
1557440	1568780	We did have one more question come in and said, is JSOF going to create any tools like Nmap, NSF, etc. to help bind the devices that may have these issues?
1571380	1582180	So we already have detection scripts. You can send us an email, the address is on our website, and you can get the detection scripts.
1582180	1591000	And also, correct me if I'm wrong, but we have the filters for... what's it called?
1593840	1606960	So we have... there's passive detection scripts for the exploits themselves. We have some passive information about how to detect the track stack and some active detection scripts.
1606960	1612660	You can email them to us, we also plan to release them open source, we just haven't gotten around to it yet.
1613440	1621400	They are also available, I think Tenable released them as a package and Zeek released them as a package.
1622160	1626260	Probably a few others have released them as a package, but those are the ones we know about.
1626260	1636460	And yeah, until we release them open source, you can just email us. Anyone whose legitimate request, we'll just send you the scripts.
1638400	1642280	Take us a couple hours and you'll get the scripts in your email.
1642880	1647460	Great, and if I remember correctly, your contact information is in the beginning of your talk, correct?
1649460	1659700	I hope so. If not, it's just ripple20 at jsof-tech.jsof-tech.com.
1662320	1665820	Hopefully someone was typing faster than I did when you said that.
1665980	1676160	So we do have one last question. What specific recommendations would you have for third-party library vendors that could minimize the possibility of vulnerabilities like this?
1678180	1686500	So code review would be a big one. Exploit mitigations would be another one.
1686500	1698160	But for third-party library suppliers like Crack that are relied on by a pretty big amount of IoT and critical devices,
1698160	1704040	I'd say penetration testing and a really good security review and secure code review,
1704040	1707920	as well as just securing the design is absolutely necessary.
1707920	1713060	Because doing this once for Crack, it's just like we found one vulnerability and it affects everyone.
1713540	1716380	Like one secure code review would protect everyone.
1716380	1722220	So the deeper you are in the supply chain, or well I guess deeper depends on where you're coming from,
1722220	1731580	but the further at the beginning of the supply chain you are, the more you should be investing in your security.
1731580	1740580	And the more your clients should be asking you about your security and probably will be going forward.
1741720	1752420	Yeah, I think one of the lessons from your talk was to recognize the fact that these things do have, like you said, that ripple effect.
1752420	1763940	And if you're a maker of libraries, be careful of where your code may end up at if it has those vulnerabilities in it.
1763960	1777660	I don't think anybody building TCP IP stacks were imagining that they could be protecting incubation technologies or other medical systems that keep people alive.
1779640	1785740	So I think a lot of these device manufacturers, they do realize where they end up.
1785740	1790580	The thing is they started writing the code 20 years ago when nobody was thinking about security.
1790740	1794700	And then the code probably worked really well and just nobody wanted to touch it.
1794700	1805780	And somewhere along the line, all these companies sort of forgot that they probably revamped their own security processes and activities.
1805780	1811120	But maybe they figured if there's something in a third party, it's like a third party problem.
1811120	1814740	Or maybe they didn't realize they've been using this library for 20 years.
1814920	1820060	So I don't think that it's that Treck didn't know where the code ends up.
1820060	1824560	It's just 20 years ago, nobody was talking about security. And that's when the code was written.
1824560	1832820	A lot of this code is really old. Not just Treck. All IoT code. A lot of the code is not very fresh code.
1834680	1839160	Well, great, guys. So we're at the top of the hour. I want to be respectful of your time.
1839200	1841800	I really appreciate you guys taking the time to take these Q&A's.
1841800	1845680	And I appreciate all the questions that have come in on the Discord chat.
1845860	1848320	Any last words before we call this, folks?
1851860	1857080	All right, guys. We really look forward to more good work from you guys.
1857760	1859680	All right. Have a good one. Thank you.
1860520	1861000	Bye.