Hopefully everything will be working. Check the audio real quick and start into the questions.
Alright. Looks like we are on Twitch. Can you verify that we've got audio available?
Yeah, let me verify that. Standby. We're going to click over here and I have to turn myself on.
Yeah, let me verify that.
Yeah, we're in.
Alright, great.
Well, I'd like to welcome everybody to the Q&A on the Hacking Supply Chain presentation by
Sholbi Oberman, Moshir Khal, and Ariel Song.
And I hope I came halfway close to getting those names right.
And I have the distinct pleasure of congratulating three brand new DEF CON speakers.
And so we have a position with DEF CON where we share a shot with the new speakers.
And so, although I'd love to be handing you a bottle right now, I'll ask you to take your favorite drink and join me in a quick shot.
Cheers on your talk.
Cheers.
Okay, so that concludes the Q&A session.
It's been good. Good questions.
Well, yeah, the jokes don't get any better, guys. I'm sorry.
No, we do actually have some questions already queued up on the chat.
So the first one is, other than the one UPS you guys demoed, what other sort of devices did you successfully exploit?
Yeah, so for the remote code execution, we exploited two different vulnerabilities on two different devices.
They appear in our white papers with full details of the exploits.
One is the UPS, and then the other one is called a DigiConnect module.
It's a system or module for Ethernet connections.
And then there's a few different devices that embed that module.
The exploit would work the same on any device embedding that system or module.
But that's just those two.
And then for information leak and just denial of service and stuff like that,
we exploited on quite a few different devices just to make sure they're vulnerable.
Great. So we have another one.
You have the track identified script that you provided,
but we were wondering if there were more active indicators found,
and if there will be an update to the track identification script.
So we're staying with the same identification script that we've used to date.
We have seen some false positives, and we're trying to investigate that and see why we get those false positives.
There's something interesting going on.
The reason we're getting false positives, and we're not quite sure what it is,
so there might be updates on that front.
And we have released, together with McAfee, we've released IPS-IDS signatures.
So I'm not sure if that's the question or regarding detection.
But for detection, we're using the same scripts, which anyone can just ask for,
and we'll send you by mail.
And there's some false positives with those scripts, unfortunately.
Great. So we have another question about,
is there a way to safety-release consumer projects based upon ESP32 or ESP8266?
If no, what hardware development kit would you recommend to start with?
Is that relevant to your talk?
I think it's a slight off topic.
Okay. Moving on.
So what research methodology was your use? Did you manually fuzz, review, or something else?
I know from your talk, you talked about static analysis,
and I was actually interested in hearing more about how you did that.
Yeah, so we mainly reverse-engineered, and we were starting with a track demo
they were offering on their website, and now they removed it.
And then we purchased this DigiConnect device, and we found some headers online.
And we had luck, because this DigiConnect device has symbols,
so we can compile custom firmware, and we have debugging capabilities.
So we reversed the firmware, and that's basically how we found most of the vulnerabilities.
And along the way, with the research, we purchased more devices,
and found some more variants of the vulnerabilities.
So, yeah.
However, on the UPS device, that's relevant regarding our methodology of exploitation,
we saw a slightly different version of Trek.
And we also had... well, we didn't invest much in creating debugging capabilities,
so we didn't have a JTAG connector, or a PDB interface, or nothing of that sort.
So we relied mainly on laborious static analysis.
And also some relatively informative crash dumps, small stack traces, stuff like that.
So, yeah.
So researching heap corruptions using static analysis, I think, may be not traditional.
So had you had better tools, would you have used them?
Or do you think you got the results because you went the way you did?
Well, if we had better tools, we would definitely use them,
but we didn't see fit to invest in creating these tools.
Just because the exploitation primitive is relatively strong.
It's a controlled overflow of the heap.
And the heap on this device is very simple, rather deterministic.
So we decided to go for the low-hanging fruit first, and paid off.
For the actual vulnerability research, I think the manual static analysis was really the way to go.
Just because there were so many different architectures and different versions,
and different types of devices.
So there wasn't one tool that could be used to emulate all of these,
or fuzz all of these, or something of that sort.
And there were a lot of special cases of different types,
even architectures that weren't supported by IDA, and stuff like that.
So, yeah, I think this was a case for manual analysis.
Well, great. So we have another question.
Your second whitepaper, is it independent of the first one, or did it build on the first one?
Completely independent.
I mean, they're related because they're the same vulnerable.
They're the same Ripple20, but different vulnerability, different device, different network.
So for the DNS vulnerability, does the shellcode have to be alphanumeric?
If so, is it possible to get an RCE on other architectures?
Well, on the UPS device, as we said earlier, we saw a slightly new version of Trek.
You can also see this in our presentation.
That on some versions, alphanumeric characters and DNS domain names are enforced.
And that's the overflow, you have to use alphanumeric characters.
But on some versions, it is not enforced.
So on the UPS device, it was enforced.
And luckily, the processor was also x86 based.
That's a solved problem.
I mean, alphanumeric shellcode and x86 processor.
So, I mean, it could probably be done in theory in a different way.
For example, you can have a location primitive that allocates non-alphanumeric data with the heap.
Such as just, you know, the DNS packet data, for example, is not alphanumeric, it's in the heap.
So in theory, you could do something different.
But we just chose the easy path.
So yeah, it is exploitable on other architectures as well.
Great.
Yeah, I think it depends on the heap implementation also.
So some strike versions use heap which is not validated and asserted as the UPS heap.
So in these circumstances, it's probably more easy.
Great. So another question came in.
Are there similar TCPIP stacks that might be interesting to look at?
That's a really interesting question.
So with one of the vulnerabilities, we're not going to say which, but we looked for reference in other TCPIP stacks.
And then we found that about half the TCPIP stacks that we looked at have the same variants of the thread vulnerabilities.
And that's research we're going to be releasing in the future.
So different TCPIP stacks seem to make the same types of mistakes.
So that's a good way to do research into TCPIP stacks.
And then as far as which TCPIP stacks to look at, there's a bunch.
Just open Google.
Depending on if you want proprietary or open source, just type in your search and you'll find that nobody's hiding the available TCPIP stacks.
There's a bunch of proprietary or open source stacks out there.
There seems to be a follow-up question that expands out a bit beyond just the TCPIP stack,
since that was just an example of a library at the root of a huge supply chain.
And did you see other libraries or other stacks that you would consider evaluating next?
Yeah, we did.
But we're still deciding what to evaluate next, so we're keeping those to ourselves at this point.
Did you see other super interesting stacks with a really long supply chain?
I mean, that's how IoT is built, just like Lego blocks, right?
So just take any device, break it apart, when you get to the smallest piece, that's your research target.
Fantastic. So another question came in.
How do you explain how you got to, quote, hundreds of millions of affected devices, unquote?
So there's quite a lot of devices of different types, and we sort of tried to sum them up.
The majority of the numbers actually come from Intel devices and HP devices.
And then Digi also adds quite a big number of devices.
But we just sort of tried to assess, given open information, how many of each of these types of devices is sold per year.
To reach hundreds of millions.
If you sort of count all the devices that contain the code, you'd probably reach billions.
If you count all the actually affected and turned it on, it will be between tens of millions and hundreds of millions.
So it sort of depends how you count.
But yeah, most of them are printers and Intel AMT.
Great.
So one question I had was, when I look at your website, the company in your presentation, it looks like there's quite a big team that was involved in pulling this together.
I mean, it sounds like the core of you all were at JSOF, but how did you rope in the collaborators that you used on this?
So the vast majority of the work was done at JSOF, with Moshe doing most of the work from the beginning, finding the vulnerabilities, etc.
And Ariel doing the exploitation work.
And then for very brief periods, we used some outside help just when we were under timeline or something like that.
Most of the people that you see there, most of the names actually work for JSOF.
You'll see a few names that don't work for JSOF.
One is the kid of one of the employees who we just mentioned because his mother was away.
And then somebody else that helped along the way.
But most of the work, 99% of the work was done in-house.
Very cool.
I have a really bad question. Are you ready for it?
How'd you pick Liani?
I mean, you talked about in your presentation why you gave the 20, but why Ripple?
Okay, so we'll give you the real answer about why 20 that nobody knows.
And the real reason why 20 is because if we called it Ripple 19, then people would associate it with the current affair, current situation.
So that's why we went for 20.
I mean, there's other reasons, but that's the behind the scenes just between you and me and all our viewers.
The Ripple is because of the Ripple effect, right?
This one vulnerability that sort of expands and the impact expands and expands and expands.
It's how we experienced it.
But it's also how over the years Threadstack sort of went from device to device and got wider and wider effect and wider and wider impact and embedded in more and more devices.
So we felt like that really captures the supply chain Ripple effect.
That's cool. I like what you're going with on that.
So how big this one got?
Obviously, this got a lot of attention.
There were a lot of people talking about this.
How has the added attention to your work changed the way that you do your work?
Has it changed your daily life?
Have you found more people are asking you questions about more stuff?
So it's been quite busy.
We've seen all kinds of responses, all kinds of questions, all kinds of reach out.
Mostly to date, most of the stuff has been sort of reactionary network operators, you know, want to figure out what to do about Ripple 20.
And then, of course, some researchers interested in the work, reaching out, asking questions, companies wanting to collaborate on helping mitigate and fix these issues.
We're hoping for sort of more long term, more companies thinking about the next, we're calling it Ripple 21.
Like, how do you prevent the next time?
How do vendors use security hygiene, exploit mitigations?
You know, how they look into their supply chain?
Because I think that's like, this is not the last time this is happening, right?
Somebody asked about good research targets.
So there's already people in the audience looking for the next supply chain vulnerability.
And, you know, I think by this time next year, we'll have another one for sure.
Probably not us going to be publishing it, but somebody will.
So I think that's the interesting talk about how do we go forward from Ripple 20.
Great. So we have a question.
How do you evaluate the patchability of such high level supply chain vulnerabilities down the line, especially with regard to less frequently updated IoT?
So we sort of know, it's called common knowledge or something that we say that some devices are unpatchable and some of the companies we know ceased operations.
Some of the companies are still investigating whether they are vulnerable.
So it's going to be difficult. Many of the devices are not going to be patched for years just for those reasons.
Some of them it's difficult to patch.
We have seen companies reach out and saying, we can't patch these devices for a while because they're too critical.
And they serve a lot of customers and we can't afford any downtime.
And they're sort of trying all kinds of virtual patching, network level mitigations.
I'm not sure how effective that is. I guess it could be effective if done right.
But we definitely assume many of the devices are not going to be patched for years.
That's my personal assumption.
So I guess a follow-up question to that would be, if you're a developer of IoT, what's the lesson to take from this?
Hey, Gabe.
So I think one lesson is look into your supply chain and understand their security.
Ask them the right questions.
And another question is prepare for vulnerabilities using good exploit mitigations and security hygiene.
So some of the vendors affected were actually able to release a statement saying, yes, we're affected, but it's not that bad.
Like Greenhills and Intel to some extent with some of their devices, because they use different types of exploit mitigations and security hygiene.
And then even though they didn't know about the Ripple 20 vulnerabilities, the impact was lesser.
And then even with the Schneider UPS devices, which of course have been patched since, the heap was a stronger version of the heap than we've seen in other devices,
and therefore was slightly more difficult to exploit than, say, the DigiConnect device that used an older allocator, which was less safe.
So third-party risk supply chain and exploit mitigations are the top two.
Great, thank you. So another question from the audience, more of a general research question.
When starting a project like this, how does the timeline work?
You say, like, let's give it a month and then you give up, or you allocate three people until they figure out something, or is it some other approach?
So it's always difficult with research, but we've been doing this for a while.
Our timeline worked more or less like this.
We took the DEFCON and also Black Hat call for papers finish date.
We took as much time as we thought we might need back.
And then we also scheduled time for the disclosure process, the coordination process, etc.
And that's when we started the specific research.
And then towards the end, towards the disclosure date, we were scrambling with the exploit.
And we actually pulled the exploit together in about three weeks from start to finish.
But we sort of just dedicated a specific amount of time to hit DEFCON and Black Hat.
Oftentimes, it's very, very hard to schedule this type of research.
You never know what you're going to find, how long it's going to take.
Great. So I guess a follow-up question would be, I mean, did you feel confident you were going to find something when you started?
Or was this mostly a hunting trip?
There's always something. I don't think we've ever encountered a target where we haven't found anything.
So, yeah, there's always something.
Great.
So I think you answered this a minute ago, but one of the questions that came up out of band was, did the UPSs devices that you demonstrated, did those have patches available?
Did I hear you say yes now?
They do have patches available. Of course, you have to patch your device using the patches, but they are available. You have to install them.
Great. So we've reached that point where my abilities to do radio completely fail me because I've run out of questions.
So folks, please keep the questions coming in the last eight minutes we've got here.
I will offer you guys a chance to elaborate on anything that you wanted to add to your talks or any additional points you wanted to make while we're looking for more questions.
So there was an additional question we started with. You had several things you talked about at the very beginning of your presentation.
These are a number of vulnerabilities, yet you showed just one.
Would you talk a little more maybe about why just the one and any other things we can look forward to seeing in the future?
I know you talked about it a little bit, but there are some other questions about what are the work you're doing?
We chose this vulnerability because we think it's sort of the most interesting vulnerability, and we had a cool exploit on a demo to show that it's real.
We think it's cool because it will bypass NAT, right? The DNS request will exit your corporate network potentially, depends how it's configured.
I think we're mostly done with Drupal 20 because we're looking forward to our next research project in the research lab.
There are some offshoots, some other stuff that we found out that we're still working on and we're going to be releasing shortly.
We do have some new research that we're going to be releasing in a few months.
And then, of course, we do a lot of work for our customers. Most of our work is just for customers. We do penetration testing and research and stuff like that.
From the research lab's side, a little bit more about Drupal 20, some new stuff we found out. Something that we sort of released only on Twitter, but we never called it documented on a white paper.
And the listeners might find interesting is that the track stack actually supports encapsulation of IP and IP, and it will forward packets to itself if the destination address is the right one, which is really cool because that can help you bypass firewalls and other network equipment that might stop exploitation.
And we found that out when we wanted to test the exploit routes over the internet, not this exploit, but the one for the DigiConnect.
And just our company firewall was blocking it, so we encapsulated the packets and they bypassed the firewall.
So that's something new, but mostly we're working towards our next research, and I think in a couple of months we should be able to release that. It's currently under disclosure, a coordinated disclosure process.
Great. Yeah, unfortunately your future research becomes DEF CON talks, you won't get to drink at those.
That's only the first time.
We did have one more question come in and said, is JSOF going to create any tools like Nmap, NSF, etc. to help bind the devices that may have these issues?
So we already have detection scripts. You can send us an email, the address is on our website, and you can get the detection scripts.
And also, correct me if I'm wrong, but we have the filters for... what's it called?
So we have... there's passive detection scripts for the exploits themselves. We have some passive information about how to detect the track stack and some active detection scripts.
You can email them to us, we also plan to release them open source, we just haven't gotten around to it yet.
They are also available, I think Tenable released them as a package and Zeek released them as a package.
Probably a few others have released them as a package, but those are the ones we know about.
And yeah, until we release them open source, you can just email us. Anyone whose legitimate request, we'll just send you the scripts.
Take us a couple hours and you'll get the scripts in your email.
Great, and if I remember correctly, your contact information is in the beginning of your talk, correct?
I hope so. If not, it's just ripple20 at jsof-tech.jsof-tech.com.
Hopefully someone was typing faster than I did when you said that.
So we do have one last question. What specific recommendations would you have for third-party library vendors that could minimize the possibility of vulnerabilities like this?
So code review would be a big one. Exploit mitigations would be another one.
But for third-party library suppliers like Crack that are relied on by a pretty big amount of IoT and critical devices,
I'd say penetration testing and a really good security review and secure code review,
as well as just securing the design is absolutely necessary.
Because doing this once for Crack, it's just like we found one vulnerability and it affects everyone.
Like one secure code review would protect everyone.
So the deeper you are in the supply chain, or well I guess deeper depends on where you're coming from,
but the further at the beginning of the supply chain you are, the more you should be investing in your security.
And the more your clients should be asking you about your security and probably will be going forward.
Yeah, I think one of the lessons from your talk was to recognize the fact that these things do have, like you said, that ripple effect.
And if you're a maker of libraries, be careful of where your code may end up at if it has those vulnerabilities in it.
I don't think anybody building TCP IP stacks were imagining that they could be protecting incubation technologies or other medical systems that keep people alive.
So I think a lot of these device manufacturers, they do realize where they end up.
The thing is they started writing the code 20 years ago when nobody was thinking about security.
And then the code probably worked really well and just nobody wanted to touch it.
And somewhere along the line, all these companies sort of forgot that they probably revamped their own security processes and activities.
But maybe they figured if there's something in a third party, it's like a third party problem.
Or maybe they didn't realize they've been using this library for 20 years.
So I don't think that it's that Treck didn't know where the code ends up.
It's just 20 years ago, nobody was talking about security. And that's when the code was written.
A lot of this code is really old. Not just Treck. All IoT code. A lot of the code is not very fresh code.
Well, great, guys. So we're at the top of the hour. I want to be respectful of your time.
I really appreciate you guys taking the time to take these Q&A's.
And I appreciate all the questions that have come in on the Discord chat.
Any last words before we call this, folks?
All right, guys. We really look forward to more good work from you guys.
All right. Have a good one. Thank you.
Bye.