start	end	text
31450	33990	Hey there! Welcome everyone.
34070	38030	We are kicking off the Q&A session here with Ayub,
38030	43450	who spoke about Only Takes a Spark Popping a Shell on 1000 Nodes,
43450	45610	which was pretty awesome, pretty cool.
45610	49070	A good way of thinking about how to scale up everything.
49510	51150	And, you know, not just hacking one system,
51150	53450	but how can you use that to really spread out.
53890	57950	So, really, I heard this is also your first time speaking at DEF CON.
57950	59030	Is that correct?
60890	62050	Exactly, yeah.
62490	63270	So, we have...
63270	66030	I was actually going to give it... yeah, go ahead.
66090	69430	No, yeah, so I was just going to say we have a tradition here at DEF CON,
69430	71410	which we call Shoot the Noob.
71430	73490	And so, really, it's just the tradition of, you know,
73490	76910	hey, taking a shot on stage for, you know, your first thing.
76930	78310	It's not always liquor, right?
78310	79790	So, people mistake that.
79790	82730	But, really, just want to do that with you now.
82730	85950	So, I want to welcome you in the traditional DEF CON fashion.
86170	87590	So, cheers!
88210	89930	Oh, boy. Okay, okay, okay.
89930	91130	Oh, you got to pour it first.
91130	91770	Wait for it.
92490	93990	I got to pour it first.
93990	94810	You had to prep.
94810	95870	There you go. Okay.
96050	97110	There we go.
97790	98410	Cheers, guys.
98410	98930	Cheers.
103470	104190	Awesome.
105430	110090	So, now that you've been officially indoctrinated to the DEF CON and DEF CON safe mode.
111150	114870	Yeah, so, one of the first things, questions I'd ask, right?
114870	116970	So, right, you recorded the talk a little bit ago.
116970	118810	What have you been working on since?
118810	120990	Anything new that you've kind of come up with?
120990	122510	Or, you know, anything you found?
122510	125450	Anything, you know, that's changed since the talk was given?
127350	129490	It has nothing to do with Spark, actually.
129490	132450	I was working on Spark, like, a few months ago now.
132690	136050	But since then, I completely switched subjects and topics.
136050	138630	I've been working on AWS a little bit.
138970	144150	I did, actually, a tool that reflectively loads DLLs and executables in memory.
144150	146010	But I wrote it using Golang.
146010	149250	So, I completely abandoned that area of research.
149250	151650	And I completely switched over to something completely different.
151650	153990	So, we'll see what comes up next.
154130	156630	Good for you. That's a pretty good way to go.
156750	159270	You get some recognition for one thing.
159270	162530	And then you move off to something totally cool, totally new.
162650	163530	Excellent.
163930	167190	Yeah, definitely like this idea of discovering new stuff.
167710	168570	Excellent.
168690	171610	So, is that a thing you find yourself doing often in your own research?
171610	177530	Is that you want to get a nice breadth of experience in a lot of different things?
177530	183990	Or how deep do you go before you feel like you're comfortable with what you've learned?
186150	187570	I think I can...
187570	194130	Basically, my idea is go as deep as you start to understand how it works, basically.
194310	196630	Before, I was looking into mainframes.
196630	203810	And that took me, I think, two years of going through these 900-page documents published by IBM
203810	209210	and these obscure systems and obscure forums actually talking about it.
209210	212270	So, I did the same thing with mainframes.
212270	213610	And then I let it go.
214110	217190	And I switched over to other areas.
217190	219950	And during that work, I think Spark came up.
219950	222090	And I was like, what the hell is this thing?
222190	223050	And I dug into it.
223050	225950	And I found there was not much research was being done on it.
225950	227970	And I thought, well, you know what would be fun?
228870	229870	So, I actually dug into it.
229870	231770	And that's how it all started, really.
231890	236550	It was just going after the next shiny thing, trying to understand how it works.
236550	240990	And once you understand how it works, you try to bend its rules to do whatever you want with it.
241210	246450	And hopefully, write a tool about it, give a talk about it, and then move on to the next stuff.
247050	248010	How deep?
248210	249970	Just enough to understand it, really.
251210	252030	That's the goal.
252030	257850	So, what was it about Spark, really, other than the fact that it was new and there wasn't a lot of research?
257850	261950	I mean, what was it that really dragged you into this one?
261950	266970	Because it's an interesting system, as you showed us in your talk.
267250	274770	So, what was the thing that made you decide, okay, fine, this is where I'm going to spend the next year of my life?
274770	276190	Yeah, and why not continue?
276190	278850	All right.
278850	283870	So, I was working on the offensive side, and then I switched to the blue side.
284310	287010	And I was helping a company secure their systems, etc.
287010	291330	And they were all on these new shiny platforms, if you will.
291330	295210	Everything was on AWS, multi-region, CICD.
295210	297390	I mean, they didn't click a single button.
297390	300750	They pushed everything to code, and everything was deployed and scaled and stuff.
300750	302670	Very, very sexy stuff.
302670	307130	So, I was trying to help them secure that stuff.
307370	316070	And after six or seven months, we thought we did a pretty good job locking pretty much everything that was supposed to be locked.
316070	319450	There was no Windows, by the way, so that's why we came to this state.
319450	325370	But anyway, and then I was talking with this data scientist, just trying to understand what they do.
325770	328770	And he mentioned something about Spark.
328770	330450	And I was like, what is that?
330450	334910	And he told me, oh, it's something that we use to make calculations and parse data.
335430	337150	I mean, what do you mean, parse data?
337650	338950	How many machines do you have?
338950	343270	And he's like, oh, actually, it's like, I have 200 machines.
343270	344510	It's like, what do you mean, you have?
344510	351050	It's like, oh, I spawned a cluster of 200 machines, and that guy over there spawned a cluster of 500 machines, etc., etc.
351050	355350	So, basically, the company had like thousands of machines running sporadically, if you will.
355350	359210	But still, and I was like, okay, well, how do you walk me through it, please?
359210	364470	And he showed me how he launched the job and how he did some calculations, and I was like, oh, my God.
365810	370230	And that's when, where the all interest Spark came in, basically.
371030	371850	That's awesome.
371850	373450	That's a great place to start.
373450	379410	Yeah, you start seeing, oh, a couple of commands, and all of a sudden, there's this many machines doing something.
380070	381730	I want to know that, too.
381730	389810	He was writing Python code on a Jupyter notebook, where, you know, there was an authentication, obviously.
390250	396610	And he was writing code, and it executed over multiple machines, and I was like, dude, this is amazing.
398330	402670	And so I dug into it, and yeah, that's what it all sounded like.
402670	409890	All right, so it's a pretty specific setup that you are talking about in your presentation.
409890	416010	How common is the setup that you're speaking about?
416010	422310	And how many other ways are there to configure this that might have more research opportunities?
422610	432710	Well, the thing is, basically, a Spark cluster can be set up in very different ways.
433590	443910	The cluster manager, so basically the process, the component that is responsible for linking applications to workers, that one can be replaceable.
444070	447850	Not whatever you want, but you can put many different other components.
447850	457730	In my talk, I only briefly demonstrated when it's Spark itself that is doing the orchestration.
457730	464290	But you can have other setups where it's actually Yarn, a product from the new framework, that is doing all this orchestration.
464910	468730	And I think in 3.0, you can even have Kubernetes doing the work.
468730	471610	So you have all your pods coming up to do all the work.
471610	473610	So it's much sexier.
473610	481230	And the one that I showed, the one that bypasses authentication, that one only works on Spark standalone mode.
481230	484010	So when it's actually Spark that's doing all the stuff.
484010	488530	If you're having Yarn in front, it's a completely different story.
488530	492710	It's a completely different protocol that's been going on.
492750	496510	That's some Hadoop shit that's too much.
496610	498260	That's a completely different beast.
498610	500610	It listens on a different port, etc.
500610	503310	The tool that I released, Sparky, handles it.
503310	506770	But I didn't go much deeper into it.
506770	511190	So there's definitely some area of research there to go into.
511190	516530	And that's the default mode when you're using AWS Managed Service called EMR.
516530	522690	So if you're using EMR and you spawn a cluster using their service, it will by default use this Yarn mode.
522690	528150	So it will spin up Yarn and then the work will be done by Spark Worker.
529030	532450	How much is its widespread ratio?
532830	538890	From the studies that I saw, it was around 50-50, maybe 40-60, depending on which website you see.
540690	546590	Basically, the more traditional websites, the companies that had Hadoop before, will stick to Yarn.
546590	551990	Whereas the new ones that have the luxury of starting a cluster of data mining from scratch,
551990	556110	will maybe opt more for a Spark standalone cluster.
557330	558730	Okay, that makes sense.
558730	564350	And 50-50 means that there's stuff out there, both more to study elsewhere
564350	567790	and more to try and hit with the research that you've done.
567790	569210	So, interesting.
570070	570910	Yeah, I think so.
570910	573650	I would say that...
573650	574910	Here's the thing.
575070	579030	The idemposite community is so much focused on Windows, I find,
579690	583770	that there's so many other great stuff to talk about and to research.
584330	589210	In Windows, if you want to make a breakthrough, you have to go through 20 years of past research
589650	591150	to try to find something new.
591150	595650	Whereas in this big, shiny, new technology, well, it's right there.
595650	598890	Buffer overflows of 1999, it's right there.
598890	602610	So, there's much opportunity there to be taken.
603650	604930	That's awesome.
604970	606710	Yeah, and I'll ask a question too.
606710	610730	You mentioned how much more efficient Spark is over Hadoop.
611210	616110	Would you say companies that are still heavily relying on Hadoop are behind the times?
616110	618970	Should they be looking at Spark, or is there better security there?
618970	620910	What's your take?
621590	622670	Oh my god.
622670	627990	Well, Hadoop has that thing, well, it actually handles Kerberos authentication.
628510	631870	Now, whether that's a good thing or not, that's up for debate.
631870	634370	But, no, not really.
634450	640010	My mantra, really, I work in Blue Team, so this is Blue Team talking.
640010	644310	Basically, production should be boring, to paraphrase what other people were saying.
644310	645310	Production should be boring.
645310	649590	So, if you have a Hadoop cluster of 3,000 machines that's doing the work,
649590	653310	and it's fine, and everything is fine, then, by all means, continue.
653310	655690	If you have a mainframe to do what you need to do.
656190	661770	I've seen a post, actually, that emulates what Spark does, but only using shell commands.
662990	665330	It works. It's much cheaper.
665330	666810	It runs on a single machine.
667190	669870	And you don't have all the partitioning and the shuffling.
669870	671370	You don't have all the network latencies.
671370	674790	You don't have all that crappy stuff that makes it very, very, very slow,
674790	680050	in comparison to keeping everything in memory and just working on a big slice of an object.
680050	690830	So, I mean, hey, whatever works for the company, with that particular set of talent and that particular set of circumstances and data, I mean, go for it.
691030	692770	That's my opinion, yeah.
692770	693890	Yeah, that's awesome.
693890	696290	Yeah, and you mentioned, so you have that Blue Team mindset.
696370	700830	I mean, so, I guess, how would you try to detect what you were doing during these attacks?
700830	708350	Like, are there any particular tidbits you try to give to a company that's trying to secure their Spark instance, other than making sure it's patched?
709210	710730	Very interesting question.
711310	712510	Very interesting.
713350	714530	Here's the thing.
716210	718150	I never thought about that.
720290	723630	That's how de-correlated the two worlds are for me.
723630	728430	It's like, I do Blue Team in the morning and I do Red Team at night, and that's how, again, they are apart.
728430	733170	So, I never thought about, hey, it would be good to release some Yara rules to detect this stuff.
733170	735230	Never crossed my mind.
736790	738890	Future research, there you go.
742710	748390	No, but it's, to me, it's easier to patch.
748990	753730	I know that some, well, a lot of people will not be able to patch.
754610	757070	To me, the first thing you try is you try to patch.
757070	760530	If you can't patch, then you want to detect, then you want to mitigate it somehow.
760530	770590	So, you need to isolate it either through network firewall rules or do something like that to, you know, basically isolate it from the rest of the reducer exposure, if you will.
770590	775770	Now, if you can't do that either, well, you got to detect it somehow.
775770	777330	How would you detect it?
781580	787220	If you have some advanced correlation in place, I think you can detect it.
787220	793660	I'm talking about specifically the exploit of replaying that single theorized object that triggers command execution.
793660	795460	I'm talking specifically about that.
795460	810260	One way to detect it is that, unlike other Spark interactions, it's, like, in that interaction specifically, you only send one command instead of the whole charade of, hey, Spark, hello, which version are you running, master?
810260	812180	Yes, I'm registering the application.
812180	813340	Okay, here are the workers.
813340	816920	You know, you have, like, 80 messages going around.
816920	819200	But instead, in that exploit, you only have one message.
819200	835460	So, if you can, if you have a tool that's advanced enough to make these kinds of correlations say that, oh, this IP address only or this session only initiated that single communication, that single packet, then maybe it's suspicious because it didn't follow up with all the stuff.
836580	844260	That could give you a hint. You can also try the on out of memory errors that are on your machines.
844260	848180	That may be a bit noisy, but that could get you going.
849780	857660	But then again, a hacker could find another way to trigger the execution because I only gave out an example using the on out of memory error.
857900	866600	Another way... yeah, these are the two main things to look for off the top of my head, basically.
867160	867760	Sure.
867760	872160	We got a follow-up question from PC Squared, it appears.
872160	877340	Really well done talk. It looks like you did the magic on the Spark stand alone, like we were mentioning there.
877440	881880	Did you get it working with Yarn 2? Not a big deal.
882700	886400	Not a big deal, if not, but I imagine it's possible.
886500	889960	Also, 100% production should be boring.
891540	892460	Amazing.
893720	895360	Oh, my God. Okay.
895460	896980	So, here's the real story.
896980	901660	So, I found this thing on Spark, and I was like, oh, my God, this is amazing. I feel so great.
901660	904520	You know what? I'm going to follow up. I'm going to see how it works on Yarn.
904520	908860	Then I pulled up Yarn, did the installation, made it work on Sparky, right?
908860	910240	So, let's look up some runes.
910240	913580	So, I turned up Wireshark, and I saw the traffic.
913800	917760	It was on Wireshark, and I was like, oh, my God, I don't want to touch that.
918760	921660	That's how foreign it was. That's how...
921660	924800	Yarn is more difficult than Spark stand alone.
925480	927460	It's really, really more difficult.
927460	930040	And I looked at it, and I was like, fuck it. I don't want to look into it.
930040	935060	That's really what went through my mind when I tried to do that on Yarn.
935340	940880	Now, if you look for a similar vulnerability, I couldn't trigger it.
940880	945100	Like, I was playing around with Yarn a little bit, but I couldn't trigger the same vulnerability.
945460	948680	Does that mean that it's not vulnerable in some way? Of course not.
948680	954820	But by that time, I was like, oh, Yarn is just too much for me.
955340	959200	I'm not going to waste my time on it. I'm going to move on to other stuff.
959760	963740	Maybe if you feel a connection with Yarn, you can dig into it. Please do so.
964000	967440	Because like I said, there's nothing out there on Yarn. I didn't find anything.
967440	971720	So, probably there are some stuff to be sought after in Yarn.
973080	975120	Well, that makes a lot of sense.
975120	985340	So, if somebody decides that they do want to pursue this, pursue the Yarn side, and would like to ask you more questions,
985340	995100	are you available for either consultations or for answering questions that people come up with when they're doing this on their own?
996220	1001360	Yeah, of course. I mean, yeah, definitely. Like, my Twitter is open.
1001360	1008380	Right at the end of this, we'll have you post any contact information you'd like for people to have access to in the TrackOne channel.
1008380	1012400	And you can be there for people.
1013800	1020620	Yeah, definitely. I mean, when you look at this stuff, like when you look at the Spark source code, it's beautifully written, by the way.
1020620	1025640	But it's written in Scala. Some of it is written in Scala. And it can be very daunting.
1025640	1040080	The syntax is weird. And when you look at information on the web that talks about Spark, I think one blog post or two maybe that are dedicated to the internals of Spark.
1040080	1047640	Everybody talks about how shuffling works and how it partitions data and stuff, but nobody talks about how it works inside.
1049580	1062220	So yeah, there's a lot of things to figure out and some simple stuff when you explain them. They look simple, but to get that information, it takes a little bit of time because nobody talks about those internals.
1063000	1069840	So yeah, if you have any questions, please do not hesitate. Just hit me on a DM or just ping me or whatever.
1072520	1080140	Yeah, it's one of the things I thought was interesting too. So through what you discovered, I was like, oh man, I wonder if he's going to report this to Apache.
1080360	1090920	And then like, oh, look, he did. So I guess, how did that interaction go? How long did it take him to fix it? Was it a good interaction? Tips for other people having to reach out to companies like that?
1092060	1093020	Okay.
1094960	1100660	I sent them the phone, I think on 24th of December or something like that. Okay, very bad of me.
1102600	1109840	Then I got a response from the Spark security team, and they said, yeah, okay, we'll look into it, etc. But
1111740	1117740	whatever, and I didn't receive a response later. And then my talk at Troopers got canceled.
1118540	1126700	And I think one week before I decided to, you know, because I was rehearsing a little bit. So I said, you know what, let me just write them again to see where they are on this.
1126900	1132740	And I got a response from the same guy saying, oh, you know what, we looked it over, and we discarded it because it's not interesting.
1133220	1135240	I was like, okay.
1137120	1148760	Let me rephrase. Oh, and when I sent them, I sent them an email with a proof of concept. It's like a Python code that actually executes the code fully. So proof of concept, it's there. So it was really detailed email.
1150100	1163800	And yeah, so the guy said that they discarded it and I didn't understand. So I wrote a second email saying, are you sure you want to do that? Because you're right in the documentation that when authentication is enabled between this and this component, etc.
1164360	1168880	So you're kind of breaking this trust by allowing this vulnerability to go on, etc, etc, etc.
1169400	1176760	And then I got a response saying, oh, my God, I'm so sorry. Yeah, we made a mistake. We thought you were talking about the RPC endpoint that we already got a report for.
1176760	1188140	Now, indeed, this is a dangerous vulnerability, etc. I'm going to put everything. And they, you know, I mean, it happens. I do try it myself and sometimes I get it wrong. So this is normal.
1188760	1199680	And then they went on to fix it. So, all in all, it took, I think, eight months. But since they acknowledged that it was a vulnerability, I think it only took like three months.
1200200	1202140	Gotcha. So persistence was key.
1202940	1217100	Persistence was key. Well, I mean, my goal was not to get it out there, to just publish it. And I didn't release the tool or I didn't even talk about it. I think I talked about it to two people who were very expert in Spark to validate that I was not saying bullshit.
1217560	1225460	And I asked them to not disclose it. That's about it. Because my intent was not to just, you know, release that. I am pro full disclosure.
1226600	1233740	But everybody should make their own choices and it depends on my mood. So for that one, I decided, you know what, let's just keep it on the low key anyway.
1234340	1248500	And, yeah, and I think three weeks after disclosing it, they released a complete correction. Because it was not impacting only one function, it was impacting also two others.
1248500	1257640	So they rewrote a class, etc. So we worked on the fix. Well, I worked on the origin of the fix and then they took it over because more competent than me.
1257640	1274200	And then the thing that was really long is the release process. They had the fix three weeks after I reported it. But since like, let's say, beginning of April, they waited until July on the other version 2.4.6 to actually release the fix.
1275020	1280220	So that's the part that was long and not actually fixing it, just releasing it. But, yeah.
1280420	1287300	It's out there and we're all more secure for your efforts. So it's, you know, better than having no one looked in that area before.
1288680	1297640	Yeah, yeah, definitely. And now we're talking about that. There was some interesting point during basically this whole Spark adventure.
1297640	1306980	And I think I briefly touched upon it during the talk is that I was so, like, I spent like a couple of hours, even days, trying to understand how the RPC stuff works.
1306980	1318300	And only to find out a couple of days later that somebody already published that vault. So, yeah, that was very interesting. I don't know if it happened. I think it happens to some people.
1318300	1329680	But basically, you're just so stuck in, like, you find yourself drawn into that code base and trying to understand how it works when all you have to do is actually Google the right keyword to get the actual exploit.
1331000	1338540	And when you do it, you feel a sense of frustration. But it's part of the game.
1338820	1352020	And that's pretty good advice for all of us who are into finding vulnerabilities and figuring out how to report them is that enumeration step goes on the software that you're working on.
1352020	1359060	And you have to read the state of the art of what people know. So that's a good thing to reinforce.
1359680	1364060	Exactly. Awesome.
1364060	1372640	So tell us a little bit more about the next thing that you're moving on to. It sounds like you already have this idea of what you're researching next.
1374920	1383600	Actually, I wrote down a small plan. Not a plan, but bullet points of what I want to do later.
1384120	1395680	I just released this tool called Reflect PE, which reflectively loads an assembly in a PE object. And shout out to name right, please.
1399860	1400580	Rob?
1402740	1406180	If needed, you can post it in the track one channel at the end.
1406180	1411460	I will. I will post it. Okay. So his name is Rob Knopp. Well, he goes by the handle of Rob Knopp. Sorry.
1411980	1417760	But yeah, I wrote a tool called Reflect PE, which reflectively loads a PE executable in memory.
1418720	1426000	And I borrowed his basically tool to reflectively load also PE assemblies. So Rob Knopp, great tool.
1427100	1445680	And why did I do that? So this is what I'm working on right now. And why did I do it is very, I find interesting, is that when you look at the same code to do the same thing, but in PowerShell and other languages, is that it's a huge script of like 2000 lines of code.
1445680	1452600	No tests whatsoever. And you just launch it. And if it doesn't work, well, too bad. You can't debug a 2000 line PowerShell script.
1452720	1460660	So I decided to write it from scratch using Golang and using like, you know, testing and good dev practices, etc.
1460820	1466620	So that anybody could just look at the code and understand what's going on and where are the steps and where it didn't work, etc.
1466620	1474020	So this is what I'm really working on. I'm trying to make this tool that will actually be easy to understand for people who just get into it.
1474020	1479720	And just want to understand how reflectively loading something works on Windows.
1480460	1487880	So that's the thing that taking most of my time right now. And my next big thing is Kubernetes.
1487880	1501300	I really want to get into it. It is all of the talks that I saw, like 90% of them talk about how to abuse a Kubernetes set up with no authentication, no airbox, no nothing.
1501300	1514900	I want to see what is there to exploit, you know, once you have all these security hardens in place, is it possible to bypass, I don't know, namespace isolation, is it possible to bypass team control, etc.
1514900	1527080	I want to really dig into it because there's, again, like I said, I really look out for these niche topics where there is not much talk, where there is not much tools available.
1527080	1532060	And try to focus on them and hopefully something gets out of this research.
1533600	1538440	That's awesome. That's awesome. Yeah, so we're kind of wrapping up here at the end of the time.
1538440	1543120	Is there anything else you want to share with the group or again, how to reach out to you?
1543120	1549060	Again, we can post your contact information, but anywhere you want to point anyone to before we call it an end here?
1551840	1560820	No, nothing except that, basically, thank you to the Spark team, first of all, because honestly, amazing.
1560820	1571360	Like when you look at the code base, you understand how talented these people are and it's really a pleasure and a privilege to actually have interacted with some of you, especially some of you that I saw on stage talking about it.
1571360	1574500	So I was pretty excited, though I tried to hide it a little bit.
1574620	1576480	Yeah, so thank you all.
1577080	1588840	But again, like for people who want to basically go into InfoSec and break into it, give talks, do research, etc.
1589000	1601120	My only advice is basically look out for all these small niche topics that are basically abandoned by everybody and go deep until you feel you understand it and then just go crazy with it.
1601120	1603740	And then most surely something will pop out.
1604160	1606420	Awesome, awesome. That's great advice.
1606620	1607860	I appreciate that.
1607940	1617100	Yeah, and I was going to say just one more thing, basically, was getting to, you know, since you just actually did a live demo, right?
1617120	1623960	One thing you're missing out on not having the live DEF CON crowd is a round of applause for having a demo work on stage.
1623960	1631260	So since you actually did kind of roll that out, I just wanted to give you a nice round of applause and get that in the Discord chat, too.
1632320	1635560	Thank you for getting that and getting it working and everything.
1636060	1637260	Thank you very much.
1637920	1642900	Cool. So I think that's it for us. So again, thank you again for joining us.
1642900	1647000	And thanks for the talk and look forward to what you have coming up next.
1647000	1648600	And hopefully see you next year at DEF CON.
1648980	1651240	Thank you very much. Thank you.
1651240	1652540	Bye-bye.