start	end	text
2180	5380	So, hello everyone. Thank you for tuning in.
5460	7680	Today, we're going to talk about Apache Spark.
7840	9680	Now, I will just assume that while some of you
9680	11600	may be familiar with the technology,
11700	13840	a lot of you did not have actual chance
13840	14820	to play with the cluster.
14820	17040	So, we'll take it right from the beginning.
17840	19560	So yeah, we'll see how Spark works,
19560	21280	what makes it so popular,
21280	23980	how to view some common and default settings,
23980	25860	and obviously, some bones along the way
25860	29220	to bypass authentication and execute code, right?
29360	30680	So, let's get going, right?
30680	34120	So, our story takes us back to 2008,
34120	37460	when the economy was not much better than today's,
37460	39720	but Internet Explorer was still a thing,
39720	42300	and the big tech companies were doing what they did best,
42300	44480	quietly collecting the world's data.
44560	46040	When you gather that much data,
46040	49160	and I'm talking about terabytes and terabytes of data,
49160	50920	performing even the simplest calculations
50920	52860	becomes quite challenging.
53020	54580	Now, you can either get a mainframe for half a million
54580	57060	dollars, plus whatever it takes to get an intern
57060	60300	to work on it, or you can actually distribute
60300	63300	your files over thousands and thousands of servers.
63520	67600	That way, your computations are small, though multiplied,
67600	70360	and that's one way to handle the load.
70360	73240	And the standard way of doing so, I would say,
73240	75980	is by using the Apache Hadoop framework.
76420	78540	Now, if you've ever worked with Hadoop,
78540	82020	you must know that it's actually quite a hostile environment.
82140	84200	It's a very complex environment.
84300	86680	So, first of all, just to give you a quick example,
86680	90000	you need to fragment your files using the HDFS file system,
90000	94500	so you end up with multiple fragments on multiple servers,
94500	97180	these files, and then you need to do coherent computations
97180	98540	on these small fragments,
98540	100280	so you need the MapReduce framework.
100340	102460	But then you end up with thousands of processes
102460	104100	running on multiple machines,
104100	106320	so you need a way to schedule them, enter YARN,
106320	108080	but then you want to do some SQL querying,
108080	110640	so you need Apache Drill, et cetera, et cetera.
110640	113160	So, you end up with this plethora of tools,
113160	114280	each with their own latencies,
114800	118400	with their own learning curve, et cetera,
118400	120560	just to cover all your needs in data analytics.
120560	122620	So, just to give you an example,
122620	126020	this is what it takes to do a simple word count on MapReduce.
126020	127420	So, this is the first page of Java.
127420	129800	This is the second page, which is downright outrageous
129800	132520	for a data processing solution, right?
133360	136340	So, in our story, we're around 2009,
136340	139680	and there's this guy named Matei Azaria,
139680	141940	who was at Berkeley, I believe, at the time,
142400	144920	who I imagine looked at this code and went,
144920	145880	are you fucking crazy?
145880	148340	That's not going to work.
148560	150600	A word count is a simple operation.
150600	154220	A word count should be done in five lines of code.
154500	155960	Apache Spark was born.
156360	157620	Now, this is not the official story.
157620	158880	The official story is that he was working
158880	161140	on something called Mesos, which is a resource manager,
161140	163000	and he invented Spark as a proof of concept,
163000	165160	but I like my version better.
165340	166860	So, anyway, needless to say,
166860	168780	we'll go through the code later on,
168780	170240	but right off the bat, you can see
170240	172380	that you can do pretty much the same processing
172380	173840	that you would do in Apache Hadoop,
173840	177320	except that you can do it at 10 times less code than Spark.
177560	179820	But it's also much faster.
180280	182840	So, it's three times faster on 10 times less nodes,
182840	184340	so it's 30 times faster.
184340	186340	You see, Hadoop will flush everything to disk
186340	187660	whenever it gets a chance,
187660	190300	whereas Spark will try to keep everything in memory.
190300	191440	It was developed in an era
191440	193160	where memory was getting cheaper and cheaper,
193160	194560	so it kind of makes sense.
194560	196200	But if you're running in a cloud environment
196200	198680	where you pay for the usage,
199820	201640	having something that runs 30 times faster
201640	204540	means that it's 30 times cheaper, actually.
204900	208860	And that partly explains the gigantic boom
208860	213320	and popularity of Spark amongst the big data companies.
213320	216220	If you're serious about your big data processing
216220	218900	and cluster front, you will be using Spark
218900	221180	for at least some part of it.
221500	222780	And so, yeah, Spark boomed,
223430	225240	was adopted by the Apache Foundation,
225240	229500	and it grew to encompass many aspects of data analytics.
229500	230880	If you're gonna do SQL, streaming,
230880	232460	machine learning, graph processing,
232460	233860	you know, it got you covered.
234180	236440	So it's developed half in Java, half in Scala,
236440	239320	but it also has some connectors in Python and R.
239320	241580	So, yeah, awesome.
241640	246320	So basically, when you have this powerful framework,
246320	248360	suddenly, you know, everything looks like a nail.
248360	249980	So if you're gonna do revenue prediction,
249980	251900	you just hook Spark to your, I don't know,
251900	254420	Redshift cluster, download those financial statements,
254420	256280	and then, you know, you can do yourself.
256280	258640	If you're gonna do fraud analysis, user targeting,
258640	260660	same, you hook it up to your Cassandra database,
260660	262360	your Astra buckets, you, you know,
262360	264680	upload or download those log files,
264680	266560	and then you do your computation.
266560	268680	So basically, what I'm saying is that
269180	271680	Apache Spark is often in the,
271680	274320	at the junction of almost every important data store
274320	275460	inside the company.
275700	278240	And that's what made it interesting, at least for me,
278240	281480	because, well, if you can access this tool,
281480	283360	you have pretty much,
283360	286480	you're pretty much open to every database that's out there.
286480	288380	So, you know, how is it protected?
288380	289240	How does it work?
289240	290920	What is its security, et cetera?
291020	292720	And so what I did, I did what I,
292720	294080	any sensible person would do,
294080	295980	I went to the documentation, to their website,
295980	297540	and I went to the security page,
297540	301740	and lo and behold, from this magnificent sentence
301740	305120	that says security in Spark is off by default, well, fuck.
306580	307800	And what does it do again?
307800	311020	Oh yeah, it's this gigantic framework cluster
311020	313200	of hundreds and hundreds of machines
313200	316420	that have access to every database inside the company.
316420	317300	And what does it do?
317300	319720	Oh yeah, it does distributed data processing,
319720	321700	or as I'd like to call it,
321700	324120	it's really distributed remote code execution.
324120	325660	So I thought this would be fun.
326120	327880	And I hope I piqued your interest
327880	329780	because really that's what sparked my,
329780	332300	the whole endeavor, no pun intended.
332620	334440	And I hope I got you interested enough
334440	337080	to actually continue along this presentation
337080	338540	so we can explore how it works
338540	343220	and some bones that I found, right?
343220	346240	So, oh yeah, in keeping with DEF CON's tradition,
346240	349380	I think it's time for drinking.
350500	354240	I should not do this online, but oh, what the heck?
355260	357760	So to DEF CON safe mode, cheers.
361360	362920	Yeah, that will kick in later.
362920	366880	Anyway, so what is Spark and how does it work?
367320	368620	I hope it's going to be one take
368620	370380	because I cannot do that five times.
370460	372000	So how does it work?
372000	374400	So as in everything in distributed processing,
374400	376620	basically we start with a bunch of machines
376620	378220	that we're going to call workers.
378860	382980	Now these workers are just basically brainless machines
382980	384880	that will execute whatever is sent their way.
384880	388040	Now on each worker, you find a JVM process,
388040	391300	so a Spark process that's called an executor
391300	393940	and that's going to do the actual execution.
394440	396700	Now you can have many executors on each worker
397180	398980	and it's the basically executor
398980	402040	that defines your power in terms of parallelism.
402040	404160	So if you have three, you have 10 workers
404160	406200	on each worker or each machine,
406200	407500	basically you have three executors
407500	409740	so you can run 30 tasks in parallel.
411080	413520	So yeah, you have like some default HTTP port
413520	415520	that gives a status of what's going on
415520	417380	on the machine you're not interested in,
417380	419220	but you have the random RPC port
419220	421420	that is set for each executor.
421420	424480	And it's this one that the rest of the cluster contacts
424480	429080	in order to send and receive a status from the worker.
430020	432260	Now the worker is fine, but the most important piece
432260	433660	or one of the most important pieces,
433660	435920	the cluster manager, and that's our second piece.
436500	438240	And the cluster manager's sole job
438240	440660	is to actually schedule the application.
440660	443020	So the work that is coming and say,
443020	444640	oh yeah, this application is going to run
444640	446420	on these two workers, that one is going to run
446420	447440	on the other three workers.
447440	451180	So it has a status of the workers, what they're doing,
451180	454340	are they online or not, are they busy or not, et cetera.
454520	456880	Every component inside Spark communicates
456880	458560	using this Spark RPC protocol
458560	461040	that we will detail a little bit later on.
461040	462400	And just to show you what, you know,
462400	465160	to demystify this Spark cluster,
465160	466460	this is what it looks like.
466880	470600	This is the basic UI of the Spark master on port 8080.
470600	472060	Oh, sorry about that.
472160	474040	So yeah, you can see that we have like, you know,
474040	475820	two workers, one which is dead.
475820	478420	We have an application that's running on two cores.
478420	480660	And since each worker has one core,
480660	482460	so this application is running on two cores.
482460	484840	It's being distributed over two machines.
485640	490140	But the most important port really is port 7077,
490140	492120	because that's the one that we're going to contact
492120	494500	in order to send an application, register it,
494500	497260	and schedule it to be, you know,
497260	500120	executed in parallel on these workers.
500640	502400	And it's going to be these basic hypotheses
502400	504840	that we're going to make along this presentation
504840	509040	that we have access to that, to this specific ports.
509360	511980	And the third component, and indeed the most component,
511980	513860	the most important, sorry, is the driver.
513860	515860	And the driver is really the brain
515860	518140	behind the application orchestration.
518140	519860	It's the one that's going to slice up your application
519860	522640	and small tasks, contact the cluster manager,
522640	524320	receive the worker, and then send,
524320	526280	distribute your workload to these worker
526280	529820	and, you know, make sure that everything is going smoothly
529820	531420	and aggregate the results.
532500	535420	And the driver in a typical case or typical scenario
535420	536840	is going to be running on your laptop.
536840	538660	So you enter a network,
538660	540380	you have the cluster manager running somewhere,
540380	542560	you have the workers running somewhere else,
542560	544340	and you're going to boot up the driver,
544340	547380	write your application in Java, Scala, Python, or whatever,
547380	548340	boot up the driver,
548340	550040	and then send it to the cluster manager
550040	551980	to be scheduled on the workers,
551980	553520	hopefully to execute some code.
554080	558300	So that's one setup that's kind of common.
558900	560620	Right, so recon.
560620	563540	So let's take this specific scenario that I just listed.
563540	564900	Let's say you end up in a network
565360	567500	and you want to find, you want to hunt for some spark
567500	568960	in order to exploit it.
569640	571240	How would you go about it?
571240	573740	Well, the first thing you want to do, I guess,
573740	577440	is basically if you're in an AWS environment,
577440	579040	you can simply use the API,
579040	581120	if you have the proper access rights, obviously,
581120	584460	to look for every machine that has spark, master,
584460	586280	or other keywords in their labels.
586280	588600	Similarly, if you're in a Kube environment, Kubernetes,
588600	589720	it's the same thing.
589720	591700	But what if you end up in a more traditional network
591700	593900	where you have to nmap the shit out of things?
593900	596020	How can you find that cluster?
596080	598920	Because you see nmap does not really support spark.
598920	601420	So we're in a kind of a pickle here.
601420	602720	So the first thing we want to do
603220	606360	is be able to fingerprint spark properly using nmap.
606360	607280	And to do that,
607280	610020	I simply whip out a spark cluster in my lab.
610020	612460	I put some Wireshark in the middle,
612460	614100	and I analyze the traffic.
614100	614800	And if you do that,
614800	616680	this is the blob that you're going to see.
619240	623000	And basically, the way to decompose this is fairly simple,
623000	624240	and it's quite repetitive, actually,
624240	625800	once you get to know Spark RPC.
625800	627880	So this is the Spark RPC I was talking about.
628100	631860	It always starts with the same thing, the same header.
631860	633360	It's going to be, most of the time,
633360	636640	21 bytes of data followed by the payload, right?
636640	639120	And these 21 bytes of data,
639500	642720	a common header when you send Spark RPC commands,
642720	645080	is composed of seven bytes of null bytes,
645080	646620	followed by two magic bytes.
646620	647560	I call them magic bytes,
647560	649940	but basically they depend on the RPC endpoint
649940	650960	that you're trying to reach
650960	652840	and the type of message that you're sending.
652900	654200	So in this case,
654200	658440	I chose the example of verify endpoint,
658440	660700	check existence message.
661260	664180	And its magic byte is C305.
664180	665200	If it's, I don't know,
665200	667620	if it's a worker sending a heartbeat message
667620	668600	to the cluster manager,
668600	671620	it's going to be 2B0F or whatever.
671780	673600	But in this case, it's C305.
673600	674820	But anyway, it doesn't matter.
675300	678280	Next, we have eight bytes of data, like random data.
678280	679760	You can put anything you want here.
679760	683100	They'll just be echoed back by the cluster manager.
683420	686920	And finally, we have four bytes of the size of the data
686920	687700	that's going to follow,
687700	689560	so the payload that will follow.
689560	691120	What is this payload?
691320	693140	Well, you have a bunch of data IP addresses
693140	694240	that we don't care about,
694240	695780	but the most important thing really,
695780	697960	the heart of the payload, if you will,
697960	700440	is this serialized object that you can see here.
700440	703800	So it's a serialized JVM object called check existence.
703800	704960	And if you look at the source code,
704960	706920	check existence is actually a class.
706920	708660	Well, it's a Scala case class,
708660	711840	but just think of it as a regular class, which is immutable.
712220	714600	And it has some interesting other properties in Scala
714600	716920	that we simply don't give a fuck about,
716920	718420	but yeah, it's a class.
718960	722760	So check existence is a class and it has an attribute name.
722760	724500	So basically the client or driver
724500	727320	is sending this serialized class to the cluster manager
727320	730440	to see if there's an endpoint bearing the name
730880	731600	that it sent.
731600	733300	And the name that it sent is obviously master.
733300	737500	So it wants to know if there is a master RPC endpoint
738020	740800	listening on the other end of the communication.
741080	744580	And if that is the case, the cluster manager,
744580	747000	well, will respond first with the 21 bytes of data
747000	749540	that you can see it echoed back what we said before, right?
749540	752700	And a different magic RPC, magic two bytes.
752860	755760	But anyway, and if there is indeed an endpoint,
755760	758720	it sends a Boolean that is set to true.
758740	760920	So obviously I took this special exchange
760920	763180	because you can use it and map to fingerprint
763180	765580	the cluster manager, but you have to imagine
765580	770600	that all Spark RPCs are these exchange of serialized data
770600	773360	that are prefixed with some type of header,
773360	775740	either a 21 bytes header or a 13 byte header
775740	777740	that we will see a bit later on.
777740	781480	But this is the main way that Spark components
781480	783700	communicate with each other, just serialized objects.
784740	787180	So once we have this exchange all mapped out,
787180	790420	then we can use it and write a Lua script
790420	792540	that will replicate the same thing.
792540	793980	Once you do that, obviously,
793980	795720	once you deconstruct it like this,
795720	797080	it becomes quite obvious to do it.
797080	799660	So I wrote an Nmap script that's on my repo.
800180	802580	Sorry, and I will push it to the real Nmap repo
802580	806100	once people have used it enough and debugged it enough.
806660	808980	So yeah, now we are able to identify a Spark cluster.
808980	811100	And you can see they've been true to their promise.
811100	813900	Authentication is indeed disabled by default.
813900	815260	So this is pretty great.
815980	817140	Now comes the interesting part.
817140	820820	Now we know how to locate Spark clusters inside a network.
820820	823700	How can we execute code, right?
823700	825520	Because that's what's really interesting.
825920	828480	And now the standard way of submitting an application
828480	831360	on to a Spark cluster is basically
831360	833500	you write your application in Java or Scala,
833500	837340	what have you, you package it as a JAR file,
837340	838500	you boot up the driver,
838500	841500	and then you send that JAR file to be executed
842540	843960	or parsed, if you will,
843960	845800	by the cluster manager and the workers.
846440	848680	Now, the problem is this,
848680	852640	I find writing Java boring, sorry,
852640	855400	and Scala makes me want to shoot myself.
855400	858100	So I decided to do it in Python.
858640	860820	And luckily, Spark supports Python.
860820	863980	In fact, they have an official wrapper called PySpark,
864820	866280	which actually takes care
866280	868160	of all the nasty stuff that I just described.
868160	869840	So it will boot up the driver for you.
869840	873280	It will serialize JVM object into Python object
873280	874600	and vice versa and send it.
874600	876620	And it will take care of all the nasty bits for you.
876620	879100	So you just write a simple Python.
880260	881420	And so, yeah, PySpark.
881420	882820	And if you pip install PySpark,
882820	884340	it really is just a thin wrapper.
884340	886580	It downloads 200 megabytes of JAR file behind.
886580	887900	So it's hilarious.
887900	889560	Anyway, so from PySpark,
889560	891380	we import these two classes,
891380	892780	SparkContext and SparkConfiguration.
892780	895900	It's very straightforward, very intuitive, I would say.
895900	897140	So you define the configuration.
897140	899200	We're going to name our application WordCount.
899200	902700	We're going to point to the cluster, port 7077, obviously.
902820	905820	Now you have to define your public IP
905820	907220	that you're going to bind on.
907220	908460	We're going to see why later.
908460	911380	But if you forget this line, it will not work.
911400	912820	No word does it say so in the doc,
912820	914620	but if you forget this line, it will not work.
914660	917880	So anyway, so you define the SparkContext.
920300	921840	SparkContext is going to be the client
921840	924700	that's going to initiate the communication,
924700	926700	handle the communication with the cluster manager.
926700	929060	And basically, when the Python interpreter
929060	932760	comes to this line, it will boot up the JVM process.
932760	933860	We talked about the driver,
933860	935060	and it will initiate communication
935060	936340	with the cluster manager.
936360	939440	So quite naively and quite intuitively, I would say,
939440	940940	what we want to do next is obviously
942140	944380	just send the code that we want to...
945120	946400	we want to be serialized
946400	948120	and we want to be executed on the worker.
948120	951660	So from subprocess P open the command ID.
952540	953640	And if you do that,
953640	955900	and this is really the first thing that I did.
956020	957980	If you do that, it actually works.
957980	961880	So I got this result back and I was like, well, hang on.
962160	964080	I know that user, that's me.
964080	967240	I have that user defined on my local machine,
967240	969300	not on the worker or the cluster machines.
969300	970280	What the fuck happened?
970280	972240	So it turns out I actually just executed the code
972240	973200	on my own machine.
973560	975520	So if you look at Wireshark, what happened,
975520	977180	because I was curious, what the fuck happened?
977180	978980	I thought this stuff was supposed to serialize
978980	981280	and send the command to be executed on the workers.
981280	985180	Well, what happened was first we actually sent this,
985180	987420	you know, 21 bytes of data, check existence class,
987420	988920	yada, yada, everything's all right.
988920	991100	There is indeed a cluster manager listening.
991420	994160	Then we send the register application class,
994160	996420	which contains all the information of the application
996420	999000	that we can... that we're going to run.
999000	1001180	So you can see that we send, you know,
1001180	1004660	word count plus another other properties, right?
1004660	1008400	The cluster manager responded with executed added class.
1008580	1011680	So they gave us actually two workers.
1011680	1012760	So this is great.
1012760	1013680	What happened next?
1013680	1016000	Well, the driver, so our instance of PySpark
1016000	1017360	simply unregistered the application
1017360	1019060	and called off the whole thing.
1019060	1020340	So what the hell?
1020520	1022660	Well, to understand what happened actually,
1022660	1025480	remember when I said that Spark contacts,
1025480	1027220	PySpark will simply serialize whatever comes
1027220	1028940	after the Spark contacts line?
1028940	1030140	Yeah, I lied.
1030660	1033480	Well, the thing is, in order to understand what happened,
1033480	1035720	we basically need to talk about two concepts.
1035720	1037120	First of all, Spark APIs,
1037120	1038960	and second, the notion of a DAG
1038960	1040680	and lazy evaluation in Spark.
1040860	1045060	So Spark APIs, counter-intuitively I would say,
1045060	1047480	Spark APIs are not methods, they're data structures.
1048160	1050280	So let's say you want to open a file on Python
1050280	1052180	or Java or Scala or whatever.
1052240	1054860	You point to a file and then you call the method,
1054860	1055740	the open method.
1055740	1058400	This will only give you an object, a Python object,
1058400	1060320	that refers to a handling memory, whatever,
1060860	1063040	but a single threaded object nonetheless.
1063740	1065560	If there is no parallelism there,
1066140	1068040	and the proper way to do it in Spark
1068040	1069580	and still get that parallelism
1069580	1071820	is actually to call text file method
1071820	1073980	of the Spark contacts class.
1073980	1076960	So here SC is an instance of Spark contacts.
1076960	1079500	So you point to a file, you call the text file method
1079500	1082000	and text file method will grab that file,
1082000	1083100	slice it into fragments
1083100	1086260	and then load those two fragments in memory.
1086920	1089360	Similarly, if you want to load a list,
1089360	1091220	just don't define a list like that.
1091220	1093640	This will give you only a single threaded object in memory.
1093640	1095960	No, you need to call the parallelize method,
1095960	1098140	which will take that list, slice it in half
1098140	1101540	or three or four, whatever Spark seems necessary.
1101740	1104240	And that way you will end up with two fragments in file
1104240	1107300	on which you can perform work in a parallel fashion.
1107540	1110080	Now, these fragments are called partitions.
1110140	1111800	And this is a very frequent term in Spark
1111800	1113040	that will come up a lot.
1113040	1117180	Partitions are the main unit of work inside Spark.
1117680	1120820	And these collection of partitions
1120820	1123040	are called resilient distributed datasets,
1123040	1125340	which is really the most fundamental API in Spark.
1125340	1127360	There are others, but they're ultimately all based
1127360	1130300	on RDDs at the core.
1130500	1133680	So once we have these partition,
1133680	1136300	then we can apply parallelized work
1136300	1138680	using something called transformations in action.
1138680	1140000	Now, what's a transformation?
1140340	1141820	A transformation...
1141820	1143280	Now, let's say I want to multiply every element
1143280	1145120	on the list by 20, let's say.
1145120	1147600	A concrete example will actually make it better.
1147780	1149720	So I define a method called multiply,
1149720	1153460	which takes an element, multiplies it by 20.
1153580	1155800	And then I just call the map transformation,
1155800	1159460	which will iterate over every element in every partition
1159460	1161940	and yield the same number of elements
1161940	1163380	in the same number of partitions.
1163380	1164620	So that's a transformation.
1164620	1166140	There are other types of transformation,
1166140	1168580	which may or may not yield the same number of partitions
1169380	1170960	and elements, by the way.
1170960	1174320	So yeah, each Spark API basically defines
1174320	1177060	these transformations and actions to work on.
1177500	1180860	And so what happens is if I write this code,
1180860	1183700	thinking that I just multiplied every element by 20,
1183700	1185840	actually nothing gets sent to the workers just yet.
1185840	1186700	Nothing happened.
1186700	1187720	Nothing gets computed.
1187720	1190340	I will have the same results as I had earlier.
1190340	1191900	In fact, I would have nothing.
1193520	1194440	Why?
1194440	1199140	Because when the driver starts parsing this application,
1199140	1203460	all it does is actually it builds an execution graph.
1204880	1206060	And this execution graph is called
1206060	1207640	the DAG and Spark terminology.
1207880	1209340	But basically all it does,
1209340	1212420	it follows all these calls that we made.
1212420	1215920	So, oh yeah, it knows that it's going to call the paralyzed.
1215920	1218200	So it knows that it's going to build an RDD.
1218200	1219260	It doesn't care what's inside.
1219260	1220820	It just knows that it's going to build an RDD
1220820	1222780	with X number of partitions.
1222860	1224680	Then, oh yeah, we're going to call a map.
1224680	1226120	It doesn't care what's inside the map.
1226120	1227340	It doesn't care that we're going to multiply
1227340	1228460	every element by 20.
1228460	1229780	That's not the driver's job.
1229780	1232460	The driver's job is only to follow,
1232460	1234000	to build this execution graph.
1234000	1235720	So it knows that it's going to apply a map.
1235720	1237720	So given a set number of partition,
1237720	1239620	it's going to have an output,
1239620	1240680	the same number of partition.
1240680	1242920	It's going to continue building up this DAG.
1242920	1244160	Maybe it's going to be a filter map,
1244160	1246460	which is another type of transformation, et cetera.
1246500	1247940	And it's going to continue building this graph
1247940	1249280	until it hits an action.
1249280	1251640	An action in Spark is something that's going to force it
1251640	1253680	to do the actual computation.
1253680	1257220	So think like save file or collect or take sample.
1257220	1260080	All these methods that you can call in an RDD
1260080	1262900	that will force it to actually have the right value.
1262960	1263900	And once you do that,
1263900	1266840	once you call a collect or an action in Spark,
1267220	1268420	then, and only then,
1268420	1270880	will the graph be sent to the executors.
1271220	1274600	The graph, mind you, not the code, simply the graph.
1274980	1276200	And it goes a little bit like this.
1276200	1277940	So the driver will send to the workers
1278460	1279640	the graph of execution.
1279640	1282400	The workers will go through this graph
1282400	1283720	and they go, oh, I need to parallelize,
1283720	1284680	but I don't have that list.
1284680	1285520	Oh, you know what?
1285540	1286360	Give me that list.
1286360	1289020	And the driver will send all these serialized objects,
1289020	1290220	one after the other.
1290380	1291500	I need to do a map.
1291500	1293500	Oh, but I don't have the function multiply.
1293740	1295320	You know, they ask the driver to give,
1295320	1297780	serialize the function, the method,
1297780	1299220	and send it to the workers, et cetera.
1299300	1301140	And then they're going to apply those methods.
1301140	1304380	So each worker will only apply these methods,
1304380	1306920	will only follow the DAG on its own partition.
1307500	1309360	So the worker number one will, you know,
1309360	1311020	loop through the elements of the first partition.
1311020	1314180	So one, two, three, multiply by 20, 20, 40, 60,
1314180	1315780	send back the result to worker number two,
1315780	1316780	will do the same.
1316780	1318460	And the driver will aggregate the results
1318460	1321160	and present it to the operator, us.
1321620	1323840	So when you think about it this way,
1323840	1326840	and you go back to our code earlier with our subprocessor,
1326840	1329960	we're just sitting naked there alone in the wilderness.
1330360	1332220	Well, of course it didn't make it to the workers.
1332220	1334640	The workers are only ever aware of the DAG.
1334640	1337000	And so we need, if we want to execute code,
1337000	1339320	we need to somehow embed it inside the DAG.
1339900	1343360	So we need to put it inside a transformation,
1343360	1344540	hence the following code.
1344540	1346800	So here we can see, we define a Lambda function,
1346800	1347820	anonymous function,
1347820	1352120	and inside we put our P open command execution stuff.
1352320	1354820	And obviously we need to follow it by an action,
1354820	1356400	otherwise nothing gets sent.
1356400	1358820	And this is the skeleton that you need to follow,
1358820	1360640	or the basic skeleton you need to follow
1360640	1363380	in order to achieve code execution on Spark
1363380	1365620	across multiple machines.
1365760	1369200	Now, if you don't want to be bothered by creating like,
1369200	1371520	you know, RDDs, handling parallelism and whatnot,
1371520	1374460	I took this skeleton, put it in a fancier way,
1374460	1375300	or something like that.
1375300	1377540	And I incorporated in a tool called Sparky
1377540	1379780	that will take care of all this stuff for you.
1379780	1381840	So you just, you know, point to a cluster,
1381840	1384480	point to the IP that you want to target,
1385100	1386460	send the command that you want,
1386460	1387780	and then how many times,
1387780	1390160	on how many workers you want it to execute,
1390160	1391620	and it will do that for you.
1391620	1394120	It will build the RDDs, build the APIs,
1394120	1395080	call the transformations,
1395080	1397020	the right ones in order to achieve the code,
1397020	1398620	the execution that you want.
1399040	1400780	And this is what it looks like really.
1400780	1405400	So again, I'm going to publish it on GitHub for everyone.
1405460	1407500	So yeah, you'll have a chance to look at the code.
1407500	1409020	So you can see here, we just point to the cluster,
1409020	1411060	we specify our public IP address,
1411060	1412560	the command that we want to execute,
1412560	1414860	and the number of workers that we want to target.
1417000	1418520	And you can see here that we have execution
1418520	1420740	on three machines, so great.
1420980	1422360	I'd also add some, you know,
1422360	1424700	I embedded some scripts to make it easy
1424700	1426880	to run in an AWS environment,
1426880	1431040	to like dump some AWS secrets if you want,
1431040	1435760	or search for files that are sometimes dumped by Spark,
1435760	1438200	you know, work in some work folder.
1438200	1441800	So you can search for AWS keys or secrets or whatever.
1441800	1445960	So it automates a lot of the pen tests in Spark.
1445960	1451080	So yeah, that will get you started.
1451620	1455760	Next, now in this type of execution that I just showed,
1455760	1458260	there are some problems that are quite annoying.
1458260	1459640	Let me go through them very quickly.
1459640	1462380	First of all, the Python version on the driver
1462380	1464920	needs to match the one on the worker,
1464920	1466620	down to the minor version,
1466620	1468300	otherwise it will throw an error.
1469360	1470380	Now there's a way...
1470380	1472140	Yeah, so you can see even if you have a worker
1472140	1474860	that's on 3.5 and your driver is on 3.7,
1474860	1475580	you're kind of screwed.
1475580	1477400	So there's a way around it.
1477740	1480420	Since the check is simply done by checking the version info,
1480420	1481840	you can just override it.
1481840	1484360	So you can see here that I put it like 3.5.
1484920	1486660	And if you do that, it will work,
1486660	1489000	so long as it is not too much of a gap
1489000	1491500	between the two versions.
1491500	1493840	So it can get away with 3.6 and 3.7
1493840	1497460	because the serialized objects are similar to some extent,
1497460	1499780	but it cannot get away with the 3.4 and the 3.7
1499780	1502220	or there's too much of a gap.
1502740	1505660	And the way around that is simply,
1505660	1507140	well, to override the file,
1507140	1508760	but also to take advantage of the fact
1508760	1511380	that a Spark cluster will use, or PySpark,
1511380	1514900	will use pickle deserializers to deserialize the object.
1515280	1518160	And in pickle, you can specify a method.
1518160	1521880	If you define the method reduce of an object,
1521880	1524360	it will get executed first before,
1524360	1526540	in the process of deserialization.
1526540	1528940	So before it fails,
1529370	1532280	that deserialization process later on,
1532280	1534660	it will have executed the command reduce,
1534660	1536320	which contains the execution command.
1536320	1539000	And so you simply need to define the object
1539000	1540800	and send the method as part of that object,
1540800	1541760	and it will get you through.
1541760	1544640	So even though the job submission fails,
1544640	1545940	you still achieve code execution
1546600	1549340	over however many workers you decided.
1549540	1552280	And I did not incorporate it inside Sparky
1552280	1554060	because it's just a hacky way.
1554060	1556920	Really the easiest way is simply to align your versions,
1556920	1558620	your version of Python with that of the workers,
1558620	1562500	but oh well, this is just a fun trick to share.
1562640	1564400	But really the most annoying problem
1564400	1567280	is actually a network problem.
1567340	1569800	See what happens when we register an application,
1569800	1570920	we send an application,
1570920	1572280	as we contact the driver,
1573660	1576560	as we, sorry, the driver contacts the cluster manager
1576560	1579640	on RPC port 7077, as we just saw,
1579640	1581680	and we'll send the register application class
1581680	1585520	along with the parameters, the application's name, et cetera.
1585520	1587560	The cluster manager will receive that application.
1587560	1589880	Oh yeah, I need three, four workers, whatever.
1590740	1592100	Gonna fetch the workers,
1592100	1595260	send them to the driver along with some status data.
1595560	1596640	But then just when you would think
1596640	1599740	that it would be the driver's job to contact the workers,
1599740	1601860	actually, it's the workers
1601860	1604700	who initiate the communication towards the driver.
1604700	1606820	So they will be the one contacting the driver
1606820	1609500	on a random RPC port called schedule import
1609500	1611940	to say, hey, we're ready, et cetera.
1612440	1615320	And that's annoying because you don't necessarily
1615320	1617640	have this flow of communication open
1618260	1619400	because the driver is, you know,
1619400	1621500	your computer is usually behind the net or whatever.
1621500	1622640	So you're kind of screwed.
1622640	1624420	Not only that, the workers will initiate
1624420	1627060	another communication on another port altogether
1627060	1628720	called the block manager port
1629070	1630860	in order to receive some data blocks,
1630860	1632920	you know, request some serialized objects,
1632920	1634840	broadcast some variables and whatnot.
1634840	1637320	So that's annoying.
1637320	1639160	So in Sparky, by default,
1639160	1641240	I basically, I default these settings
1641240	1643720	to some common ports in order to, you know,
1643720	1647300	evade or it works in some settings, but not all.
1647300	1649440	But really the solution is to use
1649440	1651220	what we call a cluster mode.
1651220	1653200	And in a cluster mode,
1653200	1655260	you don't run the driver on your laptop.
1655380	1658240	You simply contact the cluster manager on port 77,
1658240	1659400	7077, yeah.
1659400	1660500	And then you tell it,
1660500	1662160	oh, here's the application to execute.
1662160	1666660	And please also set up the driver on your own cluster.
1667860	1669540	And that works pretty well.
1669540	1672740	So in this case, we only need access to port 7077.
1672940	1676320	The downside is that you cannot use PySpark.
1676320	1677580	You cannot use Python.
1677580	1681220	So I had to redo that skeleton I showed you earlier
1681220	1681860	in Python.
1681860	1685520	I had to do it in Scala to build a jar file
1685520	1687580	that will parallelize,
1687580	1690240	that will execute whatever command you give it
1690240	1693620	across however many servers that you want.
1693620	1696080	So yeah, the code is again on GitHub,
1696080	1698500	but you don't need to touch this one.
1698500	1700400	It's this compiled jar file.
1701240	1703580	So you can just use it as it is.
1703760	1705200	It's, yeah.
1705460	1707480	So here's a quick example.
1707480	1708900	So you can see we host this jar file
1708900	1710560	that I just showed you on a server
1710560	1712460	that is reachable by the workers.
1712460	1715760	And we simply use Sparky again to build that command
1715760	1717420	to register this in cluster mode.
1717420	1719160	So we just point to this cluster,
1719160	1721560	to this jar file command we want to execute.
1721560	1724680	And then we don't need any network condition
1724680	1727960	except for that 7077 to reach the cluster.
1727960	1729660	And it works pretty well.
1729840	1731660	It will even auto delete itself.
1731660	1733180	So you don't have to worry about that.
1733180	1736320	So this gets us through the network thing.
1736780	1738620	So these are the main ways to execute code
1738620	1742520	on Spark that I found on a Spark cluster
1742520	1746280	that has no authentication, which is enabled by default.
1746420	1750040	So anyway, let's go explore some other facets of Spark.
1750260	1753140	And there's this interesting interface called,
1753140	1754660	well, it's the REST API.
1754660	1755640	And it is interesting
1755640	1757980	because when you go through the documentation,
1757980	1760840	you can see that it's available in port 6066.
1761160	1762660	And when you look at the documentation,
1762660	1765960	it's really a read-only boring API.
1765960	1767480	I was like, yeah, whatever.
1767480	1769400	And I remember a few days later,
1769400	1770760	I was digging through the source code,
1770760	1773180	like really lost inside the source code
1773180	1777280	amongst weird scholarship functions, options and whatnot.
1777280	1780080	And I remember I saw this keyword REST.
1780080	1781660	I'm like, oh my God, I know what that is.
1781660	1783460	And I grabbed it and I followed it.
1783480	1785640	And I came across this piece of code
1785640	1788920	that says something like, submit requests, create.
1788920	1790600	I was like, I like to submit stuff.
1790600	1791760	I like to create stuff.
1791760	1792900	So this thing do.
1793980	1797100	And you follow this, create submission requests.
1797100	1800000	And I was like, let's talk about app resource, main class.
1800000	1801300	I'm like, well, hang on.
1801300	1805840	App resources references a jar usually in Spark code.
1805840	1808500	So I thought this was a read-only API.
1808500	1809620	What the hell is going on?
1809620	1812560	So I followed this, create submission request.
1812560	1813580	And I came across this.
1813580	1816080	Yeah, app resources, indeed an application jar.
1816080	1817720	So basically the REST API,
1817720	1818860	even though the documentation,
1818860	1820700	it mentions it's only a read-only API,
1820700	1822160	actually in the code,
1822160	1824380	it accepts a jar file that you can send it,
1825320	1826560	which is kind of amazing.
1827040	1830440	All you have to do is simply build a JSON file.
1830440	1831820	It accepts a JSON input.
1831820	1834000	So you build a JSON file with the right properties.
1834000	1837480	So the app resource, main class, Spark properties and so on.
1837480	1839660	And then you can send your jar file
1839660	1840940	and then chief code execution.
1840940	1844240	And we can take that jar file we used earlier.
1844240	1846520	So simple app jar, send it a command arc.
1846520	1849840	So this is, it just accepts commands to execute in base 64.
1849840	1852000	So this is really, I think a, who am I,
1852000	1853520	or some shit like, no, not who am I,
1853520	1857760	but touch some file in some folder in base 64.
1857760	1858580	It's very simple.
1859080	1860580	And yeah, but there's different properties
1860580	1862760	and you can just curl this payload
1862760	1865620	and it will execute codes as simple as that.
1866480	1869000	And this is just to show you what it looks like.
1869560	1873380	But yeah, but again, if you use this method,
1873380	1875520	the driver, you don't need it to run on your machine.
1875520	1877040	It's going to run in the cluster.
1877040	1878320	So it's going to run in cluster mode.
1878320	1879660	And one thing I will mention,
1879660	1882800	if you decide to use the REST API or cluster mode in general,
1882800	1884440	you can send whatever jar file you want.
1884440	1886600	You don't have to use my specific jar file,
1886600	1889200	but if you, you will get execution on one worker.
1889200	1891180	But if you want to execute on 200, 300,
1891180	1893160	all the workers inside the cluster,
1893160	1894760	then you need to use Spark API.
1894760	1899320	So you need to use some version of the Spark,
1899320	1901300	like, you know, Spark APIs I just showed
1901300	1904840	in the Scala code earlier, or Java code earlier.
1904840	1908160	So anyway, so this was fun and I thought,
1908160	1910220	well, I wonder if anybody blogged about it.
1910220	1912780	And I Googled create submission request.
1912780	1915420	And I came across the first result on Google
1915420	1917580	was this script and it was already published.
1917580	1919640	And I was like, fuck, so close.
1919640	1921660	Not only that, there was actually an exploit
1921660	1924280	that was already, you know, published.
1924280	1925900	It was actually a Metasploit model
1926440	1928660	that was published by these fine gentlemen.
1928660	1929920	I was like, damn it.
1930180	1933240	Just, you know, sometimes you keep your head in the wheel
1933240	1935200	and you forget to do basic research,
1935200	1937800	but it was just, you know, months away.
1937960	1941460	Anyway, let's get to the real stuff, right?
1942840	1944040	So authentication.
1944280	1946280	Let's say you come across a cluster
1946280	1949180	that has all the right switches on, right?
1949180	1950800	So it's enabled authentication.
1950800	1952880	And the way to do that in Spark is fairly easy.
1952880	1955200	You just enable authenticate to true.
1955200	1957500	Then you define a secret, you can see here.
1958040	1960120	And yeah, all the components you do,
1960120	1962100	you need to do that on every component inside the cluster.
1962100	1966320	So every worker and every cluster manager and the driver.
1966360	1968000	And that way they all have a shared secret
1968000	1968820	to communicate with.
1968820	1970500	And if you have an authenticated cluster
1970500	1972720	and you try to communicate with it,
1972720	1975400	you will get this delightful error that says,
1975400	1977980	you know, you send this check existence class,
1977980	1980820	and then you get this illegal state exception, ACSL message.
1980820	1983140	Now, ACSL is an authentication protocol,
1983140	1984460	challenge response protocol,
1985480	1989320	based on a shared secret, as you might've guessed.
1989920	1992460	But yeah, basically the server sends a nonce to the client
1992460	1994700	and the client hashes that with the nonce,
1994700	1995860	hashes the secrets with the nonce,
1995860	1997840	and then sends the response plus other parameters
1997840	2000240	that we will see, but fairly simple.
2000240	2002500	But the funny thing is that even though the driver,
2002500	2004420	you configure the driver to use the secret
2004420	2006860	and you tell it to authenticate,
2006860	2010020	it will first attempt an unauthenticated session.
2010020	2014480	So it will first, you know, try check existence,
2014480	2016080	get slapped with this error,
2016080	2018260	and then it will try an authenticated message.
2018260	2019020	So it's kind of weird.
2019020	2020760	So anyway, what does it look like,
2020760	2022080	an authenticated session?
2022080	2025540	Well, first the driver will send 21 bytes of data,
2025540	2026900	the header, the usual header,
2026900	2030460	with the magic bytes, this time set to 2B03,
2030460	2032660	as opposed to C05 earlier,
2032660	2034700	followed by, you know, eight random bytes
2034700	2036480	that we don't care about, the size of the payload,
2036480	2038060	which is definitely wrong in this case,
2038060	2042140	because we're not talking, it's not 176 bytes.
2042140	2043500	But anyway, the most important thing
2043500	2044720	in this authentication sequence
2044720	2047520	is that it sends the Spark ACSL user,
2047520	2048480	which is going to be the user
2048480	2052480	used inside the challenge calculation.
2052920	2055340	So anyway, Cluster Manager responds with 21 bytes data
2055340	2056960	as usual, the header,
2056960	2059740	and then it follows by the parameters to,
2059740	2062720	ACSL parameters to perform the challenge computation.
2062720	2066560	So you have the nonce, random value, realm, QOP,
2066560	2067980	which is the, are we doing authentication
2067980	2069680	or encryption or both?
2069680	2071740	These delightful algorithms to use
2071740	2074180	in case we're doing ACSL encryption.
2074180	2075900	Now, just to reassure you,
2075900	2079800	Spark does not use ACSL encryption by default.
2079980	2081780	It uses AES, but it's definitely possible
2081780	2083820	to use it as a fallback.
2083820	2086480	And the algorithm, and this one is indeed used
2086980	2088420	to calculate the hash.
2088420	2089620	And the calculation of the hash,
2089620	2091040	we're not going to do it, honestly.
2092100	2094160	It's detailed in RFC 2831,
2094160	2095700	but you take a bunch of data,
2095700	2097620	you hash it in MD5, you take the output,
2097620	2099380	you combine it with the nonce and other stuff.
2099380	2101380	And then you hashed a bunch,
2101380	2103080	you hash other types of data,
2103080	2105400	and then you combine everything,
2105400	2107720	which gives you the response to the challenge.
2107760	2109820	Again, if you want to know more about it,
2109820	2111320	Google it, RFC 2831.
2111320	2114880	So, the driver sends the response
2114880	2116260	to the cluster manager.
2116840	2119500	If everything checks out, the cluster manager acts.
2120100	2122320	Right, and then it can follow
2123260	2126320	the classic flow of communication.
2126320	2128540	So basically, check existence class,
2128540	2131240	register application, register with application,
2132940	2135140	executors added, yada, yada, yada.
2135560	2136880	Now, here's what bothered me,
2136880	2139160	or what bothered me, what ticked me a little bit,
2139160	2143380	is that this authentication step was only ever done
2143380	2147320	when there was an RPC endpoint that was prefixed,
2147320	2149660	an RPC-ish communication that was prefixed
2149660	2151640	with those 21 bytes of data.
2151640	2154420	But it turns out, there's another header
2154420	2157280	that is sometimes sent before payloads,
2157280	2159960	that prefixes payloads.
2159960	2162520	And this header is only 13 bytes long.
2162520	2165580	And it's usually sent inside the middle,
2165580	2167500	inside the TCP stream.
2167500	2169400	So it's inside the same TCP session.
2169400	2171220	So I did not have much hope.
2171220	2173420	I was like, well, maybe the cluster actually remembers
2173420	2175260	that it's in an authenticated session.
2175260	2177120	So that's why, et cetera.
2177140	2179580	But nevertheless, I thought, well, you know what?
2179580	2181140	Let's see what it looks like.
2181140	2184440	Let's see if we start the communication from here.
2184440	2187100	So we don't send this check existence header,
2187100	2189440	which prompts the authentication message.
2189440	2192120	Let's just start communication from here,
2192120	2193680	by sending these 13 bytes of data,
2193680	2195720	followed by the register application,
2195720	2197320	because that's what we really care about, right?
2197320	2198960	We want to submit an application.
2199560	2200880	That's what we really want.
2201180	2204820	And so if you begin communication at this stage,
2204820	2207080	you first send 13 bytes of data.
2207580	2211320	It looks awfully similar to the 21 bytes header,
2211320	2212720	but basically, oh, sorry.
2212840	2215300	You have four bytes, four null bytes,
2215300	2217540	followed by the size of the payload plus the header.
2217540	2220380	So the size plus 13, a magic byte nine,
2220380	2221900	which didn't change for some reason,
2221900	2223500	and four bytes of data,
2223500	2224900	which are the actual size of the payload.
2224900	2227340	And then you send the serialized register application
2227340	2230000	with all its attributes, you know, application's name,
2230000	2230420	yeah, yeah.
2230900	2234060	And I sent this, and I was expecting a massive error,
2234060	2236400	like, you know, those JVM hundred line errors,
2236400	2239540	but instead I got a header
2239540	2241800	followed by register application class.
2241800	2243760	And I was like, is it me,
2243760	2245920	or is it just possible to bypass authentication?
2245920	2248200	I was like, hallelujah, it's amazing.
2248940	2251640	Now, bypassing authentication is fine,
2251640	2253860	but what can we do with it?
2254420	2257660	Because I was not going to recreate the 80 next messages
2257660	2261300	that were following along, it's not possible.
2261520	2263980	So what can we do with just one message?
2263980	2267580	Well, if you look back at Spark, how it works,
2267580	2269500	the register application class is the one
2269500	2274120	that prompts all that workload on the cluster manager
2274120	2276700	that's gonna contact the workers and say,
2276700	2278540	hey, I've got an application for you,
2278540	2280940	you better get ready, spawn those executors,
2280940	2282700	and get ready to take the work.
2283300	2286240	So I went ahead to the workers and I replayed my message,
2286240	2288520	hoping to see some process spawn or whatever,
2288520	2290120	but I didn't see anything.
2290540	2293600	And the reason for it, I figured out a couple of days later,
2293600	2296440	was because when you send this payload,
2296440	2299260	serialized register application, when you send it,
2299260	2301200	the cluster manager receives it.
2302320	2304660	I immediately shut down the connection,
2304660	2305620	the program terminated.
2305620	2307860	So the cluster manager did not have enough time
2307860	2309880	to actually contact the worker,
2309880	2312900	spawn the process, and let it do its thing.
2313160	2316260	Now, so I immediately shut down the connection.
2316260	2318660	So the way around it, in order to give time,
2318660	2320220	you simply need to do a sleep.
2320220	2321420	As simple as that.
2321420	2322900	You send the register application,
2322900	2324400	you sleep for five seconds.
2324420	2326860	That way you give enough time for the cluster manager
2326860	2330540	to contact the worker, spawn the executor, and do its work.
2330540	2332600	And once you do that, once you do that sleep,
2332600	2334640	I saw, like I put a watcher on the worker,
2334640	2336220	monitoring for processes,
2336220	2340160	and I saw this process getting started.
2340280	2342400	And the magnificent thing about it was that
2342400	2346180	it took as parameters, something that I was,
2346180	2349700	I was defining inside that serialized register application.
2349700	2351920	You know, the block manager report set to 8443.
2351920	2354260	And I knew it was me because I'm the one that said it
2354260	2356660	in order to have communication come from the worker
2356660	2358220	to the driver.
2358580	2362440	So I thought, okay, authentication bypass is a sure thing.
2362860	2364960	Remote code execution is a definite maybe.
2364960	2366360	Because you can influence parameters
2366360	2368560	that are being used to spawn a process.
2368560	2370380	So maybe there's some injection there.
2370380	2373020	So I turned to this register application class
2373020	2375020	and saw like, okay, what's inside?
2375020	2377200	What can we define as parameters as options
2377200	2378700	that we can take an advantage of?
2378700	2380800	So register application, again, a case class,
2380800	2382100	which extends the deploy message,
2382100	2384360	which implements the trade serializable.
2384460	2385720	Don't care about that.
2385720	2387560	Application description, what does it do?
2387560	2389440	Okay, so if you're like me,
2389440	2390920	you're going to ignore all this shit
2390920	2393960	and go straight to command, right?
2393960	2396840	Because the command class, so command case class,
2396840	2397880	it has the main class,
2397880	2400060	which is the class of the executor to be executed.
2400060	2401580	We cannot overwrite that.
2401620	2404280	You have arguments, environments, a bunch of parameters.
2404320	2406860	Now, if you just want to inject like, you know,
2407060	2409320	a semicolon or an and or a pipe,
2409320	2413180	it will not work because these parameters are sanitized.
2413600	2415980	So we need a clever, more clever way
2415980	2417700	of hijacking the execution.
2417700	2418660	And the way to do it,
2418660	2420580	I found the easiest way to do it at least,
2420580	2422540	is using these Java options.
2422540	2424180	Now, if you're familiar with the JVM,
2424180	2426100	you know that you can actually specify
2426100	2428300	or control the behavior of that JVM.
2428300	2430540	Like, I don't know which garbage collector to use,
2430540	2433620	how should strings be compressed,
2433620	2435720	the amount of heap memory, et cetera.
2435920	2439120	Well, it turns out if you dig into the hundreds
2439120	2440380	and hundreds of Java options,
2440380	2443180	there is one actually that allow code execution.
2443280	2445660	Also, yeah, this is an example of a JVM option.
2445840	2447660	So there is one that allows code execution
2447660	2450100	and it's called an un-out-of-memory error.
2450100	2452760	And you just specify a command and it will execute it.
2452760	2454940	The catch is that it only executes it
2454940	2456960	if there's an un-out-of-memory error.
2456960	2458800	So like its name said.
2459120	2464680	So if the executor or process fails to allocate memory,
2464680	2466500	heap memory, then it will throw this error
2466500	2467660	and then it will execute code.
2467660	2470520	So how to make the executor trigger this error?
2470520	2474260	Well, simple way is simply to add another JVM option, xmx,
2474260	2476500	which will set the maximum amount of memory
2476500	2479300	to a ridiculous amount, like one megabyte or two megabytes.
2479300	2482080	And once you combine these two properties,
2482080	2483920	then you have your code execution.
2484580	2486720	And so ideally, we'd like to do something like this.
2486720	2488480	Obviously, we cannot do it inside the code
2488480	2491480	because while the driver will attempt to authenticate,
2491480	2492340	which is no good.
2492340	2494940	And anyway, you cannot set the xmx inside the driver.
2494940	2498180	So we really need to forge by hand, basically,
2498180	2500900	that serialized object, embed these options,
2500900	2502300	have a properly serialized object
2502300	2505940	and send it to the cluster manager.
2505940	2509820	And again, easy way to do it using Sparky.
2511780	2513300	So I just point...
2513300	2514660	So this is just a quick example
2514660	2516420	to show you how to exploit this phone.
2517220	2520420	So this is the reversal that we want to execute, right?
2520420	2523100	So I just point to Sparky, I want to execute this code,
2523100	2526060	but as you can see, it requires,
2526060	2527660	the cluster requires authentication.
2529820	2532580	But if I add dash B, then it will take care,
2532580	2533920	it will take advantage of this phone
2533920	2535540	and you can see it bypassed authentication,
2535540	2537320	which has executed code directly.
2537320	2541020	And it's gonna execute it on as many workers as requested.
2541700	2543620	So yeah, we just bypassed authentication,
2543620	2545720	which has executed code on workers.
2546740	2547840	So great.
2548920	2550420	So yeah, so now that we understood
2550420	2554100	like the hacker intuitive way of finding this phone,
2554100	2555480	let's go through the code,
2555480	2556980	because I think it's interesting.
2557040	2558960	If you see, like, I made the distinction
2558960	2561300	between the 21 bytes, the, you know,
2561960	2563960	payload that was prefixed with 21 bytes of data
2563960	2565520	or prefixed with 13 bytes of data.
2565520	2569000	It turns out there's actually a difference in the dispatcher.
2569040	2570680	So the register application is handled
2570680	2572960	by the process one-way message.
2573020	2574360	And the process one-way message called
2574360	2576580	the receive method of the RPC handler, right?
2576580	2578680	So it delegates to the receive,
2578680	2580380	and you can see there at no point
2580380	2582540	there is any authentication going on here.
2582540	2584900	The delegate receive will actually perform the work.
2584900	2588040	If you compare that with the other receive method,
2588040	2590280	which actually, you know, asks for authentication,
2590280	2593680	you can see that it's quite beefy and different.
2593680	2595860	So the correction, basically,
2595860	2598900	the patch was to protect that same receive method
2598900	2601280	and another one used in streaming,
2601280	2602740	which was affected also,
2602740	2607100	by the same type, the same types of checks.
2607220	2610520	And this was labeled CB20209480.
2610520	2612780	I disclosed them on 24th of December,
2612780	2616100	if I'm not mistaken, which was very horrible for me.
2616100	2619640	But anyway, and yeah, it was fixed by the Apache Spark team.
2619640	2621780	So thank you very much for the work that you guys have done.
2621780	2622720	It's amazing.
2623380	2628180	Yeah, so in summary, what I wanna say is Spark is awesome.
2628280	2630440	I'm gonna say too bad security is not taken seriously,
2630440	2631720	and I'm not talking about the folks
2631720	2633180	who manage security there.
2633480	2635020	I'm just referring to the fact
2635020	2637300	that security is off by default.
2637300	2638620	I don't think that should be something
2638620	2640020	that we should have in 2020,
2640020	2642160	especially for something that's a framework
2642160	2644660	that does, you know, data computation.
2644940	2646740	And we all know how data is valuable.
2646740	2650960	So I say it in reference to that.
2651640	2653660	Anyway, I hope that will change.
2653860	2656100	And finally, yeah, we only covered Spark standalone
2656100	2658180	where the cluster manager is actually
2659400	2660820	the one doing the work.
2660820	2662120	But you can have other setups
2662120	2664320	where it's actually Yarn that's doing the work,
2664320	2668040	or I think even a Kubernetes API server in Spark 3.
2668040	2670420	So yeah, there's a lot of ground to cover.
2670420	2672680	So if you wanna dig into it, go ahead.
2673560	2674840	Yeah, the code to Sparky,
2674840	2677480	if you wanna check out these, you know, Spark APIs,
2677480	2678900	the serialized object that's being forced
2678900	2681860	in order to bypass the vuln, all that stuff, it's there.
2681860	2686320	I hope you got enough of this talk to actually dig into it
2686320	2687520	because that's really the only way
2687520	2689500	to figure out how stuff works.
2689520	2692460	So yeah, your contribution are most welcome.
2692460	2693500	And if you wanna hit me up on Twitter
2693500	2695960	to talk about it more, please do not hesitate.
2695960	2697320	And looking forward to talk to you guys
2697320	2699500	and to talk to you folks in the QA.
2699500	2700520	Thank you very much.
2700520	2701200	Bye-bye.