WEBVTT 00:31.450 --> 00:32.870 Hey there! 00:32.870 --> 00:33.990 Welcome everyone. 00:34.070 --> 00:45.610 We are kicking off the Q&A session here with Ayub, who spoke about Only Takes a Spark Popping a Shell on 1000 Nodes, which was pretty awesome, pretty cool. 00:45.610 --> 00:49.070 A good way of thinking about how to scale up everything. 00:49.510 --> 00:53.450 And, you know, not just hacking one system, but how can you use that to really spread out. 00:53.890 --> 00:57.950 So, really, I heard this is also your first time speaking at DEF CON. 00:57.950 --> 00:59.030 Is that correct? 01:00.890 --> 01:02.050 Exactly, yeah. 01:02.490 --> 01:03.270 So, we have... 01:03.270 --> 01:06.030 I was actually going to give it... yeah, go ahead. 01:06.090 --> 01:11.410 No, yeah, so I was just going to say we have a tradition here at DEF CON, which we call Shoot the Noob. 01:11.430 --> 01:16.910 And so, really, it's just the tradition of, you know, hey, taking a shot on stage for, you know, your first thing. 01:16.930 --> 01:18.310 It's not always liquor, right? 01:18.310 --> 01:19.790 So, people mistake that. 01:19.790 --> 01:22.730 But, really, just want to do that with you now. 01:22.730 --> 01:25.950 So, I want to welcome you in the traditional DEF CON fashion. 01:26.170 --> 01:27.590 So, cheers! 01:28.210 --> 01:28.970 Oh, boy. 01:28.970 --> 01:29.930 Okay, okay, okay. 01:29.930 --> 01:31.130 Oh, you got to pour it first. 01:31.130 --> 01:31.770 Wait for it. 01:32.490 --> 01:33.990 I got to pour it first. 01:33.990 --> 01:34.810 You had to prep. 01:34.810 --> 01:35.290 There you go. 01:35.290 --> 01:35.870 Okay. 01:36.050 --> 01:37.110 There we go. 01:37.790 --> 01:38.410 Cheers, guys. 01:38.410 --> 01:38.930 Cheers. 01:43.470 --> 01:44.190 Awesome. 01:45.430 --> 01:50.090 So, now that you've been officially indoctrinated to the DEF CON and DEF CON safe mode. 01:51.150 --> 01:54.870 Yeah, so, one of the first things, questions I'd ask, right? 01:54.870 --> 01:56.970 So, right, you recorded the talk a little bit ago. 01:56.970 --> 01:58.810 What have you been working on since? 01:58.810 --> 02:00.990 Anything new that you've kind of come up with? 02:00.990 --> 02:02.510 Or, you know, anything you found? 02:02.510 --> 02:05.450 Anything, you know, that's changed since the talk was given? 02:07.350 --> 02:09.490 It has nothing to do with Spark, actually. 02:09.490 --> 02:12.450 I was working on Spark, like, a few months ago now. 02:12.690 --> 02:16.050 But since then, I completely switched subjects and topics. 02:16.050 --> 02:18.630 I've been working on AWS a little bit. 02:18.970 --> 02:24.150 I did, actually, a tool that reflectively loads DLLs and executables in memory. 02:24.150 --> 02:26.010 But I wrote it using Golang. 02:26.010 --> 02:29.250 So, I completely abandoned that area of research. 02:29.250 --> 02:31.650 And I completely switched over to something completely different. 02:31.650 --> 02:33.990 So , we'll see what comes up next. 02:34.130 --> 02:35.230 Good for you. 02:35.230 --> 02:36.630 That's a pretty good way to go. 02:36.750 --> 02:39.270 You get some recognition for one thing. 02:39.270 --> 02:42.530 And then you move off to something totally cool, totally new. 02:42.650 --> 02:43.530 Excellent. 02:43.930 --> 02:47.190 Yeah, definitely like this idea of discovering new stuff. 02:47.710 --> 02:48.570 Excellent. 02:48.690 --> 02:51.610 So, is that a thing you find yourself doing often in your own research? 02:51.610 --> 02:57.530 Is that you want to get a nice breadth of experience in a lot of different things? 02:57.530 --> 03:03.990 Or how deep do you go before you feel like you're comfortable with what you've learned? 03:06.150 --> 03:07.570 I think I can... 03:07.570 --> 03:14.130 Basically, my idea is go as deep as you start to understand how it works, basically. 03:14.310 --> 03:16.630 Before, I was looking into mainframes. 03:16.630 --> 03:29.210 And that took me, I think, two years of going through these 900-page documents published by IBM and these obscure systems and obscure forums actually talking about it. 03:29.210 --> 03:32.270 So, I did the same thing with mainframes. 03:32.270 --> 03:33.610 And then I let it go. 03:34.110 --> 03:37.190 And I switched over to other areas. 03:37.190 --> 03:39.950 And during that work, I think Spark came up. 03:39.950 --> 03:42.090 And I was like, what the hell is this thing? 03:42.190 --> 03:43.050 And I dug into it. 03:43.050 --> 03:45.950 And I found there was not much research was being done on it. 03:45.950 --> 03:47.970 And I thought, well, you know what would be fun? 03:48.870 --> 03:49.870 So, I actually dug into it. 03:49.870 --> 03:51.770 And that's how it all started, really. 03:51.890 --> 03:56.550 It was just going after the next shiny thing, trying to understand how it works. 03:56.550 --> 04:00.990 And once you understand how it works, you try to bend its rules to do whatever you want with it. 04:01.210 --> 04:06.450 And hopefully, write a tool about it, give a talk about it, and then move on to the next stuff. 04:07.050 --> 04:08.010 How deep? 04:08.210 --> 04:09.970 Just enough to understand it, really. 04:11.210 --> 04:12.030 That's the goal. 04:12.030 --> 04:17.850 So, what was it about Spark, really, other than the fact that it was new and there wasn't a lot of research? 04:17.850 --> 04:21.950 I mean, what was it that really dragged you into this one? 04:21.950 --> 04:26.970 Because it's an interesting system, as you showed us in your talk. 04:27.250 --> 04:34.770 So, what was the thing that made you decide, okay, fine, this is where I'm going to spend the next year of my life? 04:34.770 --> 04:36.190 Yeah, and why not continue? 04:36.190 --> 04:38.850 All right. 04:38.850 --> 04:43.870 So, I was working on the offensive side, and then I switched to the blue side. 04:44.310 --> 04:47.010 And I was helping a company secure their systems, etc. 04:47.010 --> 04:51.330 And they were all on these new shiny platforms, if you will. 04:51.330 --> 04:55.210 Everything was on AWS, multi-region, CICD. 04:55.210 --> 04:57.390 I mean, they didn't click a single button. 04:57.390 --> 05:00.750 They pushed everything to code, and everything was deployed and scaled and stuff. 05:00.750 --> 05:02.670 Very, very sexy stuff. 05:02.670 --> 05:07.130 So, I was trying to help them secure that stuff. 05:07.370 --> 05:16.070 And after six or seven months, we thought we did a pretty good job locking pretty much everything that was supposed to be locked. 05:16.070 --> 05:19.450 There was no Windows, by the way, so that's why we came to this state. 05:19.450 --> 05:25.370 But anyway, and then I was talking with this data scientist, just trying to understand what they do. 05:25.770 --> 05:28.770 And he mentioned something about Spark. 05:28.770 --> 05:30.450 And I was like, what is that? 05:30.450 --> 05:34.910 And he told me, oh, it's something that we use to make calculations and parse data. 05:35.430 --> 05:37.150 I mean, what do you mean, parse data? 05:37.650 --> 05:38.950 How many machines do you have? 05:38.950 --> 05:43.270 And he's like, oh, actually, it's like, I have 200 machines. 05:43.270 --> 05:44.510 It's like, what do you mean, you have? 05:44.510 --> 05:51.050 It's like, oh, I spawned a cluster of 200 machines, and that guy over there spawned a cluster of 500 machines, etc., etc. 05:51.050 --> 05:55.350 So, basically, the company had like thousands of machines running sporadically, if you will. 05:55.350 --> 05:59.210 But still, and I was like, okay, well, how do you walk me through it, please? 05:59.210 --> 06:04.470 And he showed me how he launched the job and how he did some calculations, and I was like, oh, my God. 06:05.810 --> 06:10.230 And that's when, where the all interest Spark came in, basically. 06:11.030 --> 06:11.850 That's awesome. 06:11.850 --> 06:13.450 That's a great place to start. 06:13.450 --> 06:19.410 Yeah, you start seeing, oh, a couple of commands, and all of a sudden, there's this many machines doing something. 06:20.070 --> 06:21.730 I want to know that, too. 06:21.730 --> 06:29.810 He was writing Python code on a Jupyter notebook, where, you know, there was an authentication, obviously. 06:30.250 --> 06:36.610 And he was writing code, and it executed over multiple machines, and I was like, dude, this is amazing. 06:38.330 --> 06:42.670 And so I dug into it, and yeah, that's what it all sounded like. 06:42.670 --> 06:49.890 All right, so it's a pretty specific setup that you are talking about in your presentation. 06:49.890 --> 06:56.010 How common is the setup that you're speaking about? 06:56.010 --> 07:02.310 And how many other ways are there to configure this that might have more research opportunities? 07:02.610 --> 07:12.710 Well, the thing is, basically, a Spark cluster can be set up in very different ways. 07:13.590 --> 07:23.910 The cluster manager, so basically the process, the component that is responsible for linking applications to workers, that one can be replaceable. 07:24.070 --> 07:27.850 Not whatever you want, but you can put many different other components. 07:27.850 --> 07:37.730 In my talk, I only briefly demonstrated when it's Spark itself that is doing the orchestration. 07:37.730 --> 07:44.290 But you can have other setups where it's actually Yarn, a product from the new framework, that is doing all this orchestration. 07:44.910 --> 07:48.730 And I think in 3.0, you can even have Kubernetes doing the work. 07:48.730 --> 07:51.610 So you have all your pods coming up to do all the work. 07:51.610 --> 07:53.610 So it's much sexier. 07:53.610 --> 08:01.230 And the one that I showed, the one that bypasses authentication, that one only works on Spark standalone mode. 08:01.230 --> 08:04.010 So when it's actually Spark that's doing all the stuff. 08:04.010 --> 08:08.530 If you're having Yarn in front, it's a completely different story. 08:08.530 --> 08:12.710 It's a completely different protocol that's been going on. 08:12.750 --> 08:16.510 That's some Hadoop shit that's too much. 08:16.610 --> 08:18.260 That's a completely different beast. 08:18.610 --> 08:20.610 It listens on a different port, etc. 08:20.610 --> 08:23.310 The tool that I released, Sparky, handles it. 08:23.310 --> 08:26.770 But I didn't go much deeper into it. 08:26.770 --> 08:31.190 So there's definitely some area of research there to go into. 08:31.190 --> 08:36.530 And that's the default mode when you're using AWS Managed Service called EMR. 08:36.530 --> 08:42.690 So if you're using EMR and you spawn a cluster using their service, it will by default use this Yarn mode. 08:42.690 --> 08:48.150 So it will spin up Yarn and then the work will be done by Spark Worker. 08:49.030 --> 08:52.450 How much is its widespread ratio? 08:52.830 --> 08:58.890 From the studies that I saw, it was around 50-50, maybe 40-60, depending on which website you see. 09:00.690 --> 09:06.590 Basically, the more traditional websites, the companies that had Hadoop before, will stick to Yarn. 09:06.590 --> 09:16.110 Whereas the new ones that have the luxury of starting a cluster of data mining from scratch, will maybe opt more for a Spark standalone cluster. 09:17.330 --> 09:18.730 Okay, that makes sense. 09:18.730 --> 09:27.790 And 50-50 means that there's stuff out there, both more to study elsewhere and more to try and hit with the research that you've done. 09:27.790 --> 09:29.210 So, interesting. 09:30.070 --> 09:30.910 Yeah, I think so. 09:30.910 --> 09:33.650 I would say that... 09:33.650 --> 09:34.910 Here's the thing. 09:35.070 --> 09:43.770 The idemposite community is so much focused on Windows, I find, that there's so many other great stuff to talk about and to research. 09:44.330 --> 09:51.150 In Windows, if you want to make a breakthrough, you have to go through 20 years of past research to try to find something new. 09:51.150 --> 09:55.650 Whereas in this big, shiny, new technology, well, it's right there. 09:55.650 --> 09:58.890 Buffer overflows of 1999, it's right there. 09:58.890 --> 10:02.610 So , there's much opportunity there to be taken. 10:03.650 --> 10:04.930 That's awesome. 10:04.970 --> 10:06.710 Yeah, and I'll ask a question too. 10:06.710 --> 10:10.730 You mentioned how much more efficient Spark is over Hadoop. 10:11.210 --> 10:16.110 Would you say companies that are still heavily relying on Hadoop are behind the times? 10:16.110 --> 10:18.970 Should they be looking at Spark, or is there better security there? 10:18.970 --> 10:20.910 What's your take? 10:21.590 --> 10:22.670 Oh my god. 10:22.670 --> 10:27.990 Well, Hadoop has that thing, well, it actually handles Kerberos authentication. 10:28.510 --> 10:31.870 Now, whether that's a good thing or not, that's up for debate. 10:31.870 --> 10:34.370 But , no, not really. 10:34.450 --> 10:40.010 My mantra, really, I work in Blue Team, so this is Blue Team talking. 10:40.010 --> 10:44.310 Basically, production should be boring, to paraphrase what other people were saying. 10:44.310 --> 10:45.310 Production should be boring. 10:45.310 --> 10:53.310 So, if you have a Hadoop cluster of 3,000 machines that's doing the work, and it's fine, and everything is fine, then, by all means, continue. 10:53.310 --> 10:55.690 If you have a mainframe to do what you need to do. 10:56.190 --> 11:01.770 I've seen a post, actually, that emulates what Spark does, but only using shell commands. 11:02.990 --> 11:04.130 It works. 11:04.130 --> 11:05.330 It's much cheaper. 11:05.330 --> 11:06.810 It runs on a single machine. 11:07.190 --> 11:09.870 And you don't have all the partitioning and the shuffling. 11:09.870 --> 11:11.370 You don't have all the network latencies. 11:11.370 --> 11:20.050 You don't have all that crappy stuff that makes it very, very, very slow, in comparison to keeping everything in memory and just working on a big slice of an object. 11:20.050 --> 11:30.830 So, I mean, hey, whatever works for the company, with that particular set of talent and that particular set of circumstances and data, I mean, go for it. 11:31.030 --> 11:32.770 That's my opinion, yeah. 11:32.770 --> 11:33.890 Yeah, that's awesome. 11:33.890 --> 11:36.290 Yeah, and you mentioned, so you have that Blue Team mindset. 11:36.370 --> 11:40.830 I mean, so, I guess, how would you try to detect what you were doing during these attacks? 11:40.830 --> 11:48.350 Like, are there any particular tidbits you try to give to a company that's trying to secure their Spark instance, other than making sure it's patched? 11:49.210 --> 11:50.730 Very interesting question. 11:51.310 --> 11:52.510 Very interesting. 11:53.350 --> 11:54.530 Here's the thing. 11:56.210 --> 11:58.150 I never thought about that. 12:00.290 --> 12:03.630 That's how de-correlated the two worlds are for me. 12:03.630 --> 12:08.430 It's like, I do Blue Team in the morning and I do Red Team at night, and that's how, again, they are apart. 12:08.430 --> 12:13.170 So, I never thought about, hey, it would be good to release some Yara rules to detect this stuff. 12:13.170 --> 12:15.230 Never crossed my mind. 12:16.790 --> 12:18.890 Future research, there you go. 12:22.710 --> 12:28.390 No, but it's, to me, it's easier to patch. 12:28.990 --> 12:33.730 I know that some, well, a lot of people will not be able to patch. 12:34.610 --> 12:37.070 To me, the first thing you try is you try to patch. 12:37.070 --> 12:40.530 If you can't patch, then you want to detect, then you want to mitigate it somehow. 12:40.530 --> 12:50.590 So, you need to isolate it either through network firewall rules or do something like that to, you know, basically isolate it from the rest of the reducer exposure, if you will. 12:50.590 --> 12:55.770 Now, if you can't do that either, well, you got to detect it somehow. 12:55.770 --> 12:57.330 How would you detect it? 13:01.580 --> 13:07.220 If you have some advanced correlation in place, I think you can detect it. 13:07.220 --> 13:13.660 I'm talking about specifically the exploit of replaying that single theorized object that triggers command execution. 13:13.660 --> 13:15.460 I'm talking specifically about that. 13:15.460 --> 13:30.260 One way to detect it is that, unlike other Spark interactions, it's, like, in that interaction specifically, you only send one command instead of the whole charade of, hey, Spark, hello, which version are you running, master? 13:30.260 --> 13:32.180 Yes, I'm registering the application. 13:32.180 --> 13:33.340 Okay, here are the workers. 13:33.340 --> 13:36.920 You know, you have, like, 80 messages going around. 13:36.920 --> 13:39.200 But instead, in that exploit, you only have one message. 13:39.200 --> 13:55.460 So, if you can, if you have a tool that's advanced enough to make these kinds of correlations say that, oh, this IP address only or this session only initiated that single communication, that single packet, then maybe it's suspicious because it didn't follow up with all the stuff. 13:56.580 --> 13:58.140 That could give you a hint. 13:58.140 --> 14:04.260 You can also try the on out of memory errors that are on your machines. 14:04.260 --> 14:08.180 That may be a bit noisy, but that could get you going. 14:09.780 --> 14:17.660 But then again, a hacker could find another way to trigger the execution because I only gave out an example using the on out of memory error. 14:17.900 --> 14:26.600 Another way... yeah, these are the two main things to look for off the top of my head, basically. 14:27.160 --> 14:27.760 Sure. 14:27.760 --> 14:32.160 We got a follow-up question from PC Squared, it appears. 14:32.160 --> 14:33.300 Really well done talk. 14:33.300 --> 14:37.340 It looks like you did the magic on the Spark stand alone, like we were mentioning there. 14:37.440 --> 14:40.020 Did you get it working with Yarn 2? 14:40.320 --> 14:41.880 Not a big deal. 14:42.700 --> 14:46.400 Not a big deal, if not, but I imagine it's possible. 14:46.500 --> 14:49.960 Also, 100% production should be boring. 14:51.540 --> 14:52.460 Amazing. 14:53.720 --> 14:54.720 Oh, my God. 14:54.720 --> 14:55.360 Okay. 14:55.460 --> 14:56.980 So, here's the real story. 14:56.980 --> 15:00.900 So, I found this thing on Spark, and I was like, oh, my God, this is amazing. 15:00.900 --> 15:01.660 I feel so great. 15:01.660 --> 15:02.020 You know what? 15:02.020 --> 15:02.840 I'm going to follow up. 15:02.840 --> 15:04.520 I'm going to see how it works on Yarn. 15:04.520 --> 15:08.860 Then I pulled up Yarn, did the installation, made it work on Sparky, right? 15:08.860 --> 15:10.240 So, let's look up some runes. 15:10.240 --> 15:13.580 So, I turned up Wireshark, and I saw the traffic. 15:13.800 --> 15:17.760 It was on Wireshark, and I was like, oh, my God, I don't want to touch that. 15:18.760 --> 15:20.240 That's how foreign it was. 15:20.240 --> 15:21.660 That's how... 15:21.660 --> 15:24.800 Yarn is more difficult than Spark stand alone. 15:25.480 --> 15:27.460 It's really, really more difficult. 15:27.460 --> 15:29.180 And I looked at it, and I was like, fuck it. 15:29.180 --> 15:30.040 I don't want to look into it. 15:30.040 --> 15:35.060 That's really what went through my mind when I tried to do that on Yarn. 15:35.340 --> 15:40.880 Now, if you look for a similar vulnerability, I couldn't trigger it. 15:40.880 --> 15:45.100 Like, I was playing around with Yarn a little bit, but I couldn't trigger the same vulnerability. 15:45.460 --> 15:47.940 Does that mean that it's not vulnerable in some way? 15:47.940 --> 15:48.680 Of course not. 15:48.680 --> 15:54.820 But by that time, I was like, oh, Yarn is just too much for me. 15:55.340 --> 15:57.700 I'm not going to waste my time on it. 15:57.700 --> 15:59.200 I'm going to move on to other stuff. 15:59.760 --> 16:02.440 Maybe if you feel a connection with Yarn, you can dig into it. 16:02.460 --> 16:03.740 Please do so. 16:04.000 --> 16:06.440 Because like I said, there's nothing out there on Yarn. 16:06.440 --> 16:07.440 I didn't find anything. 16:07.440 --> 16:11.720 So, probably there are some stuff to be sought after in Yarn. 16:13.080 --> 16:15.120 Well, that makes a lot of sense. 16:15.120 --> 16:35.100 So, if somebody decides that they do want to pursue this, pursue the Yarn side, and would like to ask you more questions, are you available for either consultations or for answering questions that people come up with when they're doing this on their own? 16:36.220 --> 16:37.140 Yeah, of course. 16:37.140 --> 16:39.460 I mean, yeah, definitely. 16:39.620 --> 16:41.360 Like, my Twitter is open. 16:41.360 --> 16:48.380 Right at the end of this, we'll have you post any contact information you'd like for people to have access to in the TrackOne channel. 16:48.380 --> 16:52.400 And you can be there for people. 16:53.800 --> 16:54.940 Yeah, definitely. 16:54.940 --> 17:00.620 I mean, when you look at this stuff, like when you look at the Spark source code, it's beautifully written, by the way. 17:00.620 --> 17:02.520 But it's written in Scala. 17:02.520 --> 17:03.840 Some of it is written in Scala. 17:03.840 --> 17:05.640 And it can be very daunting. 17:05.640 --> 17:08.520 The syntax is weird. 17:08.740 --> 17:20.080 And when you look at information on the web that talks about Spark, I think one blog post or two maybe that are dedicated to the internals of Spark. 17:20.080 --> 17:27.640 Everybody talks about how shuffling works and how it partitions data and stuff, but nobody talks about how it works inside. 17:29.580 --> 17:34.840 So yeah, there's a lot of things to figure out and some simple stuff when you explain them. 17:34.840 --> 17:42.220 They look simple, but to get that information, it takes a little bit of time because nobody talks about those internals. 17:43.000 --> 17:45.540 So yeah, if you have any questions, please do not hesitate. 17:45.900 --> 17:49.840 Just hit me on a DM or just ping me or whatever. 17:52.520 --> 17:55.400 Yeah, it's one of the things I thought was interesting too. 17:55.400 --> 18:00.140 So through what you discovered, I was like, oh man, I wonder if he's going to report this to Apache. 18:00.360 --> 18:02.260 And then like, oh, look, he did. 18:02.560 --> 18:04.600 So I guess, how did that interaction go? 18:04.600 --> 18:06.240 How long did it take him to fix it? 18:06.240 --> 18:07.680 Was it a good interaction? 18:07.720 --> 18:10.920 Tips for other people having to reach out to companies like that? 18:12.060 --> 18:13.020 Okay. 18:14.960 --> 18:19.000 I sent them the phone, I think on 24th of December or something like that. 18:19.000 --> 18:20.660 Okay, very bad of me. 18:22.600 --> 18:29.660 Then I got a response from the Spark security team, and they said, yeah, okay, we'll look into it, etc. 18:29.660 --> 18:34.360 But whatever, and I didn't receive a response later. 18:34.740 --> 18:37.740 And then my talk at Troopers got canceled. 18:38.540 --> 18:42.680 And I think one week before I decided to, you know, because I was rehearsing a little bit. 18:42.680 --> 18:46.700 So I said, you know what, let me just write them again to see where they are on this. 18:46.900 --> 18:52.740 And I got a response from the same guy saying, oh, you know what, we looked it over, and we discarded it because it's not interesting. 18:53.220 --> 18:55.240 I was like, okay. 18:57.120 --> 18:58.380 Let me rephrase. 18:58.380 --> 19:01.260 Oh, and when I sent them, I sent them an email with a proof of concept. 19:01.260 --> 19:04.500 It's like a Python code that actually executes the code fully. 19:04.500 --> 19:06.620 So proof of concept, it's there. 19:06.620 --> 19:08.760 So it was really detailed email. 19:10.100 --> 19:13.860 And yeah, so the guy said that they discarded it and I didn't understand. 19:13.860 --> 19:17.360 So I wrote a second email saying, are you sure you want to do that? 19:17.360 --> 19:23.800 Because you're right in the documentation that when authentication is enabled between this and this component, etc. 19:24.360 --> 19:28.880 So you're kind of breaking this trust by allowing this vulnerability to go on, etc, etc, etc. 19:29.400 --> 19:31.520 And then I got a response saying, oh, my God, I'm so sorry. 19:31.520 --> 19:32.540 Yeah, we made a mistake. 19:32.540 --> 19:36.760 We thought you were talking about the RPC endpoint that we already got a report for. 19:36.760 --> 19:40.100 Now, indeed, this is a dangerous vulnerability, etc. 19:40.100 --> 19:41.280 I'm going to put everything. 19:41.420 --> 19:44.420 And they, you know, I mean, it happens. 19:44.420 --> 19:46.660 I do try it myself and sometimes I get it wrong. 19:46.660 --> 19:48.140 So this is normal. 19:48.760 --> 19:50.840 And then they went on to fix it. 19:51.000 --> 19:54.960 So, all in all, it took, I think, eight months. 19:54.980 --> 19:59.680 But since they acknowledged that it was a vulnerability, I think it only took like three months. 20:00.200 --> 20:00.620 Gotcha. 20:00.620 --> 20:02.140 So persistence was key. 20:02.940 --> 20:03.980 Persistence was key. 20:03.980 --> 20:08.720 Well, I mean, my goal was not to get it out there, to just publish it. 20:08.720 --> 20:11.000 And I didn't release the tool or I didn't even talk about it. 20:11.000 --> 20:17.100 I think I talked about it to two people who were very expert in Spark to validate that I was not saying bullshit. 20:17.560 --> 20:19.720 And I asked them to not disclose it. 20:20.240 --> 20:21.240 That's about it. 20:21.240 --> 20:23.680 Because my intent was not to just, you know, release that. 20:23.680 --> 20:25.460 I am pro full disclosure. 20:26.600 --> 20:29.460 But everybody should make their own choices and it depends on my mood. 20:29.460 --> 20:33.740 So for that one, I decided, you know what, let's just keep it on the low key anyway. 20:34.340 --> 20:44.180 And, yeah, and I think three weeks after disclosing it, they released a complete correction. 20:44.240 --> 20:48.500 Because it was not impacting only one function, it was impacting also two others. 20:48.500 --> 20:49.960 So they rewrote a class, etc. 20:49.960 --> 20:51.400 So we worked on the fix. 20:51.880 --> 20:57.640 Well, I worked on the origin of the fix and then they took it over because more competent than me. 20:57.640 --> 21:02.700 And then the thing that was really long is the release process. 21:02.700 --> 21:04.980 They had the fix three weeks after I reported it. 21:04.980 --> 21:14.200 But since like, let's say, beginning of April, they waited until July on the other version 2.4.6 to actually release the fix. 21:15.020 --> 21:18.960 So that's the part that was long and not actually fixing it, just releasing it. 21:19.220 --> 21:20.220 But, yeah. 21:20.420 --> 21:23.460 It's out there and we're all more secure for your efforts. 21:23.460 --> 21:27.300 So it's, you know, better than having no one looked in that area before. 21:28.680 --> 21:30.340 Yeah, yeah, definitely. 21:30.340 --> 21:31.900 And now we're talking about that. 21:31.900 --> 21:37.640 There was some interesting point during basically this whole Spark adventure. 21:37.640 --> 21:46.980 And I think I briefly touched upon it during the talk is that I was so, like, I spent like a couple of hours, even days, trying to understand how the RPC stuff works. 21:46.980 --> 21:52.200 And only to find out a couple of days later that somebody already published that vault. 21:52.920 --> 21:55.940 So, yeah, that was very interesting. 21:55.940 --> 21:56.880 I don't know if it happened. 21:56.880 --> 21:58.300 I think it happens to some people. 21:58.300 --> 22:09.680 But basically, you're just so stuck in, like, you find yourself drawn into that code base and trying to understand how it works when all you have to do is actually Google the right keyword to get the actual exploit. 22:11.000 --> 22:14.700 And when you do it, you feel a sense of frustration. 22:16.080 --> 22:18.540 But it's part of the game. 22:18.820 --> 22:32.020 And that's pretty good advice for all of us who are into finding vulnerabilities and figuring out how to report them is that enumeration step goes on the software that you're working on. 22:32.020 --> 22:35.920 And you have to read the state of the art of what people know. 22:35.920 --> 22:39.060 So that's a good thing to reinforce. 22:39.680 --> 22:40.440 Exactly. 22:43.220 --> 22:44.060 Awesome. 22:44.060 --> 22:48.920 So tell us a little bit more about the next thing that you're moving on to. 22:48.920 --> 22:52.640 It sounds like you already have this idea of what you're researching next. 22:54.920 --> 22:58.620 Actually, I wrote down a small plan. 22:58.620 --> 23:03.600 Not a plan, but bullet points of what I want to do later. 23:04.120 --> 23:10.580 I just released this tool called Reflect PE, which reflectively loads an assembly in a PE object. 23:10.580 --> 23:15.680 And shout out to name right, please. 23:19.860 --> 23:20.580 Rob? 23:22.740 --> 23:26.180 If needed, you can post it in the track one channel at the end. 23:26.180 --> 23:27.000 I will. 23:27.000 --> 23:27.920 I will post it. 23:27.920 --> 23:28.160 Okay. 23:28.160 --> 23:29.540 So his name is Rob Knopp. 23:29.540 --> 23:30.960 Well, he goes by the handle of Rob Knopp. 23:30.960 --> 23:31.460 Sorry. 23:31.980 --> 23:37.760 But yeah, I wrote a tool called Reflect PE, which reflectively loads a PE executable in memory. 23:38.720 --> 23:44.640 And I borrowed his basically tool to reflectively load also PE assemblies. 23:44.640 --> 23:46.000 So Rob Knopp, great tool. 23:47.100 --> 23:48.780 And why did I do that? 23:48.780 --> 23:50.700 So this is what I'm working on right now. 23:50.700 --> 24:05.680 And why did I do it is very, I find interesting, is that when you look at the same code to do the same thing, but in PowerShell and other languages, is that it's a huge script of like 2000 lines of code. 24:05.680 --> 24:07.100 No tests whatsoever. 24:07.100 --> 24:07.960 And you just launch it. 24:07.960 --> 24:09.740 And if it doesn't work, well, too bad. 24:09.740 --> 24:12.600 You can't debug a 2000 line PowerShell script. 24:12.720 --> 24:20.660 So I decided to write it from scratch using Golang and using like, you know, testing and good dev practices, etc. 24:20.820 --> 24:26.620 So that anybody could just look at the code and understand what's going on and where are the steps and where it didn't work, etc. 24:26.620 --> 24:28.360 So this is what I'm really working on. 24:28.360 --> 24:34.020 I'm trying to make this tool that will actually be easy to understand for people who just get into it. 24:34.020 --> 24:39.720 And just want to understand how reflectively loading something works on Windows. 24:40.460 --> 24:45.080 So that's the thing that taking most of my time right now. 24:45.080 --> 24:47.880 And my next big thing is Kubernetes. 24:47.880 --> 24:49.860 I really want to get into it. 24:50.280 --> 25:01.300 It is all of the talks that I saw, like 90% of them talk about how to abuse a Kubernetes set up with no authentication, no airbox, no nothing. 25:01.300 --> 25:14.900 I want to see what is there to exploit, you know, once you have all these security hardens in place, is it possible to bypass, I don't know, namespace isolation, is it possible to bypass team control, etc. 25:14.900 --> 25:27.080 I want to really dig into it because there's, again, like I said, I really look out for these niche topics where there is not much talk, where there is not much tools available. 25:27.080 --> 25:32.060 And try to focus on them and hopefully something gets out of this research. 25:33.600 --> 25:34.380 That's awesome. 25:34.380 --> 25:35.160 That's awesome. 25:35.160 --> 25:38.440 Yeah, so we're kind of wrapping up here at the end of the time. 25:38.440 --> 25:43.120 Is there anything else you want to share with the group or again, how to reach out to you? 25:43.120 --> 25:49.060 Again, we can post your contact information, but anywhere you want to point anyone to before we call it an end here? 25:51.840 --> 26:00.820 No, nothing except that, basically, thank you to the Spark team, first of all, because honestly, amazing. 26:00.820 --> 26:11.360 Like when you look at the code base, you understand how talented these people are and it's really a pleasure and a privilege to actually have interacted with some of you, especially some of you that I saw on stage talking about it. 26:11.360 --> 26:14.500 So I was pretty excited, though I tried to hide it a little bit. 26:14.620 --> 26:16.480 Yeah, so thank you all. 26:17.080 --> 26:28.840 But again, like for people who want to basically go into InfoSec and break into it, give talks, do research, etc. 26:29.000 --> 26:41.120 My only advice is basically look out for all these small niche topics that are basically abandoned by everybody and go deep until you feel you understand it and then just go crazy with it. 26:41.120 --> 26:43.740 And then most surely something will pop out. 26:44.160 --> 26:45.200 Awesome, awesome. 26:45.200 --> 26:46.420 That's great advice. 26:46.620 --> 26:47.860 I appreciate that. 26:47.940 --> 26:57.100 Yeah, and I was going to say just one more thing, basically, was getting to, you know, since you just actually did a live demo, right? 26:57.120 --> 27:03.960 One thing you're missing out on not having the live DEF CON crowd is a round of applause for having a demo work on stage. 27:03.960 --> 27:11.260 So since you actually did kind of roll that out, I just wanted to give you a nice round of applause and get that in the Discord chat, too. 27:12.320 --> 27:15.560 Thank you for getting that and getting it working and everything. 27:16.060 --> 27:17.260 Thank you very much. 27:17.920 --> 27:18.720 Cool. 27:18.920 --> 27:20.600 So I think that's it for us. 27:20.600 --> 27:22.900 So again, thank you again for joining us. 27:22.900 --> 27:27.000 And thanks for the talk and look forward to what you have coming up next. 27:27.000 --> 27:28.600 And hopefully see you next year at DEF CON. 27:28.980 --> 27:30.120 Thank you very much. 27:30.120 --> 27:31.240 Thank you. 27:31.240 --> 27:32.540 Bye-bye.