[00:14.640 --> 00:18.240] And welcome back, everybody, to DEF CON 28 Safe Mode. [00:18.360 --> 00:22.940] Continuing on with the Blue Team Village OpenSOC CTF walkthrough, [00:22.940 --> 00:24.920] we're going to be talking about Moloch today, [00:24.920 --> 00:28.880] and we have Bashar Shama here to give us a quick discussion [00:28.880 --> 00:30.520] and a walkthrough on the tool. [00:30.740 --> 00:32.960] Welcome, Bashar. [00:33.320 --> 00:35.380] Thank you. Hello, everyone. [00:35.380 --> 00:40.200] I am Bashar, and I'm going to go over how we're going to use Moloch tomorrow [00:40.200 --> 00:41.780] during the CTF. [00:41.780 --> 00:45.780] I was actually introduced to Moloch two years ago [00:45.780 --> 00:49.020] playing the same exact CTF that you're going to play tomorrow. [00:49.020 --> 00:52.620] I played it two years ago, and I really, really fell in love with Moloch. [00:52.620 --> 00:56.020] So since then, I started playing with it, [00:56.020 --> 00:58.060] tinkering with it, just use it as much as I can, [00:58.060 --> 01:00.220] and I keep using it day to day. [01:01.740 --> 01:07.520] The goal today is to really prepare you on how to use Moloch on the CTF tomorrow. [01:07.760 --> 01:11.260] So we will have time for questions at the end. [01:11.260 --> 01:17.220] Please post any questions that you have to the TechWorkshops Track 1 channel in the Discord. [01:17.620 --> 01:23.600] The moderators will monitor your questions, and we're going to ask them at the end, [01:23.600 --> 01:24.640] towards the end of the session. [01:24.640 --> 01:27.240] So keep your questions thrown away. [01:27.240 --> 01:29.880] We'll have some time at the end to go over some questions. [01:31.000 --> 01:33.960] So let's go ahead and get started. [01:34.960 --> 01:39.240] So a very brief intro on what Moloch is. [01:39.240 --> 01:41.860] It's pretty much a free open source tool. [01:42.320 --> 01:50.080] It's really a network analysis tool that you can use to analyze a large volume of packet data, or PCAPs. [01:51.040 --> 01:55.300] The simplest way to think about it is, if you've ever used Wireshark, [01:55.300 --> 02:00.520] it's like a Wireshark frontend with a huge database backend, [02:00.520 --> 02:04.940] so you can search tons and tons of PCAP data, gigs of data, hundreds of gigs of data, [02:04.940 --> 02:08.240] of PCAPs very easily, very quickly. [02:09.280 --> 02:16.240] That's a very, very, very 10,000, over 2,000 foot overview of Moloch. [02:16.480 --> 02:21.520] For our purpose here, and for the CTF tomorrow, what I have done is, actually, [02:21.520 --> 02:28.620] I have Moloch set up, and I got some, about four gigs of data as a sample [02:28.620 --> 02:32.540] from the network forensics training of March 2015. [02:32.620 --> 02:37.240] I downloaded these PCAPs, and I pretty much loaded them into Moloch [02:37.240 --> 02:42.640] to kind of show you how we can use this tool to do our investigation [02:42.640 --> 02:46.100] and answer the questions during the CTF tomorrow. [02:46.860 --> 02:53.340] So, our scenario for today is, you're part of the security team, [02:53.340 --> 03:00.100] and you receive a call saying, hey, around 1 p.m. today, or not today, [03:00.100 --> 03:04.980] 1 p.m. on March 12, 2015, our main company site has been de-fixed. [03:04.980 --> 03:08.240] Somebody has changed the way our website looks like. [03:08.300 --> 03:15.000] All we know is, we have this image of a file showing as a frog on our main site, [03:15.000 --> 03:18.460] and we don't know how this happened. Can you please help us? [03:18.680 --> 03:22.360] And all you know is, you have access to packet capture. [03:22.480 --> 03:25.300] You have packet capture, and you have access to Moloch. [03:25.300 --> 03:27.920] So, let's walk right through it. [03:28.360 --> 03:32.900] So, if you have never seen Moloch before, this is what it looks like. [03:32.900 --> 03:37.360] This is when you land or you log in. [03:37.360 --> 03:39.600] You will land pretty much in the sessions tab, [03:39.600 --> 03:42.900] and that's where you're going to probably spend most of your time tomorrow, [03:42.900 --> 03:44.040] in the sessions tab. [03:45.260 --> 03:50.380] Usually, I would like to go and just drop down here on the time, on the date, [03:50.380 --> 03:54.260] and I would do all time, all day, just to have an understanding [03:54.260 --> 03:56.320] of how big the incident is. [03:56.980 --> 04:00.600] For the purpose of tomorrow, again, it's going to be a very specific time period. [04:00.600 --> 04:06.560] But this is where I would start, and let's go ahead and deep dive into the investigation [04:06.560 --> 04:08.260] and how can we use Moloch. [04:08.260 --> 04:13.840] So, the first piece of information we know is the date of the incident. [04:14.040 --> 04:18.520] So, we know it's on 2015. [04:18.520 --> 04:22.160] So, as you can see, Moloch can give us this ability to just click through [04:23.260 --> 04:27.180] and decide on the dates that we want to investigate. [04:27.240 --> 04:28.960] I'm going to do 12. [04:28.960 --> 04:32.100] So, we have our date here. That's going to be our starting date. [04:32.320 --> 04:36.940] And then I'm going to just copy it, paste it here, and I'm going to change this to 13, [04:37.440 --> 04:42.100] which will show us here now, oh, you're looking at one day time range. [04:42.320 --> 04:45.540] So, I have my time range set up. I'm going to go search. [04:46.560 --> 04:49.680] And now I'm going to narrow it down to that 24-hour period [04:50.620 --> 04:53.440] to kind of figure out what happened during that day. [04:53.440 --> 04:59.440] Now, the other piece of information that we were given was it was our main company's website. [04:59.780 --> 05:06.140] So, how can I search for that? Probably I want to look for our host name. [05:06.140 --> 05:11.600] So, what I can do is I can type host, and then Moloch would automatically parse [05:11.600 --> 05:17.800] the different fields based on the protocols that exist in the packet. [05:17.800 --> 05:24.380] So, if it was HTTP traffic, then Moloch will say, okay, well, this is an HTTP host name. [05:24.380 --> 05:27.500] If it's an email, then it's an email host name and so on. [05:27.880 --> 05:30.960] In my case, because I want to see everything, how can I post it? [05:30.960 --> 05:35.700] I can just say host. And to specify what I'm looking for in Moloch, [05:35.700 --> 05:38.480] I would just say equal equal. [05:39.400 --> 05:47.060] So, equal equal means show me everything that matches exactly our host name. [05:47.680 --> 05:53.980] On ad.ase. Okay? That's the first bit of it. And I can run that. [05:54.200 --> 05:59.800] I also know, since it's our website, I want to say, well, it's a website, [05:59.800 --> 06:01.720] it's going to only run on two ports, for example. [06:01.720 --> 06:07.280] So, to kind of add more queries into Moloch, what you need to do is, [06:07.280 --> 06:11.580] you just do two amp signs as an end. [06:11.740 --> 06:15.540] And I'm going to open parentheses and add the ports that we need. [06:15.540 --> 06:20.600] So, port equals 80, just like we did before. [06:22.120 --> 06:30.200] And port equals 443, which we know they both are HTTP ports. [06:30.200 --> 06:34.020] Now, before we go any further, you know, under ports I can say, well, [06:34.020 --> 06:39.320] I can specify I want it to be a destination port, I can have it be a source port, or so on. [06:39.540 --> 06:43.860] For, just to keep it general, and we can see everything here, I'm just going to say, okay, [06:43.860 --> 06:48.320] these ports, this host name, and let's see what we find. [06:48.560 --> 06:51.000] And I messed up something, of course. [06:51.180 --> 06:55.140] Oh, I'm saying end here, and that should be or. [06:55.140 --> 06:58.300] So, it's either port 80 or port 443. [06:59.440 --> 07:06.920] So, in that case, I just do two pipelines, and run that. [07:07.940 --> 07:09.460] Not pipelines, but pipes. [07:09.460 --> 07:14.720] And now I'm saying, okay, show me everything for this host on port 80 and port 443. [07:14.720 --> 07:17.140] And now I can see the traffic. [07:18.300 --> 07:22.140] So, let's go and dive deeper. [07:22.960 --> 07:31.840] On this traffic, I can see all kind of requests, and let's just open something very randomly. [07:31.940 --> 07:36.500] Let's open up this request, and I see it's a get request. [07:36.500 --> 07:39.380] And before we go down, let's spend some time here. [07:39.380 --> 07:44.260] So, idcmalloc will parse all the fields in the packet. [07:44.260 --> 07:50.280] So, I can click on any of these, and I can say, okay, well, what protocol did it come from? [07:50.280 --> 07:52.220] Which IP, which ports? [07:53.260 --> 08:00.360] If it's HTTP packet, then it will also parse the method, the status code, all of these things. [08:00.680 --> 08:02.580] It will also parse the user agent. [08:02.580 --> 08:11.980] Let's say I'm interested in knowing all the user agents that happened in that specific day that visited our website. [08:12.040 --> 08:18.640] I can easily click on user agents, and I can say, okay, export unique user agents with counts. [08:18.640 --> 08:29.800] Once I click that, I will see a new page showing me the unique user agents, along with how many times we've seen this in that specific day, and the other one. [08:29.800 --> 08:34.780] So, we only see two different user agents, nothing abnormal, nothing suspicious. [08:34.780 --> 08:37.580] So, nothing to worry about here. [08:38.300 --> 08:45.360] If we scroll a bit down, we can actually see the actual requests, the raw requests. [08:45.360 --> 08:50.600] So, I can see which hosts they're requesting, where the URI they're going after, and so on. [08:51.320 --> 08:56.800] Because I know we had an incident, and I know the website has been defaced, [08:56.800 --> 09:01.440] the attacker must have sent some kind of data to our website. [09:01.440 --> 09:04.540] So, most likely they will not be doing a GET request. [09:04.540 --> 09:07.280] When we see GET, it's just pulling data from our website. [09:07.280 --> 09:10.040] They're going to be posting or sending some kind of data. [09:10.040 --> 09:12.660] So, this excludes this. [09:12.860 --> 09:18.160] So, I'm going to click GET, and we don't want GET anymore, so I'm just going to say net GET. [09:18.420 --> 09:20.940] That automatically adds that to our query. [09:22.580 --> 09:24.820] And let's hit search and see what we find. [09:24.820 --> 09:28.280] So, now we went down to only 12 entries. [09:28.280 --> 09:34.500] Okay, that's much easier to kind of go through and investigate. [09:35.080 --> 09:42.360] Again, we're not sure what happened, but one thing I can sort for is by time, [09:42.360 --> 09:47.460] to kind of understand the timeline of these different requests and the events. [09:48.220 --> 09:51.000] And this just opens on the random. [09:53.680 --> 09:55.800] So, again, same kind of thing. [09:56.040 --> 09:59.660] So, now it's a POST request. [10:02.920 --> 10:05.400] And it's just a request in index.php. [10:10.470 --> 10:12.650] And I see a test on sleep. [10:12.650 --> 10:14.850] I'm not sure what all this stuff really means. [10:14.850 --> 10:18.770] And I also see this jumper stuff. What is this? [10:19.070 --> 10:23.250] So, if we look at the header, which I've already displayed for you, [10:23.250 --> 10:25.510] it turns out it's actually encoded with gzip. [10:25.510 --> 10:30.710] Gzip is just a method of compression that web browsers use to compress the data [10:30.710 --> 10:33.170] to transfer the least amount of traffic. [10:33.370 --> 10:37.550] So, the nice thing with Molek, what we can do is I can just click uncompress, [10:38.210 --> 10:41.750] and now it automatically will decode this packet for us. [10:41.750 --> 10:45.550] And now I can easily read it and see, oh, okay, this is what loaded, [10:45.550 --> 10:47.190] this is what the page is showing. [10:47.570 --> 10:52.110] And from what we see here, nothing of interest yet. [10:52.110 --> 10:54.130] Okay, that was useful. [10:54.130 --> 10:57.730] But let's keep going. Let's find out what happened. [10:58.270 --> 11:02.330] So, another request, and see this. [11:02.330 --> 11:05.410] But this is like, this looks like an IP address. [11:05.410 --> 11:07.470] But what is all this? I don't know. [11:07.930 --> 11:13.810] Well, what we can do is Molek has Cybershift built in. [11:13.950 --> 11:17.290] Well, Cybershift is a separate open source project, [11:17.290 --> 11:22.010] that you can actually just go and do Cybershift, [11:22.010 --> 11:25.510] and you can load it outside of... [11:26.510 --> 11:28.330] Cybershift, there we go. [11:31.600 --> 11:35.540] And you can just load it outside of Molek, and you can do your decoding. [11:35.540 --> 11:36.380] But let's get back here. [11:36.380 --> 11:39.480] The beauty of this is when we do it on Molek, [11:39.480 --> 11:43.860] it will automatically take that packet data and put it in for us. [11:43.880 --> 11:45.640] So, what is Cybershift? [11:45.640 --> 11:47.880] It's pretty much a tool, a web GUI, [11:47.880 --> 11:52.780] that you can use to do different decoding of different languages, [11:53.440 --> 11:55.360] decoding methods, and so on. [11:55.360 --> 12:00.600] So, in this case, it's taking the hex code and just decoding it. [12:00.800 --> 12:05.560] And while I'm looking at this, I can see these percentage signs. [12:05.960 --> 12:09.400] Percentage sign means it's a URL kind of thing. [12:09.400 --> 12:12.860] So, I can easily just drag the URL decode. [12:12.860 --> 12:16.700] And now I can see the decoded message here. [12:16.920 --> 12:21.480] Now I can see, oh, it's trying to ping this IP address, [12:21.480 --> 12:25.000] which is the same IP that's trying to visit. [12:25.000 --> 12:27.200] Okay, this is interesting. [12:27.200 --> 12:30.140] I'm not sure if they were actually able to ping, [12:30.140 --> 12:35.120] but it looks like this IP is trying to do something here. [12:37.380 --> 12:40.540] Let's keep going, see what else can we find. [12:42.860 --> 12:46.620] Next packet, same thing. [12:46.620 --> 12:50.060] Tests, okay, nothing here. [12:50.480 --> 12:54.180] Next packet, so on. [12:55.640 --> 12:57.780] Oh, there's an NC. [12:57.840 --> 12:59.460] Okay, what is this? Hold on a second. [12:59.460 --> 13:02.220] Let's pull it up again in CyberChef. [13:02.220 --> 13:05.860] Do the same method, decode the URL. [13:05.940 --> 13:10.960] And now we see an NC IP address and port. [13:10.960 --> 13:15.040] So, NC stands for Netcat, which is the utility attackers can use [13:15.040 --> 13:19.040] to have a server connect back to them and get a show in that box. [13:19.040 --> 13:24.920] So, what this is saying is, okay, connect back to my IP over this port. [13:25.600 --> 13:28.980] So, what this means, if this actually succeeded, [13:28.980 --> 13:34.520] that means our server connected back on this IP address. [13:34.800 --> 13:37.820] Well, let's see if this actually happens. [13:37.820 --> 13:42.280] I'm going to take that port, I'm going to clean all this up, [13:42.700 --> 13:46.600] and then I'm going to add the IP address. [13:47.780 --> 13:49.500] It's not going to be the source. [13:52.140 --> 13:57.400] And port equals the port that we're looking for. [13:57.480 --> 13:58.520] Let's search for it. [13:58.520 --> 14:00.420] And we have traffic. Uh-oh. [14:00.880 --> 14:02.420] This is not good. [14:02.420 --> 14:06.020] Looks like our box, which is the source here, [14:06.020 --> 14:08.800] connected back to the server over this port. [14:09.040 --> 14:14.240] We can see how many packets and the amount of data that was sent back and forth. [14:14.600 --> 14:18.900] So, immediately, I would say, okay, well, this is the first connection [14:18.900 --> 14:22.460] that has the highest number of packets, that might have something interesting in it, [14:22.460 --> 14:24.960] so let me open it up. [14:26.540 --> 14:33.280] Scroll down, and we can actually see the whole conversation back and forth now. [14:33.280 --> 14:37.920] Well, it looks like the attacker ran a command, which is id, [14:37.920 --> 14:40.620] which is equal to whoami on Windows, [14:40.620 --> 14:45.400] to figure out who the attacker is running as on this box, [14:45.400 --> 14:47.320] and they're running as apache. [14:47.580 --> 14:50.300] Then they try to figure out which folder they're in, [14:50.300 --> 14:55.220] and they try to access file systems. [14:55.220 --> 14:56.820] Okay, this is not good. [14:57.140 --> 15:02.460] And then they did this cat-index.php, which looks like it's our website, [15:02.460 --> 15:07.740] so they're looking into what's inside our main file and so on. [15:07.900 --> 15:09.840] Okay, what is all this? [15:09.840 --> 15:12.420] Oh, and then I see another netcat command here, [15:13.440 --> 15:18.920] and saving the connection as a cm0.php file, [15:18.920 --> 15:22.220] and our box, locally. [15:22.500 --> 15:26.820] So I'm interested in knowing what this is, but let's keep going. [15:26.820 --> 15:29.020] Let's see what else they do in this box. [15:30.180 --> 15:32.760] And if we come here, they actually did it for us. [15:32.760 --> 15:35.140] They did a cat on this file that they created. [15:36.420 --> 15:40.220] Okay, when they did that, we can see, oh, it's a PHP backdoor. [15:40.480 --> 15:44.420] This is not good, so they used a PHP backdoor on our site. [15:45.060 --> 15:48.800] So how am I going to now figure out what did they do? [15:49.060 --> 15:51.840] I know because of how backdoors work, [15:51.840 --> 15:57.820] they have to visit this PHP file explicitly to kind of load the command they want to run. [15:57.820 --> 16:03.240] So what I can say to Molek is, well, show me all the URIs. [16:03.520 --> 16:06.060] So when we say a URI is... [16:07.880 --> 16:12.380] pretty much everything after the domain will be a URI, [16:12.380 --> 16:14.300] so anything from here on, that's a URI. [16:14.300 --> 16:18.480] So I want to say, show me everything that has this in it. [16:18.600 --> 16:20.980] And since I can't just do, like, all, [16:20.980 --> 16:25.120] what I do in Molek is I just use wildcards at the beginning, [16:25.120 --> 16:29.360] so I don't care what's before this, and the end, meaning I don't care what's after it, [16:29.360 --> 16:33.320] just show me anything that has this string in the URI. [16:33.580 --> 16:36.620] And when I search it, [16:37.680 --> 16:42.060] now, here, we can see all these URLs or URIs [16:42.060 --> 16:44.820] with all these different links in them. [16:44.820 --> 16:46.400] What are they trying to do? [16:46.600 --> 16:51.240] Well, how can I know, like, how can I get all this in one nice place? [16:51.240 --> 16:53.540] I can just click on info, [16:53.540 --> 16:57.240] and I want to say export unique URI with two counts. [16:58.380 --> 17:00.320] Now, Molek will take this for us and tell us, [17:00.320 --> 17:03.980] okay, well, this command was run five times, [17:03.980 --> 17:06.560] this command was run three times, and so on. [17:06.560 --> 17:11.000] So now I kind of know which commands they ran, [17:11.000 --> 17:13.320] because I can see it's command equal cat, [17:13.320 --> 17:16.900] command equal cat, or else, or whatever they're trying to do. [17:17.180 --> 17:20.280] And I kind of see here there's a JPEG file as well. [17:20.280 --> 17:26.080] Okay, so this is how they probably got that JPEG file [17:26.080 --> 17:29.460] or the image of the fraud into our website. [17:29.820 --> 17:35.920] So let's find out where did they, like, how did they do this? [17:37.240 --> 17:43.000] Let's go back here, and let's do... [17:43.000 --> 17:47.700] Instead of this, we are looking for that specific JPEG file. [17:48.220 --> 17:53.020] So I'm going to do wildcard and show me anything that has JPEG in it. [17:53.960 --> 17:55.180] Load it up. [17:56.060 --> 17:58.180] And it's going to show me a lot of stuff. [17:59.620 --> 18:03.200] I'm going to say probably it's going to be the biggest file, [18:03.200 --> 18:06.640] so I'm going to sort by the size of the data. [18:08.060 --> 18:10.740] And this is the biggest file that we have. [18:14.970 --> 18:16.350] So it's loading. [18:17.010 --> 18:19.950] And here I can see, yeah, it's an image file. [18:23.290 --> 18:25.510] But I'm not sure what it looks like. [18:25.510 --> 18:30.090] So what I can do in Moloch is I can just click on show images and files, [18:30.090 --> 18:33.310] and it will actually render it for us right here in the browser. [18:34.390 --> 18:37.370] Now, let's say, you know, I'm not comfortable with doing this [18:37.370 --> 18:41.890] or I want to dig in deeper to it or I want to understand what actually happens [18:41.890 --> 18:45.030] using Wireshark because I've never used Moloch before. [18:45.030 --> 18:50.210] At any session, at any point, you can always just click on download PCAP, [18:50.210 --> 18:53.090] and it will kind of, like, save the PCAPs for you. [18:53.470 --> 18:59.090] So you can open up in Wireshark and, you know, do your analysis manually [18:59.090 --> 19:00.710] if you'd like to. [19:04.650 --> 19:08.950] Pretty much that's how they get the file, [19:08.950 --> 19:12.270] and that's how they deface our website using a backdoor. [19:12.270 --> 19:17.810] If we have time, I think we have another... I think that's pretty much it. [19:17.810 --> 19:23.870] Okay, so with that being said, I think that's the highlight of Moloch [19:23.870 --> 19:27.290] and how to use it. I want to keep some time for questions. [19:28.590 --> 19:30.950] Please let me know any questions you've got. [19:31.170 --> 19:35.070] I'm also going to be in Discord. I'll be on Twitter if you need anything else, [19:35.070 --> 19:38.450] but I'm going to be here waiting for any questions that you guys have. [19:39.610 --> 19:40.610] Cool. [19:41.290 --> 19:42.470] Thank you, Bashar. [19:42.630 --> 19:47.990] I went ahead and put a note in the text window under the workshop track one [19:47.990 --> 19:54.370] with a link over to Recon Episec's OpenSOC Moloch Discord channel. [19:54.830 --> 19:56.250] So definitely check that out. [19:56.250 --> 19:59.570] Obviously, hit Bashar up on Twitter and Discord, [19:59.570 --> 20:03.890] but we're trying to help everybody kind of connect with the right people here [20:03.890 --> 20:05.370] so you can get the help you need. [20:06.870 --> 20:07.590] Perfect. [20:08.830 --> 20:10.990] Thanks a lot, Bashar. I appreciate it. [20:11.130 --> 20:13.570] Thank you. Thank you. I appreciate your time. Thank you. [20:13.590 --> 20:14.210] Take care.