1 00:00:14,640 --> 00:00:18,240 And welcome back, everybody, to DEF CON 28 Safe Mode. 2 00:00:18,360 --> 00:00:30,520 Continuing on with the Blue Team Village OpenSOC CTF walkthrough, we're going to be talking about Moloch today, and we have Bashar Shama here to give us a quick discussion and a walkthrough on the tool. 3 00:00:30,740 --> 00:00:32,960 Welcome, Bashar. 4 00:00:33,320 --> 00:00:34,560 Thank you. 5 00:00:34,560 --> 00:00:35,380 Hello, everyone. 6 00:00:35,380 --> 00:00:41,780 I am Bashar, and I'm going to go over how we're going to use Moloch tomorrow during the CTF. 7 00:00:41,780 --> 00:00:49,020 I was actually introduced to Moloch two years ago playing the same exact CTF that you're going to play tomorrow. 8 00:00:49,020 --> 00:00:52,620 I played it two years ago, and I really, really fell in love with Moloch. 9 00:00:52,620 --> 00:01:00,220 So since then, I started playing with it, tinkering with it, just use it as much as I can, and I keep using it day to day. 10 00:01:01,740 --> 00:01:07,520 The goal today is to really prepare you on how to use Moloch on the CTF tomorrow. 11 00:01:07,760 --> 00:01:11,260 So we will have time for questions at the end. 12 00:01:11,260 --> 00:01:17,220 Please post any questions that you have to the TechWorkshops Track 1 channel in the Discord. 13 00:01:17,620 --> 00:01:24,640 The moderators will monitor your questions, and we're going to ask them at the end, towards the end of the session. 14 00:01:24,640 --> 00:01:27,240 So keep your questions thrown away. 15 00:01:27,240 --> 00:01:29,880 We'll have some time at the end to go over some questions. 16 00:01:31,000 --> 00:01:33,960 So let's go ahead and get started. 17 00:01:34,960 --> 00:01:39,240 So a very brief intro on what Moloch is. 18 00:01:39,240 --> 00:01:41,860 It's pretty much a free open source tool. 19 00:01:42,320 --> 00:01:50,080 It's really a network analysis tool that you can use to analyze a large volume of packet data, or PCAPs. 20 00:01:51,040 --> 00:02:08,240 The simplest way to think about it is, if you've ever used Wireshark, it's like a Wireshark frontend with a huge database backend, so you can search tons and tons of PCAP data, gigs of data, hundreds of gigs of data, of PCAPs very easily, very quickly. 21 00:02:09,280 --> 00:02:16,240 That's a very, very, very 10,000, over 2,000 foot overview of Moloch. 22 00:02:16,480 --> 00:02:32,540 For our purpose here, and for the CTF tomorrow, what I have done is, actually, I have Moloch set up, and I got some, about four gigs of data as a sample from the network forensics training of March 2015. 23 00:02:32,620 --> 00:02:46,100 I downloaded these PCAPs, and I pretty much loaded them into Moloch to kind of show you how we can use this tool to do our investigation and answer the questions during the CTF tomorrow. 24 00:02:46,860 --> 00:02:57,860 So, our scenario for today is, you're part of the security team, and you receive a call saying, hey, around 1 p.m. 25 00:02:57,860 --> 00:03:00,540 today, or not today, 1 p.m. 26 00:03:00,540 --> 00:03:04,980 on March 12, 2015, our main company site has been de-fixed. 27 00:03:04,980 --> 00:03:08,240 Somebody has changed the way our website looks like. 28 00:03:08,300 --> 00:03:17,040 All we know is, we have this image of a file showing as a frog on our main site, and we don't know how this happened. 29 00:03:17,040 --> 00:03:18,460 Can you please help us? 30 00:03:18,680 --> 00:03:22,360 And all you know is, you have access to packet capture. 31 00:03:22,480 --> 00:03:25,300 You have packet capture, and you have access to Moloch. 32 00:03:25,300 --> 00:03:27,920 So, let's walk right through it. 33 00:03:28,360 --> 00:03:32,900 So, if you have never seen Moloch before, this is what it looks like. 34 00:03:32,900 --> 00:03:37,360 This is when you land or you log in. 35 00:03:37,360 --> 00:03:44,040 You will land pretty much in the sessions tab, and that's where you're going to probably spend most of your time tomorrow, in the sessions tab. 36 00:03:45,260 --> 00:03:56,320 Usually, I would like to go and just drop down here on the time, on the date, and I would do all time, all day, just to have an understanding of how big the incident is. 37 00:03:56,980 --> 00:04:00,600 For the purpose of tomorrow, again, it's going to be a very specific time period. 38 00:04:00,600 --> 00:04:08,260 But this is where I would start, and let's go ahead and deep dive into the investigation and how can we use Moloch. 39 00:04:08,260 --> 00:04:13,840 So, the first piece of information we know is the date of the incident. 40 00:04:14,040 --> 00:04:18,520 So, we know it's on 2015. 41 00:04:18,520 --> 00:04:27,180 So, as you can see, Moloch can give us this ability to just click through and decide on the dates that we want to investigate. 42 00:04:27,240 --> 00:04:28,960 I'm going to do 12. 43 00:04:28,960 --> 00:04:30,300 So, we have our date here. 44 00:04:30,300 --> 00:04:32,100 That's going to be our starting date. 45 00:04:32,320 --> 00:04:42,100 And then I'm going to just copy it, paste it here, and I'm going to change this to 13, which will show us here now, oh, you're looking at one day time range. 46 00:04:42,320 --> 00:04:44,060 So, I have my time range set up. 47 00:04:44,060 --> 00:04:45,540 I'm going to go search. 48 00:04:46,560 --> 00:04:53,440 And now I'm going to narrow it down to that 24-hour period to kind of figure out what happened during that day. 49 00:04:53,440 --> 00:04:59,440 Now, the other piece of information that we were given was it was our main company's website. 50 00:04:59,780 --> 00:05:03,280 So, how can I search for that? 51 00:05:03,280 --> 00:05:06,140 Probably I want to look for our host name. 52 00:05:06,140 --> 00:05:17,800 So, what I can do is I can type host, and then Moloch would automatically parse the different fields based on the protocols that exist in the packet. 53 00:05:17,800 --> 00:05:24,380 So, if it was HTTP traffic, then Moloch will say, okay, well, this is an HTTP host name. 54 00:05:24,380 --> 00:05:27,500 If it's an email, then it's an email host name and so on. 55 00:05:27,880 --> 00:05:30,960 In my case, because I want to see everything, how can I post it? 56 00:05:30,960 --> 00:05:32,260 I can just say host. 57 00:05:32,640 --> 00:05:38,480 And to specify what I'm looking for in Moloch, I would just say equal equal. 58 00:05:39,400 --> 00:05:47,060 So, equal equal means show me everything that matches exactly our host name. 59 00:05:47,680 --> 00:05:49,940 On ad.ase. 60 00:05:50,220 --> 00:05:51,100 Okay? 61 00:05:51,260 --> 00:05:52,660 That's the first bit of it. 62 00:05:52,660 --> 00:05:53,980 And I can run that. 63 00:05:54,200 --> 00:06:01,720 I also know, since it's our website, I want to say, well, it's a website, it's going to only run on two ports, for example. 64 00:06:01,720 --> 00:06:11,580 So, to kind of add more queries into Moloch, what you need to do is, you just do two amp signs as an end. 65 00:06:11,740 --> 00:06:15,540 And I'm going to open parentheses and add the ports that we need. 66 00:06:15,540 --> 00:06:20,600 So, port equals 80, just like we did before. 67 00:06:22,120 --> 00:06:30,200 And port equals 443, which we know they both are HTTP ports. 68 00:06:30,200 --> 00:06:39,320 Now, before we go any further, you know, under ports I can say, well, I can specify I want it to be a destination port, I can have it be a source port, or so on. 69 00:06:39,540 --> 00:06:48,320 For, just to keep it general, and we can see everything here, I'm just going to say, okay, these ports, this host name, and let's see what we find. 70 00:06:48,560 --> 00:06:51,000 And I messed up something, of course. 71 00:06:51,180 --> 00:06:55,140 Oh, I'm saying end here, and that should be or. 72 00:06:55,140 --> 00:06:58,300 So, it's either port 80 or port 443. 73 00:06:59,440 --> 00:07:06,920 So, in that case, I just do two pipelines, and run that. 74 00:07:07,940 --> 00:07:09,460 Not pipelines, but pipes. 75 00:07:09,460 --> 00:07:14,720 And now I'm saying, okay, show me everything for this host on port 80 and port 443. 76 00:07:14,720 --> 00:07:17,140 And now I can see the traffic. 77 00:07:18,300 --> 00:07:22,140 So, let's go and dive deeper. 78 00:07:22,960 --> 00:07:31,840 On this traffic, I can see all kind of requests, and let's just open something very randomly. 79 00:07:31,940 --> 00:07:36,500 Let's open up this request, and I see it's a get request. 80 00:07:36,500 --> 00:07:39,380 And before we go down, let's spend some time here. 81 00:07:39,380 --> 00:07:44,260 So, idcmalloc will parse all the fields in the packet. 82 00:07:44,260 --> 00:07:50,280 So, I can click on any of these, and I can say, okay, well, what protocol did it come from? 83 00:07:50,280 --> 00:07:52,220 Which IP, which ports? 84 00:07:53,260 --> 00:08:00,360 If it's HTTP packet, then it will also parse the method, the status code, all of these things. 85 00:08:00,680 --> 00:08:02,580 It will also parse the user agent. 86 00:08:02,580 --> 00:08:11,980 Let's say I'm interested in knowing all the user agents that happened in that specific day that visited our website. 87 00:08:12,040 --> 00:08:18,640 I can easily click on user agents, and I can say, okay, export unique user agents with counts. 88 00:08:18,640 --> 00:08:29,800 Once I click that, I will see a new page showing me the unique user agents, along with how many times we've seen this in that specific day, and the other one. 89 00:08:29,800 --> 00:08:34,780 So, we only see two different user agents, nothing abnormal, nothing suspicious. 90 00:08:34,780 --> 00:08:37,580 So, nothing to worry about here. 91 00:08:38,300 --> 00:08:45,360 If we scroll a bit down, we can actually see the actual requests, the raw requests. 92 00:08:45,360 --> 00:08:50,600 So, I can see which hosts they're requesting, where the URI they're going after, and so on. 93 00:08:51,320 --> 00:09:01,440 Because I know we had an incident, and I know the website has been defaced, the attacker must have sent some kind of data to our website. 94 00:09:01,440 --> 00:09:04,540 So, most likely they will not be doing a GET request. 95 00:09:04,540 --> 00:09:07,280 When we see GET, it's just pulling data from our website. 96 00:09:07,280 --> 00:09:10,040 They're going to be posting or sending some kind of data. 97 00:09:10,040 --> 00:09:12,660 So, this excludes this. 98 00:09:12,860 --> 00:09:18,160 So, I'm going to click GET, and we don't want GET anymore, so I'm just going to say net GET. 99 00:09:18,420 --> 00:09:20,940 That automatically adds that to our query. 100 00:09:22,580 --> 00:09:24,820 And let's hit search and see what we find. 101 00:09:24,820 --> 00:09:28,280 So, now we went down to only 12 entries. 102 00:09:28,280 --> 00:09:34,500 Okay, that's much easier to kind of go through and investigate. 103 00:09:35,080 --> 00:09:47,460 Again, we're not sure what happened, but one thing I can sort for is by time, to kind of understand the timeline of these different requests and the events. 104 00:09:48,220 --> 00:09:51,000 And this just opens on the random. 105 00:09:53,680 --> 00:09:55,800 So, again, same kind of thing. 106 00:09:56,040 --> 00:09:59,660 So, now it's a POST request. 107 00:10:02,920 --> 00:10:05,400 And it's just a request in index.php. 108 00:10:10,470 --> 00:10:12,650 And I see a test on sleep. 109 00:10:12,650 --> 00:10:14,850 I'm not sure what all this stuff really means. 110 00:10:14,850 --> 00:10:17,510 And I also see this jumper stuff. 111 00:10:17,510 --> 00:10:18,770 What is this? 112 00:10:19,070 --> 00:10:25,510 So, if we look at the header, which I've already displayed for you, it turns out it's actually encoded with gzip. 113 00:10:25,510 --> 00:10:33,170 Gzip is just a method of compression that web browsers use to compress the data to transfer the least amount of traffic. 114 00:10:33,370 --> 00:10:41,750 So, the nice thing with Molek, what we can do is I can just click uncompress, and now it automatically will decode this packet for us. 115 00:10:41,750 --> 00:10:47,190 And now I can easily read it and see, oh, okay, this is what loaded, this is what the page is showing. 116 00:10:47,570 --> 00:10:52,110 And from what we see here, nothing of interest yet. 117 00:10:52,110 --> 00:10:54,130 Okay, that was useful. 118 00:10:54,130 --> 00:10:56,270 But let's keep going. 119 00:10:56,270 --> 00:10:57,730 Let's find out what happened. 120 00:10:58,270 --> 00:11:02,330 So, another request, and see this. 121 00:11:02,330 --> 00:11:05,410 But this is like, this looks like an IP address. 122 00:11:05,410 --> 00:11:06,490 But what is all this? 123 00:11:06,490 --> 00:11:07,470 I don't know. 124 00:11:07,930 --> 00:11:13,810 Well, what we can do is Molek has Cybershift built in. 125 00:11:13,950 --> 00:11:25,510 Well, Cybershift is a separate open source project, that you can actually just go and do Cybershift, and you can load it outside of... 126 00:11:26,510 --> 00:11:28,330 Cybershift, there we go. 127 00:11:31,600 --> 00:11:35,540 And you can just load it outside of Molek, and you can do your decoding. 128 00:11:35,540 --> 00:11:36,380 But let's get back here. 129 00:11:36,380 --> 00:11:43,860 The beauty of this is when we do it on Molek, it will automatically take that packet data and put it in for us. 130 00:11:43,880 --> 00:11:45,640 So, what is Cybershift? 131 00:11:45,640 --> 00:11:55,360 It's pretty much a tool, a web GUI, that you can use to do different decoding of different languages, decoding methods, and so on. 132 00:11:55,360 --> 00:12:00,600 So, in this case, it's taking the hex code and just decoding it. 133 00:12:00,800 --> 00:12:05,560 And while I'm looking at this, I can see these percentage signs. 134 00:12:05,960 --> 00:12:09,400 Percentage sign means it's a URL kind of thing. 135 00:12:09,400 --> 00:12:12,860 So, I can easily just drag the URL decode. 136 00:12:12,860 --> 00:12:16,700 And now I can see the decoded message here. 137 00:12:16,920 --> 00:12:25,000 Now I can see, oh, it's trying to ping this IP address, which is the same IP that's trying to visit. 138 00:12:25,000 --> 00:12:27,200 Okay, this is interesting. 139 00:12:27,200 --> 00:12:35,120 I'm not sure if they were actually able to ping, but it looks like this IP is trying to do something here. 140 00:12:37,380 --> 00:12:40,540 Let's keep going, see what else can we find. 141 00:12:42,860 --> 00:12:46,620 Next packet, same thing. 142 00:12:46,620 --> 00:12:50,060 Tests, okay, nothing here. 143 00:12:50,480 --> 00:12:54,180 Next packet, so on. 144 00:12:55,640 --> 00:12:57,780 Oh, there's an NC. 145 00:12:57,840 --> 00:12:58,980 Okay, what is this? 146 00:12:58,980 --> 00:12:59,460 Hold on a second. 147 00:12:59,460 --> 00:13:02,220 Let's pull it up again in CyberChef. 148 00:13:02,220 --> 00:13:05,860 Do the same method, decode the URL. 149 00:13:05,940 --> 00:13:10,960 And now we see an NC IP address and port. 150 00:13:10,960 --> 00:13:19,040 So, NC stands for Netcat, which is the utility attackers can use to have a server connect back to them and get a show in that box. 151 00:13:19,040 --> 00:13:24,920 So, what this is saying is, okay, connect back to my IP over this port. 152 00:13:25,600 --> 00:13:34,520 So, what this means, if this actually succeeded, that means our server connected back on this IP address. 153 00:13:34,800 --> 00:13:37,820 Well, let's see if this actually happens. 154 00:13:37,820 --> 00:13:46,600 I'm going to take that port, I'm going to clean all this up, and then I'm going to add the IP address. 155 00:13:47,780 --> 00:13:49,500 It's not going to be the source. 156 00:13:52,140 --> 00:13:57,400 And port equals the port that we're looking for. 157 00:13:57,480 --> 00:13:58,520 Let's search for it. 158 00:13:58,520 --> 00:13:59,700 And we have traffic. 159 00:13:59,700 --> 00:14:00,420 Uh-oh. 160 00:14:00,880 --> 00:14:02,420 This is not good. 161 00:14:02,420 --> 00:14:08,800 Looks like our box, which is the source here, connected back to the server over this port. 162 00:14:09,040 --> 00:14:14,240 We can see how many packets and the amount of data that was sent back and forth. 163 00:14:14,600 --> 00:14:24,960 So, immediately, I would say, okay, well, this is the first connection that has the highest number of packets, that might have something interesting in it, so let me open it up. 164 00:14:26,540 --> 00:14:33,280 Scroll down, and we can actually see the whole conversation back and forth now. 165 00:14:33,280 --> 00:14:47,320 Well, it looks like the attacker ran a command, which is id, which is equal to whoami on Windows, to figure out who the attacker is running as on this box, and they're running as apache. 166 00:14:47,580 --> 00:14:55,220 Then they try to figure out which folder they're in, and they try to access file systems. 167 00:14:55,220 --> 00:14:56,820 Okay, this is not good. 168 00:14:57,140 --> 00:15:07,740 And then they did this cat-index.php, which looks like it's our website, so they're looking into what's inside our main file and so on. 169 00:15:07,900 --> 00:15:09,840 Okay, what is all this? 170 00:15:09,840 --> 00:15:22,220 Oh, and then I see another netcat command here, and saving the connection as a cm0.php file, and our box, locally. 171 00:15:22,500 --> 00:15:26,820 So I'm interested in knowing what this is, but let's keep going. 172 00:15:26,820 --> 00:15:29,020 Let's see what else they do in this box. 173 00:15:30,180 --> 00:15:32,760 And if we come here, they actually did it for us. 174 00:15:32,760 --> 00:15:35,140 They did a cat on this file that they created. 175 00:15:36,420 --> 00:15:40,220 Okay, when they did that, we can see, oh, it's a PHP backdoor. 176 00:15:40,480 --> 00:15:44,420 This is not good, so they used a PHP backdoor on our site. 177 00:15:45,060 --> 00:15:48,800 So how am I going to now figure out what did they do? 178 00:15:49,060 --> 00:15:57,820 I know because of how backdoors work, they have to visit this PHP file explicitly to kind of load the command they want to run. 179 00:15:57,820 --> 00:16:03,240 So what I can say to Molek is, well, show me all the URIs. 180 00:16:03,520 --> 00:16:06,060 So when we say a URI is... 181 00:16:07,880 --> 00:16:14,300 pretty much everything after the domain will be a URI, so anything from here on, that's a URI. 182 00:16:14,300 --> 00:16:18,480 So I want to say, show me everything that has this in it. 183 00:16:18,600 --> 00:16:33,320 And since I can't just do, like, all, what I do in Molek is I just use wildcards at the beginning, so I don't care what's before this, and the end, meaning I don't care what's after it, just show me anything that has this string in the URI. 184 00:16:33,580 --> 00:16:44,820 And when I search it, now, here, we can see all these URLs or URIs with all these different links in them. 185 00:16:44,820 --> 00:16:46,400 What are they trying to do? 186 00:16:46,600 --> 00:16:51,240 Well, how can I know, like, how can I get all this in one nice place? 187 00:16:51,240 --> 00:16:57,240 I can just click on info, and I want to say export unique URI with two counts. 188 00:16:58,380 --> 00:17:06,560 Now, Molek will take this for us and tell us, okay, well, this command was run five times, this command was run three times, and so on. 189 00:17:06,560 --> 00:17:16,900 So now I kind of know which commands they ran, because I can see it's command equal cat, command equal cat, or else, or whatever they're trying to do. 190 00:17:17,180 --> 00:17:20,280 And I kind of see here there's a JPEG file as well. 191 00:17:20,280 --> 00:17:29,460 Okay, so this is how they probably got that JPEG file or the image of the fraud into our website. 192 00:17:29,820 --> 00:17:35,920 So let's find out where did they, like, how did they do this? 193 00:17:37,240 --> 00:17:43,000 Let's go back here, and let's do... 194 00:17:43,000 --> 00:17:47,700 Instead of this, we are looking for that specific JPEG file. 195 00:17:48,220 --> 00:17:53,020 So I'm going to do wildcard and show me anything that has JPEG in it. 196 00:17:53,960 --> 00:17:55,180 Load it up. 197 00:17:56,060 --> 00:17:58,180 And it's going to show me a lot of stuff. 198 00:17:59,620 --> 00:18:06,640 I'm going to say probably it's going to be the biggest file, so I'm going to sort by the size of the data. 199 00:18:08,060 --> 00:18:10,740 And this is the biggest file that we have. 200 00:18:14,970 --> 00:18:16,350 So it's loading. 201 00:18:17,010 --> 00:18:19,950 And here I can see, yeah, it's an image file. 202 00:18:23,290 --> 00:18:25,510 But I'm not sure what it looks like. 203 00:18:25,510 --> 00:18:33,310 So what I can do in Moloch is I can just click on show images and files, and it will actually render it for us right here in the browser. 204 00:18:34,390 --> 00:18:45,030 Now, let's say, you know, I'm not comfortable with doing this or I want to dig in deeper to it or I want to understand what actually happens using Wireshark because I've never used Moloch before. 205 00:18:45,030 --> 00:18:53,090 At any session, at any point, you can always just click on download PCAP, and it will kind of, like, save the PCAPs for you. 206 00:18:53,470 --> 00:19:00,710 So you can open up in Wireshark and, you know, do your analysis manually if you'd like to. 207 00:19:04,650 --> 00:19:12,270 Pretty much that's how they get the file, and that's how they deface our website using a backdoor. 208 00:19:12,270 --> 00:19:17,810 If we have time, I think we have another... I think that's pretty much it. 209 00:19:17,810 --> 00:19:25,050 Okay, so with that being said, I think that's the highlight of Moloch and how to use it. 210 00:19:25,050 --> 00:19:27,290 I want to keep some time for questions. 211 00:19:28,590 --> 00:19:30,950 Please let me know any questions you've got. 212 00:19:31,170 --> 00:19:32,750 I'm also going to be in Discord. 213 00:19:32,750 --> 00:19:38,450 I'll be on Twitter if you need anything else, but I'm going to be here waiting for any questions that you guys have. 214 00:19:39,610 --> 00:19:40,610 Cool. 215 00:19:41,290 --> 00:19:42,470 Thank you, Bashar. 216 00:19:42,630 --> 00:19:54,370 I went ahead and put a note in the text window under the workshop track one with a link over to Recon Episec's OpenSOC Moloch Discord channel. 217 00:19:54,830 --> 00:19:56,250 So definitely check that out. 218 00:19:56,250 --> 00:20:05,370 Obviously, hit Bashar up on Twitter and Discord, but we're trying to help everybody kind of connect with the right people here so you can get the help you need. 219 00:20:06,870 --> 00:20:07,590 Perfect. 220 00:20:08,830 --> 00:20:09,890 Thanks a lot, Bashar. 221 00:20:09,890 --> 00:20:10,990 I appreciate it. 222 00:20:11,130 --> 00:20:11,810 Thank you. 223 00:20:11,810 --> 00:20:12,170 Thank you. 224 00:20:12,170 --> 00:20:12,870 I appreciate your time. 225 00:20:12,870 --> 00:20:13,570 Thank you. 226 00:20:13,590 --> 00:20:14,210 Take care.