WEBVTT 00:01.690 --> 00:06.210 So let's finally talk about visual security event analysis. 00:11.370 --> 00:20.490 So what I'm going to talk about is how you analyze log files or events that you have in a visual manner. 00:22.570 --> 00:25.090 I don't want to talk about myself too much. 00:25.530 --> 00:26.750 Maybe one thing. 00:26.750 --> 00:43.110 I've been involved in, like, log analysis and event analysis for quite a while, and it has always interested me whether there's a way to look at the data in a way that it really, like, a lot of data, in a way that it's really apparent what is going on in that dataset. 00:43.370 --> 00:49.390 And if you're looking at log files, thousands of lines of logs, it's really hard to figure out at one glance what is going on. 00:49.390 --> 00:50.830 You have to start reading through them. 00:50.830 --> 00:55.890 You have to, like, use your own mind to do pattern analysis and figure out what is really going on. 00:55.890 --> 01:22.330 So at some point, I started working for this company that I'm working for right now called ArcSight, and I got a hand on a lot of neat tools to look at data, and I got a feature in the software that is called Event Graphs, and I really liked it, and I started looking into how can we use this feature to make it more easy for a security analyst or a person interested in a log file to figure out what is really going on in that dataset. 01:22.330 --> 01:26.670 So that's how kind of my passion about Event Graphs started. 01:27.530 --> 01:34.550 My talk here is not going to be very textual or, like, theoretical, and I'm not going to talk too much about how you generate the graphs and whatever. 01:34.610 --> 01:40.990 I want to show you a lot of examples on how to look at datasets and how you configure graphs to get the most out of it. 01:41.510 --> 01:58.230 As part of the talk, I'm going to show you a tool that I have up on SourceForge, which is called Afterglow, which is a very simple Perl script that transforms CSV files into a graph or language file, so you can go home and play with these things yourself, 01:58.230 --> 02:00.010 and the tool is actually also on the CD. 02:02.570 --> 02:06.510 I'm going a little faster in the beginning so we can catch up for the time I lost already. 02:06.510 --> 02:12.130 One thing, IP addresses and host names, you're going to see in all these graphs, and the descriptions are obfuscated or changed. 02:12.130 --> 02:17.410 The addresses are completely random and any resemblance with well-known addresses or host names are purely coincidental. 02:18.510 --> 02:19.390 All right. 02:19.690 --> 02:22.490 So let's do a little bit of an introduction here. 02:22.490 --> 02:23.970 Why would you want to look at these graphs? 02:23.970 --> 02:25.770 I guess this is a pretty good example already. 02:25.770 --> 02:29.330 Do you want to look at the left-hand side, or do you want to look at the right-hand side? 02:32.670 --> 02:36.850 Well, even if I made the font bigger on the left-hand side, it's probably still not very fun. 02:37.330 --> 02:38.050 All right. 02:38.670 --> 02:40.410 Why do you want to use event graphs? 02:40.870 --> 02:43.090 I guess I already motivated that quite well. 02:43.090 --> 02:45.090 I'd rather look at the graphs. 02:45.090 --> 02:47.270 They look just much nicer. 02:48.520 --> 03:05.390 If you actually dig into it a little more and you start looking how to configure these things, you will see pretty quickly that you can visualize different properties in those graphs, and it's very easy to see certain aspects of a log file very quickly and certain things that are important to you. 03:05.390 --> 03:13.690 You don't have to wade through ten thousands of lines of logs to figure out whether there is anyone sending a lot of e-mails to someone else. 03:14.590 --> 03:33.310 If you use this in a security operating center or in a 724 response team or something, if you have graphs, they're a great way of giving you situational awareness or real-time view on things that are happening, and you can quickly react on them. 03:33.690 --> 03:41.330 We have a lot of customers that are starting to use this in their SOCs, and it doesn't just look pretty on their big screens, but it's actually really useful. 03:42.010 --> 03:43.870 It's actually also kind of an interesting story. 03:43.870 --> 03:52.750 I was at one customer, and they had all these big displays, and they were empty, and I was like, wow, why do you have these nice graphs here on the analyst screens? 03:52.750 --> 03:54.870 Why don't you have them up there on the big screens? 03:55.110 --> 04:00.830 They were like, whoa, no, no, no, we tried that for a while, but then the boss came in, and he saw all these things on the screen. 04:00.830 --> 04:01.650 He was like, oh, what is this? 04:01.650 --> 04:02.050 What's that? 04:02.050 --> 04:02.430 What's this? 04:02.430 --> 04:02.850 What's that? 04:02.850 --> 04:03.950 So we had to explain all this. 04:03.950 --> 04:04.950 We took them off. 04:08.140 --> 04:12.800 All right, and one other thing, another story that I had with these graphs, which actually two other stories. 04:12.800 --> 04:18.980 One is, one customer uses these graphs to communicate with other teams, so you have the security analysts. 04:19.000 --> 04:27.460 They look at the log files and all that, and then they go back to the operators or the operational folks, and they're like, guys, you have to do something about this machine. 04:27.460 --> 04:29.200 It's talking to all the others in the network. 04:29.280 --> 04:37.020 And in the past, they were handing them log files to prove that it's that way, and the operators were like, whatever, we don't understand what this is. 04:37.020 --> 04:42.120 But now they started giving them graphs, and the operators were like, oh, my God, my machine is talking to all these? 04:42.120 --> 04:43.720 Okay, we're going to take care of this. 04:44.200 --> 05:00.380 And even worse, I had one customer that actually just told me this week that he used the graphs to analyze some data, and he found some stuff, and one of the guys got fired just because he used the graphs and found out that one of the guys was not doing his work. 05:04.090 --> 05:09.130 All right, so I came up with, like, two big categories of when to use these event graphs. 05:09.130 --> 05:18.330 One is in a real-time monitoring scenario, where you have, like, a screen app that looks real-time at certain properties of your data file, and I'm going to show you some examples on what I mean by that. 05:18.590 --> 05:20.950 It gives you great awareness of things that are happening. 05:20.950 --> 05:26.270 You can configure these things to, for example, look at transactions going to your financial server. 05:26.270 --> 05:29.910 So right away, if someone connects to your financial service, you will know. 05:30.010 --> 05:31.670 And then you can take it a level further. 05:31.670 --> 05:37.930 You can say, well, I know that these machines are okay to connect to my financial servers, so don't show me those, but show me everything else. 05:37.930 --> 05:41.270 So as soon as something pops up there, you know you've got to look into it. 05:41.610 --> 05:53.790 The other thing is, if you take this in a more forensic or historical way, you can, you grab a data set, you run it through some tools, and you look at the graphs on what is, on certain properties, and you figure out what's going on. 05:55.390 --> 05:57.310 Let me mention quickly some related work. 05:57.310 --> 06:02.410 There's some classics from, he did all these kind of things. 06:02.410 --> 06:11.450 He tried to figure out whether there are certain ways to visualize properties of data, and he took a lot of dimensions and put them all in one graph. 06:11.830 --> 06:19.390 What I'm going to do is, I take a lot of dimensions too, and I'm always going to stay with event graphs, so like node-to-node connections. 06:19.430 --> 06:27.610 I'm not going to use any of the parallel coordinates or the right-hand-to-bottom thing that I have no idea what it shows there. 06:28.310 --> 06:31.170 Then Ehrbacher, he also, he actually went even further. 06:31.170 --> 06:41.870 He tried to go even more multidimensional and map everything into these nodes, so there's like size, there's shape, there's length of these spirals going out, and so on. 06:42.810 --> 06:46.990 Then Greg Conti, you probably have seen his talk, I hope. 06:47.750 --> 06:55.910 He mainly focuses on different ways to visualize TCP dump or packet locks, and very interesting. 06:57.510 --> 06:59.570 Then Envision IP is something I found. 06:59.570 --> 07:10.530 What you see up there is a grid of, I think it's 8x8 or 64x64, and they just map IP addresses in there, so you see certain clusters of things going on. 07:10.530 --> 07:16.630 The nice thing about this project is, there's a lot of interactivity, so you can start playing with the data, you can zoom in and so on. 07:17.030 --> 07:21.910 And then finally, a few of you might know Showkey from Steve Barry. 07:22.110 --> 07:23.490 It's basically a scatterplot. 07:23.490 --> 07:28.530 You can define three different scatterplots, and then you have a 3D of that. 07:28.810 --> 07:40.690 So on the top right, it's not a very nice picture, but on the top right it's a three-dimensional output, so you can say one axis is your source IP, one axis is destination IP, and the other one is the destination port. 07:41.630 --> 07:43.010 All right, let's move on. 07:44.310 --> 07:45.950 How do you generate an event graph? 07:45.950 --> 07:46.490 Pretty easy. 07:46.490 --> 07:49.190 You have a log file that a device is generating. 07:49.550 --> 07:52.150 You take a parser. 07:52.930 --> 07:54.170 Why do you take a parser? 07:54.170 --> 08:01.970 Well, you need to know where in my log event or my log file, where's the source IP, where's the destination IP, where's the event name, where's the port? 08:01.970 --> 08:07.550 So you parse that, you get the syntax right, and then you visualize that in an event analyzer. 08:08.250 --> 08:12.330 If you have an event, let's look at this Nord event here. 08:12.870 --> 08:20.550 You don't have to be really able to read this, but what I want to show here is in an event graph you can have different configurations. 08:20.830 --> 08:26.210 The normal event graph that you will always see is like who is talking to whom, so you have source IP, destination IP. 08:26.330 --> 08:28.870 But you can take this a lot further. 08:28.870 --> 08:40.730 You can say, well, on the right top you want to have the source IP, destination IP, and destination port, so you will see someone port scanning, for example, if there's a fan with destination ports. 08:40.830 --> 08:45.290 So you can configure these things in pretty wild ways, and I'm going to show you some of them. 08:46.990 --> 08:51.210 Now let's have a sneak preview of Afterglow. 08:51.450 --> 08:54.010 I apologize because I don't use my laptop running it. 08:54.010 --> 08:55.010 I can't show you the demo. 08:55.010 --> 08:59.130 It's not very fun anyways. 08:59.130 --> 09:02.070 I will show you the output anyway, so it doesn't really matter. 09:02.130 --> 09:03.510 So what is Afterglow? 09:03.510 --> 09:07.310 Afterglow is not a security information management tool. 09:07.330 --> 09:16.710 It's just a script which takes CSV files, so you have to somehow from your logs generate a CSV file, comma, separated values. 09:16.830 --> 09:24.130 And then you take Afterglow, the tool that I'm going to show you, and you transform the CSV files into a graph language file. 09:24.750 --> 09:32.530 And then you take a grapher tool, and right now I'm supporting two of them, that takes this graph language and generates the graph. 09:32.530 --> 09:34.210 And that's not something I did. 09:34.210 --> 09:40.130 I just transformed the CSVs into graph files, but I'm going to show you some cool things that we implemented there. 09:41.490 --> 09:55.830 So how you can use this thing is you basically cap the input file, you pipe it to Afterglow, you give it a color properties file, which I'm going to introduce a little later, and then you can use the AT&T Graphverse library, .neto, I'll give you references in a second, 09:55.830 --> 09:57.670 and you generate a GIF file. 09:58.230 --> 10:06.830 The color properties file is basically just you tell it the source nodes are red, for example, the event nodes are green, and the target nodes are blue. 10:06.830 --> 10:09.370 And I'm going to show you how to really play with this stuff. 10:09.370 --> 10:15.970 We extended this quite a bit so you can have colors depending on the values in the log files. 10:17.250 --> 10:18.850 Big shout out to Christian. 10:18.850 --> 10:22.550 Thanks a lot for building the foundation of this tool. 10:23.010 --> 10:32.170 He built me the whole Afterglow profile, the basics and everything, and I just went in and made some small changes, but thanks a lot, Chris. 10:34.330 --> 10:36.390 Let's look at examples. 10:37.750 --> 10:42.790 I mentioned in the beginning I kind of have two use cases or two big categories of things to look at. 10:42.790 --> 10:47.230 One is what I call situational awareness or real-time monitoring. 10:51.120 --> 10:55.300 So this is one of the views that I saw at a customer. 10:55.600 --> 11:02.560 Basically you see here is six independent screens of different properties of your data that you want to look at. 11:02.920 --> 11:14.280 So here the configuration is such that on the top left the customer wanted to see just the compromise attempts, and don't ask me how to filter for just those events. 11:14.280 --> 11:16.080 That's a whole different story. 11:16.840 --> 11:30.280 But if you have a way to say these are my malware events, these are my DOS events, these are my scan events, if you can separate them by something, then you can draw all these screens and you can look at these screens and see what's going on. 11:30.500 --> 11:36.620 So what this customer is actually also doing, whenever an analyst starts his shift, he resets all the graphs so they're empty. 11:36.620 --> 11:47.340 So over time, nodes start popping up and they look at them and they're like, okay, this is fine, and they might filter them out even, or they just leave them there and say, okay, I know about this. 11:47.420 --> 11:52.000 And the more things that show up, the more they go along in their workflow and work with these events. 11:52.220 --> 12:01.080 So I have a feeling this is going to be a great tool, especially in a SOC or an analysis environment, to kick off your workflow. 12:01.080 --> 12:09.440 You look at these things and you take it on and you start building filters to filter out the things you don't know about, and you just see the outliers here in these screens. 12:11.300 --> 12:22.760 The second big group of use cases is forensic analysis or historical analysis, where you take a log file that you have and you visualize it. 12:24.520 --> 12:27.100 Afterglow can generate two kinds of outputs. 12:27.100 --> 12:34.760 One is for the AT&T graphics library for .neto, and the other one is for LGL, the large graph library. 12:35.420 --> 12:37.760 I will give a pointer to that in the end as well. 12:37.760 --> 12:40.360 This thing is generating three-dimensional graphs. 12:40.360 --> 12:43.860 You can actually generate VRML output and you can fly through these things. 12:44.440 --> 12:52.740 I didn't prepare a demo because it's kind of fun to fly through it, but the analysis value is not really high or the value you get out of it. 12:52.740 --> 12:56.020 It's really cool to fly through it and see that, oh, this guy sent an email. 12:56.020 --> 12:56.480 To whom? 12:56.480 --> 12:57.520 Oh, over there. 12:57.520 --> 12:58.120 Okay. 12:59.060 --> 13:03.720 It looks great, and I apologize for the colors here, but this is really how it looks. 13:04.220 --> 13:05.900 Anyway, let's go back to 2D. 13:07.000 --> 13:14.500 This is a very simple example, one of about 12 that I'm going to show, of how to analyze a log file. 13:14.600 --> 13:26.820 So what I had here is, I believe, a snort log or a web log or something, and I said, well, show me what is happening. 13:26.820 --> 13:32.800 So the event name, the destination IP are the blue ones, and then the destination ports are the white nodes. 13:32.880 --> 13:37.060 So what you see here, all the blue circles are my web servers. 13:37.820 --> 13:41.840 The white boxes are the target ports that they're being connected to. 13:41.840 --> 13:44.620 So you see it's 80 and 443, so that's totally legitimate. 13:44.620 --> 13:49.640 Only HTTP and HTTPS traffic is going to my servers, so I probably don't have to worry about that. 13:49.640 --> 14:01.600 If I would see a node there, like port 21, and someone connects to my web servers with FTP, maybe that's okay, but if I'm not running FTP in my environment and that shows up here, I probably want to investigate that. 14:01.720 --> 14:07.600 So it's a very quick way to see whether traffic is really what it's supposed to be. 14:07.600 --> 14:11.460 Almost same configuration. 14:11.460 --> 14:14.520 Here you see source IP, destination IP, destination port. 14:14.520 --> 14:18.680 So this would potentially show you port scans or network scans. 14:18.680 --> 14:24.660 Here you see a lot of machines connecting to a lot of other ones, all on the same port. 14:25.700 --> 14:28.860 Down here, this white node here, is the port. 14:28.860 --> 14:30.200 So they all connect to the same port. 14:30.200 --> 14:41.600 Well, it could be that a worm is probing you, but it could also be that you have a web server, or a lot of web servers, and a lot of machines are connecting to those. 14:41.600 --> 14:50.260 So it might be okay, but it might be a scan, and depending on what the destination port is, you might figure out what's going on in this case. 14:50.520 --> 15:06.760 But you can certainly tell, well, this is all similar behavior here on this fan, and then actually if you see on the bottom left here, there's some little ones, it's probably worth investigating what those are, and probably those are the outliers that you are interested in. 15:09.200 --> 15:11.280 Let's keep this configuration of the graph. 15:11.280 --> 15:14.380 This is just a little bit of a different dataset and a different time. 15:15.120 --> 15:16.840 It looks much prettier. 15:18.260 --> 15:21.980 Let's zoom in on the bottom part here. 15:22.460 --> 15:30.020 What this is showing you is that there are two machines, the red nodes, that are connecting to about 15 other ones, and still on the same port. 15:30.260 --> 15:40.020 So again, in your environment, do you really have all these other 15 or 20 machines serving this one service, and exactly two machines are connecting to them? 15:40.040 --> 15:49.500 So you can go and figure out what all these machines are, and hopefully they are really the ones that are serving this service, and then you can figure out whether these other two are really allowed to connect to this. 15:50.060 --> 15:53.460 Let's have a look at this weird thing over here. 15:55.240 --> 16:14.560 What you see here is, probably you can't read this, but there is a node here that says 21 FTP, so one of the properties of this graph is that the white nodes show up exactly once per target port, so everything that happens on port 21 is going to show up around that thing there. 16:14.940 --> 16:23.520 So again, I can say, wow, I have about six FTP servers, and about five people are connecting to them. 16:23.520 --> 16:28.740 So the FTP servers would be the blue nodes, and then the red ones that connect to them would be the people using it. 16:28.760 --> 16:31.380 So, well, is this really something I have in my environment? 16:31.380 --> 16:32.380 Is this allowed? 16:32.380 --> 16:35.920 And why are only three or five people connecting to them? 16:35.920 --> 16:43.400 Is that some kind of a process that, I don't know, maybe AIX that transfers log files over, something weird like that? 16:43.820 --> 16:45.320 Or is this something malicious? 16:45.320 --> 16:47.260 Is anyone having a ware server or whatever? 16:48.980 --> 16:52.320 If you interpret these graphs, you have to be a little careful. 16:53.060 --> 16:54.700 I still have the same configuration. 16:54.820 --> 16:58.480 Destination IPs are the blue ones, and the white ones are the destination ports. 16:58.800 --> 17:03.840 Well, if you look at this on the left-hand side, I would say, wow, it's a port scan. 17:04.360 --> 17:05.960 Well, is it really a port scan? 17:09.410 --> 17:12.630 I have a book for the one that gets this right. 17:12.630 --> 17:14.290 What is the left-hand side? 17:16.230 --> 17:17.830 No, not voice over IP. 17:20.860 --> 17:21.940 Who said that? 17:23.120 --> 17:23.920 Over there. 17:23.920 --> 17:24.840 You got the book. 17:24.840 --> 17:26.240 It's the source ports. 17:26.300 --> 17:35.700 The log file reports such that the source port is really the destination port and the other way around. 17:35.700 --> 17:41.640 So when you analyze these things, be careful and make sure you're really looking at what you're looking at. 17:41.640 --> 17:45.460 So why do I know are the source ports and the destination ports? 17:45.460 --> 17:47.660 Well, they're in the 50,000 range. 17:48.100 --> 17:50.100 They're almost contiguous. 17:50.780 --> 17:53.820 And I mean, who is scanning me on these high ports? 17:53.820 --> 17:55.580 We're not a capture the flag. 17:55.580 --> 17:59.520 There, they might scan you on those high ports because services are running all over. 17:59.560 --> 18:02.900 But here, there's really no sense why someone would scan you up there. 18:02.900 --> 18:05.420 There's probably nothing there listening anyway. 18:05.520 --> 18:07.560 So be careful when you analyze these things. 18:07.560 --> 18:10.020 Sometimes it's port scan, sometimes it's not. 18:10.020 --> 18:14.240 I'm always, my first impression is always, well, false positive. 18:14.240 --> 18:16.320 Let's really figure out whether that's true. 18:19.270 --> 18:20.930 Okay, let's get fancy here. 18:21.810 --> 18:23.530 This is a firewall log. 18:23.530 --> 18:27.970 Basically, I'll show you passes and blocks from a PF firewall. 18:29.790 --> 18:36.690 If I would just visualize source IPs, destination IPs, I would see big clouds of things. 18:36.710 --> 18:38.750 Well, I want to see a little more in this graph. 18:38.750 --> 18:45.330 So what I did is I have source port, rule number, that's the squares, and then destination IP. 18:45.330 --> 18:54.250 So you will see I'm connecting to some other machine and rule number 100 is responsible for letting me in or blocking me. 18:55.070 --> 19:03.530 What I did then with Afterglow is to configure my nodes such that all the external IPs are going to show up as orange. 19:03.530 --> 19:07.630 I'm going to show you how to configure Afterglow to get exactly this graph. 19:07.890 --> 19:10.570 So the external machines are orange. 19:10.650 --> 19:12.330 My internal machines are green. 19:12.330 --> 19:14.170 And you see I only have a few of them. 19:14.250 --> 19:16.950 And then the rules are the blue boxes. 19:17.350 --> 19:22.550 And then further what I said, I wanted to see whether a connection is incoming or outgoing of my network. 19:22.550 --> 19:31.170 So I configured the edges such that outgoing connections are blue, that's the majority here, and incoming is red. 19:31.650 --> 19:34.290 And I just realized it's probably exactly the other way around. 19:34.290 --> 19:39.470 So I hope there's more incoming than outgoing because these are web servers that are servicing some stuff. 19:39.870 --> 19:46.070 Anyway, so you see big clouds of things that are showing similar behavior. 19:46.070 --> 19:50.690 So down here there's this one rule that is responsible for letting in a whole lot of traffic. 19:51.290 --> 19:53.150 And probably that's fine. 19:53.390 --> 19:54.970 There's another one up here. 19:55.510 --> 20:01.910 You can, if you know the rule set, if you know your ACLs, you can go and figure out whether it's really what it's supposed to do. 20:02.130 --> 20:05.990 But you can also figure out some things that are kind of weird. 20:06.850 --> 20:09.450 You have this whole section in the middle. 20:09.450 --> 20:17.750 You probably want to zoom in on those and filter out all the others and just look at these things and figure out whether your rule set is really doing what it's supposed to do. 20:18.390 --> 20:25.390 So the next steps would be, so the first is like overview, general impression, what is going on, is there anything really sticking out. 20:25.390 --> 20:31.190 And then my next steps would be, I would visualize the firewall blocks of outgoing traffic. 20:31.410 --> 20:34.330 Why would internal machines connect outside and be blocked? 20:34.330 --> 20:40.090 So that's either a misconfiguration of the internal box or the firewall rule set is wrong. 20:40.390 --> 20:44.090 Then the second one would be , I would probably look at blocks of incoming traffic. 20:44.090 --> 20:46.670 I want to know who's probing me , who's poking around. 20:47.090 --> 21:01.090 And if you have a live system that can look at these things, you can put all these attacker machines maybe on a blacklist and if you see them again and they're successful coming in, you might actually take some action and say, whoa, now someone probed me and now he's coming in. 21:01.430 --> 21:02.870 And then you can take this further. 21:02.870 --> 21:06.270 Third step would be to visualize the passes of outgoing traffic. 21:06.270 --> 21:08.410 Is this really what should be leaving the network? 21:08.470 --> 21:13.170 Maybe you find someone FTPing stuff out that you really don't want to FTP out. 21:13.210 --> 21:20.010 Maybe you realize that you're running peer-to-peer clients in your internal network and they're connecting out, anything like that. 21:20.350 --> 21:25.810 And you can probably take a few more steps to analyze the data set, but I guess you get the idea. 21:27.090 --> 21:30.450 So this one is probably the wackiest graph I have. 21:30.530 --> 21:35.310 It's totally going away from source IPs and destination IPs. 21:35.310 --> 21:38.370 What I did here is I wanted to debug my firewall rule set. 21:38.370 --> 21:51.770 So it's still the same data set as before, but this time what I show you is the red nodes are the rule number or the ACL number, then the destination ports, and then you see the action taken, pass or block. 21:51.930 --> 21:57.030 So just by virtue of firewall rule configurations, you will see two distinct graphs. 21:57.110 --> 22:01.390 One on the left-hand side is all the passes and the right-hand side is all the blocks. 22:02.250 --> 22:13.330 So here you can see, if you take one of these blue nodes, for example, port 53 here, you see that this one rule is responsible for letting traffic in. 22:13.430 --> 22:23.270 Well, most is okay, maybe, probably, hopefully, but you have other ones where there are two nodes or three nodes connected to the same rule. 22:24.690 --> 22:30.210 Like over here, for example, there are three ports being blocked by this one rule. 22:30.210 --> 22:31.330 It might be okay. 22:31.330 --> 22:32.330 It might not. 22:32.330 --> 22:44.490 If you see something like this on the other side, that one rule passes traffic from multiple ports, maybe that's the way you configured it, but this is certainly a great way of analyzing and debugging your firewall rule set. 22:46.630 --> 23:03.730 This is another example where I took a TCP dump log, and what I visualized here is I take the region of where the source IP resides, so you can go on the web, there are databases that show you what the location codes are for the IP addresses. 23:03.730 --> 23:06.890 They're not very accurate, but in a lot of cases, they're actually pretty good. 23:07.490 --> 23:14.310 So I show you the source region, where it's coming from in the world, then I show you the source MAC address. 23:14.370 --> 23:15.970 So this is a TCP dump log. 23:15.970 --> 23:22.850 The source MAC address is going to be the last hub, probably my router, that let the traffic into my network. 23:22.850 --> 23:25.450 And then I show the destination IP. 23:25.470 --> 23:36.370 So what I see here, I have four boxes, the four white ones, that are my internal machines, and interesting enough, I have two entry points into my network. 23:36.370 --> 23:38.850 I have two source MAC addresses showing up. 23:38.850 --> 23:42.670 And then I have these two fans in the middle part. 23:42.890 --> 23:50.110 What this is showing me is I probably either have a redundant router setup, or maybe I have a load balancer. 23:50.330 --> 23:57.090 And what is interesting here is that all these machines up here, they're connecting through this guy. 23:57.090 --> 24:01.450 All these guys here are coming through both of them, and all these down here come from the other one. 24:01.450 --> 24:08.890 So maybe this is a regional load balancer, that everything on one side of the world takes this entry point, everything on the other side is using that entry point. 24:08.890 --> 24:18.070 So if that's really the case, a great way to analyze this log is this graph here, and you can actually figure out why these two in the middle are coming in for both of them. 24:19.430 --> 24:30.710 And this is just a zoom, where you see the one server, and here it's like, you can't even read it below the node here, but it's Sand Hill or whatever in the United States. 24:35.510 --> 24:41.170 Graphs like these are a lot of times indications for warms, but again, be careful. 24:41.250 --> 24:47.550 There's a nice fan here of source IPs using the same event going to a few destination IPs. 24:47.610 --> 24:58.930 Well, it could be, again, it could be a warm that is using this one event to try to get into a network, but it might again be a web server serving just legitimate files. 24:59.150 --> 25:07.210 What you have to do in this case, you have to figure out what is this event name here that connects all these dots, and this might give you an indication of a warm or not. 25:07.330 --> 25:12.070 If you're visualizing a snort log file and this talks about warm, well, duh, it's probably a warm. 25:12.210 --> 25:25.570 But if you visualize a TCP dump log file, like an ICMP port unreachable might be an indicator for a warm trying to probe UDP ports and you get ICMP port unreachables back. 25:25.770 --> 25:29.330 So you have to investigate a little bit to figure out whether it's really a warm. 25:29.330 --> 25:49.410 I was playing Capture the Flag actually this year and last year, and I was trying to figure out the challenge really for the analysis team on Capture the Flag is that all the traffic hitting your web server or your servers seems to be coming from the same IP address, 25:49.410 --> 25:59.210 and there is malicious traffic of the other teams hacking you, but there is also benign traffic from the score bots trying to figure out whether your services are still up. 25:59.290 --> 26:01.810 So blocking all the traffic is not working. 26:01.810 --> 26:12.890 So you have to figure out a certain way of distinguishing these two types of traffic, and you can't do it by the source IP, which you would normally do in your corporate environment. 26:13.350 --> 26:21.470 So I started visualizing this log file of TCP dump, and I just wanted to have a general feeling for what's going on. 26:21.470 --> 26:32.610 So what you see here, the dark blue ovals are destination ports that are below 1024, normally services that are offered to the outside world. 26:32.750 --> 26:44.830 Then destination ports higher than 1024 are light blue, and then you see here what I just explained, the red node, this is what all the traffic that's coming in is coming from this one IP. 26:44.830 --> 26:59.450 And then I configured it such that our machines, the internal targets are green, other teams' targets are brown, and then internal sources are yellow, and internal targets are the boxes. 26:59.550 --> 27:18.050 So what you see here, we have two servers, while there's a third one showing up down here, this was actually not a server, but I think that was one of the machines on our network that someone connected to, and that could have been one of our team members connected to someone else's machine. 27:19.270 --> 27:26.950 And then the services that we offered were D6, and it was all kinds of weird things. 27:27.810 --> 27:33.630 I don't even remember anymore what it was last year, port 9999 or something, and there was some mud running there. 27:33.770 --> 27:37.830 But this gives you a very quick way of looking at what is happening in this network. 27:38.850 --> 27:47.150 Well, I took it a little further, and I had an idea, or a few people had an idea, I said, well , maybe it's a TTL of the traffic coming in. 27:47.150 --> 27:50.910 Maybe you can distinguish the good traffic from the bad traffic by looking at the TTL. 27:51.030 --> 27:57.650 So well, what I did is I said, okay, show me the source IPs, the destination IPs, and then the TTLs of that traffic. 27:58.090 --> 28:11.890 So you see here, I have eight TTLs showing up, eight distinct TTLs, and I'm going to look at the everything from 35, 50, 52, 60, 62, 63, 64, 128, and 146. 28:13.470 --> 28:20.510 Some of them are showing up if my teammates would connect out, so I could filter them out. 28:20.510 --> 28:22.090 I was not interested in those. 28:22.090 --> 28:28.390 I'm just interested in the ones that happen of the traffic coming in from this one evil IP. 28:30.430 --> 28:32.150 Let's go back quickly again. 28:32.150 --> 28:40.810 And then I figured out, well, I still can't quite tell what is okay and what's not, so I had to reconfigure my graph and look at some other properties. 28:40.810 --> 28:44.630 So what I did is I said, well, I'm not really interested in the IPs. 28:44.630 --> 28:57.510 I know everything is coming from this one IP that I'm interested in, so it only showed me the traffic from that guy, but I'm interested in the destination ports, the TCP flags, and the TTLs. 28:57.510 --> 29:05.930 So I wanted to see, like, the SYNs, the SYNX, the pushes and whatever, and their TTLs, and what port they went to. 29:06.530 --> 29:14.130 So this yields this graph here, and in addition to that, that was not quite enough to figure out what's really going on. 29:14.130 --> 29:15.690 I need to have the counts. 29:15.730 --> 29:17.010 I'm going to lose everyone here. 29:17.010 --> 29:23.510 I need the counts on all these nodes, so how many times did certain flags show up? 29:23.790 --> 29:26.090 So let's take one example here. 29:26.090 --> 29:29.770 I have port 80 here. 29:29.790 --> 29:37.150 It showed up 2,600-and-something times, and these are all the flags that showed up with the different TTLs. 29:37.150 --> 29:50.810 So there were four TTLs, one, two, three, four, that were used when someone connected these ports here, and here are the TCP flags that were being used for this traffic. 29:51.270 --> 29:52.510 So I had a theory. 29:52.510 --> 29:59.530 I said, well, I'm looking at all the connection establishments, all the SYNs, and I want to see how many times they happen. 29:59.610 --> 30:06.450 The score about traffic is probably going to all the services, like in sequence, or probably the same amount of time. 30:06.450 --> 30:18.730 So I will see the same amount of SYNs going to each of the destination ports, and if the theory about the TTLs is right, they all use the same TTL. 30:19.350 --> 30:26.570 And it turned out that we were actually able to figure out the TTLs, and for the game last year that way, and we filtered all the other traffic. 30:26.590 --> 30:29.530 But it was a little late in the game, so we still didn't win. 30:33.080 --> 30:36.560 All right, so I have some other logs here. 30:37.420 --> 30:49.860 At some point I was interested in email logs, and I wrote a little send mail parser, and that's actually part of the afterglow scripts that you can have on the CD or you can get from SourceForge. 30:50.220 --> 30:55.010 The problem in send mail, I'll mention that in the end I guess, there are always two entries. 30:55.110 --> 31:07.170 It says from so-and-so, and then there's a message ID, and then the next log entry says to so-and-so, and there's a message ID to link those two events together, or just two log entries. 31:07.190 --> 31:15.330 So I had to write a tool to actually take them, to put them together, and then output a CSV file, and all that stuff, and the result is this one here. 31:15.330 --> 31:19.030 So I'm just showing who is talking to whom by email. 31:19.370 --> 31:30.790 And then I said, well, what's probably interesting is to see everything that's coming from my domain, the one that I'm really hosting on that server, going to people outside and what is coming into my domain. 31:30.790 --> 31:39.590 So I colored the whole graph that way, and if we look at a zoom up here, you see that these people here are quite popular. 31:40.330 --> 31:46.950 A lot of people from outside, these red nodes, are sending emails to this guy that was hosting on this server. 31:47.750 --> 31:49.750 Here you just see the popular people. 31:49.990 --> 31:50.730 It might be interesting. 31:50.730 --> 31:51.050 I don't know. 31:51.050 --> 31:52.250 Maybe you want to get to know them. 31:53.090 --> 31:57.270 But I was really interested in whether I'm running an open relay. 31:58.250 --> 32:00.530 I'm not, but I'm pretending I will. 32:01.750 --> 32:10.950 So what I did is I said, well, just show me everything on the outside. 32:10.950 --> 32:20.050 All my domains or all the, like, rafi.ch or whatever my domain is, don't grade them out and just show me the rest. 32:20.050 --> 32:24.130 So you see there's still some others showing up here. 32:24.210 --> 32:30.250 So what I did then was I used afterglow to just eliminate all those. 32:30.250 --> 32:31.130 I'll show you how to do that. 32:31.130 --> 32:32.970 You can set nodes and invisible. 32:33.330 --> 32:45.130 And here I realized, oh, my God, there are actually people using my server, like this guy here, to send emails to people that are not hosted on my server. 32:45.670 --> 32:48.910 So I'm probably running an open relay. 32:51.610 --> 32:57.190 Well, some other things I was interested in, well, am I running, or is anyone spamming? 32:57.210 --> 32:59.770 And , well, probably, hopefully I see something in here. 32:59.770 --> 33:06.710 Otherwise my graph would probably not be very, my configuration would not be very good because probably on every mail server you'll find spam. 33:07.270 --> 33:11.470 So what I did is in the email log you see the size of the messages. 33:11.470 --> 33:15.530 So I just visualized who is getting what sizes of messages. 33:16.010 --> 33:21.110 And I was interested in just the ones that are above 10,000, all the others I'm not interested in. 33:21.110 --> 33:26.970 So you still see that, for example, down here there are multiple recipients that get the same size messages. 33:26.970 --> 33:41.110 So that means that someone probably is sending emails to a few people and this email is pretty big and it's probably the same one because they have quite a high size, or the same size. 33:42.290 --> 33:46.990 So that would be one indication of maybe there's something spam going on. 33:46.990 --> 33:48.170 Probably not. 33:48.230 --> 33:55.010 Well, another way of looking at whether you have spam is look at the number of recipients in an email. 33:55.010 --> 33:57.330 You can send emails to multiple people, right? 33:57.330 --> 34:02.070 So the number of recipients in the log file would indicate how many people got this email. 34:02.070 --> 34:04.710 And what I said, well, I want to see everything that's bigger than two. 34:05.070 --> 34:11.690 And all the things where someone is just sending to two other people, I don't want to see. 34:11.690 --> 34:20.390 So I have this omit threshold thing in Afterglow, I will explain it in a second, where I can drop out nodes that only connect to one other or two others. 34:20.390 --> 34:23.990 So I can filter out the things that are not very interesting. 34:23.990 --> 34:28.290 I'm interested in the bigger, the fans or the circles showing up. 34:28.430 --> 34:32.770 So when I did that I realized, well, only this one comes back here. 34:32.770 --> 34:35.750 There are, like, ten recipients and seven and eleven. 34:36.010 --> 34:39.210 And it's always from this free cloud report at news.ch. 34:39.210 --> 34:41.250 So it was probably a newsletter. 34:41.250 --> 34:43.530 And maybe it was spam. 34:43.870 --> 34:48.130 You can take this further and further. 34:48.190 --> 34:51.890 You can have very big, you look at very big emails. 34:52.050 --> 34:56.550 Maybe someone is sending out documents from the internal network to someone outside. 34:56.850 --> 35:01.230 I'm going to skip over these a little quicker to show you Afterglow. 35:02.710 --> 35:08.590 What an interesting, another one is there's a field called delay on the email server log. 35:08.850 --> 35:14.370 And that shows you how long was the email queued on the mail gateway. 35:14.710 --> 35:19.950 And what you see here is that there was apparently some problem here with these big two clouds. 35:20.450 --> 35:22.450 This one here and this one down here. 35:22.450 --> 35:25.370 These emails stayed on the server for more than ten minutes. 35:25.910 --> 35:31.530 And it happened a lot of times, as you can see, to these two email addresses. 35:31.810 --> 35:34.110 This one here and this one down here. 35:34.110 --> 35:35.830 So there's some problem with those addresses. 35:35.830 --> 35:36.970 Maybe they're wrong. 35:37.650 --> 35:45.750 And you can go in and either block them or do something about them or figure out who was sending the emails out to these guys. 35:45.790 --> 35:55.450 And this might actually be also a way to detect spammers because they send these huge lists of emails and probably a lot of them don't exist and they will be queued on system. 35:55.450 --> 35:58.430 They might retry all the time, so getting a lot of delay. 35:58.470 --> 36:00.990 So maybe you can figure out who your spammers are. 36:01.150 --> 36:03.410 Okay, let's get to the gist here, to Afterglow. 36:05.090 --> 36:06.950 You find it on SourceForge. 36:07.370 --> 36:12.010 As I mentioned earlier already, I support two output formats. 36:12.590 --> 36:19.790 The GraphList library from AT&T, also known as .neto, and the LGL, the Large Graph Layout by Alex Adai. 36:24.510 --> 36:27.250 If you, so Afterglow is just a Perl script. 36:27.250 --> 36:31.470 You can pass it the input and read some standard in. 36:31.470 --> 36:33.970 You can pass it a few parameters, figure it. 36:35.270 --> 36:40.770 One of them is if you do a minus T on it, it expects a two-column input. 36:40.770 --> 36:46.110 So just from two or size and recipient or something like that. 36:46.110 --> 36:48.890 So minus T reads two columns. 36:49.190 --> 36:51.470 If you don't do the minus T, it expects three columns. 36:51.470 --> 36:55.890 So you can do all these like source IP destination, IP destination port things. 36:56.610 --> 37:00.450 If you do a minus D, it will print the count on the nodes. 37:00.450 --> 37:08.250 That was that weird example from Capture the Flag I showed you, that you actually see how many times did that node show up in the log file. 37:08.590 --> 37:12.550 Then you can pass it an edge length to kind of optimize how your graph is going to look. 37:12.550 --> 37:15.830 If you have a length of five, then things would be really distant from each other. 37:15.830 --> 37:19.170 But if you have big clouds around one of the nodes, that might be helpful. 37:20.610 --> 37:24.370 With minus N, you can disable the node labels. 37:25.070 --> 37:29.870 So sometimes you have logs, if you can't parse them really well, you have really long tokens. 37:29.870 --> 37:32.870 So they will really clutter your graphs. 37:32.870 --> 37:37.090 They just turn them off and you get kind of a feeling that there are certain things going on. 37:37.090 --> 37:40.030 And then you have to go back to the log file and figure out what those nodes were. 37:41.110 --> 37:57.030 This minus O thing, this threshold that you can set, if you have a log file of communications like someone is talking to someone else and you see a lot of people that just connect to one other machine and nothing else. 37:57.030 --> 37:59.890 And then another machine to something totally different. 37:59.990 --> 38:01.890 But then you have these big clouds. 38:01.890 --> 38:05.410 A lot of times, I'm not interested in these one-to-one connections. 38:05.410 --> 38:08.790 I really don't want to know about those, but I'm interested in the big clouds. 38:08.970 --> 38:13.430 So if you do a minus O one, it would kill everything that has a fan out of one. 38:14.030 --> 38:17.270 And this feature is not quite complete. 38:17.270 --> 38:22.410 It doesn't do the whole traversing of the clusters and figure out whether it's connected to other clusters. 38:22.410 --> 38:23.110 Play with it. 38:23.110 --> 38:24.090 It's kind of cool. 38:24.150 --> 38:27.410 There might be like one-off cases that look kind of weird. 38:27.950 --> 38:30.490 And then the most important thing is the minus C. 38:30.490 --> 38:33.070 You can pass it a color property file. 38:33.070 --> 38:36.570 I'm going to show this in a second how this works. 38:37.070 --> 38:43.550 So the property file just has color dot, for example, source equals, and then it can pass it a Perl expression. 38:43.550 --> 38:48.570 And the expression just has to return a color, like red, this string. 38:50.010 --> 39:00.950 Then there is an array called fields that contains the columns that you passed it in. 39:00.950 --> 39:04.350 So for every line, it will evaluate these array. 39:04.490 --> 39:17.690 So you can say, like, the event node is red if the first column, so that's really the second column, if fields one matches this regular expression, so it's in 192, then make it red. 39:18.570 --> 39:21.590 Then I have the special color called invisible. 39:21.590 --> 39:28.630 If you say that the target is invisible if this expression here matches, then it won't draw that node. 39:28.630 --> 39:32.190 So you can kick out certain things by certain expressions. 39:33.530 --> 39:35.350 Or you just have a simple color. 39:35.350 --> 39:37.630 Just say color dot edge equals blue. 39:38.210 --> 39:42.170 So here is the fancy example of the file graph. 39:42.170 --> 39:43.630 I'm just very quick. 39:43.630 --> 39:44.910 So this is the property file. 39:44.910 --> 39:58.510 You say the source is olive drab if the field zero, so if the first entry that you get is 192.141.69.4, that's one of my IP addresses. 39:58.630 --> 40:01.230 It's not really, so go and play with it. 40:02.350 --> 40:04.330 So they show up green. 40:04.950 --> 40:09.230 Then everything else, like first match wins. 40:09.230 --> 40:12.150 So if it falls through, it will just be orange-red. 40:12.150 --> 40:14.330 So everything outside is going to be red. 40:14.470 --> 40:17.990 Then you configure the nodes to be blue, the targets. 40:18.410 --> 40:20.090 Like, I think you get the idea here. 40:20.090 --> 40:36.290 You can have really complicated things on the edges, for example, for ingoing and outgoing traffic, I have to say if the first column is inside my network or the third column, which is the destination IP, is inside my network, then give it a fire break color. 40:36.850 --> 40:39.310 And otherwise just use the other color. 40:39.910 --> 40:43.870 Afterglow also has two parses I'll give you. 40:44.110 --> 40:47.430 And I just realized, I just put this slide in last night at like 2.30. 40:47.630 --> 40:49.370 I hope there are no typos. 40:50.690 --> 40:53.950 I realized that these two parses are actually quite useful. 40:53.950 --> 40:56.050 I used them at Capture the Flag yesterday. 40:56.270 --> 41:00.310 If you have a TCP down block file and you want to visualize that, you have one problem. 41:00.310 --> 41:04.370 You see requests from a high port to port 80. 41:04.490 --> 41:08.370 Then you see the response going from port 80 to a high port. 41:08.370 --> 41:13.470 If you visualize that, you will see the high port and port 80 as source ports. 41:13.550 --> 41:16.490 And that was what I was showing in the other graph, basically. 41:16.490 --> 41:19.110 And then the destination port, again, same port numbers. 41:19.110 --> 41:22.790 So you need to, for the request, you need to flip the ports. 41:23.790 --> 41:25.590 That's so that you have the connections again. 41:25.590 --> 41:29.310 And this, the TCP down to CSV takes care of that. 41:29.310 --> 41:33.850 It remembers what direction the connections are going in and automatically flips them. 41:33.850 --> 41:34.990 So you get nice output. 41:35.090 --> 41:41.210 And then the send mail parser combines the send mail log entries into one again that you need. 41:41.850 --> 41:43.290 That's really it. 41:43.790 --> 41:45.270 Any questions? 41:47.960 --> 41:49.540 Wow, everyone is tired. 41:49.540 --> 41:50.120 Yeah? 42:02.030 --> 42:09.710 So the question is whether I'm thinking about using, like, a database to drive or to read from or something. 42:09.710 --> 42:12.850 Well, AfterGrowth is really just like a Unix script, right? 42:12.850 --> 42:16.470 You use pipes and you pipe in data and you get data out. 42:16.630 --> 42:19.850 So it's pretty easy to use a SQL database. 42:19.850 --> 42:22.650 For example, you use the TCP down to CSV. 42:22.650 --> 42:33.650 You pipe that into MySQL minus E, insert into blah, blah, and you get into database and then you can have it piped into something else and you read out the stuff from the database. 42:33.650 --> 42:36.350 So you plug it basically into whatever you want it. 42:36.770 --> 42:48.370 We basically just provided this AfterGlow to take the CSV, to add the color and then everything to that graph file and generate the graph file and then you use the grapher to generate the graph. 42:48.990 --> 42:52.930 But please feel free to contribute anything to the SourceForest project. 42:54.010 --> 42:58.470 And also if you're going ahead, go home and visualize and get some cool graphs. 42:58.470 --> 43:02.610 I'm always interested in getting graphs, so spam me with cool pictures. 43:07.240 --> 43:09.000 How fast is it? 43:09.820 --> 43:13.000 Well, most of the time it's actually consumed by Nido. 43:13.000 --> 43:24.240 That's what I use to generate all these graphs here because it has to generate the layout and it really depends on how many nodes or how many log entries you're passing in. 43:25.860 --> 43:31.780 It's, if you have probably about, let's say we want to have a refresh rate of like 30 seconds. 43:31.780 --> 43:35.740 I think that's good because you need time to look at it and then it's going to change completely again. 43:35.740 --> 43:36.800 That's going to be one of the problems. 43:36.800 --> 43:38.900 You don't see nice delta updates. 43:38.900 --> 43:40.400 You would have to handle that somehow. 43:40.480 --> 43:43.600 Might be something that maybe Greg and I could work on. 43:47.160 --> 43:55.340 So if you're 30 seconds, it's probably safe to use up to probably 5,000 events in 30 seconds. 43:55.340 --> 43:56.060 That should probably work. 43:56.060 --> 44:00.380 I think I did something like that yesterday with the capture the flag data. 44:01.880 --> 44:03.300 I have two more books. 44:04.640 --> 44:08.340 I guess I give them to the two gentlemen that asked the questions here. 44:09.660 --> 44:11.020 Anyone else? 44:13.160 --> 44:14.820 All right, thank you very much.