[00:01.690 --> 00:06.210] So let's finally talk about visual security event analysis. [00:11.370 --> 00:17.870] So what I'm going to talk about is how you analyze log files or events that you have [00:18.770 --> 00:27.070] in a visual manner. I don't want to talk about myself too much. Maybe one thing. I've been [00:27.070 --> 00:33.310] involved in, like, log analysis and event analysis for quite a while, and it has always [00:33.310 --> 00:38.030] interested me whether there's a way to look at the data in a way that it really, like, [00:38.210 --> 00:43.870] a lot of data, in a way that it's really apparent what is going on in that dataset. And if you're [00:43.870 --> 00:47.850] looking at log files, thousands of lines of logs, it's really hard to figure out at one [00:47.850 --> 00:51.610] glance what is going on. You have to start reading through them. You have to, like, use [00:51.610 --> 00:55.890] your own mind to do pattern analysis and figure out what is really going on. [00:55.890 --> 01:01.290] So at some point, I started working for this company that I'm working for right now called [01:01.290 --> 01:08.910] ArcSight, and I got a hand on a lot of neat tools to look at data, and I got a feature [01:08.910 --> 01:13.910] in the software that is called Event Graphs, and I really liked it, and I started looking [01:13.910 --> 01:18.450] into how can we use this feature to make it more easy for a security analyst or a person [01:18.450 --> 01:22.330] interested in a log file to figure out what is really going on in that dataset. [01:22.330 --> 01:28.770] So that's how kind of my passion about Event Graphs started. My talk here is not going [01:28.770 --> 01:32.810] to be very textual or, like, theoretical, and I'm not going to talk too much about how [01:32.810 --> 01:36.850] you generate the graphs and whatever. I want to show you a lot of examples on how to look [01:36.850 --> 01:40.990] at datasets and how you configure graphs to get the most out of it. [01:41.510 --> 01:47.170] As part of the talk, I'm going to show you a tool that I have up on SourceForge, which [01:47.170 --> 01:53.610] is called Afterglow, which is a very simple Perl script that transforms CSV files into [01:54.070 --> 01:58.390] a graph or language file, so you can go home and play with these things yourself, and the [01:58.390 --> 02:00.010] tool is actually also on the CD. [02:02.570 --> 02:06.510] I'm going a little faster in the beginning so we can catch up for the time I lost already. [02:06.510 --> 02:10.230] One thing, IP addresses and host names, you're going to see in all these graphs, and the [02:10.230 --> 02:14.330] descriptions are obfuscated or changed. The addresses are completely random and any resemblance [02:14.330 --> 02:17.410] with well-known addresses or host names are purely coincidental. [02:18.510 --> 02:23.270] All right. So let's do a little bit of an introduction here. Why would you want to look [02:23.270 --> 02:26.250] at these graphs? I guess this is a pretty good example already. Do you want to look [02:26.250 --> 02:29.330] at the left-hand side, or do you want to look at the right-hand side? [02:32.670 --> 02:36.250] Well, even if I made the font bigger on the left-hand side, it's probably still not very [02:36.250 --> 02:36.850] fun. [02:37.330 --> 02:43.090] All right. Why do you want to use event graphs? I guess I already motivated that quite well. [02:43.090 --> 02:47.270] I'd rather look at the graphs. They look just much nicer. [02:48.520 --> 02:52.030] If you actually dig into it a little more and you start looking how to configure these [02:52.030 --> 02:58.690] things, you will see pretty quickly that you can visualize different properties in those [02:58.690 --> 03:03.510] graphs, and it's very easy to see certain aspects of a log file very quickly and certain [03:03.510 --> 03:08.430] things that are important to you. You don't have to wade through ten thousands of lines [03:08.430 --> 03:13.690] of logs to figure out whether there is anyone sending a lot of e-mails to someone else. [03:14.590 --> 03:23.990] If you use this in a security operating center or in a 724 response team or something, if [03:23.990 --> 03:30.090] you have graphs, they're a great way of giving you situational awareness or real-time view [03:30.090 --> 03:33.310] on things that are happening, and you can quickly react on them. [03:33.690 --> 03:37.850] We have a lot of customers that are starting to use this in their SOCs, and it doesn't [03:37.850 --> 03:42.370] just look pretty on their big screens, but it's actually really useful. It's actually [03:42.370 --> 03:48.330] also kind of an interesting story. I was at one customer, and they had all these big displays, [03:48.330 --> 03:51.950] and they were empty, and I was like, wow, why do you have these nice graphs here on [03:51.950 --> 03:55.590] the analyst screens? Why don't you have them up there on the big screens? They were like, [03:55.590 --> 03:59.870] whoa, no, no, no, we tried that for a while, but then the boss came in, and he saw all [03:59.870 --> 04:02.570] these things on the screen. He was like, oh, what is this? What's that? What's this? What's [04:02.570 --> 04:04.950] that? So we had to explain all this. We took them off. [04:08.140 --> 04:12.040] All right, and one other thing, another story that I had with these graphs, which actually [04:12.040 --> 04:17.440] two other stories. One is, one customer uses these graphs to communicate with other teams, [04:17.440 --> 04:20.780] so you have the security analysts. They look at the log files and all that, and then they [04:20.780 --> 04:25.700] go back to the operators or the operational folks, and they're like, guys, you have to [04:25.700 --> 04:30.180] do something about this machine. It's talking to all the others in the network. And in the [04:30.180 --> 04:34.220] past, they were handing them log files to prove that it's that way, and the operators [04:34.220 --> 04:38.020] were like, whatever, we don't understand what this is. But now they started giving [04:38.020 --> 04:42.120] them graphs, and the operators were like, oh, my God, my machine is talking to all these? [04:42.120 --> 04:47.180] Okay, we're going to take care of this. And even worse, I had one customer that actually [04:47.180 --> 04:53.520] just told me this week that he used the graphs to analyze some data, and he found some stuff, [04:53.520 --> 04:58.900] and one of the guys got fired just because he used the graphs and found out that one [04:58.900 --> 05:07.430] of the guys was not doing his work. All right, so I came up with, like, two big categories [05:07.430 --> 05:11.950] of when to use these event graphs. One is in a real-time monitoring scenario, where [05:11.950 --> 05:16.170] you have, like, a screen app that looks real-time at certain properties of your data file, and [05:16.170 --> 05:19.930] I'm going to show you some examples on what I mean by that. It gives you great awareness [05:19.930 --> 05:24.770] of things that are happening. You can configure these things to, for example, look at transactions [05:24.770 --> 05:29.110] going to your financial server. So right away, if someone connects to your financial service, [05:29.110 --> 05:32.890] you will know. And then you can take it a level further. You can say, well, I know that [05:32.890 --> 05:36.950] these machines are okay to connect to my financial servers, so don't show me those, but show [05:36.950 --> 05:40.470] me everything else. So as soon as something pops up there, you know you've got to look [05:40.470 --> 05:46.710] into it. The other thing is, if you take this in a more forensic or historical way, you [05:46.710 --> 05:50.770] can, you grab a data set, you run it through some tools, and you look at the graphs on [05:50.770 --> 05:56.190] what is, on certain properties, and you figure out what's going on. Let me mention quickly [05:56.190 --> 06:02.890] some related work. There's some classics from, he did all these kind of things. He tried [06:02.890 --> 06:08.430] to figure out whether there are certain ways to visualize properties of data, and he took [06:08.650 --> 06:13.930] a lot of dimensions and put them all in one graph. What I'm going to do is, I take a lot [06:13.930 --> 06:18.390] of dimensions too, and I'm always going to stay with event graphs, so like node-to-node [06:18.790 --> 06:24.710] connections. I'm not going to use any of the parallel coordinates or the right-hand-to-bottom [06:24.710 --> 06:27.610] thing that I have no idea what it shows there. [06:28.310 --> 06:33.890] Then Ehrbacher, he also, he actually went even further. He tried to go even more multidimensional [06:33.890 --> 06:39.650] and map everything into these nodes, so there's like size, there's shape, there's length of [06:39.650 --> 06:46.110] these spirals going out, and so on. Then Greg Conti, you probably have seen his talk, I [06:46.110 --> 06:54.690] hope. He mainly focuses on different ways to visualize TCP dump or packet locks, and [06:54.710 --> 06:55.910] very interesting. [06:57.510 --> 07:02.430] Then Envision IP is something I found. What you see up there is a grid of, I think it's [07:03.190 --> 07:09.710] 8x8 or 64x64, and they just map IP addresses in there, so you see certain clusters of things [07:09.710 --> 07:14.010] going on. The nice thing about this project is, there's a lot of interactivity, so you [07:14.010 --> 07:16.630] can start playing with the data, you can zoom in and so on. [07:17.030 --> 07:22.810] And then finally, a few of you might know Showkey from Steve Barry. It's basically a [07:22.810 --> 07:28.530] scatterplot. You can define three different scatterplots, and then you have a 3D of that. [07:28.810 --> 07:33.230] So on the top right, it's not a very nice picture, but on the top right it's a three-dimensional [07:33.230 --> 07:38.850] output, so you can say one axis is your source IP, one axis is destination IP, and the other [07:38.850 --> 07:40.690] one is the destination port. [07:41.630 --> 07:47.370] All right, let's move on. How do you generate an event graph? Pretty easy. You have a log [07:47.370 --> 07:54.250] file that a device is generating. You take a parser. Why do you take a parser? Well, [07:54.250 --> 07:59.430] you need to know where in my log event or my log file, where's the source IP, where's [07:59.430 --> 08:02.910] the destination IP, where's the event name, where's the port? So you parse that, you get [08:02.910 --> 08:07.550] the syntax right, and then you visualize that in an event analyzer. [08:08.250 --> 08:14.450] If you have an event, let's look at this Nord event here. You don't have to be really able [08:14.450 --> 08:18.790] to read this, but what I want to show here is in an event graph you can have different [08:19.750 --> 08:23.670] configurations. The normal event graph that you will always see is like who is talking [08:23.670 --> 08:28.950] to whom, so you have source IP, destination IP. But you can take this a lot further. You [08:28.950 --> 08:35.210] can say, well, on the right top you want to have the source IP, destination IP, and destination [08:35.210 --> 08:39.950] port, so you will see someone port scanning, for example, if there's a fan with destination [08:39.950 --> 08:44.370] ports. So you can configure these things in pretty wild ways, and I'm going to show you [08:44.370 --> 08:45.290] some of them. [08:46.990 --> 08:53.650] Now let's have a sneak preview of Afterglow. I apologize because I don't use my laptop [08:53.650 --> 08:59.890] running it. I can't show you the demo. It's not very fun anyways. I will show you the [08:59.890 --> 09:05.350] output anyway, so it doesn't really matter. So what is Afterglow? Afterglow is not a security [09:05.350 --> 09:10.850] information management tool. It's just a script which takes CSV files, so you have [09:10.850 --> 09:17.310] to somehow from your logs generate a CSV file, comma, separated values. And then you take [09:17.310 --> 09:22.170] Afterglow, the tool that I'm going to show you, and you transform the CSV files into [09:22.390 --> 09:28.110] a graph language file. And then you take a grapher tool, and right now I'm supporting [09:28.110 --> 09:33.710] two of them, that takes this graph language and generates the graph. And that's not something [09:33.710 --> 09:37.950] I did. I just transformed the CSVs into graph files, but I'm going to show you some cool [09:37.950 --> 09:40.130] things that we implemented there. [09:41.490 --> 09:46.410] So how you can use this thing is you basically cap the input file, you pipe it to Afterglow, [09:46.410 --> 09:49.790] you give it a color properties file, which I'm going to introduce a little later, and [09:49.790 --> 09:55.830] then you can use the AT&T Graphverse library, .neto, I'll give you references in a second, [09:55.830 --> 10:01.990] and you generate a GIF file. The color properties file is basically just you tell it the source [10:01.990 --> 10:06.830] nodes are red, for example, the event nodes are green, and the target nodes are blue. [10:06.830 --> 10:10.350] And I'm going to show you how to really play with this stuff. We extended this quite a [10:10.350 --> 10:17.690] bit so you can have colors depending on the values in the log files. [10:17.690 --> 10:23.530] Big shout out to Christian. Thanks a lot for building the foundation of this tool. He built [10:23.530 --> 10:29.690] me the whole Afterglow profile, the basics and everything, and I just went in and made [10:29.690 --> 10:38.710] some small changes, but thanks a lot, Chris. Let's look at examples. I mentioned in the [10:38.710 --> 10:42.950] beginning I kind of have two use cases or two big categories of things to look at. One [10:42.950 --> 10:53.120] is what I call situational awareness or real-time monitoring. So this is one of the views that [10:53.740 --> 11:00.200] I saw at a customer. Basically you see here is six independent screens of different properties [11:00.200 --> 11:06.240] of your data that you want to look at. So here the configuration is such that on the [11:06.240 --> 11:12.560] top left the customer wanted to see just the compromise attempts, and don't ask me how [11:12.560 --> 11:17.980] to filter for just those events. That's a whole different story. But if you have a way [11:17.980 --> 11:21.900] to say these are my malware events, these are my DOS events, these are my scan events, [11:21.900 --> 11:26.280] if you can separate them by something, then you can draw all these screens and you can [11:26.280 --> 11:30.280] look at these screens and see what's going on. [11:30.500 --> 11:34.900] So what this customer is actually also doing, whenever an analyst starts his shift, he resets [11:34.900 --> 11:40.940] all the graphs so they're empty. So over time, nodes start popping up and they look at them [11:40.940 --> 11:44.800] and they're like, okay, this is fine, and they might filter them out even, or they just [11:44.800 --> 11:49.140] leave them there and say, okay, I know about this. And the more things that show up, the [11:49.140 --> 11:53.680] more they go along in their workflow and work with these events. So I have a feeling this [11:53.680 --> 12:00.200] is going to be a great tool, especially in a SOC or an analysis environment, to kick [12:00.200 --> 12:03.340] off your workflow. You look at these things and you take it on and you start building [12:03.340 --> 12:08.120] filters to filter out the things you don't know about, and you just see the outliers [12:08.120 --> 12:09.440] here in these screens. [12:11.300 --> 12:19.700] The second big group of use cases is forensic analysis or historical analysis, where you [12:19.700 --> 12:27.100] take a log file that you have and you visualize it. Afterglow can generate two kinds of outputs. [12:27.100 --> 12:33.900] One is for the AT&T graphics library for .neto, and the other one is for LGL, the large graph [12:33.900 --> 12:39.180] library. I will give a pointer to that in the end as well. This thing is generating [12:39.180 --> 12:42.980] three-dimensional graphs. You can actually generate VRML output and you can fly through [12:42.980 --> 12:49.080] these things. I didn't prepare a demo because it's kind of fun to fly through it, but the [12:49.080 --> 12:53.660] analysis value is not really high or the value you get out of it. It's really cool [12:53.660 --> 12:58.120] to fly through it and see that, oh, this guy sent an email. To whom? Oh, over there. Okay. [12:59.060 --> 13:04.640] It looks great, and I apologize for the colors here, but this is really how it looks. Anyway, [13:04.640 --> 13:05.900] let's go back to 2D. [13:07.000 --> 13:13.180] This is a very simple example, one of about 12 that I'm going to show, of how to analyze [13:13.480 --> 13:22.620] a log file. So what I had here is, I believe, a snort log or a web log or something, and [13:23.180 --> 13:29.500] I said, well, show me what is happening. So the event name, the destination IP are the [13:29.500 --> 13:33.640] blue ones, and then the destination ports are the white nodes. So what you see here, [13:33.640 --> 13:40.480] all the blue circles are my web servers. The white boxes are the target ports that they're [13:40.480 --> 13:45.280] being connected to. So you see it's 80 and 443, so that's totally legitimate. Only HTTP [13:45.280 --> 13:49.640] and HTTPS traffic is going to my servers, so I probably don't have to worry about that. [13:49.640 --> 13:54.260] If I would see a node there, like port 21, and someone connects to my web servers with [13:54.260 --> 13:59.800] FTP, maybe that's okay, but if I'm not running FTP in my environment and that shows up here, [13:59.800 --> 14:05.960] I probably want to investigate that. So it's a very quick way to see whether traffic is [14:05.960 --> 14:07.600] really what it's supposed to be. [14:07.600 --> 14:14.720] Almost same configuration. Here you see source IP, destination IP, destination port. So this [14:14.720 --> 14:21.420] would potentially show you port scans or network scans. Here you see a lot of machines connecting [14:21.420 --> 14:28.860] to a lot of other ones, all on the same port. Down here, this white node here, is the port. [14:28.860 --> 14:35.720] So they all connect to the same port. Well, it could be that a worm is probing you, but [14:35.720 --> 14:40.200] it could also be that you have a web server, or a lot of web servers, and a lot of machines [14:40.200 --> 14:45.980] are connecting to those. So it might be okay, but it might be a scan, and depending on what [14:45.980 --> 14:50.780] the destination port is, you might figure out what's going on in this case. But you [14:50.780 --> 14:55.920] can certainly tell, well, this is all similar behavior here on this fan, and then actually [14:55.920 --> 15:01.820] if you see on the bottom left here, there's some little ones, it's probably worth investigating [15:01.820 --> 15:06.760] what those are, and probably those are the outliers that you are interested in. [15:09.200 --> 15:12.860] Let's keep this configuration of the graph. This is just a little bit of a different dataset [15:12.860 --> 15:22.620] and a different time. It looks much prettier. Let's zoom in on the bottom part here. What [15:22.620 --> 15:26.360] this is showing you is that there are two machines, the red nodes, that are connecting [15:26.360 --> 15:32.000] to about 15 other ones, and still on the same port. So again, in your environment, do you [15:32.000 --> 15:38.400] really have all these other 15 or 20 machines serving this one service, and exactly two [15:38.400 --> 15:42.240] machines are connecting to them? So you can go and figure out what all these machines [15:42.240 --> 15:46.480] are, and hopefully they are really the ones that are serving this service, and then you [15:46.480 --> 15:49.500] can figure out whether these other two are really allowed to connect to this. [15:50.060 --> 15:57.580] Let's have a look at this weird thing over here. What you see here is, probably you can't [15:57.580 --> 16:05.520] read this, but there is a node here that says 21 FTP, so one of the properties of this graph [16:05.520 --> 16:10.560] is that the white nodes show up exactly once per target port, so everything that happens [16:10.560 --> 16:16.740] on port 21 is going to show up around that thing there. So again, I can say, wow, I have [16:16.740 --> 16:24.400] about six FTP servers, and about five people are connecting to them. So the FTP servers [16:24.400 --> 16:27.740] would be the blue nodes, and then the red ones that connect to them would be the people [16:27.740 --> 16:32.380] using it. So, well, is this really something I have in my environment? Is this allowed? [16:32.380 --> 16:38.440] And why are only three or five people connecting to them? Is that some kind of a process that, [16:38.440 --> 16:44.260] I don't know, maybe AIX that transfers log files over, something weird like that? Or [16:44.260 --> 16:47.260] is this something malicious? Is anyone having a ware server or whatever? [16:48.980 --> 16:54.000] If you interpret these graphs, you have to be a little careful. I still have the same [16:54.000 --> 16:57.640] configuration. Destination IPs are the blue ones, and the white ones are the destination [16:57.640 --> 17:03.840] ports. Well, if you look at this on the left-hand side, I would say, wow, it's a port scan. [17:04.360 --> 17:13.070] Well, is it really a port scan? I have a book for the one that gets this right. What is [17:13.070 --> 17:24.840] the left-hand side? No, not voice over IP. Who said that? Over there. You got the book. [17:24.840 --> 17:34.200] It's the source ports. The log file reports such that the source port is really the destination [17:34.200 --> 17:39.300] port and the other way around. So when you analyze these things, be careful and make [17:39.300 --> 17:43.860] sure you're really looking at what you're looking at. So why do I know are the source [17:43.860 --> 17:50.100] ports and the destination ports? Well, they're in the 50,000 range. They're almost contiguous. [17:50.780 --> 17:55.980] And I mean, who is scanning me on these high ports? We're not a capture the flag. There, [17:55.980 --> 18:00.560] they might scan you on those high ports because services are running all over. But here, there's [18:00.560 --> 18:04.520] really no sense why someone would scan you up there. There's probably nothing there listening [18:04.520 --> 18:09.100] anyway. So be careful when you analyze these things. Sometimes it's port scan, sometimes [18:09.100 --> 18:14.640] it's not. I'm always, my first impression is always, well, false positive. Let's really [18:14.640 --> 18:20.930] figure out whether that's true. Okay, let's get fancy here. [18:21.810 --> 18:27.970] This is a firewall log. Basically, I'll show you passes and blocks from a PF firewall. [18:29.790 --> 18:37.170] If I would just visualize source IPs, destination IPs, I would see big clouds of things. Well, [18:37.170 --> 18:41.590] I want to see a little more in this graph. So what I did is I have source port, rule [18:41.590 --> 18:49.150] number, that's the squares, and then destination IP. So you will see I'm connecting to some [18:49.150 --> 18:54.250] other machine and rule number 100 is responsible for letting me in or blocking me. [18:55.070 --> 19:02.150] What I did then with Afterglow is to configure my nodes such that all the external IPs are [19:02.150 --> 19:06.610] going to show up as orange. I'm going to show you how to configure Afterglow to get exactly [19:06.610 --> 19:12.370] this graph. So the external machines are orange. My internal machines are green. And [19:12.370 --> 19:17.950] you see I only have a few of them. And then the rules are the blue boxes. And then further [19:17.950 --> 19:22.710] what I said, I wanted to see whether a connection is incoming or outgoing of my network. So [19:22.710 --> 19:27.370] I configured the edges such that outgoing connections are blue, that's the majority [19:27.370 --> 19:34.290] here, and incoming is red. And I just realized it's probably exactly the other way around. [19:34.290 --> 19:38.550] So I hope there's more incoming than outgoing because these are web servers that are servicing [19:38.550 --> 19:46.070] some stuff. Anyway, so you see big clouds of things that are showing similar behavior. [19:46.070 --> 19:49.910] So down here there's this one rule that is responsible for letting in a whole lot of [19:49.910 --> 19:56.270] traffic. And probably that's fine. There's another one up here. You can, if you know [19:56.270 --> 20:00.650] the rule set, if you know your ACLs, you can go and figure out whether it's really what [20:00.650 --> 20:07.250] it's supposed to do. But you can also figure out some things that are kind of weird. You [20:07.250 --> 20:11.830] have this whole section in the middle. You probably want to zoom in on those and filter [20:11.830 --> 20:15.870] out all the others and just look at these things and figure out whether your rule set [20:15.870 --> 20:21.130] is really doing what it's supposed to do. So the next steps would be, so the first is [20:21.130 --> 20:24.970] like overview, general impression, what is going on, is there anything really sticking [20:24.970 --> 20:30.150] out. And then my next steps would be, I would visualize the firewall blocks of outgoing [20:30.150 --> 20:35.730] traffic. Why would internal machines connect outside and be blocked? So that's either a [20:35.730 --> 20:41.750] misconfiguration of the internal box or the firewall rule set is wrong. Then the second [20:41.750 --> 20:45.330] one would be, I would probably look at blocks of incoming traffic. I want to know who's [20:45.330 --> 20:49.910] probing me, who's poking around. And if you have a live system that can look at these [20:49.910 --> 20:54.950] things, you can put all these attacker machines maybe on a blacklist and if you see them again [20:54.950 --> 20:59.210] and they're successful coming in, you might actually take some action and say, whoa, now [20:59.210 --> 21:03.010] someone probed me and now he's coming in. And then you can take this further. Third [21:03.010 --> 21:07.530] step would be to visualize the passes of outgoing traffic. Is this really what should be leaving [21:07.530 --> 21:12.450] the network? Maybe you find someone FTPing stuff out that you really don't want to FTP [21:12.450 --> 21:17.630] out. Maybe you realize that you're running peer-to-peer clients in your internal network [21:17.630 --> 21:22.390] and they're connecting out, anything like that. And you can probably take a few more [21:22.390 --> 21:25.810] steps to analyze the data set, but I guess you get the idea. [21:27.090 --> 21:33.470] So this one is probably the wackiest graph I have. It's totally going away from source [21:33.470 --> 21:38.710] IPs and destination IPs. What I did here is I wanted to debug my firewall rule set. So [21:38.710 --> 21:43.250] it's still the same data set as before, but this time what I show you is the red nodes [21:43.250 --> 21:49.630] are the rule number or the ACL number, then the destination ports, and then you see the [21:49.630 --> 21:55.250] action taken, pass or block. So just by virtue of firewall rule configurations, you will [21:55.250 --> 22:00.110] see two distinct graphs. One on the left-hand side is all the passes and the right-hand [22:00.110 --> 22:08.310] side is all the blocks. So here you can see, if you take one of these blue nodes, for example, [22:08.310 --> 22:13.710] port 53 here, you see that this one rule is responsible for letting traffic in. Well, [22:13.710 --> 22:20.270] most is okay, maybe, probably, hopefully, but you have other ones where there are two [22:20.790 --> 22:26.590] nodes or three nodes connected to the same rule. Like over here, for example, there are [22:26.590 --> 22:32.710] three ports being blocked by this one rule. It might be okay. It might not. If you see [22:32.710 --> 22:38.870] something like this on the other side, that one rule passes traffic from multiple ports, [22:38.870 --> 22:42.430] maybe that's the way you configured it, but this is certainly a great way of analyzing [22:42.430 --> 22:44.490] and debugging your firewall rule set. [22:46.630 --> 22:53.290] This is another example where I took a TCP dump log, and what I visualized here is I [22:53.290 --> 22:58.390] take the region of where the source IP resides, so you can go on the web, there are databases [22:58.390 --> 23:04.770] that show you what the location codes are for the IP addresses. They're not very accurate, [23:04.770 --> 23:09.430] but in a lot of cases, they're actually pretty good. So I show you the source region, where [23:09.430 --> 23:15.330] it's coming from in the world, then I show you the source MAC address. So this is a TCP [23:15.330 --> 23:20.770] dump log. The source MAC address is going to be the last hub, probably my router, that [23:20.770 --> 23:27.430] let the traffic into my network. And then I show the destination IP. So what I see here, [23:27.430 --> 23:33.370] I have four boxes, the four white ones, that are my internal machines, and interesting [23:33.370 --> 23:38.850] enough, I have two entry points into my network. I have two source MAC addresses showing up. [23:38.850 --> 23:45.690] And then I have these two fans in the middle part. What this is showing me is I probably [23:45.690 --> 23:52.070] either have a redundant router setup, or maybe I have a load balancer. And what is interesting [23:52.070 --> 23:57.770] here is that all these machines up here, they're connecting through this guy. All these guys [23:57.770 --> 24:01.450] here are coming through both of them, and all these down here come from the other one. [24:01.450 --> 24:05.510] So maybe this is a regional load balancer, that everything on one side of the world takes [24:05.510 --> 24:09.250] this entry point, everything on the other side is using that entry point. So if that's [24:09.250 --> 24:13.890] really the case, a great way to analyze this log is this graph here, and you can actually [24:13.890 --> 24:20.130] figure out why these two in the middle are coming in for both of them. And this is just [24:20.270 --> 24:26.830] a zoom, where you see the one server, and here it's like, you can't even read it below [24:26.830 --> 24:30.710] the node here, but it's Sand Hill or whatever in the United States. [24:35.510 --> 24:41.450] Graphs like these are a lot of times indications for warms, but again, be careful. There's [24:41.550 --> 24:48.250] a nice fan here of source IPs using the same event going to a few destination IPs. Well, [24:48.250 --> 24:52.210] it could be, again, it could be a warm that is using this one event to try to get into [24:52.310 --> 24:59.890] a network, but it might again be a web server serving just legitimate files. What you have [24:59.890 --> 25:03.270] to do in this case, you have to figure out what is this event name here that connects [25:03.270 --> 25:08.250] all these dots, and this might give you an indication of a warm or not. If you're visualizing [25:08.390 --> 25:13.510] a snort log file and this talks about warm, well, duh, it's probably a warm. But if you [25:13.510 --> 25:21.330] visualize a TCP dump log file, like an ICMP port unreachable might be an indicator for [25:21.470 --> 25:26.770] a warm trying to probe UDP ports and you get ICMP port unreachables back. So you have to [25:26.770 --> 25:29.330] investigate a little bit to figure out whether it's really a warm. [25:29.330 --> 25:36.930] I was playing Capture the Flag actually this year and last year, and I was trying to figure [25:36.930 --> 25:43.370] out the challenge really for the analysis team on Capture the Flag is that all the traffic [25:43.370 --> 25:49.530] hitting your web server or your servers seems to be coming from the same IP address, and [25:49.530 --> 25:55.710] there is malicious traffic of the other teams hacking you, but there is also benign traffic [25:55.710 --> 26:00.050] from the score bots trying to figure out whether your services are still up. So blocking [26:00.050 --> 26:04.490] all the traffic is not working. So you have to figure out a certain way of distinguishing [26:04.490 --> 26:11.270] these two types of traffic, and you can't do it by the source IP, which you would normally [26:11.270 --> 26:12.890] do in your corporate environment. [26:13.350 --> 26:19.930] So I started visualizing this log file of TCP dump, and I just wanted to have a general [26:19.930 --> 26:26.170] feeling for what's going on. So what you see here, the dark blue ovals are destination [26:26.170 --> 26:32.850] ports that are below 1024, normally services that are offered to the outside world. Then [26:32.850 --> 26:39.730] destination ports higher than 1024 are light blue, and then you see here what I just explained, [26:39.730 --> 26:44.830] the red node, this is what all the traffic that's coming in is coming from this one IP. [26:44.830 --> 26:53.770] And then I configured it such that our machines, the internal targets are green, other teams' [26:53.770 --> 26:59.450] targets are brown, and then internal sources are yellow, and internal targets are the boxes. [26:59.550 --> 27:05.510] So what you see here, we have two servers, while there's a third one showing up down [27:05.510 --> 27:11.970] here, this was actually not a server, but I think that was one of the machines on our [27:11.970 --> 27:16.310] network that someone connected to, and that could have been one of our team members connected [27:16.310 --> 27:25.830] to someone else's machine. And then the services that we offered were D6, and it was all kinds [27:25.830 --> 27:31.930] of weird things. I don't even remember anymore what it was last year, port 9999 or something, [27:31.930 --> 27:35.930] and there was some mud running there. But this gives you a very quick way of looking [27:35.930 --> 27:37.830] at what is happening in this network. [27:38.850 --> 27:44.590] Well, I took it a little further, and I had an idea, or a few people had an idea, I said, [27:44.590 --> 27:48.830] well, maybe it's a TTL of the traffic coming in. Maybe you can distinguish the good traffic [27:48.830 --> 27:53.490] from the bad traffic by looking at the TTL. So well, what I did is I said, okay, show [27:53.490 --> 27:58.670] me the source IPs, the destination IPs, and then the TTLs of that traffic. So you see [27:58.670 --> 28:05.910] here, I have eight TTLs showing up, eight distinct TTLs, and I'm going to look at the [28:05.930 --> 28:15.810] everything from 35, 50, 52, 60, 62, 63, 64, 128, and 146. Some of them are showing up [28:16.410 --> 28:22.090] if my teammates would connect out, so I could filter them out. I was not interested in those. [28:22.090 --> 28:27.510] I'm just interested in the ones that happen of the traffic coming in from this one evil [28:27.510 --> 28:32.150] IP. Let's go back quickly again. [28:32.150 --> 28:37.870] And then I figured out, well, I still can't quite tell what is okay and what's not, so [28:37.870 --> 28:42.010] I had to reconfigure my graph and look at some other properties. So what I did is I [28:42.010 --> 28:45.950] said, well, I'm not really interested in the IPs. I know everything is coming from this [28:45.950 --> 28:50.610] one IP that I'm interested in, so it only showed me the traffic from that guy, but I'm [28:50.610 --> 29:00.490] interested in the destination ports, the TCP flags, and the TTLs. So I wanted to see, like, [29:00.490 --> 29:05.930] the SYNs, the SYNX, the pushes and whatever, and their TTLs, and what port they went to. [29:06.530 --> 29:13.190] So this yields this graph here, and in addition to that, that was not quite enough to figure [29:13.190 --> 29:17.010] out what's really going on. I need to have the counts. I'm going to lose everyone here. [29:17.010 --> 29:24.650] I need the counts on all these nodes, so how many times did certain flags show up? So let's [29:24.650 --> 29:33.810] take one example here. I have port 80 here. It showed up 2,600-and-something times, and [29:33.810 --> 29:39.750] these are all the flags that showed up with the different TTLs. So there were four TTLs, [29:39.750 --> 29:46.210] one, two, three, four, that were used when someone connected these ports here, and here [29:46.210 --> 29:53.330] are the TCP flags that were being used for this traffic. So I had a theory. I said, well, [29:53.330 --> 29:58.570] I'm looking at all the connection establishments, all the SYNs, and I want to see how many times [29:58.570 --> 30:04.310] they happen. The score about traffic is probably going to all the services, like in sequence, [30:04.310 --> 30:10.930] or probably the same amount of time. So I will see the same amount of SYNs going to [30:10.930 --> 30:16.270] each of the destination ports, and if the theory about the TTLs is right, they all use [30:16.270 --> 30:23.310] the same TTL. And it turned out that we were actually able to figure out the TTLs, and [30:23.330 --> 30:27.110] for the game last year that way, and we filtered all the other traffic. But it was a little [30:27.110 --> 30:29.530] late in the game, so we still didn't win. [30:33.080 --> 30:40.100] All right, so I have some other logs here. At some point I was interested in email logs, [30:40.510 --> 30:46.160] and I wrote a little send mail parser, and that's actually part of the afterglow scripts [30:46.160 --> 30:51.570] that you can have on the CD or you can get from SourceForge. The problem in send mail, [30:51.570 --> 30:57.850] I'll mention that in the end I guess, there are always two entries. It says from so-and-so, [30:57.850 --> 31:02.870] and then there's a message ID, and then the next log entry says to so-and-so, and there's [31:02.970 --> 31:07.830] a message ID to link those two events together, or just two log entries. So I had to write [31:07.950 --> 31:12.750] a tool to actually take them, to put them together, and then output a CSV file, and [31:13.150 --> 31:15.330] all that stuff, and the result is this one here. [31:15.330 --> 31:21.410] So I'm just showing who is talking to whom by email. And then I said, well, what's probably [31:21.410 --> 31:25.930] interesting is to see everything that's coming from my domain, the one that I'm really hosting [31:25.930 --> 31:31.470] on that server, going to people outside and what is coming into my domain. So I colored [31:31.470 --> 31:37.290] the whole graph that way, and if we look at a zoom up here, you see that these people [31:37.290 --> 31:43.970] here are quite popular. A lot of people from outside, these red nodes, are sending emails [31:43.970 --> 31:46.950] to this guy that was hosting on this server. [31:47.750 --> 31:51.350] Here you just see the popular people. It might be interesting. I don't know. Maybe you want [31:51.350 --> 31:58.610] to get to know them. But I was really interested in whether I'm running an open relay. I'm [31:58.610 --> 32:00.530] not, but I'm pretending I will. [32:01.750 --> 32:12.170] So what I did is I said, well, just show me everything on the outside. All my domains [32:12.170 --> 32:19.630] or all the, like, rafi.ch or whatever my domain is, don't grade them out and just show me [32:19.630 --> 32:26.610] the rest. So you see there's still some others showing up here. So what I did then was I [32:27.550 --> 32:31.550] used afterglow to just eliminate all those. I'll show you how to do that. You can set [32:31.550 --> 32:38.350] nodes and invisible. And here I realized, oh, my God, there are actually people using [32:38.350 --> 32:45.130] my server, like this guy here, to send emails to people that are not hosted on my server. [32:45.670 --> 32:48.910] So I'm probably running an open relay. [32:51.610 --> 32:57.190] Well, some other things I was interested in, well, am I running, or is anyone spamming? [32:57.210 --> 33:01.170] And, well, probably, hopefully I see something in here. Otherwise my graph would probably [33:01.170 --> 33:05.490] not be very, my configuration would not be very good because probably on every mail server [33:05.490 --> 33:06.710] you'll find spam. [33:07.270 --> 33:12.510] So what I did is in the email log you see the size of the messages. So I just visualized [33:12.510 --> 33:18.530] who is getting what sizes of messages. And I was interested in just the ones that are [33:18.530 --> 33:23.090] above 10,000, all the others I'm not interested in. So you still see that, for example, down [33:23.090 --> 33:27.950] here there are multiple recipients that get the same size messages. So that means that [33:27.950 --> 33:35.170] someone probably is sending emails to a few people and this email is pretty big and it's [33:35.170 --> 33:42.970] probably the same one because they have quite a high size, or the same size. So that would [33:42.970 --> 33:48.170] be one indication of maybe there's something spam going on. Probably not. [33:48.230 --> 33:52.790] Well, another way of looking at whether you have spam is look at the number of recipients [33:54.050 --> 33:58.690] in an email. You can send emails to multiple people, right? So the number of recipients [33:59.110 --> 34:03.010] in the log file would indicate how many people got this email. And what I said, well, I want [34:03.010 --> 34:09.530] to see everything that's bigger than two. And all the things where someone is just sending [34:09.530 --> 34:15.150] to two other people, I don't want to see. So I have this omit threshold thing in Afterglow, [34:15.150 --> 34:19.430] I will explain it in a second, where I can drop out nodes that only connect to one other [34:19.430 --> 34:24.450] or two others. So I can filter out the things that are not very interesting. I'm interested [34:24.450 --> 34:31.090] in the bigger, the fans or the circles showing up. So when I did that I realized, well, only [34:31.090 --> 34:36.350] this one comes back here. There are, like, ten recipients and seven and eleven. And it's [34:36.350 --> 34:41.490] always from this free cloud report at news.ch. So it was probably a newsletter. And maybe [34:41.490 --> 34:50.490] it was spam. You can take this further and further. You can have very big, you look at [34:50.490 --> 34:55.170] very big emails. Maybe someone is sending out documents from the internal network to [34:55.170 --> 35:01.230] someone outside. I'm going to skip over these a little quicker to show you Afterglow. [35:02.710 --> 35:08.590] What an interesting, another one is there's a field called delay on the email server log. [35:08.850 --> 35:15.790] And that shows you how long was the email queued on the mail gateway. And what you see [35:15.790 --> 35:21.390] here is that there was apparently some problem here with these big two clouds. This one here [35:21.390 --> 35:25.950] and this one down here. These emails stayed on the server for more than ten minutes. And [35:26.370 --> 35:32.550] it happened a lot of times, as you can see, to these two email addresses. This one here [35:32.990 --> 35:36.970] and this one down here. So there's some problem with those addresses. Maybe they're wrong. [35:37.650 --> 35:43.850] And you can go in and either block them or do something about them or figure out who [35:43.850 --> 35:47.850] was sending the emails out to these guys. And this might actually be also a way to detect [35:47.850 --> 35:53.330] spammers because they send these huge lists of emails and probably a lot of them don't [35:53.330 --> 35:57.570] exist and they will be queued on system. They might retry all the time, so getting a lot [35:57.570 --> 36:00.990] of delay. So maybe you can figure out who your spammers are. [36:01.150 --> 36:08.150] Okay, let's get to the gist here, to Afterglow. You find it on SourceForge. As I mentioned [36:08.150 --> 36:15.010] earlier already, I support two output formats. The GraphList library from AT&T, also known [36:15.010 --> 36:19.790] as .neto, and the LGL, the Large Graph Layout by Alex Adai. [36:24.510 --> 36:31.130] If you, so Afterglow is just a Perl script. You can pass it the input and read some standard [36:31.130 --> 36:37.710] in. You can pass it a few parameters, figure it. One of them is if you do a minus T on [36:37.710 --> 36:45.450] it, it expects a two-column input. So just from two or size and recipient or something [36:45.450 --> 36:50.890] like that. So minus T reads two columns. If you don't do the minus T, it expects three [36:50.890 --> 36:55.890] columns. So you can do all these like source IP destination, IP destination port things. [36:56.610 --> 37:01.950] If you do a minus D, it will print the count on the nodes. That was that weird example [37:01.950 --> 37:05.790] from Capture the Flag I showed you, that you actually see how many times did that node [37:05.790 --> 37:11.650] show up in the log file. Then you can pass it an edge length to kind of optimize how [37:11.650 --> 37:14.790] your graph is going to look. If you have a length of five, then things would be really [37:14.790 --> 37:18.310] distant from each other. But if you have big clouds around one of the nodes, that might [37:18.310 --> 37:26.590] be helpful. With minus N, you can disable the node labels. So sometimes you have logs, [37:26.590 --> 37:31.110] if you can't parse them really well, you have really long tokens. So they will really clutter [37:32.050 --> 37:36.250] your graphs. They just turn them off and you get kind of a feeling that there are certain [37:36.250 --> 37:39.090] things going on. And then you have to go back to the log file and figure out what those [37:39.090 --> 37:47.650] nodes were. This minus O thing, this threshold that you can set, if you have a log file of [37:49.510 --> 37:54.350] communications like someone is talking to someone else and you see a lot of people that [37:54.350 --> 37:58.770] just connect to one other machine and nothing else. And then another machine to something [37:58.770 --> 38:03.090] totally different. But then you have these big clouds. A lot of times, I'm not interested [38:03.090 --> 38:07.690] in these one-to-one connections. I really don't want to know about those, but I'm interested [38:07.690 --> 38:12.150] in the big clouds. So if you do a minus O one, it would kill everything that has a fan [38:12.150 --> 38:19.410] out of one. And this feature is not quite complete. It doesn't do the whole traversing [38:19.410 --> 38:23.110] of the clusters and figure out whether it's connected to other clusters. Play with it. [38:23.110 --> 38:28.690] It's kind of cool. There might be like one-off cases that look kind of weird. And then the [38:28.690 --> 38:33.430] most important thing is the minus C. You can pass it a color property file. I'm going to [38:33.430 --> 38:39.670] show this in a second how this works. So the property file just has color dot, for [38:39.670 --> 38:44.370] example, source equals, and then it can pass it a Perl expression. And the expression just [38:44.370 --> 38:53.770] has to return a color, like red, this string. Then there is an array called fields that [38:53.770 --> 39:03.410] contains the columns that you passed it in. So for every line, it will evaluate these [39:03.430 --> 39:10.750] array. So you can say, like, the event node is red if the first column, so that's really [39:10.750 --> 39:16.670] the second column, if fields one matches this regular expression, so it's in 192, then make [39:16.670 --> 39:23.950] it red. Then I have the special color called invisible. If you say that the target is invisible [39:23.950 --> 39:30.050] if this expression here matches, then it won't draw that node. So you can kick out certain [39:30.050 --> 39:36.470] things by certain expressions. Or you just have a simple color. Just say color dot edge [39:36.470 --> 39:43.710] equals blue. So here is the fancy example of the file graph. I'm just very quick. So [39:43.710 --> 39:51.310] this is the property file. You say the source is olive drab if the field zero, so if the [39:51.310 --> 39:59.710] first entry that you get is 192.141.69.4, that's one of my IP addresses. It's not really, [39:59.710 --> 40:09.230] so go and play with it. So they show up green. Then everything else, like first match wins. [40:09.230 --> 40:14.330] So if it falls through, it will just be orange-red. So everything outside is going to be red. [40:14.470 --> 40:20.090] Then you configure the nodes to be blue, the targets. Like, I think you get the idea here. [40:20.090 --> 40:25.150] You can have really complicated things on the edges, for example, for ingoing and outgoing [40:25.150 --> 40:31.530] traffic, I have to say if the first column is inside my network or the third column, [40:31.530 --> 40:36.290] which is the destination IP, is inside my network, then give it a fire break color. [40:36.850 --> 40:39.310] And otherwise just use the other color. [40:39.910 --> 40:45.630] Afterglow also has two parses I'll give you. And I just realized, I just put this slide [40:45.630 --> 40:52.950] in last night at like 2.30. I hope there are no typos. I realized that these two parses [40:52.950 --> 40:56.050] are actually quite useful. I used them at Capture the Flag yesterday. [40:56.270 --> 41:00.310] If you have a TCP down block file and you want to visualize that, you have one problem. [41:00.310 --> 41:06.510] You see requests from a high port to port 80. Then you see the response going from port [41:06.510 --> 41:12.430] 80 to a high port. If you visualize that, you will see the high port and port 80 as [41:12.430 --> 41:16.490] source ports. And that was what I was showing in the other graph, basically. [41:16.490 --> 41:20.570] And then the destination port, again, same port numbers. So you need to, for the request, [41:20.570 --> 41:26.490] you need to flip the ports. That's so that you have the connections again. And this, [41:26.490 --> 41:31.770] the TCP down to CSV takes care of that. It remembers what direction the connections are [41:31.770 --> 41:34.990] going in and automatically flips them. So you get nice output. [41:35.090 --> 41:40.390] And then the send mail parser combines the send mail log entries into one again that [41:40.390 --> 41:50.120] you need. That's really it. Any questions? Wow, everyone is tired. Yeah? [42:02.030 --> 42:08.930] So the question is whether I'm thinking about using, like, a database to drive or to read [42:08.930 --> 42:13.970] from or something. Well, AfterGrowth is really just like a Unix script, right? You use pipes [42:13.970 --> 42:19.850] and you pipe in data and you get data out. So it's pretty easy to use a SQL database. [42:19.850 --> 42:27.230] For example, you use the TCP down to CSV. You pipe that into MySQL minus E, insert into [42:27.250 --> 42:31.430] blah, blah, and you get into database and then you can have it piped into something [42:31.430 --> 42:35.530] else and you read out the stuff from the database. So you plug it basically into whatever you [42:35.530 --> 42:42.110] want it. We basically just provided this AfterGlow to take the CSV, to add the color and then [42:42.110 --> 42:46.410] everything to that graph file and generate the graph file and then you use the grapher [42:46.410 --> 42:51.890] to generate the graph. But please feel free to contribute anything to the SourceForest [42:51.890 --> 42:56.070] project. And also if you're going ahead, go home and [42:56.070 --> 43:01.030] visualize and get some cool graphs. I'm always interested in getting graphs, so spam me with [43:01.030 --> 43:13.000] cool pictures. How fast is it? Well, most of the time it's actually consumed by Nido. [43:13.000 --> 43:18.080] That's what I use to generate all these graphs here because it has to generate the layout [43:18.080 --> 43:24.240] and it really depends on how many nodes or how many log entries you're passing in. [43:25.860 --> 43:31.260] It's, if you have probably about, let's say we want to have a refresh rate of like 30 [43:31.260 --> 43:34.480] seconds. I think that's good because you need time to look at it and then it's going to [43:34.480 --> 43:38.160] change completely again. That's going to be one of the problems. You don't see nice delta [43:38.160 --> 43:42.880] updates. You would have to handle that somehow. Might be something that maybe Greg and I could [43:42.880 --> 43:49.880] work on. So if you're 30 seconds, it's probably safe [43:49.880 --> 43:56.300] to use up to probably 5,000 events in 30 seconds. That should probably work. I think [43:56.300 --> 44:00.380] I did something like that yesterday with the capture the flag data. [44:01.880 --> 44:07.680] I have two more books. I guess I give them to the two gentlemen that asked the questions [44:07.680 --> 44:14.820] here. Anyone else? All right, thank you very much.