So let's finally talk about visual security event analysis.
So what I'm going to talk about is how you analyze log files or events that you have
in a visual manner. I don't want to talk about myself too much. Maybe one thing. I've been
involved in, like, log analysis and event analysis for quite a while, and it has always
interested me whether there's a way to look at the data in a way that it really, like,
a lot of data, in a way that it's really apparent what is going on in that dataset. And if you're
looking at log files, thousands of lines of logs, it's really hard to figure out at one
glance what is going on. You have to start reading through them. You have to, like, use
your own mind to do pattern analysis and figure out what is really going on.
So at some point, I started working for this company that I'm working for right now called
ArcSight, and I got a hand on a lot of neat tools to look at data, and I got a feature
in the software that is called Event Graphs, and I really liked it, and I started looking
into how can we use this feature to make it more easy for a security analyst or a person
interested in a log file to figure out what is really going on in that dataset.
So that's how kind of my passion about Event Graphs started. My talk here is not going
to be very textual or, like, theoretical, and I'm not going to talk too much about how
you generate the graphs and whatever. I want to show you a lot of examples on how to look
at datasets and how you configure graphs to get the most out of it.
As part of the talk, I'm going to show you a tool that I have up on SourceForge, which
is called Afterglow, which is a very simple Perl script that transforms CSV files into
a graph or language file, so you can go home and play with these things yourself, and the
tool is actually also on the CD.
I'm going a little faster in the beginning so we can catch up for the time I lost already.
One thing, IP addresses and host names, you're going to see in all these graphs, and the
descriptions are obfuscated or changed. The addresses are completely random and any resemblance
with well-known addresses or host names are purely coincidental.
All right. So let's do a little bit of an introduction here. Why would you want to look
at these graphs? I guess this is a pretty good example already. Do you want to look
at the left-hand side, or do you want to look at the right-hand side?
Well, even if I made the font bigger on the left-hand side, it's probably still not very
fun.
All right. Why do you want to use event graphs? I guess I already motivated that quite well.
I'd rather look at the graphs. They look just much nicer.
If you actually dig into it a little more and you start looking how to configure these
things, you will see pretty quickly that you can visualize different properties in those
graphs, and it's very easy to see certain aspects of a log file very quickly and certain
things that are important to you. You don't have to wade through ten thousands of lines
of logs to figure out whether there is anyone sending a lot of e-mails to someone else.
If you use this in a security operating center or in a 724 response team or something, if
you have graphs, they're a great way of giving you situational awareness or real-time view
on things that are happening, and you can quickly react on them.
We have a lot of customers that are starting to use this in their SOCs, and it doesn't
just look pretty on their big screens, but it's actually really useful. It's actually
also kind of an interesting story. I was at one customer, and they had all these big displays,
and they were empty, and I was like, wow, why do you have these nice graphs here on
the analyst screens? Why don't you have them up there on the big screens? They were like,
whoa, no, no, no, we tried that for a while, but then the boss came in, and he saw all
these things on the screen. He was like, oh, what is this? What's that? What's this? What's
that? So we had to explain all this. We took them off.
All right, and one other thing, another story that I had with these graphs, which actually
two other stories. One is, one customer uses these graphs to communicate with other teams,
so you have the security analysts. They look at the log files and all that, and then they
go back to the operators or the operational folks, and they're like, guys, you have to
do something about this machine. It's talking to all the others in the network. And in the
past, they were handing them log files to prove that it's that way, and the operators
were like, whatever, we don't understand what this is. But now they started giving
them graphs, and the operators were like, oh, my God, my machine is talking to all these?
Okay, we're going to take care of this. And even worse, I had one customer that actually
just told me this week that he used the graphs to analyze some data, and he found some stuff,
and one of the guys got fired just because he used the graphs and found out that one
of the guys was not doing his work. All right, so I came up with, like, two big categories
of when to use these event graphs. One is in a real-time monitoring scenario, where
you have, like, a screen app that looks real-time at certain properties of your data file, and
I'm going to show you some examples on what I mean by that. It gives you great awareness
of things that are happening. You can configure these things to, for example, look at transactions
going to your financial server. So right away, if someone connects to your financial service,
you will know. And then you can take it a level further. You can say, well, I know that
these machines are okay to connect to my financial servers, so don't show me those, but show
me everything else. So as soon as something pops up there, you know you've got to look
into it. The other thing is, if you take this in a more forensic or historical way, you
can, you grab a data set, you run it through some tools, and you look at the graphs on
what is, on certain properties, and you figure out what's going on. Let me mention quickly
some related work. There's some classics from, he did all these kind of things. He tried
to figure out whether there are certain ways to visualize properties of data, and he took
a lot of dimensions and put them all in one graph. What I'm going to do is, I take a lot
of dimensions too, and I'm always going to stay with event graphs, so like node-to-node
connections. I'm not going to use any of the parallel coordinates or the right-hand-to-bottom
thing that I have no idea what it shows there.
Then Ehrbacher, he also, he actually went even further. He tried to go even more multidimensional
and map everything into these nodes, so there's like size, there's shape, there's length of
these spirals going out, and so on. Then Greg Conti, you probably have seen his talk, I
hope. He mainly focuses on different ways to visualize TCP dump or packet locks, and
very interesting.
Then Envision IP is something I found. What you see up there is a grid of, I think it's
8x8 or 64x64, and they just map IP addresses in there, so you see certain clusters of things
going on. The nice thing about this project is, there's a lot of interactivity, so you
can start playing with the data, you can zoom in and so on.
And then finally, a few of you might know Showkey from Steve Barry. It's basically a
scatterplot. You can define three different scatterplots, and then you have a 3D of that.
So on the top right, it's not a very nice picture, but on the top right it's a three-dimensional
output, so you can say one axis is your source IP, one axis is destination IP, and the other
one is the destination port.
All right, let's move on. How do you generate an event graph? Pretty easy. You have a log
file that a device is generating. You take a parser. Why do you take a parser? Well,
you need to know where in my log event or my log file, where's the source IP, where's
the destination IP, where's the event name, where's the port? So you parse that, you get
the syntax right, and then you visualize that in an event analyzer.
If you have an event, let's look at this Nord event here. You don't have to be really able
to read this, but what I want to show here is in an event graph you can have different
configurations. The normal event graph that you will always see is like who is talking
to whom, so you have source IP, destination IP. But you can take this a lot further. You
can say, well, on the right top you want to have the source IP, destination IP, and destination
port, so you will see someone port scanning, for example, if there's a fan with destination
ports. So you can configure these things in pretty wild ways, and I'm going to show you
some of them.
Now let's have a sneak preview of Afterglow. I apologize because I don't use my laptop
running it. I can't show you the demo. It's not very fun anyways. I will show you the
output anyway, so it doesn't really matter. So what is Afterglow? Afterglow is not a security
information management tool. It's just a script which takes CSV files, so you have
to somehow from your logs generate a CSV file, comma, separated values. And then you take
Afterglow, the tool that I'm going to show you, and you transform the CSV files into
a graph language file. And then you take a grapher tool, and right now I'm supporting
two of them, that takes this graph language and generates the graph. And that's not something
I did. I just transformed the CSVs into graph files, but I'm going to show you some cool
things that we implemented there.
So how you can use this thing is you basically cap the input file, you pipe it to Afterglow,
you give it a color properties file, which I'm going to introduce a little later, and
then you can use the AT&T Graphverse library, .neto, I'll give you references in a second,
and you generate a GIF file. The color properties file is basically just you tell it the source
nodes are red, for example, the event nodes are green, and the target nodes are blue.
And I'm going to show you how to really play with this stuff. We extended this quite a
bit so you can have colors depending on the values in the log files.
Big shout out to Christian. Thanks a lot for building the foundation of this tool. He built
me the whole Afterglow profile, the basics and everything, and I just went in and made
some small changes, but thanks a lot, Chris. Let's look at examples. I mentioned in the
beginning I kind of have two use cases or two big categories of things to look at. One
is what I call situational awareness or real-time monitoring. So this is one of the views that
I saw at a customer. Basically you see here is six independent screens of different properties
of your data that you want to look at. So here the configuration is such that on the
top left the customer wanted to see just the compromise attempts, and don't ask me how
to filter for just those events. That's a whole different story. But if you have a way
to say these are my malware events, these are my DOS events, these are my scan events,
if you can separate them by something, then you can draw all these screens and you can
look at these screens and see what's going on.
So what this customer is actually also doing, whenever an analyst starts his shift, he resets
all the graphs so they're empty. So over time, nodes start popping up and they look at them
and they're like, okay, this is fine, and they might filter them out even, or they just
leave them there and say, okay, I know about this. And the more things that show up, the
more they go along in their workflow and work with these events. So I have a feeling this
is going to be a great tool, especially in a SOC or an analysis environment, to kick
off your workflow. You look at these things and you take it on and you start building
filters to filter out the things you don't know about, and you just see the outliers
here in these screens.
The second big group of use cases is forensic analysis or historical analysis, where you
take a log file that you have and you visualize it. Afterglow can generate two kinds of outputs.
One is for the AT&T graphics library for .neto, and the other one is for LGL, the large graph
library. I will give a pointer to that in the end as well. This thing is generating
three-dimensional graphs. You can actually generate VRML output and you can fly through
these things. I didn't prepare a demo because it's kind of fun to fly through it, but the
analysis value is not really high or the value you get out of it. It's really cool
to fly through it and see that, oh, this guy sent an email. To whom? Oh, over there. Okay.
It looks great, and I apologize for the colors here, but this is really how it looks. Anyway,
let's go back to 2D.
This is a very simple example, one of about 12 that I'm going to show, of how to analyze
a log file. So what I had here is, I believe, a snort log or a web log or something, and
I said, well, show me what is happening. So the event name, the destination IP are the
blue ones, and then the destination ports are the white nodes. So what you see here,
all the blue circles are my web servers. The white boxes are the target ports that they're
being connected to. So you see it's 80 and 443, so that's totally legitimate. Only HTTP
and HTTPS traffic is going to my servers, so I probably don't have to worry about that.
If I would see a node there, like port 21, and someone connects to my web servers with
FTP, maybe that's okay, but if I'm not running FTP in my environment and that shows up here,
I probably want to investigate that. So it's a very quick way to see whether traffic is
really what it's supposed to be.
Almost same configuration. Here you see source IP, destination IP, destination port. So this
would potentially show you port scans or network scans. Here you see a lot of machines connecting
to a lot of other ones, all on the same port. Down here, this white node here, is the port.
So they all connect to the same port. Well, it could be that a worm is probing you, but
it could also be that you have a web server, or a lot of web servers, and a lot of machines
are connecting to those. So it might be okay, but it might be a scan, and depending on what
the destination port is, you might figure out what's going on in this case. But you
can certainly tell, well, this is all similar behavior here on this fan, and then actually
if you see on the bottom left here, there's some little ones, it's probably worth investigating
what those are, and probably those are the outliers that you are interested in.
Let's keep this configuration of the graph. This is just a little bit of a different dataset
and a different time. It looks much prettier. Let's zoom in on the bottom part here. What
this is showing you is that there are two machines, the red nodes, that are connecting
to about 15 other ones, and still on the same port. So again, in your environment, do you
really have all these other 15 or 20 machines serving this one service, and exactly two
machines are connecting to them? So you can go and figure out what all these machines
are, and hopefully they are really the ones that are serving this service, and then you
can figure out whether these other two are really allowed to connect to this.
Let's have a look at this weird thing over here. What you see here is, probably you can't
read this, but there is a node here that says 21 FTP, so one of the properties of this graph
is that the white nodes show up exactly once per target port, so everything that happens
on port 21 is going to show up around that thing there. So again, I can say, wow, I have
about six FTP servers, and about five people are connecting to them. So the FTP servers
would be the blue nodes, and then the red ones that connect to them would be the people
using it. So, well, is this really something I have in my environment? Is this allowed?
And why are only three or five people connecting to them? Is that some kind of a process that,
I don't know, maybe AIX that transfers log files over, something weird like that? Or
is this something malicious? Is anyone having a ware server or whatever?
If you interpret these graphs, you have to be a little careful. I still have the same
configuration. Destination IPs are the blue ones, and the white ones are the destination
ports. Well, if you look at this on the left-hand side, I would say, wow, it's a port scan.
Well, is it really a port scan? I have a book for the one that gets this right. What is
the left-hand side? No, not voice over IP. Who said that? Over there. You got the book.
It's the source ports. The log file reports such that the source port is really the destination
port and the other way around. So when you analyze these things, be careful and make
sure you're really looking at what you're looking at. So why do I know are the source
ports and the destination ports? Well, they're in the 50,000 range. They're almost contiguous.
And I mean, who is scanning me on these high ports? We're not a capture the flag. There,
they might scan you on those high ports because services are running all over. But here, there's
really no sense why someone would scan you up there. There's probably nothing there listening
anyway. So be careful when you analyze these things. Sometimes it's port scan, sometimes
it's not. I'm always, my first impression is always, well, false positive. Let's really
figure out whether that's true. Okay, let's get fancy here.
This is a firewall log. Basically, I'll show you passes and blocks from a PF firewall.
If I would just visualize source IPs, destination IPs, I would see big clouds of things. Well,
I want to see a little more in this graph. So what I did is I have source port, rule
number, that's the squares, and then destination IP. So you will see I'm connecting to some
other machine and rule number 100 is responsible for letting me in or blocking me.
What I did then with Afterglow is to configure my nodes such that all the external IPs are
going to show up as orange. I'm going to show you how to configure Afterglow to get exactly
this graph. So the external machines are orange. My internal machines are green. And
you see I only have a few of them. And then the rules are the blue boxes. And then further
what I said, I wanted to see whether a connection is incoming or outgoing of my network. So
I configured the edges such that outgoing connections are blue, that's the majority
here, and incoming is red. And I just realized it's probably exactly the other way around.
So I hope there's more incoming than outgoing because these are web servers that are servicing
some stuff. Anyway, so you see big clouds of things that are showing similar behavior.
So down here there's this one rule that is responsible for letting in a whole lot of
traffic. And probably that's fine. There's another one up here. You can, if you know
the rule set, if you know your ACLs, you can go and figure out whether it's really what
it's supposed to do. But you can also figure out some things that are kind of weird. You
have this whole section in the middle. You probably want to zoom in on those and filter
out all the others and just look at these things and figure out whether your rule set
is really doing what it's supposed to do. So the next steps would be, so the first is
like overview, general impression, what is going on, is there anything really sticking
out. And then my next steps would be, I would visualize the firewall blocks of outgoing
traffic. Why would internal machines connect outside and be blocked? So that's either a
misconfiguration of the internal box or the firewall rule set is wrong. Then the second
one would be, I would probably look at blocks of incoming traffic. I want to know who's
probing me, who's poking around. And if you have a live system that can look at these
things, you can put all these attacker machines maybe on a blacklist and if you see them again
and they're successful coming in, you might actually take some action and say, whoa, now
someone probed me and now he's coming in. And then you can take this further. Third
step would be to visualize the passes of outgoing traffic. Is this really what should be leaving
the network? Maybe you find someone FTPing stuff out that you really don't want to FTP
out. Maybe you realize that you're running peer-to-peer clients in your internal network
and they're connecting out, anything like that. And you can probably take a few more
steps to analyze the data set, but I guess you get the idea.
So this one is probably the wackiest graph I have. It's totally going away from source
IPs and destination IPs. What I did here is I wanted to debug my firewall rule set. So
it's still the same data set as before, but this time what I show you is the red nodes
are the rule number or the ACL number, then the destination ports, and then you see the
action taken, pass or block. So just by virtue of firewall rule configurations, you will
see two distinct graphs. One on the left-hand side is all the passes and the right-hand
side is all the blocks. So here you can see, if you take one of these blue nodes, for example,
port 53 here, you see that this one rule is responsible for letting traffic in. Well,
most is okay, maybe, probably, hopefully, but you have other ones where there are two
nodes or three nodes connected to the same rule. Like over here, for example, there are
three ports being blocked by this one rule. It might be okay. It might not. If you see
something like this on the other side, that one rule passes traffic from multiple ports,
maybe that's the way you configured it, but this is certainly a great way of analyzing
and debugging your firewall rule set.
This is another example where I took a TCP dump log, and what I visualized here is I
take the region of where the source IP resides, so you can go on the web, there are databases
that show you what the location codes are for the IP addresses. They're not very accurate,
but in a lot of cases, they're actually pretty good. So I show you the source region, where
it's coming from in the world, then I show you the source MAC address. So this is a TCP
dump log. The source MAC address is going to be the last hub, probably my router, that
let the traffic into my network. And then I show the destination IP. So what I see here,
I have four boxes, the four white ones, that are my internal machines, and interesting
enough, I have two entry points into my network. I have two source MAC addresses showing up.
And then I have these two fans in the middle part. What this is showing me is I probably
either have a redundant router setup, or maybe I have a load balancer. And what is interesting
here is that all these machines up here, they're connecting through this guy. All these guys
here are coming through both of them, and all these down here come from the other one.
So maybe this is a regional load balancer, that everything on one side of the world takes
this entry point, everything on the other side is using that entry point. So if that's
really the case, a great way to analyze this log is this graph here, and you can actually
figure out why these two in the middle are coming in for both of them. And this is just
a zoom, where you see the one server, and here it's like, you can't even read it below
the node here, but it's Sand Hill or whatever in the United States.
Graphs like these are a lot of times indications for warms, but again, be careful. There's
a nice fan here of source IPs using the same event going to a few destination IPs. Well,
it could be, again, it could be a warm that is using this one event to try to get into
a network, but it might again be a web server serving just legitimate files. What you have
to do in this case, you have to figure out what is this event name here that connects
all these dots, and this might give you an indication of a warm or not. If you're visualizing
a snort log file and this talks about warm, well, duh, it's probably a warm. But if you
visualize a TCP dump log file, like an ICMP port unreachable might be an indicator for
a warm trying to probe UDP ports and you get ICMP port unreachables back. So you have to
investigate a little bit to figure out whether it's really a warm.
I was playing Capture the Flag actually this year and last year, and I was trying to figure
out the challenge really for the analysis team on Capture the Flag is that all the traffic
hitting your web server or your servers seems to be coming from the same IP address, and
there is malicious traffic of the other teams hacking you, but there is also benign traffic
from the score bots trying to figure out whether your services are still up. So blocking
all the traffic is not working. So you have to figure out a certain way of distinguishing
these two types of traffic, and you can't do it by the source IP, which you would normally
do in your corporate environment.
So I started visualizing this log file of TCP dump, and I just wanted to have a general
feeling for what's going on. So what you see here, the dark blue ovals are destination
ports that are below 1024, normally services that are offered to the outside world. Then
destination ports higher than 1024 are light blue, and then you see here what I just explained,
the red node, this is what all the traffic that's coming in is coming from this one IP.
And then I configured it such that our machines, the internal targets are green, other teams'
targets are brown, and then internal sources are yellow, and internal targets are the boxes.
So what you see here, we have two servers, while there's a third one showing up down
here, this was actually not a server, but I think that was one of the machines on our
network that someone connected to, and that could have been one of our team members connected
to someone else's machine. And then the services that we offered were D6, and it was all kinds
of weird things. I don't even remember anymore what it was last year, port 9999 or something,
and there was some mud running there. But this gives you a very quick way of looking
at what is happening in this network.
Well, I took it a little further, and I had an idea, or a few people had an idea, I said,
well, maybe it's a TTL of the traffic coming in. Maybe you can distinguish the good traffic
from the bad traffic by looking at the TTL. So well, what I did is I said, okay, show
me the source IPs, the destination IPs, and then the TTLs of that traffic. So you see
here, I have eight TTLs showing up, eight distinct TTLs, and I'm going to look at the
everything from 35, 50, 52, 60, 62, 63, 64, 128, and 146. Some of them are showing up
if my teammates would connect out, so I could filter them out. I was not interested in those.
I'm just interested in the ones that happen of the traffic coming in from this one evil
IP. Let's go back quickly again.
And then I figured out, well, I still can't quite tell what is okay and what's not, so
I had to reconfigure my graph and look at some other properties. So what I did is I
said, well, I'm not really interested in the IPs. I know everything is coming from this
one IP that I'm interested in, so it only showed me the traffic from that guy, but I'm
interested in the destination ports, the TCP flags, and the TTLs. So I wanted to see, like,
the SYNs, the SYNX, the pushes and whatever, and their TTLs, and what port they went to.
So this yields this graph here, and in addition to that, that was not quite enough to figure
out what's really going on. I need to have the counts. I'm going to lose everyone here.
I need the counts on all these nodes, so how many times did certain flags show up? So let's
take one example here. I have port 80 here. It showed up 2,600-and-something times, and
these are all the flags that showed up with the different TTLs. So there were four TTLs,
one, two, three, four, that were used when someone connected these ports here, and here
are the TCP flags that were being used for this traffic. So I had a theory. I said, well,
I'm looking at all the connection establishments, all the SYNs, and I want to see how many times
they happen. The score about traffic is probably going to all the services, like in sequence,
or probably the same amount of time. So I will see the same amount of SYNs going to
each of the destination ports, and if the theory about the TTLs is right, they all use
the same TTL. And it turned out that we were actually able to figure out the TTLs, and
for the game last year that way, and we filtered all the other traffic. But it was a little
late in the game, so we still didn't win.
All right, so I have some other logs here. At some point I was interested in email logs,
and I wrote a little send mail parser, and that's actually part of the afterglow scripts
that you can have on the CD or you can get from SourceForge. The problem in send mail,
I'll mention that in the end I guess, there are always two entries. It says from so-and-so,
and then there's a message ID, and then the next log entry says to so-and-so, and there's
a message ID to link those two events together, or just two log entries. So I had to write
a tool to actually take them, to put them together, and then output a CSV file, and
all that stuff, and the result is this one here.
So I'm just showing who is talking to whom by email. And then I said, well, what's probably
interesting is to see everything that's coming from my domain, the one that I'm really hosting
on that server, going to people outside and what is coming into my domain. So I colored
the whole graph that way, and if we look at a zoom up here, you see that these people
here are quite popular. A lot of people from outside, these red nodes, are sending emails
to this guy that was hosting on this server.
Here you just see the popular people. It might be interesting. I don't know. Maybe you want
to get to know them. But I was really interested in whether I'm running an open relay. I'm
not, but I'm pretending I will.
So what I did is I said, well, just show me everything on the outside. All my domains
or all the, like, rafi.ch or whatever my domain is, don't grade them out and just show me
the rest. So you see there's still some others showing up here. So what I did then was I
used afterglow to just eliminate all those. I'll show you how to do that. You can set
nodes and invisible. And here I realized, oh, my God, there are actually people using
my server, like this guy here, to send emails to people that are not hosted on my server.
So I'm probably running an open relay.
Well, some other things I was interested in, well, am I running, or is anyone spamming?
And, well, probably, hopefully I see something in here. Otherwise my graph would probably
not be very, my configuration would not be very good because probably on every mail server
you'll find spam.
So what I did is in the email log you see the size of the messages. So I just visualized
who is getting what sizes of messages. And I was interested in just the ones that are
above 10,000, all the others I'm not interested in. So you still see that, for example, down
here there are multiple recipients that get the same size messages. So that means that
someone probably is sending emails to a few people and this email is pretty big and it's
probably the same one because they have quite a high size, or the same size. So that would
be one indication of maybe there's something spam going on. Probably not.
Well, another way of looking at whether you have spam is look at the number of recipients
in an email. You can send emails to multiple people, right? So the number of recipients
in the log file would indicate how many people got this email. And what I said, well, I want
to see everything that's bigger than two. And all the things where someone is just sending
to two other people, I don't want to see. So I have this omit threshold thing in Afterglow,
I will explain it in a second, where I can drop out nodes that only connect to one other
or two others. So I can filter out the things that are not very interesting. I'm interested
in the bigger, the fans or the circles showing up. So when I did that I realized, well, only
this one comes back here. There are, like, ten recipients and seven and eleven. And it's
always from this free cloud report at news.ch. So it was probably a newsletter. And maybe
it was spam. You can take this further and further. You can have very big, you look at
very big emails. Maybe someone is sending out documents from the internal network to
someone outside. I'm going to skip over these a little quicker to show you Afterglow.
What an interesting, another one is there's a field called delay on the email server log.
And that shows you how long was the email queued on the mail gateway. And what you see
here is that there was apparently some problem here with these big two clouds. This one here
and this one down here. These emails stayed on the server for more than ten minutes. And
it happened a lot of times, as you can see, to these two email addresses. This one here
and this one down here. So there's some problem with those addresses. Maybe they're wrong.
And you can go in and either block them or do something about them or figure out who
was sending the emails out to these guys. And this might actually be also a way to detect
spammers because they send these huge lists of emails and probably a lot of them don't
exist and they will be queued on system. They might retry all the time, so getting a lot
of delay. So maybe you can figure out who your spammers are.
Okay, let's get to the gist here, to Afterglow. You find it on SourceForge. As I mentioned
earlier already, I support two output formats. The GraphList library from AT&T, also known
as .neto, and the LGL, the Large Graph Layout by Alex Adai.
If you, so Afterglow is just a Perl script. You can pass it the input and read some standard
in. You can pass it a few parameters, figure it. One of them is if you do a minus T on
it, it expects a two-column input. So just from two or size and recipient or something
like that. So minus T reads two columns. If you don't do the minus T, it expects three
columns. So you can do all these like source IP destination, IP destination port things.
If you do a minus D, it will print the count on the nodes. That was that weird example
from Capture the Flag I showed you, that you actually see how many times did that node
show up in the log file. Then you can pass it an edge length to kind of optimize how
your graph is going to look. If you have a length of five, then things would be really
distant from each other. But if you have big clouds around one of the nodes, that might
be helpful. With minus N, you can disable the node labels. So sometimes you have logs,
if you can't parse them really well, you have really long tokens. So they will really clutter
your graphs. They just turn them off and you get kind of a feeling that there are certain
things going on. And then you have to go back to the log file and figure out what those
nodes were. This minus O thing, this threshold that you can set, if you have a log file of
communications like someone is talking to someone else and you see a lot of people that
just connect to one other machine and nothing else. And then another machine to something
totally different. But then you have these big clouds. A lot of times, I'm not interested
in these one-to-one connections. I really don't want to know about those, but I'm interested
in the big clouds. So if you do a minus O one, it would kill everything that has a fan
out of one. And this feature is not quite complete. It doesn't do the whole traversing
of the clusters and figure out whether it's connected to other clusters. Play with it.
It's kind of cool. There might be like one-off cases that look kind of weird. And then the
most important thing is the minus C. You can pass it a color property file. I'm going to
show this in a second how this works. So the property file just has color dot, for
example, source equals, and then it can pass it a Perl expression. And the expression just
has to return a color, like red, this string. Then there is an array called fields that
contains the columns that you passed it in. So for every line, it will evaluate these
array. So you can say, like, the event node is red if the first column, so that's really
the second column, if fields one matches this regular expression, so it's in 192, then make
it red. Then I have the special color called invisible. If you say that the target is invisible
if this expression here matches, then it won't draw that node. So you can kick out certain
things by certain expressions. Or you just have a simple color. Just say color dot edge
equals blue. So here is the fancy example of the file graph. I'm just very quick. So
this is the property file. You say the source is olive drab if the field zero, so if the
first entry that you get is 192.141.69.4, that's one of my IP addresses. It's not really,
so go and play with it. So they show up green. Then everything else, like first match wins.
So if it falls through, it will just be orange-red. So everything outside is going to be red.
Then you configure the nodes to be blue, the targets. Like, I think you get the idea here.
You can have really complicated things on the edges, for example, for ingoing and outgoing
traffic, I have to say if the first column is inside my network or the third column,
which is the destination IP, is inside my network, then give it a fire break color.
And otherwise just use the other color.
Afterglow also has two parses I'll give you. And I just realized, I just put this slide
in last night at like 2.30. I hope there are no typos. I realized that these two parses
are actually quite useful. I used them at Capture the Flag yesterday.
If you have a TCP down block file and you want to visualize that, you have one problem.
You see requests from a high port to port 80. Then you see the response going from port
80 to a high port. If you visualize that, you will see the high port and port 80 as
source ports. And that was what I was showing in the other graph, basically.
And then the destination port, again, same port numbers. So you need to, for the request,
you need to flip the ports. That's so that you have the connections again. And this,
the TCP down to CSV takes care of that. It remembers what direction the connections are
going in and automatically flips them. So you get nice output.
And then the send mail parser combines the send mail log entries into one again that
you need. That's really it. Any questions? Wow, everyone is tired. Yeah?
So the question is whether I'm thinking about using, like, a database to drive or to read
from or something. Well, AfterGrowth is really just like a Unix script, right? You use pipes
and you pipe in data and you get data out. So it's pretty easy to use a SQL database.
For example, you use the TCP down to CSV. You pipe that into MySQL minus E, insert into
blah, blah, and you get into database and then you can have it piped into something
else and you read out the stuff from the database. So you plug it basically into whatever you
want it. We basically just provided this AfterGlow to take the CSV, to add the color and then
everything to that graph file and generate the graph file and then you use the grapher
to generate the graph. But please feel free to contribute anything to the SourceForest
project. And also if you're going ahead, go home and
visualize and get some cool graphs. I'm always interested in getting graphs, so spam me with
cool pictures. How fast is it? Well, most of the time it's actually consumed by Nido.
That's what I use to generate all these graphs here because it has to generate the layout
and it really depends on how many nodes or how many log entries you're passing in.
It's, if you have probably about, let's say we want to have a refresh rate of like 30
seconds. I think that's good because you need time to look at it and then it's going to
change completely again. That's going to be one of the problems. You don't see nice delta
updates. You would have to handle that somehow. Might be something that maybe Greg and I could
work on. So if you're 30 seconds, it's probably safe
to use up to probably 5,000 events in 30 seconds. That should probably work. I think
I did something like that yesterday with the capture the flag data.
I have two more books. I guess I give them to the two gentlemen that asked the questions
here. Anyone else? All right, thank you very much.