[00:19.670 --> 00:25.930]  Okay, good morning everybody. Welcome to DEF CON 28, Blue Team Village. We are going to start this
[00:25.930 --> 00:31.670]  morning's workshop off with Lennart Koopman from Greylog. He's going to be doing an introduction
[00:31.670 --> 00:36.930]  to the OpenStock CTF tools. Definitely worthwhile listening to. Hope everybody enjoys. I'm going to
[00:36.930 --> 00:43.410]  turn it over to Lennart now to get to the meat and potatoes of it. Thank you. That sounds great.
[00:43.410 --> 00:49.970]  Thank you so much. First time I'm doing a talk at a remote conference. I think we're all learning
[00:49.970 --> 00:55.490]  new things in this year. I wanted to, before I start with anything else, I just wanted to say
[00:55.490 --> 01:03.410]  thank you to the organizers not only of DEF CON, but also the organizers of Blue Team Village,
[01:04.130 --> 01:07.810]  because I think that's just an amazing job they're all doing. They're doing that every year,
[01:07.810 --> 01:12.750]  but now this year I think it's even harder. And I as a speaker, and I think I was the first in
[01:12.750 --> 01:17.710]  the lineup here, so some things were new to them too. I feel like they did an absolutely
[01:17.710 --> 01:22.330]  fantastic job. So thanks everyone, and I hope everyone has an absolutely fantastic time at
[01:23.770 --> 01:30.570]  at DEF CON and at Blue Team Village specifically. So what I'm going to do today is I'm going to
[01:30.570 --> 01:39.410]  walk you through a very, very high-level overview of what Greylog is. And we're doing that because
[01:39.410 --> 01:45.890]  Greylog is also part of OpenSOC. And OpenSOC, the CTF of the Blue Team Village, if you're
[01:45.890 --> 01:51.670]  competing in it, you're going to be using Greylog a lot. So I was asked if I could give an introduction
[01:51.670 --> 01:56.950]  to it, especially I think for everyone, but even a little focused on the people that will be using
[01:56.950 --> 02:01.390]  it during OpenSOC. So I think it's going to be a fantastic overview if you're new to Greylog,
[02:01.390 --> 02:05.670]  or if you haven't looked at Greylog in a while, or if you're generally interested in open source
[02:05.670 --> 02:11.590]  software, DFIR, block management, whatever. I'm going to walk you through it, like I said,
[02:11.590 --> 02:17.990]  very high level. And very important, we're going to have a bunch of time for Q&A and questions at
[02:17.990 --> 02:24.330]  the end. That means that if you have any questions, we're going to go through them
[02:24.330 --> 02:30.170]  then, and I'm going to make sure that we leave enough time there. We will also be, when I say we,
[02:30.170 --> 02:33.970]  we Greylog people are going to be around in the Blue Team Village
[02:35.210 --> 02:42.230]  Discord, and there is a Greylog channel there under the Sponsors tab. And we'll be around
[02:42.230 --> 02:45.970]  there. We're in multiple time zones. There's a good chance you're going to meet
[02:45.970 --> 02:50.010]  someone from us over there. I believe we have a little flag at our username or something that
[02:50.010 --> 02:54.150]  shows that we're part of Greylog, and we'll be happy to also answer any kind of questions that
[02:54.150 --> 03:03.510]  you might have, maybe during the CTF or OpenSoc itself, or maybe in general about Greylog. So
[03:03.510 --> 03:12.230]  we'll be around. We'll be happy to answer any questions you have. Next slide is the agenda.
[03:12.230 --> 03:19.130]  I'm going to be very careful with not moving around too much and not skipping through slides
[03:19.130 --> 03:24.130]  too fast. I hope there's not too large of a delay between what I'm saying and what you're seeing.
[03:25.290 --> 03:30.510]  So that that is all in sync, but I'm going to be careful that that works. If there are any issues,
[03:30.510 --> 03:35.010]  you can always post in the Discord channel. There are moderators there, and they can make me aware
[03:35.430 --> 03:41.690]  if there's something going massively wrong in any way. The agenda for today, you'll see
[03:41.690 --> 03:47.890]  this here on the left, I think. I only have like three real slides, because I want to really
[03:47.890 --> 03:52.730]  show you Greylog as a product. I want to show you what it does. So it's going to be a big focus on a live
[03:52.730 --> 03:58.010]  demo. But I'm going to go over one slide, just real quick, about me. Then one about Greylog,
[03:58.010 --> 04:03.050]  give you a little bit of background. Then we can take a quick look at the Greylog architecture.
[04:04.230 --> 04:08.590]  Not kind of in the level of detail for when you would get ready to deploy this yourself, but
[04:09.130 --> 04:14.870]  just to give you an idea of how the parts of Greylog are kind of coupled together,
[04:14.870 --> 04:19.650]  so that you can, when you use it later, maybe a few things might make a little more sense. I just
[04:19.650 --> 04:25.930]  feel we should take a quick, just a minute, at the components of Greylog itself. Then, like I said,
[04:25.930 --> 04:32.450]  live demo. We're going to focus on analysis, because I think that's, as a user and as
[04:32.450 --> 04:36.970]  an analyst, that's probably the most important part for you, especially if you're part of OpenSoc
[04:36.970 --> 04:44.730]  this year. But we're also going to look at some differences if you're used to other tools.
[04:44.730 --> 04:49.250]  I think that could also be a really important kind of question that you might want to ask.
[04:49.250 --> 04:56.290]  If you are used to another log management tool out there, maybe if you have questions about how
[04:56.290 --> 05:00.550]  Greylog compares to that, from kind of how you use this, does Greylog do that, is there another way
[05:00.550 --> 05:05.090]  of doing that, so you can apply the knowledge that you have already, so that you can
[05:07.430 --> 05:12.610]  really apply this in a good way with a new product like Greylog. And then, like I said,
[05:12.610 --> 05:18.330]  we're going to have a lot of time for Q&A. For the questions that you have, please use the channel
[05:18.330 --> 05:22.790]  in the Blue Team Village Discord. If you're not in that yet, it's a good moment to just quickly
[05:22.790 --> 05:28.170]  sign up for that. Go to the Blue Team Village Discord, and in Discord you see these categories
[05:28.170 --> 05:33.490]  on the left in your sidebar. There's one that's called the Flamingo Hotel category. Made me miss
[05:33.490 --> 05:38.690]  Las Vegas very much when I saw that this morning. But there's the Flamingo Hotel category, and
[05:38.690 --> 05:43.650]  there's a channel in there that's called Text Workshop Track One. We are here right now,
[05:43.650 --> 05:47.430]  we're Workshop Track One, and that's a text channel. So that's where you can post your
[05:47.430 --> 05:52.290]  questions. The moment we're going into the Q&A phase, at the end of this, I'm going to go to
[05:52.290 --> 05:56.170]  that channel, and we can start to look at your questions and start answering them. All right?
[05:56.570 --> 06:02.010]  So, like I said, Text Workshop Track One. If you get lost somewhere in there after you join,
[06:02.010 --> 06:06.630]  like I said, there are moderators swarming that place, and they'll be happy to help you
[06:06.630 --> 06:11.150]  and guide you to the right channel. So just ask anywhere, and you should find it.
[06:11.750 --> 06:18.770]  About me, it's real quick. Lennart Koopman. My Twitter handle is at underscore Lennart,
[06:18.770 --> 06:25.270]  L-E-N-N-A-R-T. If you want to follow me on your own risk, you should definitely do that.
[06:25.570 --> 06:30.710]  My background is actually software development and architecture. So I've started my career as
[06:30.810 --> 06:37.390]  a software developer, and also started the Greylock project in 2009 already. I realized
[06:37.390 --> 06:42.410]  with horror that it's been 11 years already. It's been 11 great years, but 11 years is a pretty
[06:42.410 --> 06:48.750]  long time, and I felt like it was more like three years. But Greylock as an open source
[06:48.750 --> 06:57.030]  log management product started in 2009, I believe, actually pretty much on the day in 2009,
[06:57.030 --> 07:06.910]  like in early August. We had a nice little celebration at our last party
[07:06.910 --> 07:14.370]  over in Vegas. That was fun. I think there are pictures. I then started a company behind Greylock
[07:14.370 --> 07:21.110]  in 2013. That was after the open source project was showing some commercial interest
[07:21.630 --> 07:26.170]  from people out there. So they asked me, hey, can you give me support? And I said, hey, I don't know
[07:26.170 --> 07:30.790]  how to write an invoice. So that was a good moment to start a company behind it. That was 2013.
[07:31.130 --> 07:36.190]  I'm currently the CTO over here at Greylock. That's really what I started as, too.
[07:37.170 --> 07:42.370]  And I was born and raised in Germany. That's where my accent is coming from, if you can make it up.
[07:42.390 --> 07:49.130]  But I am living in beautiful Houston, Texas now, which is also where one of our offices is.
[07:49.130 --> 07:55.010]  The other offices are in Germany and then also straight out across the United States.
[07:55.070 --> 07:59.810]  I guess I have to say, we are hiring. If you want to go to greylock.org careers,
[07:59.810 --> 08:04.250]  there might be something that you could be interested in that's remote in Germany or the
[08:04.250 --> 08:10.450]  United States. But enough about me. Real quick about Greylock. This would be the moment at a
[08:10.450 --> 08:14.410]  conference or at a physical talk where I would ask who knows about Greylock, and then I would
[08:14.410 --> 08:20.270]  see hands go up or not go up. And then I could adapt this a little to the audience. So I'm just
[08:20.270 --> 08:26.630]  gonna tell you the whole story real quick. Open source log management. So you send Greylock or
[08:26.630 --> 08:33.530]  instruct Greylock to pull logs from somewhere. It will ingest these logs, it will parse them,
[08:34.070 --> 08:40.610]  and it will make them available for you for searching, analysis, threat hunting, for
[08:41.510 --> 08:46.530]  alerting, and also maybe to filter some out and say I'm actually not interested in these.
[08:46.530 --> 08:52.730]  Throw them away, route them somewhere else. Really a way to handle logs that are being generated in
[08:52.730 --> 08:59.930]  your environment. The three main use cases for that are dev or ops or DevOps, depending on what
[08:59.930 --> 09:05.130]  you're doing. Definitely very interesting for developers to do root cause analysis, to do
[09:05.130 --> 09:09.830]  monitoring, to do performance monitoring. Maybe when they know that there's an issue on the platform
[09:09.830 --> 09:16.250]  that they need to research or that they need to investigate, they use Greylock to go into
[09:16.250 --> 09:22.050]  the logs to figure out what exactly happened and to make sure that either it doesn't happen again
[09:22.050 --> 09:27.570]  or to put an alert on it or simply to resolve the issue that's causing it. Same for
[09:27.570 --> 09:35.210]  ops. For ops that could be just classic IT system ops. This could be network operations. I see a lot
[09:35.210 --> 09:40.270]  of people using it for that. The fastest growing use case has been for a while now security use
[09:40.270 --> 09:47.290]  cases, which I guess you as the audience here is probably very aware of, which basically means
[09:47.290 --> 09:52.250]  you're using it to keep a track record and a history of everything that's going on in the
[09:52.250 --> 09:57.470]  environment. That means you need to have a system that is able to really ingest all of your logs
[09:57.470 --> 10:04.670]  and to make sense of them and keep them for long enough so you can run your analysis against it.
[10:05.950 --> 10:11.950]  And it will then allow you to set up pretty complex alerting, like you would know it from
[10:11.950 --> 10:17.950]  some sim systems out there, for example. That's a feature that came out, I think, about a year
[10:17.950 --> 10:24.010]  or a year and a half ago. A new alerting engine that lets you help be a little more proactive
[10:24.010 --> 10:28.430]  about what's going on in your environment. But one of the other really classic use cases is you need
[10:28.430 --> 10:33.350]  to investigate an IP address. You need to investigate some kind of indicator of compromise. Something
[10:33.350 --> 10:38.270]  new came out and there's a new patch. You want to make sure that that specific file has not been
[10:38.270 --> 10:44.230]  anywhere in your environment in the last six months. Let's go in and let's see if that happened
[10:44.230 --> 10:52.030]  or not. And then, of course, classic DFIR incident response, figuring out if there was an attacker,
[10:52.030 --> 10:56.910]  what that attacker did, all of that fun stuff. And, of course, compliance, just having the
[10:56.910 --> 11:01.810]  logs somewhere. I think that is a valid use case. And for that, you need to centralize them.
[11:02.150 --> 11:07.770]  And I think Greylock is a really good choice for that, too. Like I said, started in 2009,
[11:07.770 --> 11:13.530]  really constantly grown since then. Very, very happy to have been a Blue Team Village sponsor
[11:13.530 --> 11:19.390]  really from the beginning. Like I said, absolutely fantastic people. Make sure to watch as much of
[11:19.390 --> 11:23.470]  their content as you can. And next year when we're hopefully all back in Vegas, make sure to stop by
[11:23.470 --> 11:32.190]  in person. It's always a lot of fun. And Greylock is also part of OpenSOC, the CTF that runs during
[11:32.190 --> 11:38.810]  Blue Team Village that has got so much positive feedback over the last years. Also, of course,
[11:38.810 --> 11:42.770]  extremely happy about that. And like I said, we'll be around to help you with any questions
[11:42.770 --> 11:50.150]  that comes up during your OpenSOC participation. Now, during OpenSOC or during the Blue Team
[11:51.090 --> 11:55.750]  Village, no booth this year. I was always happy to spend a day over there and just
[11:55.750 --> 12:01.130]  chat with people. Maybe give out some shirts and some Greylock swag. Because that is not
[12:01.130 --> 12:06.250]  happening this year. Make sure to stop by the Greylock channel in the Blue Team Village Discord,
[12:06.250 --> 12:13.230]  because I hear that we will be sending out swag from there. And then we'll be there to answer
[12:13.230 --> 12:21.450]  any of your questions. Very important is that OpenSOC is not running the latest version of
[12:21.450 --> 12:27.050]  Greylock yet. They're slightly behind it for some very good reasons. So the demo that I'm going to
[12:27.050 --> 12:31.690]  show you is the latest version. If you see something that's slightly different, definitely
[12:31.690 --> 12:38.030]  reach out to us in the Discord channel and we'll be happy to give you a maybe a workaround or
[12:38.030 --> 12:44.330]  tell you where that is. Or if it's simply not there and what else you could do about that. So
[12:44.330 --> 12:48.750]  just keep that in mind. But there's some structural differences about where
[12:48.750 --> 12:54.630]  some elements are. But overall, it's pretty much the same. And especially the core
[12:54.630 --> 12:58.870]  ideas and the core principles are of course the same, because they simply haven't changed in the
[12:58.870 --> 13:05.510]  last months. And then most importantly, have fun. Stay in touch. It's open source. So if you
[13:05.510 --> 13:10.330]  get interested in this, just start using it. Start running it yourself. It's designed for that.
[13:10.770 --> 13:16.330]  And there are many ways to install and run Greylock, especially if you maybe just want to
[13:16.330 --> 13:21.190]  try it out really quick. There's some ways to get something up and running really fast. There's
[13:21.250 --> 13:26.050]  a big community around this that can always help you out. So I want to say have fun and open
[13:26.050 --> 13:30.470]  SOG. I've only heard fantastic things about how much fun it is. So I'm sure you will have fun.
[13:30.470 --> 13:34.990]  And even if you're not participating in it, just download it. Give it a try. It's open source.
[13:35.290 --> 13:43.430]  That's what it's all about. So the next slide here promises a live demo. Before we do that
[13:43.430 --> 13:48.790]  I want to send you straight to the docs real quick. So we can look at the...
[13:50.650 --> 13:56.010]  I have to find where they are. First link here. Architectural considerations. And we will not be
[13:56.670 --> 14:04.190]  detailed. No worries. But I want to show you just really quick the parts that Greylock
[14:04.190 --> 14:12.310]  consists of. If you ever get stuck somewhere with Greylock, always go to docs.greylock.org
[14:12.310 --> 14:18.510]  if you feel like just looking it up yourself. That is up here. Docs.greylock.org. That's where
[14:18.510 --> 14:25.870]  all of our documentation lives. And if you can't figure it out there because your use case might
[14:25.870 --> 14:30.990]  not be described in there, then, like I said, reach out to us. But that's always a good place
[14:30.990 --> 14:36.390]  to get started. And I'm just using... you won't have to deal with the architecture because the
[14:36.390 --> 14:41.570]  open SOG folks are already setting that up for you. But just so you understand it, Greylock
[14:41.570 --> 14:46.110]  really consists of three big parts. Or three and a half big parts, I would say. There's Greylock
[14:46.110 --> 14:51.210]  itself, which consists of Greylock server and Greylock web interface. But it's really one
[14:51.990 --> 14:57.330]  system that you're running or one executable that you're running. And it has two dependencies.
[14:57.530 --> 15:03.190]  It connects to MongoDB. It does that to store some state, to store some cluster communication,
[15:03.190 --> 15:08.550]  to store some configuration that lives in MongoDB. And because there's really not much data, that
[15:08.550 --> 15:12.570]  MongoDB is really, really small in the end. Actually, most people will just run it on the
[15:12.570 --> 15:21.790]  same node as the Greylock server. And that way, you can almost forget that it's there and set it
[15:21.790 --> 15:27.870]  up in a replica set to make it highly available. The Elasticsearch part, though, that's what we're
[15:27.870 --> 15:34.030]  using to store the messages. And so that will, of course, you have to scale that properly because
[15:34.030 --> 15:38.530]  that gets all the load from searches. And it gets, of course, all the load from actually writing the
[15:38.530 --> 15:45.090]  messages to disk. And what happens is, if you connect to the Greylock web interface, it will
[15:45.090 --> 15:52.250]  basically execute API calls against the Greylock server. And it will say, hey, my user of the web
[15:52.250 --> 15:56.430]  interface is trying to search for these messages. And then the server says, yep, I'm going to search
[15:56.430 --> 16:02.110]  for them, queries Elasticsearch, gets the data back, does all its funky Greylock stuff with it,
[16:02.110 --> 16:07.210]  sends it back to the web interface, and the web interface displays it. I'm mentioning that because
[16:07.210 --> 16:14.670]  all the communication between the web interface and the Greylock server is really API calls,
[16:14.670 --> 16:21.790]  nothing else happening. All it is, it's just authenticated HTTP REST API calls. That means
[16:21.790 --> 16:26.290]  that you can also automate everything. So if you are getting at a point where you want to
[16:26.290 --> 16:32.250]  write a script to look at data, for example, you can always use the Greylock API. There's an API
[16:32.250 --> 16:36.930]  browser in Greylock, and you can start to write your Python scripts and start to pull that data
[16:36.930 --> 16:41.490]  out of Greylock and do something with it. That's kind of my secret sauce tip for if you're working
[16:41.490 --> 16:47.470]  in OpenSoC for sure. If you're good at Python or another language, then definitely keep that in
[16:47.470 --> 16:52.550]  mind that you can do that. But overall, Greylock server accepts the messages, you send the messages
[16:52.550 --> 16:58.730]  to the Greylock server, use the web interface to analyze, display, configure, set up alerts,
[16:58.730 --> 17:03.610]  all of that fun stuff. And then Greylock behind the scenes will do all the management of Elastic
[17:03.610 --> 17:08.590]  Search. You can define your retention times, all of that. So if you're maybe used to the Elastic
[17:08.590 --> 17:13.670]  Stack where you have to do a lot of stuff in Elastic Search itself, for Greylock you really
[17:13.670 --> 17:17.850]  just have to tell Greylock where the Elastic Search is and then Greylock takes care of the
[17:17.850 --> 17:22.850]  rest for you so you don't have to become an Elastic Search expert. I think that's a very
[17:22.850 --> 17:29.710]  important part. If you want to, you can scroll down here and look at what complex looks like
[17:29.710 --> 17:34.610]  with multiple nodes, clustering, load balancer in front. I think that is definitely out of scope
[17:34.610 --> 17:41.550]  right now for what you are about to do here or if you just want to give it a try. But be aware
[17:41.550 --> 17:46.630]  that this is all documented here and then there's also another setup guide that talks you through
[17:46.630 --> 17:53.010]  that. So it should be a good place to get started. All right, so now we're going to the real life demo
[17:53.010 --> 18:01.650]  part. And what I'm going to do is I'm going to walk you through... I made some
[18:01.650 --> 18:07.870]  notes here. I'm going to work with one, two, three, four areas really. And I want to spend the most time
[18:07.870 --> 18:12.930]  in the search and analysis. Like I said, I think that's where most of your time will be spent.
[18:12.930 --> 18:18.910]  But I just want to show you a few other parts of the system, at least real quick, so you are aware
[18:18.910 --> 18:25.270]  that they are there. So I always like to start in the system area. And that is simply
[18:26.330 --> 18:30.490]  because I think that gives you an idea of kind of the philosophy behind Greylock itself.
[18:31.010 --> 18:37.130]  Our idea is that you should have to set up a Greylock in a config file only initially
[18:37.130 --> 18:44.010]  and only once. Basically tell Greylock, where's my MongoDB search? Give it some data it needs
[18:44.970 --> 18:49.610]  to encrypt passwords, to set up TLS on the web interface, kind of this basic setup.
[18:49.810 --> 18:53.930]  And then from there on, we believe that you should be able to do everything from within the web
[18:53.930 --> 19:00.290]  interface. Like I said, those web interface calls are really just API calls. So if you want to use
[19:00.290 --> 19:06.470]  automation, you can just use curl to set up stuff automatically. But during, I would say,
[19:06.470 --> 19:12.090]  after a base setup, everything is configured in Greylock itself. And then Greylock keeps
[19:12.090 --> 19:17.070]  the configuration basically in MongoDB for you. What that means is, I just want to give you a real
[19:17.070 --> 19:22.370]  quick overview. We're going to go into that very deeply. But if we look at our pipelines, for
[19:22.370 --> 19:30.350]  example, that is where all of the message processing of Greylock is happening. This is not some API
[19:30.350 --> 19:37.970]  call where you have to post some cryptic language to. This is not a huge YAML file or JSON configuration
[19:37.970 --> 19:43.590]  of that. This is all, no, here we go. Here comes my Linux syslog. And I'm going to say,
[19:43.590 --> 19:48.410]  I want to parse some stuff out of it. And you just basically configure Greylock here with a
[19:48.410 --> 19:54.130]  very efficient and small rule language that then says, OK, everything has Linux syslog. Apply this
[19:54.130 --> 19:58.470]  regular expression on it. Extract some fields. You'll see here on the right, this is all the
[19:58.470 --> 20:04.230]  functions you can execute on it. You can drop a message. You can check what a message looks
[20:04.230 --> 20:09.650]  like and then define other actions on it. You can thread Intel lookups on it. You can run a
[20:09.650 --> 20:15.910]  Whois lookup on it. DNS reverse lookup on it. Run it against your own lookup tables. You can
[20:15.910 --> 20:20.790]  check if something is in a certain subnet. All of this processing happens. And this is very important.
[20:20.890 --> 20:23.770]  This is a very important thing that you need to understand, especially if you come from other
[20:23.770 --> 20:37.250]  systems. All the processing happens when the message arrives in Greylock, OK?
[20:37.450 --> 20:41.890]  That means that once a message has been written,
[20:42.530 --> 20:46.950]  that means that that message basically becomes immutable. That means that you cannot change it
[20:46.950 --> 20:52.030]  afterwards. That has some positive side effects and some side effects that you might have to
[20:52.030 --> 20:57.190]  work around. What that means, most importantly, for you, if you're coming from a system that
[20:57.190 --> 21:04.270]  allows you to transform data while you search, Greylock simply won't do that for you. Especially
[21:04.270 --> 21:08.950]  not in a way that you can then further search on. Everything that's in Elasticsearch is in
[21:08.950 --> 21:14.570]  Elasticsearch as it is. You can't transform the messages during search time. Very important to
[21:14.570 --> 21:19.770]  keep in mind. Of course, we've built our whole web interface and the processing in a way that
[21:19.770 --> 21:25.730]  you can still do all the good stuff with it. And it also means that we're getting a huge
[21:26.850 --> 21:31.930]  usability and performance hit from that in a good way, as in it's much easier to use. We hear a lot
[21:31.930 --> 21:36.970]  of people who come from these schema-on-read systems who say that is useful for a bunch of
[21:36.970 --> 21:43.790]  queries. But overall, just being sure what data you're already looking at and not having to mess
[21:43.790 --> 21:49.190]  with super complex queries all the time is actually something that in the long term people
[21:49.190 --> 21:56.230]  really, really start to like. So just be aware of that. You'll see in the inputs area, you'll see
[21:56.230 --> 22:00.930]  where all your messages are coming in. We've got a whole section here for authentication, if you
[22:00.930 --> 22:08.870]  want to connect that to LDAP or an AD. We have API tokens. There's all kinds of authentication
[22:08.870 --> 22:14.310]  mechanisms happening here. And also, that's the last thing I really want to touch on real quick,
[22:14.310 --> 22:20.130]  we have content packs. So if you want to export any kind of configuration, dashboards, reports,
[22:20.130 --> 22:26.890]  alerts, anything like that, and insert it into a different setup, you can export and import
[22:26.890 --> 22:31.750]  these things here using a feature called content packs. We've got outputs. There's a whole index
[22:31.750 --> 22:37.570]  management here, how long you want to keep data. Then after that data has been deleted, you want
[22:37.570 --> 22:43.810]  to archive, you want to close it, you want to delete it. All of that happens here. It's, I think, for you
[22:43.810 --> 22:48.870]  for now, and especially if you're just getting started, not super relevant yet. Just be aware that
[22:48.870 --> 22:53.950]  Greylock has that, Greylock does that for you. So always a good idea to maybe just click around in
[22:53.950 --> 22:59.570]  the system area a little. The other thing I just also really want to tell you, so you know that it's
[22:59.570 --> 23:08.070]  there. I am honestly not sure how much of that can be used during a CTF, but know that there
[23:08.070 --> 23:14.210]  is an alerting engine. I have here, this is kind of my small demo setup. I have an alert setup
[23:14.210 --> 23:20.150]  here that alerts me every time there's a high priority IDS alert, Greylock, and I have another
[23:20.150 --> 23:25.930]  one, and we can actually look at those. I live in southeast Texas, so we're pretty good at storms,
[23:25.930 --> 23:30.310]  and I live in the United States, so we're pretty good at not burying our power lines.
[23:30.310 --> 23:36.330]  So that combination will definitely require for you to have some kind of an uninterrupted
[23:36.330 --> 23:43.210]  power supply. So I'm reading the logs from one of my UPSs, sending those into Greylock, and I'm
[23:43.210 --> 23:48.930]  triggering alerts to my iPhone when the power goes out, because I really want to know about that,
[23:48.930 --> 23:52.570]  because maybe if it doesn't come back up within five minutes, I'm just going to start to shut
[23:52.570 --> 23:57.890]  down some servers remotely real quick, or do something else. So I have an alert setup here,
[23:57.890 --> 24:04.490]  let me look at this first. Under event definitions, I have a condition set up here called
[24:04.490 --> 24:12.090]  UPS without external power. Click on that. Really what it does is, it says when you find a message
[24:12.650 --> 24:19.330]  that has its status field set to OB discharge, which is the UPS reporting that it is now
[24:19.330 --> 24:26.170]  discharging, which means it has no power, I say when that happens, I don't need aggregations,
[24:26.170 --> 24:31.390]  I don't need groupings, I don't need any other further rules and statistics. Just the moment
[24:31.390 --> 24:37.210]  that happens, I want to be sent a message. Set this up here on notifications, I'm using Ops Genie,
[24:37.210 --> 24:43.310]  which is the system that Greylock can post these alerts to, and it will then send an alert on my
[24:43.310 --> 24:48.610]  phone. I say when that happens, send me a notification, and you can actually see here,
[24:48.610 --> 24:55.590]  if I go to my alerts and events, and I filter my events by UPS, you will see, because I'm looking
[24:55.590 --> 25:00.410]  here at the last 360 days right now, usually you would probably look at the last day or so,
[25:00.410 --> 25:06.290]  you'll see now a history of all the days when my UPS was out of power. So you see here, for
[25:06.290 --> 25:13.470]  example, just about almost exactly a month ago, my UPS was without external power and it did
[25:13.470 --> 25:19.230]  trigger a notification to my phone. So you can set up these things, you can set up
[25:19.230 --> 25:25.090]  all kinds of conditions, I can say only send me this if certain events happen in a
[25:25.090 --> 25:29.370]  certain order, all of that is there. Also, I don't think anything from this demo today, but
[25:30.010 --> 25:37.330]  just something to be aware of and know that it's there if you want to use it.
[25:38.170 --> 25:41.930]  Then we have the streams, and after that I'm going to go into search and analysis and give
[25:41.930 --> 25:47.130]  you a quick overview of that. But we have the streams here. Streams are basically, think of
[25:48.150 --> 25:52.690]  categorization of messages when they're going into Greylock. They are an important concept
[25:52.690 --> 25:57.970]  for Greylock because, for example, our retention settings are sitting on top of that. So if I say
[25:57.970 --> 26:04.790]  I want to keep my NetFlow logs for two days and my domain controller logs for two months,
[26:04.790 --> 26:09.770]  I would route them into two different streams with a very simple rule. This one here, for example,
[26:09.770 --> 26:18.790]  if the field Greylock type matches exactly the value NetFlow, then route this into the
[26:18.790 --> 26:24.530]  stream NetFlow. Another one here would be this one. Yeah, if the message matches the regular
[26:24.530 --> 26:30.170]  expression filter log at the beginning of the string, then route it into my firewall logs.
[26:30.170 --> 26:37.370]  I think here's one for, yeah, this is anything that has a snort process ID in it. Route that into my
[26:37.370 --> 26:42.070]  IDS alerts. So you're basically pre-configuring or pre-configuring messages and you put all kinds
[26:42.070 --> 26:48.910]  of stuff on top of that. Like further processing, extractions, enrichments, you can put access
[26:48.910 --> 26:53.810]  rights on top of that and say only team A is allowed to see my AD controller logs, but my
[26:54.670 --> 27:00.590]  support team is allowed to see, for example, my database debug logs or something like that.
[27:00.710 --> 27:07.090]  Same for retention times. A bunch of concepts in Greylock streams. I think for you, if you're
[27:07.090 --> 27:11.650]  getting started with this or using it during OpenSoC, I think it's just a great way to make
[27:11.650 --> 27:17.170]  certain types of messages available with one click. Okay, so we're now going into search and
[27:17.170 --> 27:21.970]  analysis. If I wanted to see all my firewall logs, I just click on the firewall stream
[27:22.530 --> 27:29.450]  and it opens a search page that has the stream selector here on top already pre-configured to
[27:29.450 --> 27:33.510]  only look at my firewall logs. I can select multiple here. So if I wanted to see all my
[27:33.510 --> 27:40.630]  firewall and all logs, you just do that. This thing's going to update. Here we go. And now
[27:40.630 --> 27:44.710]  we also got my NetFlow logs in it. For this example, though, I'm only going to look at my
[27:44.710 --> 27:49.930]  firewall logs. I'm going to take this out again. Every time I make a change to a query, you will
[27:49.930 --> 27:54.790]  see that this is... you get this little yellow icon here, which means, hey, you haven't executed
[27:54.790 --> 27:59.210]  your new search yet. I don't believe this is the case in the version you're going to be playing
[27:59.210 --> 28:03.810]  with. But if you download it fresh, that's definitely going to be the case. So I have to manually execute
[28:03.810 --> 28:09.510]  that one more time. What we're looking at here is our search page. So I think this is what you're
[28:09.510 --> 28:14.890]  going to spend the most time. So let me walk through the main areas, and then we can run more.
[28:16.130 --> 28:23.010]  Maybe execute a search query or two, and then I would say we go into Q&A. And I will ask a
[28:23.010 --> 28:27.890]  moderator to let me know when we run out of time during the Q&A. I think we started a little
[28:27.890 --> 28:35.090]  late. So just let me know what the best time for you as the organizers is. But you have three
[28:35.090 --> 28:40.930]  main areas here. You've got the search query bar here on top. This is basically where you are
[28:40.930 --> 28:47.810]  selecting what exactly you want to be searching for, in which stream, what query, and also in
[28:47.810 --> 28:53.010]  which time range. That's always the first thing you've got to think about. Then you've got a site
[28:53.010 --> 28:58.050]  bar here on the left. I think the important part of this is the fields area that shows you all the
[28:58.050 --> 29:04.170]  fields that are in these messages. So pre-parsed fields in the message.
[29:04.650 --> 29:09.410]  And then, of course, you've got the actual result set. So everything you're seeing here, everything
[29:09.410 --> 29:17.250]  under the... is your result set. So that is actually everything that Draylog returns for your search
[29:17.250 --> 29:21.450]  query. So let's say I don't want to search in the last five minutes. I want to search in the last
[29:21.450 --> 29:30.710]  hour. Change that. Execute it. Here we go. So now we're looking... you saw that this message
[29:30.710 --> 29:36.750]  count chart here changed. We're now looking at the whole hour. And I think that we can also...
[29:36.750 --> 29:43.730]  I don't think that's this one yet. But you can also definitely generate a widget that shows
[29:43.730 --> 29:47.390]  you how many messages were found in total. We can do that in just a second, actually.
[29:48.890 --> 29:54.390]  So let's say we're looking at all my firewall logs here in an hour. You see that we have a little
[29:54.390 --> 29:58.650]  bit of a spike here. And this is... firewall logs are always great because they have a lot of data
[29:58.650 --> 30:03.510]  in it, how much value you actually get out of them. So I think it's up to you. But it's a great example.
[30:03.730 --> 30:09.890]  And overall, the searches work the same for everything, no matter if it's a firewall log, a
[30:09.890 --> 30:16.050]  Windows log, a Linux log, a Amazon API log, anything like that. So if I click on it, you see here this
[30:16.050 --> 30:21.650]  is the original message that came into Greylock. Very hard to read. If I click on it, you get the details
[30:21.650 --> 30:27.070]  and you get what Greylock has actually returned. So you see that Greylock here has parsed this out
[30:27.070 --> 30:32.190]  into some fields. Like, for example, the destination address and destination port,
[30:32.190 --> 30:38.170]  header length, which interface it came in, which IP version it was, the length of the whole
[30:38.670 --> 30:45.370]  of the whole packet that was let through, which protocol it was. And you'll also see here that
[30:45.370 --> 30:49.830]  in this case, the action was block. Everything that I'm looking at here on these logs,
[30:50.550 --> 30:55.110]  I'm only sending me the block logs. So I'm not interested in what the firewall let through. I'm
[30:55.110 --> 31:01.510]  only interested in what the firewall actually blocked. So with all of this, you also see source
[31:01.510 --> 31:08.170]  address and source port. Let's say we only want to see the messages from a certain source,
[31:08.170 --> 31:14.110]  where the source address is set to this value. What I could do is I could go into the search
[31:14.110 --> 31:22.370]  query bar on top here and say source underscore address equals, and then 1356, whatever I want
[31:22.370 --> 31:30.430]  to look for. Much faster is to just simply click on the value. This context menu opens,
[31:30.430 --> 31:36.150]  and you say add to query. And it will just build that for you and immediately execute.
[31:36.270 --> 31:43.470]  So what we're now looking at is we're looking at all blocked packets from the last hour where
[31:43.470 --> 31:47.070]  this was the source address. So basically all the packets this source address sent us
[31:47.830 --> 31:56.070]  locked. Okay, so let's try to get a little bit of a better overview here of what that looked like.
[31:56.070 --> 32:02.370]  So I want to find out what's this all TCP or both. Do you see we have a field here called
[32:02.370 --> 32:10.430]  protocol. Try to get the top values. I want to see they're all set. I see something that's a
[32:10.430 --> 32:19.610]  protocol. Show top values. This opens, and you see that no, actually this source only sent
[32:20.730 --> 32:27.930]  UDP messages. Okay, let's go a little deeper and say is this always the same kind of package?
[32:27.930 --> 32:35.410]  Do they all have the same length? Like very, very uniform? Same thing. Data length. Show top values.
[32:35.470 --> 32:40.410]  Here we go. We see that actually all these packages are UDP and exactly 278 bytes long.
[32:40.410 --> 32:45.170]  That looks like we're just blocking the same package over and over and over again.
[32:45.190 --> 32:50.550]  Now what would be interesting, this setup isn't doing it, but if we were to run a
[32:50.550 --> 32:55.510]  lookup. That's a built-in lookup table onto the source address or our GeoIP to see where
[32:55.510 --> 33:00.890]  this is coming from. Who owns this IP address? Look up the underlying ASN. Do all of that stuff.
[33:00.890 --> 33:04.990]  I'm simply not doing that here in this setup, but that's all stuff that you could do.
[33:05.730 --> 33:14.370]  And now let's maybe say I want to see... now let's actually go back. I'm going to execute
[33:14.370 --> 33:19.310]  this query again. And what you're going to see is all the data in the result set, because the
[33:19.310 --> 33:25.490]  result set change, has now updated. Now we see for the last one hour, for everything that was blocked,
[33:25.490 --> 33:32.730]  UDP, TCP, and ICMP. Okay, I'm going to delete this widget here. I want to show you the aggregation
[33:32.730 --> 33:37.370]  builder. That's really the last part I want to show you here. For the search query language,
[33:37.370 --> 33:42.330]  go to docs.greylock.org. There's a whole page that explains the search query language for you.
[33:42.470 --> 33:46.190]  I don't want to go into too much detail here, because you can simply read it up there.
[33:46.190 --> 33:49.330]  It's pretty simple. It's in the end, if you've been using the Elastic Stack,
[33:49.330 --> 33:53.430]  it's the same language. We're just passing it through Elastic. So if you're used to that,
[33:53.430 --> 34:01.370]  same thing. So whenever Greylock creates a widget, that is under the hood. In fact,
[34:01.370 --> 34:05.610]  an aggregation and an aggregation builder. So you will always see this little icon,
[34:05.610 --> 34:12.310]  top right of the widget, that you can click on and say edit. And now edit is going to show you
[34:12.310 --> 34:18.410]  the aggregation builder. And it has one very important drop down box, which is the visualization
[34:18.410 --> 34:28.050]  type. Let's say you want to visualize this not as a data table, but as a bar chart. And say,
[34:28.050 --> 34:32.970]  I really want to see three bars here, with UDP the largest, TCP on second, and then a tiny one
[34:32.970 --> 34:40.710]  for ICMP based on the count. You switch it up. Art chart. Here we go. Save it. I could now make
[34:40.930 --> 34:46.910]  a dashboard out of this. Here we see for the last hour, we mostly brought UDP stuff. Okay,
[34:46.910 --> 34:51.790]  interesting. You could, of course, if you really wanted to, make it... I think they're much harder
[34:51.790 --> 34:57.150]  to read, but sure, could do that. Greylock is not going to keep you from building weird charts
[34:57.150 --> 35:04.790]  either. So you want to make this a long chart? Sure, not a problem. Can do that. So this is our
[35:04.790 --> 35:09.290]  aggregation builder. You can do sorting. You can add multiple rows. We're going to do that in just
[35:09.290 --> 35:19.130]  one second. And you can also add metrics. So let's say I want to get the total bytes that were blocked
[35:19.130 --> 35:25.510]  over the last one hour. Okay, so you know we got the data length field here. Click on that,
[35:25.510 --> 35:32.210]  press chart. That is going to create a new widget, and that widget is basically just the same
[35:32.210 --> 35:36.830]  aggregation builder, but with other predefined values. Okay, you're going to see that in one second.
[35:36.850 --> 35:46.450]  Here we go. So now we got the average data length by timestamp. Click on edit. See, okay, that's our
[35:46.450 --> 35:53.730]  line chart. That's how much data was transferred per minute, in this case, on average. But you
[35:53.730 --> 35:57.590]  know what? Let's make this a data table, so you see what this looks like under the hood.
[35:57.690 --> 36:02.410]  It's like for each minute, it calculates the average data length. And with this kind of column
[36:02.930 --> 36:08.930]  row structure, you can put that into a bar chart. You can put this into a line chart.
[36:08.930 --> 36:14.370]  But I think what we're really interested in here, we don't want to look at the average data length.
[36:14.610 --> 36:19.230]  We want to look at the sum of the data length.
[36:19.230 --> 36:29.170]  Here we go. Data line chart. And now we see how many bytes are blocked in total. I just
[36:29.170 --> 36:36.970]  had to know that this was called sum. You can do standard deviation, max, min, count, cardinality,
[36:36.970 --> 36:43.170]  average, variance, percentile. You can do all of that stuff with it if you wanted to. Okay.
[36:43.790 --> 36:46.710]  One more thing I want to show you, and then we're going to go into Q&A.
[36:47.330 --> 36:51.270]  I really want to make sure that you start playing around with aggregation because that one
[36:51.270 --> 36:56.650]  is extremely powerful. It will really help you solve these challenges that OpenSock is going
[36:56.650 --> 37:01.650]  to give you. One more thing I want to show you is, let's say we want to build an aggregate.
[37:01.650 --> 37:14.130]  We say grouped by source address which protocol was used. Okay. So let's go into the source
[37:14.130 --> 37:21.870]  address. Get the top values of that as kind of a start. Add it. And we can add multiple rows here.
[37:21.870 --> 37:28.150]  So let's set this to protocol. I want to add a second type of row which is protocol.
[37:29.070 --> 37:33.970]  It's going to update. And you actually see here that these have all only communicated
[37:34.830 --> 37:40.250]  with UDP or TCP. They're not really doing multiple things. So to explain this a little
[37:40.250 --> 37:46.390]  better, let's build pairs of source address to destination. Take the protocol out. Add a new field,
[37:46.390 --> 37:53.970]  destination address. Here we go. So now you see this is not really what I wanted to show.
[37:53.970 --> 37:59.390]  It should be the other way around. Maybe destination address to source address. Here we go. That's what
[37:59.390 --> 38:04.910]  I wanted. So you get this grouping where you say this destination address was reached by these
[38:04.910 --> 38:09.890]  source addresses. And you can go lower and lower and lower as you wish and add more and more stuff.
[38:09.890 --> 38:14.010]  So let's say as the metric, here's the third column, I actually don't want the count.
[38:14.010 --> 38:21.790]  I do actually want the total bytes that were transferred. So the sum of the data length.
[38:22.590 --> 38:30.690]  So without any kind of complex queries, you establish that this destination address
[38:30.690 --> 38:37.250]  was hit by this source address with this many bytes over the last hour. And I can make
[38:37.250 --> 38:41.090]  this a little larger, put this wherever I want. I could now save this as a dashboard,
[38:41.090 --> 38:45.230]  share it with colleagues. There's a whole concept of parameters in there. It's all stuff
[38:45.230 --> 38:51.710]  I don't want to go into today. But that's all stuff that you can do. And you can also really
[38:51.710 --> 38:56.630]  start to combine this in very many different ways. What I can tell you is this might be,
[38:56.630 --> 39:01.590]  this might look like it's a pretty good learning curve associated with all these fields in the data
[39:01.590 --> 39:06.650]  table and the visualizations. I would encourage you just to play around with it. The worst thing
[39:06.650 --> 39:11.390]  that can happen is that it shows you either a chart that doesn't make sense or that it shows
[39:11.390 --> 39:18.070]  you an error when something really doesn't compute. Okay, you can't break anything. Just really start
[39:18.070 --> 39:21.850]  to play around with this stuff. Maybe think about something you want to visualize and see how you
[39:21.850 --> 39:27.830]  would approach that. If you get stuck, go into Discord and there might be someone who can help
[39:27.830 --> 39:33.950]  you from our team. And also, whenever you feel like, okay, this stops making sense,
[39:33.950 --> 39:39.190]  go back to the data table and look at what underlying data we're really looking at. Once
[39:39.190 --> 39:44.430]  you have multiple dimensions, you can do stack, bar charts, you can do all kinds of stuff. So,
[39:44.430 --> 39:48.850]  I would encourage you to play around with that. This is really an absolute core idea of
[39:49.630 --> 39:56.630]  what CorelDoc does and what you can do with it. So, with that, I would say that's my demo.
[39:57.230 --> 40:04.910]  And as many minutes as we have left, I would use for Q&A. And like I said, if a moderator could
[40:04.910 --> 40:11.250]  let me know if we're kind of starting to run short on time or when it's time for me to stop.
[40:11.390 --> 40:15.130]  I would say we're going to do five minutes. I really, I honestly hope we have a little more
[40:15.130 --> 40:20.150]  time. I'm going to do five minutes of Q&A. I'm breaking my promise here a little of having
[40:20.710 --> 40:25.790]  time, but I'm going to promise I'm personally going to be around in that Discord channel
[40:25.790 --> 40:31.290]  for the next days. So, everything that's left after these five minutes, I'm just going to
[40:31.930 --> 40:36.210]  answer there. And like I said, there are a bunch of other people too, and I hear that it's whack.
[40:37.630 --> 40:41.870]  So, Linux on the desktop, I do not know how to hide it when you can't see it. So,
[40:41.870 --> 40:47.150]  I'm going to go straight into that. I'm going to go straight onto this one here,
[40:47.150 --> 40:49.610]  and I'm going to try to find some questions, okay?
[40:50.550 --> 40:53.030]  We have about two minutes, if you don't mind.
[40:53.170 --> 40:55.550]  Okay, understood. Thank you very much.
[40:55.550 --> 40:59.930]  We'd just like to move the questions over to the text window, or the chat, I'm sorry,
[40:59.930 --> 41:02.190]  if we run out of time. Thank you.
[41:02.250 --> 41:08.910]  Understood. So, I see a few questions here. I'm going to skim through a lot of them,
[41:08.910 --> 41:17.190]  but I see a question about if there is an OVA. There is, if you go to Greylock.org,
[41:17.190 --> 41:22.250]  and you go to Get Greylock, big button here, these are all the ways that you can install it.
[41:22.250 --> 41:28.450]  There's Docker images, OVA, virtual appliances. We're integrating with Chef, Puppet, Ansible,
[41:29.050 --> 41:33.790]  and there are Debian and RPM packages to get started really quick, and the documentation
[41:33.790 --> 41:39.490]  is going to guide you through doing that. Okay, let me scroll down a little more here.
[41:40.550 --> 41:46.310]  This is available on Linux, and the best way to set up the environment is described in the
[41:46.310 --> 41:50.550]  documentation. In fact, it only runs on Linux. It does not run on Windows.
[41:51.490 --> 41:57.790]  So, I would definitely recommend to run this on Linux. All of the official
[41:58.590 --> 42:02.050]  installation methods are going to be based on Linux, for sure.
[42:03.270 --> 42:07.210]  I'm going to pick one more, and then I'm going to stick around in the channel, and also in the
[42:07.210 --> 42:16.350]  Greylock channel, please. Let's see if I find one more. There we go. That's a good one. Is there
[42:16.350 --> 42:21.930]  some kind of how-to for reading in a log file written by another application? So, you have
[42:21.930 --> 42:26.190]  another server, and there's something running on it that writes a local log file, right? How do I
[42:26.190 --> 42:33.950]  add these into my Greylock now? There are several ways of doing that. Greylock has a concept called
[42:33.950 --> 42:40.430]  sidecar to even remotely manage of collectors. That is, however, not required. You can also
[42:40.430 --> 42:47.550]  just use any kind of open source collector that's out there. I personally recommend FileBeats or
[42:47.550 --> 42:53.550]  NXLog. FileBeats is coming from Elastic. They can send straight to our inputs, and you really just
[42:53.550 --> 42:59.630]  tell it, hey, here's a log file, and please send me everything that goes into this Greylock system
[42:59.630 --> 43:06.870]  with a buffer locally. So, look at FileBeat or NXLog. There's also WinLogBeat and a PacketBeat,
[43:06.870 --> 43:14.950]  and NXLog is also great for reading properly. And so, with that, I will keep my promise and stick
[43:14.950 --> 43:20.190]  around for more questions that are coming in. Follow me on Twitter if you have questions in
[43:20.190 --> 43:26.470]  the long term. That's at underscore Leonard, L-E-N-N-A-R-T, and I hope you enjoyed that,
[43:26.470 --> 43:32.330]  and I really, really hope that you had a fantastic time playing OpenSoC or trying out Greylock.
[43:32.330 --> 43:35.810]  We're building this for you, for the community in the end, and so let us know
[43:35.810 --> 43:39.470]  if anything doesn't work, if anything doesn't make sense, if you want to prove anything,
[43:39.470 --> 43:45.070]  it's open source. Use it, play it, improve it. That's it. Thank you very much.
[43:46.790 --> 43:50.330]  Thanks a lot, Leonard. I appreciate your time. A great presentation. We look forward
[43:50.330 --> 43:51.470]  to hearing more from you guys.