start	end	text
15040	20280	We have another great speaker for you today, Dr. Pinky, presenting
21000	24340	Discovering ELK, the First Time Lessons Learned Over Two Years.
24660	28740	And with that, I will let Dr. Pinky take it away.
29440	33800	Hey guys, thank you so much for joining me today. I know it's a little late in the evening.
34160	37560	I know things start a little bit late with Pacific, but we're kind of good in the afternoon crash,
37560	42060	evening crash, depending on your time zone. So hang in with me and let's have some fun, OK?
42840	46220	So about me real fast, other than the fact that I have pink hair, obviously.
46740	51980	I'm a Linux and Windows threat hunter. I prefer Linux, but for some reason, Windows seems to have the marketplace.
51980	56480	Don't know why. Can't be its usability or anything. So I end up there most of the time.
56780	60600	I'm a huge fan of scotch, whiskey, and bourbon. You know, the main things you actually care.
61200	65780	I'm over here in San Antonio, Texas, and as a result, I help coordinate various groups here,
65780	71420	including B-Side San Antonio and Saha. And of course, the mandatory disclaimer,
71420	76580	all views, thoughts, opinions, anything else I say here are my own and are representative of my employer.
77080	81340	With that out of the way, let's go into what we're talking about today.
81500	86080	First off, I'll just lay out some quick expectations, just so we're all on the same page.
86420	91260	And I know most of you, since we're blue teamers for the most part here, are probably familiar with Elastic Stack,
91260	95820	but I'm going to get into it just so we're using all the same terminology with the same understanding.
96640	101300	And then we'll go into four fun lessons I learned the hard way these last two years.
101300	106020	And I'll admit, some of them are pretty embarrassing. So hopefully you get a few good laughs out of this.
106500	113720	So with that, let's get into it. So after two years, you may think, hey, I should be an expert by now.
113720	117580	But no way. This keeps changing every day, it feels like.
117580	123600	And every time I think I get a grasp of something, it changes. So that's the fun thing about our community, right?
124300	129880	This talk is geared toward analysts, threat hunters, things of that nature, not toward building or deploying stacks.
129880	136680	So anything I say here could, you know, crash your product or your, you know, stack and product.
136980	140960	Sorry, life stack. Don't do it. Play first, then deploy.
141640	144440	This is aimed at giving you a starting point to research and play.
144440	148640	Again, please don't go, hey, let's go live with this right away.
149300	153800	And again, this assumes basic familiarity with the Elastic Stack.
153840	157540	So with that out of the way, what is the Elastic Stack?
157600	163700	The most basic definition you could give is that the Elastic Stack consists of multiple open source projects,
163700	171800	which, when used together, takes data from any source and enables search, analysis, and the creation of visualizations and dashboards.
171800	176260	It was originally called the Elk Stack, and I've noticed it's still constantly used today.
176640	183320	And it was called that way because it was originally just Elastic, Logstash, and Kibana for those open source projects.
183720	188020	However, back in version 5, they introduced Beats.
188020	191140	And Beats is a fantastic thing we'll get into briefly in this talk.
191140	195120	But in summary, it's just a way to gather your data and ship it for you.
195120	196620	So you don't have to worry about that.
196820	200840	And for some reason, I guess I didn't like the sound of elk or anything like that.
200840	202520	So it got called the Elastic Stack.
204820	208980	So what is the Elastic Stack and its most basic components today?
209340	212020	Note that I'm just going over the four primary ones.
212020	214620	You can add many more things to this.
214760	218660	But again, this is just so we can understand the basic terms.
219160	224100	You have Kibana, which is what analysts, threat hunters, whatever, are going to be the most used to.
224100	229160	It's the web interface. It's your visualization tool that lets you, again, visualize, search, and navigate.
229160	231220	Just a handy little web GUI.
231420	234060	And all this data is actually stored in Elastic.
234320	236560	Elastic is the heart of the stack. Stores the logs.
236560	239800	Actually does the really sexy indexing in the background.
239840	243100	Does all the things that we take for granted, to be honest.
244480	248560	Logstash is the other undervalued aspect, which is the pipeline.
248680	252620	So it takes data, parses it, and spits it back out.
252620	257200	And more than that, it can actually enrich it, like we'll get into with lesson number two.
258060	265560	Beats is located on your actual endpoints and collects configured logs or whatever files you have and sends it to Logstash.
265880	270500	So with that out of the way, very first lesson I learned.
271540	274020	Isn't Elastic and Kibana the same?
274020	277780	For the longest time, I didn't understand how these were different.
278220	283240	And hopefully after this, you'll understand that they aren't, if you're like younger me.
283240	288100	But why would I ever think this? Because, I mean, come on. Their names are two separate things.
288120	290840	I just went over it. It should be obvious.
291740	298040	However, in a lot of implementations, not all, Kibana and Elastic can be on the same page.
298920	304840	The same webpage, that is. So in the screenshot I grabbed here from GitHub, you can actually see, you have Discover, which is Kibana.
304880	309680	Kibana is featured at the top. But down at the bottom, you have Management and Monitoring.
309680	313480	And some of that can start getting into Elastic and Logstash.
314200	319060	Additionally, if you notice the error up there, it says, error request to Logstash failed.
319060	320820	I'm sorry, Elastic Search.
322160	330420	And that kind of introduces this confusion to beginners of, wait, if I'm using Kibana, why am I getting errors for Elastic?
330740	333880	Again, as you get more familiar with this, it makes a bit more sense.
334020	336600	But from a beginner, this can kind of confuse you.
337680	345600	Additionally, without fully understanding the full stack, you may think that Elastic is just part of Kibana, which they're different.
346520	349820	And to make it worse, when you go to Management, they both have the word index.
349820	353180	Again, this should be, you know, kind of easy stuff you get used to.
353280	358340	But for someone brand new to the stack, you're just going to go, Kibana, Elastic, cool, same thing.
359040	365040	So what is actually happening here is that a normal analyst is probably really going to interact with Elastic itself.
365040	371480	That's the goal. You should just be able to do everything you need through Discover, Visualize, and Dashboard, hopefully.
371540	376480	Maybe a little bit of machine learning, maybe a little bit of the timeline, maybe a bit of graph.
376540	379680	But overall, you should be sticking with the Kibana features itself.
380140	388880	As a result, you're probably primarily interacting with Kibana, but occasionally get messages like these with Elastic, and there's more you can run into.
389700	394520	Additionally, when Elastic goes down, Kibana doesn't really work.
394520	399680	The data disappears and you just get ugly errors, and depending how bad it is, it's spitting errors all the time.
399680	406220	And you're like, well, if they're not, you know, together, then why am I seeing those errors?
406420	409160	So that causes a lot of confusion to beginners.
410440	416040	So as a result, let's try to go into this a little bit more and understand better some of these differences.
416600	421360	And for this, I'm just going to go into the three primary features, at least I think are important.
421360	426560	The purpose of them, how you access them, and their search capabilities, because guess what?
426560	429360	You can search in both of them, which introduces more confusion.
430060	437080	So remember, Elastic's purpose in life is to obtain the logs, index them, and hold them in an internal database.
437580	442620	Because it's already the database, you can actually have search functionality through it.
442620	451740	It's just typically through cURL, or you can interact with a few other ways, like Python, for example, that we'll get into later.
451900	458820	Versus Kibana, its purpose is actually just to provide you, the user, the ability to create data visualizations.
459100	461980	And again, we can make that a bit more sexy with the dashboards.
462000	464760	And it does provide some search capabilities.
464760	469500	But if all we cared about was search, we probably wouldn't have ever introduced Kibana.
470500	475860	To access Elastic, again, you can use cURL generally through GET requests.
475880	481100	However, we typically interact with it with Kibana, which does all this in the background for us.
481200	487060	And again, with Kibana, you just open your favorite web browser, navigate to the IP or URL, and life's good.
487980	494300	And for the search capability for Elastic, again, you're going to do it through those GET requests or through a RESTful API.
494300	497480	And it uses the Elastic Search Query DSL.
497480	503500	And if you're used to interacting with that, you're going to have issues when you go to Kibana because it's going to keep scolding you.
503500	509140	Hey, stop using that. Use my search capability, which is the Kibana standard query language.
509220	512500	You can make it work. It allows you to search it directly.
512500	517580	But in general, if you're new at this, just try to get familiar with Kibana standard query language.
517660	521140	And again, you just access that through your handy web interface.
522600	527880	So, to kind of help summarize this, Elastic is your distributed search engine.
527880	533860	It takes in the logs, not even through Logstash. You can just feed in the logs directly in the index system.
534200	539560	Again, it can receive directly through HTTP request methods or through Logstash.
539900	544560	If you have another replacement for Logstash, that's fine. They'll accept it too.
545180	550860	And it's truly the heart, the engine, that helps make searches happen so quickly for Kibana.
550860	556020	Kibana does none of the heavy-duty effort to make that happen.
557120	562640	So, Kibana, on the other hand, is your browser-based analytics and search dashboard for Elastic.
562960	566680	And again, it's what your analyst or you are going to typically interact with.
566860	570880	Again, it allows you to conduct your searches, create visualizations, and dashboards.
571480	574900	After all this, you may be asking, why do I care?
575360	579180	First off, incorrect terminology can, one, make you sound a little silly,
579180	582540	two, result in you potentially being locked up behind your back.
583040	588240	But mainly, the reason you should care is that understanding the differences
588240	592580	and the interaction between the two can help you better improve your troubleshooting skills.
592580	597140	That way, when you encounter those errors, you can better understand what needs to happen.
597280	599800	Alternatively, you can just submit a better ticket.
600100	604520	Additionally, as we'll get into soon, if you can't quite make the right search query,
604520	607980	understanding overall the interaction with the stack is going to help you with that.
608480	612260	So, even if you think, hey, I'm never going to deploy the stack, I don't care,
612260	615280	you should care at least at a very basic level the differences.
616900	623980	So, with that out of the way, second thing I still say to this day,
624500	628460	why can't I search for, enter the field name, I see it's a field.
628460	632640	In this case, host name, I'll admit this was me probably three weeks ago.
633280	637500	So, again, me on a new and or existing setup all the time.
638560	644240	Take a moment and briefly see if you can figure out what's the problem if you use this.
645720	649660	Okay, you got your five seconds. A little yellow triangle.
649660	654820	I like calling it the yellow triangle of doom. I'm sure it has a better name, but it's what I call it.
656020	659480	So, what is the symptom when you're suffering from this?
659480	663640	You know a field exists, but you can't reference it. You go to Kibana, you go and search,
663640	668660	and you're like, take this field name and search for this value. And it's like, no records.
668660	673900	And you're 100% positive that one, the field exists, and two, something should have that.
674260	679560	So, what you do is that you find a record within Kibana, you know, you can just go discover,
679560	683920	roll through the records, and look at them until you find that field name you care about.
683920	690300	You can then search for that specific field name and its value and still have trouble.
690520	694580	At that point, you'll probably notice, hey, I have that little yellow triangle.
695220	700140	Roll over it, and you'll most likely see the message, no cached mapping for this field.
701360	706100	And the really quick way to just fix this is go into management, go to index patterns,
706940	711300	find your index pattern, in this case, just Logstash at the bottom of the screen,
711300	716360	and click the little refresh button. That'll fix it. You can search it, and life is happy.
717320	720420	But you should be asking yourself, why does this happen?
720940	726500	It happens whenever you get either one or more new fields introduced to an index pattern.
727440	733100	And remember, your index pattern is going to be the interaction between Elastic and Kibana happening here.
733100	738300	So, what happens is that Kibana has a cached version of the mapping given by Elastic,
738300	743300	and it doesn't have this field. So, therefore, it doesn't really know how to give you those results.
744040	749760	So, refreshing the index pattern allows for the creation of this mapping, fixing it.
750860	756540	So, why would this ever happen? You know, we're in this nice, stable organization. This will never happen.
757940	762640	Garbage. You're either going to do it through log enrichment, where you might see,
762640	765740	or you're interpreting a new log, and you want to improve it somehow.
766480	769500	Alternatively, and kind of tied with that, is improved parsing.
769960	774920	Let's say you have this message on the right here of the screen, where you have network connection detected,
774920	781040	and all this stuff. Depending on how your setup is, this may not actually be its own fielding.
781360	786140	So, you might not be able to go through and be like, hey, how do I query the image, or the user,
786140	790440	or the protocol, or anything else you see on there? Maybe it's all just in that one message,
790440	793020	which makes it kind of ugly for Kibana search purposes.
793520	798180	Therefore, maybe you need to go back and improve that parsing and implement that.
798180	803320	And when you do that, and you try to query, let's say, the user, it's still not going to work because you didn't refresh it.
804120	809200	And also, this can happen if you have a new update in any of the aspects you used,
809600	817060	such as if you just upgraded Elastic from an older version to a newer version that uses the Elastic Home schema.
817060	822140	You're going to have a lot of new field names, and you're going to need to refresh that.
822140	827980	Additionally, if you implemented any custom events that we'll talk about soon, you're also going to run into that.
829080	833360	So, don't just refresh blindly. Think about why it happened and if you can figure it out.
835400	843720	So, speaking of that, when is this suitable to actually use Logstash for log enrichment or just improving your contextual awareness?
844180	848440	To me, it comes down to three primary things, either to improve your search capabilities,
848440	853800	such as if you're doing a search on a Windows environment and you really don't want to see the service accounts,
853800	856240	you just want to see what the users themselves are doing.
857260	861960	Therefore, you could make this really complex query to remove all those IPs or host names,
861960	868100	or you can make a complex query to include all the different hosts and their subnets they're in.
868780	870820	And again, filter out those service accounts.
872900	878440	Or you can just mark all service accounts as they go through Logstash, so the Windows logs.
878680	884080	And if it sees that, you can just tag your service account and then you can simply tell Kibana,
884080	887480	don't use or exclude all records with this tag.
887660	892040	Makes it a lot easier, makes it easier on the analysts, makes sure you make less mistakes,
892860	896220	and just overall improves your experience.
896900	900160	Additionally, sometimes you just need to transform or fix data.
900160	906960	So an example of this is often Linux will send timestamp, maybe in a weird format that you're not using.
906960	910340	So your Windows and your Linux may send it in a different format.
911040	915120	And as a result, that makes it hard to compare things and be able to see,
915120	918200	did something happen at the same time or in a similar window?
918520	923500	Alternatively, maybe one of your systems is set up to use UTC, but the other is using local.
923620	927400	That also doesn't help. These are all different things you can use to fix.
927920	931720	And additionally, why not improve your context?
931720	936420	So rather than having you have to go like, OK, let me see if this IP is a known bad IP.
936440	940160	Let me see if this URL is a bad URL or something like that.
940160	947560	You can add the log session, have a query and be like, hey, is this IP that's coming in or going down part of a botnet?
947760	950420	Or is someone visiting a known malware site?
950460	954560	There's a lot of things you can do, and I'm not going to get into all of these per se,
954560	959280	but these are just examples to try to get you thinking about how can you improve your environment?
961300	966220	So in order to understand this, again, yes, you, the analyst, may not ever be doing this.
967100	970560	But understand how this works so you can make more reasonable requests.
970720	974160	And you're not doing things such as saying, I want to find a bad guy.
974600	978960	You get nothing else. You just say, find bad guy. I'm sure we've all heard that a lot.
978960	983640	We get irritated. We don't want to do that. We don't want to be the same people saying that.
983640	990240	So as a result, if you're asking for something to be fixed or improved, have a better understanding of what is possible.
990800	995000	So the overall format of a Logstash configuration files, you have input and output.
995000	999840	Those are the two mandatory fields. Input tells Logstash, what port should I listen on?
999840	1003160	What protocol should I listen on? And how should I interpret it?
1003340	1008260	The output says, once I'm done with it, where do I output it to and where?
1009080	1015460	Additionally, there's a filter section that can go in between, and it's optional.
1015460	1018780	However, all the fun things we just talked about are in there.
1019120	1029940	And just to go a little bit more, you can have TCP, you can have UDP, you can have Beats, things like that for all of how Logstash should be listening to it.
1029940	1036000	The codec, just briefly, is going to be how should I interpret it? JSON, CSV, something like that.
1037360	1042660	Output. Note that here, you can obviously see the hosting, which is what we're primarily going to use.
1042880	1049600	That also means if you see an error, like back in Lesson 1, a VIP can't resolve that, it may be a DNS problem.
1050020	1054580	So again, just trying to tie these together and better improve our understanding of how the whole stack works.
1055780	1059000	So with that in mind, there's a thing you can do called tags.
1059000	1062680	I mentioned this before, like tagging user accounts, tagging the machine.
1063520	1070620	Why would you ever want to do this? Again, it helps provide contextual information, because even if I don't filter for it,
1070620	1080200	if I just see an IPNC, hey, this is my domain controller, or hey, this is a workstation, or hey, this belongs to the commerce or the DMZ subnet.
1080200	1085740	This is all things that helps us quickly go like, okay, I have a better idea of what your purpose in life is.
1086180	1090260	Alternatively, you can just say, hey, this is a Linux system, this is a Windows system.
1090260	1094060	Again, you can probably figure that out with 5 or 10 seconds more thought.
1094280	1097900	But this all add up and make you operate better and more efficiently.
1098600	1108300	You can also identify the top of logs. So if you're wanting to parse only, or if you only want to look for a patchy like error logs to see what's going on, you can do that.
1108300	1114500	Access logs, same thing. You can also, you know, obviously identify admin accounts, user accounts.
1114680	1118120	Pretty much, if you think you can make it happen, you can do it.
1119300	1131560	In this example you see on the slide, it's a really simple, basic one where it says, hey, if you get any logs that come from autobeat, which is something we'll talk about soon, don't worry about it.
1131820	1136580	It's going to add a tag of Linux because autobeat only exists on Linux.
1137300	1143760	Again, you can add a lot more complex information here. This is just to give you the beginnings to make you a little dangerous.
1144240	1149280	Additionally, it's important to look at tags because Logstash will add tags when a parser fails.
1149720	1159960	So even if you didn't set up a custom one, if something's failing along the line, it's going to let you know, hey, this part in particular errored. Go look at it.
1161000	1161600	So.
1164000	1168040	Again, some of the different things you may want to consider about with Logstash in general.
1169100	1171780	And again, this is more about the filter section.
1172380	1175600	Timestamps that aren't sent through Beats may not come in a format you like.
1175600	1180220	They may come in different formats, they may come in the wrong time zone, something like that.
1180500	1185380	Additionally, one thing I particularly like doing is I just want to split the field and make them into a new one.
1185560	1191480	So rather than saying mycomputer.local, maybe I want to see the hostname of my computer in the domain local.
1192140	1194700	Maybe you want to have all three. You can do that.
1195320	1202860	You can remove field and tags. This really kind of aggravates me, but has its use, is that Beats will generate an automated tag.
1202900	1210540	However, this can be a really lengthy tag, so you might want to rename it or you just want to remove it because you don't care that it's some complex thing.
1211720	1215020	And again, the Translate plugin is going to be how you rename the fields.
1215020	1227420	And more than just being like, hey, let me rename this, you know, Beats tag, it can be a lot better and actually do like, let me convert the HTTP codes into their actual meaning so you don't have to keep pulling up a little cheat sheet.
1227460	1231820	As much as you promise you know all the error codes or all the different codes, we don't.
1232660	1237640	So that helps you, again, save that little bit of time and be a bit more efficient.
1239020	1251140	So the overall goal when you're filtering and improving your Logstash pipeline is to improve those logs by adding additional information that would have normally been done manually.
1251900	1262880	Some of the more interesting plugins or plugins you might use regularly is GeoIP, which again adds just geographical info, says, hey, this IP is from China, Russia, America, whatever.
1262880	1267080	Obviously, it's just doing it based on what the servers report.
1267700	1271720	If you're that suspicious about it, double check it, but it can give you that quick insight.
1272380	1278160	DNS Confer standard or reverse lookups, again, handy, saves you that little bit of time.
1279040	1285700	Threats Classifier is one that's really interesting where it can enrich your logs and add a little bit of context based on the MITRE ATT&CK matrix.
1286260	1292140	So again, saving you a little bit of time, obviously, especially with something like that, double check it.
1293100	1301520	And UserAgent helps parse your UserAgent strings into fields, so you can actually search on it a bit more efficiently and not have to do maybe quite so many wildcards.
1303780	1311080	So overall, if you, the analyst, can better understand Logstash, you can make a better reasonable request of new features and improvements.
1311260	1319300	It allows you to better understand the errors, so again, you can submit that ticket or ask your neighbor and be like, hey, I'm getting this error and I think it means this.
1320460	1327100	And overall, you can improve the creation of your dashboards or visualizations through asking to improve those pipelines.
1327440	1336280	Again, I probably gave you here just enough to be dangerous and maybe really slow down your pipeline to avoid too many of those filters, but have fun.
1338220	1347120	So with that out of the way, have you ever found yourself running some custom scripts to obtain information from your endpoints and just wish you could view it in Elastic?
1347120	1353460	Maybe you're just running, hey, I want to get all the processes and import it in, just so I can see.
1353660	1362400	Maybe you're just doing data analytics, any data from an incident response investigation, anything else that's not natively created in the log.
1363500	1370720	Again, with me without coffee, maybe I just want to see a pretty visualization with it rather than just viewing it naturally.
1371220	1378040	If this is you too, you're not alone, and we're going to go into how you actually take these custom scripts and throw it into Elastic.
1379260	1389760	So again, me without caffeine in the morning is like, I'm tired of reading the CSV, it's 7 a.m., it's ugly, and I don't remember how to control us because I haven't had coffee.
1391560	1398000	And I have to compare it to my dashboard. Why can't they be together so I can make better dashboards or just do better searches?
1398620	1408500	Well, you can. First off, if you really hate yourself, you can just curl everything. You could go curl in that CSV and put it into Elastic.
1409680	1414200	And I don't recommend it because at least I seem to forget curl any time over the weekend.
1415040	1421360	You can also use Python. We all love Python. And probably a few of you are now arguing, it's all good.
1421780	1430300	So if you want to use Python, all you have to do is install the Elasticsearch package through pip. You'll connect it to your Elastic, you'll do your steps, and you'll profit.
1431040	1438220	What it actually allows you to do is you can pull data from your indices, and you can make your edits, and then you push back to it.
1438540	1445660	It allows you to actually conduct the searches, because remember, Elastic is more than just storing the information. You can search with it.
1446140	1455140	Overall, it gives you the ability to use Elastic without the bonnet. Again, you're not going to get the visualizations, you're not going to get the dashboards, but there's a lot you can do with it.
1456780	1468640	So, how would you do this? First off, hopefully you have your data in a JSON format. Elastic in general prefers things to be in JSON, even Logstash. Why is that?
1468640	1481080	JSON is already a structured language. You know, it's basically a really fancy key value type of thing. So, it makes it really quick to parse, versus if you have syslog coming in, which we'll talk about soon.
1481440	1490860	That's all separated by spaces, and not all the fields are guaranteed, and there's no way to easily say it's not there. So, JSON in general will speed everything up.
1491360	1503680	So, if you don't have your data already in a JSON format, you're going to have to make it into one. Then, you'll load it into the Python JSON object, and then you'll upload it into Elastic, simply by doing...
1503680	1516920	calling your index, specify the index name, specify the type of the document, give it an ID number, note you want it to probably be unique if you're creating a new one, and then you put your JSON object there.
1517820	1529180	You can do this for every single little thing you do. If you need to update a lot of different indices, that's going to be a lot of copy and paste, and you're probably going to make mistakes.
1529760	1539460	So, alternatively, you can just use Elastic's bulk feature. Again, this isn't Elastic itself, this is the Elastic search package through Python.
1539460	1548460	And what this does is that it allows you to do better performance. Just with one call, you can do multiple indexing and deletion operations, again, just in one API call.
1548960	1560140	So, if you're interested in trying to do this, but you're like, man, I really suck at converting things into JSON, don't worry, I'm with you.
1560140	1572240	So, what you can do is you can go to GitHub and go download the Elasticsearch loader. Again, link's right there on the slide. Fantastic one.
1572540	1583140	What this does is that it'll take JSON, CSV, or Parquet files, and it'll go ahead, convert if needed, and push it for you just in one command.
1584440	1591100	Additionally, you can set up some custom mappings to help make sure your data is interpreted the right way. Why would you want to do this?
1591680	1601580	Do you want the number one to be an integer or a string? Do you want port 80 to be a number or a string? Do you want to be able to do less than or equal to?
1601800	1610380	Those type of things you sometimes want to make sure are interpreted a specific way. Maybe with your IPs, you want it to actually be IPs and not a string.
1610380	1613760	These are all different things you can try to do with the custom mappings.
1614780	1622240	But what is the downside in doing all this I just talked about, whether it's CURL or Python or using the Elasticsearch loader?
1622720	1635640	You can't really do any enrichment or tagging that we just talked about. And again, if you don't set up those custom mappings, which you can do in the Python method as well, it can be incorrect and you're going to have to fix that.
1638040	1642700	So instead, if you really don't want to do that much Python, FileBeat is going to be your friend.
1643180	1651160	What FileBeat does is that it's going to take any logs you specify and forward it either to Elastic or LogStash. It's up to you.
1651400	1656820	And it supports some basic data processing and data enrichments. What's the benefit to that?
1657160	1662400	Well, again, LogStash is already going to be a little bit overwhelmed with anything you're trying to tell it to do.
1662400	1670940	So you can do some of this data enhancements like adding tags on the endpoint, have it do it there, and then send it. Save yourself a little bit of time.
1671020	1680940	Remember, everything you can save improves overall performance. You just need to decide, are your endpoints struggling in processing power? Where's LogStash?
1681080	1686520	There's a bunch of different solutions you can do out there, but we're not necessarily going to get into those nitty gritty details.
1687420	1689740	So overall, how does FileBeat work?
1689740	1697680	First off, you're going to use your favorite file format in the world that Elastic loves to use, or the Elastic Stack, and that's YAML.
1699380	1703640	And YAML is great because you need to specify the spaces specifically, so good luck.
1705020	1712340	So other than you'll crush yourself without having with that, what you do with FileBeat, at least, is again, you'll do...
1712340	1717680	And what I have on the right is a very basic one. It doesn't include everything you need. It's just the configuration.
1717680	1726980	We say for the inputs, I'm going to grab type log. The FileBeat website, depending on what version you have, will tell you what types it supports.
1726980	1729540	And this is just going to be a Linux log type.
1730560	1738000	And what it does is it says, hey, collect type log and collect it from, in this case, var log and anything that ends in .log.
1738840	1742240	So FileBeat reads that and it says, hey, let's do this.
1742240	1748060	And what it does is it starts a harvester, which is just, you know, child processes for each input type.
1748060	1751940	So you can have type log, whatever other types. It's going to start one for each.
1753320	1756380	And additionally, it's going to create one for each file.
1756780	1762600	Don't worry, FileBeat's not going to kill your computer by creating like 100 children. It controls it. It has a mask.
1763480	1771460	What it does is that each one goes ahead, grabs the log, opens it, reads it, sends it to an aggregator.
1771460	1781280	And once it has everything, or when it reaches a certain threshold, it'll send it to either Logstash or Elastic, depending on how you set it up.
1782100	1792700	So the input aspect, again, starts the harvesters and manages them. The harvester, all they care about is opening the file, reading it, and submitting it to the aggregator.
1795060	1801580	So, in the previous slide, you just saw how it ended in star dot log.
1802160	1815700	You're going to have to specify the type of the document and where it is. FileBeat is not recursive, so any additional directors you have, let's say under bar log, you're going to have to specify those and keep going into it.
1816020	1822680	Additionally, with those data enrichment aspects, FileBeat by default is not going to parse those necessarily.
1822700	1831340	Great, it's not going to consider what type of application it is and automatically parse those for you to make it say, hey, this is a string, this is an IP, all that jazz.
1832040	1839180	Instead, what you can do is you can use modules. Modules, the ones I've listed here, are just the beginning. There's a lot more.
1839180	1846000	It includes things like system, which are going to be your typical, like bar log ones, things you see in DMessage, things like that.
1846400	1853240	Nginx for using Nginx, MySQL, IIS, so it's not just Linux, don't worry, and so many more.
1853400	1861480	An example you see on the right, it's Apache. We can enable or disable just the access and error, specify where they are, and it works.
1862640	1868660	So, again, this simplifies your config. All you do is enable it, specify where the path is, and then you're good to go.
1869400	1874260	There's a lot more settings you can specify, and that's all going to be in Elastic Documentation.
1876360	1881800	So, you're probably like, gosh, that's been a while.
1882680	1888680	And you really don't care about the fact that I want to go back to my Linux origins because I'm tired of talking about Windows and that incy little bit of Linux in the slides.
1890020	1896100	And a lot of the verbal examples I've provided have been Windows, and obviously some of this can be converted to networking devices.
1896540	1898220	But what about Linux?
1899820	1906740	With Linux, when we generally forward logs, we use syslog, rsyslog, or syslogng. Typically, those are the most popular ones.
1907640	1913520	Some implementations such as syslogng can forward directly to Elastic. Fantastic.
1914040	1919840	However, others are going to require help, whether using FileBeat or using a Python script, whatever the case may be.
1920700	1927600	But overall, this only helps your services. It only helps you find things that you can configure in there and be able to read from this log file.
1928180	1933700	And you get into a little bit of weird things when you try to talk about, I want to forward things from log in, log out.
1933700	1940000	I want to see my sudo or sus privileges being used. I want to see when someone edits a very particular file.
1940940	1951460	Things like that, it's kind of getting to the point of, can we have sysmon for Linux? Because let's face it, sysmon is great on Windows. It helps us find so many more bad things.
1953440	1957320	Linux doesn't have a super famous way of making this happen.
1958620	1965100	But that's what Audity is for. Audity is available for basically all distros out there of Linux.
1965740	1971580	The only ones I've seen a little bit of issues with are Gentoo and maybe a little bit of Arch, but you can still make them work.
1972540	1979140	And what it does is it allows you to log basically anything. Here on the slide is just some more common things you're going to do.
1979200	1986700	It allows you to log the date, time, and type of outcome of an event. And don't worry, I'm going to define what an event is in just a moment.
1986700	1994820	An association of that event and who triggered the event. Was it Root? Was it user Pinky? Who was it?
1995480	1999980	Any modifications to the audit configuration to try to maybe circumvent it.
2000680	2008460	All uses of authentication mechanisms like SSH or Kerberos. Because remember, Linux can't be joined to your Active Directory network.
2009160	2013040	And any changes to trusted databases like XC Password.
2013040	2024440	And as a little heads up, the little tweet on the slide, that will actually create or generate alerts, or events in this case, that is new process creation, just like SysMob.
2025680	2034060	So, event. I said I was going to explain that. What is it? What it is, is it's the result of a rule being triggered. What's a rule?
2034460	2039940	It's what you see later in the slide, but it's everything you specify in XC audit rules.
2040860	2047940	So, there are two overall types of rules. Files to monitor, and monitoring system calls.
2048540	2059780	So, generally when you want to specify a file, just as a quick, dirty intro, is that you're going to do "-w", for watch, the file to monitor, "-p", any permissions you want to trigger on.
2059800	2065180	And this generally comes into read, write, execute, and change in attributes. So, for.
2065560	2068140	And "-k", what you want the name of the rule to be.
2068140	2072860	Make this something that makes sense, so if it gets fired off, you understand it.
2073480	2084020	An example, in the bullet below, it says, hey, I want to watch Etsy Shadow, and anything that is written to it, or if the attribute changes, notify me with the event name, Shadow.
2085020	2089880	For system calls, it's a little weird, and I'll be honest, every time I see it, I have to go back to the main page.
2090580	2094960	But what it is, is that you have "-a", an action, followed by a list.
2094960	2103520	You have "-S", the system call, "-f", which is the field, which often ends up just being like, hey, specify the architecture.
2104560	2108160	So, is it going to be 32-bit, 64-bit, that type of deal.
2108580	2111400	And then, "-k", the name of the rule.
2112620	2116560	So, action must be one of two values, always or never.
2116820	2120000	So, always alert on it, or never.
2120960	2128000	And then, you have one of the following for the list, task, entry, exit, user, exclude.
2128220	2134800	I find the most common one is always an exit, and what this does, is that it will always alert when that system call exits.
2135880	2139520	So, entry would be the opposite, when that system call is initially called.
2140000	2149480	In this case, it's going to work on 32-bit architectures, and what this means is not just 32-bit distros, but 32-bit calls for the system calls.
2150420	2154420	And all these system calls you see here are all for changing the time.
2154700	2162160	So, adjusting the time, setting the time of day, doing clock set time, those are all different ways you can adjust the time.
2162480	2168180	So, the rule of calls, or the event that will be named, is time in this case.
2169880	2174900	So, how do you get started with AuditD, because it's amazing.
2175220	2178580	First off, you're going to install AuditD. It's already on your system.
2178580	2183320	You're going to edit the config if you need. You're going to create all your rules by hand.
2183960	2196600	No, you're not. Go to that GitHub, download it. It's a fantastic one that has a lot of the most common things you'll need, including things like empty password, the change of time, a lot of things that I catch bad are going to be in there.
2197180	2202720	You're going to restart the service, and you're going to be like, I have errors. She lied to me.
2202720	2209840	No, some of those things are distro-specific, or require specific users like Hrani, and you may need to comment it out.
2210440	2213240	Do that. Keep fixing it until it works.
2213840	2223000	Then you can view the actual logs, which, depending on how you set up your rules, is going to generate an event, or the proper ways to use AUsearch.
2223060	2226520	You can look at the main page, do it all there. So, great.
2227360	2236180	If you followed all these, you know, and we weren't there to talk and just actually worked through this, you'd be like, I have Audity installed. It's working. It's generating alerts.
2237160	2243560	Gosh, it's annoying how it does it every time I use Sue, and I'm trying to be a good person and not use Root, and it's OK, because if I use Root, it's going to cause alerts anyway.
2244840	2248520	So, hey, it's working. What next?
2250320	2263520	You're going to install autobeat, because you want to, you know, you want to move those events and its logs into Kibana and Elasticsearch so you can visualize it pretty, you know, just your text files all day.
2263960	2271160	So, you're going to point autobeats.config to autodesk.config, or you put those rules just from the GitHub into autobeat.yaml.
2271840	2283560	And note, if you created it from scratch, or like you just extracted autobeat rather than installing it from the package manager, you need to make sure autobeat.yaml is owned by Root, otherwise it complies.
2284040	2287780	You're going to start the service and get this exact error on the screen.
2288960	2289660	Why?
2290420	2294100	Well, it says, hey, failed to send it because it's already running.
2294520	2295620	That's weird.
2296680	2299240	I lied to you. You need to stop and disable Audity.
2299760	2305540	The problem with this is that autobeats ends up actually running its own version of Audity.
2305760	2312920	So, it partially helps avoid some of those alerts that happen of any time it reads those logs.
2314660	2317060	Additionally, Audity has its own formatting.
2317560	2318360	Because, again, it's going to...
2319080	2320740	Audity doesn't use JSON.
2320900	2325780	It's just more typical Linux formatting stuff, you know, space, tabs, all that.
2325780	2334820	As a result, Audity is going to take longer or cause more overhead for Logstash to parse and cause you, if you have to figure Logstash, a lot of grief.
2335500	2341460	And sometimes if Audity and autobeats are both on, it can cause a kernel panic. That all sucks.
2341760	2343260	So, why did you bother?
2343960	2350780	Do you really want to see all the times when your Audity rules fail in your Elastic?
2351640	2357600	Probably not because it's going to confuse you the next morning or the next analyst who looks at it and goes, oh, God, it's broken.
2357760	2366160	You want to test it locally to avoid debugging, you know, through it and having to figure out where the error is and without freaking out your fellow analysts.
2367600	2376380	So, again, install Audity, test it, disable it and stop it, install autobeats, get it working, and life will be good.
2377940	2382780	So, now you may be wondering, what did we just cover? Oh, God.
2383400	2386000	First, Elastic and Kibana are different.
2386760	2390800	Hopefully, you won't make the same mistake if you're guilty of saying it sometimes.
2391720	2395180	Logstash will make your Kibana queries way better, I promise.
2396140	2399780	Feeding in custom documents is possible and actually not that bad.
2399800	2403800	But the first time, like anything, new experiences suck, after that's fun.
2404400	2407040	Audity is incredibly painful to set up.
2407320	2411080	Actually, no, tuning it is extremely painful, but it's worth it.
2411080	2414920	Because, again, don't be blind on your Linux systems.
2415900	2422780	And the lesson I want to briefly hint at you in this lesson or this slideshow, YAML sucks.
2422820	2429980	Probably 90% of your errors throughout all of this will be white space issues because you are dirty and use spaces and types interchangeably.
2430520	2433160	And if they're not set the same, life sucks.
2435140	2438020	So, if you want to see these slides, they're actually on my GitLab.
2439280	2440600	You can grab it from there.
2441300	2445860	And I'll take a few minutes to answer any questions, but if not, I'll be in the chat for a bit.
2446140	2449940	And if you don't think of it until 30 days from now, whatever, ask me on Twitter.
2450860	2451800	That's all I have.
2453500	2457720	Thank you very much, Dr. Pinky, for that wonderful presentation.
2457720	2468460	As always, we encourage you to join our Blue Team Village Discord server and ask questions in Text Talk Track 1.
2468460	2475900	I will try to take a quick look here and see if we have any questions for you.
2477600	2480160	And I am not seeing anything.
2480160	2491120	So, with that, if you do have questions, feel free to direct them to the presenter as she will be hanging around for a little bit.
2491120	2492540	Thank you again.