start	end	text
13580	18660	Hey, welcome back. Hey, AppSecVillage, we're getting down to the last few speakers of our
18660	25160	time together here on day three of AppSecVillage at DEF CON 2020. I want to thank you for coming
25160	29700	and for making this possible. I want to make sure to thank our sponsors and the folks that
29700	34780	run AppSecVillage. They've done a wonderful job both years that we've been here. This
34780	39300	has become my home away from home when I come to DEF CON. I look forward to it every year
39300	44460	and I hope you do too. And I hope to see you again next year when we do this, hopefully in
44460	49520	person, but we're going to do it one way or the other. Our next discussion is going to be Secure
49520	55080	Your Code, Injections and Logging by Philip Kren. Philip lives to demo interesting technology.
55200	59360	Having worked as a web infrastructure and database engineer for more than 10 years,
59360	64840	Philip is now working as a developer advocate at Elastic. Based in Vienna, Austria, he is
64840	70500	consistently traveling Europe and beyond to speak and discuss on topics such as open source
70500	75850	software, search technology, databases, infrastructure, and security in general.
76160	79560	Please welcome Philip to the AppSecVillage stage.
80620	87780	Hi, I'm Philip. Let's dive into the next session. Let me share my screen.
88760	94720	And there we go. Secure Your Code, Injections and Logging. Let's see what we can do here.
94840	101500	So when we talk about security, we normally keep telling ourselves that everything is fine,
101500	106400	especially when we say everybody's job is security. It's normally nobody's job,
106400	111300	and we can all pretend that this is all good. And then you look at the OWASP top 10,
111300	118420	and you see at number one is still injections. So good old SQL injection, maybe LLAP injection,
118420	124740	NoSQL injection, or whatever other form of unsecured user input you are throwing at your
124740	130080	application, and that is wreaking havoc. And then you might reconsider and say like, well,
130080	136660	maybe this is not so fine anymore. And then you look further down the OWASP top 10 list,
136660	141340	and then you see, for example, number 10, insufficient logging and monitoring,
141340	146420	where you then realize that on average, it takes something like 200 days or so until you normally
146420	152460	find a breach, maybe a bit quicker if the attacker asked for some ransomware or ransom money,
152460	159560	but it might take a while. And then you both or you might end up at the point where you say,
159560	164400	well, this is no longer fine or good, and everything is terrible and on fire.
164400	168820	And we don't want to get to this point. So what can we do about that?
169440	174540	I'm Philipp. I work for Elastic, the company behind Elasticsearch, Kibana, Beats, Logstash.
174540	179280	You might have heard of us or might be using us already somewhere. My official title is
179280	184400	developer advocate. So I mostly talk about the good stuff that we do. And today is no exception.
184400	190540	So let's see what we can do here. I generally build highly monitored Hello World applications.
191000	195700	Today is no exception. And we will pretty much dive into that right away.
195700	201880	So like in any good cooking show, I have prepared something and a very bad application.
201880	208920	So xiera.wtf is where this is running. And I will let this run. So while this is pre-recorded,
208920	214420	I will let the demo run. So you can actually destroy it or play around with it live afterwards
214420	221320	as well. If you're wondering where Xiera is coming from, it has something to do with my name.
221320	228340	If you're bored, try to figure it out. And obviously, WTF is the right domain for any demo.
228340	237400	So let's hope this works out. And let's head over to my very bad and well, average demo application.
237400	245440	So we have some Elastic employees here. And I added our CEO Shai, one of our co-founders Simon,
245440	253140	and myself. And well, maybe we want to add more employees. And to do that, obviously,
253140	261960	you need to log in. If you want to regularly log in, my username is Philip, and my password is
261960	273700	secret. This works. And then we can say, other edition is, I don't know, maybe in New York.
273720	282000	And they have some salary and they also have a password of let's say secret. Okay, so that worked.
282000	290420	But looking at this form, this is not secure. And by design, this is using the Hello World
290420	297640	application of SQL injections. So we can just use one parameter like Philip and in the other feed,
297640	307120	we will just do an or true. So I will do an or true and try to log in like that.
307800	316420	This worked, unfortunately, and I can now add a bad user in, I don't know, from Gotham,
316420	320860	and have a much higher salary. And they also have some random password.
321400	326480	Okay, so we have added the bad user, nothing super surprising here.
328020	334100	This has the classic SQL injection problem, I don't escape my user input, I just pass in
334100	339160	whatever I'm doing here. I'm hashing my password, but I'm just breaking out of that and doing
339160	345680	or true, for example, or whatever other variation you want to have of this.
346200	352040	You've seen this plenty of times. It's called an SQL injection, or some people call it accidental
352040	357700	GraphQL, if that is more your thing, and which might be the deeper thing. But classically,
357700	365780	this is an SQL injection. And I'm pretty sure all or most of you have seen little bobby tables.
366180	373800	If you give your child the right name, like drop table students, you will have a lot of fun
373800	379820	at school, or your school just should write better systems. But that's a different discussion.
379820	386060	So let's have another look around. So I can also display users. So for example,
386060	392560	looking at our CEO shy here, this looks all nice and good. And you can go back and forth.
392640	399180	What I'm doing here is I'm just passing the ID of my user directly in. So I could just change
399180	407900	that to two and pass over to Simon. This is insecure, again, like unsurprisingly, since
407900	414600	this is a broken application. And before we try to actually throw some bad parameters as it,
414600	422020	I want to quickly jump to SQL map. If you've never seen or used SQL map, it's an automatic
422020	427840	SQL injection and database takeover tool. And you can basically say like, hey, there is this
427840	434460	parameter like the ID and I would like to check that and it will go through a lot of variations
434460	441860	of SQL injections to actually say like, well, this looks like it might be interesting or not.
441860	450920	Okay, so let's open my shell and let me paste in my command so you don't have to see me type this.
450920	456620	So I'm running SQL map against that URL that I've shown you. And at the end, we add a purge
456620	461840	because I've run this before. And purge will just make sure we don't use any records that
461840	469960	we've created before. So we will start fresh. So we will start running this and SQL map will just
470640	477140	figure out what is possible here. It will ask me a couple of questions of how it should treat this.
478180	482680	So you can see it's already trying to do stuff and it figured out that I'm using
482680	489820	MySQL here. So yes, this is the right database. And I want to skip tests for other databases
489820	498100	because this is MySQL, we can focus on this. And I'm okay to run these risk level queries here.
499300	502720	So you can see it's trying out various things.
504780	513260	And at the end, it will then tell me what is actually vulnerable. It will also ask me
513260	519220	if it should run or check any other parameters, which we will not do here because there's only
519220	524920	one parameter that is interesting. I don't want to test anything else, so we're done here.
524920	531100	So looking at the result, you can see, for example, okay, this is running against NGINX
531100	538580	and against the more or less recent version of MySQL. We would have some logs in this folder
538580	542780	here, though they're not super interesting. They're mostly what we have here and all the SQL
542780	550580	commands that have been running. So we will stick to what we see here. So you can see in 81 HTTP
550580	557420	requests, we figured out a couple of things. So for example, we see a Boolean-based blind attack
557420	563500	where you can do something like ID 1 and something equals something, or you can have a
563500	569720	time-based attack where we add a sleep and we see indeed the query takes that amount of sleep plus
569720	576000	a little more maybe, or that you can have a union query and that you can try to join data together.
576000	584140	So this is clearly vulnerable to something. So let's have a look of what I have actually in here.
585540	591160	Again, a very bad query. I'm just taking that parameter. I'm not escaping it in any way. I'm
591160	599400	just trimming any spaces off of it. And then I take this query and I run it. The good thing here,
599400	604700	by the way, is that I actually logged the query that people are sending to my system. So this
604700	610100	will help us afterwards to figure out what has actually been happening here. So we will log this
610100	621180	into a var log app log file. The bad thing here is that I'm using MySQL multi-query. So this will
621180	628360	allow me to run multiple queries in one go. So besides that select, I might do another insert,
628360	635920	for example. So if I add this to the end of the URL and it will automatically then be URL escaped
635920	646620	and work properly, I could just say insert into employees some bad actor. So if I run that,
646620	653060	let's see in our application. Right now we have a bad user, but we don't have a bad actor.
653060	660820	So if I have to that parameter here, I just add after semicolon the insert statement,
661200	668480	run this, it looks like nothing happened. But when I go back here, you now have a bad actor.
668840	676840	So this works as expected or not expected. Maybe this is mostly up to your expectation. By the way,
676840	682680	this doesn't escape like the visualization in any way or the output. So you could run any
682680	688580	random JavaScript here, which I didn't do because it will be kind of annoying to have various pop-ups
689320	695280	pop up. You could break the entire site with some weird JavaScript. So there is no other
695280	704820	protection here. So we don't do any escaping. So this is easy to pick apart. So what's going on in
704820	710340	our application? Now is the interesting part, like at first before I want to secure it, I want to get
710340	717040	a bit of a better idea of what even has been going on in my application. And yeah, if you don't have
717040	722640	any log files, nobody can remove your log files. So let's assume we want to have log files and we
722640	728160	also want to ship them off and have them hopefully in a secure location. So even if somebody breaks
728160	734700	into that instance and manipulates it, they cannot actually do anything with your logs and you still
734700	741820	keep some trace of what has been happening. So we want to collect all the things and having a log
741820	746800	file is a very good starting point. Maybe ship it off in case somebody breaks into the instance.
747120	753960	The main downside is these log files have the ugly tendency to get very large. So maybe at some
753960	758880	point you end up as this little submarine, basically, and then you have gigabytes and
758880	764940	gigabytes of log files in which you try to find something. Maybe successfully, maybe not so
764940	772880	successfully. Scrap is great until you have multiple servers and various big log files,
772880	777540	because then it's not going to be so much fun. So you probably want to use something else that
777540	783920	is making your life a bit easier. One of the most wily tools there probably is the Elk Stack. You
783920	790120	can see Elasticsearch, Logstash, and Kibana, one sitting on top of the other, and that makes up the
790120	796660	good old Elk. So Elasticsearch is the thing that stores your data, Logstash is the thing that gets,
796660	803620	parses, and enriches your data, and Kibana can visualize it. By the way, when I say parse,
803620	808300	you normally have some logline, and out of that logline you want to extract some piece of
808300	813500	information. So, for example, you have a log level, or you have the URL where something came
813500	818620	from, or maybe you have some user information, or a timestamp. All of these you might want to
818620	823700	extract to actually be able to filter down and then say, oh, give me all the logs from this
823700	830240	specific time frame, or for this specific user, or for this specific application, or URL within
830240	837520	the application, or any way you want to slice and dice your data. When I say enrich, it might be
837520	843300	something like you have the IP address of the person requesting the website, and then you could
843300	849180	add the GeoIP information to that IP address, and then you could draw it out on a map, and then you
849180	853960	could draw where are all my visitors coming from, for example. Or you could just filter down and
853960	860720	say like, oh, give me all the users coming from Russia or China, because that's kind of unusual.
860720	865900	I normally don't have any visitors from there. Let's figure out what they have been up to.
866780	872860	That works. The only thing is that by now the stack has evolved slightly, so we still have
872860	877260	Kibana for the visualization Elasticsearch that stores it. Sometimes, by the way, people are
877260	881720	slightly confused and ask like, what is the database behind Elasticsearch? There is no
881720	887820	other database anymore. There is the library Apache Lucene, which writes it down to disk,
887820	893700	but otherwise there is no other data store behind that anymore. Logstash is still that
893700	900260	versatile, bit more like ETL tool, so extract, transform, load, so it can get data from various
900260	906080	systems, change and transform that, then push that out to Elasticsearch or other systems.
906440	914120	But to make life a bit easier and also slimmer, we have added bits in various versions so far.
914120	920740	So, for example, for log files, we have Filebit, and it basically tails a file over the network
921300	927140	and a bit on steroids. So, we basically will say like, this is a log file that I'm interested in,
927140	932780	take that, tail it, store it into Elasticsearch. To keep my demo simple, I will actually skip
932780	937960	Logstash and Filebit will tail my log file, put the data into Elasticsearch, and then we can
937960	949780	visualize it in Kibana directly. So, let's see what we have actually here. So, maybe before we dive
949780	956800	into the visualization, let's take a very quick look at what we have done here. So, this is the
956800	963320	configuration of this entire demo, and this is all being set up automatically, so you can get all the
963320	969880	configs afterwards. What I have done here is, there are maybe two interesting things in this
969880	975680	file. So, this is the Filebit configuration HTML. What will be interesting at first is,
975680	981680	so, I'm getting some log files from the system, and there are so-called modules. These are four
981680	987400	well-known things that you have often installed. So, for example, I have Nginx running here,
987400	994720	as we have seen in SQLMAP before, and it will know automatically for the server that I'm running,
994720	1000100	where is that log file located by default, and what does it look like. So, basically, by saying,
1000100	1006800	use this Nginx module, it will then pick up the right file, be able to parse it in its individual
1006800	1013360	pieces, and then store it, and we're done. We don't need to care about any log file locations
1013360	1019680	or anything anymore. With that var log app log, which my application is writing, it's not so
1019680	1025100	standardized. So, for this one, I have this configuration here, where I basically tell
1025100	1032420	FileGrid, this is the file that you need to collect. So, it is of the type log, this is the path,
1032420	1038000	you could also use a wildcard and point to a directory, for example, and then what is also
1038000	1043840	interesting, I will store this with an additional field called application app, and this is what we
1043840	1048760	will be able to filter on right away, because I'm collecting all these other logs, so we might
1048760	1057320	see a lot of logs coming together in Kibana right away. So, let's take a quick look. So,
1057320	1061680	maybe you have used Discover before, which was the kind of like old way to do that.
1061680	1069800	We have now a dedicated logs UI, where you can filter down and live stream data. So,
1069800	1076400	with the live streaming, this is a bit more like tail F, where you just see all the log messages
1076400	1081900	that come in right away. So, whatever happened here to my SQL, we might investigate that later
1081900	1086960	on, but it will just live tail anything that is happening here. But what I'm interested in right
1086960	1096900	now is application, as I have said that before, and let me change my screen zoom in slightly,
1096900	1107300	and my application's name was app. So, I just want to filter down only on the application that
1107300	1113860	I'm running here. I don't want to see anything about Apache, or Nginx, or anything else that
1113860	1121800	I have running. Here we go. So, this is the log file of my log that I have collected. So,
1121800	1129200	you can see here from the login page, this is where I was logging in successfully with Philip
1129200	1139420	and Sigrid. I can see that here I created, for example, my other realistician that you saw.
1139420	1148420	Then here you can see the SQL injection as I was running it already. And then you can see here,
1148420	1156360	these are the tests that SQL map has been running. And you can see as or whatever has happened here.
1156360	1162800	You can also live stream these here. So, you would just see any SQL queries that come in here.
1162800	1169020	So, for example, this one here at the very end, this was the last query that I ran. When I ran or
1169020	1176720	added to the parameter of the ID one, I added the insert statement and added that bad actor
1176720	1182040	value to my data store. So, this is what happened here in the very last statement.
1182360	1188400	So, this is all pretty easy to see what my application has been doing. As long as you
1188400	1193160	write out the right log statements that make sense, you can then easily collect them and
1193160	1198240	see what is happening here. And you can also correlate that, for example, with the nginx logs,
1198240	1206300	for example, afterwards. I hope that makes sense so far. Oh, by the way, you could also search in
1206300	1212340	this one here with highlighting. You could then just say, let's quickly do that. In highlighting,
1212340	1218560	I want to search for insert. And I'm only interested in the insert statements right now.
1218560	1226260	So, we have one insert here. So, highlighting, another point done. Let's move on.
1226960	1233600	If you try to delete any record or drop the entire table, this will not work, by the way,
1233600	1238780	because if I was running this live, it would be very annoying if suddenly all the data was
1238780	1244560	gone and I couldn't demo anything anymore. So, the user that is running the SQL queries doesn't
1244560	1250900	have the permissions to delete or drop anything. Feel free to try it. You can see it in the logs,
1250900	1258340	but your requests will fail because those commands are not allowed here. Now, we know that something
1258340	1264840	is happening, but we also want to protect against it. And that is where mod security is coming into
1264840	1271180	play here. So, what is mod security if you have never used it? It's an open source tool. It's
1271460	1279420	a web application firewall. It's basically intercepting your traffic and either monitoring
1279420	1287020	stuff or even blocking or fixing things as it goes along. And it is based on rules where, for example,
1287020	1296220	this is an SQL injection, so I will deny that request, for example. It has two sets of rules.
1296220	1301520	There is a free core rule set that you can use that I'm using in my demo as well here.
1301520	1307740	So, for example, it can have a real-time blacklist lookup to deny certain things.
1307740	1315480	It can try to figure out an HTTP denial of service attack. It can detect SQL injections and other
1315480	1323460	things. There is also a commercial version which has a totally separate rule set which has some
1323460	1329760	more additional or maybe more advanced features. Though I'm not using that here, I'm only sticking
1329760	1338700	to the free version of the core rule set. So, to make this simple, I have the same vulnerable
1338700	1345920	application running behind Apache as well, only that Apache is running on port 8080. So, the code
1345920	1352620	is exactly the same. It's even referencing the same files and everything. It's just Apache has
1352620	1360060	more security added as well. So, if I run my SQL map against that...
1362340	1372840	So, let's run the SQL map command against Apache. So, I'm adding this against port 8080 now. Let's run this.
1374840	1378460	This won't be as successful as before.
1379640	1384220	So, you can see it is running. It hasn't figured out the database yet.
1385180	1391300	And it will not because it cannot really see any of the typical MySQL things.
1392020	1401010	Do I want to reduce the number of requests? No, I don't want to reduce the number of requests.
1401010	1406510	It should run all the requests here. So, please try everything you can.
1411030	1418170	And it's actually telling me that 126 times it ran into a 403, so into a forbidden.
1418170	1422990	It actually says that potentially this is running behind a web application firewall.
1422990	1430050	So, we could, with the dash dash tamper command, we could run various ways to tamper the requests
1430050	1438410	so that maybe they are not being detected by mod security or a web application firewall in general.
1438410	1446470	So, with sqlmap dash dash list tamper, it will show you all the different ways or all the different
1447110	1452770	scripts that are included to avoid tampering. There are some which are specific to mod security,
1452770	1456790	for example, but they don't make any difference for our example here.
1456790	1463610	So, I will keep this simple and I will just add the tampering. So, I will use a random agent.
1464150	1468650	So, if somebody tried to block on a specific user agent, we would avoid that.
1468650	1476430	And then we add spaces to the comments to maybe avoid detection, which won't be successful here.
1476890	1486070	But, well, yeah. Now, it did detect that this is protected by something.
1486070	1490930	I still want to continue my investigation.
1492370	1501630	But again, it won't be successful and it won't circumvent mod security in the current default
1501630	1505470	implementation that I have. I don't want to reduce the number of requests again.
1505470	1518400	I just want to run this plainly. Okay. And we have the same result, 118 requests were forbidden.
1518400	1526120	So, we have just 403 results here. Well, that is what you get.
1527860	1537860	Which leads us, by the way, so we could do two things here to actually look at. So, in the logs,
1538580	1544400	let's trash this filter here. And in the logs, just let me make this slightly smaller so I can
1544400	1556920	see what I'm typing here. So, we could, for example, say Apache error module and I want any.
1557240	1565460	So, and now you can see these are all Apache messages that we have been collected. So,
1566340	1572420	here now I'm in the live streaming mode. You can see these were rejected by mod
1572420	1578440	security already. We can, by the way, look at the details of one request here.
1579300	1587380	So, you can see that this is running. Filebit has collected it. I have this enriched this,
1587380	1591900	by the way, with the cloud information. So, you can see that this is running in an AWS data center.
1591900	1597760	You can see the AMI ID on which this is based, the instance ID. So, you could correlate this,
1597760	1603940	for example, to a specific instance ID. Or I have also added the host information,
1603940	1609060	which is like the operating system. So, for example, if an attack is only successful against
1609300	1614660	a specific version of Ubuntu, for example, you could easily filter down on this as well. So,
1614660	1621200	here you could just say, like, give me everything that is Ubuntu 18.04.4, because I know only this
1621200	1625500	one is vulnerable to some specific attack. And then you could just see the logs only for this
1625500	1631020	one, which is not the case here, but just as an additional idea of what kind of information you
1631020	1637220	might want to add. And then you can see this was an error. And you can see here the error came from
1637220	1644940	mod security, actually. And since I have live streaming on, let me run my tests once more.
1644940	1650680	And we should see... let me close this one here. We should see,
1651700	1657000	from this timestamp here, we should see stuff move on a bit.
1658720	1665400	I mean, it will take a little while. So, I run my attack. Apache is generating the log file. File
1665400	1671660	bit is picking up the logs. And then it will be stored in Elasticsearch. And after that,
1671660	1678240	you can then see it in Kibana. And you can see here now it is appearing in Kibana that this was
1678240	1686360	the attack that has just been happened. So, you can see the anomaly score is 65. And it was because
1686360	1693620	of these individual pieces that mod security figured out. So, we have an SQL injection,
1693620	1701540	cross-site scripting, remote filing inclusion, local file inclusion, remote code execution.
1701540	1708860	So, all of these factor into a total score. And then, based on the threshold, an attack might be...
1708860	1715120	or a request might be blocked or not. And this one here has been because it definitely breached
1715120	1724940	the rules here. So, the other thing that I actually wanted to show you is in Kibana, you can also have
1724940	1729280	as a dashboard, you can see a bit of an overview of what is happening in your system. So, for example,
1729280	1734580	you could see here, these were the average or the general requests that have been running. And let me
1734580	1741620	refresh this view because I think this data is slightly outdated now. So, this one here just
1741620	1747340	shows you the number of requests you can see here. These were the requests that I have just been
1747340	1752300	running with the SQL map. So, that's why they are peaking here a bit. And the different shades of
1752300	1758180	green are not ideal. Let me change this to more this pinkish. And I don't know, this I'll change
1758180	1764480	to blue so we can keep them a bit better apart. But you can see, for example, these here are the
1764480	1770640	403s. And we ran into a lot of 403s with more security now. Or you could see where the requests
1770640	1777180	are coming from. So, this here would be me. We surprisingly have some requests from the U.S. as
1777180	1786540	well. Nothing from Russia or China today, which is almost disappointing. But depending on the day,
1786540	1791300	you might have more requests from there. And then you could, for example, see which URLs were hit the
1791300	1798060	most and what response codes you got, what clients or browsers people used, how many errors you have.
1798060	1805100	And here you can see these are, for example, the errors that are the Apache log that have been
1805100	1810400	generated by more security. And you can see all of those. By the way, you could filter just on the
1810400	1815800	notices here or just on the errors if you have too much noise or just on one specific browser
1815800	1821960	or user agent, just to drill down into that data. Since this is all based on a search engine,
1821960	1825880	this is kind of the value of having a search engine. You can easily drill down and filter
1825880	1831860	into all of that data. So, this is what you can see in the dashboard here. But let's take another
1831860	1839540	look at ModSecurity. So, in ModSecurity, you have the setting SecRuleEngine and you have it either
1839540	1845920	off detection only or on. So, I have set it to on and that's what blocked the request, especially
1845920	1851380	when you want to roll this out to a production site. You probably want to start with detection
1851380	1857840	only to see what would happen if I would run this or if this would be active. And then you might
1857840	1863780	find interesting things like in San Francisco, there's Union Square. And I think ModSecurity
1863780	1870640	had a bug at some point where it saw Union anywhere in the post request, for example, and assumed
1870640	1876380	that this was an SQL Union attack, even though it was a regular address that was just
1876380	1882460	Union Square. It would block that. And I think the rules got slightly smarter now, so that this is
1882460	1888000	not having this false positive. But it's just one thing that you might run into and you might create
1888140	1893140	a bad user experience by being a bit too over-ambitious with your security settings. So,
1893140	1898600	that's why it makes sense to actually start with detection only and then review what is
1898600	1907460	happening. You can also limit how big file uploads are, for example, here or how big a post request
1907460	1913860	can be. So, these here would just limit how big requests are so that nobody can overload your
1913860	1919080	server with unnecessarily large requests. And obviously, it would make sense to change those
1919080	1926860	to some reasonable value for your setup. In terms of logging, what would be very helpful
1926860	1934280	is if you turned the logging format to JSON, because the non-JSON format is pretty terrible
1934280	1940040	to parse and nobody wants to do that. By the way, how would you parse that? How do you normally
1940040	1947040	parse? Writing a regular expression. Who enjoys writing a regular expression? I guess not so many.
1947040	1951640	Those who say they do, I would always say that this is the Stockholm syndrome, that you got
1951640	1956400	so used to writing regular expressions that you kind of accepted that this is the right way to do
1956400	1963360	stuff. But I am lazy and you should be lazy too. So, if you can avoid it, don't write regular
1963360	1969180	expressions. It's just... maybe it's job security. So, maybe that's why you would want to do it. But
1969180	1974480	otherwise, it's just a lot of work and it might go wrong and there are too many edge cases. So,
1974480	1978940	if you can have something in a structured format, ideally JSON, since Elasticsearch also stores
1978940	1985000	JSON, that would definitely be preferred. So, that is what I have done here. I have
1985760	1994680	taken that JSON format. Who remembers where or as what we have stored it? How would I see my
1995360	2003860	logs for that? Let me switch back here. Let's remove this. It was application.
2006570	2022400	Space, colon, and it was mod security. And if I stop streaming and filter down on this one,
2022400	2029720	it will show me just the logs from mod security. Which has, even though it's a JSON format, has a
2029720	2038060	pretty terrible format. So, let's look at one of those here. So, we're looking at the details. By
2038060	2043620	the way, the other option would be to see the surrounding. So, timestamp wise, it could show
2043620	2049400	you a couple of messages before and after, potentially from Nginx or Apache or whatever
2049400	2055700	other logs you collected around that time, which might be kind of related to just to have this
2055700	2061240	overview of this specific timeframe. That would have been the other option, but we don't need that
2061240	2066800	here. So, here, these are all the pieces of information we have connected. So, application
2066800	2073080	mod security, that was the field on which I actually filtered down. And you can see, this was
2073080	2081740	the pattern which we used to actually figure out that this was this specific attack here. So,
2081740	2089660	it was detected as a remote code execution Windows command injection, which I'm not sure that was
2089660	2095440	really what happened, but it was detected as that. And you can see, if you like writing regular
2095440	2102700	expressions, if you're in that Stockholm syndrome, then you will feel right at home here. So,
2102700	2108720	enjoy the glory of the regular expression here. And then you can see all the details of this rule
2108720	2115440	and how it is structured. Yeah, which might not be the most readable format, but this is what is
2115440	2120920	being used here. And you have various other fields where you can see what has been up here.
2123280	2128920	Cool. And again, we added the cloud information, we added the host information, just in case this
2128920	2134920	makes sense. If you don't use that, don't edit, because it will just add more weight to your logs
2134920	2141880	in Elasticsearch, but it's just easy to collect. Just to give you an idea how I got those,
2141880	2152100	just add this one here. I just said add cloud metadata and add host metadata, and those two
2152100	2162860	would just add that to the logs. So, here is the Ubuntu set, for example, the settings about Ubuntu,
2162860	2168180	and this one has the one about AWS. And after that here, by the way, I just renamed one field,
2168180	2173720	because otherwise I have a collision of how fields are named, and this is just an easy way to
2173720	2180040	fix up some data when you collect it. And that's all there is to this.
2182320	2186640	You can also write custom rules. So, I have added a custom rule,
2186640	2191660	and this might be very useful, like you have, I don't know, a forum software, and somebody is
2191660	2199300	posting ads for Rolexes or Viagra or whatever other stuff you don't want to have there.
2199300	2206460	I have created a different rule. So, what I tried to do here is, on my create.php file,
2206460	2215540	I'm adding a rule that will deny any request that detects a fake SHI. So, SHI is our CEO.
2215540	2219980	Let's assume there should be no other SHI in the company, otherwise somebody is trying to add a
2219980	2227760	fake SHI. So, when a post happens, I check the body of that post and say, like, if there is the
2227760	2237040	string SHI or Bannon in the post, I want to reject that request. So, let's head over to the browser
2237040	2244920	to see how that was going. So, this one is not running against Nginx, but let's head over to
2244920	2249840	Apache. So, again, this is running exactly the same thing. It's using the same database, the same
2249840	2256800	files. When I try to log in here, by the way, the SQL injection is not working anymore. So, if I
2256800	2268880	tried Philip and I tried or true, this would then, for example, reject my request because it detected
2268880	2276960	an SQL injection and would just kick me out. So, I would have to log in correctly. So,
2276960	2283400	secret was the password. If I do that, I can now create a regular user. So, let's say I want to
2283400	2291020	create SHI other and he is also based in San Francisco and he has some salary and he has a
2291020	2299280	password. And when I try to send in that other SHI, again, I get a rejection because the fake
2299280	2310420	SHI rule kicked in here. If I stream the logs for this one, it should, as the very last rule,
2310420	2317400	in a moment show me or already now show me that we just detected this fake SHI and rejected the
2317400	2326460	fake SHI. So, this is showing that my rule worked and we rejected the fake SHI. So, even if you have
2326460	2333060	some broken application where you cannot easily add like a spam filter or fix something else,
2333060	2339800	you could just write some custom rules in what security to reject certain things or filter out
2339800	2346580	certain things that you don't think should end up in your application. And then you have a more or
2346580	2352120	less smiling SHI again because there cannot be any fake SHIs. And this was the log message that we
2352120	2359960	have just seen, basically. ModSecurity is not 100% foolproof though. So, for example, if you would
2359960	2368640	run this one here with these parentheses or one equals one and parentheses, this will still allow
2368640	2375840	you to bypass the login. So, let me just quickly copy that and try it in the form.
2375840	2385840	So, here we have a forbidden. I'll try to... I need to create another random user to actually pass
2385840	2393740	or get logged out. So, let's do that. Now, I'm trying to log in again and now I'm...
2394800	2402520	foobar doesn't exist, but I add that or query. And this, even though there is ModSecurity here
2402520	2409660	at play, I still bypass that. So, ModSecurity is not 100%. There are potentially ways to work
2409660	2415400	around it, but it will filter out some things and you might have to tweak your rules according to
2415400	2423500	what might be valid input or not. Yes, everything is terrible and everything is on fire and what
2423500	2430600	the fuck were we even thinking? I guess that's a very good description of security. So, to wrap
2430600	2437080	this up, security incidents generally come in three levels. The for your information, what the
2437080	2445680	fuck, and oh my god. And today I think we were only in the between for your information, what the
2445680	2453440	fuck. It was definitely not the oh my god yet, though you frequently see those in the media
2453440	2460160	anyway. Yeah, write better code than me and use the right libraries or frameworks, especially
2460160	2466760	around injections. Nobody should have SQL injections anymore by using the right tools
2466760	2473500	or libraries, unless you just pass around plain strings like me, which is not a smart idea.
2474040	2479880	Is anybody using ModSecurity, for example? Yeah, quite a few people. So, for example, if you're
2479880	2488220	using GitLab and deploy it in Kubernetes, since almost a year ago on the Ingress app,
2488960	2496180	ModSecurity has been added to GitLab. So, this is just one of many places where ModSecurity
2496180	2503940	is being used as of today. And for the Elastic Stack, that's also pretty widely used just for
2503940	2510320	the security use case. Three companies which I think I can name publicly would be Slack,
2510320	2515720	anything related to security around Slack. Mozilla has a project and Cisco is also a pretty heavy
2515720	2523400	user just to get logs into the Elastic Stack around security. And that's not all. We recently
2523400	2530180	added SIEM and Endpoint Security. So, those would play into that. And since I think I have like one
2530180	2537520	minute or so extra, let me give you a quick idea of what SIEM might look like. So, here I have
2537520	2545860	already, let me zoom out one level again. So, here these are all the requests in the last hour. You
2545860	2551020	can see these are all the hosts. I have a single host here to keep it simple. All my login requests
2551020	2556980	into that one were successful and I could then dive into all the hosts that I have identified.
2556980	2563260	You can see it also gives you a good overview of what is running on that machine here. And for
2563260	2568340	example, I might be interested in not the login attempts because, well, they were all successful.
2568340	2573740	That's not so interesting. Let's have a quick look at the events. In the events, we can see everything
2573740	2579780	that happened on that host here. So, for example, maybe somebody said like that application is kind
2579780	2585320	of fishy. And let's have a look at what is happening on that host. So, that is what we are
2585320	2590800	trying to get here. And these are all the events that we have collected over time. So, we have
2590800	2597500	NetFlow. So, any network connections that have been going on, user logins, processes that have
2597500	2605360	been stopped or started, sessions, etc. And then at the events further down here, you could see,
2605360	2614760	for example, that this is already our application with Apache. So, let's drag and drop this one
2614760	2623380	into our timeline filter here because I want to see anything that touches this port 8080.
2623480	2628660	And then you could see, okay, this was Apache that was serving the request. You could see
2628660	2635220	where it was coming from. You could see that many packets, etc. And then dive further down
2635220	2642800	into that information and figure out why maybe your Apache server is throwing so many errors
2642800	2648660	because ModSecurity is blocking lots of bad stuff, for example, or whatever else you might
2648660	2655120	find interesting in here. And you can slice and dice your data any way you want. Okay, enough of
2655120	2663460	this. Let's close this out. So, ModSecurity and logging go very well hand in hand. If you just
2663460	2669260	run ModSecurity without looking what it actually blocks or rightfully or wrongfully blocks,
2669260	2673820	you're pretty much blind. So, you want to combine this with some proper logging,
2673820	2679740	as well as logging in general in your application to figure out if somebody has potentially breached
2679740	2685240	something or not. And especially if somebody breached it, when did that happen and what
2685240	2690060	were they actually able to do to your system to actually know afterwards and not just
2690060	2698660	stir some murky water. If you want to try out all of this, the code is on my GitHub page in
2698660	2703580	ModSecurity.log. If you want to try out the dashboards, I will also let this run. I hope
2703580	2710060	nobody kills it. You will automatically be logged into Kibana with a read-only user.
2710440	2714260	So, this is running a reverse proxy that automatically logs you in with a read-only
2714260	2720620	user. So, if you go to dashboard.xiera.wtf, you will end up in Kibana already logged in and you
2720620	2726260	can play around with the logs UI or with the dashboards and just see what is going on there.
2726260	2732340	The application is running on xiera.wtf, so any requests that you generate, you should be able to
2732340	2739380	find in Kibana as well. Just if lots of users are doing that, it might overload the system.
2739380	2744420	No guarantees that it will be up or you might see a lot of noise because of everybody else doing it,
2744420	2750520	but happy exploring around that. If you want to get the slides, that is the QR code,
2750520	2758460	I will share the link if I can live as well. And with that, let's head over to the questions.
2759180	2762480	Find me on this course. I'm happy to answer whatever you have or
2762480	2767580	ping me on Twitter. I'm also happy to answer there. Thanks a lot for joining.