WEBVTT 00:13.580 --> 00:15.060 Hey, welcome back. 00:15.060 --> 00:22.820 Hey, AppSecVillage, we're getting down to the last few speakers of our time together here on day three of AppSecVillage at DEF CON 2020. 00:22.820 --> 00:26.980 I want to thank you for coming and for making this possible. 00:26.980 --> 00:31.280 I want to make sure to thank our sponsors and the folks that run AppSecVillage. 00:31.600 --> 00:34.580 They've done a wonderful job both years that we've been here. 00:34.580 --> 00:37.360 This has become my home away from home when I come to DEF CON. 00:37.360 --> 00:40.900 I look forward to it every year and I hope you do too. 00:40.900 --> 00:46.940 And I hope to see you again next year when we do this, hopefully in person, but we're going to do it one way or the other. 00:47.000 --> 00:52.300 Our next discussion is going to be Secure Your Code, Injections and Logging by Philip Kren. 00:52.300 --> 00:55.080 Philip lives to demo interesting technology. 00:55.200 --> 01:02.720 Having worked as a web infrastructure and database engineer for more than 10 years, Philip is now working as a developer advocate at Elastic. 01:03.120 --> 01:15.850 Based in Vienna, Austria, he is consistently traveling Europe and beyond to speak and discuss on topics such as open source software, search technology, databases, infrastructure, and security in general. 01:16.160 --> 01:19.560 Please welcome Philip to the AppSecVillage stage. 01:20.620 --> 01:22.240 Hi, I'm Philip. 01:22.240 --> 01:24.520 Let's dive into the next session. 01:24.520 --> 01:27.780 Let me share my screen. 01:28.760 --> 01:30.440 And there we go. 01:30.500 --> 01:33.220 Secure Your Code, Injections and Logging. 01:33.220 --> 01:34.720 Let's see what we can do here. 01:34.840 --> 01:44.320 So when we talk about security, we normally keep telling ourselves that everything is fine, especially when we say everybody's job is security. 01:44.320 --> 01:48.740 It's normally nobody's job, and we can all pretend that this is all good. 01:48.740 --> 01:54.480 And then you look at the OWASP top 10, and you see at number one is still injections. 01:54.480 --> 02:07.720 So good old SQL injection, maybe LLAP injection, NoSQL injection, or whatever other form of unsecured user input you are throwing at your application, and that is wreaking havoc. 02:07.720 --> 02:13.520 And then you might reconsider and say like, well, maybe this is not so fine anymore. 02:13.780 --> 02:32.460 And then you look further down the OWASP top 10 list, and then you see, for example, number 10, insufficient logging and monitoring, where you then realize that on average, it takes something like 200 days or so until you normally find a breach, maybe a bit quicker if the attacker asked for some ransomware or ransom money, 02:32.460 --> 02:34.620 but it might take a while. 02:34.620 --> 02:44.400 And then you both or you might end up at the point where you say, well, this is no longer fine or good, and everything is terrible and on fire. 02:44.400 --> 02:46.240 And we don't want to get to this point. 02:46.240 --> 02:48.820 So what can we do about that? 02:49.440 --> 02:50.520 I'm Philipp. 02:50.520 --> 02:54.540 I work for Elastic, the company behind Elasticsearch, Kibana, Beats, Logstash. 02:54.540 --> 02:58.160 You might have heard of us or might be using us already somewhere. 02:58.160 --> 03:00.180 My official title is developer advocate. 03:00.180 --> 03:02.940 So I mostly talk about the good stuff that we do. 03:02.940 --> 03:04.400 And today is no exception. 03:04.400 --> 03:06.360 So let's see what we can do here. 03:07.020 --> 03:10.540 I generally build highly monitored Hello World applications. 03:11.000 --> 03:12.480 Today is no exception. 03:12.480 --> 03:15.700 And we will pretty much dive into that right away. 03:15.700 --> 03:21.880 So like in any good cooking show, I have prepared something and a very bad application. 03:21.880 --> 03:25.400 So xiera.wtf is where this is running. 03:25.400 --> 03:27.340 And I will let this run. 03:27.340 --> 03:30.340 So while this is pre-recorded, I will let the demo run. 03:30.340 --> 03:35.340 So you can actually destroy it or play around with it live afterwards as well. 03:35.340 --> 03:41.320 If you're wondering where Xiera is coming from, it has something to do with my name. 03:41.320 --> 03:43.560 If you're bored, try to figure it out. 03:43.560 --> 03:48.340 And obviously, WTF is the right domain for any demo. 03:48.340 --> 03:50.340 So let's hope this works out. 03:50.340 --> 03:57.400 And let's head over to my very bad and well, average demo application. 03:57.400 --> 04:01.080 So we have some Elastic employees here. 04:01.080 --> 04:07.000 And I added our CEO Shai, one of our co-founders Simon, and myself. 04:07.460 --> 04:11.500 And well, maybe we want to add more employees. 04:11.500 --> 04:14.740 And to do that, obviously, you need to log in. 04:14.740 --> 04:23.080 If you want to regularly log in, my username is Philip, and my password is secret. 04:23.660 --> 04:25.160 This works. 04:25.160 --> 04:33.700 And then we can say, other edition is, I don't know, maybe in New York. 04:33.720 --> 04:38.940 And they have some salary and they also have a password of let's say secret. 04:38.980 --> 04:42.000 Okay, so that worked. 04:42.000 --> 04:47.960 But looking at this form, this is not secure. 04:47.960 --> 04:52.720 And by design, this is using the Hello World application of SQL injections. 04:52.720 --> 05:00.080 So we can just use one parameter like Philip and in the other feed, we will just do an or true. 05:00.100 --> 05:07.120 So I will do an or true and try to log in like that. 05:07.800 --> 05:18.720 This worked, unfortunately, and I can now add a bad user in, I don't know, from Gotham, and have a much higher salary. 05:18.760 --> 05:20.860 And they also have some random password. 05:21.400 --> 05:26.480 Okay, so we have added the bad user, nothing super surprising here. 05:28.020 --> 05:35.560 This has the classic SQL injection problem, I don't escape my user input, I just pass in whatever I'm doing here. 05:35.560 --> 05:45.680 I'm hashing my password, but I'm just breaking out of that and doing or true, for example, or whatever other variation you want to have of this. 05:46.200 --> 05:48.460 You've seen this plenty of times. 05:48.600 --> 05:56.340 It's called an SQL injection, or some people call it accidental GraphQL, if that is more your thing, and which might be the deeper thing. 05:56.340 --> 05:59.600 But classically, this is an SQL injection. 05:59.600 --> 06:05.780 And I'm pretty sure all or most of you have seen little bobby tables. 06:06.180 --> 06:18.120 If you give your child the right name, like drop table students, you will have a lot of fun at school, or your school just should write better systems. 06:18.120 --> 06:19.820 But that's a different discussion. 06:19.820 --> 06:22.900 So let's have another look around. 06:22.900 --> 06:25.440 So I can also display users. 06:25.440 --> 06:30.480 So for example, looking at our CEO shy here, this looks all nice and good. 06:30.480 --> 06:32.560 And you can go back and forth. 06:32.640 --> 06:38.240 What I'm doing here is I'm just passing the ID of my user directly in. 06:38.240 --> 06:41.980 So I could just change that to two and pass over to Simon. 06:43.480 --> 06:49.920 This is insecure, again, like unsurprisingly, since this is a broken application. 06:49.960 --> 06:58.240 And before we try to actually throw some bad parameters as it, I want to quickly jump to SQL map. 06:58.240 --> 07:05.380 If you've never seen or used SQL map, it's an automatic SQL injection and database takeover tool. 07:05.380 --> 07:21.860 And you can basically say like, hey, there is this parameter like the ID and I would like to check that and it will go through a lot of variations of SQL injections to actually say like, well, this looks like it might be interesting or not. 07:21.860 --> 07:30.920 Okay, so let's open my shell and let me paste in my command so you don't have to see me type this. 07:30.920 --> 07:35.120 So I'm running SQL map against that URL that I've shown you. 07:35.120 --> 07:38.440 And at the end, we add a purge because I've run this before. 07:38.440 --> 07:43.000 And purge will just make sure we don't use any records that we've created before. 07:43.000 --> 07:44.780 So we will start fresh. 07:44.780 --> 07:53.000 So we will start running this and SQL map will just figure out what is possible here. 07:53.000 --> 07:57.140 It will ask me a couple of questions of how it should treat this. 07:58.180 --> 08:04.400 So you can see it's already trying to do stuff and it figured out that I'm using MySQL here. 08:04.400 --> 08:06.740 So yes, this is the right database. 08:06.740 --> 08:13.100 And I want to skip tests for other databases because this is MySQL, we can focus on this. 08:13.100 --> 08:18.100 And I'm okay to run these risk level queries here. 08:19.300 --> 08:22.720 So you can see it's trying out various things. 08:24.780 --> 08:31.880 And at the end, it will then tell me what is actually vulnerable. 08:32.000 --> 08:41.320 It will also ask me if it should run or check any other parameters, which we will not do here because there's only one parameter that is interesting. 08:41.600 --> 08:44.920 I don't want to test anything else, so we're done here. 08:44.920 --> 08:55.420 So looking at the result, you can see, for example, okay, this is running against NGINX and against the more or less recent version of MySQL. 08:55.420 --> 09:00.320 We would have some logs in this folder here, though they're not super interesting. 09:00.320 --> 09:05.160 They're mostly what we have here and all the SQL commands that have been running. 09:05.260 --> 09:08.500 So we will stick to what we see here. 09:08.500 --> 09:13.960 So you can see in 81 HTTP requests, we figured out a couple of things. 09:13.960 --> 09:30.880 So for example, we see a Boolean-based blind attack where you can do something like ID 1 and something equals something, or you can have a time -based attack where we add a sleep and we see indeed the query takes that amount of sleep plus a little more maybe, 09:30.880 --> 09:36.000 or that you can have a union query and that you can try to join data together. 09:36.000 --> 09:40.500 So this is clearly vulnerable to something. 09:40.500 --> 09:44.140 So let's have a look of what I have actually in here. 09:45.540 --> 09:47.340 Again, a very bad query. 09:47.340 --> 09:49.080 I'm just taking that parameter. 09:49.080 --> 09:51.080 I'm not escaping it in any way. 09:51.080 --> 09:53.580 I'm just trimming any spaces off of it. 09:53.760 --> 09:57.920 And then I take this query and I run it. 09:58.580 --> 10:04.280 The good thing here, by the way, is that I actually logged the query that people are sending to my system. 10:04.280 --> 10:09.080 So this will help us afterwards to figure out what has actually been happening here. 10:09.080 --> 10:14.260 So we will log this into a var log app log file. 10:14.320 --> 10:20.240 The bad thing here is that I'm using MySQL multi-query. 10:20.240 --> 10:23.760 So this will allow me to run multiple queries in one go. 10:23.760 --> 10:29.480 So besides that select, I might do another insert, for example. 10:29.840 --> 10:43.100 So if I add this to the end of the URL and it will automatically then be URL escaped and work properly, I could just say insert into employees some bad actor. 10:43.100 --> 10:48.580 So if I run that, let's see in our application. 10:49.400 --> 10:53.060 Right now we have a bad user, but we don't have a bad actor. 10:53.060 --> 11:04.000 So if I have to that parameter here, I just add after semicolon the insert statement, run this, it looks like nothing happened. 11:04.000 --> 11:08.480 But when I go back here, you now have a bad actor. 11:08.840 --> 11:13.440 So this works as expected or not expected. 11:13.820 --> 11:16.120 Maybe this is mostly up to your expectation. 11:16.120 --> 11:21.020 By the way, this doesn't escape like the visualization in any way or the output. 11:21.020 --> 11:30.460 So you could run any random JavaScript here, which I didn't do because it will be kind of annoying to have various pop-ups pop up. 11:30.480 --> 11:34.120 You could break the entire site with some weird JavaScript. 11:34.160 --> 11:36.080 So there is no other protection here. 11:36.080 --> 11:38.600 So we don't do any escaping. 11:38.600 --> 11:42.360 So this is easy to pick apart. 11:43.600 --> 11:45.840 So what's going on in our application? 11:45.840 --> 11:54.420 Now is the interesting part, like at first before I want to secure it, I want to get a bit of a better idea of what even has been going on in my application. 11:55.080 --> 12:00.120 And yeah, if you don't have any log files, nobody can remove your log files. 12:00.120 --> 12:06.640 So let's assume we want to have log files and we also want to ship them off and have them hopefully in a secure location. 12:06.640 --> 12:17.740 So even if somebody breaks into that instance and manipulates it, they cannot actually do anything with your logs and you still keep some trace of what has been happening. 12:18.120 --> 12:23.460 So we want to collect all the things and having a log file is a very good starting point. 12:23.460 --> 12:26.800 Maybe ship it off in case somebody breaks into the instance. 12:27.120 --> 12:32.480 The main downside is these log files have the ugly tendency to get very large. 12:32.480 --> 12:42.400 So maybe at some point you end up as this little submarine, basically, and then you have gigabytes and gigabytes of log files in which you try to find something. 12:42.760 --> 12:46.020 Maybe successfully, maybe not so successfully. 12:47.320 --> 12:55.360 Scrap is great until you have multiple servers and various big log files, because then it's not going to be so much fun. 12:55.360 --> 12:59.620 So you probably want to use something else that is making your life a bit easier. 12:59.820 --> 13:03.860 One of the most wily tools there probably is the Elk Stack. 13:03.860 --> 13:11.580 You can see Elasticsearch, Logstash, and Kibana, one sitting on top of the other, and that makes up the good old Elk. 13:11.640 --> 13:20.660 So Elasticsearch is the thing that stores your data, Logstash is the thing that gets, parses, and enriches your data, and Kibana can visualize it. 13:20.660 --> 13:28.420 By the way, when I say parse, you normally have some logline, and out of that logline you want to extract some piece of information . 13:29.000 --> 13:37.080 So, for example, you have a log level, or you have the URL where something came from, or maybe you have some user information, or a timestamp. 13:37.080 --> 13:54.360 All of these you might want to extract to actually be able to filter down and then say, oh, give me all the logs from this specific time frame, or for this specific user, or for this specific application, or URL within the application, or any way you want to slice and dice your data. 13:55.160 --> 14:11.680 When I say enrich, it might be something like you have the IP address of the person requesting the website, and then you could add the GeoIP information to that IP address, and then you could draw it out on a map, and then you could draw where are all my visitors coming from, 14:11.680 --> 14:12.500 for example. 14:12.500 --> 14:20.720 Or you could just filter down and say like, oh, give me all the users coming from Russia or China, because that's kind of unusual. 14:20.720 --> 14:22.640 I normally don't have any visitors from there. 14:22.700 --> 14:25.900 Let's figure out what they have been up to. 14:26.780 --> 14:28.100 That works. 14:28.300 --> 14:36.080 The only thing is that by now the stack has evolved slightly, so we still have Kibana for the visualization Elasticsearch that stores it. 14:36.080 --> 14:41.000 Sometimes, by the way, people are slightly confused and ask like, what is the database behind Elasticsearch? 14:41.000 --> 14:43.260 There is no other database anymore. 14:43.420 --> 14:51.620 There is the library Apache Lucene, which writes it down to disk, but otherwise there is no other data store behind that anymore. 14:52.260 --> 15:06.080 Logstash is still that versatile, bit more like ETL tool, so extract, transform, load, so it can get data from various systems, change and transform that, then push that out to Elasticsearch or other systems. 15:06.440 --> 15:14.120 But to make life a bit easier and also slimmer, we have added bits in various versions so far. 15:14.120 --> 15:23.020 So, for example, for log files, we have Filebit, and it basically tails a file over the network and a bit on steroids. 15:23.020 --> 15:30.320 So, we basically will say like, this is a log file that I'm interested in, take that, tail it, store it into Elasticsearch. 15:30.320 --> 15:40.360 To keep my demo simple, I will actually skip Logstash and Filebit will tail my log file, put the data into Elasticsearch, and then we can visualize it in Kibana directly. 15:41.840 --> 15:46.580 So, let's see what we have actually here. 15:46.580 --> 15:55.340 So, maybe before we dive into the visualization, let's take a very quick look at what we have done here. 15:55.340 --> 16:04.560 So, this is the configuration of this entire demo, and this is all being set up automatically, so you can get all the configs afterwards. 16:04.760 --> 16:10.220 What I have done here is, there are maybe two interesting things in this file. 16:10.220 --> 16:13.280 So, this is the Filebit configuration HTML. 16:13.360 --> 16:20.920 What will be interesting at first is, so, I'm getting some log files from the system, and there are so-called modules. 16:20.980 --> 16:24.680 These are four well-known things that you have often installed. 16:24.680 --> 16:38.540 So, for example, I have Nginx running here, as we have seen in SQLMAP before, and it will know automatically for the server that I'm running, where is that log file located by default, and what does it look like. 16:38.540 --> 16:50.100 So, basically, by saying, use this Nginx module, it will then pick up the right file, be able to parse it in its individual pieces, and then store it, and we're done. 16:50.100 --> 16:54.800 We don't need to care about any log file locations or anything anymore. 16:54.800 --> 17:00.520 With that var log app log, which my application is writing, it's not so standardized. 17:00.520 --> 17:08.120 So, for this one, I have this configuration here, where I basically tell FileGrid, this is the file that you need to collect. 17:08.240 --> 17:26.020 So, it is of the type log, this is the path, you could also use a wildcard and point to a directory, for example, and then what is also interesting, I will store this with an additional field called application app, and this is what we will be able to filter on right away, 17:26.020 --> 17:33.440 because I'm collecting all these other logs, so we might see a lot of logs coming together in Kibana right away. 17:33.620 --> 17:36.500 So, let's take a quick look. 17:36.520 --> 17:41.680 So, maybe you have used Discover before, which was the kind of like old way to do that. 17:41.680 --> 17:49.600 We have now a dedicated logs UI, where you can filter down and live stream data. 17:49.600 --> 17:58.700 So, with the live streaming, this is a bit more like tail F, where you just see all the log messages that come in right away. 17:58.700 --> 18:05.460 So, whatever happened here to my SQL, we might investigate that later on, but it will just live tail anything that is happening here. 18:05.460 --> 18:21.540 But what I'm interested in right now is application, as I have said that before, and let me change my screen zoom in slightly, and my application's name was app. 18:21.560 --> 18:28.060 So, I just want to filter down only on the application that I'm running here. 18:28.060 --> 18:35.200 I don't want to see anything about Apache, or Nginx, or anything else that I have running. 18:35.680 --> 18:36.800 Here we go. 18:36.840 --> 18:41.640 So, this is the log file of my log that I have collected. 18:41.640 --> 18:51.080 So, you can see here from the login page, this is where I was logging in successfully with Philip and Sigrid. 18:52.260 --> 18:59.420 I can see that here I created, for example, my other realistician that you saw. 18:59.420 --> 19:05.520 Then here you can see the SQL injection as I was running it already. 19:06.100 --> 19:12.740 And then you can see here, these are the tests that SQL map has been running. 19:12.740 --> 19:16.360 And you can see as or whatever has happened here. 19:16.360 --> 19:19.000 You can also live stream these here. 19:19.000 --> 19:22.800 So, you would just see any SQL queries that come in here. 19:22.800 --> 19:27.520 So, for example, this one here at the very end, this was the last query that I ran. 19:27.880 --> 19:39.020 When I ran or added to the parameter of the ID one, I added the insert statement and added that bad actor value to my data store. 19:39.020 --> 19:42.040 So, this is what happened here in the very last statement. 19:42.360 --> 19:47.500 So, this is all pretty easy to see what my application has been doing. 19:47.500 --> 19:54.920 As long as you write out the right log statements that make sense, you can then easily collect them and see what is happening here. 19:54.920 --> 19:59.540 And you can also correlate that, for example, with the nginx logs, for example, afterwards. 20:02.440 --> 20:04.320 I hope that makes sense so far. 20:04.380 --> 20:07.560 Oh, by the way, you could also search in this one here with highlighting. 20:07.560 --> 20:11.200 You could then just say, let's quickly do that. 20:11.200 --> 20:15.540 In highlighting, I want to search for insert. 20:15.540 --> 20:18.560 And I'm only interested in the insert statements right now. 20:18.560 --> 20:20.720 So, we have one insert here. 20:21.120 --> 20:24.900 So, highlighting, another point done. 20:24.900 --> 20:26.260 Let's move on. 20:26.960 --> 20:41.000 If you try to delete any record or drop the entire table, this will not work, by the way, because if I was running this live, it would be very annoying if suddenly all the data was gone and I couldn't demo anything anymore. 20:41.000 --> 20:47.820 So, the user that is running the SQL queries doesn't have the permissions to delete or drop anything. 20:47.820 --> 20:49.440 Feel free to try it. 20:49.540 --> 20:55.580 You can see it in the logs, but your requests will fail because those commands are not allowed here. 20:56.020 --> 21:01.360 Now, we know that something is happening, but we also want to protect against it. 21:01.360 --> 21:05.800 And that is where mod security is coming into play here. 21:05.800 --> 21:08.880 So, what is mod security if you have never used it? 21:08.880 --> 21:10.740 It's an open source tool. 21:10.740 --> 21:13.220 It's a web application firewall. 21:13.220 --> 21:23.980 It's basically intercepting your traffic and either monitoring stuff or even blocking or fixing things as it goes along. 21:23.980 --> 21:32.540 And it is based on rules where, for example, this is an SQL injection, so I will deny that request, for example. 21:32.840 --> 21:36.220 It has two sets of rules. 21:36.220 --> 21:41.520 There is a free core rule set that you can use that I'm using in my demo as well here. 21:41.520 --> 21:47.740 So, for example, it can have a real-time blacklist lookup to deny certain things. 21:47.740 --> 21:52.600 It can try to figure out an HTTP denial of service attack. 21:52.600 --> 21:56.300 It can detect SQL injections and other things. 21:56.300 --> 22:07.040 There is also a commercial version which has a totally separate rule set which has some more additional or maybe more advanced features. 22:07.040 --> 22:12.940 Though I'm not using that here, I'm only sticking to the free version of the core rule set. 22:13.580 --> 22:25.400 So, to make this simple, I have the same vulnerable application running behind Apache as well, only that Apache is running on port 8080. 22:25.400 --> 22:27.240 So, the code is exactly the same. 22:27.240 --> 22:29.900 It's even referencing the same files and everything. 22:29.900 --> 22:34.680 It's just Apache has more security added as well. 22:34.680 --> 22:40.060 So, if I run my SQL map against that... 22:42.340 --> 22:46.680 So, let's run the SQL map command against Apache. 22:46.680 --> 22:51.280 So, I'm adding this against port 8080 now. 22:51.280 --> 22:52.840 Let's run this. 22:54.840 --> 22:58.460 This won't be as successful as before. 22:59.640 --> 23:01.820 So, you can see it is running. 23:01.820 --> 23:04.220 It hasn't figured out the database yet. 23:05.180 --> 23:11.300 And it will not because it cannot really see any of the typical MySQL things. 23:12.020 --> 23:19.070 Do I want to reduce the number of requests? 23:19.070 --> 23:21.010 No, I don't want to reduce the number of requests. 23:21.010 --> 23:23.350 It should run all the requests here. 23:23.470 --> 23:26.510 So, please try everything you can. 23:31.030 --> 23:38.170 And it's actually telling me that 126 times it ran into a 403, so into a forbidden. 23:38.170 --> 23:42.990 It actually says that potentially this is running behind a web application firewall. 23:42.990 --> 23:58.410 So, we could, with the dash dash tamper command, we could run various ways to tamper the requests so that maybe they are not being detected by mod security or a web application firewall in general. 23:58.410 --> 24:09.970 So, with sqlmap dash dash list tamper, it will show you all the different ways or all the different scripts that are included to avoid tampering. 24:09.970 --> 24:16.790 There are some which are specific to mod security, for example, but they don't make any difference for our example here. 24:16.790 --> 24:21.490 So, I will keep this simple and I will just add the tampering. 24:21.490 --> 24:23.610 So, I will use a random agent. 24:24.150 --> 24:28.650 So, if somebody tried to block on a specific user agent, we would avoid that. 24:28.650 --> 24:36.430 And then we add spaces to the comments to maybe avoid detection, which won't be successful here. 24:36.890 --> 24:41.390 But, well, yeah. 24:41.830 --> 24:46.070 Now, it did detect that this is protected by something. 24:46.070 --> 24:50.930 I still want to continue my investigation. 24:52.370 --> 25:02.530 But again, it won't be successful and it won't circumvent mod security in the current default implementation that I have. 25:02.530 --> 25:05.470 I don't want to reduce the number of requests again. 25:05.470 --> 25:08.090 I just want to run this plainly. 25:13.160 --> 25:13.880 Okay. 25:13.880 --> 25:18.400 And we have the same result, 118 requests were forbidden. 25:18.400 --> 25:22.320 So, we have just 403 results here. 25:23.660 --> 25:26.120 Well, that is what you get. 25:27.860 --> 25:33.660 Which leads us, by the way, so we could do two things here to actually look at. 25:33.660 --> 25:40.980 So, in the logs, let's trash this filter here. 25:40.980 --> 25:47.100 And in the logs, just let me make this slightly smaller so I can see what I'm typing here. 25:47.320 --> 25:56.920 So, we could, for example, say Apache error module and I want any. 25:57.240 --> 26:03.880 So , and now you can see these are all Apache messages that we have been collected. 26:03.900 --> 26:08.600 So, here now I'm in the live streaming mode. 26:08.600 --> 26:13.660 You can see these were rejected by mod security already. 26:13.660 --> 26:18.440 We can, by the way, look at the details of one request here. 26:19.300 --> 26:23.460 So, you can see that this is running. 26:24.300 --> 26:25.780 Filebit has collected it. 26:25.780 --> 26:28.780 I have this enriched this, by the way, with the cloud information. 26:28.780 --> 26:31.900 So, you can see that this is running in an AWS data center. 26:31.900 --> 26:35.720 You can see the AMI ID on which this is based, the instance ID. 26:35.720 --> 26:39.820 So, you could correlate this, for example, to a specific instance ID. 26:39.820 --> 26:46.060 Or I have also added the host information, which is like the operating system. 26:46.060 --> 26:54.520 So, for example, if an attack is only successful against a specific version of Ubuntu, for example, you could easily filter down on this as well. 26:54.520 --> 27:03.320 So, here you could just say, like, give me everything that is Ubuntu 18.04.4, because I know only this one is vulnerable to some specific attack. 27:03.320 --> 27:12.640 And then you could just see the logs only for this one, which is not the case here, but just as an additional idea of what kind of information you might want to add. 27:12.660 --> 27:14.520 And then you can see this was an error. 27:14.520 --> 27:18.740 And you can see here the error came from mod security, actually. 27:18.740 --> 27:24.940 And since I have live streaming on, let me run my tests once more. 27:24.940 --> 27:28.540 And we should see... let me close this one here. 27:28.860 --> 27:37.000 We should see, from this timestamp here, we should see stuff move on a bit. 27:38.720 --> 27:41.820 I mean, it will take a little while. 27:41.820 --> 27:43.480 So, I run my attack. 27:43.480 --> 27:45.200 Apache is generating the log file. 27:45.200 --> 27:48.180 File bit is picking up the logs. 27:48.180 --> 27:50.800 And then it will be stored in Elasticsearch. 27:50.800 --> 27:53.080 And after that, you can then see it in Kibana. 27:53.080 --> 28:01.180 And you can see here now it is appearing in Kibana that this was the attack that has just been happened. 28:01.260 --> 28:05.080 So, you can see the anomaly score is 65. 28:05.080 --> 28:10.120 And it was because of these individual pieces that mod security figured out. 28:10.120 --> 28:21.540 So, we have an SQL injection, cross-site scripting, remote filing inclusion, local file inclusion, remote code execution. 28:21.540 --> 28:25.040 So , all of these factor into a total score. 28:25.080 --> 28:28.860 And then, based on the threshold, an attack might be... 28:28.860 --> 28:31.240 or a request might be blocked or not. 28:31.240 --> 28:36.660 And this one here has been because it definitely breached the rules here. 28:37.000 --> 28:48.780 So, the other thing that I actually wanted to show you is in Kibana, you can also have as a dashboard, you can see a bit of an overview of what is happening in your system. 28:48.780 --> 28:54.180 So, for example, you could see here, these were the average or the general requests that have been running. 28:54.180 --> 28:59.260 And let me refresh this view because I think this data is slightly outdated now. 29:00.320 --> 29:04.380 So, this one here just shows you the number of requests you can see here. 29:04.380 --> 29:08.640 These were the requests that I have just been running with the SQL map. 29:08.640 --> 29:10.960 So, that's why they are peaking here a bit. 29:10.960 --> 29:13.520 And the different shades of green are not ideal. 29:13.520 --> 29:16.580 Let me change this to more this pinkish. 29:16.580 --> 29:21.500 And I don't know, this I'll change to blue so we can keep them a bit better apart. 29:21.500 --> 29:25.440 But you can see, for example, these here are the 403s. 29:25.440 --> 29:29.220 And we ran into a lot of 403s with more security now. 29:29.220 --> 29:31.500 Or you could see where the requests are coming from. 29:31.500 --> 29:33.840 So, this here would be me. 29:34.500 --> 29:37.020 We surprisingly have some requests from the U.S. 29:37.020 --> 29:37.880 as well. 29:38.840 --> 29:44.660 Nothing from Russia or China today, which is almost disappointing. 29:45.200 --> 29:48.780 But depending on the day, you might have more requests from there. 29:48.780 --> 29:58.060 And then you could, for example, see which URLs were hit the most and what response codes you got, what clients or browsers people used, how many errors you have. 29:58.060 --> 30:06.980 And here you can see these are, for example, the errors that are the Apache log that have been generated by more security. 30:06.980 --> 30:08.440 And you can see all of those. 30:08.440 --> 30:19.380 By the way, you could filter just on the notices here or just on the errors if you have too much noise or just on one specific browser or user agent, just to drill down into that data. 30:19.380 --> 30:23.860 Since this is all based on a search engine, this is kind of the value of having a search engine. 30:23.860 --> 30:27.400 You can easily drill down and filter into all of that data. 30:27.400 --> 30:30.140 So, this is what you can see in the dashboard here. 30:30.880 --> 30:33.900 But let's take another look at ModSecurity. 30:34.320 --> 30:41.880 So, in ModSecurity, you have the setting SecRuleEngine and you have it either off detection only or on. 30:41.880 --> 30:49.500 So, I have set it to on and that's what blocked the request, especially when you want to roll this out to a production site. 30:49.500 --> 30:56.940 You probably want to start with detection only to see what would happen if I would run this or if this would be active. 30:56.940 --> 31:02.820 And then you might find interesting things like in San Francisco, there's Union Square. 31:02.820 --> 31:17.540 And I think ModSecurity had a bug at some point where it saw Union anywhere in the post request, for example, and assumed that this was an SQL Union attack, even though it was a regular address that was just Union Square. 31:17.540 --> 31:18.840 It would block that. 31:18.840 --> 31:24.400 And I think the rules got slightly smarter now, so that this is not having this false positive. 31:24.400 --> 31:32.960 But it's just one thing that you might run into and you might create a bad user experience by being a bit too over-ambitious with your security settings. 31:32.960 --> 31:39.540 So, that's why it makes sense to actually start with detection only and then review what is happening. 31:39.860 --> 31:48.740 You can also limit how big file uploads are, for example, here or how big a post request can be. 31:48.960 --> 31:56.340 So, these here would just limit how big requests are so that nobody can overload your server with unnecessarily large requests. 31:56.340 --> 32:01.780 And obviously, it would make sense to change those to some reasonable value for your setup. 32:02.580 --> 32:17.240 In terms of logging, what would be very helpful is if you turned the logging format to JSON, because the non-JSON format is pretty terrible to parse and nobody wants to do that. 32:17.240 --> 32:19.340 By the way, how would you parse that? 32:19.340 --> 32:20.760 How do you normally parse? 32:20.760 --> 32:22.360 Writing a regular expression. 32:22.360 --> 32:24.940 Who enjoys writing a regular expression? 32:25.360 --> 32:27.040 I guess not so many. 32:27.040 --> 32:37.180 Those who say they do, I would always say that this is the Stockholm syndrome, that you got so used to writing regular expressions that you kind of accepted that this is the right way to do stuff. 32:38.160 --> 32:40.900 But I am lazy and you should be lazy too. 32:40.900 --> 32:44.080 So, if you can avoid it, don't write regular expressions. 32:44.080 --> 32:47.140 It's just... maybe it's job security. 32:47.140 --> 32:49.020 So, maybe that's why you would want to do it. 32:49.020 --> 32:54.380 But otherwise, it's just a lot of work and it might go wrong and there are too many edge cases. 32:54.380 --> 33:02.420 So, if you can have something in a structured format, ideally JSON, since Elasticsearch also stores JSON, that would definitely be preferred. 33:02.420 --> 33:04.300 So, that is what I have done here. 33:04.300 --> 33:08.420 I have taken that JSON format. 33:08.880 --> 33:13.200 Who remembers where or as what we have stored it? 33:13.200 --> 33:17.460 How would I see my logs for that? 33:17.720 --> 33:20.000 Let me switch back here. 33:21.280 --> 33:22.440 Let's remove this. 33:22.440 --> 33:23.860 It was application. 33:26.570 --> 33:31.770 Space, colon, and it was mod security. 33:32.110 --> 33:46.140 And if I stop streaming and filter down on this one, it will show me just the logs from mod security. 33:46.600 --> 33:51.240 Which has, even though it's a JSON format, has a pretty terrible format. 33:51.240 --> 33:54.220 So, let's look at one of those here. 33:56.020 --> 33:58.000 So, we're looking at the details. 33:58.000 --> 34:00.980 By the way, the other option would be to see the surrounding. 34:01.640 --> 34:18.180 So, timestamp wise, it could show you a couple of messages before and after, potentially from Nginx or Apache or whatever other logs you collected around that time, which might be kind of related to just to have this overview of this specific timeframe. 34:18.180 --> 34:21.960 That would have been the other option, but we don't need that here. 34:22.520 --> 34:26.280 So, here, these are all the pieces of information we have connected. 34:26.280 --> 34:30.860 So, application mod security, that was the field on which I actually filtered down. 34:31.280 --> 34:41.060 And you can see, this was the pattern which we used to actually figure out that this was this specific attack here. 34:41.060 --> 34:52.620 So, it was detected as a remote code execution Windows command injection, which I'm not sure that was really what happened, but it was detected as that. 34:52.620 --> 35:02.360 And you can see, if you like writing regular expressions, if you're in that Stockholm syndrome, then you will feel right at home here. 35:02.360 --> 35:05.560 So, enjoy the glory of the regular expression here. 35:05.560 --> 35:10.820 And then you can see all the details of this rule and how it is structured. 35:11.640 --> 35:16.540 Yeah, which might not be the most readable format, but this is what is being used here. 35:16.540 --> 35:20.920 And you have various other fields where you can see what has been up here. 35:23.280 --> 35:24.160 Cool. 35:24.300 --> 35:29.900 And again, we added the cloud information, we added the host information, just in case this makes sense. 35:29.900 --> 35:37.940 If you don't use that, don't edit, because it will just add more weight to your logs in Elasticsearch, but it's just easy to collect. 35:37.940 --> 35:45.440 Just to give you an idea how I got those, just add this one here. 35:45.440 --> 35:57.040 I just said add cloud metadata and add host metadata, and those two would just add that to the logs. 35:57.120 --> 36:05.300 So, here is the Ubuntu set, for example, the settings about Ubuntu, and this one has the one about AWS. 36:05.300 --> 36:16.720 And after that here, by the way, I just renamed one field, because otherwise I have a collision of how fields are named, and this is just an easy way to fix up some data when you collect it. 36:17.380 --> 36:20.040 And that's all there is to this. 36:22.320 --> 36:24.720 You can also write custom rules. 36:24.720 --> 36:39.300 So, I have added a custom rule, and this might be very useful, like you have, I don't know, a forum software, and somebody is posting ads for Rolexes or Viagra or whatever other stuff you don't want to have there. 36:39.300 --> 36:42.000 I have created a different rule. 36:42.000 --> 36:53.440 So, what I tried to do here is, on my create.php file, I'm adding a rule that will deny any request that detects a fake SHI. 36:53.440 --> 36:55.540 So, SHI is our CEO. 36:55.540 --> 37:00.980 Let's assume there should be no other SHI in the company, otherwise somebody is trying to add a fake SHI. 37:00.980 --> 37:13.940 So, when a post happens, I check the body of that post and say, like, if there is the string SHI or Bannon in the post, I want to reject that request. 37:13.940 --> 37:19.700 So, let's head over to the browser to see how that was going. 37:19.700 --> 37:25.680 So, this one is not running against Nginx, but let's head over to Apache. 37:25.820 --> 37:28.400 So, again, this is running exactly the same thing. 37:28.520 --> 37:30.840 It's using the same database, the same files. 37:31.460 --> 37:36.420 When I try to log in here, by the way, the SQL injection is not working anymore. 37:36.420 --> 37:53.040 So, if I tried Philip and I tried or true, this would then, for example, reject my request because it detected an SQL injection and would just kick me out. 37:53.120 --> 37:56.740 So, I would have to log in correctly. 37:56.740 --> 37:58.320 So, secret was the password. 37:58.320 --> 38:02.000 If I do that, I can now create a regular user. 38:02.000 --> 38:11.020 So, let's say I want to create SHI other and he is also based in San Francisco and he has some salary and he has a password . 38:11.380 --> 38:21.200 And when I try to send in that other SHI, again, I get a rejection because the fake SHI rule kicked in here. 38:22.760 --> 38:37.920 If I stream the logs for this one, it should, as the very last rule, in a moment show me or already now show me that we just detected this fake SHI and rejected the fake SHI. 38:38.380 --> 38:44.880 So, this is showing that my rule worked and we rejected the fake SHI. 38:44.880 --> 39:04.300 So, even if you have some broken application where you cannot easily add like a spam filter or fix something else, you could just write some custom rules in what security to reject certain things or filter out certain things that you don't think should end up in your application. 39:04.980 --> 39:10.200 And then you have a more or less smiling SHI again because there cannot be any fake SHIs. 39:10.200 --> 39:13.880 And this was the log message that we have just seen, basically. 39:15.120 --> 39:18.260 ModSecurity is not 100% foolproof though. 39:18.260 --> 39:31.140 So, for example, if you would run this one here with these parentheses or one equals one and parentheses, this will still allow you to bypass the login. 39:31.140 --> 39:35.840 So, let me just quickly copy that and try it in the form. 39:35.840 --> 39:39.240 So, here we have a forbidden. 39:39.760 --> 39:48.380 I'll try to... I need to create another random user to actually pass or get logged out. 39:49.160 --> 39:50.520 So, let's do that. 39:50.520 --> 39:53.740 Now, I'm trying to log in again and now I'm... 39:54.800 --> 39:59.540 foobar doesn't exist, but I add that or query. 39:59.880 --> 40:05.780 And this, even though there is ModSecurity here at play, I still bypass that. 40:05.780 --> 40:07.260 So, ModSecurity is not 100%. 40:07.260 --> 40:18.680 There are potentially ways to work around it, but it will filter out some things and you might have to tweak your rules according to what might be valid input or not. 40:19.900 --> 40:25.260 Yes, everything is terrible and everything is on fire and what the fuck were we even thinking? 40:26.840 --> 40:29.920 I guess that's a very good description of security. 40:29.920 --> 40:35.440 So, to wrap this up, security incidents generally come in three levels. 40:35.440 --> 40:39.260 The for your information, what the fuck, and oh my god. 40:40.100 --> 40:46.280 And today I think we were only in the between for your information, what the fuck. 40:46.280 --> 40:54.480 It was definitely not the oh my god yet, though you frequently see those in the media anyway. 40:55.300 --> 41:01.500 Yeah, write better code than me and use the right libraries or frameworks, especially around injections. 41:02.300 --> 41:13.500 Nobody should have SQL injections anymore by using the right tools or libraries, unless you just pass around plain strings like me, which is not a smart idea. 41:14.040 --> 41:17.420 Is anybody using ModSecurity, for example? 41:17.420 --> 41:18.820 Yeah, quite a few people. 41:18.820 --> 41:32.360 So, for example, if you're using GitLab and deploy it in Kubernetes, since almost a year ago on the Ingress app, ModSecurity has been added to GitLab. 41:32.940 --> 41:39.720 So, this is just one of many places where ModSecurity is being used as of today. 41:39.900 --> 41:45.520 And for the Elastic Stack, that's also pretty widely used just for the security use case. 41:45.520 --> 41:52.800 Three companies which I think I can name publicly would be Slack, anything related to security around Slack. 41:52.880 --> 42:01.140 Mozilla has a project and Cisco is also a pretty heavy user just to get logs into the Elastic Stack around security. 42:01.140 --> 42:02.760 And that's not all. 42:02.760 --> 42:06.600 We recently added SIEM and Endpoint Security. 42:06.860 --> 42:08.720 So, those would play into that. 42:08.720 --> 42:16.000 And since I think I have like one minute or so extra, let me give you a quick idea of what SIEM might look like. 42:16.000 --> 42:22.380 So, here I have already, let me zoom out one level again. 42:22.380 --> 42:25.780 So, here these are all the requests in the last hour. 42:25.780 --> 42:27.040 You can see these are all the hosts. 42:27.040 --> 42:29.560 I have a single host here to keep it simple. 42:29.680 --> 42:36.980 All my login requests into that one were successful and I could then dive into all the hosts that I have identified. 42:36.980 --> 42:42.420 You can see it also gives you a good overview of what is running on that machine here. 42:42.980 --> 42:48.340 And for example, I might be interested in not the login attempts because, well, they were all successful. 42:48.340 --> 42:49.640 That's not so interesting. 42:49.640 --> 42:52.020 Let's have a quick look at the events. 42:52.080 --> 42:56.900 In the events, we can see everything that happened on that host here. 42:56.900 --> 43:00.560 So, for example, maybe somebody said like that application is kind of fishy. 43:00.560 --> 43:04.160 And let's have a look at what is happening on that host. 43:04.160 --> 43:06.880 So, that is what we are trying to get here. 43:06.880 --> 43:10.160 And these are all the events that we have collected over time. 43:10.160 --> 43:12.060 So, we have NetFlow. 43:12.060 --> 43:21.420 So, any network connections that have been going on, user logins, processes that have been stopped or started, sessions, etc. 43:21.420 --> 43:31.900 And then at the events further down here, you could see, for example, that this is already our application with Apache. 43:31.900 --> 43:43.380 So, let's drag and drop this one into our timeline filter here because I want to see anything that touches this port 8080. 43:43.480 --> 43:48.280 And then you could see, okay, this was Apache that was serving the request. 43:48.280 --> 43:49.940 You could see where it was coming from. 43:49.940 --> 43:53.000 You could see that many packets, etc. 43:53.040 --> 44:10.360 And then dive further down into that information and figure out why maybe your Apache server is throwing so many errors because ModSecurity is blocking lots of bad stuff, for example, or whatever else you might find interesting in here. 44:10.360 --> 44:13.200 And you can slice and dice your data any way you want. 44:13.800 --> 44:15.980 Okay, enough of this. 44:16.780 --> 44:18.520 Let's close this out. 44:18.560 --> 44:22.540 So, ModSecurity and logging go very well hand in hand. 44:22.540 --> 44:30.480 If you just run ModSecurity without looking what it actually blocks or rightfully or wrongfully blocks, you're pretty much blind. 44:30.480 --> 44:40.960 So, you want to combine this with some proper logging, as well as logging in general in your application to figure out if somebody has potentially breached something or not. 44:40.960 --> 44:52.320 And especially if somebody breached it, when did that happen and what were they actually able to do to your system to actually know afterwards and not just stir some murky water. 44:52.680 --> 45:00.100 If you want to try out all of this, the code is on my GitHub page in ModSecurity.log. 45:00.100 --> 45:03.300 If you want to try out the dashboards, I will also let this run. 45:03.300 --> 45:05.740 I hope nobody kills it. 45:05.860 --> 45:10.060 You will automatically be logged into Kibana with a read-only user. 45:10.440 --> 45:14.980 So, this is running a reverse proxy that automatically logs you in with a read-only user. 45:14.980 --> 45:26.260 So, if you go to dashboard.xiera.wtf, you will end up in Kibana already logged in and you can play around with the logs UI or with the dashboards and just see what is going on there. 45:26.260 --> 45:35.300 The application is running on xiera.wtf, so any requests that you generate, you should be able to find in Kibana as well. 45:35.740 --> 45:39.380 Just if lots of users are doing that, it might overload the system. 45:39.380 --> 45:47.060 No guarantees that it will be up or you might see a lot of noise because of everybody else doing it, but happy exploring around that. 45:47.480 --> 45:54.240 If you want to get the slides, that is the QR code, I will share the link if I can live as well. 45:54.700 --> 45:58.460 And with that, let's head over to the questions. 45:59.180 --> 46:00.500 Find me on this course. 46:00.500 --> 46:04.560 I'm happy to answer whatever you have or ping me on Twitter. 46:04.560 --> 46:06.340 I'm also happy to answer there. 46:06.340 --> 46:07.580 Thanks a lot for joining.