[00:13.580 --> 00:18.660]  Hey, welcome back. Hey, AppSecVillage, we're getting down to the last few speakers of our
[00:18.660 --> 00:25.160]  time together here on day three of AppSecVillage at DEF CON 2020. I want to thank you for coming
[00:25.160 --> 00:29.700]  and for making this possible. I want to make sure to thank our sponsors and the folks that
[00:29.700 --> 00:34.780]  run AppSecVillage. They've done a wonderful job both years that we've been here. This
[00:34.780 --> 00:39.300]  has become my home away from home when I come to DEF CON. I look forward to it every year
[00:39.300 --> 00:44.460]  and I hope you do too. And I hope to see you again next year when we do this, hopefully in
[00:44.460 --> 00:49.520]  person, but we're going to do it one way or the other. Our next discussion is going to be Secure
[00:49.520 --> 00:55.080]  Your Code, Injections and Logging by Philip Kren. Philip lives to demo interesting technology.
[00:55.200 --> 00:59.360]  Having worked as a web infrastructure and database engineer for more than 10 years,
[00:59.360 --> 01:04.840]  Philip is now working as a developer advocate at Elastic. Based in Vienna, Austria, he is
[01:04.840 --> 01:10.500]  consistently traveling Europe and beyond to speak and discuss on topics such as open source
[01:10.500 --> 01:15.850]  software, search technology, databases, infrastructure, and security in general.
[01:16.160 --> 01:19.560]  Please welcome Philip to the AppSecVillage stage.
[01:20.620 --> 01:27.780]  Hi, I'm Philip. Let's dive into the next session. Let me share my screen.
[01:28.760 --> 01:34.720]  And there we go. Secure Your Code, Injections and Logging. Let's see what we can do here.
[01:34.840 --> 01:41.500]  So when we talk about security, we normally keep telling ourselves that everything is fine,
[01:41.500 --> 01:46.400]  especially when we say everybody's job is security. It's normally nobody's job,
[01:46.400 --> 01:51.300]  and we can all pretend that this is all good. And then you look at the OWASP top 10,
[01:51.300 --> 01:58.420]  and you see at number one is still injections. So good old SQL injection, maybe LLAP injection,
[01:58.420 --> 02:04.740]  NoSQL injection, or whatever other form of unsecured user input you are throwing at your
[02:04.740 --> 02:10.080]  application, and that is wreaking havoc. And then you might reconsider and say like, well,
[02:10.080 --> 02:16.660]  maybe this is not so fine anymore. And then you look further down the OWASP top 10 list,
[02:16.660 --> 02:21.340]  and then you see, for example, number 10, insufficient logging and monitoring,
[02:21.340 --> 02:26.420]  where you then realize that on average, it takes something like 200 days or so until you normally
[02:26.420 --> 02:32.460]  find a breach, maybe a bit quicker if the attacker asked for some ransomware or ransom money,
[02:32.460 --> 02:39.560]  but it might take a while. And then you both or you might end up at the point where you say,
[02:39.560 --> 02:44.400]  well, this is no longer fine or good, and everything is terrible and on fire.
[02:44.400 --> 02:48.820]  And we don't want to get to this point. So what can we do about that?
[02:49.440 --> 02:54.540]  I'm Philipp. I work for Elastic, the company behind Elasticsearch, Kibana, Beats, Logstash.
[02:54.540 --> 02:59.280]  You might have heard of us or might be using us already somewhere. My official title is
[02:59.280 --> 03:04.400]  developer advocate. So I mostly talk about the good stuff that we do. And today is no exception.
[03:04.400 --> 03:10.540]  So let's see what we can do here. I generally build highly monitored Hello World applications.
[03:11.000 --> 03:15.700]  Today is no exception. And we will pretty much dive into that right away.
[03:15.700 --> 03:21.880]  So like in any good cooking show, I have prepared something and a very bad application.
[03:21.880 --> 03:28.920]  So xiera.wtf is where this is running. And I will let this run. So while this is pre-recorded,
[03:28.920 --> 03:34.420]  I will let the demo run. So you can actually destroy it or play around with it live afterwards
[03:34.420 --> 03:41.320]  as well. If you're wondering where Xiera is coming from, it has something to do with my name.
[03:41.320 --> 03:48.340]  If you're bored, try to figure it out. And obviously, WTF is the right domain for any demo.
[03:48.340 --> 03:57.400]  So let's hope this works out. And let's head over to my very bad and well, average demo application.
[03:57.400 --> 04:05.440]  So we have some Elastic employees here. And I added our CEO Shai, one of our co-founders Simon,
[04:05.440 --> 04:13.140]  and myself. And well, maybe we want to add more employees. And to do that, obviously,
[04:13.140 --> 04:21.960]  you need to log in. If you want to regularly log in, my username is Philip, and my password is
[04:21.960 --> 04:33.700]  secret. This works. And then we can say, other edition is, I don't know, maybe in New York.
[04:33.720 --> 04:42.000]  And they have some salary and they also have a password of let's say secret. Okay, so that worked.
[04:42.000 --> 04:50.420]  But looking at this form, this is not secure. And by design, this is using the Hello World
[04:50.420 --> 04:57.640]  application of SQL injections. So we can just use one parameter like Philip and in the other feed,
[04:57.640 --> 05:07.120]  we will just do an or true. So I will do an or true and try to log in like that.
[05:07.800 --> 05:16.420]  This worked, unfortunately, and I can now add a bad user in, I don't know, from Gotham,
[05:16.420 --> 05:20.860]  and have a much higher salary. And they also have some random password.
[05:21.400 --> 05:26.480]  Okay, so we have added the bad user, nothing super surprising here.
[05:28.020 --> 05:34.100]  This has the classic SQL injection problem, I don't escape my user input, I just pass in
[05:34.100 --> 05:39.160]  whatever I'm doing here. I'm hashing my password, but I'm just breaking out of that and doing
[05:39.160 --> 05:45.680]  or true, for example, or whatever other variation you want to have of this.
[05:46.200 --> 05:52.040]  You've seen this plenty of times. It's called an SQL injection, or some people call it accidental
[05:52.040 --> 05:57.700]  GraphQL, if that is more your thing, and which might be the deeper thing. But classically,
[05:57.700 --> 06:05.780]  this is an SQL injection. And I'm pretty sure all or most of you have seen little bobby tables.
[06:06.180 --> 06:13.800]  If you give your child the right name, like drop table students, you will have a lot of fun
[06:13.800 --> 06:19.820]  at school, or your school just should write better systems. But that's a different discussion.
[06:19.820 --> 06:26.060]  So let's have another look around. So I can also display users. So for example,
[06:26.060 --> 06:32.560]  looking at our CEO shy here, this looks all nice and good. And you can go back and forth.
[06:32.640 --> 06:39.180]  What I'm doing here is I'm just passing the ID of my user directly in. So I could just change
[06:39.180 --> 06:47.900]  that to two and pass over to Simon. This is insecure, again, like unsurprisingly, since
[06:47.900 --> 06:54.600]  this is a broken application. And before we try to actually throw some bad parameters as it,
[06:54.600 --> 07:02.020]  I want to quickly jump to SQL map. If you've never seen or used SQL map, it's an automatic
[07:02.020 --> 07:07.840]  SQL injection and database takeover tool. And you can basically say like, hey, there is this
[07:07.840 --> 07:14.460]  parameter like the ID and I would like to check that and it will go through a lot of variations
[07:14.460 --> 07:21.860]  of SQL injections to actually say like, well, this looks like it might be interesting or not.
[07:21.860 --> 07:30.920]  Okay, so let's open my shell and let me paste in my command so you don't have to see me type this.
[07:30.920 --> 07:36.620]  So I'm running SQL map against that URL that I've shown you. And at the end, we add a purge
[07:36.620 --> 07:41.840]  because I've run this before. And purge will just make sure we don't use any records that
[07:41.840 --> 07:49.960]  we've created before. So we will start fresh. So we will start running this and SQL map will just
[07:50.640 --> 07:57.140]  figure out what is possible here. It will ask me a couple of questions of how it should treat this.
[07:58.180 --> 08:02.680]  So you can see it's already trying to do stuff and it figured out that I'm using
[08:02.680 --> 08:09.820]  MySQL here. So yes, this is the right database. And I want to skip tests for other databases
[08:09.820 --> 08:18.100]  because this is MySQL, we can focus on this. And I'm okay to run these risk level queries here.
[08:19.300 --> 08:22.720]  So you can see it's trying out various things.
[08:24.780 --> 08:33.260]  And at the end, it will then tell me what is actually vulnerable. It will also ask me
[08:33.260 --> 08:39.220]  if it should run or check any other parameters, which we will not do here because there's only
[08:39.220 --> 08:44.920]  one parameter that is interesting. I don't want to test anything else, so we're done here.
[08:44.920 --> 08:51.100]  So looking at the result, you can see, for example, okay, this is running against NGINX
[08:51.100 --> 08:58.580]  and against the more or less recent version of MySQL. We would have some logs in this folder
[08:58.580 --> 09:02.780]  here, though they're not super interesting. They're mostly what we have here and all the SQL
[09:02.780 --> 09:10.580]  commands that have been running. So we will stick to what we see here. So you can see in 81 HTTP
[09:10.580 --> 09:17.420]  requests, we figured out a couple of things. So for example, we see a Boolean-based blind attack
[09:17.420 --> 09:23.500]  where you can do something like ID 1 and something equals something, or you can have a
[09:23.500 --> 09:29.720]  time-based attack where we add a sleep and we see indeed the query takes that amount of sleep plus
[09:29.720 --> 09:36.000]  a little more maybe, or that you can have a union query and that you can try to join data together.
[09:36.000 --> 09:44.140]  So this is clearly vulnerable to something. So let's have a look of what I have actually in here.
[09:45.540 --> 09:51.160]  Again, a very bad query. I'm just taking that parameter. I'm not escaping it in any way. I'm
[09:51.160 --> 09:59.400]  just trimming any spaces off of it. And then I take this query and I run it. The good thing here,
[09:59.400 --> 10:04.700]  by the way, is that I actually logged the query that people are sending to my system. So this
[10:04.700 --> 10:10.100]  will help us afterwards to figure out what has actually been happening here. So we will log this
[10:10.100 --> 10:21.180]  into a var log app log file. The bad thing here is that I'm using MySQL multi-query. So this will
[10:21.180 --> 10:28.360]  allow me to run multiple queries in one go. So besides that select, I might do another insert,
[10:28.360 --> 10:35.920]  for example. So if I add this to the end of the URL and it will automatically then be URL escaped
[10:35.920 --> 10:46.620]  and work properly, I could just say insert into employees some bad actor. So if I run that,
[10:46.620 --> 10:53.060]  let's see in our application. Right now we have a bad user, but we don't have a bad actor.
[10:53.060 --> 11:00.820]  So if I have to that parameter here, I just add after semicolon the insert statement,
[11:01.200 --> 11:08.480]  run this, it looks like nothing happened. But when I go back here, you now have a bad actor.
[11:08.840 --> 11:16.840]  So this works as expected or not expected. Maybe this is mostly up to your expectation. By the way,
[11:16.840 --> 11:22.680]  this doesn't escape like the visualization in any way or the output. So you could run any
[11:22.680 --> 11:28.580]  random JavaScript here, which I didn't do because it will be kind of annoying to have various pop-ups
[11:29.320 --> 11:35.280]  pop up. You could break the entire site with some weird JavaScript. So there is no other
[11:35.280 --> 11:44.820]  protection here. So we don't do any escaping. So this is easy to pick apart. So what's going on in
[11:44.820 --> 11:50.340]  our application? Now is the interesting part, like at first before I want to secure it, I want to get
[11:50.340 --> 11:57.040]  a bit of a better idea of what even has been going on in my application. And yeah, if you don't have
[11:57.040 --> 12:02.640]  any log files, nobody can remove your log files. So let's assume we want to have log files and we
[12:02.640 --> 12:08.160]  also want to ship them off and have them hopefully in a secure location. So even if somebody breaks
[12:08.160 --> 12:14.700]  into that instance and manipulates it, they cannot actually do anything with your logs and you still
[12:14.700 --> 12:21.820]  keep some trace of what has been happening. So we want to collect all the things and having a log
[12:21.820 --> 12:26.800]  file is a very good starting point. Maybe ship it off in case somebody breaks into the instance.
[12:27.120 --> 12:33.960]  The main downside is these log files have the ugly tendency to get very large. So maybe at some
[12:33.960 --> 12:38.880]  point you end up as this little submarine, basically, and then you have gigabytes and
[12:38.880 --> 12:44.940]  gigabytes of log files in which you try to find something. Maybe successfully, maybe not so
[12:44.940 --> 12:52.880]  successfully. Scrap is great until you have multiple servers and various big log files,
[12:52.880 --> 12:57.540]  because then it's not going to be so much fun. So you probably want to use something else that
[12:57.540 --> 13:03.920]  is making your life a bit easier. One of the most wily tools there probably is the Elk Stack. You
[13:03.920 --> 13:10.120]  can see Elasticsearch, Logstash, and Kibana, one sitting on top of the other, and that makes up the
[13:10.120 --> 13:16.660]  good old Elk. So Elasticsearch is the thing that stores your data, Logstash is the thing that gets,
[13:16.660 --> 13:23.620]  parses, and enriches your data, and Kibana can visualize it. By the way, when I say parse,
[13:23.620 --> 13:28.300]  you normally have some logline, and out of that logline you want to extract some piece of
[13:28.300 --> 13:33.500]  information. So, for example, you have a log level, or you have the URL where something came
[13:33.500 --> 13:38.620]  from, or maybe you have some user information, or a timestamp. All of these you might want to
[13:38.620 --> 13:43.700]  extract to actually be able to filter down and then say, oh, give me all the logs from this
[13:43.700 --> 13:50.240]  specific time frame, or for this specific user, or for this specific application, or URL within
[13:50.240 --> 13:57.520]  the application, or any way you want to slice and dice your data. When I say enrich, it might be
[13:57.520 --> 14:03.300]  something like you have the IP address of the person requesting the website, and then you could
[14:03.300 --> 14:09.180]  add the GeoIP information to that IP address, and then you could draw it out on a map, and then you
[14:09.180 --> 14:13.960]  could draw where are all my visitors coming from, for example. Or you could just filter down and
[14:13.960 --> 14:20.720]  say like, oh, give me all the users coming from Russia or China, because that's kind of unusual.
[14:20.720 --> 14:25.900]  I normally don't have any visitors from there. Let's figure out what they have been up to.
[14:26.780 --> 14:32.860]  That works. The only thing is that by now the stack has evolved slightly, so we still have
[14:32.860 --> 14:37.260]  Kibana for the visualization Elasticsearch that stores it. Sometimes, by the way, people are
[14:37.260 --> 14:41.720]  slightly confused and ask like, what is the database behind Elasticsearch? There is no
[14:41.720 --> 14:47.820]  other database anymore. There is the library Apache Lucene, which writes it down to disk,
[14:47.820 --> 14:53.700]  but otherwise there is no other data store behind that anymore. Logstash is still that
[14:53.700 --> 15:00.260]  versatile, bit more like ETL tool, so extract, transform, load, so it can get data from various
[15:00.260 --> 15:06.080]  systems, change and transform that, then push that out to Elasticsearch or other systems.
[15:06.440 --> 15:14.120]  But to make life a bit easier and also slimmer, we have added bits in various versions so far.
[15:14.120 --> 15:20.740]  So, for example, for log files, we have Filebit, and it basically tails a file over the network
[15:21.300 --> 15:27.140]  and a bit on steroids. So, we basically will say like, this is a log file that I'm interested in,
[15:27.140 --> 15:32.780]  take that, tail it, store it into Elasticsearch. To keep my demo simple, I will actually skip
[15:32.780 --> 15:37.960]  Logstash and Filebit will tail my log file, put the data into Elasticsearch, and then we can
[15:37.960 --> 15:49.780]  visualize it in Kibana directly. So, let's see what we have actually here. So, maybe before we dive
[15:49.780 --> 15:56.800]  into the visualization, let's take a very quick look at what we have done here. So, this is the
[15:56.800 --> 16:03.320]  configuration of this entire demo, and this is all being set up automatically, so you can get all the
[16:03.320 --> 16:09.880]  configs afterwards. What I have done here is, there are maybe two interesting things in this
[16:09.880 --> 16:15.680]  file. So, this is the Filebit configuration HTML. What will be interesting at first is,
[16:15.680 --> 16:21.680]  so, I'm getting some log files from the system, and there are so-called modules. These are four
[16:21.680 --> 16:27.400]  well-known things that you have often installed. So, for example, I have Nginx running here,
[16:27.400 --> 16:34.720]  as we have seen in SQLMAP before, and it will know automatically for the server that I'm running,
[16:34.720 --> 16:40.100]  where is that log file located by default, and what does it look like. So, basically, by saying,
[16:40.100 --> 16:46.800]  use this Nginx module, it will then pick up the right file, be able to parse it in its individual
[16:46.800 --> 16:53.360]  pieces, and then store it, and we're done. We don't need to care about any log file locations
[16:53.360 --> 16:59.680]  or anything anymore. With that var log app log, which my application is writing, it's not so
[16:59.680 --> 17:05.100]  standardized. So, for this one, I have this configuration here, where I basically tell
[17:05.100 --> 17:12.420]  FileGrid, this is the file that you need to collect. So, it is of the type log, this is the path,
[17:12.420 --> 17:18.000]  you could also use a wildcard and point to a directory, for example, and then what is also
[17:18.000 --> 17:23.840]  interesting, I will store this with an additional field called application app, and this is what we
[17:23.840 --> 17:28.760]  will be able to filter on right away, because I'm collecting all these other logs, so we might
[17:28.760 --> 17:37.320]  see a lot of logs coming together in Kibana right away. So, let's take a quick look. So,
[17:37.320 --> 17:41.680]  maybe you have used Discover before, which was the kind of like old way to do that.
[17:41.680 --> 17:49.800]  We have now a dedicated logs UI, where you can filter down and live stream data. So,
[17:49.800 --> 17:56.400]  with the live streaming, this is a bit more like tail F, where you just see all the log messages
[17:56.400 --> 18:01.900]  that come in right away. So, whatever happened here to my SQL, we might investigate that later
[18:01.900 --> 18:06.960]  on, but it will just live tail anything that is happening here. But what I'm interested in right
[18:06.960 --> 18:16.900]  now is application, as I have said that before, and let me change my screen zoom in slightly,
[18:16.900 --> 18:27.300]  and my application's name was app. So, I just want to filter down only on the application that
[18:27.300 --> 18:33.860]  I'm running here. I don't want to see anything about Apache, or Nginx, or anything else that
[18:33.860 --> 18:41.800]  I have running. Here we go. So, this is the log file of my log that I have collected. So,
[18:41.800 --> 18:49.200]  you can see here from the login page, this is where I was logging in successfully with Philip
[18:49.200 --> 18:59.420]  and Sigrid. I can see that here I created, for example, my other realistician that you saw.
[18:59.420 --> 19:08.420]  Then here you can see the SQL injection as I was running it already. And then you can see here,
[19:08.420 --> 19:16.360]  these are the tests that SQL map has been running. And you can see as or whatever has happened here.
[19:16.360 --> 19:22.800]  You can also live stream these here. So, you would just see any SQL queries that come in here.
[19:22.800 --> 19:29.020]  So, for example, this one here at the very end, this was the last query that I ran. When I ran or
[19:29.020 --> 19:36.720]  added to the parameter of the ID one, I added the insert statement and added that bad actor
[19:36.720 --> 19:42.040]  value to my data store. So, this is what happened here in the very last statement.
[19:42.360 --> 19:48.400]  So, this is all pretty easy to see what my application has been doing. As long as you
[19:48.400 --> 19:53.160]  write out the right log statements that make sense, you can then easily collect them and
[19:53.160 --> 19:58.240]  see what is happening here. And you can also correlate that, for example, with the nginx logs,
[19:58.240 --> 20:06.300]  for example, afterwards. I hope that makes sense so far. Oh, by the way, you could also search in
[20:06.300 --> 20:12.340]  this one here with highlighting. You could then just say, let's quickly do that. In highlighting,
[20:12.340 --> 20:18.560]  I want to search for insert. And I'm only interested in the insert statements right now.
[20:18.560 --> 20:26.260]  So, we have one insert here. So, highlighting, another point done. Let's move on.
[20:26.960 --> 20:33.600]  If you try to delete any record or drop the entire table, this will not work, by the way,
[20:33.600 --> 20:38.780]  because if I was running this live, it would be very annoying if suddenly all the data was
[20:38.780 --> 20:44.560]  gone and I couldn't demo anything anymore. So, the user that is running the SQL queries doesn't
[20:44.560 --> 20:50.900]  have the permissions to delete or drop anything. Feel free to try it. You can see it in the logs,
[20:50.900 --> 20:58.340]  but your requests will fail because those commands are not allowed here. Now, we know that something
[20:58.340 --> 21:04.840]  is happening, but we also want to protect against it. And that is where mod security is coming into
[21:04.840 --> 21:11.180]  play here. So, what is mod security if you have never used it? It's an open source tool. It's
[21:11.460 --> 21:19.420]  a web application firewall. It's basically intercepting your traffic and either monitoring
[21:19.420 --> 21:27.020]  stuff or even blocking or fixing things as it goes along. And it is based on rules where, for example,
[21:27.020 --> 21:36.220]  this is an SQL injection, so I will deny that request, for example. It has two sets of rules.
[21:36.220 --> 21:41.520]  There is a free core rule set that you can use that I'm using in my demo as well here.
[21:41.520 --> 21:47.740]  So, for example, it can have a real-time blacklist lookup to deny certain things.
[21:47.740 --> 21:55.480]  It can try to figure out an HTTP denial of service attack. It can detect SQL injections and other
[21:55.480 --> 22:03.460]  things. There is also a commercial version which has a totally separate rule set which has some
[22:03.460 --> 22:09.760]  more additional or maybe more advanced features. Though I'm not using that here, I'm only sticking
[22:09.760 --> 22:18.700]  to the free version of the core rule set. So, to make this simple, I have the same vulnerable
[22:18.700 --> 22:25.920]  application running behind Apache as well, only that Apache is running on port 8080. So, the code
[22:25.920 --> 22:32.620]  is exactly the same. It's even referencing the same files and everything. It's just Apache has
[22:32.620 --> 22:40.060]  more security added as well. So, if I run my SQL map against that...
[22:42.340 --> 22:52.840]  So, let's run the SQL map command against Apache. So, I'm adding this against port 8080 now. Let's run this.
[22:54.840 --> 22:58.460]  This won't be as successful as before.
[22:59.640 --> 23:04.220]  So, you can see it is running. It hasn't figured out the database yet.
[23:05.180 --> 23:11.300]  And it will not because it cannot really see any of the typical MySQL things.
[23:12.020 --> 23:21.010]  Do I want to reduce the number of requests? No, I don't want to reduce the number of requests.
[23:21.010 --> 23:26.510]  It should run all the requests here. So, please try everything you can.
[23:31.030 --> 23:38.170]  And it's actually telling me that 126 times it ran into a 403, so into a forbidden.
[23:38.170 --> 23:42.990]  It actually says that potentially this is running behind a web application firewall.
[23:42.990 --> 23:50.050]  So, we could, with the dash dash tamper command, we could run various ways to tamper the requests
[23:50.050 --> 23:58.410]  so that maybe they are not being detected by mod security or a web application firewall in general.
[23:58.410 --> 24:06.470]  So, with sqlmap dash dash list tamper, it will show you all the different ways or all the different
[24:07.110 --> 24:12.770]  scripts that are included to avoid tampering. There are some which are specific to mod security,
[24:12.770 --> 24:16.790]  for example, but they don't make any difference for our example here.
[24:16.790 --> 24:23.610]  So, I will keep this simple and I will just add the tampering. So, I will use a random agent.
[24:24.150 --> 24:28.650]  So, if somebody tried to block on a specific user agent, we would avoid that.
[24:28.650 --> 24:36.430]  And then we add spaces to the comments to maybe avoid detection, which won't be successful here.
[24:36.890 --> 24:46.070]  But, well, yeah. Now, it did detect that this is protected by something.
[24:46.070 --> 24:50.930]  I still want to continue my investigation.
[24:52.370 --> 25:01.630]  But again, it won't be successful and it won't circumvent mod security in the current default
[25:01.630 --> 25:05.470]  implementation that I have. I don't want to reduce the number of requests again.
[25:05.470 --> 25:18.400]  I just want to run this plainly. Okay. And we have the same result, 118 requests were forbidden.
[25:18.400 --> 25:26.120]  So, we have just 403 results here. Well, that is what you get.
[25:27.860 --> 25:37.860]  Which leads us, by the way, so we could do two things here to actually look at. So, in the logs,
[25:38.580 --> 25:44.400]  let's trash this filter here. And in the logs, just let me make this slightly smaller so I can
[25:44.400 --> 25:56.920]  see what I'm typing here. So, we could, for example, say Apache error module and I want any.
[25:57.240 --> 26:05.460]  So, and now you can see these are all Apache messages that we have been collected. So,
[26:06.340 --> 26:12.420]  here now I'm in the live streaming mode. You can see these were rejected by mod
[26:12.420 --> 26:18.440]  security already. We can, by the way, look at the details of one request here.
[26:19.300 --> 26:27.380]  So, you can see that this is running. Filebit has collected it. I have this enriched this,
[26:27.380 --> 26:31.900]  by the way, with the cloud information. So, you can see that this is running in an AWS data center.
[26:31.900 --> 26:37.760]  You can see the AMI ID on which this is based, the instance ID. So, you could correlate this,
[26:37.760 --> 26:43.940]  for example, to a specific instance ID. Or I have also added the host information,
[26:43.940 --> 26:49.060]  which is like the operating system. So, for example, if an attack is only successful against
[26:49.300 --> 26:54.660]  a specific version of Ubuntu, for example, you could easily filter down on this as well. So,
[26:54.660 --> 27:01.200]  here you could just say, like, give me everything that is Ubuntu 18.04.4, because I know only this
[27:01.200 --> 27:05.500]  one is vulnerable to some specific attack. And then you could just see the logs only for this
[27:05.500 --> 27:11.020]  one, which is not the case here, but just as an additional idea of what kind of information you
[27:11.020 --> 27:17.220]  might want to add. And then you can see this was an error. And you can see here the error came from
[27:17.220 --> 27:24.940]  mod security, actually. And since I have live streaming on, let me run my tests once more.
[27:24.940 --> 27:30.680]  And we should see... let me close this one here. We should see,
[27:31.700 --> 27:37.000]  from this timestamp here, we should see stuff move on a bit.
[27:38.720 --> 27:45.400]  I mean, it will take a little while. So, I run my attack. Apache is generating the log file. File
[27:45.400 --> 27:51.660]  bit is picking up the logs. And then it will be stored in Elasticsearch. And after that,
[27:51.660 --> 27:58.240]  you can then see it in Kibana. And you can see here now it is appearing in Kibana that this was
[27:58.240 --> 28:06.360]  the attack that has just been happened. So, you can see the anomaly score is 65. And it was because
[28:06.360 --> 28:13.620]  of these individual pieces that mod security figured out. So, we have an SQL injection,
[28:13.620 --> 28:21.540]  cross-site scripting, remote filing inclusion, local file inclusion, remote code execution.
[28:21.540 --> 28:28.860]  So, all of these factor into a total score. And then, based on the threshold, an attack might be...
[28:28.860 --> 28:35.120]  or a request might be blocked or not. And this one here has been because it definitely breached
[28:35.120 --> 28:44.940]  the rules here. So, the other thing that I actually wanted to show you is in Kibana, you can also have
[28:44.940 --> 28:49.280]  as a dashboard, you can see a bit of an overview of what is happening in your system. So, for example,
[28:49.280 --> 28:54.580]  you could see here, these were the average or the general requests that have been running. And let me
[28:54.580 --> 29:01.620]  refresh this view because I think this data is slightly outdated now. So, this one here just
[29:01.620 --> 29:07.340]  shows you the number of requests you can see here. These were the requests that I have just been
[29:07.340 --> 29:12.300]  running with the SQL map. So, that's why they are peaking here a bit. And the different shades of
[29:12.300 --> 29:18.180]  green are not ideal. Let me change this to more this pinkish. And I don't know, this I'll change
[29:18.180 --> 29:24.480]  to blue so we can keep them a bit better apart. But you can see, for example, these here are the
[29:24.480 --> 29:30.640]  403s. And we ran into a lot of 403s with more security now. Or you could see where the requests
[29:30.640 --> 29:37.180]  are coming from. So, this here would be me. We surprisingly have some requests from the U.S. as
[29:37.180 --> 29:46.540]  well. Nothing from Russia or China today, which is almost disappointing. But depending on the day,
[29:46.540 --> 29:51.300]  you might have more requests from there. And then you could, for example, see which URLs were hit the
[29:51.300 --> 29:58.060]  most and what response codes you got, what clients or browsers people used, how many errors you have.
[29:58.060 --> 30:05.100]  And here you can see these are, for example, the errors that are the Apache log that have been
[30:05.100 --> 30:10.400]  generated by more security. And you can see all of those. By the way, you could filter just on the
[30:10.400 --> 30:15.800]  notices here or just on the errors if you have too much noise or just on one specific browser
[30:15.800 --> 30:21.960]  or user agent, just to drill down into that data. Since this is all based on a search engine,
[30:21.960 --> 30:25.880]  this is kind of the value of having a search engine. You can easily drill down and filter
[30:25.880 --> 30:31.860]  into all of that data. So, this is what you can see in the dashboard here. But let's take another
[30:31.860 --> 30:39.540]  look at ModSecurity. So, in ModSecurity, you have the setting SecRuleEngine and you have it either
[30:39.540 --> 30:45.920]  off detection only or on. So, I have set it to on and that's what blocked the request, especially
[30:45.920 --> 30:51.380]  when you want to roll this out to a production site. You probably want to start with detection
[30:51.380 --> 30:57.840]  only to see what would happen if I would run this or if this would be active. And then you might
[30:57.840 --> 31:03.780]  find interesting things like in San Francisco, there's Union Square. And I think ModSecurity
[31:03.780 --> 31:10.640]  had a bug at some point where it saw Union anywhere in the post request, for example, and assumed
[31:10.640 --> 31:16.380]  that this was an SQL Union attack, even though it was a regular address that was just
[31:16.380 --> 31:22.460]  Union Square. It would block that. And I think the rules got slightly smarter now, so that this is
[31:22.460 --> 31:28.000]  not having this false positive. But it's just one thing that you might run into and you might create
[31:28.140 --> 31:33.140]  a bad user experience by being a bit too over-ambitious with your security settings. So,
[31:33.140 --> 31:38.600]  that's why it makes sense to actually start with detection only and then review what is
[31:38.600 --> 31:47.460]  happening. You can also limit how big file uploads are, for example, here or how big a post request
[31:47.460 --> 31:53.860]  can be. So, these here would just limit how big requests are so that nobody can overload your
[31:53.860 --> 31:59.080]  server with unnecessarily large requests. And obviously, it would make sense to change those
[31:59.080 --> 32:06.860]  to some reasonable value for your setup. In terms of logging, what would be very helpful
[32:06.860 --> 32:14.280]  is if you turned the logging format to JSON, because the non-JSON format is pretty terrible
[32:14.280 --> 32:20.040]  to parse and nobody wants to do that. By the way, how would you parse that? How do you normally
[32:20.040 --> 32:27.040]  parse? Writing a regular expression. Who enjoys writing a regular expression? I guess not so many.
[32:27.040 --> 32:31.640]  Those who say they do, I would always say that this is the Stockholm syndrome, that you got
[32:31.640 --> 32:36.400]  so used to writing regular expressions that you kind of accepted that this is the right way to do
[32:36.400 --> 32:43.360]  stuff. But I am lazy and you should be lazy too. So, if you can avoid it, don't write regular
[32:43.360 --> 32:49.180]  expressions. It's just... maybe it's job security. So, maybe that's why you would want to do it. But
[32:49.180 --> 32:54.480]  otherwise, it's just a lot of work and it might go wrong and there are too many edge cases. So,
[32:54.480 --> 32:58.940]  if you can have something in a structured format, ideally JSON, since Elasticsearch also stores
[32:58.940 --> 33:05.000]  JSON, that would definitely be preferred. So, that is what I have done here. I have
[33:05.760 --> 33:14.680]  taken that JSON format. Who remembers where or as what we have stored it? How would I see my
[33:15.360 --> 33:23.860]  logs for that? Let me switch back here. Let's remove this. It was application.
[33:26.570 --> 33:42.400]  Space, colon, and it was mod security. And if I stop streaming and filter down on this one,
[33:42.400 --> 33:49.720]  it will show me just the logs from mod security. Which has, even though it's a JSON format, has a
[33:49.720 --> 33:58.060]  pretty terrible format. So, let's look at one of those here. So, we're looking at the details. By
[33:58.060 --> 34:03.620]  the way, the other option would be to see the surrounding. So, timestamp wise, it could show
[34:03.620 --> 34:09.400]  you a couple of messages before and after, potentially from Nginx or Apache or whatever
[34:09.400 --> 34:15.700]  other logs you collected around that time, which might be kind of related to just to have this
[34:15.700 --> 34:21.240]  overview of this specific timeframe. That would have been the other option, but we don't need that
[34:21.240 --> 34:26.800]  here. So, here, these are all the pieces of information we have connected. So, application
[34:26.800 --> 34:33.080]  mod security, that was the field on which I actually filtered down. And you can see, this was
[34:33.080 --> 34:41.740]  the pattern which we used to actually figure out that this was this specific attack here. So,
[34:41.740 --> 34:49.660]  it was detected as a remote code execution Windows command injection, which I'm not sure that was
[34:49.660 --> 34:55.440]  really what happened, but it was detected as that. And you can see, if you like writing regular
[34:55.440 --> 35:02.700]  expressions, if you're in that Stockholm syndrome, then you will feel right at home here. So,
[35:02.700 --> 35:08.720]  enjoy the glory of the regular expression here. And then you can see all the details of this rule
[35:08.720 --> 35:15.440]  and how it is structured. Yeah, which might not be the most readable format, but this is what is
[35:15.440 --> 35:20.920]  being used here. And you have various other fields where you can see what has been up here.
[35:23.280 --> 35:28.920]  Cool. And again, we added the cloud information, we added the host information, just in case this
[35:28.920 --> 35:34.920]  makes sense. If you don't use that, don't edit, because it will just add more weight to your logs
[35:34.920 --> 35:41.880]  in Elasticsearch, but it's just easy to collect. Just to give you an idea how I got those,
[35:41.880 --> 35:52.100]  just add this one here. I just said add cloud metadata and add host metadata, and those two
[35:52.100 --> 36:02.860]  would just add that to the logs. So, here is the Ubuntu set, for example, the settings about Ubuntu,
[36:02.860 --> 36:08.180]  and this one has the one about AWS. And after that here, by the way, I just renamed one field,
[36:08.180 --> 36:13.720]  because otherwise I have a collision of how fields are named, and this is just an easy way to
[36:13.720 --> 36:20.040]  fix up some data when you collect it. And that's all there is to this.
[36:22.320 --> 36:26.640]  You can also write custom rules. So, I have added a custom rule,
[36:26.640 --> 36:31.660]  and this might be very useful, like you have, I don't know, a forum software, and somebody is
[36:31.660 --> 36:39.300]  posting ads for Rolexes or Viagra or whatever other stuff you don't want to have there.
[36:39.300 --> 36:46.460]  I have created a different rule. So, what I tried to do here is, on my create.php file,
[36:46.460 --> 36:55.540]  I'm adding a rule that will deny any request that detects a fake SHI. So, SHI is our CEO.
[36:55.540 --> 36:59.980]  Let's assume there should be no other SHI in the company, otherwise somebody is trying to add a
[36:59.980 --> 37:07.760]  fake SHI. So, when a post happens, I check the body of that post and say, like, if there is the
[37:07.760 --> 37:17.040]  string SHI or Bannon in the post, I want to reject that request. So, let's head over to the browser
[37:17.040 --> 37:24.920]  to see how that was going. So, this one is not running against Nginx, but let's head over to
[37:24.920 --> 37:29.840]  Apache. So, again, this is running exactly the same thing. It's using the same database, the same
[37:29.840 --> 37:36.800]  files. When I try to log in here, by the way, the SQL injection is not working anymore. So, if I
[37:36.800 --> 37:48.880]  tried Philip and I tried or true, this would then, for example, reject my request because it detected
[37:48.880 --> 37:56.960]  an SQL injection and would just kick me out. So, I would have to log in correctly. So,
[37:56.960 --> 38:03.400]  secret was the password. If I do that, I can now create a regular user. So, let's say I want to
[38:03.400 --> 38:11.020]  create SHI other and he is also based in San Francisco and he has some salary and he has a
[38:11.020 --> 38:19.280]  password. And when I try to send in that other SHI, again, I get a rejection because the fake
[38:19.280 --> 38:30.420]  SHI rule kicked in here. If I stream the logs for this one, it should, as the very last rule,
[38:30.420 --> 38:37.400]  in a moment show me or already now show me that we just detected this fake SHI and rejected the
[38:37.400 --> 38:46.460]  fake SHI. So, this is showing that my rule worked and we rejected the fake SHI. So, even if you have
[38:46.460 --> 38:53.060]  some broken application where you cannot easily add like a spam filter or fix something else,
[38:53.060 --> 38:59.800]  you could just write some custom rules in what security to reject certain things or filter out
[38:59.800 --> 39:06.580]  certain things that you don't think should end up in your application. And then you have a more or
[39:06.580 --> 39:12.120]  less smiling SHI again because there cannot be any fake SHIs. And this was the log message that we
[39:12.120 --> 39:19.960]  have just seen, basically. ModSecurity is not 100% foolproof though. So, for example, if you would
[39:19.960 --> 39:28.640]  run this one here with these parentheses or one equals one and parentheses, this will still allow
[39:28.640 --> 39:35.840]  you to bypass the login. So, let me just quickly copy that and try it in the form.
[39:35.840 --> 39:45.840]  So, here we have a forbidden. I'll try to... I need to create another random user to actually pass
[39:45.840 --> 39:53.740]  or get logged out. So, let's do that. Now, I'm trying to log in again and now I'm...
[39:54.800 --> 40:02.520]  foobar doesn't exist, but I add that or query. And this, even though there is ModSecurity here
[40:02.520 --> 40:09.660]  at play, I still bypass that. So, ModSecurity is not 100%. There are potentially ways to work
[40:09.660 --> 40:15.400]  around it, but it will filter out some things and you might have to tweak your rules according to
[40:15.400 --> 40:23.500]  what might be valid input or not. Yes, everything is terrible and everything is on fire and what
[40:23.500 --> 40:30.600]  the fuck were we even thinking? I guess that's a very good description of security. So, to wrap
[40:30.600 --> 40:37.080]  this up, security incidents generally come in three levels. The for your information, what the
[40:37.080 --> 40:45.680]  fuck, and oh my god. And today I think we were only in the between for your information, what the
[40:45.680 --> 40:53.440]  fuck. It was definitely not the oh my god yet, though you frequently see those in the media
[40:53.440 --> 41:00.160]  anyway. Yeah, write better code than me and use the right libraries or frameworks, especially
[41:00.160 --> 41:06.760]  around injections. Nobody should have SQL injections anymore by using the right tools
[41:06.760 --> 41:13.500]  or libraries, unless you just pass around plain strings like me, which is not a smart idea.
[41:14.040 --> 41:19.880]  Is anybody using ModSecurity, for example? Yeah, quite a few people. So, for example, if you're
[41:19.880 --> 41:28.220]  using GitLab and deploy it in Kubernetes, since almost a year ago on the Ingress app,
[41:28.960 --> 41:36.180]  ModSecurity has been added to GitLab. So, this is just one of many places where ModSecurity
[41:36.180 --> 41:43.940]  is being used as of today. And for the Elastic Stack, that's also pretty widely used just for
[41:43.940 --> 41:50.320]  the security use case. Three companies which I think I can name publicly would be Slack,
[41:50.320 --> 41:55.720]  anything related to security around Slack. Mozilla has a project and Cisco is also a pretty heavy
[41:55.720 --> 42:03.400]  user just to get logs into the Elastic Stack around security. And that's not all. We recently
[42:03.400 --> 42:10.180]  added SIEM and Endpoint Security. So, those would play into that. And since I think I have like one
[42:10.180 --> 42:17.520]  minute or so extra, let me give you a quick idea of what SIEM might look like. So, here I have
[42:17.520 --> 42:25.860]  already, let me zoom out one level again. So, here these are all the requests in the last hour. You
[42:25.860 --> 42:31.020]  can see these are all the hosts. I have a single host here to keep it simple. All my login requests
[42:31.020 --> 42:36.980]  into that one were successful and I could then dive into all the hosts that I have identified.
[42:36.980 --> 42:43.260]  You can see it also gives you a good overview of what is running on that machine here. And for
[42:43.260 --> 42:48.340]  example, I might be interested in not the login attempts because, well, they were all successful.
[42:48.340 --> 42:53.740]  That's not so interesting. Let's have a quick look at the events. In the events, we can see everything
[42:53.740 --> 42:59.780]  that happened on that host here. So, for example, maybe somebody said like that application is kind
[42:59.780 --> 43:05.320]  of fishy. And let's have a look at what is happening on that host. So, that is what we are
[43:05.320 --> 43:10.800]  trying to get here. And these are all the events that we have collected over time. So, we have
[43:10.800 --> 43:17.500]  NetFlow. So, any network connections that have been going on, user logins, processes that have
[43:17.500 --> 43:25.360]  been stopped or started, sessions, etc. And then at the events further down here, you could see,
[43:25.360 --> 43:34.760]  for example, that this is already our application with Apache. So, let's drag and drop this one
[43:34.760 --> 43:43.380]  into our timeline filter here because I want to see anything that touches this port 8080.
[43:43.480 --> 43:48.660]  And then you could see, okay, this was Apache that was serving the request. You could see
[43:48.660 --> 43:55.220]  where it was coming from. You could see that many packets, etc. And then dive further down
[43:55.220 --> 44:02.800]  into that information and figure out why maybe your Apache server is throwing so many errors
[44:02.800 --> 44:08.660]  because ModSecurity is blocking lots of bad stuff, for example, or whatever else you might
[44:08.660 --> 44:15.120]  find interesting in here. And you can slice and dice your data any way you want. Okay, enough of
[44:15.120 --> 44:23.460]  this. Let's close this out. So, ModSecurity and logging go very well hand in hand. If you just
[44:23.460 --> 44:29.260]  run ModSecurity without looking what it actually blocks or rightfully or wrongfully blocks,
[44:29.260 --> 44:33.820]  you're pretty much blind. So, you want to combine this with some proper logging,
[44:33.820 --> 44:39.740]  as well as logging in general in your application to figure out if somebody has potentially breached
[44:39.740 --> 44:45.240]  something or not. And especially if somebody breached it, when did that happen and what
[44:45.240 --> 44:50.060]  were they actually able to do to your system to actually know afterwards and not just
[44:50.060 --> 44:58.660]  stir some murky water. If you want to try out all of this, the code is on my GitHub page in
[44:58.660 --> 45:03.580]  ModSecurity.log. If you want to try out the dashboards, I will also let this run. I hope
[45:03.580 --> 45:10.060]  nobody kills it. You will automatically be logged into Kibana with a read-only user.
[45:10.440 --> 45:14.260]  So, this is running a reverse proxy that automatically logs you in with a read-only
[45:14.260 --> 45:20.620]  user. So, if you go to dashboard.xiera.wtf, you will end up in Kibana already logged in and you
[45:20.620 --> 45:26.260]  can play around with the logs UI or with the dashboards and just see what is going on there.
[45:26.260 --> 45:32.340]  The application is running on xiera.wtf, so any requests that you generate, you should be able to
[45:32.340 --> 45:39.380]  find in Kibana as well. Just if lots of users are doing that, it might overload the system.
[45:39.380 --> 45:44.420]  No guarantees that it will be up or you might see a lot of noise because of everybody else doing it,
[45:44.420 --> 45:50.520]  but happy exploring around that. If you want to get the slides, that is the QR code,
[45:50.520 --> 45:58.460]  I will share the link if I can live as well. And with that, let's head over to the questions.
[45:59.180 --> 46:02.480]  Find me on this course. I'm happy to answer whatever you have or
[46:02.480 --> 46:07.580]  ping me on Twitter. I'm also happy to answer there. Thanks a lot for joining.