[00:00.000 --> 00:04.660]  Welcome to the talk on detecting lateral movement in zero-touch environments. We're going to dive
[00:04.660 --> 00:12.840]  into how that lateral movement works, how to detect it, and why you really want a zero-touch
[00:12.840 --> 00:20.760]  environment. So before we jump in too much, some disclaimers and acknowledgments. First off,
[00:20.760 --> 00:25.960]  I work for the MITRE Corporation, but I am not representing or speaking for them today.
[00:27.080 --> 00:32.440]  All of the research done in this talk was performed in the pursuit of a master's degree
[00:32.440 --> 00:37.940]  through SANS, and so thank you to them for supporting the research, and thank you to my
[00:37.940 --> 00:44.700]  advisor, but I'm also not speaking for them today. And lastly, thank you to my wife Madeline,
[00:44.700 --> 00:53.520]  whom I'm also not speaking for today. So apart from all of that, who am I?
[00:54.680 --> 01:03.240]  My happy place professionally is in the middle of security and DevOps. And that has quite naturally
[01:03.240 --> 01:10.940]  led me to doing a lot of work with the cloud, automating not just applications and operations,
[01:10.940 --> 01:16.860]  but the security portions of that. A little bit of background on me. I wrote my first
[01:16.860 --> 01:22.460]  vulnerable code in elementary school. I was writing a blogging platform for my family,
[01:22.460 --> 01:28.740]  and they were quite surprised when it was hacked through a SQL injection that I inadvertently
[01:28.740 --> 01:35.440]  added. I began learning to write exploit code in middle school, when I received as a present
[01:35.440 --> 01:41.540]  the book Hacking the Art of Exploitation. And I've been involved in security one way or another
[01:41.540 --> 01:47.940]  ever since. This is my first time speaking at DEF CON, and I am super excited that the
[01:47.940 --> 01:55.060]  Cloud Village has me here. And one last little kind of personal philosophy thing is I really
[01:55.060 --> 02:00.780]  enjoy learning through hacking. I imagine a lot of you do, too. And you'll definitely see that
[02:00.780 --> 02:06.160]  flavor as we go through the presentation today of looking at things from both an attacker's
[02:06.160 --> 02:13.920]  point of view and a defender's point of view. So why should I care about DevOps if you're
[02:13.920 --> 02:19.600]  not already sold? Well, if you're running any applications or any services,
[02:20.600 --> 02:26.420]  you've got to deal with it. That's the way it is now. Welcome to the world. However,
[02:27.640 --> 02:32.080]  that's true regardless of whether you're hosting applications for the outside world,
[02:32.080 --> 02:38.540]  just for yourself, whoever it is. If you're a red team and you need to spin up infrastructure to
[02:38.540 --> 02:46.140]  support a penetration test or something like that, DevOps to the rescue. Blue team, you're
[02:46.140 --> 02:52.040]  trying to keep continuous monitoring and alerting of what's going on in your environment, DevOps.
[02:52.860 --> 03:00.300]  Application owner, trying to make sure your application is stable and available and secure,
[03:00.300 --> 03:07.480]  DevOps. Another great perk for us is that DevOps is essentially cloud native now. All of the major
[03:07.480 --> 03:16.340]  cloud providers have offered ways to automate their environments. And so being able to develop
[03:16.340 --> 03:23.640]  infrastructure as code, applications as code, security compliance as code, monitoring as code,
[03:23.640 --> 03:28.240]  all of these are really enabled by the cloud. They can be done in other environments,
[03:28.240 --> 03:35.280]  but the cloud makes it easy. And lastly, DevOps is better for security. We're integrating security
[03:35.280 --> 03:39.760]  earlier in the process, rather than trying to layer it on later.
[03:42.380 --> 03:47.720]  So I promised an attacker's point of view. Today, we're going to look at kind of a simplified
[03:47.720 --> 03:52.220]  example environment. I'm sure, you know, you've seen a bunch of other presentations that have got
[03:52.220 --> 03:58.500]  the hard to read, squint and see, that's AWS, right? Not Azure. We're simplifying things for
[03:58.500 --> 04:04.780]  this talk. You're somewhere out on the internet, and you really want to get to an application server.
[04:05.620 --> 04:12.260]  Might want to get in and then exfil some of the sensitive data on it, you might want to get in
[04:12.260 --> 04:17.940]  and use their computing resources to mine cryptocurrency. Whatever the reason is,
[04:17.940 --> 04:24.620]  you're going for that application server. So there's three main options we're going to look at.
[04:24.720 --> 04:30.160]  First is straight through the front door, directly in from the internet hitting the application
[04:30.160 --> 04:37.480]  server. Pretty simple. Yeah, there's maybe a firewall or an IDS or something that you have
[04:37.480 --> 04:45.640]  to evade. But overall, pretty direct, pretty simple. Except it's not 2003 anymore, and we've
[04:45.640 --> 04:52.080]  gotten better at protecting our applications and the servers they run on. So while it's still
[04:52.080 --> 04:59.280]  possible, and there are plenty of big, well-known events where it's happened, it's not as popular or
[05:00.500 --> 05:07.400]  as reliable as it used to be. So that brings us to our second option, and that's, let's compromise
[05:07.400 --> 05:13.440]  the workstation of an employee, and then move laterally to the application server we really
[05:13.440 --> 05:20.020]  want to be on. For the purposes of this presentation, I'm going to say we phished that
[05:20.020 --> 05:28.240]  employee, but it could be any number of things. From there, most organizations have a bastion host
[05:28.240 --> 05:35.140]  that employees have to log into before they can get into their production environment. That's a
[05:35.140 --> 05:41.080]  pretty common practice, and so we see that on the screen. Our second option is compromise an employee
[05:41.080 --> 05:49.120]  workstation, use valid credentials that that employee has access to, move through the bastion
[05:49.120 --> 05:57.040]  host to the application server. And then our third option is to take the route that DevOps uses to
[05:57.040 --> 06:03.480]  publish applications itself. From that employee workstation, push something into the source code
[06:03.480 --> 06:10.300]  repo, get it through the test servers undetected, have the configuration server deploy it for us.
[06:11.300 --> 06:18.020]  Or maybe we can compromise each of those servers in turn in order to get there in the end.
[06:18.600 --> 06:24.200]  In any case, that's a lot more hops to have to go through and a lot more chances to be detected
[06:24.200 --> 06:32.040]  or fail in some other way. So we're going to focus on detecting number two, both because it's pretty
[06:32.040 --> 06:39.920]  common and because that's the easiest route that attackers have, and so that's most likely what
[06:39.920 --> 06:45.680]  they're going to go for. And this being common is borne out by repeated studies. We've seen
[06:45.680 --> 06:52.180]  reports from FireEye and the Verizon data breach report say that social engineering and then using
[06:52.180 --> 07:01.670]  valid credentials is incredibly popular for malicious actors right now. So what does it look
[07:01.670 --> 07:09.070]  to do a traditional application deployment from that employee's point of view? Developers give
[07:09.070 --> 07:15.690]  ops a deployment package and hopefully some install instructions, and then ops logs into
[07:15.690 --> 07:22.070]  the app server through the bastion host and manually installs the software. Pretty simple.
[07:22.790 --> 07:26.750]  And then you get a patch. Well, you've got to log in and install the patch.
[07:26.930 --> 07:30.650]  There's an update to one of the dependencies. We've got to log in and install that.
[07:31.050 --> 07:35.810]  Anytime anything changes, someone's logging into that application server.
[07:39.850 --> 07:45.510]  So that's, you know, pretty normal, but not ideal, and we'll get into that.
[07:47.410 --> 07:53.530]  Traditional lateral movement takes advantage of that fact. Because you've got people logging in
[07:53.530 --> 08:03.250]  to the server manually, they have highly privileged credentials. And very often,
[08:03.250 --> 08:08.570]  those credentials are stored in plain text on their employee workstations. Exactly what we
[08:08.570 --> 08:14.750]  were talking about compromising earlier. That might take the form of SSH keys,
[08:14.750 --> 08:21.810]  it might take the form of API tokens or keys. And, you know, especially relevant for the cloud
[08:21.810 --> 08:29.350]  village, that's the way AWS stores local credentials. Plain text file in that path on the
[08:29.350 --> 08:37.510]  screen. Attackers then use those plain text credentials from the station they've already
[08:38.030 --> 08:44.710]  compromised to move further into your environment and do whatever evil they want to get up to.
[08:44.750 --> 08:53.150]  They've hit their target. So what is ZeroTouch and why does it apply to any of this?
[08:53.570 --> 09:00.630]  So Google are the ones who have defined ZeroTouch networking and ZeroTouch production concepts.
[09:00.630 --> 09:06.770]  But they're far from the only people who use them. This is a technique used by many mature
[09:06.770 --> 09:12.430]  DevOps organizations to automate as much as possible and get humans out of their
[09:12.430 --> 09:21.130]  production environments. What does that look like? Here's the three rules that Google use
[09:21.130 --> 09:28.530]  to define their ZeroTouch production. And the first one is really big. Every change
[09:28.530 --> 09:37.560]  is made by automation. No humans are involved. By getting that human traffic out of the environment,
[09:38.620 --> 09:46.340]  we can do a lot of really interesting but simple things to detect when attackers are trying to
[09:46.340 --> 09:54.980]  pivot into our protected environment. The second rule, pre-validated by software, is important
[09:54.980 --> 10:00.460]  because if we try to go to that attack path number three, the one that uses the DevOps pipeline
[10:00.460 --> 10:05.480]  itself, that means we're checking to see if there's anything malicious before we deploy.
[10:05.480 --> 10:12.720]  Now, that doesn't always work. But it gives us as defenders a better opportunity than someone
[10:12.720 --> 10:18.190]  who's just pivoting through the bastion host and hiding in the noise. And then lastly,
[10:19.440 --> 10:27.260]  emergency changes can be made via a break glass mechanism. But they have to be auditable.
[10:27.340 --> 10:35.960]  We'll come back to that. And what this looks like in our picture is essentially disallowing
[10:35.960 --> 10:43.000]  communication from that bastion host to the application server. And all normal traffic is
[10:43.000 --> 10:48.100]  going through that path number three where it goes through the source repo and the test servers
[10:48.100 --> 10:53.860]  and the configuration servers. So we've got good control over what's being deployed to the
[10:53.860 --> 11:04.320]  application server and good confidence that it's not malicious. All right. So now let's look at
[11:04.320 --> 11:10.300]  lateral movement in each of these two types of environments. We've compromised our employee
[11:10.300 --> 11:17.120]  workstation, found some valid credentials that we can use to pivot. And in this case, we're going
[11:17.120 --> 11:23.460]  to look at the traditional path where operators are regularly logging into the environment to
[11:23.460 --> 11:29.900]  deploy patches, update software, any of those kinds of things. I'm about to play an animation.
[11:30.280 --> 11:34.300]  And there are going to be several connections shown on the screen by arrows.
[11:34.320 --> 11:39.300]  Try and find the one that looks a little bit different than all the rest.
[11:39.640 --> 11:48.640]  This is going to be our malicious lateral movement. Did you see it?
[11:50.020 --> 11:57.880]  Have you seen it yet? It's a little bit tricky to find. And this is assuming that it stands out at
[11:57.880 --> 12:04.200]  all. If you're using an encrypted communication channel like SSH or HTTPS, which are super
[12:04.200 --> 12:14.620]  common, this use of valid credentials might not stand out in any way. Now let's compare that to
[12:15.320 --> 12:21.560]  someone trying to use lateral movement on a zero touch network. Same thing. I'm going to display
[12:21.560 --> 12:26.500]  several arrows in an animation and see if you can pick out the one that looks a little bit different
[12:26.500 --> 12:35.410]  and indicates our malicious lateral movement. I'll bet that was a lot easier. And you maybe even
[12:35.410 --> 12:42.050]  spotted it before the animation finished playing. In this case, it stands out like a sore thumb
[12:42.050 --> 12:48.750]  because those valid credentials, even if the channel is encrypted, are going an entirely
[12:48.750 --> 12:58.690]  different route through the network than normal, regular, valid traffic would. So how do we go
[12:58.690 --> 13:05.790]  about detecting that from a programmatic and automatic standpoint? First, we need to define
[13:05.790 --> 13:12.130]  the protected servers. In the case of our example, that's just the application server. But it could
[13:12.130 --> 13:16.770]  be a lot of things. Could be application servers, web servers, the DevOps servers themselves.
[13:17.090 --> 13:23.990]  Anything that humans shouldn't be logging into on a regular basis. Then we need to define all
[13:23.990 --> 13:29.810]  of the human access points into the environment. Again, in this example, that's just the bastion
[13:29.810 --> 13:34.710]  host. But depending on your network architecture, there might be any number of things that you need
[13:34.710 --> 13:41.930]  to put into that list. And then watch for any connections made from those manual access points
[13:41.930 --> 13:49.450]  to your protected servers. Real simple, but very effective if it's not supposed to be happening.
[13:49.450 --> 13:56.190]  At that point, you can alert, investigate, whatever your normal process is, and enjoy having detected
[13:56.190 --> 14:03.790]  malicious lateral movement. Now, I mentioned earlier that auditing was important. And the
[14:03.790 --> 14:09.690]  reason that this is still a valid attack path into the environment is most organizations can't
[14:09.690 --> 14:15.470]  get rid of their bastion hosts entirely. They're still needed for incident response or forensics,
[14:15.470 --> 14:22.410]  some kind of unusual emergency purpose. And so, even though that's valid traffic, this method
[14:22.410 --> 14:29.770]  would still alert on it. However, that's not a problem, because anytime that emergency traffic
[14:29.770 --> 14:36.670]  does occur, we wanted it to be auditable anyway. And this is helpfully generating those audit alerts
[14:36.670 --> 14:44.690]  for us. So, now we're going to cross our fingers and see if the live demo goes well.
[14:50.750 --> 14:58.630]  What I've got up on the screen are three lab machines. One, a bastion host,
[14:58.630 --> 15:03.590]  one, our application server, and one, a puppet configuration server.
[15:05.210 --> 15:08.870]  What I'm going to do, just to prove to you all that this is actually working,
[15:08.870 --> 15:14.970]  and that I've got nothing up my sleeves today, is we're going to put some valid traffic into
[15:14.970 --> 15:22.130]  the environment. What this puppet command will do is generate traffic between the puppet configuration
[15:22.130 --> 15:28.990]  server and the application server, as well as traffic out to the wider internet over both SSH
[15:28.990 --> 15:33.630]  and HTTPS. And we'll keep that going throughout the rest of the demonstration.
[15:36.150 --> 15:52.010]  Then, we'll go over to the bastion host and try to execute our... let me fix that real quick.
[15:52.010 --> 16:01.070]  Then we'll go over to the bastion host and... nope. My routes are wrong today,
[16:01.070 --> 16:04.550]  so I'm going to play the prerecorded version of the demo instead.
[16:12.580 --> 16:22.550]  All right, but you'll get all the same information here. So, for the demo, we're going to look at a
[16:22.550 --> 16:30.950]  couple of scenarios. First up, let me tell you what you're seeing. In this environment, I've got
[16:30.950 --> 16:37.390]  a bastion host that's used for manually accessing it when needed, an application server, and a
[16:37.390 --> 16:42.910]  configuration server running puppet. So, just to make sure that there's nothing up my sleeves,
[16:43.490 --> 16:49.190]  what we're going to do for this demo is run some puppet configuration in the background.
[16:50.250 --> 16:55.990]  This will be traffic both within the environment to the puppet server, and external to the
[16:55.990 --> 17:02.310]  environment SSH connections and HTTPS connections out to the wider internet. So, let's get that
[17:02.310 --> 17:12.050]  started. And then, while this runs, jump over to the bastion host, and as you can see, we've already
[17:12.050 --> 17:18.990]  compromised the bastion host. And we see some SSH credentials that we found, and in fact, some SSH
[17:18.990 --> 17:28.050]  credentials for the application server that has the goodies we want to get to. So, we'll run our evil
[17:28.770 --> 17:37.810]  script file. And, you know, in real life, this might be installing a crypto miner or any number of things.
[17:37.810 --> 17:45.270]  In this case, just modifying the configuration files that Puppet is managing to show that we've
[17:45.270 --> 17:53.230]  actually gotten in and had the ability to do evil if we wanted to. So, now we're going to jump over to
[17:54.270 --> 18:05.810]  the network monitor. This is a plain instance of Security Onion with all of the usual defaults and
[18:05.810 --> 18:16.130]  alerts installed. And what we're looking at right now are the ZIP notices. And let me turn on the
[18:16.130 --> 18:26.870]  refresh. But you'll see a whole lot of nothing. Zero. No results found. No results found. If we
[18:26.870 --> 18:43.950]  switch over to Squirt and refresh it, same thing. No result. No result. And instance of a network
[18:43.950 --> 18:54.470]  monitor. It's not going to alert on connections like this. Let me enable the rules we've been
[18:54.470 --> 19:07.480]  talking about. And then we'll come back and we'll try this again. And now that I've enabled the
[19:07.480 --> 19:13.540]  zero-touch rules, let's try it again and see what's changed. As you can see, the lab server
[19:13.540 --> 19:19.000]  is still emulating real traffic by reaching out to the Puppet configuration server and to the
[19:19.000 --> 19:26.880]  external internet. Let's go over to the Bastion and run our evil script again. Same thing, utilizing
[19:26.880 --> 19:35.960]  those compromised SSH credentials to reach in and modify the app server. Now we'll jump over to
[19:36.540 --> 19:43.300]  the network monitor. And as you can see, we're starting to get alerts. Zeke has noticed
[19:44.700 --> 19:52.700]  a couple of connections from the Bastion host to the app server. And if we switch over,
[19:52.700 --> 20:02.800]  you'll see the same thing in Squirt as well. Notice how they work slightly differently.
[20:02.800 --> 20:10.440]  The Squirt gives us many more alerts because it's looking individually at the packets traversing
[20:10.440 --> 20:16.240]  the network versus Zeke gives us total number of connections. But either way, we're seeing
[20:16.240 --> 20:22.440]  the alerts that we're violating this zero-touch policy and that it's a good indicator of lateral
[20:22.440 --> 20:40.650]  movement. All right, so you can see from the demo, you know, this isn't something that adds to your
[20:40.650 --> 20:46.340]  ability to detect lateral movement as long as you conform to the expectations of the environment.
[20:47.080 --> 20:51.260]  Demonstrated here with Zeke and Snort, but there's no reason you can't use
[20:51.940 --> 20:58.640]  any of your tools of choice. But this is the cloud village. So what about cloud?
[20:59.780 --> 21:05.730]  Totally still doable in the cloud, despite the fact I use VMware for my lab environment.
[21:07.120 --> 21:13.520]  If you're not familiar, all the major cloud providers provide some form of traffic mirroring,
[21:13.520 --> 21:18.720]  whatever they call that, traffic mirroring in AWS, virtual network tap in Azure,
[21:19.280 --> 21:25.640]  and packet mirroring for Google Cloud Platform. And that'll give you full network visibility.
[21:25.640 --> 21:29.540]  You can install something just like Security Onion or your favorite network monitor
[21:31.040 --> 21:38.540]  feeding directly off of that. You can also install your own within the environment. And
[21:38.540 --> 21:45.240]  because this is looking for such simple things as connections, we don't even need the full
[21:45.240 --> 21:51.940]  traffic in order to tell when this activity is occurring. We can just use flow logs. And so
[21:51.940 --> 21:58.800]  that'll cut down on the amount of traffic that we have to capture if we go that route. Now,
[21:58.800 --> 22:04.120]  given that the cloud is a much more dynamic environment than traditional data centers,
[22:04.120 --> 22:08.420]  this means that tagging an asset inventory is really important to get right.
[22:09.240 --> 22:15.020]  You have to know what those protected servers are and what the human access points are in order to
[22:15.020 --> 22:21.080]  detect connections between them. And if you've got automatic scaling or something like that,
[22:21.080 --> 22:25.700]  keeping those lists up to date can be a little bit tricky, but it's definitely doable,
[22:26.520 --> 22:34.100]  especially since those lists in most tools can be entire subnets rather than specific machines.
[22:35.440 --> 22:41.820]  But there are also some challenges with the cloud. And the big one is visibility into
[22:41.820 --> 22:49.960]  management traffic. With something like Puppet or Ansible, your traditional configuration servers,
[22:49.960 --> 22:54.300]  that's all going over a network that you can view. However, in the cloud,
[22:54.300 --> 22:59.180]  some of your management traffic goes over the management plane in the background.
[22:59.180 --> 23:06.800]  And these features like traffic mirroring are not able to capture that data for you.
[23:06.800 --> 23:12.680]  It all happens in AWS or in Azure without you being able to see it.
[23:12.720 --> 23:18.020]  And that poses a visibility challenge. Doesn't mean that it's not worth doing,
[23:18.020 --> 23:24.360]  but you have to understand how your environment usually works in order to be able to protect it.
[23:27.150 --> 23:32.870]  So, some next steps if you're sold on this idea. First off, if you're not Zerotouch yet,
[23:32.870 --> 23:40.870]  do it. It's got a lot of benefits aside from this particular detection. It's hard to do,
[23:40.870 --> 23:45.610]  and it's hard to get right, but it is worth it. And then you can implement this detection on
[23:45.610 --> 23:53.290]  your platform of choice. I showed off Snort and Zeek here, but it's very simple. There's no need
[23:53.290 --> 23:57.310]  and buy specialized software, buy specialized hardware,
[23:57.310 --> 24:01.290]  install anything you're probably not already using anyway.
[24:02.610 --> 24:09.330]  And then when you do implement this detection, tailor it to your specific environment. Here I
[24:09.330 --> 24:15.250]  was looking for any connections between the human access points and our protected servers. But
[24:16.670 --> 24:21.610]  your environment may or may not be able to support that depending on its specific architecture.
[24:22.150 --> 24:28.570]  The good news is that this approach is valid for a lot of different options, and you just need to
[24:28.570 --> 24:34.170]  figure out which ones work specifically for you. You might only turn it on for SSH connections.
[24:34.170 --> 24:39.350]  You might only look at particular areas of your network that are high risk.
[24:39.410 --> 24:43.270]  However you do it, make sure you tailor it, because that'll give you the most value.
[24:44.390 --> 24:51.590]  And then there are ways to extend this work far beyond what I demonstrated in today's video.
[24:52.140 --> 24:58.480]  For example, correlate it with other events that are generated off of suspicious traffic.
[24:58.830 --> 25:05.530]  Imagine tying this alert about potential lateral movement with potential data exfiltration that's
[25:05.530 --> 25:13.070]  generated by Zeke, and then presenting your security team with a map of how the attacker
[25:13.720 --> 25:18.970]  moved through your entire network. It's pretty powerful once you start combining it with other
[25:18.970 --> 25:28.430]  things. Lastly, I want to leave you with some lessons learned from doing this research project.
[25:28.430 --> 25:33.270]  First off, you have to know your network. That's the basis of this entire project,
[25:33.270 --> 25:39.250]  is you have to know your network well enough to know what's normal and what's abnormal.
[25:39.370 --> 25:46.170]  And then putting in things to look for abnormalities specific to your environment
[25:46.170 --> 25:50.190]  as opposed to things that apply to everyone's environment.
[25:50.830 --> 25:55.890]  I learned this principle first from Mick Douglas and Justin Henderson, who teach the
[25:55.890 --> 26:03.930]  sans-tactical-seam class. And it really resonated, it was really powerful in being able to do things
[26:03.930 --> 26:09.710]  that don't come pre-built with your software, because you're looking for things specific to you.
[26:10.230 --> 26:15.350]  And then second, don't be afraid to look for stupid simple things on your network.
[26:16.170 --> 26:20.670]  All we're looking for in this is the presence of a connection.
[26:21.970 --> 26:28.490]  But knowing our environment the way we do, that lets us say there's something weird going on,
[26:28.490 --> 26:32.650]  I need to investigate. And that is super valuable.
[26:34.830 --> 26:41.370]  So, thank you. If you have any questions after the talk and the Q&A is done,
[26:41.370 --> 26:45.910]  best place to reach me is on Twitter. My handle's on the screen right now.
[26:46.170 --> 26:50.530]  And then we'll see if we've got any questions live.
[26:50.530 --> 26:56.250]  Do you happen to know any tools that might help implement this kind of a
[26:56.250 --> 27:02.930]  zero-touch environment or getting to know your network better or something along those lines?
[27:02.930 --> 27:07.750]  Sure. So, getting to know your network better, you're just going to have to get familiar with
[27:07.750 --> 27:13.490]  how these operate. And my biggest recommendation is if you're starting to do DevOps as a security
[27:13.490 --> 27:21.290]  person, is make time to spend with the operations and the developers. Because they're going to know
[27:21.290 --> 27:26.550]  how things normally go. And you'll be able to learn from them what they expect. And then use
[27:26.550 --> 27:33.930]  your knowledge of security to figure out what shouldn't be going on. In terms of tools to help
[27:33.930 --> 27:41.290]  implement this, you know, typical DevOps tools, there's a periodic table of DevOps that's out
[27:41.290 --> 27:48.430]  there, I think by Xenia Labs, that shows a lot of the different options. For the demo today, I was
[27:48.430 --> 27:55.190]  using Puppet as a configuration server to manage the configuration of the application server and
[27:55.190 --> 28:02.210]  making sure that it had the files and configuration settings that I expected. There are a lot of them,
[28:02.730 --> 28:09.070]  apart from Puppet, Ansible, SaltStack, things of that nature. And then for building out a
[28:09.070 --> 28:14.270]  whole DevOps pipeline, look at that periodic table. You'll need several different things
[28:14.270 --> 28:19.730]  along there to handle your source code and your testing and things like that.