WEBVTT 00:00.000 --> 00:01.520 Hello, everybody. 00:01.520 --> 00:04.540 It's great to be here at Cloud Village. 00:04.640 --> 00:10.740 And in the next 30 minutes or so, we're going to talk about static analysis of infrastructure as code. 00:11.300 --> 00:27.860 Just as a side note, if you're interested in contributing to any of those open source projects around AWS security, IAM security, infrastructure as code, or Google Cloud, feel free to reach out. 00:27.860 --> 00:32.220 It's Barak Shoster at Twitter, or just follow my activity at GitHub. 00:32.220 --> 00:43.280 I really like to work on open source projects, and it is one of the things that is great about BridgeCode that is contributing more and more open source security tooling back to the community. 00:44.860 --> 00:45.900 All right. 00:45.900 --> 00:52.880 So on the agenda today, we're going to talk about the state of open source Terraform modules. 00:52.880 --> 00:55.460 Are they secured or not? 00:55.620 --> 01:03.220 We'll go over how to run Chekhov using two different integrations, pre-commit hook and CI deployment. 01:04.540 --> 01:12.640 And at the end of this talk, you should be able to just download Chekhov and run it on your own manifests. 01:12.800 --> 01:18.160 Before diving into deployment, let's talk about the problem space. 01:18.160 --> 01:29.800 As engineers, like myself, and probably you who are listening, our goal is to move faster and in much agile way. 01:31.340 --> 01:40.380 And when we develop a new product, we need some way to deploy the infrastructure that this product runs on. 01:40.760 --> 02:03.640 Infrastructure as code came to the world to help us just do all this manual work at the cloud providers like the AWS, Azure, GCP consoles of setting up a VM automatically and having this job of setting those instances reproducible, version controlled, 02:04.160 --> 02:06.000 and peer reviewed. 02:06.000 --> 02:13.340 Since for the first time, it's in code, you can peer review it as a pull request or merge request in your VCS. 02:14.100 --> 02:21.060 When you are working fast, you're producing probably more code, and with more code comes more bugs. 02:21.560 --> 02:37.640 So in a lot of times, when you want to just provision a new cloud resource, let's take S3, for example, you might miss some of the default configurations and keep the S3 unencrypted by default or without versioning or without access logs. 02:37.640 --> 02:41.540 And the same goes for other resources like compute resources. 02:41.640 --> 02:48.200 You might forget access keys in your plain text environment variables instead of using a secured store. 02:48.740 --> 02:59.460 And when you're copying pre-made templates from the internet that are unvetted, those issues might repeat more and more. 02:59.960 --> 03:07.640 So Gartner said, and this is often quoted, that 95 of the cloud security issues are related to configurations errors. 03:08.600 --> 03:25.530 And the trend that is being done at the past two or three years, I know that Terraform exists for a little more than seven years, is that we're codifying each and every activity that we're doing in YAML files, HCL files, etc. 03:28.570 --> 03:40.150 These tooling, those manifests that we write, the infrastructure as code, are about provisioning new resources in a reproducible way and about managing them. 03:40.150 --> 03:45.950 But it lacks the security functionality or best practices vetting functionality. 03:45.950 --> 03:48.530 And there are some great tools around this area. 03:48.530 --> 03:57.210 There is OPA, the open policy agent of the CNCF, and also Chekhov, TFSEC, CFNAG. 03:57.210 --> 04:00.910 Each of those is tackling the problem from a different angle. 04:01.590 --> 04:09.630 So let's talk about the problem that we're trying to solve using Chekhov, which is open source and under Apache 2 license. 04:09.710 --> 04:25.850 What we try to understand is the default configuration of any kind of infrastructure as code manifest, meaning Kubernetes, Terraform, CloudFormation, serverless framework, etc. 04:25.850 --> 04:29.410 is secured by default. 04:29.610 --> 04:33.010 So we asked ourselves that question. 04:33.010 --> 04:41.870 Let's scan as first open source modules of those cloud manifests. 04:42.010 --> 04:48.990 So we decided to use Chekhov to scan thousands of open source repositories. 04:49.770 --> 04:58.430 Before diving into the results, there are some interesting stats about the trend of contributing and creating new infrastructure modules. 04:58.430 --> 05:02.110 And this is focused only on Terraform modules. 05:02.510 --> 05:15.450 So back in 2017, 18, and 19, we had a mild growth in the amount of new open source modules contributed around Terraform. 05:15.670 --> 05:29.590 And specifically at the COVID era, around February, probably due to work from home productivity benefits, the contributions of new modules went sky high, which is great. 05:29.590 --> 05:35.810 We'll have open source, more code that someone else is writing, is less code for me to write. 05:36.070 --> 05:45.570 But the thing is, the more code that is being contributed, the more misconfigs that are being found in the open source repositories. 05:45.570 --> 05:52.170 So 48% of those modules have some kind of misconfiguration. 05:52.170 --> 06:01.570 So if we take about 2,500 modules, we'll see a lot of misconfigs there. 06:01.690 --> 06:08.610 And those open source modules, by GitHub statistics, had more than 26 million downloads. 06:08.890 --> 06:14.390 And that's a lot of downloads of misconfigured EC2 instances, S3 buckets, etc. 06:14.850 --> 06:20.490 So just to make things clear, writing code in Terraform does not mean that it's insecure. 06:20.490 --> 06:22.150 It's the other way around. 06:22.270 --> 06:30.570 Terraform gives the ability to peer review the infrastructure's code and to have an impact on the configuration before provisioning. 06:31.330 --> 06:41.750 The thing is, is that this practice of reviewing this infrastructure configuration is still not in place in a lot of organizations. 06:41.750 --> 06:47.610 And what we see is that backup and recovery is very often missed. 06:47.650 --> 06:53.250 Logging and auditing, encryption, Kubernetes configuration, are also missed. 06:53.250 --> 07:04.090 You can understand Kubernetes that has so many knobs and switches, but encryption should really be turned on by default on most of the cloud services. 07:04.830 --> 07:18.510 And if we break it down to the different cloud vendors, we can see that there is a different missing piece on all of those open source modules. 07:18.510 --> 07:31.810 For example, in Google, there is often misconfigured networking piece, while on AWS, Kubernetes is the most often misconfigured code. 07:32.070 --> 07:36.090 So where do those bad configurations come from, usually? 07:36.090 --> 07:47.310 So you can copy them from a blog post, from Terraform registry, from GitHub, from your internal repository where you probably have a platform team creating those manifests. 07:47.830 --> 07:59.610 And it's really not, usually it's not as a result of a bad actor, but merely lack of knowledge or lack of time of all of those best practices. 07:59.610 --> 08:08.950 You have over 160 services only on AWS, and the list of their configurations grows on and on. 08:08.950 --> 08:17.070 So it's really a lot to miss, and a lot to be familiar with as a DevOps SRE or platform engineer. 08:17.590 --> 08:19.310 And here comes Checkout. 08:19.310 --> 08:22.890 So Checkout was released on December 2019. 08:23.210 --> 08:35.870 It already has more than 1,000 GitHub stars, 600,000 downloads, tons of best opinionated best practices that comes within it. 08:35.870 --> 08:45.530 So more than 400 checks that are opinionated, are based on the CIS benchmark, SOC2, PCI, and other best practices. 08:45.530 --> 08:50.090 And it has a bunch of CICD integrations. 08:52.510 --> 08:58.230 So what Checkout is doing is giving you a policy as code. 08:58.230 --> 09:05.110 It gives you the ability to define a best practice, and to version the best practice itself. 09:05.350 --> 09:15.190 Where this policy, for example, makes sure everything is encrypted by default, can be peer-reviewed, can be automated as part of this software development lifecycle. 09:15.330 --> 09:21.010 And specifically in Checkout, we decided that policies should be written in a familiar language like Python. 09:21.570 --> 09:24.270 So let's take this example. 09:24.270 --> 09:29.070 We have here a block that defines a database. 09:29.230 --> 09:37.450 And what is missing here is that storage is not encrypted by default on the specific database configuration. 09:37.450 --> 09:40.190 How would a Checkout policy look like? 09:40.310 --> 09:53.290 Well , when creating a custom policy, you can give it a name, an identifier, the type of resource that you would like to inspect, which is AWS DB instance, a category, which is encryption. 09:53.290 --> 10:02.130 And the scan itself is just, let's look for storage encrypted and make sure that it equals true. 10:02.350 --> 10:04.590 If it does, let's pass it. 10:04.610 --> 10:06.690 Otherwise, let's fail the check. 10:07.250 --> 10:14.870 And Checkout is doing all of the wrapping of making a very clear reporting option, a very clear CI integration. 10:14.930 --> 10:20.450 And we'll take a look on how does that look like on CLI. 10:21.050 --> 10:30.450 So what I'm going to do now is I'm going to demonstrate Checkout on a vulnerable by design project called TerraGoat. 10:30.450 --> 10:36.250 And if we will have time, we'll do the same for Kubernetes Goat and CloudFormation Goat. 10:36.310 --> 10:39.550 So let's spin up the demo. 10:41.760 --> 10:43.120 All right. 10:43.120 --> 10:49.600 So what I have here is just a very simple terminal. 10:49.600 --> 10:50.660 I'm using Mac. 10:50.660 --> 10:59.200 Before this talk, I've executed brew install Checkout or pip install Checkout. 10:59.200 --> 11:01.240 Both would work for you. 11:01.520 --> 11:07.040 And what I have here are those three projects of vulnerable by design infrastructure. 11:07.380 --> 11:14.100 But even on regular open source module, you will likely find a misconfig. 11:14.100 --> 11:20.460 So let's dive into TerraGoat and take a look on the project structure. 11:20.460 --> 11:28.800 So I have a Terraform directory that has resources among the three cloud providers. 11:28.800 --> 11:39.300 Specifically on AWS, I have a Terraform file for S3 buckets. 11:39.440 --> 11:45.520 So over here I have a bucket that is public, not encrypted, without access log and without versioning. 11:46.600 --> 11:56.800 And I have another file for EC2 instances that has plain text access keys. 11:56.980 --> 12:03.620 But if I wouldn't tell you that, it would have taken you a lot of time to see that. 12:03.620 --> 12:08.460 So let's run Checkout and see what it can do. 12:09.240 --> 12:12.900 So the basic parameter is minus D. 12:12.900 --> 12:18.540 Choosing a directory that Checkout will scan the infrastructure's code for. 12:18.960 --> 12:27.780 So what I'm going to do is I'm going to scan the AWS directory and see how the results look like. 12:28.660 --> 12:30.920 So it should take a few seconds. 12:30.920 --> 12:41.580 And what Checkout found here is 58 passing checks, meaning 58 times that cloud resources were with best practices on. 12:41.580 --> 12:44.080 And 53 failchecks. 12:44.080 --> 12:48.560 So over here I can see all of those passing use cases. 12:48.800 --> 12:53.860 So I have here a database application that is not open on RDP port, which is good. 12:53.860 --> 12:55.180 I've done some good work. 12:55.180 --> 12:59.440 But if I want to on the bad things that it has found. 12:59.700 --> 13:03.280 So over here I have a bucket that is not encrypted at rest. 13:03.280 --> 13:06.720 And this is why we have a failed check. 13:07.740 --> 13:12.320 And the resource is defined between line 1 and 13. 13:12.320 --> 13:13.480 And I can see it here. 13:13.480 --> 13:16.420 And I can see that there is no encryption block. 13:16.420 --> 13:18.980 I've also written it to myself as a comment. 13:19.640 --> 13:29.800 And if I'll take a look on the next set of resources that are right after S3, here I have an unencrypted RDS. 13:29.800 --> 13:31.980 And that is publicly accessible. 13:33.620 --> 13:40.300 And also plain text access keys found in this EC2 block. 13:40.300 --> 13:41.560 Here they are. 13:42.680 --> 13:45.900 Why is plain text access keys bad? 13:45.900 --> 13:49.280 Because they're often used for crypto mining attack. 13:49.280 --> 13:53.180 People can take those access keys and deploy a crypto miner. 13:53.180 --> 13:56.860 And basically my AWS bill will go up and up and up. 13:57.720 --> 13:58.860 All right. 13:58.860 --> 14:03.020 So we have installed check off using pip install. 14:03.020 --> 14:06.020 We've executed it on our local director of AWS. 14:06.020 --> 14:09.120 And we found a bunch of issues that we should solve. 14:10.460 --> 14:14.340 But that was like a one time execution. 14:14.340 --> 14:21.280 How do I make sure that check off will run on each and every change that I'm doing to my infrastructure as code? 14:21.360 --> 14:25.080 And I want check off to tell me, hey, you have this specific issue. 14:25.080 --> 14:31.080 You should solve it before committing it to your GitHub account. 14:31.300 --> 14:33.260 And from there to the production. 14:33.800 --> 14:41.520 So what check off has is actually a pre-commit integration where you can do exactly that. 14:41.520 --> 14:43.780 You can configure a pre-commit hook. 14:44.220 --> 15:04.200 And on that scenario, before each commit, check off will scan your local directory, your local repository, and will prevent you from committing that bad code into your GitHub if it does not follow those best practices. 15:04.280 --> 15:07.620 So let's take a look on how does that look like. 15:09.780 --> 15:19.880 So if I go up here, and what I have configured is a pre-commit config YAML. 15:20.240 --> 15:27.940 Mentioning that I should run the latest version of check off always on every commit. 15:27.940 --> 15:38.200 So what I've done before this talk is I've created a new Terraform file called S3 new. 15:38.540 --> 15:40.620 And it's not committed. 15:40.660 --> 15:43.120 Let's try to commit it. 15:45.200 --> 15:54.180 So what the pre-commit hook is doing right now is running check off upon commit. 15:54.180 --> 15:55.840 And it fails my commit. 15:55.840 --> 16:06.900 I cannot push new code to my source control unless it fixes those issues that are found in my S3 new Terraform file. 16:07.680 --> 16:09.140 Which is cool. 16:09.140 --> 16:13.680 Now I don't need to run check off manually on each and every change. 16:13.840 --> 16:17.460 I can just install the pre-commit hook. 16:17.460 --> 16:24.100 And now my workstation cannot upload bad S3 buckets. 16:24.760 --> 16:26.520 All right. 16:26.520 --> 16:27.800 So we have that. 16:27.800 --> 16:29.640 The misconfig is gone. 16:29.980 --> 16:36.800 And I cannot use my workstation to write bad Terraform code. 16:36.820 --> 16:41.660 But what about workstations that did not deploy a pre-commit hook? 16:43.500 --> 16:49.840 The best way to handle that use case is to install check off also in your CI CD pipeline. 16:50.480 --> 16:58.260 Where on each change request, pull request, check off will execute as part of your CI. 16:58.940 --> 17:03.980 And we'll call that check off job, for example, infrastructure security tests. 17:03.980 --> 17:07.100 And it works just like any unit test. 17:07.100 --> 17:13.680 Where you're running application tests, you can do the same with those 400 best practices of check off. 17:14.880 --> 17:26.120 If check off approves or if the test passes successfully, you can deploy your infrastructure code to production. 17:26.120 --> 17:32.020 If check off fails, on the other hand, it will block your pull request from being merged. 17:32.020 --> 17:36.040 Just like unit tests that are blocked from being merged. 17:36.360 --> 17:49.340 So to do that specific piece, we have developed actually a community contributed piece is a check off action. 17:49.560 --> 17:53.360 Let's take a look on how it looks like. 17:58.930 --> 18:10.630 So if you go to BridgeCrew.io check off action, you can see a project that is contributed by this handsome fellow called Chris Mavarkis. 18:11.350 --> 18:18.750 And to configure a check off action in your GitHub action is as simple as that. 18:18.750 --> 18:25.490 You just reference this specific action on your GitHub. 18:25.730 --> 18:35.590 And you can choose which checks to run, whether to run them all, skip a specific one, scan a specific directory, or use any of the other flags that check off has. 18:36.170 --> 18:47.790 And if you actually do that and enhance that into your CI pipeline, assuming it's on GitHub action, it will look like this. 18:47.790 --> 18:56.270 So over here I have a pull request on a repository that demonstrates a GitHub action of check off. 18:57.150 --> 19:06.230 So here I have created a bunch of resources to create a new web server in Terraform. 19:06.230 --> 19:22.270 So I have this AWS instance and the GitHub action have added annotations to my code saying, hey, your AWS instance has hardcoded access keys and it's using unencrypted EBS. 19:22.270 --> 19:26.350 And I should really solve those issues here in that code block. 19:26.430 --> 19:34.230 And I also have issues with my security group, which has port 22 open to the entire internet. 19:34.570 --> 19:36.770 And I should really solve that too. 19:37.350 --> 19:44.990 So check off will do a set of annotations for each and every resource that is violating those best practices. 19:51.690 --> 20:01.730 So I have this pretty CI pipeline that will check each and every commit. 20:01.730 --> 20:10.590 Not only if I'm doing that on my endpoint, but also if I'm doing that directly on GitHub or from an endpoint that does not have pre-commit hook deployed on. 20:14.250 --> 20:22.290 So what if I want to see the same being done for Kubernetes? 20:23.130 --> 20:34.030 So there is actually a very cool additional project, very similar to Terracode, that is about Kubernetes configuration. 20:34.150 --> 20:42.170 So it is a project created by Madhu and it's called Kubernetes code. 20:42.170 --> 20:50.410 It has some amazing scenarios of bad configured Kubernetes deployment manifests. 20:50.650 --> 21:04.470 And it has also a great Katakoda work through that I really recommend doing just to get familiar with what are the best practices and bad practices when configuring a Kubernetes cluster or a job. 21:04.490 --> 21:11.550 What I have here on my local endpoint is actually this 21:15.420 --> 21:17.480 project already cloned. 21:18.120 --> 21:24.680 And what I'm going to do is I'm going to run chkov-d kubernetes-code. 21:30.070 --> 21:35.030 I'm going to run chkov-d on the scenarios directory. 21:35.050 --> 21:39.630 And I've actually created my own scenario just before this talk. 21:40.070 --> 21:43.330 And I'm going to see what the results look like. 21:43.330 --> 21:50.550 So chkov will now do the same that it has done for Terraform, but now for Kubernetes manifests. 21:50.930 --> 21:57.130 So we have here a bunch of issues on my deployment channel. 21:57.130 --> 21:59.090 I like a lot of them. 21:59.150 --> 22:05.710 I have here service account tokens that are mounted not when necessary. 22:06.250 --> 22:10.890 I have UID conflicts. 22:10.890 --> 22:18.030 I have usage of root on my containers. 22:18.090 --> 22:23.770 And I should really solve all of those deployments configurations across my pods. 22:24.570 --> 22:50.730 The one thing that is nice, if you're not familiar with that specific configuration and what's the rationale behind it and how to solve it, you can always go to those documentation pages that we have publicly made accessible explaining what's the rationale behind this specific check and how to automate the remediation of that specific configuration. 22:50.830 --> 22:57.130 So let's say that you're encountering a specific config and you don't realize why you need to solve it. 22:57.130 --> 23:04.650 Here's the explanation where you have the step-by-step guidelines to fix that. 23:05.010 --> 23:12.530 And obviously if you want to try to fix that automatically, you can try the platform, but you can also use only the open source tools. 23:13.450 --> 23:29.090 As for the last piece, there is actually a very cool way to use checkup for reporting mechanism, just like reporting over passing and failing unit tests. 23:29.090 --> 23:34.190 Checkout has a reach output format that has JUnit XML. 23:34.270 --> 23:43.670 And if you integrate that into a CI system like AWS CodeBuild, you can actually get a very nice report showing the amount of passing and failing. 23:43.670 --> 23:51.390 And in some systems like Jenkins, also to show the trend over time of those configurations. 23:53.550 --> 23:58.910 Yeah, so I guess that's more or less about it on Checkov. 23:58.950 --> 24:13.650 Just to give some more references of how to use it, you can always go into Checkov.io, start on the installation page and getting started, which is the pip install piece. 24:14.210 --> 24:24.650 And right after you have those next chapters that explains to you what are the existing resource scans. 24:24.770 --> 24:36.050 So Checkov has a list of let's see 422 best practices that are opinionated that you can execute. 24:36.050 --> 24:46.310 But if you want to ignore some of those and run only specific ones, you can always suppress tests that you won't like to see. 24:46.310 --> 25:12.570 For example, if you have a public bucket and it should be public for website assets, you can just use a very similar annotation to the one that you would use for JUnit test, for example, or PyTest, which is Checkov skip, mentions the Check ID and a description of why this resource should not adhere these best practices. 25:13.250 --> 25:23.330 So over here I have an S3 bucket that should be public because it's serving as a static hosting for web content. 25:24.190 --> 25:37.890 And you can do a very similar thing using a Kubernetes annotation and mention why you want to skip a specific check and then Checkov will not fail on that specific Check ID. 25:38.770 --> 25:57.610 There is some documentation also of how to integrate Checkov with CI systems like GitHub Actions, Jenkins, GitLab CI, and also screenshots of how that looks like on those specific environments. 25:59.000 --> 26:05.970 All right, so if you have any questions, now would be a great time. 26:05.970 --> 26:22.990 I really encourage you to try our open source tools, follow our blog where we update on those, and if you have any questions on contributions that you'd like to make, feel free to reach out over email or Twitter. 26:23.650 --> 26:26.490 I would be more than happy to do so. 26:26.490 --> 26:29.990 Just a quick question for me, what's next for Checkov? 26:29.990 --> 26:35.010 Any other integrations that are in the works or any additional new features? 26:35.430 --> 26:42.210 Right, so we had a bunch of requirements coming from the community. 26:42.210 --> 26:44.070 Some have asked for... 26:44.070 --> 26:46.850 So the latest feature is actually ARM templates. 26:46.850 --> 26:57.430 We added ARM templates to Checkov so you can use it not only for Terraform or CloudFormation Kubernetes, but also for this one. 26:57.550 --> 27:07.050 And we also got requirements from the community for Ansible, and I think that's the big next manifest probably. 27:07.470 --> 27:15.190 We don't have any resources scheduled yet to handle this one, but any contribution would be accepted. 27:17.010 --> 27:27.010 And yeah, it already has most of the CI integrations and Docker, a brew install, so it should be pretty straightforward to use. 27:27.390 --> 27:35.030 Do you think if people used TerraGo modules more, this would help the number of misconfigurations to go down? 27:35.250 --> 27:41.270 So TerraGrant is a do-not-repeat-yourself framework for Terraform. 27:41.270 --> 27:46.530 It's a templating piece that helps to reduce the amount of code that we're writing. 27:46.550 --> 28:02.690 So if you're writing less code, you probably create less misconfigs, but if the templated configuration lacks the right defaults, so it will not reduce that specific piece. 28:02.690 --> 28:05.350 So the answer is yes and no. 28:05.350 --> 28:07.870 Checkov specifically does not support TerraGrant. 28:09.210 --> 28:13.230 Completely, but it does show value in some cases. 28:13.230 --> 28:19.370 So if you have a TerraGrant project, I would be more than happy to see the results of Checkov on it. 28:19.370 --> 28:26.110 We're researching how to make it work even better with templated modules.