Hello, everybody. It's great to be here at Cloud Village. And in the next 30 minutes or so,
we're going to talk about static analysis of infrastructure as code.
Just as a side note, if you're interested in contributing to any of those open source
projects around AWS security, IAM security, infrastructure as code, or Google Cloud,
feel free to reach out. It's Barak Shoster at Twitter, or just follow my activity at GitHub.
I really like to work on open source projects, and it is one of the things that is great about
BridgeCode that is contributing more and more open source security tooling back to the community.
All right. So on the agenda today, we're going to talk about the state of open source
Terraform modules. Are they secured or not? We'll go over how to run Chekhov
using two different integrations, pre-commit hook and CI deployment.
And at the end of this talk, you should be able to just download Chekhov and run it on your own
manifests. Before diving into deployment, let's talk about the problem space. As engineers,
like myself, and probably you who are listening, our goal is to move faster and in much agile way.
And when we develop a new product, we need some way to deploy the infrastructure that
this product runs on. Infrastructure as code came to the world to help us just
do all this manual work at the cloud providers like the AWS, Azure, GCP consoles of setting up
a VM automatically and having this job of setting those instances reproducible, version controlled,
and peer reviewed. Since for the first time, it's in code, you can peer review it as a pull request
or merge request in your VCS. When you are working fast, you're producing probably more code,
and with more code comes more bugs. So in a lot of times, when you want to just provision
a new cloud resource, let's take S3, for example, you might miss some of the default configurations
and keep the S3 unencrypted by default or without versioning or without access logs.
And the same goes for other resources like compute resources. You might forget access
keys in your plain text environment variables instead of using a secured store. And when
you're copying pre-made templates from the internet that are unvetted, those issues might repeat
more and more. So Gartner said, and this is often quoted, that 95 of the cloud security issues
are related to configurations errors. And the trend that is being done at the past two or three
years, I know that Terraform exists for a little more than seven years, is that we're codifying
each and every activity that we're doing in YAML files, HCL files, etc.
These tooling, those manifests that we write, the infrastructure as code, are about provisioning
new resources in a reproducible way and about managing them. But it lacks the security
functionality or best practices vetting functionality. And there are some great
tools around this area. There is OPA, the open policy agent of the CNCF, and also Chekhov,
TFSEC, CFNAG. Each of those is tackling the problem from a different angle.
So let's talk about the problem that we're trying to solve using Chekhov, which is open source and
under Apache 2 license. What we try to understand is the default configuration of any kind
of infrastructure as code manifest, meaning Kubernetes, Terraform, CloudFormation,
serverless framework, etc. is secured by default. So we asked ourselves that question.
Let's scan as first open source modules of those cloud manifests.
So we decided to use Chekhov to scan thousands of open source repositories.
Before diving into the results, there are some interesting stats about the trend of
contributing and creating new infrastructure modules. And this is focused only on Terraform
modules. So back in 2017, 18, and 19, we had a mild growth in the amount of new open source modules
contributed around Terraform. And specifically at the COVID era, around February,
probably due to work from home productivity benefits, the contributions of new modules
went sky high, which is great. We'll have open source, more code that someone else is writing,
is less code for me to write. But the thing is, the more code that is being contributed,
the more misconfigs that are being found in the open source repositories. So 48% of those modules
have some kind of misconfiguration. So if we take about 2,500 modules, we'll see a lot of
misconfigs there. And those open source modules, by GitHub statistics, had more than 26 million
downloads. And that's a lot of downloads of misconfigured EC2 instances, S3 buckets, etc.
So just to make things clear, writing code in Terraform does not mean that it's insecure.
It's the other way around. Terraform gives the ability to peer review the infrastructure's code
and to have an impact on the configuration before provisioning.
The thing is, is that this practice of reviewing this infrastructure configuration
is still not in place in a lot of organizations. And what we see is that backup and recovery
is very often missed. Logging and auditing, encryption, Kubernetes configuration,
are also missed. You can understand Kubernetes that has so many knobs and switches,
but encryption should really be turned on by default on most of the cloud services.
And if we break it down to the different cloud vendors, we can see that there is a different
missing piece on all of those open source modules. For example, in Google, there is often
misconfigured networking piece, while on AWS, Kubernetes is the most often misconfigured code.
So where do those bad configurations come from, usually? So you can copy them from a blog post,
from Terraform registry, from GitHub, from your internal repository where you probably have a
platform team creating those manifests. And it's really not, usually it's not as a result of a
bad actor, but merely lack of knowledge or lack of time of all of those best practices. You have
over 160 services only on AWS, and the list of their configurations grows on and on.
So it's really a lot to miss, and a lot to be familiar with as a DevOps SRE or platform engineer.
And here comes Checkout. So Checkout was released on December 2019. It already has
more than 1,000 GitHub stars, 600,000 downloads, tons of best opinionated best practices
that comes within it. So more than 400 checks that are opinionated, are based on the CIS
benchmark, SOC2, PCI, and other best practices. And it has a bunch of CICD integrations.
So what Checkout is doing is giving you a policy as code. It gives you the ability to define a
best practice, and to version the best practice itself. Where this policy, for example, makes
sure everything is encrypted by default, can be peer-reviewed, can be automated as part of this
software development lifecycle. And specifically in Checkout, we decided that policies should be
written in a familiar language like Python. So let's take this example. We have here a
block that defines a database. And what is missing here is that storage is not encrypted by default
on the specific database configuration. How would a Checkout policy look like?
Well, when creating a custom policy, you can give it a name, an identifier, the type of resource that
you would like to inspect, which is AWS DB instance, a category, which is encryption.
And the scan itself is just, let's look for storage encrypted and make sure that it equals true.
If it does, let's pass it. Otherwise, let's fail the check. And Checkout is doing all of the wrapping
of making a very clear reporting option, a very clear CI integration. And we'll take a look on
how does that look like on CLI. So what I'm going to do now is I'm going to demonstrate Checkout
on a vulnerable by design project called TerraGoat. And if we will have time, we'll do the same for
Kubernetes Goat and CloudFormation Goat. So let's spin up the demo. All right. So what I have here
is just a very simple terminal. I'm using Mac. Before this talk, I've executed brew install
Checkout or pip install Checkout. Both would work for you. And what I have here are those three
projects of vulnerable by design infrastructure. But even on regular open source module, you will
likely find a misconfig. So let's dive into TerraGoat and take a look on the project structure.
So I have a Terraform directory that has resources among the three cloud providers.
Specifically on AWS, I have a Terraform file for S3 buckets. So over here I have a bucket
that is public, not encrypted, without access log and without versioning. And I have another file
for EC2 instances that has plain text access keys. But if I wouldn't tell you that,
it would have taken you a lot of time to see that. So let's run Checkout and see what it can do.
So the basic parameter is minus D. Choosing a directory that Checkout will scan the
infrastructure's code for. So what I'm going to do is I'm going to scan the AWS directory
and see how the results look like. So it should take a few seconds. And what Checkout found here
is 58 passing checks, meaning 58 times that cloud resources were with best practices on.
And 53 failchecks. So over here I can see all of those passing use cases. So I have here a database
application that is not open on RDP port, which is good. I've done some good work. But if I want to
on the bad things that it has found. So over here I have a bucket that is not encrypted at rest.
And this is why we have a failed check. And the resource is defined between line 1 and 13. And
I can see it here. And I can see that there is no encryption block. I've also written it to myself
as a comment. And if I'll take a look on the next set of resources that are right after S3,
here I have an unencrypted RDS. And that is publicly accessible.
And also plain text access keys found in this EC2 block. Here they are.
Why is plain text access keys bad? Because they're often used for crypto mining attack.
People can take those access keys and deploy a crypto miner. And basically my AWS bill will go
up and up and up. All right. So we have installed check off using pip install.
We've executed it on our local director of AWS. And we found a bunch of issues that we should solve.
But that was like a one time execution. How do I make sure that check off will run on each and
every change that I'm doing to my infrastructure as code? And I want check off to tell me, hey,
you have this specific issue. You should solve it before committing it to your GitHub account.
And from there to the production. So what check off has is actually a pre-commit integration
where you can do exactly that. You can configure a pre-commit hook. And on that scenario,
before each commit, check off will scan your local directory, your local repository,
and will prevent you from committing that bad code into your GitHub if it does not
follow those best practices. So let's take a look on how does that look like.
So if I go up here, and what I have configured is a pre-commit config YAML.
Mentioning that I should run the latest version of check off always on every commit. So
what I've done before this talk is I've created a new Terraform file called S3 new.
And it's not committed. Let's try to commit it. So what the pre-commit hook is doing right now
is running check off upon commit. And it fails my commit. I cannot push new code to my source
control unless it fixes those issues that are found in my S3 new Terraform file. Which is cool.
Now I don't need to run check off manually on each and every change. I can just install the
pre-commit hook. And now my workstation cannot upload bad S3 buckets. All right. So we have that.
The misconfig is gone. And I cannot use my workstation to write bad Terraform code.
But what about workstations that did not deploy a pre-commit hook?
The best way to handle that use case is to install check off also in your CI CD pipeline.
Where on each change request, pull request, check off will execute as part of your CI.
And we'll call that check off job, for example, infrastructure security tests.
And it works just like any unit test. Where you're running application tests,
you can do the same with those 400 best practices of check off.
If check off approves or if the test passes successfully, you can deploy your infrastructure
code to production. If check off fails, on the other hand, it will block your pull request from
being merged. Just like unit tests that are blocked from being merged. So to do that specific
piece, we have developed actually a community contributed piece is a check off action.
Let's take a look on how it looks like. So if you go to BridgeCrew.io check off action,
you can see a project that is contributed by this handsome fellow called Chris Mavarkis.
And to configure a check off action in your GitHub action is as simple as that. You just
reference this specific action on your GitHub. And you can choose which checks to run, whether
to run them all, skip a specific one, scan a specific directory, or use any of the other
flags that check off has. And if you actually do that and enhance that into your CI pipeline,
assuming it's on GitHub action, it will look like this. So over here I have a pull request
on a repository that demonstrates a GitHub action of check off.
So here I have created a bunch of resources to create a new web server in Terraform.
So I have this AWS instance and the GitHub action have added annotations to my code
saying, hey, your AWS instance has hardcoded access keys and it's using unencrypted EBS.
And I should really solve those issues here in that code block.
And I also have issues with my security group, which has port 22 open to the entire internet.
And I should really solve that too. So check off will do a set of annotations for each and
every resource that is violating those best practices. So I have this pretty CI pipeline
that will check each and every commit. Not only if I'm doing that on my endpoint,
but also if I'm doing that directly on GitHub or from an endpoint that does not have pre-commit
hook deployed on. So what if I want to see the same being done for Kubernetes?
So there is actually a very cool additional project, very similar to Terracode,
that is about Kubernetes configuration. So it is a project created by Madhu
and it's called Kubernetes code. It has some amazing scenarios of
bad configured Kubernetes deployment manifests. And it has also a great Katakoda work through
that I really recommend doing just to get familiar with what are the best practices
and bad practices when configuring a Kubernetes cluster or a job.
What I have here on my local endpoint is actually this project already cloned.
And what I'm going to do is I'm going to run chkov-d kubernetes-code.
I'm going to run chkov-d on the scenarios directory. And I've actually created my
own scenario just before this talk. And I'm going to see what the results look like.
So chkov will now do the same that it has done for Terraform, but now for Kubernetes manifests.
So we have here a bunch of issues on my deployment channel. I like a lot of them.
I have here service account tokens that are mounted not when necessary.
I have UID conflicts. I have usage of root on my containers. And I should really solve
all of those deployments configurations across my pods. The one thing that is nice,
if you're not familiar with that specific configuration and what's the rationale behind
it and how to solve it, you can always go to those documentation pages that we have publicly
made accessible explaining what's the rationale behind this specific check and how to automate
the remediation of that specific configuration. So let's say that you're encountering a specific
config and you don't realize why you need to solve it. Here's the explanation where you have
the step-by-step guidelines to fix that. And obviously if you want to try to fix that
automatically, you can try the platform, but you can also use only the open source tools.
As for the last piece, there is actually a very cool way to use checkup for reporting mechanism,
just like reporting over passing and failing unit tests. Checkout has a reach output format
that has JUnit XML. And if you integrate that into a CI system like AWS CodeBuild,
you can actually get a very nice report showing the amount of passing and failing. And in some
systems like Jenkins, also to show the trend over time of those configurations.
Yeah, so I guess that's more or less about it on Checkov. Just to give some more references
of how to use it, you can always go into Checkov.io, start on the installation page
and getting started, which is the pip install piece. And right after you have those next
chapters that explains to you what are the existing resource scans. So Checkov has a list of
let's see 422 best practices that are opinionated that you can execute. But if you want to ignore
some of those and run only specific ones, you can always suppress tests that you won't like
to see. For example, if you have a public bucket and it should be public for website assets,
you can just use a very similar annotation to the one that you would use for JUnit
test, for example, or PyTest, which is Checkov skip, mentions the Check ID and a description
of why this resource should not adhere these best practices. So over here I have an S3 bucket
that should be public because it's serving as a static hosting for web content.
And you can do a very similar thing using a Kubernetes annotation
and mention why you want to skip a specific check and then Checkov will not fail
on that specific Check ID. There is some documentation also of how to integrate
Checkov with CI systems like GitHub Actions, Jenkins, GitLab CI, and also screenshots of how
that looks like on those specific environments.
All right, so if you have any questions, now would be a great time. I really encourage you to
try our open source tools, follow our blog where we update on those, and if you have any
questions on contributions that you'd like to make, feel free to reach out over email or Twitter.
I would be more than happy to do so. Just a quick question for me, what's next for Checkov?
Any other integrations that are in the works or any additional new features?
Right, so we had a bunch of requirements coming from the community. Some have asked for...
So the latest feature is actually ARM templates. We added ARM templates to Checkov so you can
use it not only for Terraform or CloudFormation Kubernetes, but also for this one.
And we also got requirements from the community for Ansible, and I think that's the big
next manifest probably. We don't have any resources scheduled yet to handle this one,
but any contribution would be accepted. And yeah, it already has most of the CI integrations
and Docker, a brew install, so it should be pretty straightforward to use.
Do you think if people used TerraGo modules more, this would help the number of misconfigurations
to go down? So TerraGrant is a do-not-repeat-yourself
framework for Terraform. It's a templating piece that helps to reduce the amount of code
that we're writing. So if you're writing less code, you probably create less misconfigs, but
if the templated configuration lacks the right defaults, so it will not reduce that specific
piece. So the answer is yes and no. Checkov specifically does not support TerraGrant.
Completely, but it does show value in some cases. So if you have a TerraGrant project,
I would be more than happy to see the results of Checkov on it.
We're researching how to make it work even better with templated modules.