So, yeah, I'm here presenting today on Cloud Native Attack Detection and Simulation.
It's an expansion of a talk I did at Forward CloudSec, and we've spoken previously on some
of this at some other conferences, too. But this is the first time I've had the opportunity,
really, to gather it all together into one big presentation. So I'm really happy to be here.
Hopefully, everyone will get something out of it. So there's a few key things to go through
today, really. The first thing is that I want to spend some time kind of comparing and contrasting
on-premise detection and cloud detection, what the key differences are between them, and some
critical architectural differences, but also just in terms of what an attacker's likely to try
versus what they did on-premise, how a lot of that varies. Yeah, exactly, digging into the
likely attacker activity, we're going to see what that actually comes out as on the cloud side of
things, what you should be looking for, where you should be looking for it, and so on. And lastly,
the simulation piece, sort of buzzword is this is the idea of bringing DevOps to detection.
We've seen a real strong uptake in a lot of the DevOps tooling in the cloud space,
because it makes everything easier if you can automate stuff. So we've spent quite a lot of
time building out some tooling that we've open-sourced to do just that, and we'll talk
through that in some more detail. But just before we get started, I'm sure pretty much everyone's
seen some sort of threat actor pyramid. I've got a slightly different take on it to what you often
see, in that rather than putting sort of China and Russia at the top and a bunch of script kiddies
at the bottom, we're talking more about capabilities than we are about individual groups.
So what we've got at the top is bespoke tactics, techniques, and procedures, things that attackers
are developing and designing specifically for whatever campaign they're on. And the middle
bracket, custom tooling, is where attackers are in a position where perhaps they're not
developing their own ODAs or doing their own security research, but they are building their
own implants, their own tools, and so on. And sort of the bottom end of the pile is people grabbing
tools off GitHub and firing them off willy-nilly across the world. And the reason I bring
this up is because, to me, the interesting part of detection sits at the top end there, at the bespoke
TTPs, the custom tooling. And realistically, that's also a very expensive place to sit if you're
trying to build detective capability. And so a lot of what I do is around driving the costs down
or making it easier for organizations and for people to pick up the top end, the advanced
end of detection, and use it within their organization. And that will come up quite a
few times through today's talk. So first off, on-premise versus cloud. So there's quite a few
key differences here, really, when it comes to attack detection. And one of the biggest is just
in terms of the telemetry that we've got. So on-premise, historically, the vast majority of the
useful telemetry and data that we'd be threat hunting in sat around the endpoint. So we're
running endpoint detection response products, you know, pick your favorite from CrowdStrike, Tanium,
Cyberese, and those sorts of things. And so we'd be running those on all of our endpoints, probably
our servers too, and gathering information like process execution trees, you know, what's
happening in memory, looking for specific syscalls and Windows API calls that might be malicious.
And we do a lot of our detection on that. If you've paid dark trace or vector a lot of money,
you might also have some network telemetry, but it's usually a secondary source. Likewise, with
data you might be getting out of your applications. The one key exception there is I'm going to class
Active Directory as an application in this little diagram here. Active Directory, obviously, is one
of the big points we can get a lot of useful detection information out of on-premise, especially
when it comes to sort of lateral movement and privilege escalation. But by and large, still,
you know, we're focused in really on the endpoints of the servers. Now, when we go to cloud, we've
still got all of that to some degree, especially if you're still running virtual machines in a lot
of your cloud environments, you know, you're running containers, we've got container security
monitoring solutions now that work with that. But actually, we've now got this box that sits around
it all, which I'm going to call control plane telemetry. So that is data that we're getting out
of the cloud itself and from a given cloud provider about what's being done from an administrative
perspective inside the cloud account that you're looking at, or the Azure subscription or Google
Cloud projects about who's creating, modifying, destroying what resources, where, when, and how
they're doing that. And so that really actually forms then the core of your detective capability
in the cloud. Attackers are still going to do some of what they used to on the endpoints
and at the network level. But really, a lot of the threats we've seen, and in fact, a lot of
the successful attacks, Capital One springs to mind, you know, happened against the control plane
where an attacker's got hold of some credentials for the AWS account, the Azure subscription,
and they're targeting that. Now, what comes with that is just a dearth of knowledge, really,
in terms of what that actually looks like. We've had sort of 20, 30 years to build up an
understanding of what an attack on premise looks like. We know what an attacker is going to try
on a Windows network full of, you know, Windows laptops connected to an Active Directory
infrastructure. It's all pretty well understood. And new techniques come out all the time.
But at the end of the day, the broad attack vectors and a lot of the stuff we should be
looking for are all pretty consistent, or at least reasonably well known, even if not
always that easy to spot. The problem then comes when we move to the cloud. Not only are we
dealing with more platforms than we used to, but also we just don't really have an idea of what
a lot of this stuff looks like now. Attackers aren't targeting workstations and virtual
machines. We're targeting serverless functions, Kubernetes clusters, the control plane. And so,
what that looks like. What an attacker is likely to try. And it's just nowhere near as well
understood. That said, from my experiences working in this space, I think there's a few key things
to call out, really. One of which is that there's this concept of uncertainty of malicious intent.
On premise, there's a lot of stuff that's very malicious and it's easy to signature off. That
doesn't really exist in the cloud. We'll get into that in a bit more detail in a second.
But one of the other things that comes with it is that context is key. Understanding what an
attacker is doing in a specific environment rather than just what they're doing in general
in a Windows estate. These things vary an awful lot in the cloud, whereas they didn't historically
on premise to anything like the same degree. And again, get into that more in a second.
And one of the big wins from a detection perspective in the cloud is that visibility
tends to be a lot easier. And so, if you look at Google Cloud, for instance, we can turn on
audit logs. In fact, they're turned on by default. And we can feed those into a Google Cloud
organization-wide log sync and get all of the logs for all of our projects at the control plane
layer fed straight into a single location and ship it from there off into our scene.
And there's no more sort of, we've got our EDR on 90% of our laptops, but there's these 10%
of legacy Windows 2000 boxes that we can't do anything with. There's nothing like that.
The problem comes, actually, that now shadow IT is even more of a problem than it used to be,
because it used to be that if you had shadow IT in your on-premise environments, you could probably
still walk around to a switch and have a look at what was plugged into it. And now you've got
pretty much zero visibility of marketing walking out to one of their third-party suppliers and
saying, hey, build us a website for this marketing campaign. Here's an AWS account that we registered
yesterday with one of our credit cards. Keeping track of that's really, really hard. And actually,
one of the tricks we've seen a few organizations use is to get really friendly with the finance
department and ask finance to essentially report all transactions for the cloud providers. And then
you can chase down which teams are doing what that way. It's a bit of a low-tech solution by
our usual approaches, but it works pretty well. And lastly, while we've got this fantastic ability
to automate lots of things in the cloud now with all the APIs they expose, the problem now is that
we're not the only ones leveraging that, right? Attackers are also automating a lot of what they
do. The classic example here, again, with AWS is attackers leveraging access keys that are linked
to GitHub for Bitcoin mining. You can subscribe to a public feed of commits to public GitHub
repositories, and attackers tend to feed that straight into essentially their own continuous
delivery platforms, which monitor the commits to public repositories, look for things that look
like AWS access keys, and feed those straight into a system that will then deploy Bitcoin miners in
the account of the compromised creds. It's gotten bad enough now that the meantime to compromise
between keys hitting GitHub and Bitcoin mining startings, about 20 to 30 seconds, and Amazon
themselves started monitoring that same feed and revoking keys when they find them. So while
obviously that's pretty low-tech and in the grand scheme of things not that interesting,
the fact that we're seeing that automated makes me think that in the long run
we're going to start seeing attackers building out tooling and capabilities that focus on
getting in and doing as much damage or as much of what they want to do as rapidly as possible
before the defenders have woken up and realize there's a problem.
And so I mentioned mindset shift here. We have this idea of uncertainty of malicious intent.
So there are some things that we know are obviously bad. If you've got Mimikatz running on a system,
that's probably not supposed to be happening. Likewise, someone accessing
LSAS memory or, you know, there's a variety of fairly common things that are obviously bad and
we can signature on them, right? That's how antivirus has been working for the last 20 years.
Some things are usually bad. HTA spawning PowerShell is a classic example there, but
actually that's not always bad. We were doing some threat hunting on one organization's estates
and found that when we ran a query for PowerShell being spawned by MSHTA files,
we got like 15,000 hits or something. So we immediately thought there'd been a major
compromise and started digging into it. And it turns out that actually Hewlett-Packard
enterprise printer drivers for a certain set of models work by, or the driver installations work
by using an MSHTA file to spawn PowerShell to do the installation. So it's probably bad,
it might not be. And then you've also got things like users requesting Kerberos tickets for
services they've not spoken to before, you know, network comms that isn't normal, but doesn't
immediately stick out as malicious and things that bear further investigation rather than things that
you immediately signature on. The problem in cloud is that there are very few things that are
obviously or usually bad. There's a few, but there's not many. The vast majority of what I'd
expect an attacker to do and what I see people doing when they're doing offensive security
testing revolves around abusing legitimate functionality, API calls that the cloud
providers expose to, you know, install persistence mechanisms, escalate their privileges, access
data they shouldn't be, all of these kinds of things. And so everything becomes about context.
So let's say we've got a, we're in AWS, we've got an IAM user, we've had some access keys added to
that IAM user. In principle, that might be okay, that might not. So we dig into it. It turns out
that was made, or that change was made by a user account or a set of access keys associated with
some kind of continuous delivery pipeline that we're running on premise. Okay, that's probably
fine. You know, actually that might be expected in this environment. But if we've got a change
made by an administrator from a weird network location, maybe without his 2FA that you'd
usually be using or something else, that actually might not be. And so we might want to further
investigate. But that might well be entirely dependent on the environment. And there are two
things that will come out of this. One of which is that over time attackers are going to become
more and more context aware. I'm already seeing that sort of continuous integration, continuous
delivery systems are favorite targets of a lot of attackers. I'll get into that in a bit.
And so being able to masquerade as sort of known trusted entities will become more of a thing over
time. But equally, that then presents a real problem for managed detection and response
providers or MSSPs who are trying to monitor a wide variety of client estates. Because there's
no such thing anymore as, you know, we detect known bad and that's great. We've got to be able
to monitor on a per environment basis, not even per customer or client. And so that's where things
like user behavior analytics starts becoming really useful. You know, is this behavior normal
for this environment, for this specific entity? Or is this something that we've never seen before?
Even just things like least frequency analysis, you know, for this environment, how often do we
see this particular API call made? You know, if it's never been called before, then that might
warrant some investigation. If it happens 500 times a day, then well, it's probably fine.
Equally, the other thing that comes out of this is that there are some things that just don't
exist from on premise in terms of actions and attackers likely to perform. And so let's say,
again, we're working in AWS, we've got an attacker who comes in and they run IAM create policy version
to create a new version of the policy document for one of the policies that's attached to them.
And so this is changing the permissions they've got by altering the permissions definitions,
basically. And actually, if you can do create policy version on any resource, and you have
policies attached to you, then that essentially gives you admin access. There's a concept that
doesn't really exist to the same degree on premise. So there's a number of these sorts of actions
where we don't really have an on premise equivalent. And these are kind of new techniques that
defenders are going to have to get their heads around.
This is then massively complicated by the fact that your average enterprise really isn't
one cloud provider with a few cloud accounts these days, right? I can't remember the last
time I went to a client and they said, we're just in AWS or we're just in Azure, and it turned out
to be true. F-Secure is a great example for that. In fact, we use AWS very heavily, but we have got
bits of Google Cloud and Azure lying around as well for various reasons. But even if you're
just looking at that, that's a fairly complex thing to deal with in and of itself. Multiple
cloud providers and completely different technology stacks and different log formats, all of that kind
of thing. When you then expand that to include everything else that exists in most organizations,
cloud estates these days, you know, we're probably pulling container images down from Docker Hub,
storing our source code in GitHub or something like that. We're probably using Slack for
comms and we've got Office 365 for our emails and all of these are fairly juicy targets for
an attacker in one respect or another. And they all need monitoring to some degree, which presents
a pretty big problem for a lot of organizations really, just getting their heads around all of
this. So when it comes to actually designing how you do detection in the cloud, I think it would
be worth just pausing for a second and talking through a few key observations that we've made
about how to successfully try and integrate sort of that much of a disparate estate, you know,
a wide variety of technology platforms. And the big one really is to centralize everything.
Have a single scene that handles your on-premise, your cloud, your everything, ship all of your
logs into it from everywhere, you know, all into the same single place. One thing that we've
definitely noticed is that if you have a security operations center or a threat hunting team,
what have you, who are monitoring quite a disparate estate, the more things you make
them go look at, the more likely it is that some things are going to get missed or people
aren't really paying as much attention to one thing as the others. So put it all in one place.
And the other thing you'll note here with this diagram is that for each of the major cloud
providers, there's like one account that we're shipping logs off to before they go into the scene.
And there's a few big benefits to this, really. One of which is that if we're shipping all of
our logs to a centralized location, we can lock that down to some degree so that only our security
people can access it, which means that all the logs, even when they're in the cloud still,
are in a place that an attacker probably doesn't immediately have access to if they've compromised
one of the project's accounts. And it means that it's then relatively straightforward as well for
defenders if, let's say, there's some issues with the scene or they need more data than that's
actually being shipped off to the scene for various reasons. Spunk charging per gigabyte
is usually one of the examples there behind why that doesn't happen. You might need more data.
So you go into this one location for each cloud provider, and that's where that data lives.
And there's also a few key data sources to be looking at here. And I'm going to pick a couple
out to dig into in detail. But you'll notice from this table, the control plane audit logs
have bolded at the top there. And there's a very good reason for that. And that is that,
in my experience, something like probably 70-80% of attacker activity is going to show up in those
control plane logs when we're looking at the cloud. Just because even if an attacker is targeting
virtual machines or what have you, they're eventually likely to want to pivot out into
the underlying control plane layer and start messing around with other resources in the
environment, data stores, or escalate their privileges or anything like that. That's all
going to come down through the control plane. So for AWS, you've got CloudTrail, you've got audit
log in both Azure and GCP, and actually Kubernetes has that for the Kubernetes API too. And as we
say here, this logs almost every control plane level event. I say almost because people keep
proving me wrong with that with AWS CloudTrail. We've seen a few people now digging around trying
to find API calls that aren't correctly logged, and there are a few here and there. AWS do their
best to catch them, but I'm sure that same issue will exist in Azure and GCP. But by and large,
you're going to get logging for pretty much everything that happens. The key thing to
remember here is to make sure that you have it all turned on properly. AWS is particularly bad
for this in that CloudTrail has a number of different knobs and settings you can tweak
that vary what it logs and how it logs it. So often we find that either multi-region logging
hasn't been turned on, which means you're only getting data for US East 1 or wherever it is
you've got the trail deployed. And equally, if you don't have global events enabled, then you
don't get anything for IAM or a couple of other critical services. So it's well worth digging
into your provider's documentation and working out exactly what you should have turned on there
and making sure you've got that. Now, Cloud Native Detection Services. So GuardDuty is
probably the big name here, but you've also got Advanced Threat Protection from Azure and
Security Command Center from Google has some of this built into its tool.
And what we've got here is automatic detection of lower sophistication attacker activity.
And what I mean by that is this is essentially sort of antivirus for cloud. We're spotting known
bad things like network communications to known malware command and control addresses.
In the case of GuardDuty, it'll detect when someone tries to run commands from AWS API commands
from Kali Linux if they haven't changed the user agent on the basis that obviously everyone using
Kali Linux must be bad, so therefore we should be firing alerts for it. And while they have their
flaws, actually they're still a pretty cost effective way of detecting sort of low complexity,
low sophistication attacks. So I would always turn it on. I just wouldn't rely on it. You know,
you need more than this to be able to spot more advanced attackers.
I'm also leaving out Sentinel when we're talking about this. So Sentinel is Microsoft's scene
that they go with or they pair with Azure. And actually Sentinel's pretty capable by and of
itself. And there's a pretty great team of people building some pretty serious capability into that
that goes well beyond what we've seen in GuardDuty and what they've got in the
advanced threat protection package. And so if you're in Azure and you've not got anything like
this set up yet, I'd definitely be looking quite hard at Sentinel. We've been pretty impressed with
it where we've been running into it so far. Service specific telemetry is the last one I
want to dig into here when we're talking about log sources. So here we're talking about the kind
of logs you get out of things like storage accounts, access and object logs, executions
of serverless functions, audit logs from your key management system keys, this sort of thing where
we've got specific telemetry about what's happening inside a specific service or set of resources.
This is a bit of an interesting one because everything here is going to be case by case.
Sometimes this data could be extremely valuable. Sometimes it could be basically useless.
So you're going to end up needing to tune this on a per environment basis and choose what to turn
on and where, you know, as relevant depending on what use you think you're going to get out
of the data. So one good example here is S3 access logs in AWS. If we've got an S3 bucket that's
doing nothing but hosting images and CSS files for a static website, we really don't care who's
accessing it. On the other hand, if you have your organization's most important data in some S3
buckets, it's probably well worth having those logs turned on so that you can spot who's accessing
them where, when, and how. So, yeah, this one takes a bit more thought. But once you've got,
like, the key main sources turned on, the audit logs, that sort of thing, this is well worth
looking into, I think. So next up, one of the real problems we've got at the moment is having
talked now about what data you can get to detect attacks with, we've got to work out what an attack
actually looks like, right? I've spoken a little bit about that to some degree, but we've got a
real problem actually at the moment just in terms of getting hold of data to show what attackers
are actually doing in the cloud. So I've been talking on and off to MITRE's cloud team about
this for the MITRE attack framework, and they're doing their best, but they're really struggling
to get their hands on decent threat intel, and I think the rest of the industry's in the same boat.
So this is the on-premise attack framework on the left, and the cloud one on the right, for those
who aren't familiar with MITRE attack, it's a taxonomy of likely attacker tools, techniques,
and procedures, TTPs, things that attackers likely to try in a given environment. So the one on the
left, as I say, is Windows. As you can see, there's a lot of things in there that we've
noted attackers doing over time. For cloud, much less so, as you can see on the right there.
And so if we were to purely rely on what we know attackers are doing actively,
we're barely going to spot anything, because there's just not very much information out there
about this. And where we see reports of, oh, we think an attacker might be doing X, Y, and Z,
it's actually often really hard to validate that. You can't just sort of jump onto Google and dig
out an old Mandiant or CrowdStrike PDF, you know, from the last APT29 report or whatever,
and there's a lot less information sharing on this. And part of that, I think, is actually the
cloud provider's fault. I have to say, we've not had great experiences working with some of them
on this. They're always very keen to hide any evidence of sort of real malicious activity,
just because it's potentially damaging to their reputation. But that doesn't help us
from a defensive perspective. So that said, you know, there are a few places you can draw
inspiration from, I think, for working out what an attacker is likely to do in your environment.
Certainly Scott Piper at Summit Root is going to be one of the big ones for AWS,
and the guys over at Rhino Security have been putting out some great research for years in
this space. If you're a fan of Google Cloud, then GitLab's internal red team have done some
pretty fantastic stuff there. And there's a variety of tools, too, that are pretty useful
for working this out. AWS especially, Paku from Rhino, is full of modules of things that an
attacker might want to do. And so given how powerful it is, you know, we use it pretty
frequently when we're doing offensive security testing of one kind or another. It's pretty
likely that a lot of the lower skilled attackers or people using public tooling are going to be
taking Paku and trying to do stuff with it. So if you can signature on most of what Paku's doing,
or at least be in a position where you've got the right data to be able to spot it if it does
happen, that puts you in a pretty decent place. So go have a look at a lot of the existing tooling
that's out there, and see what penetration testers say they're doing, you know, read around
on the blog posts they publish, and base a lot of your detective activity on what looks like it's
most relevant from what you see that way. And that's a pretty great way, in my experience,
to expand beyond sort of MITRE ATT&CK and the other common industry frameworks.
But even going beyond that, we can take a look at a bit of a high level as to what an attacker's
actually likely to do. So there's four key areas, really, that I think I see
us, other pen testing firms, red teams, and so on, exploiting on engagements.
The big ones really are identity management, pivoting around between environments,
source code management, continuous delivery, and application vulnerabilities.
You know, we'll dig into those one after the other. So first up, identity management exploitation.
So here what we've got is where we've got an attacker who's managed to get hold of some
credentials in some fashion or another, either from access keys they've stolen out of source
code or what have you, service account details that way, or they're targeting a single sign-on
mechanism, they found some credentials somewhere and they're doing that. What we end up with
is an attacker leveraging credentials to get into the environment and start doing things from there.
So this is pretty common. Verizon's 2020 data breach investigation report reckons that
somewhere in the region of about 80% of cloud breaches involved lost or stolen credentials.
So this really is the big ones we're paying attention to.
This also comes up in another context, which I'll get into in a minute, but actually
making sure that you've got sensible procedures around how you handle credentials, enforcing
multi-factor authentication and so on is really important. The other thing to note here is that
these credentials are often found in what seem like fairly stupid locations.
Performing penetration tests and stuff, we've often found them in file shares, SharePoint sites,
source code repositories to a degree too. People leave them lying around. And so that can be
pretty problematic. But even in cases where people haven't left the credentials lying around,
actually, a lot of the single sign-on tools that are used, a lot of the ways we do identity
management in the cloud aren't as well understood as some of the legacy on-premise equivalents.
So people make stupid mistakes, or say stupid mistakes. People make mistakes through lack of
education, lack of understanding of what's going on. In this particular case, we had an organization
deploy AWS Cognito, but pretty much by default and with the default configurations enabled.
For those who aren't familiar, Cognito is a single sign-on user management tool that's designed to be
integrated into web applications. So the idea is that you can use this to do your user management,
so you don't have to implement that in your own web app, and there's an AWS service that
takes care of it for you. You can also plug it into a few AWS services natively. So in this
particular case, they'd used it as the front end for a fairly large data store. And because
everything had been left at default, you had open registration enabled, so anyone could come
along and register an account. It's what you want if you've got a public-facing web application,
probably not if it's the authentication mechanism for a sensitive data store. And then it turns out
that by default, if you don't configure multiple user account groups, then everyone goes into the
default group, and the default group is administrator in the data storage thing that we were looking at.
So we essentially just registered ourselves an account coming in from the internet and had admin
access to this big data store. And so the important thing to note here is they'd not
deliberately misconfigured it. They'd not even accidentally misconfigured it. They'd left it as
default, and it turns out the defaults had really bitten quite badly in the back side in this case.
Next up, pivoting around between environments. So this is quite a common one we see.
Sorry, and we find that it's really common, obviously, for attackers to go in with phishing
campaigns, either for phishing for credentials or getting a foothold on a network with, you know,
pick your favorite impromptu, Cobalt Strike, Covenant, whatever, and pivot around from there,
move up into the cloud. And so actually, that can be a pretty effective route. We're seeing it an
awful lot that way around. We have also seen a few cases where attackers have somehow breached
the cloud environment, either for an application vulnerability or something else. And they've been
able to pivot around inside the cloud until they get to, say, an S3 bucket or some other data store
that's got something sensitive in it. And in a couple of cases, we found either sort of SSH keys
that allowed us to pivot back on premise over the VPNs they had set up in the cloud, or credentials
to sort of third party management solutions that they were using, that we were then able to go in
and manipulate what was happening there. So actually, the danger here is sort of in both
directions, as much as not in the diagram. You can either start on premise and pivot into the cloud,
or you can start on cloud and pivot on premise. And you'd probably pivot between cloud environments
as well. Not something I've ever done personally, but I'm sure it'd be feasible. And so as an
example, one thing I've seen pulled off on a number of occasions now, why I generally recommend
not using Active Directory for your cloud single sign-on, is that we've seen cases where an
organization's on-premise Active Directory has been compromised, or we'd compromised it as part
of a red team. We've got ourselves in, we've added some users that we have access to, to the
right groups in Active Directory to get cloud access, AWS, Azure, what have you. And then by
virtue of having that, we then become administrators in the cloud, at which point we've owned the cloud
as well. So either it's well worth investing in hardening your Active Directory, if you are
going to do Active Directory single sign-on to the cloud, and especially Azure AD. Or equally,
we've seen a number of organizations now go out and buy Ping, Okta, some other third-party single
sign-on, and use that for their cloud access, completely separate to all their on-prem stuff.
But it's definitely worth segregating or separating your Active Directory environment
and your cloud environments as much as possible, given how weak most legacy AD environments are.
Next up, source code management and continuous delivery. And so here, this is where we've got an
attacker who's targeting code repositories or the pipelines that are deploying these things into the
cloud. We're manipulating code being deployed as applications, or we're manipulating infrastructure
as code that controls the cloud environment itself. And if we can commit code, we can have
the pipeline push it out. If we can manipulate the pipeline, we can modify the code as it goes
through the pipeline. Essentially, your cloud is as secure as the things deploying the applications
and infrastructure. And that's something I think often gets overlooked. It's really important to
also take a look at your source code repositories and pipelines. Next up, application exploitation.
So we've got an attacker coming in from the web, and we're going back to sort of the early days of
application security, where SQL injection and remote code execution was something you found
on a regular basis. Obviously, it's a lot less common now. But we've seen new classes of
vulnerabilities crop up, or we've seen things like server-side request forgery become more relevant
now than they used to be on premise, thanks to instance metadata services and so on. But
being able to pivot into the cloud via an application attack is something that is obviously
very useful to an attacker, but it's something that gets exploited a fair bit, too, in my experience.
It's usually vulnerabilities in commercial off-the-shelf components, or open source apps
that are being deployed that have been misconfigured, in my experience, and we see a lot of that.
But generally speaking, obviously, this is still a significant threat, and that's why
companies still keep investing in AppSec teams. On the plus side, though, when it comes to
application exploitation, it's generally a lot easier to spot this stuff in the cloud than it
was on premise, or at least a lot easier to gather the right data for it. In this case,
we're running an Azure Kubernetes cluster, we've got some applications running in, and by the
time the traffic hits the application, we actually exploit it, it's passed through probably a web
application firewall on the front, we've got some kind of API gateway, we've got the ingress
controller going into the Kubernetes cluster, we've got the logs from the pod, from the container that
the app's running in, we've got whatever application logs are being generated, and we can ship all of
those out into our scene, and we can monitor and react to that. So there's quite a lot more we can
do here than on premise. Also worth noting, if you do decide to go down the route of detecting
against things happening in your applications, you should probably focus in on looking at
authenticated traffic. I'm sure we're all aware that most applications on the internet get
bombarded with SQL map scans and all that sort of stuff. The easiest way to discard all that
and focus in on the interesting data is by looking at what your authenticated users are doing.
So we talked about a variety of different things that an attacker is likely to try,
and this is essentially the paths we go down when we're looking to work out what we should
be spotting in a given environment. And threat model your environment, and work out what an
attacker is likely to try, how they're going to do it, what the likely attack paths are,
prioritize those based on what you think is most likely to be exploited, either because it's easy
or because it's the most obvious or whatever else. Then define those attack paths, ideally in some
sort of code format so it's easy to share around or machine read. Verify that the right logs,
telemetry is available to defenders, and then go in and do some actual testing to see whether
if we fire these test cases off, we get the right log data, we get something that we can action.
But if you're not familiar enough, really, with your cloud environments to be able to jump in and
do this, or you don't have the ability to do it at the scale your organization is working at,
you've got to pick somewhere to start. And one of the easiest places, some of the biggest quick
wins, are around either the objective end of the kill chain, where an attacker is looking to get
access to key data sources or to start ransomware-ing everything or whatever else, and look
at what those are, put key monitoring around that, but also look at infrequent actions that are
generally highly privileged or affect your security model. Things like anything that
involves turning off or altering the telemetry you're getting, modifying cloud trail, turning
guard duty off in AWS, changing your audit log configs in Azure, or anything around manipulating
your permissions model, your IAM policies in AWS, RBAC in Azure, anything that's modifying
any of that. Those will make really good points to build out some initial detection around.
So let's say now we've got some detection that we've built, and we think we've got some use
cases implemented to spot some stuff. How do we validate those? As I say, by this point,
we should have a set of likely attack paths identified, so we know what we're testing for,
we've got some use cases for that. We then go and execute those attack paths, usually manually,
and we then review whether we actually fired off some alerts or not, and then we work out what we
need to do next based on how much of our detection actually worked as expected. Now, some of this,
actually, we can quite easily automate, in theory at least, certainly executing the attack paths
where it's targeting the control plane layer or we're targeting things like Kubernetes that are
designed to be automated. It's generally fairly easy to automate the attack paths as well.
One of the more interesting ones is doing an automated gap analysis, essentially spotting
what fired that should have done, what didn't fire, and so on and so forth, so that we can
essentially track over time in an automated way our improvements, what's working, what's not.
And so these are all great ideas, but actually the industry is still mostly working on a manual
basis for all of this, especially in the cloud. So one thing I think, one really important lesson
to take away from the rest of the cloud community is treat everything as code. DevOps guys have
really got this down, and we're starting to see the security industry get on board as well in the
detection space. And the big reason for that, really, is that if all of your detection is
defined as code, it makes it much easier both to share knowledge internally, because anyone can
just take a look at your rules, and if they're in a common format that they already understand,
you know, some people have written some new ones, they can just go in and they can start reading it
and get their heads around it. And if we have reasonably common code formats, it also makes it
much easier to share. And so if we can share information on how to spot, you know, what given
threat groups are up to, specific TTPs, and we can either post those on GitHub or share them between
sort of some of the closed threat intel sharing groups, it means that everyone's detection gets
better for comparatively minimal effort on the part of an individual organization.
A few big things to pick out in this space, Sigma by Florian Roth. So we've got a set of
seam agnostic rules, essentially, that there's then a compiler for that produces your Splunk or
your QRadar or your Elasticsearch queries off the back of this common rule set. So someone
writes a Sigma rule for a particular TTP, and then you have the ability to use that in every
platform going pretty much. There's a lot of compilers in there. The other interesting one
in this space is Jupyter Notebooks. So this is more sort of threat hunting, you know, proactive
investigation rather than reactive alerting based on known TTPs. But the idea of a Jupyter Notebook,
for those who aren't familiar, is it's essentially a way of having a document with Python embedded
in it. And so you can run code inside your documents, basically, and tie explanations of
what's happening and the data that an analyst is seeing, and directly to the executing code,
which makes it quite a nice way to share information, share threat hunts. And the Spectrops
guys in particular have been doing some fun stuff with that. But it's starting to take off now. It's
a data science thing, but it's been quite nicely repurposed now. And if you're looking for another
conference talk to watch, John Lambert's the GitHubification of InfoSec from T2 last year.
It was a really interesting presentation on how the industry needs to work better together
to start making it more expensive for attackers to compromise organizations by sharing knowledge
better than we do right now. Now, one of the interesting problems we've had historically
in the cloud is that a lot of the existing solutions for automatically testing attacker
activity don't extend to the cloud. They're entirely focused on on-premise at the moment.
So things like MITRE Caldera, Atomic Red Team, there's a number of tools out there that have
been very successful at automating attack simulation on-premise estates. But actually,
we didn't have that for the cloud. So I went away and built one called Leonidas, which we've
open-sourced. So Leonidas is essentially a web API where we have a variety of different
attack definitions defined within it, and each API endpoint is a test case you can execute.
So under the hood, actually, the way it works, we took a slightly different approach to how a lot
of these tools work in that it's been designed to make it as easy as possible for you to share
the ability to execute these test cases and to detect them between your team members internally
and within the wider community, too. So let's say we've got a security team, we've got some
analysts, we've got some red teams, purple teams, whatever, and one of them's read a new blog post
about some new TTP in the cloud that they're interested in. So they sit down and they define
this in our format. We've got like a YAML-based definitions format. And you write the TTP,
it gets checked into a Git repository. We then have a CICD pipeline that essentially builds out
a copy of the API based on the definitions that are in the Git repository. And that runs as a
serverless function at the moment, a Lambda function AWS, but we're building out support
for Azure and GCP, too. And so you've then got this API exposed, and we can have some target
resources that we're looking to attack. Your purple team comes in or your red team comes in,
executes some attacks against these target resources, and we have our logs shipped out to
our scene. And so we can then verify whether when we triggered this new TTP, whether we
have the right logs in place, whether we spotted it or not, and whether we need to tweak things
further. So from these definitions, we define the actual test case itself. In this particular case,
this is an AWS test case. We're looking at enumerating cloud trails, which as an attacker
you probably want to do to see if you're being monitored or not. And so we've got a single
line of code here, Python calling the describe trails API function on the cloud trail APIs in
AWS. And behind the scenes, the framework abstracts away a lot of things around what identity we're
executing this as, which region are we targeting, all these kinds of things. And so the actual
code that's written for these test cases is pretty minimal, which obviously then makes it easier to
write a lot of these quite fast, or make it easy for less skilled analysts to get involved in
developing these. We also define what you need to detect the use case as part of the same definition
file, right? So we've got both the test case and the detection case defined together. Here we've
got a detection case that says we've got a event name of describe trails with a particular event
source, and we're likely to find that in cloud trail audit logs. So we can then compile that,
well, we usually compile it down to Sigma and then rely on Sigma's compilers to compile it out for
the different SIEM solutions. But we can also easily compile Lucene queries for Elasticsearch
or pretty much whatever you want off it. You know, the data's there, it's just a case of writing
something to generate the right queries. And also because we try and be responsible about these
things, we define the permissions that are necessary for Leonidas to execute a specific
test case. And so what we actually do here then is generate the right IAM permissions for AWS,
or the necessary permissions for your service accounts in Azure or what have you,
in order to allow us to execute the test case as safely as possible. And we can also set
restrictions on which resource it can target and so on to make sure you're only, you can only
leverage it against test environments rather than your production accounts. Or we can also generate
some documentation off the back of it. All of this is defined in a reasonably human-readable
but not necessarily friendly file format. And so instead, what we do here is we then generate
markdown files off the back of that and use mkdocs, for those who are familiar with it, to generate a
pretty website, which makes it a lot easier for those who are less familiar with some of this
stuff to understand what's actually happening, what this test case is, how it works, and ties
it back out to the MITRE framework. So you can click on the MITRE links there and it takes you
out to the relevant TTP there. So we can do quite a lot really with it. So in the spirit of this
being both a presentation and sort of a tool demo, it's demo time. Let's hope this works.
So what we have here is a case configuration file for Leo, which is essentially a Python
script that allows you to easily talk to that API and comes with it. So we can just define some
fairly simple test cases here that say we're going to do an AWS attack path that's going to work out
who we are, have a look at some defensive measures that might be in place, set up some persistence,
and then steal some secrets. So what we're going to do now is actually run that.
Fingers crossed this actually works. Let's hope the demo gods don't bite me in the backside.
Here we go. So we're hitting the endpoint and we've got here, we've got Leonidas hosted up
as a lambda function behind an API gateway. So we've run a test case to work out who we are,
and this has come back with us being Leonidas. So we've now had a look and see if we've
got any guard duty detectors running. We have. It's not ideal, but we're going to carry on anyway.
We just hope that we can move faster than the defenders can. So next up, we're going to
enumerate the cloud trails that we've got running. And so this was with the idea of spotting whether
we've got anyone who's actually monitoring us, who's going to be hunting down our attacker
activity within that. There we go. So we've got two cloud trails running here, both of which are
probably going to pick us up, but that's fine. We'll carry on anyway. So we've now added an IAM
user in this particular case, Defcon user here. And that's to allow us to come back later on if
we wanted to. Let's add an API key to it. And so we've got programmatic access. So that's come
back with some access keys and things, which I'll be deleting straight after this talk.
We then have a look and see what secrets we've got running in secrets manager.
And so in this particular case, we've got a secret in here already that I created
another time. And we're then actually going to execute the access commands to then actually dig
out the secret value here. And you can see we've got a test API key lying around inside the secret
string. So we just run a essentially seven-step kill chain here with a short YAML file defining
what test cases we want to execute. And that's all just hitting an API that's running up in AWS.
So now on top of that, we can actually just interact with it through the Swagger interface
if we wanted to. So you can test out the test cases here if you wanted to as you're developing
new ones. And you can run all this locally as well as in Lambda, obviously. And this is just
really to help with development. But you've also got Swagger definitions here if you wanted to
import that into something like Postman or Insomnia or build it into other tooling.
And the API is pretty easy to work with. And the other thing that is worth me calling out
quickly here is that Leonidas actually generates its own logs. So I'm just going to take a quick
look. Let's load up the CloudWatch log streams from what we've just done. No, in fact, we won't
be doing that. So we'll get into that in a second. But one of the really powerful things
with this, then, we've got an API. We've got a set of things we can execute automatically.
And we can then build that into pretty much any other security testing that we've got going on,
especially some of the on-premise testing that we're already doing. Let's say we're using
something like Caldera or Atomic Red Team to do on-premise simulation already. Actually,
we can tie this together, and we can have simulations kicking off on-premise. Let's say
simulate some phishing, a bit of lateral movement, and some Active Directory privilege escalation,
and then pivot into the cloud from there, and then start using Leonidas to trigger
cloud test cases off the back of that with the credentials that we've just stolen in the
simulated attack on-premise. And to me, that's really exciting. We've been able to do
on-premise automation for a while, and we've done a fair bit of manual stuff in the cloud like this,
but now we can tie all that together and execute hybrid kill chains in a completely automated
fashion. So with the log data that I mentioned, Leonidas generates its own log telemetry on
I executed this test case, this identity, these parameters were passed in, this was the resulting
data that came back, and that all feeds into a CloudWatch log stream in AWS with Stackdriver
or Azure Monitor for the Azure ones. And so we can ship that out somewhere ourselves and monitor
that separately. And one thing that strikes me as being really exciting here, or some potential for
some really exciting work, is then integrating your seam alerts and your tagged notables, whatever,
that are being generated by these test cases, and essentially performing automatic gap analysis,
working out for every Leonidas test case that's fired, as denoted by the Leonidas logs that are
coming in, did we get what we expected out the other side of it? And we can then track over time
as we write new detection cases and tweak things, how much better we're getting, which gives us some
nice data both to make us feel good about ourselves, but also it helps justify to whoever
it is that's signing the paychecks, and that what we're doing is worthwhile, and that we're improving
over time, and we can demonstrate that, and then so on and so forth. So one of the reasons I bring
up that sort of continuous integration aspect specifically is that what we've got here is the
ability to support us on a continuous journey. We can consistently improve our detection capability,
write new use cases, write new test cases, embed them in Leonidas, immediately start
simulating them alongside all the other ones we've already got, and it supports us on that
journey. Detections are moving target, attackers are consistently coming up with new techniques,
new mechanisms for targeting you, new tooling, so on and so forth. So we need to be able to move with
the times as well, and being able to define all this stuff as code and automate as much of it
as possible makes it easier to get better faster with fewer people. And so it's worth actually
treating your detection improvement projects as a software dev project. I find that that works
pretty well. You want to be identifying new threats and risks on a continuous basis,
or at least taking the time to do that regularly and feeding that into some sort of design process
to come up with new use cases for you to detect with. We then simulate that, we evaluate whether
what we've changed as a result of that actually made some improvements or not, and we cycle
round and round. And that's how you get better at this stuff over time. So there's a few things to
conclude on there, I think, really. First off, actually, like the cloud brings a lot of benefits
from a security perspective. And loads of people have been talking about that in the prevention
space for a while. There's a lot from the detection side as well. We've got visibility
like nothing we've ever had on premise before. If you've got a list of all your cloud accounts,
actually, you can get all of your log data comparatively easily. And we don't end up in
this situation where we've got some visibility of some things and not of others. And we lose
attackers because they've pivoted off the box that they landed on and they persisted inside
some legacy printer with some old dodgy firmware and we've got no monitoring for that.
We don't have those kinds of problems in the cloud anymore. The problem that comes with that is
all of this, the benefits that the cloud bring, all of the APIs and the automatability, means
that our attackers are already automating this stuff. We were talking about the crypto mining
earlier, but that's going to become more and more of a problem over time. We're going to start seeing
attackers automating more complex, more interesting kill chains to go after potentially
more damaging things. At the end of the day, Bitcoin mining is not great. You don't really
want someone having you pay for their cryptocurrency. But it pales in comparison
to attackers automating, like stealing loads of really important data or defacing someone's
front page of their website or whatever else. And we're going to start seeing attackers doing that
in the long term, I think. So we need to get better at this now. We need to be in front of this whole
cloud detection thing from the go. And we need to be getting better at this as an industry.
And so to go along with that, we released Leonidas, which is a tool for automating a lot
of this stuff, both the attack simulation and the attack detection piece and write the detection
cases. Right now, AWS only. We are not actually that far off Azure and GCP now. We're starting
to test some of those out. There's currently 40 test cases in there, 41 for AWS. But we're
pushing more out all the time. And you can grab it over our GitHub now, fsecurelabs.leonidas.
So thanks for listening. And at that point, I'll take questions.
Just one question from the audience.
What do you recommend they use for an identity store other than cloud SSL?
So basically, I'd recommend anything that's not Active Directory. And so we've seen a lot
of organizations pick up things like Ping, or Okta, or Auth0, Authy. And there's a number
of providers that offer single sign-on platforms that are designed to essentially do single sign-on
for large cloud estates. And the reason that I recommend that is purely that it breaks that link
between your on-premise legacy identity store and the cloud. And while that brings in a bunch of
extra overhead from a management perspective, all of your sort of HR joiners, leavers, processes,
all of that kind of thing, you know, it's extra steps to add to that. It also then means that
if your Active Directory is breached, you don't get immediately owned in the cloud as well,
which is something that, you know, in my experience working with Active Directory
over the past few years, is something that would happen to quite a few organizations, I think.