WEBVTT 00:01.700 --> 00:02.940 Hello, DEF CON. 00:02.940 --> 00:07.880 I speak to you today from my own version of the Batcave, the ultimate in social distancing. 00:08.160 --> 00:11.160 This is Hacking the Hybrid Cloud, and I am Shah Metcalf. 00:11.540 --> 00:19.180 I'm the founder of Trimark, a professional services company that helps organizations better secure their Microsoft platform, including the Microsoft Cloud and VMware infrastructure. 00:19.360 --> 00:22.980 I'm a Microsoft certified master in Active Directory and a Microsoft MVP. 00:23.080 --> 00:29.360 I've spoken at a number of conferences and I'm excited to be back speaking at DEF CON, even if I'm not physically at DEF CON. 00:29.360 --> 00:34.840 Last year, I keynoted the DEF CON Cloud Village and three years ago gave a talk on Hacking the Cloud at DEF CON. 00:35.280 --> 00:44.700 This talk will take a journey through the hybrid cloud, starting with the comparison between cloud and virtualization discussing how on-prem Active Directory domain controllers can be compromised. 00:45.020 --> 00:53.320 I will also walk through the cloud hosted managed Active Directory deployments from AWS, Azure, and GCP. 00:53.640 --> 00:59.660 Since this is hybrid talk, I will walk through some hybrid components and see how they can be attacked. 00:59.760 --> 01:11.400 I'll touch on cloud administration and identity access management, or IAM concepts, and finally walk through escalation scenarios for compromising cloud-hosted on-prem domain controllers, AWS, and Azure. 01:12.220 --> 01:16.700 First, we need to talk about what hybrid cloud is, at least in the context of this talk. 01:16.700 --> 01:19.460 There are likely lots of definitions of hybrid cloud. 01:19.580 --> 01:23.440 Due to this, I'm defining hybrid cloud as something on-prem and something in the cloud. 01:23.440 --> 01:28.480 This typically means infrastructure on-prem and in the cloud, more IaaS than strict SaaS. 01:28.960 --> 01:32.400 Now that we've defined hybrid cloud, let's look at three common scenarios. 01:32.480 --> 01:37.480 The first is the on-prem Active Directory environment with Office 365 SaaS services. 01:37.540 --> 01:39.700 SaaS stands for software as a service. 01:39.740 --> 01:45.560 Many organizations are leveraging the cloud for infrastructure as a service, or IaaS, as their cloud data center. 01:45.560 --> 01:51.880 The third scenario here is an on-prem Active Directory with a cloud-managed AD as an additional or resource forest. 01:51.940 --> 01:57.440 Since cloud can be used in many ways, it's likely that most organizations use a combination of these, among others. 01:58.380 --> 02:06.060 This section covers virtualization concepts and controls as foundational information to better set up and describe potential attacks. 02:07.020 --> 02:11.240 Conceptually, the cloud is virtualization, certainly from an IaaS perspective. 02:11.240 --> 02:13.960 Amazon has used Xen and a switch to Nitro. 02:14.300 --> 02:20.000 Azure uses a custom version of Hyper-V Core, and Google Cloud Platform uses KVM. 02:21.520 --> 02:28.780 Now, there is a cloud fabric that ties the virtualization elements together with orchestration, which is what makes the cloud the cloud. 02:29.300 --> 02:33.440 So, this diagram shows Azure IaaS with a VM and related components. 02:33.440 --> 02:35.740 This is similar to on-prem virtualization. 02:37.500 --> 02:39.300 And Amazon's Nitro is pretty cool. 02:39.300 --> 02:44.800 The Nitro hypervisor can take advantage of Nitro-optimized hardware, including a Nitro security chip. 02:44.800 --> 02:47.660 Any instance created prior to Nitro is still running Xen. 02:50.400 --> 03:03.700 AWS also has this concept of connectivity between the AWS-managed Active Directory that they host themselves, and EC2, which is their IaaS environment within the AWS cloud. 03:03.700 --> 03:11.380 And then, of course, since you can host Windows servers and Windows systems and other types of systems, you can also integrate and connect to other cloud environments. 03:11.380 --> 03:14.820 This demonstrates how cloud can get complex pretty fast. 03:15.840 --> 03:21.800 And you can run VMware on-prem and in AWS and Azure and GCP. 03:23.480 --> 03:30.300 Domain controllers control the keys to the kingdom and require special security protections that are often ignored on virtual platforms. 03:30.780 --> 03:38.480 Compromising physical domain controllers boils down to either gaining physical access or compromising an out-of-band management system like iLO. 03:38.540 --> 03:43.640 iLO hosts a web server on port 2381, and we can scan for this using PowerShell. 03:43.840 --> 03:54.300 So, if iLO is hosted on the same network as production systems and clients, such as workstations, we can scan and identify iLO. 03:54.420 --> 04:01.920 iLO has had a number of security issues with firmware patches available, but the issue is that pretty much no one updates firmware on physical servers. 04:03.080 --> 04:17.360 Airbus Security has identified many of these security issues with iLO, from accessing memory to an RCE to get read-write memory access to the ability to execute arbitrary commands via DMA to gain host access through iLO even when iLO is disabled. 04:17.440 --> 04:26.180 And I have a link at the bottom that includes a list even beyond what I have up here and a Go scanner to discover vulnerable servers running iLO. 04:27.280 --> 04:40.340 When VMware is hosting virtual DCs, which is probably around 95% of the organizations are better, certainly the larger ones, there's different ways that you could compromise VMware to get access to a virtual domain controller. 04:40.460 --> 04:44.280 This could involve compromising an account with rights to VMware. 04:44.280 --> 04:59.840 Often this is a regular user account that is a member of an 80 group called VMware Admins, or we just need to compromise an account that has VMware access to, or at least one virtual domain controller, potentially more. 04:59.860 --> 05:04.860 If we can compromise a system which runs vCenter, we can ultimately compromise VMware. 05:05.220 --> 05:08.700 Also, every ESXi server has a root account. 05:08.700 --> 05:17.640 And most of the time, every ESXi server has the same root password, which may be the organization default password, which could be guest or found in a password file. 05:18.260 --> 05:24.580 We can compromise the vCenter system, we can compromise the vSphere environment. 05:25.740 --> 05:35.440 And the VMware VIX API has functionality that allows for direct access to guest OSs, which is used by solutions such as VMware Site Recovery Manager. 05:35.440 --> 05:39.500 This requires VMware tools to be installed on the guest OS or VM. 05:39.500 --> 05:52.120 The VIX API allows guest operations such as starting, stopping the application, file directory manipulation, uploading, downloading files, all within the guest OS without requiring any network connectivity to the virtual machine. 05:52.120 --> 05:59.840 This functionality may also be used by vSphere accounts with limited privileges to access a guest OS without the need to authenticate. 06:00.120 --> 06:14.140 There is a scenario where an account with vCenter privileges can use the vSphere API to interact with the local OS installation of VMware tools to allow a pass-through session for an unprivileged account to connect to that virtual machine and execute local commands. 06:14.140 --> 06:27.500 Since this applies to virtual domain controllers, Active Directory could be compromised in this manner by allowing execution of commands locally on the virtual DC with accounts not normally allowed local logon access to the DCs. 06:29.260 --> 06:31.900 For Hyper-V, the attacks are similar to VMware. 06:31.900 --> 06:42.600 We compromise an account with rights or find a way to compromise a server, either through a server admin account, discover the local admin password, or a modified group policy to gain access to the server. 06:44.520 --> 06:53.660 So I decided I would take a look at the so-called big three cloud hosting providers, Managed Active Directory Environments, and then focus on what this means to Pentesters and Red Teamers. 06:54.120 --> 07:03.140 The Managed Active Directory Environment does not provide AD admin rights to the AD environment, so the customer will never have domain admin rights or even domain controller access. 07:03.340 --> 07:10.540 The cloud provider controls AD infrastructure management and delegates rights to OUs and AD components to the customer. 07:11.400 --> 07:15.880 Let's start with Amazon AWS Directory Services from Microsoft Active Directory. 07:15.880 --> 07:17.560 That's quite the mouthful. 07:18.680 --> 07:27.740 The AWS Managed AD has an AWS Delegated Groups OU that contains the delegation groups which the customer is provided membership in and access to. 07:27.740 --> 07:32.600 The AWS Reserved OU is controlled exclusively by the AWS system. 07:32.600 --> 07:44.060 There is a customer OU created in the Managed AD domain that is typically named after the AD subdomain or the first part of the fully qualified domain name or the domain's NetBIOS name. 07:44.060 --> 07:47.900 This customer OU contains OUs for computers and users. 07:48.860 --> 07:55.480 AWS Managed AD spins up two domain controllers which are in different network segments but in the same Active Directory site. 07:55.480 --> 08:01.920 The default domain admin account retains the default name administrator, though this is moved to the AWS Reserved OU. 08:02.280 --> 08:17.980 When a customer creates an AWS Managed AD, their first account in the AD environment is called admin and is granted full delegated rights to the AD environment for that customer, not to the domain controllers or AD admin rights to the full AD. 08:18.840 --> 08:26.840 The domain password is default, still seven characters, but there are five AWS-created fine-grained password policies which are delegated to the customer to modify. 08:27.140 --> 08:36.340 Based on the group policy linked to the domain controller's OU, I was able to identify a pretty decent audit policy, though it was missing Kerberos auditing, which means Kerberos activity would be missed. 08:37.760 --> 08:49.080 The first customer account, admin, is created with the description that notes the customer OU distinguished name, so it's very nice that they provide this description so that way you can quickly and easily identify where this OU is. 08:50.020 --> 08:53.800 There are several delegation groups in the AWS Managed AD environment. 08:53.800 --> 08:59.600 All these are named with the prefix AWS-delegated and are contained in the AWS Delegated Groups OU. 08:59.860 --> 09:04.140 The AWS Delegated Administrators Group is the primary customer admin group. 09:04.200 --> 09:11.300 There are also delegation groups for managed service accounts, adding workstations to the main Kerberos delegation, DNS, and server administration. 09:12.720 --> 09:16.080 Let's take a look at Azure Active Directory Domain Services. 09:16.180 --> 09:30.840 While Azure Active Directory is not exactly Active Directory, since it's a cloud directory with no Kerberos or NTLM authentication, group policy, or LDAP, Azure AD Domain Services is Active Directory, just like you would have on a corporate network, and is Microsoft's managed Active Directory. 09:32.440 --> 09:38.000 So this environment, Azure AD DS, has several top-level OUs that are created. 09:38.000 --> 09:47.740 These are prefixed with AADD, and the AADDC computers and AADDC users OUs are the primary customer OUs. 09:48.720 --> 09:57.860 The Azure AD DS domain controllers are in the same network, and at least in this instance that I created, had sequential IPs, .4 and .5. 09:57.860 --> 10:05.120 The default domain admin account is called DCAASAdmin, and remains in the default user's OU. 10:05.180 --> 10:08.020 There is one fine-grained password policy that's pre-created. 10:08.020 --> 10:09.880 I'm not going to try to read the name. 10:09.980 --> 10:17.840 As it is, it's a bit tough with all the AA and DDs, but the customer has delegated modifier rights to this fine-grained password policy object. 10:18.060 --> 10:25.220 I can't see any auditing configuration set in GPO, so I have to assume that they are set via local policy, which I can't see without domain controller access. 10:27.860 --> 10:37.860 Azure AD DS has a few delegation groups which are prefixed with AADDC, and are in the AADDC users OU. 10:37.860 --> 10:45.980 The primary customer group is AADDC administrators, which is delegated full control to many OUs and AD components. 10:46.140 --> 10:52.920 I'm pretty sure when Microsoft came up with the names for these groups, they were not thinking about having to say this out loud in a row. 10:53.040 --> 10:55.260 So thank you, Microsoft, for that. 10:55.900 --> 11:03.340 Moving on to the third and final managed AD environment that I reviewed recently, this is the Google Cloud Platform Managed Active Directory. 11:07.090 --> 11:16.990 It's interesting to note that the GCP-managed AD seems to cost about twice what the Amazon AWS and Microsoft Azure-managed AD environments cost, because they're about $100 each. 11:18.430 --> 11:25.650 The GCP-managed AD environment has a single root OU for the customer called Cloud, with a single child OU called Computers. 11:25.650 --> 11:29.170 The Cloud Service Objects OU contains delegation groups. 11:30.350 --> 11:42.630 GCP-managed AD is different from the others that I just talked about in that domain controllers are running Windows Server 2019, though the Active Directory forest functional level is still set to 2012 R2. 11:43.210 --> 11:48.650 Also, the Active Directory Recycle Bin, which enables rapid undelete capability, is not enabled, which is interesting. 11:48.770 --> 11:54.250 The default domain administrator account keeps the default name Administrator, and it's disabled. 11:54.250 --> 12:02.150 So, GCP spins up a second domain admin account called CloudSVCAdmin, and this account is enabled. 12:02.610 --> 12:07.450 The domain password policy is default, and the customer can create fine-grained password policies. 12:07.730 --> 12:14.090 And with this, the DC event auditing is not configured through group policy, so again, I couldn't identify how it may be configured. 12:15.130 --> 12:18.030 GCP-managed AD has several delegation groups. 12:18.030 --> 12:22.650 They start with Cloud Service, and these are stored in the Cloud Service Objects OU. 12:22.650 --> 12:28.090 The Cloud Service All Administrators and the Cloud Service Administrators are the primary customer admin groups. 12:28.270 --> 12:33.330 GCP delegation groups include computer admins, managed service account admins, and others. 12:33.850 --> 12:38.850 This slide captures some of the common themes of the three managed AD environments I reviewed. 12:38.850 --> 12:51.810 I used a version of the Trimark Active Directory assessment tool that I wrote to perform the review of the different AD environments, which enabled me to capture data for all three and identify the key security features and delegation in about a day to put this talk together, 12:51.810 --> 12:53.030 or at least this part of it. 12:53.030 --> 12:59.470 The bottom of the slide will provide a link to an Active Directory security review PowerShell script that I wrote and published in June. 12:59.650 --> 13:10.450 The link points to the article where I highlight key AD security items to check and what to check for, and the PowerShell script simplifies the data collection part, so analysis is much simpler. 13:10.570 --> 13:11.890 So it's good to check that out. 13:11.890 --> 13:15.470 It would be helpful if you're looking at an environment such as these. 13:16.850 --> 13:24.210 So since the customer does not actually have AD admin rights to a managed AD environment, there's not much point in attempting to escalate to DA. 13:24.630 --> 13:32.230 Better to identify which managed AD environment it is and enumerate the managed AD privilege groups to determine which accounts are highly privileged. 13:32.550 --> 13:38.770 Interesting note about Azure Active Directory domain services, Microsoft's managed AD. 13:38.770 --> 13:49.810 It can be managed by Azure AD accounts, and these are synchronized into Azure AD DS or even by on-prem AD accounts that are synced in from the on-prem AD environment. 13:49.810 --> 13:57.330 And these are synced using the on-prem Azure AD Connect system through Azure AD to Azure AD domain services. 13:57.330 --> 14:03.720 And if password hash sync is enabled, which it usually is, then the on-prem AD account password hash is included. 14:03.990 --> 14:21.470 Normally, password hash sync does a hash of a hash, but when syncing through to Azure AD domain services, that original AD password hash is retained and synchronized, so that way the password can actually be used in Azure Active Directory domain services. 14:23.690 --> 14:27.310 So, there are several cloud components that connect on-prem to cloud. 14:27.310 --> 14:30.150 This connectivity has some very interesting implications. 14:31.250 --> 14:35.850 Amazon's AD Connector is effectively an on-prem AD authentication proxy. 14:36.210 --> 14:42.290 AD Connector is designed to provide an easy way to establish a trusted relationship between your Active Directory and AWS. 14:42.310 --> 15:03.170 So, when AD Connector is configured, it enables the ability to sign into AWS applications with on-prem AD credentials, join AWS Windows instances to the on-prem AD domain, and provides federated sign-in to the AWS Management Console by mapping these on-prem identities to AWS IAM roles. 15:04.810 --> 15:09.310 And AD Connector can't be used with custom applications since it's just for AWS. 15:09.750 --> 15:14.890 Basically, a user opens the secure custom sign-in page, supplies their Active Directory username and password. 15:14.890 --> 15:22.170 This authentication request is sent over SSL to the AD Connector, and then the AD Connector performs LDAP authentication to Active Directory. 15:22.170 --> 15:25.660 The interesting part happens at this point. 15:25.910 --> 15:33.580 Once the user has been authenticated, the AD Connector calls the STS assume role method to get temporary security credentials for that user. 15:33.850 --> 15:40.560 Using those temporary security credentials, AD Connector constructs a sign-in URL that users use to access the console. 15:40.870 --> 15:47.110 And if a user is mapped to multiple roles, the user will be presented with a choice at the sign-in time as to which role they want to assume. 15:47.110 --> 15:51.130 User session is valid for an hour, which is pretty standard for cloud apps. 15:51.730 --> 16:03.410 Since the AD Connector environment is hosted in AWS, this would need to be targeted in order to potentially impersonate the users and be able to leverage that STS assume role method. 16:05.770 --> 16:26.230 Microsoft's pass-through authentication is similar because with Microsoft pass-through authentication, or PTA, it provides users the ability to authenticate to Azure Active Directory, which is passed through to the on-prem AD and validated in Azure AD by getting the status back from the PTA agent in the on-prem environment. 16:27.030 --> 16:30.010 PTA is enabled via Azure AD Connect. 16:30.010 --> 16:35.130 The users use the same passwords to sign into both on-prem and cloud-based applications. 16:35.310 --> 16:44.050 The communication between an agent and Azure AD is secured using certificate-based authentication, and these certificates are automatically renewed every few months by Azure AD. 16:44.230 --> 16:49.050 I couldn't figure out what happens if you get one of these certificates, but that could be pretty interesting. 16:50.370 --> 16:59.370 But ultimately, this supports user sign-in into all web browser-based applications and into Microsoft Office client applications that use modern authentication. 17:02.860 --> 17:16.800 Adam Chester, aka XPN, has done a great blog post on how to leverage a compromised Azure AD Connect server with PTA enabled in order to gather usernames and cleartext passwords that were used to authenticate via PTA. 17:16.800 --> 17:21.940 So since password authentication is managed by Azure AD Connect, you can compromise that server. 17:21.940 --> 17:29.500 And then during this authentication process, Azure AD sends the cleartext password, not hashed, to authenticate the user. 17:29.500 --> 17:31.380 So it's sent to Azure AD Connect. 17:31.380 --> 17:42.180 Azure AD Connect PTA component is going to check and validate the username and password with Active Directory, then send back the status to Azure AD for that. 17:42.340 --> 17:52.680 So Adam identified that you could inject a DLL to compromise the credentials that were used for PTA and then have those credentials for anyone that was using PTA. 17:53.780 --> 18:03.320 Azure Active Directory Seamless Single Sign-On, or Azure AD Seamless SSO, automatically signs users in when they are on their corporate devices connected to the corporate network. 18:04.040 --> 18:09.300 This graphic is interesting and nice, but I prefer this one because it has a lot more of the flow. 18:09.380 --> 18:17.100 But basically what ends up happening is that users don't need to type in their passwords to sign into Azure AD or even really type in their usernames. 18:17.100 --> 18:24.320 It provides the users easy access to their cloud-based applications without any additional on-prem components like federation. 18:24.320 --> 18:30.360 This is done by converting the on-prem authentication to cloud authentication through a computer account in the Active Directory. 18:30.800 --> 18:49.860 So Michael Grafner from DS Internals posted this article a few years ago, or a couple of years ago, where he was talking about how the Azure AD has a publicly available endpoint that accepts these Kerberos tickets and translates them into SAML and JWT tokens that can be leveraged for cloud apps. 18:51.320 --> 18:57.480 Effectively, if you can compromise the Azure AD Seamless Single Sign-On computer account, the password has to be associated with it. 18:57.480 --> 19:04.860 The computer account is called Azure AD SSO ACC and can be identified as in the computer's container. 19:04.860 --> 19:18.140 If you're able to compromise this and get this password hash, you can generate a silver ticket for any user you want to impersonate, and the service that you use in the silver ticket is going to be aadg.windows.net.nsatc.net. 19:18.140 --> 19:27.440 You inject this ticket into the local Kerberos cache, and Azure AD Seamless Single Sign-On, the computer account password doesn't actually change. 19:27.440 --> 19:31.700 Microsoft does recommend that it gets rolled and changed at least every 30 days. 19:31.700 --> 19:43.070 What's interesting is because it's a computer account, there is no actual computer that is behind the scenes that is using that computer account other than the Azure AD Connect client. 19:43.480 --> 19:50.400 So normally a computer in Active Directory would update the computer account password that's associated with that computer. 19:50.400 --> 19:54.160 Since there isn't really a computer here, that password isn't getting changed. 19:54.160 --> 20:00.460 So maybe in the future Microsoft will code into Azure AD Connect to automatically update the computer account password for this. 20:02.000 --> 20:06.920 Three years ago at DEF CON 25 in Las Vegas, I co-presented a talk called Hacking the Cloud. 20:06.920 --> 20:24.620 I talked about how Azure AD Connect's password hash sync permissions provided Mimikatz DC-sync rights, and that the author of Mimikatz DC-sync, Vincent Leteau, was inspired by password hash sync to create DC-sync because PHS requests password hashes for sync users on a regular basis. 20:25.520 --> 20:31.020 We can scan Active Directory domain permissions looking for potential Azure AD Connect service counts. 20:31.720 --> 20:42.080 Or if the Azure AD Connect install would use the express install option, then there's a helpful description on the account noting on which server Azure AD Connect is installed. 20:42.180 --> 20:46.620 We can then check in Active Directory to figure out where that computer is stored. 20:46.620 --> 20:50.040 Very often this is in the server's OU or somewhere similar. 20:50.260 --> 20:58.160 Once we know the server name, we can check for, and its location, the actual distinguished name of the OU that it's in. 20:58.160 --> 21:02.960 We can check AD for any group policies that add AD groups to the local administrators group. 21:03.080 --> 21:19.320 Compromising any account that is a member of the local administrators group on the Azure AD Connect server would result in compromise of the Azure AD Connect server and ultimately Active Directory, assuming that it has password hash synchronization rights configured in the environment. 21:20.100 --> 21:26.860 Also, if we can modify a group policy that links to the Azure AD Connect server OU, then we can compromise the server that way as well. 21:28.240 --> 21:33.440 When talking about cloud environments, it's important to mention how administration is performed. 21:33.440 --> 21:36.920 This is commonly referred to as Identity Access Management, or IAM. 21:37.480 --> 21:45.380 So we've gone from groups in our on-prem environment with Active Directory to roles, which is effectively these rights that are packaged together. 21:45.740 --> 21:51.220 Azure and AWS and GCP have their own methods, but a lot of the concepts are effectively the same. 21:51.860 --> 21:55.380 Azure IAM uses the concept of an owner, a contributor, and a reader. 21:56.860 --> 22:05.800 The owner has full access, the contributor has the ability to create and manage the resources but can't grant access to others, and then a reader can just view things. 22:06.900 --> 22:15.940 And so Azure has the tenant, which is the top-level container of the Azure environment, and this contains subscriptions. 22:17.660 --> 22:25.160 The tenant admins are effectively the owner role on the tenant, and subscription admin would be effectively the owner role on the subscription. 22:28.050 --> 22:38.130 What's interesting is that Microsoft notes that Azure roles and Azure AD roles are separate and different, mostly, and I'll cover that in a little bit. 22:41.620 --> 22:49.720 AWS has a root account, often called or referred to as a payer account, which is the primary account, though it really shouldn't be used regularly. 22:49.720 --> 22:51.560 Then there are account admins. 22:51.560 --> 22:55.160 So in AWS, the account is similar to the Azure tenant. 22:55.160 --> 22:58.980 It's the top-level container for the environment for that customer. 22:58.980 --> 23:01.940 The account admins have full control over the account. 23:01.960 --> 23:07.720 If an account is a root account and an admin account, then that account has full organizational control. 23:07.940 --> 23:12.440 It's important to note here that any overscoped roles provide escalation capability. 23:12.920 --> 23:24.640 And Rhino Security Labs has an extensive article covering 21 AWS IAM privilege escalation methods, which allow an attacker to escalate from a compromised low-privilege account to full admin privileges. 23:24.640 --> 23:35.100 So I've listed a bunch of them here, I apologize for the small text, but really what you want to do is you want to read the blog article that they wrote, which is linked at the bottom and in the slides. 23:35.160 --> 23:42.800 They also released a scanning tool, which is on GitHub, to identify these vulnerabilities in an AWS user account. 23:42.860 --> 23:49.820 And if you have an account with IAM read access for all users, the script can be run against every user in the account to detect these vulnerabilities account-wide. 23:49.820 --> 24:04.820 So if you're asked to pen test AWS, or you're providing security consulting services around AWS, or even you work in an organization and you're moving to AWS and you have some of the IAM roles configured, you definitely want to scan them for potential escalation methods. 24:06.020 --> 24:09.240 API keys, I'm just going to cover this really quickly. 24:09.240 --> 24:13.400 They provide permanent access to an environment, often with extensive privileges. 24:13.500 --> 24:20.280 These keys can often bypass the typical username and password used to authenticate as an account if it's configured as such. 24:21.680 --> 24:26.780 Well, I've covered a number of scenarios on how to attack on-prem and hybrid cloud components. 24:26.780 --> 24:34.120 Let's focus on on-prem Active Directory domain controllers hosted in cloud infrastructure as a service. 24:34.120 --> 24:37.420 So first up is Amazon AWS EC2. 24:38.160 --> 24:44.460 Many organizations leverage Amazon AWS EC2 or IaaS as a cloud data center. 24:44.460 --> 24:49.220 Some of these host on-prem AD domain controllers in AWS. 24:49.820 --> 24:54.900 So we start with Acme Corporation, which has the on-prem AD environment. 24:54.900 --> 25:01.100 They have stood up an Amazon AWS account to use EC2 as a cloud data center. 25:01.620 --> 25:09.960 Then Acme configures an IAM role for the server admins and they stand up a couple of on-prem Active Directory domain controllers in AWS. 25:11.160 --> 25:18.100 Acme then configures Federation so that users in the on-prem AD can authenticate to AWS using their AD credentials. 25:18.540 --> 25:27.180 To simplify access for server admins, the on-prem AD group AWS EC2 admins is created and then that's associated with AWS IAM role. 25:27.280 --> 25:37.480 The server team accounts are then added to the on-prem AD group called AWS EC2 admins to enable the team to easily administer the AWS environment. 25:39.200 --> 25:52.720 But if an attacker is able to compromise one of the user accounts that's a member of the AD group AWS admins, then they are able to connect to the AWS console as an AWS admin. 25:53.260 --> 25:59.620 And then now the attacker can configure the virtual domain controller instance in AWS and run commands on it. 25:59.640 --> 26:10.300 Once the command runs to modify the domain controller's copy of AD, the changes replicate back to the on-prem domain controllers and then we end up with the on-prem Active Directory being compromised. 26:10.520 --> 26:19.780 So effectively in this scenario, the attacker can compromise an account a regular user account that doesn't have any Active Directory rights. 26:19.780 --> 26:22.400 It's just a member of a group in Active Directory. 26:22.400 --> 26:28.080 This group is associated with the AWS IAM role through federation. 26:28.120 --> 26:45.060 And so once the attacker compromises this account, which has these rights granted in the group in Active Directory, but then are relayed and related to this AWS IAM role, then they can leverage that in order to compromise any of the systems that are in there that they have rights to. 26:45.720 --> 26:50.560 So I've got a summary slide here that covers this very briefly. 26:50.860 --> 26:58.720 But keep in mind that Amazon SSM, which is effectively the Amazon agent, is installed by default on most Amazon-provided instances or templates. 26:58.720 --> 27:05.880 You do need the role to execute, but once you have the rights over that environment as a server admin would, you can normally go ahead and get that. 27:06.020 --> 27:08.440 Hopefully you're logging this and looking at the logs. 27:08.440 --> 27:10.960 Amazon CloudTrail can pull these logs. 27:10.960 --> 27:14.400 And the logs need to be configured to not be deleted. 27:14.640 --> 27:19.420 So very often an attacker is just going to delete the logs so you can't see what they did. 27:20.540 --> 27:25.700 For most of 2019, I was digging in Office 365 and Azure Active Directory looking at features. 27:25.700 --> 27:29.020 As I went through each of them, I found one that was very interesting. 27:30.160 --> 27:33.920 Remember earlier how I noted that Azure roles and Azure AD roles are separate? 27:33.920 --> 27:35.400 Well, not exactly. 27:35.600 --> 27:42.320 Here it describes a way that global administrators could actually elevate their access and have some control over Azure. 27:42.960 --> 27:46.500 Surely the global administrator role description notes this, right? 27:47.020 --> 27:50.680 Well, no, it didn't, at least not as of May. 27:51.120 --> 27:57.520 The global administrator role provides full admin rights to Azure AD and ultimately all the Office 365 services. 27:57.520 --> 28:04.300 The Microsoft Online document provides key information about this, and there's nothing here showing Azure capability. 28:04.300 --> 28:06.020 I took this screenshot in May. 28:06.560 --> 28:20.400 After contacting Microsoft and providing information about this escalation path, the document's been updated in the past month or so, stating that, yes, there is a way for a global administrator to have Azure rights, which is a bit different than what it said before. 28:21.740 --> 28:32.940 Once we have access to the Azure AD portal, which is typically all Azure AD users by default, we can view several different configuration settings for Azure Active Directory, which controls many aspects of Office 365. 28:33.080 --> 28:37.640 This page shows the directory properties, which now includes the new managed security defaults. 28:37.640 --> 28:41.800 Towards the bottom is access management for Azure resources toggle. 28:41.800 --> 28:43.620 So we can see what that is. 28:43.980 --> 28:54.480 According to Microsoft documentation, toggling this option from no to yes adds the global administrator account to the user access administrator role in Azure RBAC at the root scope. 28:54.760 --> 28:57.420 This has control over all subscriptions in the tenant. 28:57.420 --> 29:01.600 And this option is only available to accounts that are members of the global administrator role. 29:01.600 --> 29:06.880 While this option is configured in the directory properties section, this is actually a per account configuration option. 29:06.880 --> 29:13.960 So this magic button provides ability to manage Azure roles, but no direct Azure rights as into VMs. 29:13.960 --> 29:20.420 Further in the documents documentation, there's additional context on what elevate access is and how it works. 29:20.420 --> 29:28.860 When developing Azure Active Directory in Azure, Microsoft had to decide if the global admin should have Azure rights initially, and opted for no, they shouldn't. 29:28.860 --> 29:32.120 But they provided this capability to gain Azure access. 29:32.120 --> 29:39.140 This is that backdoor of sorts, which has been somewhat highlighted now that the managed security default option is right below it. 29:40.320 --> 29:44.000 This graphic shows what happens in the connection point between Azure AD and Azure. 29:44.000 --> 29:52.240 My biggest concern here is that for many organizations, the group that manages Azure Active Directory and Office 365 are often a different group from those that manage Azure. 29:52.240 --> 29:57.720 This means that someone could elevate access, think a rogue admin, and no one probably would notice. 29:57.720 --> 30:04.180 This is potentially a serious threat from an insider threat perspective, especially around the detection part of this issue, which I'll cover in a bit. 30:05.040 --> 30:09.000 This note in the Microsoft document highlights an interesting phenomena with this option. 30:09.000 --> 30:14.860 Only while the account is a member of the global administrator role can the account toggle this option to yes or no. 30:14.860 --> 30:24.420 This means that if an account is no longer in the global administrator role, and the option has been set to yes, they cannot change it back to no, and no one else can change it either. 30:25.180 --> 30:31.760 I also found an API that seems related, which means that an attacker would not need to visit the Azure AD portal to perform this action. 30:31.760 --> 30:35.100 They just have to run this API call. 30:35.900 --> 30:41.000 And Ryan added this REST API elevation method to PowerAzure, which you should check out. 30:41.000 --> 30:43.140 I have a link at the bottom of the slide. 30:44.960 --> 30:53.100 Now, what I'm going to do is walk through a scenario where we have Acme, and Acme has stood up. 30:53.260 --> 30:56.080 They have a new Office 365 environment. 30:56.080 --> 30:58.100 They've had Azure for a very long time. 30:58.100 --> 31:00.620 They're using Azure as their cloud data center. 31:00.620 --> 31:01.740 They've got domain controllers there. 31:01.740 --> 31:03.880 They've got a bunch of different servers in Azure. 31:04.100 --> 31:06.480 But they're just now spinning up Office 365. 31:06.480 --> 31:07.560 They're going to use it for email. 31:07.560 --> 31:09.260 They're going to use it for a number of different things. 31:09.260 --> 31:20.280 So they have members of the Active Directory team, Exchange, SharePoint, a bunch of different people, probably 30 to 50 people are members of the global administrator role. 31:20.280 --> 31:31.300 Not everyone has had their account configured for MFA, which is realistic since many admins, this is from last year's Black Hat talk that I gave with Mark Morrow, and it was only about 8%. 31:31.860 --> 31:36.660 Hopefully, it's much higher than that, but it's still probably less than 15% if I had to guess. 31:36.820 --> 31:40.640 And so this means that most global admin accounts don't have MFA configured. 31:40.640 --> 31:46.800 And so if an attacker is able to password spray and guess the account password, then they're able to take over this account. 31:48.320 --> 31:50.440 So by default, this option is set to no. 31:50.440 --> 32:08.020 So if an attacker compromises a global admin account, they can go in and actually toggle this from no to yes, which gives them Azure User Access Administrator role rights at the root of Azure, that Azure tenant that's configured on the Azure side. 32:08.160 --> 32:15.460 So once that toggle option is set to yes, we can see a new account appears in the User Access Administrator Azure RBAC role. 32:15.460 --> 32:18.820 I call this attack account hacker, so I don't forget which one I'm using. 32:18.820 --> 32:23.020 But you'll notice that I also added a couple others that look like they belong. 32:23.080 --> 32:24.940 There's really nothing that's supposed to be in this role. 32:32.100 --> 32:37.300 Monitoring for this can be a challenge since the primary method is running an Azure CLI command at the root scope. 32:37.540 --> 32:44.500 I can't find root level Azure RBAC roles listed in the portal, so I can't go there to see what the membership is. 32:44.620 --> 32:54.560 So something that could be monitored through an API or if you're pulling logs into your sim, you could alert on RBAC role or role membership modification. 32:54.920 --> 33:00.360 But when I run that command, I get this output, which means I would have to parse it. 33:00.900 --> 33:05.920 And so this is the message that appears when attempting to remove root level role assignments from a subscription. 33:05.920 --> 33:11.220 Removing an account from a root level Azure RBAC role requires running an Azure PowerShell command. 33:11.220 --> 33:17.080 When an account toggles elevate access from yes to no, it is automatically removed from the User Access Administrator. 33:17.760 --> 33:24.860 Once the attacker has User Access Administrator rights at the root level, this account can modify roles for any subscription in Azure. 33:24.900 --> 33:27.300 Owner's obvious and probably monitored, maybe. 33:27.300 --> 33:29.180 What about something more innocuous? 33:29.660 --> 33:41.540 So I found Virtual Machine Contributor, and this is pretty interesting because the documentation states it lets you manage virtual machines, but not access to them and not the virtual network or storage account that they're connected to. 33:41.820 --> 33:48.480 And it includes this ability, which is Microsoft.compute slash virtual machine slash run command. 33:48.480 --> 33:53.460 This provides the ability to run commands on the VM as system such as PowerShell. 33:54.200 --> 33:57.860 And so we can click on run PowerShell script and we can do that. 33:57.860 --> 34:06.780 The other thing that's interesting here is that it provides the ability to re-enable the administrator account, which on a domain controller in Azure would be the RID 500 account for the domain. 34:06.800 --> 34:19.980 So we have seen some environments when we look at them from an Active Directory assessment perspective, they have had the default domain administrator account disabled, but very often it has an old password, say 10 years old or more. 34:19.980 --> 34:26.200 And it may actually have a Kerberos service principal name or spin associated with it, which means that we can Kerberos it. 34:26.200 --> 34:35.860 So if an attacker could just go ahead and Kerberos it, then re-enable it through here, they could potentially leverage that default admin account, which was originally disabled. 34:37.560 --> 34:42.160 So we're going to add the attacker controlled account to the virtual machine contributor. 34:43.800 --> 34:48.320 And now that our attacker has the ability to run commands, we can update the administrators group. 34:48.320 --> 34:52.980 This will happen on the Azure based domain controller then replicate to the domain controllers that are on-prem. 34:53.060 --> 35:02.080 Note that being able to run commands on an Azure VM is not specific to customer on-prem DCs hosted on Azure, but also other systems hosted there as well. 35:02.620 --> 35:04.500 So we take a look to see what would happen. 35:04.500 --> 35:10.100 We can go switch to a computer on-prem or run another command on the DC to see what the result is. 35:10.100 --> 35:14.200 And we've now compromised the on-prem active directory from Azure. 35:14.980 --> 35:23.560 If PowerShell logging is configured appropriately and sent to a sim and monitored, this event should identify unusual activity showing up in the PowerShell log. 35:24.460 --> 35:33.100 Another option an attacker has with these rights is running invoke mimikatz from the internet or even a compromise system on the corporate network or within the Azure tenant. 35:33.420 --> 35:47.220 So here I'm running mimikatz just to get the curb CD password hash because with that I can create golden tickets, which means I can persist forever in the on-prem active directory even though I've never touched that or had access to the corporate network. 35:47.220 --> 35:51.600 So that means once I get a foothold, I can have domain admin possibly forever. 35:52.200 --> 35:57.720 But what if instead of running invoke mimikatz directly from GitHub, we loaded it into Azure Blob Storage? 35:57.720 --> 36:00.040 So effectively, we're going to use a PowerShell script. 36:00.040 --> 36:01.060 We're going to upload. 36:01.060 --> 36:02.840 I'm not going to call it invoke mimikatz. 36:02.840 --> 36:05.680 I'll call it invmmk.txt. 36:05.760 --> 36:13.800 And Azure Blob Storage is very nice because it gives me a URL in which I can interact and connect to this file. 36:14.420 --> 36:16.080 So that would work as well. 36:16.700 --> 36:18.440 Let's take a moment to recap. 36:18.440 --> 36:25.460 From global admin to Azure user access administrator to Azure admin or virtual machine contributor. 36:25.500 --> 36:37.240 So if an attacker compromises an organization's global administrator account, either because they're just starting with Office 365 or didn't realize the risks around protecting global admins, either way, the global admin accounts weren't locked down with PIM, 36:37.240 --> 36:38.700 conditional access, or MFA. 36:38.700 --> 36:47.760 Or a global admin session token was stolen because this admin was using their regular web browser on the regular user workstation, which was compromised. 36:47.760 --> 37:02.880 This would mean that the attacker was about an hour to perform these actions, compromise the account, elevate access to Azure, get access to get Azure rights through Azure role membership, remove elevate access rights, perform malicious actions on any or all Azure VMs and all subscriptions, 37:02.880 --> 37:05.620 then remove role membership in Azure or not. 37:05.620 --> 37:09.900 The total time required only a few minutes, perhaps as much as 15 minutes total. 37:10.900 --> 37:15.840 In many organizations, there are two different groups that manage Azure AD, Office 365, and Azure. 37:15.840 --> 37:22.640 As I mentioned earlier, they have no expectations that the other team has the ability to affect their environment. 37:23.820 --> 37:26.340 So why is this issue important? 37:26.340 --> 37:34.020 Well, customers don't usually expect that the global administrator role that says it has Office 365 admin rights can also control Azure subscriptions. 37:35.380 --> 37:44.020 We need to be very careful what accounts are members of roles in Azure at the root level, as well as on the subscriptions, since this impacts the security posture. 37:44.020 --> 37:56.240 It's not obvious that removing the account from global administrator role does not remove the account from the Azure RBAC role, user access administrator, when the access management for Azure resources toggle option is still set to yes. 37:59.100 --> 38:04.300 And the key points around detection is there's no way to detect this configuration in Azure Active Directory at all. 38:04.300 --> 38:06.380 There's no property to query on accounts. 38:06.380 --> 38:11.200 You have to run an Azure CLI command to check the role group membership, but that's on the Azure side. 38:11.320 --> 38:18.500 When I walked through the Azure AD to Azure access elevation, I attempted to identify a clear event that I could alert on and was unable to. 38:18.660 --> 38:24.180 The core directory, directory management set company information log notes that something has changed, but not what. 38:24.180 --> 38:28.440 The capability for an Azure AD global admin is quote unquote by design. 38:28.440 --> 38:34.560 However, not expected, not well documented by Microsoft when reading about what Azure AD global admins can do. 38:34.560 --> 38:43.420 Certainly now the documentation has been updated to mention this, but I feel like there should be better logging and alerting around this sort of configuration. 38:44.620 --> 38:56.400 I'm concerned that this is a scenario where the Microsoft cloud customer can't detect, can't remediate and ultimately can't prevent since there's no real gate to lock if an Azure AD global admin account is exposed. 38:56.420 --> 38:58.560 And then they can jump over on the Azure side. 38:58.600 --> 39:08.380 So the only real good answer I can find for this is to ensure that global administrator groups are managed or their roles are managed with Azure AD PIM. 39:08.420 --> 39:12.280 But even this doesn't stop a rogue admin from doing whatever they want. 39:12.680 --> 39:15.960 The best mitigation is to prevent compromise of global admin accounts. 39:15.960 --> 39:18.200 Using PIM reduces this risk tremendously. 39:18.220 --> 39:24.500 Outside of that, monitor global admin role membership in Azure AD and the user access administrator group at the Azure root level. 39:24.620 --> 39:27.740 Regarding sensitive systems, severely restrict who has access to them. 39:27.740 --> 39:33.580 Placing them in a separate subscription will protect against common over-permissioning, but not against a sort of attack. 39:33.660 --> 39:39.880 So leveraging a separate Azure tenant is the best method for isolating and securing sensitive systems, including domain controllers. 39:41.040 --> 39:49.560 I reported this issue to Microsoft security in September of 2019, continued corresponding with them, providing additional details through the rest of the year. 39:49.560 --> 39:58.800 Again, my biggest concern with this is that on the Azure AD side, there is no way to detect, remediate, or prevent a global admin from flipping the switch and gaining full Azure rights. 40:00.080 --> 40:07.440 Instead of an external attacker, what if there was a malicious insider, someone who did not benefit from the company's spared no expense mantra? 40:08.200 --> 40:10.420 And we know what happened in that scenario. 40:11.280 --> 40:14.140 But how bad could it really get? 40:14.200 --> 40:15.760 Let's play what if. 40:15.760 --> 40:19.640 What if an attacker takes control of Azure resources? 40:19.980 --> 40:23.900 What if the attacker then removes all the accounts from all the roles? 40:24.160 --> 40:27.040 And what if they decide to ransom the Azure environment? 40:27.040 --> 40:28.240 Azure ransomware? 40:28.300 --> 40:29.420 Azureware? 40:30.020 --> 40:31.720 Yeah, that could be pretty bad. 40:32.680 --> 40:37.920 So let's take the cloud architecture concepts to the next level, which is pretty much where organizations are headed. 40:38.200 --> 40:45.220 Let's say you start with the typical Microsoft server architecture with on-prem Active Directory, and then we're going to add Azure to it. 40:45.680 --> 40:48.640 And then we're going to add AWS to the mix. 40:48.860 --> 40:51.460 And then we're going to add Google Cloud Platform. 40:51.820 --> 40:57.540 And so I'm just going to provide a simplified view of this because really the architecture diagrams get pretty messy. 40:58.400 --> 41:02.100 We have AWS, we have Azure, and we have Google Cloud Platform. 41:02.460 --> 41:08.840 And I simplify this, like I said, showing the on-prem data center and federation to the three different cloud environments. 41:08.840 --> 41:16.880 But ultimately this federation just means that we have this sort of configuration where everything can talk to everything else and that they're interconnected. 41:17.580 --> 41:22.220 And this means a compromise in one will likely lead to a compromise in all cloud environments. 41:22.220 --> 41:29.180 Since the IAM story is quite different in each and difficult to track roles and privileges across them, it's likely the accounts are over-permissioned in this scenario. 41:29.340 --> 41:36.120 And the use of federation provides the ability to jump across between these different environments. 41:37.000 --> 41:44.460 So a compromise of all three cloud environments will likely result in on-prem compromise since there's likely on-prem AD domain controllers in one or more of the cloud environments. 41:44.460 --> 41:51.020 Federation or any type of authentication flow, really, between these environments enables a new type of lateral movement. 41:51.320 --> 42:07.080 The attacker could leverage over-privileged Azure and AWS roles and the rights provided with them and showing which ones can provide system access to VMs, despite how they're described, is one way that we can identify and lock down those controls or those roles. 42:08.020 --> 42:16.840 But the attacker could jump across the federation to gain privileged access to cloud-hosted services and systems once they compromise a single system or even an account. 42:17.280 --> 42:29.200 Because ultimately, gaining privileged access to the IaaS service provider, so Azure, Amazon, AWS, GCP, means that the attacker could do anything with the VMs hosted in the environment, including accessing and modifying data on the systems. 42:29.200 --> 42:33.480 So it's very important that administrative roles need to be limited and gated. 42:34.800 --> 42:40.520 I recently heard a customer tell me that an executive was concerned about putting all their eggs in one basket. 42:40.940 --> 42:48.040 So they signed up for Azure and AWS and Google Cloud Platform, so now their eggs are in all the baskets. 42:48.360 --> 42:52.380 And the security capabilities and controls across these environments is quite different. 42:52.560 --> 42:55.940 And it's up to the operation and security teams to figure that part out. 42:58.060 --> 43:03.080 So the cloud provides many benefits and power, though cloud integration often complicates security. 43:03.120 --> 43:07.600 Organizations will have to do better to map security between on-prem and cloud systems and authentication. 43:07.880 --> 43:13.060 It's important as a security professional or security hobbyist to learn about these things. 43:13.060 --> 43:15.620 The cloud is the Wild West at this point. 43:15.620 --> 43:17.560 It is the whole brave new world. 43:17.720 --> 43:22.380 If you want to go far in your career, focus on cloud and cloud security because it is wide open. 43:22.380 --> 43:32.000 There's a lot of areas that haven't been explored, and especially in those areas across on-prem and the hybrid environment. 43:32.080 --> 43:39.840 So there's a lot of work that needs to be done there, and there's a lot that we need to do moving forward to improve the security in this environment. 43:40.200 --> 43:41.400 That's been my time. 43:41.400 --> 43:42.780 Thank you very much for yours.