All righty. Well, good morning, good afternoon, or good evening, wherever you guys may be today. I am Jura. Stand with me. I've got Patrick. And we are here to talk with Sean Metcalf about his talk, Hacking the Hybrid Cloud. We've both taken some time to watch this. Hopefully you guys have too. I thought it was an excellent talk, incredibly comprehensive, and did a great job of kind of giving everybody, even if you don't have a lot of knowledge about Active Directory or the way that some of these deployments work, of kind of bringing you up to speed. So if you are kind of joining and are looking for a talk to watch and you haven't caught this one yet, make sure you go back and check that out. Sean, why don't you take a moment to introduce yourself. Tell us where you are, how you kind of got into this, and how we got to where we are today. Sure thing. I'm Sean Metcalf, Pyrotek3 on Twitter. Hello everyone. I've spoken at a few DEF CONs in the past. I really enjoy the security research element of technology and IT. I've been working with Active Directory since pretty much when it came out in the year 2000. First as operations admin, engineer, designer, architect, what have you. And then pivoted pretty hard into the security aspect in say 2003, 2004. But I'd say probably the interesting thing to me was looking at the security aspects of these systems along the way. A lot of times you have people that are administrators and engineers and they understand the security elements of it. They understand the challenges, but they just don't necessarily have the backing to push in that direction. So certainly for me, since I was coming up through the Microsoft area, I didn't feel like the security space or InfoSec was really a space for me because I had worked with Microsoft Windows and most of the security people in 2000 to 2010 and certainly beyond are like, oh, Microsoft Windows, that's a toy OS. What do you know about security in that area? So after I had been working with Active Directory for a long time and I started seeing these attacks go from hypothetical to reality, I said, hey, there's really something here. And I started putting together some talks to help people better understand Active Directory security. And then in 2017, I did a talk with Taya about hacking the cloud and what that actually means. And at the time, Taya was working on the Azure Red team. And so I learned a lot from Taya about how these things interconnect. Certainly looked a lot at Azure AD and Office 365 at that time. As I was continuing that journey, in the past year, I identified some issues or concerns that I had with hybrid cloud. So these connection points between your on-prem Active Directory and Azure AD or Office 365 or Azure and beyond. And certainly Dirk-Jan Milema, he did a great talk last year at DEF CON and has been publishing some great information about doing just that from a comprehensive attack perspective. So certainly the research of others have been inspiring my research recently and things that I've been looking at. Excellent. Sean, so I guess also just for starters, can you kind of like explain in a nutshell what like the hybrid cloud... I think people kind of understand the cloud and people understand on-prem. Can you kind of talk a little bit about what you mean with the hybrid cloud? Sure thing. So it's good to have a good definition or it's good to have a solid definition of what it actually is because, like you said, it can be confusing. So for me, at least the definition I put together for this talk is hybrid cloud is when you have on-premises infrastructure. So you have something like Active Directory and then you have something in the cloud. It could be you've extended your on-prem environment into cloud IaaS infrastructure as a service. So you're using the cloud sort of as an extension of your data center. A lot of organizations are moving into the cloud and using the Azure or AWS or even GCP as their data center now. So that's one part of it. Another part of it is when you have all your on-prem infrastructure that's handling your typical things like Active Directory logons, things like that. But then you have services that are up in the cloud. So your SaaS side, so software as a service. You have things like Salesforce or you have Workday or you have Office 365 or things along those lines where you have some connection points. And so certainly from the Office 365 on-prem AD side, there's a connection point called Azure Active Directory Connect. And there's some interesting things that have occurred with that connection point between your on-prem and that cloud environment. I pointed out some back in 2017, Dirk Jans published a whole bunch of information about that. Others like Adam Chesters has identified some things where the additional connection points of what Microsoft's been doing to simplify this authentication flow between on-prem and the cloud. And I think that's really one of the things that I wanted to dive into as part of my talk was discussing the roles and how they're typically over permission and also looking at this authentication flow that gets very complicated. And that's why I closed out the talk with a bunch of different slides where you have three different cloud environments and you have your on-prem environment and you're working through the federation of it and how they all get interconnected in very interesting ways. The IAM story there, the Identity Access Management story is very complicated when you have multiple cloud environments. And it gets difficult to track what all these roles are. And typically they get over permission just like they do on-prem. So the other thing I wanted to establish as a narrative of my talk was that we have virtualization on-prem. We're using VMware primarily, predominantly. And then there's also Hyper-V to some extent. And then in the cloud, we have effectively virtualization writ large. Be it Amazon AWS, which started with Zen, uses more of Amazon's Nitro now, to Azure leveraging a custom version Hyper-V core, to GCP using Zen to straight up in their environment. So this concept of virtualization and the way that you attack or could attack virtualization on-prem is very similar to how you can do so in the cloud. And I think that was an important part of what I was talking about. And I even mentioned how you could attack on-prem AD domain controllers, as well as those on-prem AD domain controllers hosted in the cloud. So regardless of where they are, they're vulnerable. And then the attacks across those different platforms can have some similarities, but also some pretty significant differences. Excellent. Also, is there really any kind of difference or what should people think about if they have their AD environment in Azure or if they have it as an on-prem AD environment? Are there differences they should think about? Sure. So my friend and colleague Demetrius, his first question is, do you trust the cloud? Because everything starts with that trust. So we usually presume or assume that our customers trust the cloud because they already have things in it. But we still ask the question that to what degree or to what level do you trust the cloud? Because you have to have that trust. I have concerns certainly about placing on-prem Active Directory domain controllers into cloud environments, IaaS environments, regardless of what they are, because ultimately there's a significant amount of trust. With the cloud provider. The other part of that is oftentimes you'll set up your IaaS environment, AWS or Azure, where your server admins are the ones that manage that entire environment. Well, your server admins who manage that environment can run commands potentially on the domain controllers in those environments. So if an attacker can compromise one of those server admin accounts, then they can compromise your on-prem AD. I walked through for the scene of the talk with Amazon over Federation, a way that you can link your IAM role in AWS with your on-prem AD group. And then you can have an on-prem regular user that's a member of that group that gets compromised because he's a regular user. And then leverage that account in order to compromise the domain controllers hosted in AWS, just because that one account was compromised. It has that extended role access and privileges. And then because it's in the cloud, that can roll right back again and compromise the full on-prem Active Directory environment. So there's some very interesting things that happen with that. So I also talk about one interesting one from Azure AD to Azure, because of this sort of backdoor that Microsoft put into it and has talked about, but it wasn't clear that if you were a global admin in Azure Active Directory, that you could then go jump over to Azure and have full control of that environment. So I walked through that as well, because these sort of connection points are interesting, but a lot of times operations is like, let's go, let's get it done. And they're being told by the executives, we have to get into the cloud. Our number one goal right now is to get in the cloud. I mean, we've had a surge of interest and companies that have moved to the cloud. We've been very busy talking and consulting with customers that are moving to the cloud, because they're like, are we doing this right? Because operations gets pulled along and thrown in and say, you got to migrate all these systems in the cloud. And then you have security team going, I just don't know what I'm supposed to do with this. Like, what are we supposed to know? Because the thing is on-premise is well understood, right? We at least know what our egress ingress points are. At least we hope that they do. Pen testers and red keepers are good at helping to point out what those actually are. We have a decent idea of auditing and logging. And while things aren't great, at least there's some knowledge of where some of those weaker points are. In the cloud, we have to leverage whatever the cloud provider gives to us. So Azure, AWS, GCP, they have different capabilities. They have different controls. And then the nomenclature, the names for all these things are different across all of them. So one thing that may be called something in Azure, it may be called something different in AWS and something else in GCP. So what I find myself doing is saying, okay, a subnet, whatever that may be called in this cloud environment, that's what you need to do. Or let's have a zone where we segment off this one type of system from the others. And then we make sure that we can track and control what systems that can connect with it. I will say that one of the things that we have the opportunity with in the cloud now is the ability to actually implement network segmentation, which we've failed for years on prem, right? Because now we're setting everything up. We should be building these zones. A zone for domain controllers, a zone for servers, a zone for clients, a zone for other things, and then control the communication between them. There's some ways that that could be done on prem, but the cloud gives the capability and ability now to move forward in that direction. Cool. We've got some coming in from chat here. So Mugs asks, Azure services evolve quickly. Do you see AAD evolving rapidly, or is it pretty well baked at this point as a fundamental underpinning of PaaS and software as a service services, generally speaking? Right. So Azure Active Directory and the Office 365 cloud, that's kind of an extension of that, is evolving very rapidly. Microsoft keeps hiring developers and adding new features. And it's funny because Mark Morrow, who I did a talk with about the cloud last year at Black Hat, I was at Redmond with him earlier this year for an Azure AD identity event. And as Microsoft was talking about some of the new features, I looked over at him, I was like, yeah, so like Active Directory, right? He's not in his head. So I think there's going to be some more and more parity between the Azure AD side and the AD side, even though Azure AD is not Active Directory, at least not the same, because the authentication is very different, the management is very different. But when you look at the way that the features are being deployed and configured, MFA is getting baked into it. Conditional access I consider is kind of an identity-based firewall. You can control who can access from where with what sort of credentials, be it MFA or passwordless. So you have some of that control capability baked into it. So I see it just growing. And the challenge is that the cloud changes so frequently. I mean, myself trying to keep up with it, it's a lot. I mean, Microsoft sends out emails to Office 365 customers, I think every week, saying these are the new features. Azure Active Directory is growing and developing, maturing, and will continue to. I think on the IaaS side, so Azure, AWS, GCP, I think pretty much the platform is relatively stable from the perspective of hosting virtual machines, instances, whatever you want to call them. And the management of those will grow and develop. And the orchestration around that will grow and develop. The IAM, the ability to manage them will grow and develop. But I think the core hosting components are pretty static. I think that the security elements and controls will grow and develop and improve in that area as well. Excellent. So with regard to security or hardening, R0 wants to know, what's the best AD sync scenario direction for hybrid environments? Is it Azure to on-prem or on-prem to Azure? Or does it not matter? So typically, you're going to have your on-prem environment sync to a cloud environment. And last year, I identified that there were a lot of these sync tools. There's a different sync engine for just about every platform. I mentioned Azure AD Connect for the Microsoft Azure AD Office 365 component. But oftentimes, it's going to go that way because your core identity is going to be rooted in Active Directory on-prem, at least for most organizations. As organizations would go more cloud first, the challenge is that I've heard customers say, well, we're going to migrate from AD to Azure AD. And I'm like, it just doesn't work that way. So you don't have LDAP. You don't have Kerberos. You don't have NTLM for authentication. You don't have group policy. So everything's very different. So it's not going to work the same way. And there's different controls and capabilities there. So the identity is basically still going to be rooted in your on-prem AD unless you go with a large federation provider that then becomes your identity provider, your IDP for that identity for your users. At that point, then there's some other things you could do. But I think for the most part, we're going to see on-prem AD, at least for the next few years, syncing up to the cloud or some sort of weird amalgamation of combined identities. For years, there was this concept of like a metaverse sync engine where you have different identities in different places. You may have employees in your on-prem HR system, and you may have contractors in some other system. And then you have this kind of collected understanding of identity across those. I think that with federation, that gets simpler. And yet with the cloud, it gets more complex because the identities need to be managed even more so now. And it's already been talked about several times about how identity is the new vulnerability or the new attack surface for organizations. Because we've seen that when an admin identity gets compromised, then the whole environment can get compromised. And in the cloud, that gets even more complicated because you don't necessarily know what users on-prem could be mapped to other things in a cloud environment as far as what rights they have there. You can check them out in AD, and they're a member of the right groups, and they can do all these other things, but they may be configured as an admin in Workday or an admin in Salesforce. And that's not going to be clear. Of course, the same thing can be true for, we're said, for VMware on-prem or another application. But it just... that problem just gets exacerbated, so to speak, into the cloud environment as well. I'm curious, what do you think, for an organization that is kind of maybe thinking about moving towards more of a cloud strategy, do you think that, because like you said, you've got upper management kind of saying, hey, this is what we've got to do. We've got to move to the cloud, wave our hands around, say move to the cloud. Is that kind of move towards the cloud in any of these kind of different scenarios, do you that kind of move from a strictly security perspective? Is that more risky to do that than sticking with on-prem? And how much more risky do you think people are taking on by doing that? I think it's a balancing act, honestly, and it's a tough one. So the issue is that when this train, this cloud train is moving and operations and security in an organization just kind of has to get in line and just get on board with the cloud train, the challenge is that there's a knowledge gap. And with the cloud, there's a huge learning curve. I've certainly seen it. I'm sure others have. First of all, you can't just download some software or trial and run it on a VM on your existing system. You can get a trial for 30 days, but very often you're not going to learn what you need to. You're not going to learn what you need to in 30 days on a trial. So it's going to cost some money. And then you have to find the time to work with this new paradigm of an environment, whatever they may be. So there are some pretty big barriers for entry as far as I'm concerned with it. And so the challenges with the security of the cloud is you have to kind of understand the security capabilities of that cloud environment. The other thing I mentioned in the talk is that we've talked to a customer where their executive didn't want to have all their eggs in one basket. So they signed up with all of the big three cloud providers, Amazon, AWS, Microsoft Azure, and Google Cloud Platform. Now their eggs are in all the baskets. And it's even more challenging from that perspective, because operations and security, they've got to figure out what that looks like and how to normalize across all three. And of course there's programs or applications or systems that will help you kind of merge all that together in a way that makes sense. But then you have one system that overarches and controls or has control over all of these. So I think it's a complicated issue. I think that certainly there are a lot of security benefits that can be gained from the cloud, but you have to know what you're doing and know how to do it right. I mean, if you're going into Office 365 and Azure AD, you could absolutely, as a relatively new company or relatively small company, go cloud only and not have the on-prem AD, assuming that you don't have all the applications that are tied into it. I did an Active Directory security assessment in the last year where they had one Active Directory forest that had to stay around because the application they needed for whatever they were doing that was in that forest, it was hard-coded to the name of the Active Directory forest. And that application, the company that made it isn't around anymore. This is 10, 15 years old. So there's no way that they can move off that environment until they can figure out how to have someone else come in and recode that application potentially. But there's some weird scenarios like this that are challenges, but ultimately the migration from on-prem to the cloud, I think the challenges that are going to be applications and authentication flow. Some of the applications aren't going to support federation. I've told customers, look, telecontracting now, any new applications have to support federation because this is the time to actually get those onboarded. So that way you're not in a situation 10 years from now where you can't move things. Now, as I said, the managed AD for the major cloud providers, Azure, AWS, GCP, those are there to support those applications. But as I discussed in the talk, there's some interesting things there because now you're providing an AD environment managed by the cloud provider, but you're giving the application owner effectively the admin rights to that environment. And they're not going to be as cognizant of how well to protect those credentials potentially as someone else may be. So there's some interesting things with the managed AD environments as well from a security perspective. So I say the, my answer is it's complicated, but I think eventually it hopefully will get less complicated and be easier to actually onboard these security tools like Microsoft security defaults for Office 365, which are enabled or turned on by default now, once you have a new tenant. What do you, having given that you've now kind of taken a big picture look at all of the different platform options there, do you, is there one that kind of sticks out to you as one platform is doing it better than the others? I'd say from a managed AD environment, and I looked at Azure AD domain services, Microsoft managed AD in 2017. I looked at Amazon AWS as well in 2017. I like the fact that Amazon AWS had all the different delegation groups that they have had. The initial goals of those two environments were quite different. So that explained why they were, they were configured that way. There's some interesting aspects around Amazon or Azure Active Directory domain services, Microsoft's managed AD and how they sync, which could be some interesting things for pen testers or red teamers. I would say that there's, there's certainly elements of each that I like. I like how Amazon has delegated a lot of different things. I like GCP's approach to some of the delegation that they're doing as far as just saying, okay, we're just going to give you access to this component. For example, Azure ADDS and Amazon AWS both pre-create a fine-grained password policy and say, here, you have rights to modify this. GCP is just like, go ahead and create as many as you want, you know, kind of do your own thing. So there's definitely things I like about them, the different ones. But I'd say probably Amazon AWS is probably a little more mature from, from the delegation perspective and, and the, you can set this up and, and create a from your on-prem to, to this managed AD environment and leverage this as an extension of your on-prem AD. I was going to combine two of the questions that we got here, Gers, from both Mugs and CrunchBank. How much better off are we if we don't sync credentials from the on-prem to the cloud platform and rely on SSO? And if you're unwilling to do that, what would you be missing out on using something like the ADPTA? Good question. So the question really comes down to authentication and the risk profile of the customer themselves. So for your single sign on from your on-prem to your cloud, it's typically going to be federation, but that means you're going to have to manage and maintain your own ADFS servers, your Microsoft Active Directory Federation servers, which are a pain. They are not that easy to set up. It's, it's difficult to get right, get right. There's a lot of different things you need to get correct. So then you're looking at Ping or Okta or one of those companies that simplify that process for you, which is going to cost money. If you're a smaller company, you're probably going to do something like password hash sync, at least to the Microsoft platform where you're going to, it's Azure AD Connect is going to have the permissions and AD to pull the hashes for all the different users that are synced. Go ahead and hash those and send them up to Azure AD. And then when the user logs into Azure AD, it compares the password that the user has typed to the hash function that it runs through the on-prem AD hash function, plus the Azure AD hash function to see what that password actually is. The benefit of this password hash sync is that Microsoft gets a ton of password data from security researchers privately, law enforcement, dark web, stuff like that. So at least if you're doing password hash sync, you have a pretty good chance of knowing if that credential has been compromised. So that, that is a nice benefit. And if there's any sort of problem where like ransomware just crushes your on-prem AD and your users can't authenticate, you can flip a bit and the users can authenticate directly to Azure AD. So that way, at least they still have access to their collaboration environment. So a couple of nice benefits there. PTA is interesting from the perspective of providing access without having to do password hash sync or federation. I think that ultimately password hash sync is probably the direction that everything's going to be going in. There's certainly concerns, as I've highlighted, that other people have identified with the different options. Like I said, federation's hard and a federation server has to be available to the internet. So you have a federation server that's basically a web server that's listening to requests from the internet. We have customers that have been had denial of service attacks, basically, because accounts have been locked out with older versions of AD FS. So there's some challenges there. You got to get up to AD FS on server 2019 to enable smart lockout and really have that protect against this denial of service accounts are locked out. So I think it's definitely a complicated story and it's going to be per organization. I'd say the larger organizations pretty much already have federation because they have SaaS apps that they're using. And they pretty well understand that the smaller organizations are going to use password hash sync or PTA. Cool, I'm going to kind of combine maybe a little bit of a look forward along with with a fairly good question that we got from chat. You mentioned in your talk that, you know, cloud is in cloud security is a large area, it's growing area, and it's an opportunity for people that that are in security or looking to further specialize to see some success over the next few years. I think that the challenge for that is how do you how do you get in there? And how do you mess around with this stuff? How do you learn this kind of thing? Specifically, you've been asked if you're going to set up a hybrid training lab for red teamers of some kind. But you know, beyond maybe that, you know, if someone wanted to learn this in their spare time on the side, what's the best way for them to get into this kind of stuff? I would say that the best way to get into it would be to use a trial. I mean, that'll get you 30 days of experience, I would say read the documentation. There's videos out there as well, we're kind of walking through new features, that's a good way to learn and understand what's there. For me, there's nothing better than actually doing and playing around with it. I mean, for speaking specifically to Azure Active Directory and Office 365, you can get an account for about $20 a month, I think, that gives you Office 365, but they have cheaper ones than that, I think that you can use. But again, that is a barrier to entry, because there's a cost with the cloud, that's not there for a lot of the on-prem things. So you can, there's different trials that you can sign up for. So you could have one account that might be $8 or $10 a month, and then kind of have trials for the others. And from what I've seen with some of my testing, you can create a bunch of different accounts, and then have those kind of last for a period of time, a short period of time, before they kind of just get pulled out because they're not subscribed. So it is a challenge. I would say that every organization should have a test account. And if your job is dealing with the cloud in any way, shape, or form, ask your company to make sure there's a test account so you can better understand this, because if you're going in this direction, company's going in this direction, the most important thing they can do is to make sure that the operation security people understand the elements and ramifications of this. The cloud to me is the wild west. We just have not explored enough of it and gotten to the point where we truly understand it. We're not testing it as well from a security perspective, from a red team pen test perspective. And I think from the security perspective, I don't think the controls have been configured the way that they should be for most organizations, just because there's a lack of understanding. New things come out, new features are there, and they just don't get implemented. So given that, if you can't mitigate the risk technically and practically, here comes the lawyer to solve problems with lawyers and contracts. What kinds of things do you think that somebody could build into an SLA to protect the client in case you've got a compromise at the cloud service provider level? That's a great question. I'm not sure. That's probably above my head as far as what that would look like. And there's mitigations for that sort of thing. I mean, certainly from the IaaS side, Amazon AWS has some components around how you can configure your own encryption key. That would mitigate that pretty strongly, I would think. I believe GCP has the same capability. Azure probably has the same capability as well. So there are some options there. Encryption is easy. Key management is hard. So if you're managing your own keys, that makes things a little more difficult. But I think that those are some mitigations that could be done. There are certainly regulatory requirements based on things. But the thing is that these cloud providers have all said, we meet all these criteria. So it does make it challenging. I don't have a good answer for that. The cloud providers compromise at a pretty major level, which probably will happen at some point. I don't know that it'll be the big three, because they spend a tremendous amount of money on security. But a cloud provider compromise, I think, is going to happen at some point. It's just the nature of what we do, like the Intel processors were compromised, which I think is interesting in the past few years. So when you have hypervisors that are kind of the core to what you're doing, and then people are going to look at the hypervisors, which they have. And I think that's why Microsoft has something like $100,000 or $200,000 bug bounty for Azure core hypervisor bugs. Cool. I think there's kind of one last question. This kind of relates to how some of this stuff operates out of the box, right? Do any of these cloud services that you have taken a look at meet any of the kind of HIPAA requirements out of the box when it comes to using Active Directory creds to try to link those together with any electronic medical records programs that you know? So I don't look at the regulations per se. So I can't speak to that or really how they work across the different cloud providers. I can tell you that the regulations, there's always the regulation of how something's supposed to be because it's generally understood, that's the best way to do it. Based on Active Directory security systems that I've done on-prem and what I've seen customers do, which have been HIPAA compliant or PCI compliant, when you have a domain in an Active Directory force, which is a PCI domain, and then you firewall that off, but I can compromise a domain controller in one domain to jump over to a domain controller in that other domain, then that breaks the whole concept of PCI for that. So these things have to be separated. The security controls have to be better understood from the reality of it. And that's why pentesting and red teaming, I think, is important. And certainly security assessments like what we do, but the pentesting and red teaming, showing what these issues are and walking through how they could be compromised despite controls that people academically think are good, I think that's a big part of it. And I think that any aspiring or junior pentester or red teamer watching this right now should absolutely start learning the cloud as best they can. I mean, again, with the constraints of what it costs, but there's a lot of documentation out there, especially from Microsoft. And there's videos about walking through what these features are and what they do, and then start figuring out some ideas and some hypothesis of what this might mean, and then spin up an environment and see what it looks like in a trial and play around with it. And then your trial is going to expire and then give it a couple more months, set up another trial to play around with it. And hopefully the cloud providers will provide some sort of student or beginner access where they can actually just start playing around with these things and get a better understanding of it. I'm sure that there's some learning programs that I'm not aware of that someone will say afterwards, but that would be very helpful. Cool. Well, hey, we are just about out of time, but I want to thank you for putting together your talk and doing it pre-recorded, which I know is definitely a different experience than most of us are used to kind of getting up there and doing it live, but then also kind of playing around with this, with our kind of experiment of doing the live Q&A. I know I had a good time watching your talk. I thought it was great. And I hope everybody in the chat thought the same thing. If anybody needs to get ahold of you or, you know, kind of follow you for more of your developments, where can folks find you on the internet? Twitter's the best, at Pyrotek3, that's P-Y-R-O-T-E-K, number three. Good stuff. Well, thank you very much again, and everybody, everybody hanging out in chat, you know, we're going to take a 30-minute break so you can watch the next talk if you haven't watched it yet, and we'll have a new crew of goons and the next speaker up here in just a little bit. Thanks a bunch. We'll talk to everybody soon. Thank you.