WEBVTT 00:00.000 --> 00:03.400 Next up, I have the pleasure of introducing Dimitri Berioza. 00:03.400 --> 00:10.400 Dimitri is a senior security researcher with Vector AI, where he works on threat detection in cloud and on-prem networks. 00:10.400 --> 00:15.680 His interests include reverse engineering, secure software development, and CTF competitions. 00:15.680 --> 00:29.340 In this talk, we're going to examine logging facilities in Azure, concentrating on events generated by Azure AD and Microsoft 365, where we can discuss multiple problems that he's observed in monitoring them. 00:29.340 --> 00:32.400 So please give a warm welcome for Dimitri. 00:34.120 --> 00:35.220 Thank you. 00:35.420 --> 00:36.660 Thank you, Mike. 00:39.000 --> 00:40.020 Hello, everyone. 00:40.020 --> 00:41.660 Really excited to be here. 00:44.240 --> 00:45.460 Some housekeeping. 00:45.460 --> 00:49.080 The slides, you can just scan the QR code to get the slides. 00:49.220 --> 00:54.640 If we run out of time and there are questions, I'll be happy to discuss things outside. 00:54.640 --> 00:57.420 So I'll just hang around outside the door. 00:57.420 --> 01:06.280 So Dimitri Berioza is my name, and I'll be talking about logs, which are the most exciting topic in cybersecurity. 01:06.760 --> 01:10.580 All right, so Mike already mentioned a few things about me. 01:10.580 --> 01:15.980 There are my handles on Discord, Twitter, and connect with me on LinkedIn. 01:16.760 --> 01:23.100 Before Vector, I worked at IBM as a pen tester, and I come from a software engineering background. 01:25.120 --> 01:26.440 So why this talk? 01:26.440 --> 01:33.720 I currently do security research in the Azure space, and I also respond to incidents that our customers have. 01:34.060 --> 01:40.680 And in that, I concentrate on Azure AD and Microsoft 365. 01:40.680 --> 01:43.360 I'll talk about that a little bit more later. 01:43.580 --> 01:57.740 And in the course of doing that for a few years, I stumbled upon multiple issues that complicate log analysis, that complicate creating automated detection queries, and that's what I'll describe for you today. 01:57.740 --> 02:00.080 It's not an Azure bashing talk. 02:00.080 --> 02:02.360 I love Azure. 02:03.060 --> 02:07.460 And actually, some of the same issues, I think, exist in other cloud providers. 02:07.460 --> 02:14.120 My goal here is to raise awareness for defenders and those on the offensive side, the red timers, pen testers. 02:14.120 --> 02:18.180 And also, hopefully, maybe there's someone from Azure here. 02:18.180 --> 02:19.140 Raise your hand. 02:19.140 --> 02:19.980 No? 02:20.080 --> 02:20.940 Oh, wow. 02:20.940 --> 02:21.300 Okay. 02:21.300 --> 02:22.080 Hello. 02:22.680 --> 02:23.560 All right. 02:23.560 --> 02:26.280 So please pay attention. 02:28.340 --> 02:32.220 So maybe, maybe this will affect some actual change. 02:32.380 --> 02:33.480 Who knows? 02:33.480 --> 02:34.980 Miracles do happen. 02:35.600 --> 02:37.620 And of course, opinions are my own. 02:37.720 --> 02:44.900 My employer just sponsored me to come here, but they don't control the contents of the talk. 02:44.900 --> 02:48.800 So a quick refresher on what Azure is, for those who are not familiar. 02:49.400 --> 02:54.720 I'll speak a little bit about how monitoring was done before cloud and how it's done now. 02:54.760 --> 03:01.780 And actually, the bulk of the talk will be on the problems that I observed and my kind of wish list for what actually should happen. 03:02.600 --> 03:05.600 So Azure, for those who don't know, one of the big three cloud providers. 03:05.600 --> 03:07.260 Actually, I think it's number two now. 03:07.260 --> 03:09.960 They have about a quarter of the market share. 03:10.680 --> 03:13.780 Huge array of services, over 200 now. 03:13.780 --> 03:19.440 It's infrastructure-as-a-service, a platform-as-a-service, software-as-a-service, like the full set. 03:19.680 --> 03:21.280 And it's massive. 03:21.280 --> 03:25.200 It's impossible to talk even about logging in the whole Azure environment. 03:25.200 --> 03:28.400 So I'll just concentrate on Azure AD and Microsoft 365. 03:28.900 --> 03:34.000 So Azure AD, directory and identity management service, really important. 03:34.000 --> 03:35.640 Provides authentication authorization. 03:35.640 --> 03:39.840 So if you want to do anything in Azure, you have to get through that service first. 03:39.840 --> 03:40.820 You have to have an identity. 03:40.820 --> 03:42.520 You have to have certain permissions. 03:42.520 --> 03:51.080 And, of course, they support the whole set of features, Azure, for federation, multi-factor authentication. 03:51.080 --> 03:52.040 You mentioned it. 03:52.040 --> 03:57.340 And has, by what stats are about two years old, almost half a billion daily active users. 03:57.340 --> 03:58.660 So it's kind of a big deal. 03:59.220 --> 04:06.620 Microsoft 365, you're probably familiar with it from back in the Office days, when Office was just on our personal computers. 04:06.620 --> 04:11.060 This is essentially Office in the cloud with a bunch of additions. 04:11.060 --> 04:14.880 So there's also Teams, Exchange Online, Skype, SharePoint, OneDrive. 04:14.880 --> 04:15.960 There's a whole bunch of it. 04:15.960 --> 04:19.600 It's its own big lineup of products. 04:19.600 --> 04:21.240 Also very popular. 04:21.240 --> 04:23.460 I think it's a market leader at this point. 04:23.640 --> 04:26.780 And lots of monthly active users. 04:26.860 --> 04:38.420 So I think if you pick a random person, if they deal with Azure in any way at all, it'll be probably through, well, first of all, through Azure AD. 04:38.420 --> 04:41.180 And most likely they'll have a Microsoft 365 account. 04:41.180 --> 04:43.120 They'll have email hosted in the cloud. 04:43.120 --> 04:45.980 So both of these are really high-profile services. 04:45.980 --> 04:48.860 And this is where it kind of fits in the whole ecosystem. 04:48.860 --> 04:52.560 So Azure AD is a platform-as-a-service service. 04:52.720 --> 04:59.820 And the software-as-a-service Microsoft 365, or Office 365 as it used to be called, is a good example of that. 05:00.780 --> 05:04.640 So event monitoring is based on logs, of course. 05:04.640 --> 05:08.620 Microsoft Sentinel is their native CM solution. 05:08.620 --> 05:14.000 It ingests logs from cloud services, but also can, I think, ingest external logs. 05:14.280 --> 05:16.400 And you can build detections in it. 05:16.400 --> 05:19.000 I think there are a bunch of built-in detections. 05:19.240 --> 05:24.420 There's also something called Azure Log Analytics, which is a great log analysis platform. 05:24.420 --> 05:28.880 You can kind of run ad hoc queries in there and do investigations. 05:28.960 --> 05:31.540 I love Kusto language that they implement. 05:31.540 --> 05:33.440 It's a very powerful query language. 05:33.440 --> 05:39.260 And queries are very, very fast because I think they massively index on the back end. 05:39.260 --> 05:47.100 So it was a breath of fresh air when I could run a query of a massive data set and really get results really quickly. 05:47.520 --> 05:50.040 So logs can be analyzed in Log Analytics. 05:50.040 --> 05:54.660 It can be streamed internally if you have your own CM in your environment. 05:54.920 --> 06:00.000 And third-party solutions that you can buy, they can help you analyze logs. 06:00.000 --> 06:03.900 But they cannot help you produce logs or different logs or better logs. 06:03.900 --> 06:05.180 Microsoft controls that. 06:05.180 --> 06:07.580 And that's kind of part of the problem. 06:07.800 --> 06:13.280 So logs, there are different types of logging available in Azure AD and Microsoft 365. 06:13.300 --> 06:18.040 Azure AD mostly deals with sign-ins, with changes to directory settings. 06:18.040 --> 06:20.700 It's based on a more modern API called Graph API. 06:20.700 --> 06:24.900 I think Microsoft's plan is to migrate all of their services to that. 06:25.140 --> 06:29.000 And it currently exists in base version, in a beta. 06:29.000 --> 06:34.400 I see Microsoft using the beta version in their production products. 06:34.400 --> 06:35.920 And yet it's a beta. 06:35.920 --> 06:37.920 I don't know if they want their customers to use it. 06:37.920 --> 06:39.660 And the schemas are very different. 06:39.660 --> 06:46.060 So it's a bit of a mystery of what the recommendation is for their customers to actually standardize on. 06:46.060 --> 06:50.500 Microsoft 365, kind of a completely different beast in terms of logging. 06:50.500 --> 06:52.500 They have something called Management Activity API. 06:52.500 --> 06:53.980 There are a bunch of logs there. 06:54.000 --> 06:57.640 They're sort of split by general functionality types. 06:57.640 --> 07:01.560 So there's a separate log for Exchange, separate log for SharePoint. 07:01.560 --> 07:06.180 And there's audit general, kind of all other events fall into that. 07:06.180 --> 07:09.380 And I think the plan is to migrate to Graph API eventually. 07:09.980 --> 07:16.800 So if you think back to if you dealt with on-prem networks before, migration to cloud is great. 07:16.800 --> 07:19.540 We get a lot of really neat features. 07:19.640 --> 07:21.860 A lot of things are easier in the cloud. 07:21.860 --> 07:25.400 But when it comes to logging, I think it's kind of a mixed bag. 07:25.400 --> 07:32.140 The problem is that when you control your own environment, when you control your own network, you can set up logging pretty much anywhere. 07:32.140 --> 07:36.400 You could have logging on individual machines, on the edges of your network. 07:36.400 --> 07:38.640 You can put network tabs anywhere you want. 07:38.640 --> 07:40.800 You can go with software. 07:40.800 --> 07:44.340 You can have hardware solutions plugging in, like physical cables. 07:44.340 --> 07:45.800 So you have a lot of choices. 07:45.800 --> 07:48.060 If you don't like one vendor, you ditch them. 07:48.060 --> 07:48.820 You go to another. 07:48.820 --> 07:49.720 No problem. 07:49.760 --> 07:51.740 In the cloud, things are more streamlined. 07:51.740 --> 07:53.800 However, you lose that choice. 07:53.800 --> 07:58.920 So whatever logging you want to have, it only comes from Microsoft. 07:58.920 --> 08:00.880 You don't really have much of a choice. 08:01.200 --> 08:06.420 And if the quality of those logs is great, if everything is really well thought through, perfect. 08:06.460 --> 08:11.140 But unfortunately, if you see issues, you don't have much choice. 08:11.140 --> 08:18.980 You can either migrate to another cloud provider, and it's a huge undertaking, or complain to Microsoft and hope that things get fixed. 08:18.980 --> 08:25.200 Unfortunately, that doesn't happen quickly enough, I guess, when that happens. 08:25.560 --> 08:34.620 So my expectations for log quality in the cloud are such that, first of all, it's easy to consume API, timely logging of events. 08:34.620 --> 08:36.660 The speed is really important here. 08:36.860 --> 08:41.280 Clear and complete documentation for everything that I see in the logs. 08:41.280 --> 08:46.220 I don't want to guess what I'm looking at when I'm analyzing an incident, for example. 08:46.220 --> 08:49.240 All log records should be complete. 08:49.520 --> 08:53.500 And I want the following data consistently available everywhere. 08:53.600 --> 08:56.700 What the operation was and all the corresponding parameters. 08:56.980 --> 08:59.300 All the identity and session information. 08:59.300 --> 09:03.000 I want to know everything about the user that performed that action. 09:03.000 --> 09:08.440 I want full IP information, geolocation, device information, threat and reputation indicators. 09:08.440 --> 09:13.200 They are slowly becoming available, and they really help in incident response. 09:13.200 --> 09:15.160 I also want to correlate between logs. 09:15.160 --> 09:18.700 Somebody logged in, and then they did something in Exchange. 09:18.700 --> 09:20.300 They did something in SharePoint. 09:20.300 --> 09:23.980 I want to connect all the dots there and see the full attack. 09:24.400 --> 09:26.120 That's my wish list. 09:26.120 --> 09:32.980 Unfortunately, that doesn't fully happen to my liking. 09:33.580 --> 09:39.440 Let me run through a big list of things that I see that complicate my work as a defender. 09:40.320 --> 09:45.060 First of all, there are some difficulties in log consumption. 09:45.220 --> 09:52.520 In some of the logs, primarily in Microsoft 365 logs, the schema that you observe is fluid. 09:52.520 --> 10:04.040 Every new event type that you get, if you look at the log as a table, if that new event type has five new fields, you get five new columns. 10:04.040 --> 10:08.160 You can see the log growing not only vertically, but horizontally. 10:08.160 --> 10:13.080 Microsoft itself and Log Analytics may handle it, but actually they don't always handle it. 10:13.080 --> 10:19.100 I've seen this error of limit of maximum 500 columns reached, which is kind of insane. 10:19.360 --> 10:24.180 But if you start to consume these logs externally, it's a big problem. 10:24.180 --> 10:27.380 You need to pick and choose which fields you want to see. 10:27.580 --> 10:34.520 You want to have the foresight to predict what fields you will want to see for this new operation that Microsoft decides to log. 10:34.520 --> 10:36.380 So it's really, really hard to handle. 10:36.980 --> 10:43.060 And also, when you pull on the APIs, you have to be really careful because of delays. 10:43.060 --> 10:45.200 And I'll talk about those further. 10:45.260 --> 10:46.840 Sometimes events come late. 10:46.840 --> 10:51.860 So if you pulled on the log API, you got your logs for today. 10:51.860 --> 10:56.420 Maybe another event is still hanging in there, arrives late. 10:56.820 --> 11:08.680 To get it, you need to call the same API with the same parameters again, and then deduplicate the events to remove the duplicates that you have already seen to get the few events that were late. 11:08.680 --> 11:13.340 So it just adds work to you as a defender. 11:13.940 --> 11:16.020 There are occasionally log flow issues. 11:16.020 --> 11:21.660 In my memory, there were a few log outages there, the ones that were announced. 11:21.660 --> 11:25.740 And one of them that I remember lasted several weeks, and we noticed it just by accident. 11:25.740 --> 11:31.080 So things fell silent, and we didn't have any mechanism in place to actually detect that. 11:31.080 --> 11:34.500 So that was kind of a big problem when you're blind for that long. 11:34.640 --> 11:48.580 Nothing really you can do about it, rather than maybe invest some kind of solution that monitors log flow, and tells you that, oh, you know, this is the normal log patterns, and things all of a sudden have dropped off. 11:48.580 --> 11:49.980 Something is going on. 11:50.980 --> 11:57.420 Also, another kind of, I think, a general issue with all cloud providers, logs are enabled and disabled in band. 11:57.420 --> 12:01.840 So you log in as admin, and you can stop logging. 12:01.840 --> 12:11.840 That's kind of a problem, because if your admin gets popped, as a red teamer, one of the first things to do, I would actually stop logs to blind the security tools. 12:11.840 --> 12:29.080 So I think we need more mechanisms for preventing that, for an event as big as stopping the logs, it has to be more, I guess, difficult for attacker to control that. 12:29.080 --> 12:31.680 And things can actually be changed more stealthily. 12:31.680 --> 12:41.240 You can not only stop logs completely, but stop individual logs, and in exchange, you can stop auditing for specific users. 12:41.380 --> 12:48.140 So as an attacker, you could be selective, and kind of try to stay under the radar. 12:48.140 --> 13:01.320 So the outcome for this, I think defenders just need to pay attention to how their logs are flowing, really pay attention to what events they see that actually trigger log disabling. 13:01.560 --> 13:08.100 And attackers, I think it's a great technique to actually just turn off the logs, even temporarily, to blind the defense. 13:08.100 --> 13:11.840 And Azure just, I think, needs to be aware of this. 13:12.540 --> 13:15.500 So events are also getting delayed. 13:15.500 --> 13:22.100 Microsoft SLA says, I think it's between like 5 and 15 minutes, they pretty much guarantee that you will get events that happen. 13:22.180 --> 13:26.840 Which is also, like 15 minutes is already kind of, you know, it's not very comfortable. 13:26.980 --> 13:31.820 I actually see in Graph API-based logs delays up to 40 minutes. 13:32.240 --> 13:35.640 And in Microsoft 365, sometimes more than 24 hours. 13:35.640 --> 13:38.640 Like if you look, these are recent stats that I pulled. 13:38.640 --> 13:44.380 This is actually one day and 14 hours for this event to arrive in the logs. 13:44.380 --> 13:47.740 Which is just, I don't think this should happen. 13:47.860 --> 13:49.880 Delay really hurts defenders. 13:49.880 --> 13:53.360 You cannot react as it is. 13:53.360 --> 14:04.460 Like even if, let's say, logs arrive in 30 minutes, there were some attackers that are known that actually started doing their work within 30 minutes of the breach. 14:04.460 --> 14:08.700 So you are almost, as a defender, you're almost late by design. 14:08.700 --> 14:16.700 So by the time you see the events, if attacker is, you know, really competent and they have scripted a lot of their actions, they're already done. 14:16.700 --> 14:19.140 You're just, you know, cleaning up and picking up the pieces. 14:19.320 --> 14:31.640 So I think the lesson here is that as defenders, you, I think, need to run, maybe run to some stats to see when, how long it takes for logs to actually get to you. 14:34.060 --> 14:40.100 Attackers take advantage of these delays because I think defenders will be continuously late. 14:40.100 --> 14:43.580 And Azure, I think, should really speed things up. 14:43.580 --> 14:47.200 We need logs to arrive very, very quickly. 14:47.400 --> 14:49.400 There's also something called login tax. 14:49.400 --> 14:59.320 Depending on your level of license that you pay for, you get different, I guess, granularity, different level of security details in the logs. 14:59.320 --> 15:00.180 And that's a big problem. 15:00.180 --> 15:03.580 So you essentially have to pay premium for better security. 15:03.580 --> 15:06.220 There was this great quote from a U.S. 15:06.220 --> 15:13.980 senator that charging people for premium features, necessary not to get hacked, is like selling car and then charging extra for seat belts and airbags. 15:13.980 --> 15:16.300 I completely agree with that. 15:16.340 --> 15:18.240 I understand that Microsoft is a business. 15:18.240 --> 15:19.540 They need to make money. 15:19.540 --> 15:22.560 But this is kind of, I think, a wrong place to make money. 15:22.560 --> 15:25.680 You need to make sure all of your customers are really secure. 15:25.720 --> 15:29.620 And there was a recent incident that really kicked things into high gear. 15:29.620 --> 15:32.000 CISA got involved because a bunch of U.S. 15:32.000 --> 15:35.300 departments got hacked in Azure. 15:35.320 --> 15:43.060 And I think they pushed Microsoft to actually expand access to these types of logs that are not visible to ordinary users. 15:43.060 --> 15:43.980 And that's amazing. 15:43.980 --> 15:45.820 I hope this trend continues. 15:46.380 --> 15:50.780 So for defenders, not much to suggest here. 15:50.780 --> 15:56.000 If things that you want are paid, you have a difficult choice. 15:56.000 --> 16:01.320 You'd have to pay extra or agree with not fully seeing everything. 16:02.460 --> 16:09.580 While these features are still being developed and coming online, attackers can take advantage of some things being hidden from the majority of the customers. 16:09.580 --> 16:14.260 Because not everybody can afford something like $37 per user or something. 16:14.740 --> 16:18.220 It's a big expense to get those extra features. 16:18.220 --> 16:23.120 And Azure, please, don't charge extra for such a basic security feature. 16:23.120 --> 16:28.040 I would say documentation on logs is generally deficient. 16:28.040 --> 16:32.860 The best you can probably hope for is the names of operations that you'll see in the logs. 16:32.860 --> 16:39.340 But if you actually go into the actual log record and start looking at the individual fields, most of them are a mystery. 16:39.900 --> 16:45.740 Individual, let's say, enum values, you get authentication code 278. 16:45.740 --> 16:52.000 And you look online and there's nothing there and you're lost. 16:52.000 --> 16:55.560 You have to hunt through blogs and ask questions and discussion boards. 16:55.560 --> 16:57.120 And this is fine. 16:57.120 --> 16:59.840 I understand that it's just probably part of your daily work. 16:59.840 --> 17:02.100 But as a defender, you shouldn't be doing that. 17:02.100 --> 17:10.320 You should be looking at the logs and analyzing incidents, not digging into all kinds of blogs just to find bits of information. 17:10.320 --> 17:16.280 Microsoft really needs to document everything that they log to help us as defenders. 17:16.280 --> 17:20.560 And attackers, again, any defender's loss is an attacker's gain. 17:20.560 --> 17:25.060 You can't really hide in cases that are not well documented. 17:25.740 --> 17:29.140 I find that Microsoft does a lot of changes that are unannounced. 17:29.280 --> 17:35.540 In my experience, I was at one point monitoring mailbox login events in Exchange logs. 17:35.540 --> 17:37.440 And then all of a sudden they disappeared. 17:37.480 --> 17:40.300 And I noticed maybe several months later. 17:40.300 --> 17:41.760 Because it was just completely silent. 17:41.760 --> 17:43.120 So you have your queries running. 17:43.120 --> 17:44.380 They're not finding anything. 17:44.380 --> 17:47.160 And that's because that event is not logged anymore. 17:47.160 --> 17:56.100 Or I had queries that detected attackers trying to enumerate users coming from the outside of the enterprise. 17:56.100 --> 18:02.620 They would try different combinations of usernames, like a first name, dot, last name, or last name, dot, first name, etc., etc. 18:02.620 --> 18:04.620 They tried those enumeration attacks. 18:04.620 --> 18:05.800 I used to see those logs. 18:05.800 --> 18:06.880 They're gone now. 18:06.880 --> 18:09.380 Again, silently, without announcement. 18:09.420 --> 18:10.900 It's kind of upsetting. 18:10.940 --> 18:13.780 And also we had a case underlined there. 18:13.780 --> 18:14.960 You can see it in the slides. 18:14.960 --> 18:15.900 There's a link. 18:15.900 --> 18:29.480 We documented a case where passwordless authentication, when it was introduced and encouraged, the error codes that you get for that are completely different from anything you got before. 18:29.820 --> 18:30.560 Undocumented. 18:30.560 --> 18:32.460 We discovered them by accident. 18:33.340 --> 18:35.260 Just didn't know how to react to that. 18:35.260 --> 18:39.720 But this is important if you're doing threat hunting and responding to incidents. 18:40.180 --> 18:44.360 So it's really the worst kind of failure when you don't know that you have a problem. 18:44.360 --> 18:53.100 So for anonymous changes, I don't know how much defenders can do rather than just study and continuously look at the logs and learn. 18:53.100 --> 18:54.880 But it shouldn't be this hard. 18:54.880 --> 18:56.740 I think Microsoft can do better. 18:56.860 --> 19:03.080 And again, anything that's difficult for defenders, attackers can put in their TTPs. 19:03.540 --> 19:06.620 So cloud is a recon paradise. 19:07.020 --> 19:11.540 Get type events, when you enumerate something, almost never get logged. 19:11.540 --> 19:16.400 I think it's the same thing in AWS and maybe the same thing in other places. 19:16.720 --> 19:20.740 So when an attacker gets access, they can look around without you noticing. 19:20.740 --> 19:21.920 I actually did an experiment. 19:21.920 --> 19:28.420 I ran a bunch of open source Microsoft 365 Azure AD attack tools, the ones that do enumeration. 19:28.480 --> 19:30.520 Virtually no signal in the logs. 19:30.520 --> 19:36.900 As soon as you try to maybe try different passwords, that's when things start to show up in the logs. 19:36.900 --> 19:38.640 Before that, it's silent. 19:38.640 --> 19:42.940 So if you're a red teamer, have a ball. 19:43.000 --> 19:47.940 As soon as you get in, you're free to look around and not be noticed. 19:48.040 --> 19:52.720 Again, Azure doesn't even give you an option to turn this on. 19:52.980 --> 19:56.000 I'm sure there's some serious reason for not logging them. 19:56.000 --> 20:01.100 Maybe there's space concerns, like the logs will just explode if you log all of these events. 20:01.100 --> 20:06.780 But at least give me an option to turn them on if I'm extra careful in my particular environment. 20:06.780 --> 20:10.320 I may want to actually turn these things on. 20:11.620 --> 20:15.020 There are a bunch of inconsistencies I see in the data that's being logged. 20:15.020 --> 20:22.560 So when you run your detection queries, you have to be really careful to make sure that you kind of clean up, let's say, the IDs that you see. 20:22.560 --> 20:29.340 Because, as you can see, the same user can be logged with different letter case in different log records. 20:29.440 --> 20:32.920 Sometimes the log ID is different altogether. 20:32.920 --> 20:36.600 And Exchange really throws a wrench in the works. 20:36.600 --> 20:42.700 This huge chunk of texture is an actual user ID that gets logged for some of the users. 20:42.700 --> 20:55.020 So you can imagine, if you're investigating an incident and you're trying to consolidate different logs and tell a story of what happened, it's really difficult to connect the dots if you just want to join on username. 20:55.120 --> 20:58.860 There are unique IDs available that you may want to... 20:58.860 --> 21:03.560 It's probably a better way of consolidating, but again, I don't have to discover this. 21:03.560 --> 21:05.880 This really has to be clearly explained to me. 21:08.180 --> 21:11.180 Occasionally things are just missing, not available. 21:11.560 --> 21:15.300 I'll try to go quicker because I think there's a lot more material. 21:15.300 --> 21:17.880 For IP also, lots of inconsistencies. 21:17.880 --> 21:26.300 You usually get the clean IP v4 or v6 address, but sometimes you get IPs logged together with ports. 21:26.480 --> 21:31.300 So you have to call a port number, you have to cut that off to clean things up. 21:31.300 --> 21:38.580 I've seen empty IPs, I've seen local and private IPs, I've seen all zero IPs, I've seen all 255 IPs. 21:39.080 --> 21:43.020 None of this is useful, but we still see that. 21:43.020 --> 21:46.040 So it's hard to deal with. 21:47.520 --> 21:52.260 Geolocation also, it's a useful feature, it's only available in sign-ins log. 21:52.780 --> 21:55.440 Sometimes it breaks, sometimes it's empty. 21:55.440 --> 21:59.380 You can see here the real record that I pulled last night. 21:59.640 --> 22:01.240 Country, state, everything is empty. 22:01.240 --> 22:03.320 So that's not helping everyone. 22:03.320 --> 22:05.360 I understand that geolocation is kind of tricky. 22:05.540 --> 22:10.720 You can be easily fooled, and it does get fooled by Autor, by VPNs, proxies, etc. 22:10.720 --> 22:14.400 But when an attacker is not careful, they're revealing their real location. 22:14.400 --> 22:16.540 It's helpful in this investigation. 22:16.540 --> 22:19.820 So it really has to work consistently. 22:19.840 --> 22:22.200 Device information also hit and miss sometimes. 22:22.200 --> 22:25.440 I think it's mostly grabbed from the user agent, and it's a mixed bag. 22:25.440 --> 22:32.400 You get sometimes a full user agent, sometimes you get pieces of it parsed out, and parsed not always consistently. 22:32.420 --> 22:35.440 So some consistency in all of the logs would be great. 22:36.120 --> 22:39.000 Sometimes you get broken values in the logs. 22:39.000 --> 22:42.600 So as part of your query, you may want to parse a field that's JSON. 22:43.700 --> 22:51.080 I think when things run out of space, you actually get broken JSON, as you can see in this example here. 22:51.080 --> 22:54.020 These three dots is not something you click on to expand. 22:54.020 --> 22:57.480 These are literal three dots at the end of JSON. 22:57.480 --> 23:00.420 And of course, your parser will choke on that. 23:00.660 --> 23:03.220 So something to be aware of. 23:03.940 --> 23:10.200 Again, for all of these, the running lesson is defenders be aware. 23:10.400 --> 23:14.000 Try to factor that in your analysis queries. 23:14.000 --> 23:18.040 Attackers, maybe something to take advantage of, because these are not some fancy zero days. 23:18.040 --> 23:25.780 You could probably add stealth to your actions by just taking advantage of all these little deficiencies. 23:25.780 --> 23:28.220 And Azure, please fix stuff. 23:28.480 --> 23:40.940 All right, so all of this is mostly about log analysis and the signal you see in the logs, but you could actually occasionally be attacked through the logs. 23:40.940 --> 23:44.320 I just want to explain this example here. 23:44.320 --> 23:50.480 So log pollution is a type of attack where attacker controls the content in your log maliciously. 23:50.480 --> 23:58.480 And it could be coupled with something called CSV injection, which is an Excel formula you could embed in a CSV field. 23:58.480 --> 24:05.700 And because most of us open CSVs in Excel, Excel actually interprets those formulas and executes them. 24:05.700 --> 24:09.280 And sometimes those formulas can actually run external commands. 24:09.280 --> 24:20.360 So there was a case of log pollution with CSV injection that was documented by Rhino Security five years ago in AWS and Azure. 24:20.360 --> 24:23.300 I think it got fixed since, but I found another one. 24:23.300 --> 24:26.820 And this one is pre-authentication, so I'll just do a demo of that. 24:26.860 --> 24:39.260 So the idea is to get a malicious payload in the logs and social engineer the admin to open the log to get this payload to detonate on their machine. 24:39.260 --> 24:44.220 It's harder to do these days because Microsoft put some defenses in Microsoft Office. 24:44.620 --> 24:54.920 You could do it with the old version of the Office, or you could maybe social engineer admin to drop this particular setting in their Excel. 24:54.920 --> 24:59.860 It's not easy to do, so it's not like a zero-click thing, but it's something to be aware of. 24:59.860 --> 25:06.760 And when I reported it to Microsoft, they didn't seem really excited about it, so I'm just going to tell you guys about it. 25:08.160 --> 25:11.800 So anyway, this is the payload. 25:11.800 --> 25:14.940 You may recognize it as a DDE formula from Excel. 25:14.940 --> 25:18.240 It just pops a calc because we're all white hats here, right? 25:19.880 --> 25:27.020 It attempts to sign in into Azure with an existing account and a bad password, so actually you don't want to log in. 25:27.020 --> 25:32.500 You just want to try to log in with that special user agent field. 25:32.500 --> 25:44.900 And then when a defender, admin, opens the log as a CSV file, if he has the right version of Office, like the old version of Office, or if these defenses are dropped, it will detonate. 25:44.900 --> 25:46.520 So this is the demo. 25:48.840 --> 25:50.220 Takes a bit of time. 25:54.720 --> 26:00.200 So the first step is to override the user agent with an exploit that I just showed. 26:06.840 --> 26:14.340 I think you can do it in pretty much any browser nowadays, or use Zap, or use Burp, it doesn't matter. 26:25.400 --> 26:28.580 So there's a field for user agent. 26:29.220 --> 26:31.260 It's under network conditions. 26:31.260 --> 26:37.780 Instead of using browser default, we use this custom string that we drop in there. 26:38.760 --> 26:45.040 And then, the second step, where we try to log in as an existing user. 27:00.830 --> 27:15.030 So I type my username, and then I intentionally type the invalid password, so that there's just a record of me trying to log in and failing. 27:15.030 --> 27:18.110 But the user agent gets into the log. 27:18.110 --> 27:19.610 So this fails. 27:19.790 --> 27:24.630 Then I call up the admin and say, hello, this is Microsoft support. 27:24.630 --> 27:26.890 I want to help you investigate an incident. 27:26.890 --> 27:29.170 We're seeing something fishy in your system. 27:29.170 --> 27:30.990 Can you please pull up the logs? 27:32.210 --> 27:35.730 It takes a bit of time for the records to actually get there. 27:41.650 --> 27:47.290 And then, so admin says, well, Microsoft told me to do this. 27:47.290 --> 27:48.410 I must do this. 27:48.410 --> 27:50.790 They open the logs in Log Analytics. 27:51.130 --> 27:54.790 I added this query to kind of filter things down to make things clearer. 27:54.790 --> 27:56.350 You don't have to run any queries. 27:56.350 --> 27:59.510 As long as you open that log, that payload should be in there. 27:59.510 --> 28:01.690 And you export it as CSV. 28:01.690 --> 28:03.610 You open it in Excel. 28:06.530 --> 28:10.290 You tell the user to click through these useless prompts. 28:12.290 --> 28:13.330 And boom. 28:13.330 --> 28:14.170 All right. 28:18.270 --> 28:19.190 Thank you. 28:19.410 --> 28:20.310 All right. 28:20.310 --> 28:28.650 So the lesson here is to kind of be aware that this may happen and attackers maybe try and, you know, integrate this into your tactics. 28:28.650 --> 28:29.790 So why is this happening? 28:29.790 --> 28:30.570 I have no idea. 28:30.570 --> 28:31.470 I'm speculating. 28:31.470 --> 28:34.870 I think logs are an afterthought for many software developers. 28:34.870 --> 28:37.930 I know when I was developing software, it was an afterthought for us. 28:38.010 --> 28:41.590 Features were the focus, and then everything else comes after. 28:41.650 --> 28:46.370 Sometimes maybe other fixes take priority to fixing these things. 28:46.370 --> 28:53.030 Some of the logs come from legacy products probably, so they just integrate with them and forget about them. 28:53.090 --> 28:57.370 And sometimes provider doesn't have an incentive to fix things. 28:57.370 --> 28:59.850 But whatever the reason is, I think things have to be fixed. 28:59.850 --> 29:08.110 So I think what should happen is full documentation for everything, logs treated as an API, so all the changes announced. 29:09.210 --> 29:14.590 You know, you have time to react if you have queries running against the current scheme of the logs. 29:14.590 --> 29:16.510 Logs are timely, arrive quickly. 29:16.530 --> 29:17.510 Values are clean. 29:17.510 --> 29:19.370 This is my Christmas wish list. 29:19.370 --> 29:26.990 I don't know how much of it is actually going to happen, but Microsoft, this is what the customers long for. 29:27.310 --> 29:37.930 And as a maximum goal, I think this thing should be refactored because it feels like it's pulled together from many different pieces and held together, like with duct tape. 29:39.450 --> 29:47.810 Ideally, a refactor is in order because I have seen logs in AWS and Okta, and I think they do a much better job, much cleaner job of doing this. 29:48.470 --> 29:49.670 So that's it. 29:51.250 --> 29:54.110 Logging, you know, we see it's in the cloud. 29:54.110 --> 29:55.250 This has a lot of advantages. 29:55.250 --> 29:56.190 It's uniform. 29:56.190 --> 30:00.170 It's kind of better understood, probably, than on-prem. 30:00.170 --> 30:06.350 But there's a lack of alternatives and no strong incentive for a provider to fix things. 30:06.350 --> 30:08.690 As defenders, we just want to be aware of these issues. 30:08.690 --> 30:13.110 And red teamers, until the issues are fixed, maybe try to take advantage of them. 30:13.110 --> 30:13.850 That's it. 30:13.850 --> 30:15.910 I think I'm out of time. 30:16.970 --> 30:19.350 Do we have time for questions? 30:19.570 --> 30:20.170 Okay . 30:20.590 --> 30:21.950 Sure, go ahead. 30:21.950 --> 30:36.960 Yeah, I think actually Microsoft monitors for... 30:37.580 --> 30:38.600 Oh, yes, sorry. 30:38.600 --> 30:45.360 Is there software that can monitor for these things happening and sending alerts? 30:45.360 --> 30:47.400 I think Microsoft has logging for that. 30:47.400 --> 30:52.360 Other external players have products that monitor for things like that. 30:52.360 --> 30:55.840 Our product company I'm working for does that. 30:56.380 --> 30:59.740 Or you can roll your own, execute your own queries. 31:00.180 --> 31:01.600 One more question. 31:04.840 --> 31:05.540 Yes. 31:05.540 --> 31:11.140 Do you have a query to turn AWS JSON log back to where it was supposed to be at? 31:11.140 --> 31:17.660 No, because when it's logged and sent, it only breaks down the entire AWS logs into whole fields. 31:18.300 --> 31:21.840 So the question is, do I have a query for what? 31:21.840 --> 31:22.740 I'm sorry, for... 31:22.740 --> 31:27.060 For changing Azure logs at AWS. 31:27.060 --> 31:29.160 That's back to their AWS JSON format. 31:29.160 --> 31:33.180 Because when you get JSON, you just have to call it with them. 31:34.900 --> 31:36.940 Okay, I don't think I got that. 31:36.940 --> 31:39.280 Maybe we can discuss this outside. 31:39.280 --> 31:40.240 You'll explain it to me.