[00:00.000 --> 00:03.400]  Next up, I have the pleasure of introducing Dimitri Berioza.
[00:03.400 --> 00:10.400]  Dimitri is a senior security researcher with Vector AI, where he works on threat detection in cloud and on-prem networks.
[00:10.400 --> 00:15.680]  His interests include reverse engineering, secure software development, and CTF competitions.
[00:15.680 --> 00:29.340]  In this talk, we're going to examine logging facilities in Azure, concentrating on events generated by Azure AD and Microsoft 365, where we can discuss multiple problems that he's observed in monitoring them.
[00:29.340 --> 00:32.400]  So please give a warm welcome for Dimitri.
[00:34.120 --> 00:35.220]  Thank you.
[00:35.420 --> 00:36.660]  Thank you, Mike.
[00:39.000 --> 00:40.020]  Hello, everyone.
[00:40.020 --> 00:41.660]  Really excited to be here.
[00:44.240 --> 00:45.460]  Some housekeeping.
[00:45.460 --> 00:49.080]  The slides, you can just scan the QR code to get the slides.
[00:49.220 --> 00:54.640]  If we run out of time and there are questions, I'll be happy to discuss things outside.
[00:54.640 --> 00:57.420]  So I'll just hang around outside the door.
[00:57.420 --> 01:06.280]  So Dimitri Berioza is my name, and I'll be talking about logs, which are the most exciting topic in cybersecurity.
[01:06.760 --> 01:10.580]  All right, so Mike already mentioned a few things about me.
[01:10.580 --> 01:15.980]  There are my handles on Discord, Twitter, and connect with me on LinkedIn.
[01:16.760 --> 01:23.100]  Before Vector, I worked at IBM as a pen tester, and I come from a software engineering background.
[01:25.120 --> 01:26.440]  So why this talk?
[01:26.440 --> 01:33.720]  I currently do security research in the Azure space, and I also respond to incidents that our customers have.
[01:34.060 --> 01:40.680]  And in that, I concentrate on Azure AD and Microsoft 365.
[01:40.680 --> 01:43.360]  I'll talk about that a little bit more later.
[01:43.580 --> 01:57.740]  And in the course of doing that for a few years, I stumbled upon multiple issues that complicate log analysis, that complicate creating automated detection queries, and that's what I'll describe for you today.
[01:57.740 --> 02:00.080]  It's not an Azure bashing talk.
[02:00.080 --> 02:02.360]  I love Azure.
[02:03.060 --> 02:07.460]  And actually, some of the same issues, I think, exist in other cloud providers.
[02:07.460 --> 02:14.120]  My goal here is to raise awareness for defenders and those on the offensive side, the red timers, pen testers.
[02:14.120 --> 02:18.180]  And also, hopefully, maybe there's someone from Azure here.
[02:18.180 --> 02:19.140]  Raise your hand.
[02:19.140 --> 02:19.980]  No?
[02:20.080 --> 02:20.940]  Oh, wow.
[02:20.940 --> 02:21.300]  Okay.
[02:21.300 --> 02:22.080]  Hello.
[02:22.680 --> 02:23.560]  All right.
[02:23.560 --> 02:26.280]  So please pay attention.
[02:28.340 --> 02:32.220]  So maybe, maybe this will affect some actual change.
[02:32.380 --> 02:33.480]  Who knows?
[02:33.480 --> 02:34.980]  Miracles do happen.
[02:35.600 --> 02:37.620]  And of course, opinions are my own.
[02:37.720 --> 02:44.900]  My employer just sponsored me to come here, but they don't control the contents of the talk.
[02:44.900 --> 02:48.800]  So a quick refresher on what Azure is, for those who are not familiar.
[02:49.400 --> 02:54.720]  I'll speak a little bit about how monitoring was done before cloud and how it's done now.
[02:54.760 --> 03:01.780]  And actually, the bulk of the talk will be on the problems that I observed and my kind of wish list for what actually should happen.
[03:02.600 --> 03:05.600]  So Azure, for those who don't know, one of the big three cloud providers.
[03:05.600 --> 03:07.260]  Actually, I think it's number two now.
[03:07.260 --> 03:09.960]  They have about a quarter of the market share.
[03:10.680 --> 03:13.780]  Huge array of services, over 200 now.
[03:13.780 --> 03:19.440]  It's infrastructure-as-a-service, a platform-as-a-service, software-as-a-service, like the full set.
[03:19.680 --> 03:21.280]  And it's massive.
[03:21.280 --> 03:25.200]  It's impossible to talk even about logging in the whole Azure environment.
[03:25.200 --> 03:28.400]  So I'll just concentrate on Azure AD and Microsoft 365.
[03:28.900 --> 03:34.000]  So Azure AD, directory and identity management service, really important.
[03:34.000 --> 03:35.640]  Provides authentication authorization.
[03:35.640 --> 03:39.840]  So if you want to do anything in Azure, you have to get through that service first.
[03:39.840 --> 03:40.820]  You have to have an identity.
[03:40.820 --> 03:42.520]  You have to have certain permissions.
[03:42.520 --> 03:51.080]  And, of course, they support the whole set of features, Azure, for federation, multi-factor authentication.
[03:51.080 --> 03:52.040]  You mentioned it.
[03:52.040 --> 03:57.340]  And has, by what stats are about two years old, almost half a billion daily active users.
[03:57.340 --> 03:58.660]  So it's kind of a big deal.
[03:59.220 --> 04:06.620]  Microsoft 365, you're probably familiar with it from back in the Office days, when Office was just on our personal computers.
[04:06.620 --> 04:11.060]  This is essentially Office in the cloud with a bunch of additions.
[04:11.060 --> 04:14.880]  So there's also Teams, Exchange Online, Skype, SharePoint, OneDrive.
[04:14.880 --> 04:15.960]  There's a whole bunch of it.
[04:15.960 --> 04:19.600]  It's its own big lineup of products.
[04:19.600 --> 04:21.240]  Also very popular.
[04:21.240 --> 04:23.460]  I think it's a market leader at this point.
[04:23.640 --> 04:26.780]  And lots of monthly active users.
[04:26.860 --> 04:38.420]  So I think if you pick a random person, if they deal with Azure in any way at all, it'll be probably through, well, first of all, through Azure AD.
[04:38.420 --> 04:41.180]  And most likely they'll have a Microsoft 365 account.
[04:41.180 --> 04:43.120]  They'll have email hosted in the cloud.
[04:43.120 --> 04:45.980]  So both of these are really high-profile services.
[04:45.980 --> 04:48.860]  And this is where it kind of fits in the whole ecosystem.
[04:48.860 --> 04:52.560]  So Azure AD is a platform-as-a-service service.
[04:52.720 --> 04:59.820]  And the software-as-a-service Microsoft 365, or Office 365 as it used to be called, is a good example of that.
[05:00.780 --> 05:04.640]  So event monitoring is based on logs, of course.
[05:04.640 --> 05:08.620]  Microsoft Sentinel is their native CM solution.
[05:08.620 --> 05:14.000]  It ingests logs from cloud services, but also can, I think, ingest external logs.
[05:14.280 --> 05:16.400]  And you can build detections in it.
[05:16.400 --> 05:19.000]  I think there are a bunch of built-in detections.
[05:19.240 --> 05:24.420]  There's also something called Azure Log Analytics, which is a great log analysis platform.
[05:24.420 --> 05:28.880]  You can kind of run ad hoc queries in there and do investigations.
[05:28.960 --> 05:31.540]  I love Kusto language that they implement.
[05:31.540 --> 05:33.440]  It's a very powerful query language.
[05:33.440 --> 05:39.260]  And queries are very, very fast because I think they massively index on the back end.
[05:39.260 --> 05:47.100]  So it was a breath of fresh air when I could run a query of a massive data set and really get results really quickly.
[05:47.520 --> 05:50.040]  So logs can be analyzed in Log Analytics.
[05:50.040 --> 05:54.660]  It can be streamed internally if you have your own CM in your environment.
[05:54.920 --> 06:00.000]  And third-party solutions that you can buy, they can help you analyze logs.
[06:00.000 --> 06:03.900]  But they cannot help you produce logs or different logs or better logs.
[06:03.900 --> 06:05.180]  Microsoft controls that.
[06:05.180 --> 06:07.580]  And that's kind of part of the problem.
[06:07.800 --> 06:13.280]  So logs, there are different types of logging available in Azure AD and Microsoft 365.
[06:13.300 --> 06:18.040]  Azure AD mostly deals with sign-ins, with changes to directory settings.
[06:18.040 --> 06:20.700]  It's based on a more modern API called Graph API.
[06:20.700 --> 06:24.900]  I think Microsoft's plan is to migrate all of their services to that.
[06:25.140 --> 06:29.000]  And it currently exists in base version, in a beta.
[06:29.000 --> 06:34.400]  I see Microsoft using the beta version in their production products.
[06:34.400 --> 06:35.920]  And yet it's a beta.
[06:35.920 --> 06:37.920]  I don't know if they want their customers to use it.
[06:37.920 --> 06:39.660]  And the schemas are very different.
[06:39.660 --> 06:46.060]  So it's a bit of a mystery of what the recommendation is for their customers to actually standardize on.
[06:46.060 --> 06:50.500]  Microsoft 365, kind of a completely different beast in terms of logging.
[06:50.500 --> 06:52.500]  They have something called Management Activity API.
[06:52.500 --> 06:53.980]  There are a bunch of logs there.
[06:54.000 --> 06:57.640]  They're sort of split by general functionality types.
[06:57.640 --> 07:01.560]  So there's a separate log for Exchange, separate log for SharePoint.
[07:01.560 --> 07:06.180]  And there's audit general, kind of all other events fall into that.
[07:06.180 --> 07:09.380]  And I think the plan is to migrate to Graph API eventually.
[07:09.980 --> 07:16.800]  So if you think back to if you dealt with on-prem networks before, migration to cloud is great.
[07:16.800 --> 07:19.540]  We get a lot of really neat features.
[07:19.640 --> 07:21.860]  A lot of things are easier in the cloud.
[07:21.860 --> 07:25.400]  But when it comes to logging, I think it's kind of a mixed bag.
[07:25.400 --> 07:32.140]  The problem is that when you control your own environment, when you control your own network, you can set up logging pretty much anywhere.
[07:32.140 --> 07:36.400]  You could have logging on individual machines, on the edges of your network.
[07:36.400 --> 07:38.640]  You can put network tabs anywhere you want.
[07:38.640 --> 07:40.800]  You can go with software.
[07:40.800 --> 07:44.340]  You can have hardware solutions plugging in, like physical cables.
[07:44.340 --> 07:45.800]  So you have a lot of choices.
[07:45.800 --> 07:48.060]  If you don't like one vendor, you ditch them.
[07:48.060 --> 07:48.820]  You go to another.
[07:48.820 --> 07:49.720]  No problem.
[07:49.760 --> 07:51.740]  In the cloud, things are more streamlined.
[07:51.740 --> 07:53.800]  However, you lose that choice.
[07:53.800 --> 07:58.920]  So whatever logging you want to have, it only comes from Microsoft.
[07:58.920 --> 08:00.880]  You don't really have much of a choice.
[08:01.200 --> 08:06.420]  And if the quality of those logs is great, if everything is really well thought through, perfect.
[08:06.460 --> 08:11.140]  But unfortunately, if you see issues, you don't have much choice.
[08:11.140 --> 08:18.980]  You can either migrate to another cloud provider, and it's a huge undertaking, or complain to Microsoft and hope that things get fixed.
[08:18.980 --> 08:25.200]  Unfortunately, that doesn't happen quickly enough, I guess, when that happens.
[08:25.560 --> 08:34.620]  So my expectations for log quality in the cloud are such that, first of all, it's easy to consume API, timely logging of events.
[08:34.620 --> 08:36.660]  The speed is really important here.
[08:36.860 --> 08:41.280]  Clear and complete documentation for everything that I see in the logs.
[08:41.280 --> 08:46.220]  I don't want to guess what I'm looking at when I'm analyzing an incident, for example.
[08:46.220 --> 08:49.240]  All log records should be complete.
[08:49.520 --> 08:53.500]  And I want the following data consistently available everywhere.
[08:53.600 --> 08:56.700]  What the operation was and all the corresponding parameters.
[08:56.980 --> 08:59.300]  All the identity and session information.
[08:59.300 --> 09:03.000]  I want to know everything about the user that performed that action.
[09:03.000 --> 09:08.440]  I want full IP information, geolocation, device information, threat and reputation indicators.
[09:08.440 --> 09:13.200]  They are slowly becoming available, and they really help in incident response.
[09:13.200 --> 09:15.160]  I also want to correlate between logs.
[09:15.160 --> 09:18.700]  Somebody logged in, and then they did something in Exchange.
[09:18.700 --> 09:20.300]  They did something in SharePoint.
[09:20.300 --> 09:23.980]  I want to connect all the dots there and see the full attack.
[09:24.400 --> 09:26.120]  That's my wish list.
[09:26.120 --> 09:32.980]  Unfortunately, that doesn't fully happen to my liking.
[09:33.580 --> 09:39.440]  Let me run through a big list of things that I see that complicate my work as a defender.
[09:40.320 --> 09:45.060]  First of all, there are some difficulties in log consumption.
[09:45.220 --> 09:52.520]  In some of the logs, primarily in Microsoft 365 logs, the schema that you observe is fluid.
[09:52.520 --> 10:04.040]  Every new event type that you get, if you look at the log as a table, if that new event type has five new fields, you get five new columns.
[10:04.040 --> 10:08.160]  You can see the log growing not only vertically, but horizontally.
[10:08.160 --> 10:13.080]  Microsoft itself and Log Analytics may handle it, but actually they don't always handle it.
[10:13.080 --> 10:19.100]  I've seen this error of limit of maximum 500 columns reached, which is kind of insane.
[10:19.360 --> 10:24.180]  But if you start to consume these logs externally, it's a big problem.
[10:24.180 --> 10:27.380]  You need to pick and choose which fields you want to see.
[10:27.580 --> 10:34.520]  You want to have the foresight to predict what fields you will want to see for this new operation that Microsoft decides to log.
[10:34.520 --> 10:36.380]  So it's really, really hard to handle.
[10:36.980 --> 10:43.060]  And also, when you pull on the APIs, you have to be really careful because of delays.
[10:43.060 --> 10:45.200]  And I'll talk about those further.
[10:45.260 --> 10:46.840]  Sometimes events come late.
[10:46.840 --> 10:51.860]  So if you pulled on the log API, you got your logs for today.
[10:51.860 --> 10:56.420]  Maybe another event is still hanging in there, arrives late.
[10:56.820 --> 11:08.680]  To get it, you need to call the same API with the same parameters again, and then deduplicate the events to remove the duplicates that you have already seen to get the few events that were late.
[11:08.680 --> 11:13.340]  So it just adds work to you as a defender.
[11:13.940 --> 11:16.020]  There are occasionally log flow issues.
[11:16.020 --> 11:21.660]  In my memory, there were a few log outages there, the ones that were announced.
[11:21.660 --> 11:25.740]  And one of them that I remember lasted several weeks, and we noticed it just by accident.
[11:25.740 --> 11:31.080]  So things fell silent, and we didn't have any mechanism in place to actually detect that.
[11:31.080 --> 11:34.500]  So that was kind of a big problem when you're blind for that long.
[11:34.640 --> 11:48.580]  Nothing really you can do about it, rather than maybe invest some kind of solution that monitors log flow, and tells you that, oh, you know, this is the normal log patterns, and things all of a sudden have dropped off.
[11:48.580 --> 11:49.980]  Something is going on.
[11:50.980 --> 11:57.420]  Also, another kind of, I think, a general issue with all cloud providers, logs are enabled and disabled in band.
[11:57.420 --> 12:01.840]  So you log in as admin, and you can stop logging.
[12:01.840 --> 12:11.840]  That's kind of a problem, because if your admin gets popped, as a red teamer, one of the first things to do, I would actually stop logs to blind the security tools.
[12:11.840 --> 12:29.080]  So I think we need more mechanisms for preventing that, for an event as big as stopping the logs, it has to be more, I guess, difficult for attacker to control that.
[12:29.080 --> 12:31.680]  And things can actually be changed more stealthily.
[12:31.680 --> 12:41.240]  You can not only stop logs completely, but stop individual logs, and in exchange, you can stop auditing for specific users.
[12:41.380 --> 12:48.140]  So as an attacker, you could be selective, and kind of try to stay under the radar.
[12:48.140 --> 13:01.320]  So the outcome for this, I think defenders just need to pay attention to how their logs are flowing, really pay attention to what events they see that actually trigger log disabling.
[13:01.560 --> 13:08.100]  And attackers, I think it's a great technique to actually just turn off the logs, even temporarily, to blind the defense.
[13:08.100 --> 13:11.840]  And Azure just, I think, needs to be aware of this.
[13:12.540 --> 13:15.500]  So events are also getting delayed.
[13:15.500 --> 13:22.100]  Microsoft SLA says, I think it's between like 5 and 15 minutes, they pretty much guarantee that you will get events that happen.
[13:22.180 --> 13:26.840]  Which is also, like 15 minutes is already kind of, you know, it's not very comfortable.
[13:26.980 --> 13:31.820]  I actually see in Graph API-based logs delays up to 40 minutes.
[13:32.240 --> 13:35.640]  And in Microsoft 365, sometimes more than 24 hours.
[13:35.640 --> 13:38.640]  Like if you look, these are recent stats that I pulled.
[13:38.640 --> 13:44.380]  This is actually one day and 14 hours for this event to arrive in the logs.
[13:44.380 --> 13:47.740]  Which is just, I don't think this should happen.
[13:47.860 --> 13:49.880]  Delay really hurts defenders.
[13:49.880 --> 13:53.360]  You cannot react as it is.
[13:53.360 --> 14:04.460]  Like even if, let's say, logs arrive in 30 minutes, there were some attackers that are known that actually started doing their work within 30 minutes of the breach.
[14:04.460 --> 14:08.700]  So you are almost, as a defender, you're almost late by design.
[14:08.700 --> 14:16.700]  So by the time you see the events, if attacker is, you know, really competent and they have scripted a lot of their actions, they're already done.
[14:16.700 --> 14:19.140]  You're just, you know, cleaning up and picking up the pieces.
[14:19.320 --> 14:31.640]  So I think the lesson here is that as defenders, you, I think, need to run, maybe run to some stats to see when, how long it takes for logs to actually get to you.
[14:34.060 --> 14:40.100]  Attackers take advantage of these delays because I think defenders will be continuously late.
[14:40.100 --> 14:43.580]  And Azure, I think, should really speed things up.
[14:43.580 --> 14:47.200]  We need logs to arrive very, very quickly.
[14:47.400 --> 14:49.400]  There's also something called login tax.
[14:49.400 --> 14:59.320]  Depending on your level of license that you pay for, you get different, I guess, granularity, different level of security details in the logs.
[14:59.320 --> 15:00.180]  And that's a big problem.
[15:00.180 --> 15:03.580]  So you essentially have to pay premium for better security.
[15:03.580 --> 15:06.220]  There was this great quote from a U.S.
[15:06.220 --> 15:13.980]  senator that charging people for premium features, necessary not to get hacked, is like selling car and then charging extra for seat belts and airbags.
[15:13.980 --> 15:16.300]  I completely agree with that.
[15:16.340 --> 15:18.240]  I understand that Microsoft is a business.
[15:18.240 --> 15:19.540]  They need to make money.
[15:19.540 --> 15:22.560]  But this is kind of, I think, a wrong place to make money.
[15:22.560 --> 15:25.680]  You need to make sure all of your customers are really secure.
[15:25.720 --> 15:29.620]  And there was a recent incident that really kicked things into high gear.
[15:29.620 --> 15:32.000]  CISA got involved because a bunch of U.S.
[15:32.000 --> 15:35.300]  departments got hacked in Azure.
[15:35.320 --> 15:43.060]  And I think they pushed Microsoft to actually expand access to these types of logs that are not visible to ordinary users.
[15:43.060 --> 15:43.980]  And that's amazing.
[15:43.980 --> 15:45.820]  I hope this trend continues.
[15:46.380 --> 15:50.780]  So for defenders, not much to suggest here.
[15:50.780 --> 15:56.000]  If things that you want are paid, you have a difficult choice.
[15:56.000 --> 16:01.320]  You'd have to pay extra or agree with not fully seeing everything.
[16:02.460 --> 16:09.580]  While these features are still being developed and coming online, attackers can take advantage of some things being hidden from the majority of the customers.
[16:09.580 --> 16:14.260]  Because not everybody can afford something like $37 per user or something.
[16:14.740 --> 16:18.220]  It's a big expense to get those extra features.
[16:18.220 --> 16:23.120]  And Azure, please, don't charge extra for such a basic security feature.
[16:23.120 --> 16:28.040]  I would say documentation on logs is generally deficient.
[16:28.040 --> 16:32.860]  The best you can probably hope for is the names of operations that you'll see in the logs.
[16:32.860 --> 16:39.340]  But if you actually go into the actual log record and start looking at the individual fields, most of them are a mystery.
[16:39.900 --> 16:45.740]  Individual, let's say, enum values, you get authentication code 278.
[16:45.740 --> 16:52.000]  And you look online and there's nothing there and you're lost.
[16:52.000 --> 16:55.560]  You have to hunt through blogs and ask questions and discussion boards.
[16:55.560 --> 16:57.120]  And this is fine.
[16:57.120 --> 16:59.840]  I understand that it's just probably part of your daily work.
[16:59.840 --> 17:02.100]  But as a defender, you shouldn't be doing that.
[17:02.100 --> 17:10.320]  You should be looking at the logs and analyzing incidents, not digging into all kinds of blogs just to find bits of information.
[17:10.320 --> 17:16.280]  Microsoft really needs to document everything that they log to help us as defenders.
[17:16.280 --> 17:20.560]  And attackers, again, any defender's loss is an attacker's gain.
[17:20.560 --> 17:25.060]  You can't really hide in cases that are not well documented.
[17:25.740 --> 17:29.140]  I find that Microsoft does a lot of changes that are unannounced.
[17:29.280 --> 17:35.540]  In my experience, I was at one point monitoring mailbox login events in Exchange logs.
[17:35.540 --> 17:37.440]  And then all of a sudden they disappeared.
[17:37.480 --> 17:40.300]  And I noticed maybe several months later.
[17:40.300 --> 17:41.760]  Because it was just completely silent.
[17:41.760 --> 17:43.120]  So you have your queries running.
[17:43.120 --> 17:44.380]  They're not finding anything.
[17:44.380 --> 17:47.160]  And that's because that event is not logged anymore.
[17:47.160 --> 17:56.100]  Or I had queries that detected attackers trying to enumerate users coming from the outside of the enterprise.
[17:56.100 --> 18:02.620]  They would try different combinations of usernames, like a first name, dot, last name, or last name, dot, first name, etc., etc.
[18:02.620 --> 18:04.620]  They tried those enumeration attacks.
[18:04.620 --> 18:05.800]  I used to see those logs.
[18:05.800 --> 18:06.880]  They're gone now.
[18:06.880 --> 18:09.380]  Again, silently, without announcement.
[18:09.420 --> 18:10.900]  It's kind of upsetting.
[18:10.940 --> 18:13.780]  And also we had a case underlined there.
[18:13.780 --> 18:14.960]  You can see it in the slides.
[18:14.960 --> 18:15.900]  There's a link.
[18:15.900 --> 18:29.480]  We documented a case where passwordless authentication, when it was introduced and encouraged, the error codes that you get for that are completely different from anything you got before.
[18:29.820 --> 18:30.560]  Undocumented.
[18:30.560 --> 18:32.460]  We discovered them by accident.
[18:33.340 --> 18:35.260]  Just didn't know how to react to that.
[18:35.260 --> 18:39.720]  But this is important if you're doing threat hunting and responding to incidents.
[18:40.180 --> 18:44.360]  So it's really the worst kind of failure when you don't know that you have a problem.
[18:44.360 --> 18:53.100]  So for anonymous changes, I don't know how much defenders can do rather than just study and continuously look at the logs and learn.
[18:53.100 --> 18:54.880]  But it shouldn't be this hard.
[18:54.880 --> 18:56.740]  I think Microsoft can do better.
[18:56.860 --> 19:03.080]  And again, anything that's difficult for defenders, attackers can put in their TTPs.
[19:03.540 --> 19:06.620]  So cloud is a recon paradise.
[19:07.020 --> 19:11.540]  Get type events, when you enumerate something, almost never get logged.
[19:11.540 --> 19:16.400]  I think it's the same thing in AWS and maybe the same thing in other places.
[19:16.720 --> 19:20.740]  So when an attacker gets access, they can look around without you noticing.
[19:20.740 --> 19:21.920]  I actually did an experiment.
[19:21.920 --> 19:28.420]  I ran a bunch of open source Microsoft 365 Azure AD attack tools, the ones that do enumeration.
[19:28.480 --> 19:30.520]  Virtually no signal in the logs.
[19:30.520 --> 19:36.900]  As soon as you try to maybe try different passwords, that's when things start to show up in the logs.
[19:36.900 --> 19:38.640]  Before that, it's silent.
[19:38.640 --> 19:42.940]  So if you're a red teamer, have a ball.
[19:43.000 --> 19:47.940]  As soon as you get in, you're free to look around and not be noticed.
[19:48.040 --> 19:52.720]  Again, Azure doesn't even give you an option to turn this on.
[19:52.980 --> 19:56.000]  I'm sure there's some serious reason for not logging them.
[19:56.000 --> 20:01.100]  Maybe there's space concerns, like the logs will just explode if you log all of these events.
[20:01.100 --> 20:06.780]  But at least give me an option to turn them on if I'm extra careful in my particular environment.
[20:06.780 --> 20:10.320]  I may want to actually turn these things on.
[20:11.620 --> 20:15.020]  There are a bunch of inconsistencies I see in the data that's being logged.
[20:15.020 --> 20:22.560]  So when you run your detection queries, you have to be really careful to make sure that you kind of clean up, let's say, the IDs that you see.
[20:22.560 --> 20:29.340]  Because, as you can see, the same user can be logged with different letter case in different log records.
[20:29.440 --> 20:32.920]  Sometimes the log ID is different altogether.
[20:32.920 --> 20:36.600]  And Exchange really throws a wrench in the works.
[20:36.600 --> 20:42.700]  This huge chunk of texture is an actual user ID that gets logged for some of the users.
[20:42.700 --> 20:55.020]  So you can imagine, if you're investigating an incident and you're trying to consolidate different logs and tell a story of what happened, it's really difficult to connect the dots if you just want to join on username.
[20:55.120 --> 20:58.860]  There are unique IDs available that you may want to...
[20:58.860 --> 21:03.560]  It's probably a better way of consolidating, but again, I don't have to discover this.
[21:03.560 --> 21:05.880]  This really has to be clearly explained to me.
[21:08.180 --> 21:11.180]  Occasionally things are just missing, not available.
[21:11.560 --> 21:15.300]  I'll try to go quicker because I think there's a lot more material.
[21:15.300 --> 21:17.880]  For IP also, lots of inconsistencies.
[21:17.880 --> 21:26.300]  You usually get the clean IP v4 or v6 address, but sometimes you get IPs logged together with ports.
[21:26.480 --> 21:31.300]  So you have to call a port number, you have to cut that off to clean things up.
[21:31.300 --> 21:38.580]  I've seen empty IPs, I've seen local and private IPs, I've seen all zero IPs, I've seen all 255 IPs.
[21:39.080 --> 21:43.020]  None of this is useful, but we still see that.
[21:43.020 --> 21:46.040]  So it's hard to deal with.
[21:47.520 --> 21:52.260]  Geolocation also, it's a useful feature, it's only available in sign-ins log.
[21:52.780 --> 21:55.440]  Sometimes it breaks, sometimes it's empty.
[21:55.440 --> 21:59.380]  You can see here the real record that I pulled last night.
[21:59.640 --> 22:01.240]  Country, state, everything is empty.
[22:01.240 --> 22:03.320]  So that's not helping everyone.
[22:03.320 --> 22:05.360]  I understand that geolocation is kind of tricky.
[22:05.540 --> 22:10.720]  You can be easily fooled, and it does get fooled by Autor, by VPNs, proxies, etc.
[22:10.720 --> 22:14.400]  But when an attacker is not careful, they're revealing their real location.
[22:14.400 --> 22:16.540]  It's helpful in this investigation.
[22:16.540 --> 22:19.820]  So it really has to work consistently.
[22:19.840 --> 22:22.200]  Device information also hit and miss sometimes.
[22:22.200 --> 22:25.440]  I think it's mostly grabbed from the user agent, and it's a mixed bag.
[22:25.440 --> 22:32.400]  You get sometimes a full user agent, sometimes you get pieces of it parsed out, and parsed not always consistently.
[22:32.420 --> 22:35.440]  So some consistency in all of the logs would be great.
[22:36.120 --> 22:39.000]  Sometimes you get broken values in the logs.
[22:39.000 --> 22:42.600]  So as part of your query, you may want to parse a field that's JSON.
[22:43.700 --> 22:51.080]  I think when things run out of space, you actually get broken JSON, as you can see in this example here.
[22:51.080 --> 22:54.020]  These three dots is not something you click on to expand.
[22:54.020 --> 22:57.480]  These are literal three dots at the end of JSON.
[22:57.480 --> 23:00.420]  And of course, your parser will choke on that.
[23:00.660 --> 23:03.220]  So something to be aware of.
[23:03.940 --> 23:10.200]  Again, for all of these, the running lesson is defenders be aware.
[23:10.400 --> 23:14.000]  Try to factor that in your analysis queries.
[23:14.000 --> 23:18.040]  Attackers, maybe something to take advantage of, because these are not some fancy zero days.
[23:18.040 --> 23:25.780]  You could probably add stealth to your actions by just taking advantage of all these little deficiencies.
[23:25.780 --> 23:28.220]  And Azure, please fix stuff.
[23:28.480 --> 23:40.940]  All right, so all of this is mostly about log analysis and the signal you see in the logs, but you could actually occasionally be attacked through the logs.
[23:40.940 --> 23:44.320]  I just want to explain this example here.
[23:44.320 --> 23:50.480]  So log pollution is a type of attack where attacker controls the content in your log maliciously.
[23:50.480 --> 23:58.480]  And it could be coupled with something called CSV injection, which is an Excel formula you could embed in a CSV field.
[23:58.480 --> 24:05.700]  And because most of us open CSVs in Excel, Excel actually interprets those formulas and executes them.
[24:05.700 --> 24:09.280]  And sometimes those formulas can actually run external commands.
[24:09.280 --> 24:20.360]  So there was a case of log pollution with CSV injection that was documented by Rhino Security five years ago in AWS and Azure.
[24:20.360 --> 24:23.300]  I think it got fixed since, but I found another one.
[24:23.300 --> 24:26.820]  And this one is pre-authentication, so I'll just do a demo of that.
[24:26.860 --> 24:39.260]  So the idea is to get a malicious payload in the logs and social engineer the admin to open the log to get this payload to detonate on their machine.
[24:39.260 --> 24:44.220]  It's harder to do these days because Microsoft put some defenses in Microsoft Office.
[24:44.620 --> 24:54.920]  You could do it with the old version of the Office, or you could maybe social engineer admin to drop this particular setting in their Excel.
[24:54.920 --> 24:59.860]  It's not easy to do, so it's not like a zero-click thing, but it's something to be aware of.
[24:59.860 --> 25:06.760]  And when I reported it to Microsoft, they didn't seem really excited about it, so I'm just going to tell you guys about it.
[25:08.160 --> 25:11.800]  So anyway, this is the payload.
[25:11.800 --> 25:14.940]  You may recognize it as a DDE formula from Excel.
[25:14.940 --> 25:18.240]  It just pops a calc because we're all white hats here, right?
[25:19.880 --> 25:27.020]  It attempts to sign in into Azure with an existing account and a bad password, so actually you don't want to log in.
[25:27.020 --> 25:32.500]  You just want to try to log in with that special user agent field.
[25:32.500 --> 25:44.900]  And then when a defender, admin, opens the log as a CSV file, if he has the right version of Office, like the old version of Office, or if these defenses are dropped, it will detonate.
[25:44.900 --> 25:46.520]  So this is the demo.
[25:48.840 --> 25:50.220]  Takes a bit of time.
[25:54.720 --> 26:00.200]  So the first step is to override the user agent with an exploit that I just showed.
[26:06.840 --> 26:14.340]  I think you can do it in pretty much any browser nowadays, or use Zap, or use Burp, it doesn't matter.
[26:25.400 --> 26:28.580]  So there's a field for user agent.
[26:29.220 --> 26:31.260]  It's under network conditions.
[26:31.260 --> 26:37.780]  Instead of using browser default, we use this custom string that we drop in there.
[26:38.760 --> 26:45.040]  And then, the second step, where we try to log in as an existing user.
[27:00.830 --> 27:15.030]  So I type my username, and then I intentionally type the invalid password, so that there's just a record of me trying to log in and failing.
[27:15.030 --> 27:18.110]  But the user agent gets into the log.
[27:18.110 --> 27:19.610]  So this fails.
[27:19.790 --> 27:24.630]  Then I call up the admin and say, hello, this is Microsoft support.
[27:24.630 --> 27:26.890]  I want to help you investigate an incident.
[27:26.890 --> 27:29.170]  We're seeing something fishy in your system.
[27:29.170 --> 27:30.990]  Can you please pull up the logs?
[27:32.210 --> 27:35.730]  It takes a bit of time for the records to actually get there.
[27:41.650 --> 27:47.290]  And then, so admin says, well, Microsoft told me to do this.
[27:47.290 --> 27:48.410]  I must do this.
[27:48.410 --> 27:50.790]  They open the logs in Log Analytics.
[27:51.130 --> 27:54.790]  I added this query to kind of filter things down to make things clearer.
[27:54.790 --> 27:56.350]  You don't have to run any queries.
[27:56.350 --> 27:59.510]  As long as you open that log, that payload should be in there.
[27:59.510 --> 28:01.690]  And you export it as CSV.
[28:01.690 --> 28:03.610]  You open it in Excel.
[28:06.530 --> 28:10.290]  You tell the user to click through these useless prompts.
[28:12.290 --> 28:13.330]  And boom.
[28:13.330 --> 28:14.170]  All right.
[28:18.270 --> 28:19.190]  Thank you.
[28:19.410 --> 28:20.310]  All right.
[28:20.310 --> 28:28.650]  So the lesson here is to kind of be aware that this may happen and attackers maybe try and, you know, integrate this into your tactics.
[28:28.650 --> 28:29.790]  So why is this happening?
[28:29.790 --> 28:30.570]  I have no idea.
[28:30.570 --> 28:31.470]  I'm speculating.
[28:31.470 --> 28:34.870]  I think logs are an afterthought for many software developers.
[28:34.870 --> 28:37.930]  I know when I was developing software, it was an afterthought for us.
[28:38.010 --> 28:41.590]  Features were the focus, and then everything else comes after.
[28:41.650 --> 28:46.370]  Sometimes maybe other fixes take priority to fixing these things.
[28:46.370 --> 28:53.030]  Some of the logs come from legacy products probably, so they just integrate with them and forget about them.
[28:53.090 --> 28:57.370]  And sometimes provider doesn't have an incentive to fix things.
[28:57.370 --> 28:59.850]  But whatever the reason is, I think things have to be fixed.
[28:59.850 --> 29:08.110]  So I think what should happen is full documentation for everything, logs treated as an API, so all the changes announced.
[29:09.210 --> 29:14.590]  You know, you have time to react if you have queries running against the current scheme of the logs.
[29:14.590 --> 29:16.510]  Logs are timely, arrive quickly.
[29:16.530 --> 29:17.510]  Values are clean.
[29:17.510 --> 29:19.370]  This is my Christmas wish list.
[29:19.370 --> 29:26.990]  I don't know how much of it is actually going to happen, but Microsoft, this is what the customers long for.
[29:27.310 --> 29:37.930]  And as a maximum goal, I think this thing should be refactored because it feels like it's pulled together from many different pieces and held together, like with duct tape.
[29:39.450 --> 29:47.810]  Ideally, a refactor is in order because I have seen logs in AWS and Okta, and I think they do a much better job, much cleaner job of doing this.
[29:48.470 --> 29:49.670]  So that's it.
[29:51.250 --> 29:54.110]  Logging, you know, we see it's in the cloud.
[29:54.110 --> 29:55.250]  This has a lot of advantages.
[29:55.250 --> 29:56.190]  It's uniform.
[29:56.190 --> 30:00.170]  It's kind of better understood, probably, than on-prem.
[30:00.170 --> 30:06.350]  But there's a lack of alternatives and no strong incentive for a provider to fix things.
[30:06.350 --> 30:08.690]  As defenders, we just want to be aware of these issues.
[30:08.690 --> 30:13.110]  And red teamers, until the issues are fixed, maybe try to take advantage of them.
[30:13.110 --> 30:13.850]  That's it.
[30:13.850 --> 30:15.910]  I think I'm out of time.
[30:16.970 --> 30:19.350]  Do we have time for questions?
[30:19.570 --> 30:20.170]  Okay .
[30:20.590 --> 30:21.950]  Sure, go ahead.
[30:21.950 --> 30:36.960]  Yeah, I think actually Microsoft monitors for...
[30:37.580 --> 30:38.600]  Oh, yes, sorry.
[30:38.600 --> 30:45.360]  Is there software that can monitor for these things happening and sending alerts?
[30:45.360 --> 30:47.400]  I think Microsoft has logging for that.
[30:47.400 --> 30:52.360]  Other external players have products that monitor for things like that.
[30:52.360 --> 30:55.840]  Our product company I'm working for does that.
[30:56.380 --> 30:59.740]  Or you can roll your own, execute your own queries.
[31:00.180 --> 31:01.600]  One more question.
[31:04.840 --> 31:05.540]  Yes.
[31:05.540 --> 31:11.140]  Do you have a query to turn AWS JSON log back to where it was supposed to be at?
[31:11.140 --> 31:17.660]  No, because when it's logged and sent, it only breaks down the entire AWS logs into whole fields.
[31:18.300 --> 31:21.840]  So the question is, do I have a query for what?
[31:21.840 --> 31:22.740]  I'm sorry, for...
[31:22.740 --> 31:27.060]  For changing Azure logs at AWS.
[31:27.060 --> 31:29.160]  That's back to their AWS JSON format.
[31:29.160 --> 31:33.180]  Because when you get JSON, you just have to call it with them.
[31:34.900 --> 31:36.940]  Okay, I don't think I got that.
[31:36.940 --> 31:39.280]  Maybe we can discuss this outside.
[31:39.280 --> 31:40.240]  You'll explain it to me.