[00:41.340 --> 00:45.640]  Thanks for tuning into our talk. I'm Spencer Cureton, and with me is Alan Sagenovic.
[00:45.640 --> 00:52.720]  We are on the abuse operations team at Heroku, where we keep users from doing bad things on, to, and from the platform.
[00:52.920 --> 00:55.820]  We're like mall cops for our corner of the internet.
[00:56.360 --> 01:00.120]  This talk focuses on abuse and how we hunt for it on Heroku.
[01:00.580 --> 01:05.380]  So Heroku is a platform as a service that was purchased by Salesforce about 10 years ago.
[01:05.380 --> 01:12.900]  It provides an easy interface to quickly go from idea to online application without having to manage and configure the underlying infrastructure.
[01:13.400 --> 01:20.400]  So to put it simply, we run anyone's code on the internet as directed from unknown sources millions of times a day.
[01:21.140 --> 01:24.260]  Okay, we are RCE as a service.
[01:24.340 --> 01:31.980]  The part of Heroku we're talking about is referred to as the free tier, or the common runtime, where all you need is an email address to create an account.
[01:31.980 --> 01:36.160]  Heroku has other offerings that are not like this, but we won't be talking about them today.
[01:36.200 --> 01:41.420]  Our focus will be on how we defend the common runtime from abusive bad actors.
[01:42.540 --> 01:45.680]  So just defining abuse is a challenge within itself.
[01:46.000 --> 01:49.940]  We define abuse as the unintended monetization of anything.
[01:50.240 --> 01:58.300]  So if you have a free service, a free trial, or any offering that has an angle to extract value, somebody is already trying to figure out a way to make money off of it.
[01:58.300 --> 02:04.980]  It's a very fine line between a business trying to get off the ground and an abuser trying to take advantage of your services.
[02:05.140 --> 02:07.620]  Sometimes it's obvious, like theft of resources.
[02:07.860 --> 02:09.400]  Other times it's not so clear.
[02:09.600 --> 02:11.740]  For example, think about a social media poll.
[02:11.740 --> 02:17.220]  It's intended for honest input, but it's gameable because it's not one person, one vote.
[02:17.240 --> 02:18.920]  It's one account, one vote.
[02:18.920 --> 02:22.380]  So the person in control of the most accounts wins.
[02:22.380 --> 02:27.460]  The problem isn't new, the scams aren't new, but we see them leveraging more and more of the cloud.
[02:27.780 --> 02:31.480]  And we feel like we have a unique perspective that we would like to share.
[02:32.640 --> 02:41.500]  So in a more practical layer, we see a portion of this bigger abuse picture realized through misuse, malice, and crime on our platform.
[02:41.960 --> 02:48.160]  Legal documents like the Acceptable Use Policy, the AUP, provide a list of prohibited content and prohibited actions.
[02:48.160 --> 02:54.640]  So it's the list of don'ts for users to follow, and also acts as a set of guidelines for our team to uphold.
[02:55.180 --> 02:58.340]  Obviously, this can't cover every type of abuse activity.
[02:58.360 --> 03:04.840]  And even if it did, it would act as a laundry list for an abuser to know what kind of badness could be performed from the platform.
[03:05.440 --> 03:08.580]  Let's look at how we group violations of the AUP.
[03:09.060 --> 03:12.340]  There are unintentional and intentional violations.
[03:12.340 --> 03:20.580]  Now, unintentional could be a customer that deploys a poorly written app that consumes more than its share of resources, and now they've become a noisy neighbor.
[03:21.000 --> 03:26.400]  So think about like a runaway web scraper. Not malicious, but kind of like they were accidentally jerks.
[03:26.880 --> 03:31.140]  Then we have the intentional abusers, and these are the folks that make this job really interesting.
[03:31.480 --> 03:37.040]  They have skills that range from script pity to rogue advanced developer, which we call our apex actors.
[03:38.160 --> 03:43.600]  Skids will be skids and deploy code found on GitHub, YouTube, or whatever their buddy gave them to run.
[03:43.860 --> 03:50.520]  At times, we'll get what feels like a swarm of abuse centered around a particular app, like a proxy kit or a peer-to-peer app.
[03:50.780 --> 03:59.420]  Then there are the quick and loud abusers that could care less about being detected and will attempt to stand up a DDoS farm and attack a target as quickly as possible.
[03:59.880 --> 04:04.740]  Now, the actors that are patient, skilled, and criminal are the more dangerous group.
[04:04.740 --> 04:08.740]  These actors attempt to iterate past our detection capabilities.
[04:08.740 --> 04:14.540]  They may try to social engineer support staff in order to gain insight into how they were detected after being suspended,
[04:14.540 --> 04:17.960]  and will try anything to maximize their use of the platform.
[04:18.160 --> 04:22.660]  Now, these users are not exploiting technical vulnerabilities within the platform.
[04:22.740 --> 04:25.980]  They take advantage of gaps in business logic.
[04:25.980 --> 04:29.020]  So technically, everything's working as designed.
[04:29.200 --> 04:32.100]  The majority are not too advanced, but are very persistent.
[04:32.100 --> 04:37.700]  So if you like to think about APT, small lowercase a, big capital P.
[04:37.760 --> 04:39.700]  And again, financially motivated.
[04:39.860 --> 04:43.740]  Now, persistent here means that after an account has been suspended,
[04:43.740 --> 04:51.060]  the same actor will try to create a new account with a different email address or a sub-inbox variant of the original email address.
[04:51.100 --> 04:55.060]  We'll cover a few campaigns during the show and tell section of this talk.
[04:56.400 --> 05:01.460]  Overall, we define our team, Abuse Operations, as a specialized DFIR team
[05:01.460 --> 05:05.440]  that requires a wide view across the business as shown in this slide.
[05:05.580 --> 05:12.680]  Our goal is to push bad actors from the platform and put a spotlight on abuse patterns for engineering teams and the business to address.
[05:12.700 --> 05:18.620]  In order to identify those bad actors and stay on top of them, we need to be proactive and hunt.
[05:18.660 --> 05:22.780]  Now I'll hand it over to Alan to discuss the art of the hunt. Alan?
[05:23.180 --> 05:29.120]  Hi, thank you, Spencer. I'm Alan. I'm Akam on the Discord, if you want to find me later.
[05:29.640 --> 05:35.420]  Now, we're quickly going to breeze through this, how we hunt, mostly just to give better context around the next section,
[05:35.420 --> 05:40.340]  which is really going to be the meat of it. Now, there's a lot on these slides. I'm not going to talk about every point.
[05:40.340 --> 05:43.060]  Just kind of the more interesting stuff, the more pertinent stuff.
[05:43.060 --> 05:49.380]  If you want to hear the philosophy behind this, go look at the talk that we did at that other conference that happened a little earlier this week.
[05:50.360 --> 05:58.000]  Besides, I think that you guys at the Blue Team Village probably do some semblance of this already.
[05:58.000 --> 06:05.100]  So we gather suspicious indicators from any of a number of different places, as you can see listed.
[06:05.160 --> 06:14.540]  For example, an app with an abnormally high CPU would generate some sort of event-based alarm that would give us a hint to go look at it.
[06:14.540 --> 06:20.320]  On the slide, as you can see, there's a short list of some of the indicators that we collect of the various places we collect them from.
[06:20.320 --> 06:28.240]  Now, because we're highly instrumented, and because Heroku runs on Heroku, this is how we can actually gain home-field advantage,
[06:28.240 --> 06:31.040]  by putting these internal and external indicators together.
[06:31.040 --> 06:35.620]  Now, for this moment, just understand that the indicators that we're getting are considered really low-value.
[06:35.620 --> 06:40.180]  By themselves, they don't have any kind of context, or rarely enough context to actually make a decision.
[06:41.800 --> 06:48.200]  Now, given a starting suspicious indicator or two, we form some sort of hypothesis about the intent of the actor.
[06:48.200 --> 06:52.580]  We start looking at related indicators and try to prove or disprove that hypothesis.
[06:52.620 --> 06:59.740]  Now, each of these indicators is not proof of intent of any kind individually, but as you can see by this list,
[06:59.740 --> 07:03.080]  you can see that they collectively start making a really strong case.
[07:03.080 --> 07:08.320]  Collection of low-value indicators does become a very good footprint of malicious activity.
[07:10.380 --> 07:15.120]  To increase the confidence of what we're looking at, we keep asking that question, what else you got?
[07:15.120 --> 07:21.980]  For instance, a face-rolled email address, like the one shown, is a pretty good indicator that the user doesn't really value the account.
[07:22.020 --> 07:25.060]  Same with app names, if they got a face-rolled app name, too.
[07:25.060 --> 07:29.720]  Rumor is a great indicator of badness, though, because it could be just for discarding it.
[07:29.720 --> 07:30.940]  Test, discard.
[07:30.960 --> 07:35.120]  However, a face-rolled email address with a collection of face-rolled app names,
[07:35.700 --> 07:40.600]  in addition to multiple attempts to verify a credit card, is worth a deeper look.
[07:43.490 --> 07:45.110]  We try to establish intent.
[07:45.410 --> 07:48.590]  There are two goals here. One is to determine what the actor is trying to do.
[07:48.590 --> 07:52.010]  And the other is to determine what the actor is trying to do before they do it.
[07:52.030 --> 07:57.810]  Now, the key piece here is that goal number two is something that we reserve for actors that we've actually seen before.
[07:57.810 --> 08:00.030]  Actors that have tried to do something bad before.
[08:00.030 --> 08:03.630]  This is how we avoid a situation of doing pre-crime.
[08:03.630 --> 08:08.350]  We don't want to necessarily kick people off the platform before they've done bad.
[08:08.350 --> 08:12.950]  But if we have indicators that they are a same returning actor that has already been declared persona non grata,
[08:12.950 --> 08:15.030]  then maybe we want to know as early as possible.
[08:15.030 --> 08:16.490]  Now, there is an exception to this.
[08:16.550 --> 08:19.630]  There is software that we don't let people deploy to the platform.
[08:19.790 --> 08:25.710]  But we're not talking about that in any great degree, although we'll mention it later in contrast to other things.
[08:27.930 --> 08:31.690]  Now, making the decision and taking the action is kind of the crux of the matter.
[08:32.010 --> 08:36.510]  We've collected a bunch of indicators that give us a good idea of what they're trying to do.
[08:36.510 --> 08:42.670]  So we take the action. If the intent is bad, then the action is suspension, evidence collection, so on and so forth.
[08:42.670 --> 08:46.970]  If it's not bad, watch lists, allow lists, or false positive checks.
[08:47.750 --> 08:54.190]  And sometimes it's not easy to determine what the intent is, in which case we'll throw it back out of the stack.
[08:54.190 --> 08:57.870]  We'll examine it again should it pop up again in a fresh alert.
[08:59.090 --> 09:01.810]  We investigate for more indicators.
[09:02.470 --> 09:06.650]  Now, really the question here is, have we found all the accounts that are associated with the badness?
[09:06.810 --> 09:12.950]  And have we found all the indicators associated with all the accounts that point to the badness?
[09:12.950 --> 09:17.290]  We cycle this a couple of different times to make certain that we've got as big a view as possible,
[09:17.290 --> 09:22.070]  even retroactively, so we can pick up early starts, early campaigns, so on and so forth.
[09:22.070 --> 09:27.050]  And this is all done to strengthen our ability to do automation.
[09:30.150 --> 09:33.970]  Automation is... we're doing it because, obviously, we never want to see the same abuse again,
[09:33.970 --> 09:37.410]  and this is how we reduce all the toil of all the things that we do.
[09:37.590 --> 09:42.930]  The automation of the footprint includes the alerting and the actioning on a subset of indicators,
[09:42.930 --> 09:45.230]  including what we call a partial match.
[09:46.990 --> 09:49.570]  And then we do it all over again.
[09:49.810 --> 09:51.910]  Now, some of that requires a little bit more.
[09:52.110 --> 09:56.370]  But in a nutshell, if you're a visual learner, here's what the loop looks like.
[09:57.490 --> 10:00.370]  Okay? Let's go next.
[10:00.810 --> 10:05.390]  Within our tooling, we have... is where our codified indicators live.
[10:05.390 --> 10:09.830]  So we take the indicators that we've identified, and it could be something as simple as a substring match
[10:09.830 --> 10:12.730]  on a registration of an email address.
[10:12.730 --> 10:16.950]  Or it could be something more complex, like a Levenstein distance to a particular word,
[10:16.950 --> 10:18.470]  or any of these kinds of things.
[10:18.470 --> 10:23.350]  And we codify them as, literally, as code into any of these tooling.
[10:23.350 --> 10:28.230]  Now, the column on the left, we have an event collection agent, similar to a classic SEM.
[10:28.230 --> 10:32.630]  We have database query tools that'll run automatically or manually when we need to check stuff.
[10:32.630 --> 10:34.270]  And we have malware analysis tools.
[10:34.270 --> 10:39.610]  So we'll do our queries within those tools so that we can actually get these indicators,
[10:39.610 --> 10:41.310]  these codified indicators out.
[10:42.210 --> 10:46.110]  Now, we also have, as everybody, we have a ticket system.
[10:46.110 --> 10:49.930]  Now, the ticket system is not normally considered part of the tooling in a lot of cases.
[10:49.930 --> 10:55.750]  We like to consider it part of the tooling because our ever-increasing goal to reduce Toro
[10:55.750 --> 11:01.550]  means that we also have to automate ticketing where it makes sense or where it's possible as well.
[11:01.550 --> 11:03.150]  And we have our suspension tools.
[11:03.230 --> 11:07.210]  Suspend at various levels, but the really important thing there is that they're set up in such a way
[11:07.210 --> 11:11.550]  that they have business rules and guardrails so that we don't accidentally suspend the entire world
[11:11.550 --> 11:13.070]  if we make a mistake.
[11:14.190 --> 11:16.690]  Now, we track two general types of activities.
[11:16.690 --> 11:19.670]  We track campaigns and prohibited softwares, as we previously said.
[11:19.670 --> 11:24.370]  Campaigns track actors, or sometimes they're victims, whatever makes more sense.
[11:24.370 --> 11:29.550]  And it also tracks the custom software that they might write to be able to do the things that they're trying to do.
[11:30.230 --> 11:34.750]  Whereas prohibited software is tracking malware, tracking C2, tracking known abusive software,
[11:34.750 --> 11:39.370]  including the stuff that has a deploy to Heroku button sitting in GitHub.
[11:40.670 --> 11:43.410]  Now, ultimately, we're looking to track actors.
[11:43.610 --> 11:48.010]  There doesn't seem to be a great way to do this, and we're building up towards more of that.
[11:48.010 --> 11:56.010]  But what we found is that actors really do end up being a collection of TTPs codified as a constellation of indicators,
[11:56.010 --> 11:57.730]  and we call that a footprint.
[11:57.890 --> 12:00.850]  Now, footprints can overlap, partially or fully.
[12:01.230 --> 12:05.990]  And how we implement footprint detection is how we detect actor behavior changes,
[12:05.990 --> 12:07.890]  and multiple actors doing the same thing.
[12:07.910 --> 12:13.190]  Tracking footprints is kind of difficult, though, and we're looking to be improving the process as we go on this year
[12:14.330 --> 12:16.230]  so we can speed up this whole loop.
[12:16.230 --> 12:21.510]  But even today, with what we have, we can detect pivots.
[12:21.930 --> 12:25.710]  A pivot is what we call a change to a TTP of a returning actor.
[12:25.710 --> 12:27.850]  Now, different from a pen tester pivot.
[12:28.210 --> 12:32.110]  If a footprint is a collection of indicators, then given enough of them,
[12:32.210 --> 12:36.550]  a partial match usually indicates a change in behavior of the same actor.
[12:37.190 --> 12:38.390]  Actors are people, too.
[12:38.430 --> 12:43.050]  When an actor gets suspended, they do have this tendency to test how by isolating for variables
[12:43.050 --> 12:47.270]  and making very small changes over time until they get it working again.
[12:47.570 --> 12:53.750]  So from a threshold detection, like shown, works really well as an accurate way to detect their return
[12:53.750 --> 12:55.610]  and still take some action.
[12:56.010 --> 13:00.170]  Then updating the footprint with the new indicators, reflecting the change,
[13:00.170 --> 13:02.450]  just keeps that footprint accurate.
[13:02.450 --> 13:06.790]  And this is designed to make it difficult for a persistent actor to regain that foothold.
[13:06.890 --> 13:10.350]  And it frustrates the heck out of them, and that's always a pleasure.
[13:11.110 --> 13:13.350]  We also have what we call a splash.
[13:13.970 --> 13:16.930]  When we block a particular actor under a footprint,
[13:16.930 --> 13:21.030]  sometimes what we'll see is that we'll see a whole bunch of different footprints
[13:21.030 --> 13:26.510]  or different partial footprint matches against that actor's footprint.
[13:26.510 --> 13:29.850]  And we call this a splash. It's like slapping a mud puddle.
[13:30.070 --> 13:33.830]  It's a whole bunch of different actors that are suddenly trying to get back onto the platform
[13:33.830 --> 13:38.650]  that have actually all been hiding under the same TTP umbrella, the same footprint.
[13:39.150 --> 13:41.550]  Sometimes they don't even know that they're doing it.
[13:41.550 --> 13:45.470]  Now, they're trying to... often it turns out that they're sharing TTPs,
[13:45.470 --> 13:47.910]  they're sharing techniques of doing things on the platform
[13:47.910 --> 13:52.170]  through everything from YouTube videos to forums to whatever.
[13:52.830 --> 13:55.890]  And sometimes when we find the source of those,
[13:55.890 --> 14:00.430]  if they're doing really particular badness, you know, phishing attacks and stuff like that,
[14:00.430 --> 14:03.750]  and watching them discuss how they're going to make changes,
[14:03.750 --> 14:06.310]  if we can keep on top of those changes and add them to the footprint
[14:06.310 --> 14:11.250]  before they're actually even deployed, then we can keep on top of them as well.
[14:11.250 --> 14:13.490]  Shout out to our fantastic Threat Intel team.
[14:13.490 --> 14:17.530]  They kind of help us find all that extra info if that is the need.
[14:17.930 --> 14:21.430]  Now, when we have all that tooling in place,
[14:21.430 --> 14:23.350]  when pivot detection is working and designed,
[14:23.350 --> 14:27.490]  and when we manage to stop the actor, sometimes even before they've begun,
[14:27.490 --> 14:29.910]  then they only have two choices.
[14:29.910 --> 14:33.530]  They can either go somewhere else, or they can entirely retool.
[14:34.410 --> 14:37.110]  Retooling requires them to change everything,
[14:37.110 --> 14:40.770]  because we've got footprints for nearly everything that they're doing.
[14:40.770 --> 14:42.930]  I don't want to say absolutely everything. We don't know.
[14:42.930 --> 14:44.590]  But pretty damn close.
[14:44.990 --> 14:48.130]  Now, this makes them indistinguishable from a new actor.
[14:48.130 --> 14:51.690]  But honestly, we don't really care at that point.
[14:51.690 --> 14:54.930]  If we end up being a show-up actor, so be it.
[14:54.930 --> 14:56.470]  We'll track them as a new actor.
[14:56.470 --> 14:59.510]  Sometimes they've actually retooled to the point where they decide to do something completely different
[14:59.510 --> 15:01.350]  than what they've been doing before.
[15:02.890 --> 15:04.430]  Now, we don't usually...
[15:05.390 --> 15:08.410]  Because we're pretty thorough in our collection,
[15:08.410 --> 15:12.370]  sometimes we still find evidence that a new actor is actually an old actor retooled,
[15:12.370 --> 15:14.970]  and that's kind of fascinating for different reasons.
[15:14.970 --> 15:19.610]  And when we find that evidence, we can still put it together and make great work of that.
[15:19.610 --> 15:23.010]  And sometimes that little tenuous link, as you probably know,
[15:23.010 --> 15:25.350]  is something that's really, really simple.
[15:25.350 --> 15:27.790]  Something as simple as putting a couple of numbers,
[15:27.790 --> 15:30.790]  the same 1, 2, 3, 4 numbers in there,
[15:31.690 --> 15:34.470]  constantly using four dots in their email address.
[15:34.850 --> 15:40.290]  Now, given that as our hunt loop, let's go back to Spencer for the show and tell.
[15:42.190 --> 15:43.790]  All right. Thanks, Alan.
[15:44.190 --> 15:46.930]  Let's look at a few campaigns that have touched our hearts.
[15:46.970 --> 15:49.530]  And, well, we'd like to think we've touched theirs, too.
[15:50.230 --> 15:54.210]  The first actor appeared on our radar by spinning up 100 apps per user account
[15:54.210 --> 15:57.690]  across multiple accounts, all from free email providers.
[15:57.850 --> 16:00.010]  So a user account is limited to running five apps
[16:00.010 --> 16:03.830]  until they verify their account by adding a payment method, like the credit card.
[16:03.830 --> 16:06.090]  After that, 100 apps is the new max.
[16:06.190 --> 16:12.110]  So here we have new accounts immediately adding a credit card and going full tilt to 100 apps.
[16:12.570 --> 16:15.510]  That's not normal. Normal users don't do this.
[16:15.510 --> 16:19.150]  That activity stands out and gets on our radar fast.
[16:19.170 --> 16:22.890]  And to top things off, fraudulent credit cards were used to verify each account.
[16:22.890 --> 16:25.110]  And all of the apps were identical.
[16:26.210 --> 16:31.970]  So we inspected one of the apps and saw that they were running a Tor hidden service accessible through SSH.
[16:31.990 --> 16:35.730]  Here's a screenshot of the script that would run once their app was deployed.
[16:37.990 --> 16:42.930]  We suspended the accounts and very soon the actor was back and we had their attention.
[16:44.650 --> 16:49.010]  Soon after, we saw their underlying intent, mining Monero.
[16:49.010 --> 16:53.990]  The apps were mining with LibXMR and sending all traffic through Tor.
[16:54.630 --> 16:58.790]  Now after seeing this, we believe that the initial wave of apps that were running Tor
[16:58.790 --> 17:03.590]  were really part of some capabilities testing, so some real live bad actor DevOps.
[17:04.770 --> 17:07.750]  We continued to suspend the accounts and the actor would pivot,
[17:07.750 --> 17:11.390]  each time trying to minimize their footprint to avoid detection.
[17:11.410 --> 17:15.810]  So the top image is from an app that packed the Tor and mining binaries in a zip file,
[17:15.810 --> 17:19.790]  which would then be unzipped with a unique passphrase at the time of deployment.
[17:19.910 --> 17:24.870]  The bottom image is from a later pivot, this time using GPG encrypted payloads
[17:24.870 --> 17:27.470]  with unique names and unique passphrases.
[17:27.470 --> 17:31.670]  So decrypt, unpack, and mine, sending everything through Tor.
[17:32.070 --> 17:34.190]  Now this user was really fast.
[17:34.370 --> 17:40.270]  They appeared to be retooling their apps and back online within just four hours of being suspended.
[17:41.630 --> 17:46.990]  And this type of activity at this scale threatened the performance of some of the shared resources.
[17:46.990 --> 17:50.210]  Other customers were impacted due to these noisy neighbor situations,
[17:50.210 --> 17:51.950]  so we had to stay on top of this actor.
[17:51.950 --> 17:57.030]  But they were automating their operation and getting faster, mostly during our off hours.
[17:57.150 --> 18:02.030]  So playing never-ending whack-a-mole was unsustainable and really a path toward team burnout.
[18:02.710 --> 18:08.370]  So we took a step back and we looked at all of the data provided from this actor's use of the platform.
[18:08.370 --> 18:09.610]  Home field advantage, right?
[18:09.610 --> 18:14.490]  This turned into a list of indicators which we could use to form a TTP footprint.
[18:14.950 --> 18:18.030]  The actual footprint is very close to what's on this slide.
[18:18.250 --> 18:21.310]  As you can imagine, to make the most from cryptocurrency mining,
[18:21.310 --> 18:24.750]  the actor needs to spread out across as many accounts as possible.
[18:25.210 --> 18:29.710]  So to make it worth their time, they try to automate any part of their operation, like account creation.
[18:30.050 --> 18:34.250]  That kind of automation generates patterns that we can use for detection.
[18:34.250 --> 18:38.650]  In this case, we look for groups of accounts that signed up from an IP address,
[18:38.650 --> 18:43.130]  that happened to be a Tor exit node, from a particular free email provider,
[18:43.130 --> 18:47.490]  and had an email address that matched a certain naming pattern.
[18:47.570 --> 18:50.210]  If a new account matched those three indicators,
[18:50.210 --> 18:53.650]  we already have high confidence that it's an account controlled by this actor.
[18:53.970 --> 18:58.790]  Next, the accounts would quickly create five apps, all with a positive size,
[18:58.790 --> 19:02.890]  and they would leave the names as the default Heroku haiku.
[19:02.890 --> 19:06.790]  Then sampling some of the apps, we could confirm if they had a GPG blob,
[19:06.790 --> 19:11.330]  and if they had the startup script, which decrypted and unpacked the payload.
[19:11.510 --> 19:14.270]  Again, normal users don't do this.
[19:15.090 --> 19:18.870]  So when we joined these indicators together, the false positive rate was zero.
[19:18.870 --> 19:20.610]  Very high confidence, right?
[19:20.770 --> 19:26.990]  So now matching these 10 indicators, low value by themselves, gave us a high value decision.
[19:26.990 --> 19:30.290]  We can now identify this actor without having to pick everything apart,
[19:30.290 --> 19:32.830]  every time, on every pivot.
[19:33.030 --> 19:36.370]  Just focus on what's different, which saved us a bunch of time,
[19:36.370 --> 19:37.930]  while wasting the actors.
[19:38.270 --> 19:41.050]  They soon gave up, and appeared to have left the platform.
[19:42.350 --> 19:44.090]  Let's look at another campaign.
[19:44.330 --> 19:47.150]  This actor came to our attention through a phishing report.
[19:47.210 --> 19:49.200]  We suspended their account, and very soon,
[19:49.590 --> 19:52.110]  we saw that the same phish was reported again,
[19:52.110 --> 19:54.230]  so we tracked them as a campaign.
[19:54.230 --> 19:58.090]  They were targeting customers of a specific insurance provider.
[19:58.410 --> 20:02.250]  When inspecting one of their apps, we see the following directory structure,
[20:02.250 --> 20:06.370]  which contains message-sending capabilities in the mailer.py file.
[20:06.630 --> 20:08.850]  Here, they would use third-party messaging add-ons
[20:08.850 --> 20:13.010]  to either distribute their messages or handle their loot.
[20:13.730 --> 20:16.250]  We suspended the accounts and hunted for more,
[20:16.250 --> 20:19.710]  and found an entirely different app used for the same campaign.
[20:19.830 --> 20:22.230]  This time, it was a JavaScript React app,
[20:22.230 --> 20:24.210]  which had some added functionality.
[20:24.310 --> 20:26.770]  Like, now they were going after credit card information,
[20:26.770 --> 20:28.550]  as well as credentials.
[20:29.430 --> 20:32.250]  This screenshot shows the collection of credit card number,
[20:32.250 --> 20:35.170]  expiration date, and CVV in French.
[20:37.630 --> 20:40.810]  Really quick, I want to share these screenshots from another pivot,
[20:40.810 --> 20:43.310]  where the actor repurposed somebody else's kit.
[20:43.310 --> 20:45.030]  You can probably Google for some of these keywords,
[20:45.030 --> 20:47.470]  and still find it out there somewhere on the internet.
[20:48.770 --> 20:51.650]  We developed a TTP footprint for this actor.
[20:51.650 --> 20:53.450]  This time, it was more app-focused,
[20:53.450 --> 20:56.550]  since they didn't need to spread out across multiple accounts
[20:56.550 --> 20:58.710]  to profit like the crypto-miner did.
[20:58.950 --> 21:01.210]  We also used these unique indicators to update rules
[21:01.210 --> 21:02.910]  in our detection and response tooling,
[21:02.910 --> 21:05.450]  just whenever possible, whenever it was appropriate.
[21:05.710 --> 21:08.010]  But this actor was still very persistent.
[21:08.370 --> 21:09.910]  We want to break their spirit.
[21:09.910 --> 21:11.650]  We don't want to help them improve their coding skills
[21:11.650 --> 21:13.330]  as we go back and forth.
[21:13.330 --> 21:15.690]  Typically, being in proximity to the sensitive information
[21:15.690 --> 21:19.550]  that they were gathering is like being in proximity to toxic waste.
[21:19.550 --> 21:22.430]  Unless you're the bad actor, then it has value.
[21:22.430 --> 21:25.470]  But it's only valuable if the information is valid.
[21:25.750 --> 21:28.470]  So considering this, we did our best to get that information
[21:28.470 --> 21:31.490]  into the appropriate hands, so that passwords, payment information
[21:31.490 --> 21:33.770]  could all get reset and nullified.
[21:33.970 --> 21:36.790]  Let's strip the value and really waste their time.
[21:37.290 --> 21:39.490]  Also, we found ways to notify the other service providers
[21:39.490 --> 21:41.070]  that were used in the campaign,
[21:41.070 --> 21:43.470]  which put even more pressure on this actor.
[21:43.910 --> 21:46.210]  These two things disrupted their operations enough
[21:46.210 --> 21:48.910]  that it appears to have pushed them from the platform.
[21:49.330 --> 21:52.350]  But some of the external indicators that we used to track them,
[21:52.350 --> 21:54.150]  like what you can find on EARLScan,
[21:54.150 --> 21:57.410]  suggest that they're still active, just not on our platform.
[21:59.170 --> 22:01.570]  The third campaign is from a couple of years ago.
[22:01.570 --> 22:04.170]  It started as a DMCA takedown request,
[22:04.170 --> 22:07.930]  claiming that a Heroku app was distributing a copyrighted e-book.
[22:10.450 --> 22:13.090]  When we checked out the link that was reported,
[22:13.090 --> 22:14.990]  the PDF was not the full e-book,
[22:14.990 --> 22:18.450]  but had the same title and same cover art as the real book.
[22:18.670 --> 22:21.730]  But it also had a link to download the book on another site.
[22:22.150 --> 22:23.430]  Nothing weird there, right?
[22:23.670 --> 22:25.750]  This screenshot is not from what was reported,
[22:25.750 --> 22:27.690]  but it was from one of the other apps that we found
[22:27.690 --> 22:29.790]  while researching this campaign.
[22:31.290 --> 22:32.750]  When we investigate an app,
[22:32.750 --> 22:34.470]  we're looking for just enough information
[22:34.470 --> 22:36.850]  to establish intent with high confidence,
[22:36.850 --> 22:39.470]  while hunting for unique indicators and general functionality.
[22:39.970 --> 22:41.230]  Now, with an app like this,
[22:41.230 --> 22:43.230]  we don't have enough time to become subject matter experts
[22:43.230 --> 22:44.710]  and pick everything apart.
[22:44.710 --> 22:48.850]  So we typically start with something like the index.php file,
[22:48.850 --> 22:51.750]  or look for some other unique files that really stand out,
[22:51.750 --> 22:52.890]  directories.
[22:53.010 --> 22:55.130]  So in this case, these were some of the files
[22:55.130 --> 22:56.930]  and directories that stood out.
[22:56.990 --> 23:00.170]  func.php, the document generator PHP files,
[23:00.170 --> 23:01.610]  and the keyw directory.
[23:01.610 --> 23:05.150]  Now, func is Indonesian for function, by the way.
[23:05.150 --> 23:07.450]  It's going to be important in a few seconds.
[23:07.990 --> 23:11.110]  Let's see what's in the keyw directory first.
[23:11.810 --> 23:13.610]  Lots of text files.
[23:15.870 --> 23:18.090]  Lots of text files.
[23:19.490 --> 23:22.530]  Each filled with lots of book titles,
[23:22.530 --> 23:24.270]  from New York Times bestsellers
[23:24.270 --> 23:27.030]  to obscure auto-wiring diagrams.
[23:27.090 --> 23:29.670]  Now, these screenshots are just from one of the files.
[23:29.970 --> 23:33.270]  Overall, this app had over 300,000 titles.
[23:34.190 --> 23:36.250]  To understand how these titles are being used,
[23:36.250 --> 23:40.290]  we went back to the root and searched for the keyw string in the PHP files.
[23:40.290 --> 23:42.750]  Now, in this screenshot, we see that they were creating
[23:42.750 --> 23:46.270]  sitemap XML files for each book index file.
[23:46.610 --> 23:47.370]  Huh.
[23:48.370 --> 23:51.570]  And they would make sitemaps for each book title.
[23:52.010 --> 23:54.410]  These sitemaps would be sent to search engines
[23:56.050 --> 23:56.570]  for...
[23:57.290 --> 23:58.590]  Black Hat SEO!
[23:59.270 --> 24:01.570]  We saw other tactics like cloaking,
[24:01.570 --> 24:03.670]  where search engines would return different results
[24:03.670 --> 24:05.090]  than a normal visitor.
[24:05.450 --> 24:07.890]  Some apps would perform multiple redirects.
[24:07.890 --> 24:09.070]  And then they would do link manipulation
[24:09.070 --> 24:11.290]  by taking these long book titles, slicing them up,
[24:11.290 --> 24:12.510]  creating a whole bunch of different links,
[24:12.510 --> 24:14.770]  and sending them back out to the search engines.
[24:15.410 --> 24:16.930]  And I'm sure there's more, but, I mean,
[24:16.930 --> 24:18.490]  this was more than enough to establish intent.
[24:18.490 --> 24:20.870]  We don't want this kind of activity on our platform.
[24:21.490 --> 24:24.210]  Now, we had to see how widespread it was.
[24:25.210 --> 24:27.270]  Turned out, there were thousands of accounts
[24:27.270 --> 24:28.890]  participating in this campaign.
[24:29.470 --> 24:32.750]  Many were found just by looking at patterns from the sign-up data,
[24:32.750 --> 24:36.170]  like source IP and user-agent string combinations,
[24:36.610 --> 24:39.150]  as well as the use of email sub-inboxing patterns
[24:39.150 --> 24:40.850]  to create multiple accounts.
[24:40.850 --> 24:42.330]  So, you know, like how you can move the dot around
[24:42.330 --> 24:43.690]  in a Gmail account?
[24:44.210 --> 24:45.210]  It's been fixed.
[24:45.450 --> 24:48.130]  The overall campaign included a variety of apps
[24:48.130 --> 24:50.190]  that were ultimately doing the same thing.
[24:50.550 --> 24:52.450]  Some were even sniping orphaned subdomains
[24:52.450 --> 24:55.030]  of reputable customers in order to boost their rankings
[24:55.030 --> 24:56.150]  with the search engines.
[24:56.210 --> 24:57.550]  It's been fixed as well.
[24:57.670 --> 25:00.750]  So, abusing our systems to abuse the search engine algorithms.
[25:00.810 --> 25:02.870]  I mean, it's just abuse all the way down.
[25:03.450 --> 25:06.030]  Now, here's a quick look at some of the other indicators.
[25:06.030 --> 25:09.530]  Now, we mentioned hunting within the sign-up data.
[25:09.530 --> 25:12.410]  These accounts were created within the typical working hours
[25:12.410 --> 25:14.530]  of the Indonesian time zone,
[25:14.530 --> 25:17.270]  and only on five days out of the week.
[25:17.510 --> 25:19.350]  They weren't non-stop sign-ups, but you see these bursts
[25:19.350 --> 25:21.790]  of accounts being created a few times a day
[25:21.790 --> 25:23.250]  within those working days.
[25:23.390 --> 25:26.650]  Other fun indicators were typos and Indonesian words.
[25:26.990 --> 25:29.370]  One that had me scratching my head at first was
[25:29.370 --> 25:34.070]  S-U-K-S-E-S, which translates to how it sounds.
[25:34.070 --> 25:35.230]  Success.
[25:35.630 --> 25:38.150]  So, we successfully booted them from the platform.
[25:39.490 --> 25:41.810]  Now, I wish we could dive deeper and share more,
[25:41.810 --> 25:43.850]  but we want to briefly mention some of the lessons
[25:43.850 --> 25:46.790]  that we've learned while fighting abuse on a platform.
[25:46.870 --> 25:50.550]  First, demotivating a bad actor can be very effective
[25:50.550 --> 25:52.150]  and satisfying.
[25:52.610 --> 25:54.870]  Know your environment and hunt for internal indicators,
[25:54.870 --> 25:57.350]  no matter how little value they appear to have,
[25:57.350 --> 25:59.750]  because collectively they can show intent,
[25:59.750 --> 26:02.970]  build confidence, and help make a high-value decision.
[26:02.970 --> 26:05.830]  Also, check if there are ways to include external indicators,
[26:05.830 --> 26:07.790]  like OSINT or threat intelligence,
[26:07.790 --> 26:09.110]  and enrich that data.
[26:09.450 --> 26:12.690]  For us, not focusing on things like suspension count,
[26:12.690 --> 26:15.570]  but rather focusing on patterns that the bad actors were using
[26:15.570 --> 26:17.310]  to take advantage of our services.
[26:17.310 --> 26:19.390]  Those patterns are worth reporting back to the business
[26:19.390 --> 26:21.430]  so they can be addressed and fixed,
[26:21.430 --> 26:24.190]  and ultimately create a platform that's less hospitable
[26:24.190 --> 26:25.410]  for abuse.
[26:26.750 --> 26:28.290]  Thanks again for attending our talk.
[26:28.290 --> 26:29.510]  If you're interested in more of this,
[26:29.510 --> 26:32.270]  but maybe for more of a philosophical or business approach,
[26:32.270 --> 26:33.710]  check out our Black Hat Talk.
[26:33.710 --> 26:34.590]  Thank you.
[26:35.510 --> 26:36.890]  Thank you.
[26:36.910 --> 26:39.350]  Any questions in text talks in Track 1?