WEBVTT 00:41.340 --> 00:42.960 Thanks for tuning into our talk. 00:42.960 --> 00:45.640 I'm Spencer Cureton, and with me is Alan Sagenovic. 00:45.640 --> 00:52.720 We are on the abuse operations team at Heroku, where we keep users from doing bad things on, to, and from the platform. 00:52.920 --> 00:55.820 We're like mall cops for our corner of the internet. 00:56.360 --> 01:00.120 This talk focuses on abuse and how we hunt for it on Heroku. 01:00.580 --> 01:05.380 So Heroku is a platform as a service that was purchased by Salesforce about 10 years ago. 01:05.380 --> 01:12.900 It provides an easy interface to quickly go from idea to online application without having to manage and configure the underlying infrastructure. 01:13.400 --> 01:20.400 So to put it simply, we run anyone's code on the internet as directed from unknown sources millions of times a day. 01:21.140 --> 01:24.260 Okay, we are RCE as a service. 01:24.340 --> 01:31.980 The part of Heroku we're talking about is referred to as the free tier, or the common runtime, where all you need is an email address to create an account. 01:31.980 --> 01:36.160 Heroku has other offerings that are not like this, but we won't be talking about them today. 01:36.200 --> 01:41.420 Our focus will be on how we defend the common runtime from abusive bad actors. 01:42.540 --> 01:45.680 So just defining abuse is a challenge within itself. 01:46.000 --> 01:49.940 We define abuse as the unintended monetization of anything. 01:50.240 --> 01:58.300 So if you have a free service, a free trial, or any offering that has an angle to extract value, somebody is already trying to figure out a way to make money off of it. 01:58.300 --> 02:04.980 It's a very fine line between a business trying to get off the ground and an abuser trying to take advantage of your services. 02:05.140 --> 02:07.620 Sometimes it's obvious, like theft of resources. 02:07.860 --> 02:09.400 Other times it's not so clear. 02:09.600 --> 02:11.740 For example, think about a social media poll. 02:11.740 --> 02:17.220 It's intended for honest input, but it's gameable because it's not one person, one vote. 02:17.240 --> 02:18.920 It's one account, one vote. 02:18.920 --> 02:22.380 So the person in control of the most accounts wins. 02:22.380 --> 02:27.460 The problem isn't new, the scams aren't new, but we see them leveraging more and more of the cloud. 02:27.780 --> 02:31.480 And we feel like we have a unique perspective that we would like to share. 02:32.640 --> 02:41.500 So in a more practical layer, we see a portion of this bigger abuse picture realized through misuse, malice, and crime on our platform. 02:41.960 --> 02:48.160 Legal documents like the Acceptable Use Policy, the AUP, provide a list of prohibited content and prohibited actions. 02:48.160 --> 02:54.640 So it's the list of don'ts for users to follow, and also acts as a set of guidelines for our team to uphold. 02:55.180 --> 02:58.340 Obviously, this can't cover every type of abuse activity. 02:58.360 --> 03:04.840 And even if it did, it would act as a laundry list for an abuser to know what kind of badness could be performed from the platform. 03:05.440 --> 03:08.580 Let's look at how we group violations of the AUP. 03:09.060 --> 03:12.340 There are unintentional and intentional violations. 03:12.340 --> 03:20.580 Now, unintentional could be a customer that deploys a poorly written app that consumes more than its share of resources, and now they've become a noisy neighbor. 03:21.000 --> 03:23.020 So think about like a runaway web scraper. 03:23.020 --> 03:26.400 Not malicious, but kind of like they were accidentally jerks. 03:26.880 --> 03:31.140 Then we have the intentional abusers, and these are the folks that make this job really interesting. 03:31.480 --> 03:37.040 They have skills that range from script pity to rogue advanced developer, which we call our apex actors. 03:38.160 --> 03:43.600 Skids will be skids and deploy code found on GitHub, YouTube, or whatever their buddy gave them to run. 03:43.860 --> 03:50.520 At times, we'll get what feels like a swarm of abuse centered around a particular app, like a proxy kit or a peer-to-peer app. 03:50.780 --> 03:59.420 Then there are the quick and loud abusers that could care less about being detected and will attempt to stand up a DDoS farm and attack a target as quickly as possible. 03:59.880 --> 04:04.740 Now, the actors that are patient, skilled, and criminal are the more dangerous group. 04:04.740 --> 04:08.740 These actors attempt to iterate past our detection capabilities. 04:08.740 --> 04:17.960 They may try to social engineer support staff in order to gain insight into how they were detected after being suspended, and will try anything to maximize their use of the platform. 04:18.160 --> 04:22.660 Now, these users are not exploiting technical vulnerabilities within the platform. 04:22.740 --> 04:25.980 They take advantage of gaps in business logic. 04:25.980 --> 04:29.020 So technically, everything's working as designed. 04:29.200 --> 04:32.100 The majority are not too advanced, but are very persistent. 04:32.100 --> 04:37.700 So if you like to think about APT, small lowercase a, big capital P. 04:37.760 --> 04:39.700 And again, financially motivated. 04:39.860 --> 04:51.060 Now, persistent here means that after an account has been suspended, the same actor will try to create a new account with a different email address or a sub-inbox variant of the original email address. 04:51.100 --> 04:55.060 We'll cover a few campaigns during the show and tell section of this talk. 04:56.400 --> 05:05.440 Overall, we define our team, Abuse Operations, as a specialized DFIR team that requires a wide view across the business as shown in this slide. 05:05.580 --> 05:12.680 Our goal is to push bad actors from the platform and put a spotlight on abuse patterns for engineering teams and the business to address. 05:12.700 --> 05:18.620 In order to identify those bad actors and stay on top of them, we need to be proactive and hunt. 05:18.660 --> 05:22.000 Now I'll hand it over to Alan to discuss the art of the hunt. 05:22.220 --> 05:22.780 Alan? 05:23.180 --> 05:24.700 Hi, thank you, Spencer. 05:24.700 --> 05:25.540 I'm Alan. 05:25.540 --> 05:29.120 I'm Akam on the Discord, if you want to find me later. 05:29.640 --> 05:36.900 Now, we're quickly going to breeze through this, how we hunt, mostly just to give better context around the next section, which is really going to be the meat of it. 05:37.420 --> 05:38.760 Now, there's a lot on these slides. 05:38.760 --> 05:40.340 I'm not going to talk about every point. 05:40.340 --> 05:43.060 Just kind of the more interesting stuff, the more pertinent stuff. 05:43.060 --> 05:49.380 If you want to hear the philosophy behind this, go look at the talk that we did at that other conference that happened a little earlier this week. 05:50.360 --> 05:58.000 Besides, I think that you guys at the Blue Team Village probably do some semblance of this already. 05:58.000 --> 06:05.100 So we gather suspicious indicators from any of a number of different places, as you can see listed. 06:05.160 --> 06:14.540 For example, an app with an abnormally high CPU would generate some sort of event-based alarm that would give us a hint to go look at it. 06:14.540 --> 06:20.320 On the slide, as you can see, there's a short list of some of the indicators that we collect of the various places we collect them from. 06:20.320 --> 06:31.040 Now, because we're highly instrumented, and because Heroku runs on Heroku, this is how we can actually gain home-field advantage, by putting these internal and external indicators together. 06:31.040 --> 06:35.620 Now, for this moment, just understand that the indicators that we're getting are considered really low-value. 06:35.620 --> 06:40.180 By themselves, they don't have any kind of context, or rarely enough context to actually make a decision. 06:41.800 --> 06:48.200 Now, given a starting suspicious indicator or two, we form some sort of hypothesis about the intent of the actor. 06:48.200 --> 06:52.580 We start looking at related indicators and try to prove or disprove that hypothesis. 06:52.620 --> 07:03.080 Now, each of these indicators is not proof of intent of any kind individually, but as you can see by this list, you can see that they collectively start making a really strong case. 07:03.080 --> 07:08.320 Collection of low-value indicators does become a very good footprint of malicious activity. 07:10.380 --> 07:15.120 To increase the confidence of what we're looking at, we keep asking that question, what else you got? 07:15.120 --> 07:21.980 For instance, a face-rolled email address, like the one shown, is a pretty good indicator that the user doesn't really value the account. 07:22.020 --> 07:25.060 Same with app names, if they got a face-rolled app name, too. 07:25.060 --> 07:29.720 Rumor is a great indicator of badness, though, because it could be just for discarding it. 07:29.720 --> 07:30.940 Test, discard. 07:30.960 --> 07:40.600 However, a face-rolled email address with a collection of face-rolled app names, in addition to multiple attempts to verify a credit card, is worth a deeper look. 07:43.490 --> 07:45.110 We try to establish intent. 07:45.410 --> 07:46.870 There are two goals here. 07:46.870 --> 07:48.590 One is to determine what the actor is trying to do. 07:48.590 --> 07:52.010 And the other is to determine what the actor is trying to do before they do it. 07:52.030 --> 07:57.810 Now, the key piece here is that goal number two is something that we reserve for actors that we've actually seen before. 07:57.810 --> 08:00.030 Actors that have tried to do something bad before. 08:00.030 --> 08:03.630 This is how we avoid a situation of doing pre-crime. 08:03.630 --> 08:08.350 We don't want to necessarily kick people off the platform before they've done bad. 08:08.350 --> 08:15.030 But if we have indicators that they are a same returning actor that has already been declared persona non grata, then maybe we want to know as early as possible. 08:15.030 --> 08:16.490 Now, there is an exception to this. 08:16.550 --> 08:19.630 There is software that we don't let people deploy to the platform. 08:19.790 --> 08:25.710 But we're not talking about that in any great degree, although we'll mention it later in contrast to other things. 08:27.930 --> 08:31.690 Now, making the decision and taking the action is kind of the crux of the matter. 08:32.010 --> 08:36.510 We've collected a bunch of indicators that give us a good idea of what they're trying to do. 08:36.510 --> 08:37.470 So we take the action. 08:37.470 --> 08:42.670 If the intent is bad, then the action is suspension, evidence collection, so on and so forth. 08:42.670 --> 08:46.970 If it's not bad, watch lists, allow lists, or false positive checks. 08:47.750 --> 08:54.190 And sometimes it's not easy to determine what the intent is, in which case we'll throw it back out of the stack. 08:54.190 --> 08:57.870 We'll examine it again should it pop up again in a fresh alert. 08:59.090 --> 09:01.810 We investigate for more indicators. 09:02.470 --> 09:06.650 Now, really the question here is, have we found all the accounts that are associated with the badness? 09:06.810 --> 09:12.950 And have we found all the indicators associated with all the accounts that point to the badness? 09:12.950 --> 09:22.070 We cycle this a couple of different times to make certain that we've got as big a view as possible, even retroactively, so we can pick up early starts, early campaigns, so on and so forth. 09:22.070 --> 09:27.050 And this is all done to strengthen our ability to do automation. 09:30.150 --> 09:37.410 Automation is... we're doing it because, obviously, we never want to see the same abuse again, and this is how we reduce all the toil of all the things that we do. 09:37.590 --> 09:45.230 The automation of the footprint includes the alerting and the actioning on a subset of indicators, including what we call a partial match. 09:46.990 --> 09:49.570 And then we do it all over again. 09:49.810 --> 09:51.910 Now, some of that requires a little bit more. 09:52.110 --> 09:56.370 But in a nutshell, if you're a visual learner, here's what the loop looks like. 09:57.490 --> 09:58.530 Okay? 09:58.910 --> 10:00.370 Let's go next. 10:00.810 --> 10:05.390 Within our tooling, we have... is where our codified indicators live. 10:05.390 --> 10:12.730 So we take the indicators that we've identified, and it could be something as simple as a substring match on a registration of an email address. 10:12.730 --> 10:18.470 Or it could be something more complex, like a Levenstein distance to a particular word, or any of these kinds of things. 10:18.470 --> 10:23.350 And we codify them as, literally, as code into any of these tooling. 10:23.350 --> 10:28.230 Now, the column on the left, we have an event collection agent, similar to a classic SEM. 10:28.230 --> 10:32.630 We have database query tools that'll run automatically or manually when we need to check stuff. 10:32.630 --> 10:34.270 And we have malware analysis tools. 10:34.270 --> 10:41.310 So we'll do our queries within those tools so that we can actually get these indicators, these codified indicators out. 10:42.210 --> 10:46.110 Now, we also have, as everybody, we have a ticket system. 10:46.110 --> 10:49.930 Now, the ticket system is not normally considered part of the tooling in a lot of cases. 10:49.930 --> 11:01.550 We like to consider it part of the tooling because our ever-increasing goal to reduce Toro means that we also have to automate ticketing where it makes sense or where it's possible as well. 11:01.550 --> 11:03.150 And we have our suspension tools. 11:03.230 --> 11:13.070 Suspend at various levels, but the really important thing there is that they're set up in such a way that they have business rules and guardrails so that we don't accidentally suspend the entire world if we make a mistake. 11:14.190 --> 11:16.690 Now, we track two general types of activities. 11:16.690 --> 11:19.670 We track campaigns and prohibited softwares, as we previously said. 11:19.670 --> 11:24.370 Campaigns track actors, or sometimes they're victims, whatever makes more sense. 11:24.370 --> 11:29.550 And it also tracks the custom software that they might write to be able to do the things that they're trying to do. 11:30.230 --> 11:39.370 Whereas prohibited software is tracking malware, tracking C2, tracking known abusive software, including the stuff that has a deploy to Heroku button sitting in GitHub. 11:40.670 --> 11:43.410 Now, ultimately, we're looking to track actors. 11:43.610 --> 11:48.010 There doesn't seem to be a great way to do this, and we're building up towards more of that. 11:48.010 --> 11:57.730 But what we found is that actors really do end up being a collection of TTPs codified as a constellation of indicators, and we call that a footprint. 11:57.890 --> 12:00.850 Now, footprints can overlap, partially or fully. 12:01.230 --> 12:07.890 And how we implement footprint detection is how we detect actor behavior changes, and multiple actors doing the same thing. 12:07.910 --> 12:16.230 Tracking footprints is kind of difficult, though, and we're looking to be improving the process as we go on this year so we can speed up this whole loop. 12:16.230 --> 12:21.510 But even today, with what we have, we can detect pivots. 12:21.930 --> 12:25.710 A pivot is what we call a change to a TTP of a returning actor. 12:25.710 --> 12:27.850 Now, different from a pen tester pivot. 12:28.210 --> 12:36.550 If a footprint is a collection of indicators, then given enough of them, a partial match usually indicates a change in behavior of the same actor. 12:37.190 --> 12:38.390 Actors are people, too. 12:38.430 --> 12:47.270 When an actor gets suspended, they do have this tendency to test how by isolating for variables and making very small changes over time until they get it working again. 12:47.570 --> 12:55.610 So from a threshold detection, like shown, works really well as an accurate way to detect their return and still take some action. 12:56.010 --> 13:02.450 Then updating the footprint with the new indicators, reflecting the change, just keeps that footprint accurate. 13:02.450 --> 13:06.790 And this is designed to make it difficult for a persistent actor to regain that foothold. 13:06.890 --> 13:10.350 And it frustrates the heck out of them, and that's always a pleasure. 13:11.110 --> 13:13.350 We also have what we call a splash. 13:13.970 --> 13:26.510 When we block a particular actor under a footprint, sometimes what we'll see is that we'll see a whole bunch of different footprints or different partial footprint matches against that actor's footprint. 13:26.510 --> 13:27.850 And we call this a splash. 13:27.850 --> 13:29.850 It's like slapping a mud puddle. 13:30.070 --> 13:38.650 It's a whole bunch of different actors that are suddenly trying to get back onto the platform that have actually all been hiding under the same TTP umbrella, the same footprint. 13:39.150 --> 13:41.550 Sometimes they don't even know that they're doing it. 13:41.550 --> 13:52.170 Now, they're trying to... often it turns out that they're sharing TTPs, they're sharing techniques of doing things on the platform through everything from YouTube videos to forums to whatever. 13:52.830 --> 14:08.430 And sometimes when we find the source of those, if they're doing really particular badness, you know, phishing attacks and stuff like that, and watching them discuss how they're going to make changes, if we can keep on top of those changes and add them to the footprint before they're actually even deployed, 14:08.430 --> 14:11.250 then we can keep on top of them as well. 14:11.250 --> 14:13.490 Shout out to our fantastic Threat Intel team. 14:13.490 --> 14:17.530 They kind of help us find all that extra info if that is the need. 14:17.930 --> 14:29.910 Now, when we have all that tooling in place, when pivot detection is working and designed, and when we manage to stop the actor, sometimes even before they've begun, then they only have two choices. 14:29.910 --> 14:33.530 They can either go somewhere else, or they can entirely retool. 14:34.410 --> 14:40.770 Retooling requires them to change everything, because we've got footprints for nearly everything that they're doing. 14:40.770 --> 14:42.230 I don't want to say absolutely everything. 14:42.230 --> 14:42.930 We don't know. 14:42.930 --> 14:44.590 But pretty damn close. 14:44.990 --> 14:48.130 Now, this makes them indistinguishable from a new actor. 14:48.130 --> 14:51.690 But honestly, we don't really care at that point. 14:51.690 --> 14:54.930 If we end up being a show-up actor, so be it. 14:54.930 --> 14:56.470 We'll track them as a new actor. 14:56.470 --> 15:01.350 Sometimes they've actually retooled to the point where they decide to do something completely different than what they've been doing before. 15:02.890 --> 15:04.430 Now, we don't usually... 15:05.390 --> 15:14.970 Because we're pretty thorough in our collection, sometimes we still find evidence that a new actor is actually an old actor retooled, and that's kind of fascinating for different reasons. 15:14.970 --> 15:19.610 And when we find that evidence, we can still put it together and make great work of that. 15:19.610 --> 15:25.350 And sometimes that little tenuous link, as you probably know, is something that's really, really simple. 15:25.350 --> 15:34.470 Something as simple as putting a couple of numbers, the same 1, 2, 3, 4 numbers in there, constantly using four dots in their email address. 15:34.850 --> 15:40.290 Now, given that as our hunt loop, let's go back to Spencer for the show and tell. 15:42.190 --> 15:42.990 All right. 15:42.990 --> 15:43.790 Thanks, Alan. 15:44.190 --> 15:46.930 Let's look at a few campaigns that have touched our hearts. 15:46.970 --> 15:49.530 And, well, we'd like to think we've touched theirs, too. 15:50.230 --> 15:57.690 The first actor appeared on our radar by spinning up 100 apps per user account across multiple accounts, all from free email providers. 15:57.850 --> 16:03.830 So a user account is limited to running five apps until they verify their account by adding a payment method, like the credit card. 16:03.830 --> 16:06.090 After that, 100 apps is the new max. 16:06.190 --> 16:12.110 So here we have new accounts immediately adding a credit card and going full tilt to 100 apps. 16:12.570 --> 16:13.730 That's not normal. 16:13.810 --> 16:15.510 Normal users don't do this. 16:15.510 --> 16:19.150 That activity stands out and gets on our radar fast. 16:19.170 --> 16:22.890 And to top things off, fraudulent credit cards were used to verify each account. 16:22.890 --> 16:25.110 And all of the apps were identical. 16:26.210 --> 16:31.970 So we inspected one of the apps and saw that they were running a Tor hidden service accessible through SSH. 16:31.990 --> 16:35.730 Here's a screenshot of the script that would run once their app was deployed. 16:37.990 --> 16:42.930 We suspended the accounts and very soon the actor was back and we had their attention. 16:44.650 --> 16:49.010 Soon after, we saw their underlying intent, mining Monero. 16:49.010 --> 16:53.990 The apps were mining with LibXMR and sending all traffic through Tor. 16:54.630 --> 17:03.590 Now after seeing this, we believe that the initial wave of apps that were running Tor were really part of some capabilities testing, so some real live bad actor DevOps. 17:04.770 --> 17:11.390 We continued to suspend the accounts and the actor would pivot, each time trying to minimize their footprint to avoid detection. 17:11.410 --> 17:19.790 So the top image is from an app that packed the Tor and mining binaries in a zip file, which would then be unzipped with a unique passphrase at the time of deployment. 17:19.910 --> 17:27.470 The bottom image is from a later pivot, this time using GPG encrypted payloads with unique names and unique passphrases. 17:27.470 --> 17:31.670 So decrypt, unpack, and mine, sending everything through Tor. 17:32.070 --> 17:34.190 Now this user was really fast. 17:34.370 --> 17:40.270 They appeared to be retooling their apps and back online within just four hours of being suspended. 17:41.630 --> 17:46.990 And this type of activity at this scale threatened the performance of some of the shared resources. 17:46.990 --> 17:51.950 Other customers were impacted due to these noisy neighbor situations, so we had to stay on top of this actor. 17:51.950 --> 17:57.030 But they were automating their operation and getting faster, mostly during our off hours. 17:57.150 --> 18:02.030 So playing never-ending whack-a-mole was unsustainable and really a path toward team burnout. 18:02.710 --> 18:08.370 So we took a step back and we looked at all of the data provided from this actor's use of the platform. 18:08.370 --> 18:09.610 Home field advantage, right? 18:09.610 --> 18:14.490 This turned into a list of indicators which we could use to form a TTP footprint. 18:14.950 --> 18:18.030 The actual footprint is very close to what's on this slide. 18:18.250 --> 18:24.750 As you can imagine, to make the most from cryptocurrency mining, the actor needs to spread out across as many accounts as possible. 18:25.210 --> 18:29.710 So to make it worth their time, they try to automate any part of their operation, like account creation. 18:30.050 --> 18:34.250 That kind of automation generates patterns that we can use for detection. 18:34.250 --> 18:47.490 In this case, we look for groups of accounts that signed up from an IP address, that happened to be a Tor exit node, from a particular free email provider, and had an email address that matched a certain naming pattern. 18:47.570 --> 18:53.650 If a new account matched those three indicators, we already have high confidence that it's an account controlled by this actor. 18:53.970 --> 19:02.890 Next, the accounts would quickly create five apps, all with a positive size, and they would leave the names as the default Heroku haiku. 19:02.890 --> 19:11.330 Then sampling some of the apps, we could confirm if they had a GPG blob, and if they had the startup script, which decrypted and unpacked the payload. 19:11.510 --> 19:14.270 Again, normal users don't do this. 19:15.090 --> 19:18.870 So when we joined these indicators together, the false positive rate was zero. 19:18.870 --> 19:20.610 Very high confidence, right? 19:20.770 --> 19:26.990 So now matching these 10 indicators, low value by themselves, gave us a high value decision. 19:26.990 --> 19:32.830 We can now identify this actor without having to pick everything apart, every time, on every pivot. 19:33.030 --> 19:37.930 Just focus on what's different, which saved us a bunch of time, while wasting the actors. 19:38.270 --> 19:41.050 They soon gave up, and appeared to have left the platform. 19:42.350 --> 19:44.090 Let's look at another campaign. 19:44.330 --> 19:47.150 This actor came to our attention through a phishing report. 19:47.210 --> 19:54.230 We suspended their account, and very soon, we saw that the same phish was reported again, so we tracked them as a campaign. 19:54.230 --> 19:58.090 They were targeting customers of a specific insurance provider. 19:58.410 --> 20:06.370 When inspecting one of their apps, we see the following directory structure, which contains message-sending capabilities in the mailer.py file. 20:06.630 --> 20:13.010 Here, they would use third-party messaging add-ons to either distribute their messages or handle their loot. 20:13.730 --> 20:19.710 We suspended the accounts and hunted for more, and found an entirely different app used for the same campaign. 20:19.830 --> 20:24.210 This time, it was a JavaScript React app, which had some added functionality. 20:24.310 --> 20:28.550 Like, now they were going after credit card information, as well as credentials. 20:29.430 --> 20:35.170 This screenshot shows the collection of credit card number, expiration date, and CVV in French. 20:37.630 --> 20:43.310 Really quick, I want to share these screenshots from another pivot, where the actor repurposed somebody else's kit. 20:43.310 --> 20:47.470 You can probably Google for some of these keywords, and still find it out there somewhere on the internet. 20:48.770 --> 20:51.650 We developed a TTP footprint for this actor. 20:51.650 --> 20:58.710 This time, it was more app-focused, since they didn't need to spread out across multiple accounts to profit like the crypto-miner did. 20:58.950 --> 21:05.450 We also used these unique indicators to update rules in our detection and response tooling, just whenever possible, whenever it was appropriate. 21:05.710 --> 21:08.010 But this actor was still very persistent. 21:08.370 --> 21:09.910 We want to break their spirit. 21:09.910 --> 21:13.330 We don't want to help them improve their coding skills as we go back and forth. 21:13.330 --> 21:19.550 Typically, being in proximity to the sensitive information that they were gathering is like being in proximity to toxic waste. 21:19.550 --> 21:22.430 Unless you're the bad actor, then it has value. 21:22.430 --> 21:25.470 But it's only valuable if the information is valid. 21:25.750 --> 21:33.770 So considering this, we did our best to get that information into the appropriate hands, so that passwords, payment information could all get reset and nullified. 21:33.970 --> 21:36.790 Let's strip the value and really waste their time. 21:37.290 --> 21:43.470 Also, we found ways to notify the other service providers that were used in the campaign, which put even more pressure on this actor. 21:43.910 --> 21:48.910 These two things disrupted their operations enough that it appears to have pushed them from the platform. 21:49.330 --> 21:57.410 But some of the external indicators that we used to track them, like what you can find on EARLScan, suggest that they're still active, just not on our platform. 21:59.170 --> 22:01.570 The third campaign is from a couple of years ago. 22:01.570 --> 22:07.930 It started as a DMCA takedown request, claiming that a Heroku app was distributing a copyrighted e-book. 22:10.450 --> 22:18.450 When we checked out the link that was reported, the PDF was not the full e-book, but had the same title and same cover art as the real book. 22:18.670 --> 22:21.730 But it also had a link to download the book on another site. 22:22.150 --> 22:23.430 Nothing weird there, right? 22:23.670 --> 22:29.790 This screenshot is not from what was reported, but it was from one of the other apps that we found while researching this campaign. 22:31.290 --> 22:39.470 When we investigate an app, we're looking for just enough information to establish intent with high confidence, while hunting for unique indicators and general functionality. 22:39.970 --> 22:44.710 Now, with an app like this, we don't have enough time to become subject matter experts and pick everything apart. 22:44.710 --> 22:52.890 So we typically start with something like the index.php file, or look for some other unique files that really stand out, directories. 22:53.010 --> 22:56.930 So in this case, these were some of the files and directories that stood out. 22:56.990 --> 23:01.610 func.php, the document generator PHP files, and the keyw directory. 23:01.610 --> 23:05.150 Now, func is Indonesian for function, by the way. 23:05.150 --> 23:07.450 It's going to be important in a few seconds. 23:07.990 --> 23:11.110 Let's see what's in the keyw directory first. 23:11.810 --> 23:13.610 Lots of text files. 23:15.870 --> 23:18.090 Lots of text files. 23:19.490 --> 23:27.030 Each filled with lots of book titles, from New York Times bestsellers to obscure auto-wiring diagrams. 23:27.090 --> 23:29.670 Now, these screenshots are just from one of the files. 23:29.970 --> 23:33.270 Overall, this app had over 300,000 titles. 23:34.190 --> 23:40.290 To understand how these titles are being used, we went back to the root and searched for the keyw string in the PHP files. 23:40.290 --> 23:46.270 Now, in this screenshot, we see that they were creating sitemap XML files for each book index file. 23:46.610 --> 23:47.370 Huh. 23:48.370 --> 23:51.570 And they would make sitemaps for each book title. 23:52.010 --> 23:56.570 These sitemaps would be sent to search engines for... 23:57.290 --> 23:58.590 Black Hat SEO! 23:59.270 --> 24:05.090 We saw other tactics like cloaking, where search engines would return different results than a normal visitor. 24:05.450 --> 24:07.890 Some apps would perform multiple redirects. 24:07.890 --> 24:14.770 And then they would do link manipulation by taking these long book titles, slicing them up, creating a whole bunch of different links, and sending them back out to the search engines. 24:15.410 --> 24:18.490 And I'm sure there's more, but, I mean, this was more than enough to establish intent. 24:18.490 --> 24:20.870 We don't want this kind of activity on our platform. 24:21.490 --> 24:24.210 Now, we had to see how widespread it was. 24:25.210 --> 24:28.890 Turned out, there were thousands of accounts participating in this campaign. 24:29.470 --> 24:40.850 Many were found just by looking at patterns from the sign-up data, like source IP and user-agent string combinations, as well as the use of email sub-inboxing patterns to create multiple accounts. 24:40.850 --> 24:43.690 So, you know, like how you can move the dot around in a Gmail account? 24:44.210 --> 24:45.210 It's been fixed. 24:45.450 --> 24:50.190 The overall campaign included a variety of apps that were ultimately doing the same thing. 24:50.550 --> 24:56.150 Some were even sniping orphaned subdomains of reputable customers in order to boost their rankings with the search engines. 24:56.210 --> 24:57.550 It's been fixed as well. 24:57.670 --> 25:00.750 So, abusing our systems to abuse the search engine algorithms. 25:00.810 --> 25:02.870 I mean, it's just abuse all the way down. 25:03.450 --> 25:06.030 Now, here's a quick look at some of the other indicators. 25:06.030 --> 25:09.530 Now, we mentioned hunting within the sign-up data. 25:09.530 --> 25:17.270 These accounts were created within the typical working hours of the Indonesian time zone, and only on five days out of the week. 25:17.510 --> 25:23.250 They weren't non-stop sign-ups, but you see these bursts of accounts being created a few times a day within those working days. 25:23.390 --> 25:26.650 Other fun indicators were typos and Indonesian words. 25:26.990 --> 25:34.070 One that had me scratching my head at first was S-U-K-S-E-S, which translates to how it sounds. 25:34.070 --> 25:35.230 Success. 25:35.630 --> 25:38.150 So, we successfully booted them from the platform. 25:39.490 --> 25:46.790 Now, I wish we could dive deeper and share more, but we want to briefly mention some of the lessons that we've learned while fighting abuse on a platform. 25:46.870 --> 25:52.150 First, demotivating a bad actor can be very effective and satisfying. 25:52.610 --> 26:02.970 Know your environment and hunt for internal indicators, no matter how little value they appear to have, because collectively they can show intent, build confidence, and help make a high-value decision. 26:02.970 --> 26:09.110 Also, check if there are ways to include external indicators, like OSINT or threat intelligence, and enrich that data. 26:09.450 --> 26:17.310 For us, not focusing on things like suspension count, but rather focusing on patterns that the bad actors were using to take advantage of our services. 26:17.310 --> 26:25.410 Those patterns are worth reporting back to the business so they can be addressed and fixed, and ultimately create a platform that's less hospitable for abuse. 26:26.750 --> 26:28.290 Thanks again for attending our talk. 26:28.290 --> 26:33.710 If you're interested in more of this, but maybe for more of a philosophical or business approach, check out our Black Hat Talk. 26:33.710 --> 26:34.590 Thank you. 26:35.510 --> 26:36.890 Thank you. 26:36.910 --> 26:39.350 Any questions in text talks in Track 1?