00:00:00.000-->00:00:05.125 >>We're gonna go ahead and start with a number. A dollar seventy-one. This is what it 00:00:05.125-->00:00:10.125 cost Jeremy and I to run a five minute test against Netflix dot com in production against one of 00:00:12.667-->00:00:19.125 our evacuated regions or data centers and cause a five minute service outage. We're gonna 00:00:19.125-->00:00:25.958 kinda go back a little bit in time and start with um this medieval picture here. You can 00:00:25.958-->00:00:30.958 kind of imagine sieging a you know with a battering ram while you're attacking a castle and 00:00:33.917-->00:00:37.333 there's archers sitting on top of the castle right and they're shooting arrows down. And 00:00:37.333-->00:00:41.625 they're picking people off. You can kinda think of it as your firewall right? And you know 00:00:41.625-->00:00:45.667 we're doing our denial of service aka we're trying to brute our way in. And so we're 00:00:45.667-->00:00:52.583 losing a lot of life right like we're losing we're doing work but we're actually getting hurt. 00:00:52.583-->00:00:58.083 This is an image of Genghis Khan uh sieging a castle. Up here are bodies infected with the Bubonic 00:00:58.083-->00:01:03.792 Plague. He's catapulting them over the city walls, or the castle walls. That contagion 00:01:03.792-->00:01:09.167 spreads to it's inhabitants right?. So for a lot less work and a lot less death for his 00:01:09.167-->00:01:16.083 attack basically he's able to cause sort of an amplification, right? And the inhabitants die. 00:01:16.083-->00:01:22.708 And it's much more effective. And so that's really kinda the ethos of the talk today. We're 00:01:22.708-->00:01:27.833 gonna present why application denial service attacks matter and why they're extremely 00:01:27.833-->00:01:32.375 relevant in microservice architectures. We're going to start off with just explaining 00:01:32.375-->00:01:37.667 what application DDoS is. We'll step into an introduction to microservices. Who here is 00:01:37.667-->00:01:42.708 familiar with microservices architectures? That's awesome, great. So we'll quickly go 00:01:42.708-->00:01:47.375 through that. We'll talk about application DDoS in microservices. I'll walk you 00:01:47.375-->00:01:51.667 through a framework we've developed to help you identify application DDoS in your own 00:01:51.667-->00:01:58.500 environments. We'll also be um doing a case study where we'll look at what we did against 00:01:58.500-->00:02:02.875 Netflix dot com and we'll introduce you to the two tools we are open sourcing today 00:02:02.875-->00:02:08.875 Repulsive Grizzly which is our application DDoS framework. And Cloudy Kraken which is our AWS 00:02:08.875-->00:02:14.500 red teaming orchestration framework. We'll then do a demo. And we'll discuss a mitigation 00:02:14.500-->00:02:18.792 strategies and a call to action and some future work ideas we have. So application denial 00:02:18.792-->00:02:25.000 service is just a denial service focus at the application layer logic. And you've probably 00:02:25.000-->00:02:29.583 attended talks that focus on the network layer um you know you've probably heard terms like 00:02:29.583-->00:02:33.917 amplification attacks etcetera. Uh, we've decided to focus on the application layer because 00:02:33.917-->00:02:38.083 we've found that in certain circumstances you can cause applications to become very 00:02:38.083-->00:02:43.083 unstable for a lot less requests. When we're doing research we identified that 00:02:45.458-->00:02:49.167 application DDoS isn't that novel and we've pulled this from the Akamai's state of the 00:02:49.167-->00:02:53.667 internet security report. If you notice in the upper left hand corner here. It accounts for 00:02:53.667-->00:02:58.917 basically point six percent of all DDoS. So it's, it's not very common. And then if we actually 00:02:58.917-->00:03:04.583 look at what kind of application DDoS. If we look at the numbers here. Most of the DDoS that 00:03:04.583-->00:03:11.125 oncomine saw were get requests. So it's just like somebody hitting a URL. And only ten 00:03:11.125-->00:03:16.583 percent of all application DDoS were post. Basically I'm sending a request to the web server. 00:03:16.583-->00:03:21.125 Potentially a little bit more sophistication. So although it's not very common, it's happened. 00:03:21.125-->00:03:25.000 Which means that attackers are privy. They know that this is an exploitation vector and they're 00:03:25.000-->00:03:31.125 using it in certain circumstances. So quick introduction to microservices. 00:03:31.125-->00:03:34.958 Microservice is basically our-a collection of small loosely coupled but collaborative 00:03:34.958-->00:03:39.792 services. So think of them as like really lightweight services that boot quickly um can solve 00:03:39.792-->00:03:45.875 simple problems um you know are often pretty small blobs of codes sometimes are single 00:03:45.875-->00:03:50.083 purpose sometimes are just a few purposes. But the id-i-idea being that instead of having 00:03:50.083-->00:03:55.375 like a giant monolithe right, a huge JAVA application, uh you know you would potentially have 00:03:55.375-->00:04:00.750 an environment where that what once used to be your giant JAVA application is now fifty sixty 00:04:00.750-->00:04:04.500 different services that are kinda connected together. And it provides some really unique 00:04:04.500-->00:04:09.708 benefits for companies that have really large environments and have large customer demands. So 00:04:09.708-->00:04:14.958 who uses microservices? I work for Netflix. So we do. Who-Anybody here work for a 00:04:14.958-->00:04:20.750 company that uses microservices? Awesome. So here's some companies that use it and 00:04:20.750-->00:04:24.208 microservice architectures there's- there's a couple different approaches. We're 00:04:24.208-->00:04:29.042 gonna be focusing today on what's known as the API gateway architecture. So you might work 00:04:29.042-->00:04:32.917 for a company or are familiar with microservice architectures that are more grid based, mesh 00:04:32.917-->00:04:39.417 based. Those'll be outside the scope of the discussion today. So just a quick primer. I 00:04:39.417-->00:04:43.333 mentioned we'll be focusing on an API gateway in-infrastructure. And so you 00:04:43.333-->00:04:47.458 can see in the image here. Basically and API gateway is your single entry point. It's 00:04:47.458-->00:04:51.417 gonna be that system that sits on the edge, or that's internet accessible. And it's gonna 00:04:51.417-->00:04:53.417 provide an interface for your middle tier and backend services. So think of it as like 00:04:53.417-->00:04:55.417 basically just a place where you can invoke calls that will then federate through your middle 00:04:55.417-->00:05:00.417 tier and backend services. And those middle tier and backend services they might provide 00:05:06.792-->00:05:11.167 libraries for the API gateway and maybe those libraries let-let the gateways make rest 00:05:11.167-->00:05:17.542 calls. Maybe it's um GRPC or some other RPC framework. And the specific examples will be 00:05:17.542-->00:05:22.542 discussed when today gets rest based. Another concept I wanted to touch-touch on was circuit 00:05:24.875-->00:05:30.292 breakers. So if we think of the circuit breaker you could of this as that gateway. So then 00:05:30.292-->00:05:34.417 once again a sort of centralized API service and the supplier you could think of as a middle tier 00:05:34.417-->00:05:38.208 service and the client can be like your web browser, right? And so there ends up being like 00:05:38.208-->00:05:42.958 a connection problem betw-at some point in time. And the API gateway starts getting these 00:05:42.958-->00:05:47.083 timeouts and after a certain number of timeouts are triggered it fast fails, right? So it 00:05:47.083-->00:05:50.958 triggers the circuit and it says I'm no longer gonna try to make requests to the middle tier 00:05:50.958-->00:05:55.083 service. I'm just gonna return to generic error. Maybe I'll return some sort of a fallback 00:05:55.083-->00:06:00.792 experience or some-something so that the person using my site gets some value. But the idea 00:06:00.792-->00:06:06.500 being that um it really gives your middle tier services times-time to recover. A couple 00:06:06.500-->00:06:12.458 things you have to take into consideration are how do you know what timeout to choose and 00:06:12.458-->00:06:17.333 how long should the breaker be triggered? Cache is another important concept that's often 00:06:17.333-->00:06:20.833 leveraged in microservice architectures and really the focus here is just to speed up 00:06:20.833-->00:06:25.833 response time. So, um the idea being that if we know what a user or particular use case is. 00:06:28.667-->00:06:33.542 Let's Cache that data upfront, return it very fast. That reduces the load on the services 00:06:33.542-->00:06:38.042 fronted by the Cache and so you ultimately, potentially need less servers in your middle tier 00:06:38.042-->00:06:43.917 and your backend. Okay, Now that we've got the introductions out of the way. I wanna talk about 00:06:43.917-->00:06:48.958 some common application DDoS techniques and application DDoS is not newer novel. Um we've 00:06:48.958-->00:06:54.167 found references from it eh you know a what fifteen or twenty years ago and a lot of the focus 00:06:54.167-->00:06:59.708 has traditionally been on regular IO things like CPU, Mem, Cache, Disk, Network sequel 00:06:59.708-->00:07:04.667 etcetera. But now that our application environments are-are becoming a little more 00:07:04.667-->00:07:09.750 sophisticated there's actually more interesting attack vectors that we should be exploring. 00:07:09.750-->00:07:15.375 Things like queuing and batching like how do middle tier and backend services how do they 00:07:15.375-->00:07:20.292 queue and process requests? What are the library timeouts as-as we mentioned before um you know 00:07:20.292-->00:07:25.583 circuit breakers are-are potentially gonna trigger if-if certain timeouts are hit. So can 00:07:25.583-->00:07:30.750 we take advantage of an application that doesn't tune it's timeouts correctly? What 00:07:30.750-->00:07:35.375 about health checks? Obviously the services need to let each other know when they're healthy. 00:07:35.375-->00:07:39.375 What if we could focus our attacks specifically on the health check? So even if the 00:07:39.375-->00:07:45.000 service was fine, but we can cause the health check to fail, we might be able to cause some 00:07:45.000-->00:07:49.958 service instability. And then things like-that don't auto scale. You can imagine um a 00:07:49.958-->00:07:56.458 giant JAVA d-monolithe or a backend database um and when we say autoscale think of this as a 00:07:56.458-->00:08:02.083 concept of my service has gotten a lot of work and I'm gonna boot more of it. Um and that's really 00:08:02.083-->00:08:06.458 what we mean by auto scaling so as you can imagine a giant database it migh-ni-might not be 00:08:06.458-->00:08:11.083 able to boot a lot of versions of that very quickly so that might be another area you wanna 00:08:11.083-->00:08:17.250 focus your attack on. And so really ah what I wanna drive home to point is that there's a 00:08:17.250-->00:08:21.708 difference here between monolithic denials service and microservice denial service. I 00:08:21.708-->00:08:27.250 would say most monolithic application DDoS is one to one meaning like you're sending in 00:08:27.250-->00:08:31.417 some work to the service and it's kinda happening on that box right. And so it's-you know it's 00:08:31.417-->00:08:36.875 calc-doing some calculation and so it's kinda like a one to one work-work per request raito and 00:08:36.875-->00:08:41.042 now that's-I say it's most because there are monolithic applications where you might 00:08:41.042-->00:08:44.625 actually be able to get a little bit of amplification going. But in general it's gonna be like on 00:08:44.625-->00:08:51.250 a single system. In microservice application DDoS it's often one to many cause if you can imagine 00:08:51.250-->00:08:55.958 we might make a request at that gateway that gonna federate out to tons of middle tier and 00:08:55.958-->00:08:59.792 backend services and if we construct those corre-requests correctly we might be able to 00:08:59.792-->00:09:05.500 cause a lot of work and each one of those services in the middle tier and the backend they have 00:09:05.500-->00:09:08.500 different characteristics they have different health checks they have different timeouts 00:09:08.500-->00:09:13.500 they have different um you know potentially system builds and configurations. There's a lot of 00:09:13.500-->00:09:18.333 things that we might be able to cause havoc on if-if the system can't handle those requests that 00:09:18.333-->00:09:23.375 we are sending through. So here's like a new school microservice API DDoS example. 00:09:23.375-->00:09:29.875 And like here's little Jimmy Wright our '90's kid. Here's Jeremy he's typin with uh gloves 00:09:29.875-->00:09:35.208 on right that's how we all hack right. So the idea being here we have an edge you can think of 00:09:35.208-->00:09:39.250 the edge as basically these are things you'd be able to hit in your browser so we have our 00:09:39.250-->00:09:44.375 proxys our website maybe this is like static assets and what not another proxy. We have our API 00:09:44.375-->00:09:49.417 gateway here. API gateway once again provides an interface to middle tier and backend services 00:09:49.417-->00:09:55.000 right. So we fire up our script it's a Python script let's assume it's I don't know two 00:09:55.000-->00:09:59.917 hundred threads or something we're posting to recommendations and then there's a JSON blob 00:09:59.917-->00:10:06.583 that's says range uh I can't see it says range zero to ten thousand. Okay. So that call 00:10:06.583-->00:10:11.125 flows in and the API gateway starts making many client requests. So let's just assume 00:10:11.125-->00:10:15.792 it starts pegging those middle tier services for those ten thousand recommendations. The 00:10:15.792-->00:10:19.292 middle tier services started making many calls to the backend services maybe to retrieve data. 00:10:19.292-->00:10:25.708 The backend service queues start filling up with expensive requests. And now we've reached 00:10:25.708-->00:10:29.500 that sort of sweet spot right. The client timeouts start happening circuit breakers might 00:10:29.500-->00:10:33.500 trigger and maybe some of the data comes back, maybe not all of it. Maybe a fallback 00:10:33.500-->00:10:37.083 experience is triggered and once again a fallback experience you can kinda think of is like hey I 00:10:37.083-->00:10:41.583 don't know what to do so I'm gonna give you some data to work with. That way you know your 00:10:41.583-->00:10:47.167 customers can still sort of browse the site. Cool. So let's-let's d-get a little bit 00:10:47.167-->00:10:52.167 more into that. Another example same sort of attack. We have a single request asking for a 00:10:52.167-->00:10:57.000 thousand objects not in Cache. Now I mentioned that objects not being Cache is extremely 00:10:57.000-->00:11:03.750 important because Cache is fast it's hella fast. So if-if we're trying to exploit an application 00:11:03.750-->00:11:08.375 DDoS um vulnerability and we're only targeting things that are in Cache we're not gonna be 00:11:08.375-->00:11:13.458 successful. So we have to perform a Cache Miss attack. And now if you-if you Google Cache 00:11:13.458-->00:11:17.500 Miss you're gonna find stuff for like intel processors. This is much higher in the stack. And 00:11:17.500-->00:11:20.542 really all we're doing here is to figure out what's in the Cache. Obviously if you're 00:11:20.542-->00:11:23.417 testing this in your own environment or for your own company you might have a good 00:11:23.417-->00:11:28.417 idea what's in the Cache. Um so figure out what's in the Cache and then just makes calls that 00:11:28.417-->00:11:34.042 require lookups outside the Cache. And often if you specify really large requests, ranges 00:11:34.042-->00:11:39.500 and object sizes you can actually perform this. So we'll stop back for the example again. 00:11:39.500-->00:11:44.000 We have that single request asking for a thousand objects not in the Cache. The middle 00:11:44.000-->00:11:47.083 tier library that we have whatever doesn't support batching so the gateway 00:11:47.083-->00:11:52.750 basically has to make a RPC call to every-for every object we're asking. So since we're asking 00:11:52.750-->00:11:57.667 for a thousand objects and two middle tier services have to be returned for this specific 00:11:57.667-->00:12:02.042 request the result in two thousand RPC calls. Those middle tier services need to call the 00:12:02.042-->00:12:07.208 backend services and they have to call three backend services or six thousand calls. So you 00:12:07.208-->00:12:10.875 started seeing-seeing the-the trend here right. Like there's-there's an opportunity 00:12:10.875-->00:12:15.875 for a lot more requests to happen once we get to the API gateway. So what is the workflow 00:12:18.250-->00:12:22.125 for identifying the application DDoS? This first thing you need to do is identify the most 00:12:22.125-->00:12:26.208 latent service calls. What are the calls that are going to be the most expensive? Touched most 00:12:26.208-->00:12:30.708 middle tier and backend services. And once you identify those you want to investigate 00:12:30.708-->00:12:34.583 gateways to manipulate them. How do we make them more expensive? How do we get those calls to 00:12:34.583-->00:12:41.250 touch more services? Once we've determined those circumstances we want to learn more about the 00:12:41.250-->00:12:45.958 API gateways error conditions like how do we know our attack is working? Um how do we know 00:12:45.958-->00:12:50.750 when we are being um blocked by firewall? What are like the thresholds and timeouts um of 00:12:50.750-->00:12:55.417 the particular clients that we're targeting? And once you build that story up you need to 00:12:55.417-->00:12:58.667 actually tune your payload to fly under the WAF. And we'll ta-discuss a couple techniques 00:12:58.667-->00:13:04.417 you can use to do that. We'll test our hypothesis at a small scale and then we'll scale up 00:13:04.417-->00:13:08.417 our test using the Orchestration framework and our Repulsive Grizzly attack framework which 00:13:08.417-->00:13:14.833 we are open sourcing today. [single person clapping] Thank you. Heh, hehe cool. [Laughing] 00:13:14.833-->00:13:18.583 So um I gotta find latent service calls. Now I-I will admit this is-this is error 00:13:18.583-->00:13:23.000 prone. But this is a-a good first step if you're just kinda getting started with this 00:13:23.000-->00:13:27.875 process. Open up Developer Console in Chrome click the Preserve Log button and just 00:13:27.875-->00:13:32.667 start browsing the site. After some period of time, sort by time. And then look at those 00:13:32.667-->00:13:36.625 requests like those post requests your API gateway. I'll mention why this is a little 00:13:36.625-->00:13:42.417 error prone as you can imagine like just because a call doesn't show up as like latent. Doesn't 00:13:42.417-->00:13:46.250 mean it couldn't be made latent right. So this you might miss some opportunities here. 00:13:46.250-->00:13:49.792 Actually I think a better approach would be to potentially automate this out. You could 00:13:49.792-->00:13:54.083 imagine a spidering tool that would sort of crawl to your applications doing a fair bit of 00:13:54.083-->00:13:57.583 sampling to figure out what calls are the most latent. So I think there's some room for us 00:13:57.583-->00:14:02.583 to improve on ways to actually identify those calls automatically. Oops sorry. It 00:14:07.958-->00:14:12.958 froze. Sorry about that. Can you hold this for a second. I'm like literally, the laptop froze. 00:14:26.833-->00:14:31.833 Sorry about that guys. Gimme just a second here. We have a back up. Yeah, I'm-I'm fully 00:14:33.917-->00:14:38.917 bricked. We good? >>Shoulda got real jobs! [nervous laughter] >>Cool. We're good we got a back 00:14:56.417-->00:15:01.417 up. Yeah. Cool. Thanks guys. Hunter two. [Laughter] Cool. Alright. So let's assume that 00:15:12.750-->00:15:17.542 we've identified an interesting latent service call. Um the first thing I might do is I 00:15:17.542-->00:15:21.875 found this-this call I-I'm not sure if it's interesting or not yet. Um it might be a little bit 00:15:21.875-->00:15:25.250 harder to see in the back so I'll kinda walk through it. Here we have a post request to some 00:15:25.250-->00:15:30.417 licensing end point. It has a bunch of encrypted data and what I di-you know bay sixty four 00:15:30.417-->00:15:34.917 decoded it and started messing with it- it just returned to fast error code. So like there 00:15:34.917-->00:15:38.417 wasn't really anything for me to tweak even though the call was latent changing the IO didn't 00:15:38.417-->00:15:42.500 really result in an increased latency. So this is kind of an example of a call that, nah, 00:15:42.500-->00:15:46.750 might not be a good attack vector. So here's another inter-another call that might be 00:15:46.750-->00:15:50.208 a little bit more interesting. Once again this post to that recommendations endpoint we've 00:15:50.208-->00:15:54.750 kinda been using as our example here. And you'll notice in the JSON body here that there's an 00:15:54.750-->00:16:00.167 array of items. I observe that when I add more items to that list, it resulted in a longer 00:16:00.167-->00:16:05.292 response time. If I added too many items to th-that list I got a special error code. Okay. 00:16:05.292-->00:16:10.833 That's kinda cool. So changing that IO resulted in an increased latency of the API calls. So a 00:16:10.833-->00:16:16.875 more accurate way to find latent service calls is to actually have visibility into what your 00:16:16.875-->00:16:21.208 middle tier and backends services are doing. And this is a dashboard that we have in 00:16:21.208-->00:16:26.333 Netflix that helps us actually identify that. I'm zooming in on three areas that I think are 00:16:26.333-->00:16:30.250 interesting. The first is request per second. So how many times is this particular service 00:16:30.250-->00:16:35.833 being invoked? The next is Cache response. Does this service actually Cache content? And then 00:16:35.833-->00:16:39.542 the most interesting bit to hear is actually just latency. And here it's actually doing a 00:16:39.542-->00:16:44.875 rollup of latency based on ninety percent of requests um and we see we have one call here 00:16:44.875-->00:16:49.083 that averages two seconds. So that's interesting. So a technique that I might use here 00:16:49.083-->00:16:53.458 is I know that these services can be invoked via the API gateway. Now maybe I didn't find 00:16:53.458-->00:16:58.000 em with that original discovery method that I discussed. But I could probably step my way back 00:16:58.000-->00:17:02.042 through. Go back to the documentation on the API gateway and figure out how to invoke 00:17:02.042-->00:17:05.625 those latent service calls. So if you're in a position where you can actually see this. 00:17:05.625-->00:17:10.625 You're gonna have a much higher chance of finding those latent calls. So once we've identified 00:17:13.292-->00:17:17.125 those latent calls let's discuss some attack patterns we can leverage to make those calls 00:17:17.125-->00:17:23.125 more expensive. The first is range. We'll also discuss object out per object in. 00:17:23.125-->00:17:26.500 M-manipulating request size and then just a combination and there's other vectors here but 00:17:26.500-->00:17:31.667 these are the three that we've found to be the most effective. So the first technique is range 00:17:31.667-->00:17:37.500 and you'll see here that we have a request for items once again recommendations and we have a 00:17:37.500-->00:17:42.542 from and a to here. So if we go from one to two what if we change it from one to two 00:17:42.542-->00:17:46.375 hundred to twenty thousand to two million and you're probably thinking to yourself huh this 00:17:46.375-->00:17:52.708 feels a lot like s-like what a scraper might do. Oop. Did it just start doing that? Oh there 00:17:52.708-->00:17:57.708 we go. Cool. It's glitching a little bit. That's-that's too bad. Sorry about that hopefully 00:18:00.917-->00:18:07.458 it doesn't give you a headache back there guys. Um so what we basically observed was you know 00:18:07.458-->00:18:11.958 we can increase the range and this technique is really similar to like what content scrapers 00:18:11.958-->00:18:16.958 use. That is obnoxious man. [Laughing] We'll keep going. And we'll skip over that note. Cool. 00:18:31.750-->00:18:35.625 Alright we're gonna go to the next one. Object out per object in so here we've identified like 00:18:35.625-->00:18:40.625 a direct object reference right so we have an ID here so what if we send more of those in? Yeah 00:18:43.917-->00:18:48.458 can you just see if you can get that working? Thanks. Just reboot it. Cool. Okay so the 00:18:48.458-->00:18:53.292 idea being here that if we enumerate out more objects. If we send more objects in maybe 00:18:53.292-->00:18:58.625 we'll see an increase in response size. Or s-sorry an increase response time. Yeah 00:18:58.625-->00:19:03.625 dude, thank you. Cool. Request size is another technique we could take advantage of so as 00:19:06.750-->00:19:11.167 you can see in the corner here we have like this-this element called art size and it has a 00:19:11.167-->00:19:17.083 range of three forty two by one ninety two um so if we imagine pinning like a zero onto the end 00:19:17.083-->00:19:22.167 of there. If that art size is calculated in real time you could imagine that that might 00:19:22.167-->00:19:26.750 result in increased latency. So once again that's another place we can potentially toggle the 00:19:26.750-->00:19:30.417 switch here and you probably notice that a lot of the steals similar to what a content 00:19:30.417-->00:19:34.083 scraper might do. But you know you're trying to pull a catalog off of some site. You're kinda 00:19:34.083-->00:19:38.250 like manipulating the range of your requests. That's really ultimately what a lot of these 00:19:38.250-->00:19:42.667 techniques are. That's like the same stuff that a content scraper uses. Or you can use a 00:19:42.667-->00:19:46.500 combination right. So we can kinda coggle everything let's just turn all the knobs that we 00:19:46.500-->00:19:50.625 possibly can. What about like languages right. You guys probably noticed like English 00:19:50.625-->00:19:56.958 and Spanish. What if we put like French, Cantonese? Um or what if we touched these object fields 00:19:56.958-->00:20:00.667 that it's obviously looking for description title artwork. Maybe if we put more object fields in 00:20:00.667-->00:20:05.667 there we would touch more microservices. Next thing you wanna do is build a list of 00:20:07.875-->00:20:13.917 indicators on API health. And so as I mentioned before the API gateway um, you know you kinda 00:20:13.917-->00:20:17.375 wanna know if you're-you're attack is being successful. So the first thing is like what's a 00:20:17.375-->00:20:23.958 healthy response probably like HTTP two hundred right. When does your API gateway timeout? 00:20:23.958-->00:20:27.792 What this basically means is like. You API gateway is under so much distress that like it 00:20:27.792-->00:20:32.833 literally cannot function anymore. Specifically in our test example we got a five o two 00:20:32.833-->00:20:36.250 bad gateway. Your environment you might get a five hundred. Maybe you get a stack trace. 00:20:36.250-->00:20:40.125 Maybe you get something on the server. Um but there might be an indicator that your API gateway 00:20:40.125-->00:20:44.042 is not healthy. The next is, what about those middle tier services. What if they're not 00:20:44.042-->00:20:48.750 healthy. Um we might see something like a five o three service unavailable or that 00:20:48.750-->00:20:51.625 might let us know that one of the circuit breakers has actually been triggered. What 00:20:51.625-->00:20:57.250 about a WAF what if we've you know we sent too much work sent too many requests. We're getting 00:20:57.250-->00:21:01.625 blocked. We might get a four o three forbidden. Or the rate limiter so if you're in an 00:21:01.625-->00:21:05.833 environment that actually has a rate limiter, know that that's not very common, but um you know 00:21:05.833-->00:21:10.333 you might you might see something like a four twenty nine. And then framework 00:21:10.333-->00:21:15.125 exceptions. These are kinda interesting and sorta unique and novel like you might end up in a 00:21:15.125-->00:21:18.958 position like the application wanted to do some work but it just literally gave up and said 00:21:18.958-->00:21:23.375 you're asking me for too much work. And then there's other indicators. So if we zoom in 00:21:23.375-->00:21:28.708 here. We got an HTTP two hundred okay so but look at the latency here right. That's like sixteen 00:21:28.708-->00:21:32.375 seconds. And that's a huge response. So you know HTTP two hundred plus latency that's 00:21:32.375-->00:21:37.208 kinda like the Holy Grail. That means like you are causing a lot of work um on the backend. 00:21:37.208-->00:21:40.750 Another thing you might see is like an empty response. You might send something. Might take 00:21:40.750-->00:21:44.750 sixteen seconds to return and then you get nothing back. So there's you'll start gettin 00:21:44.750-->00:21:47.875 kinda these weird errors conditions when you send enough traffic up the services. And 00:21:47.875-->00:21:52.625 obviously look for correlations. Like while you're running your tests. Um you know be browsing 00:21:52.625-->00:21:56.083 your site like are there other systems that impacted that you didn't even th-take into 00:21:56.083-->00:22:02.750 consideration. So once we've kinda built up that latent request. We need to find the 00:22:02.750-->00:22:07.083 sweet spot. And really that's gonna be finding the right balance of number of requests 00:22:07.083-->00:22:12.292 and per the logical work per request cause as we mentioned before we have a lot of knobs we 00:22:12.292-->00:22:17.250 can- we can tweak to make more work happen per request. So you know there's gonna be some spot 00:22:17.250-->00:22:21.417 where the service is healthy right. And there'll be a spot where there's a service 00:22:21.417-->00:22:27.208 impacted. When there's enough requests and enough logical work per requests. Now if we send too 00:22:27.208-->00:22:31.542 many requests too fast or too much um work per request too fast we're gonna get rate 00:22:31.542-->00:22:35.250 limited. And once again you could think of rate limiting as like the firewall might kick in 00:22:35.250-->00:22:40.833 might block us. Now if we don't send a lot of requests but the logical work per request is high 00:22:40.833-->00:22:44.250 the service might scale up right. It might boot more of itself and it might stay 00:22:44.250-->00:22:48.000 healthy. So really our job is to find the skull and crossbones right like we wanna be in a 00:22:48.000-->00:22:52.458 sweet spot. Just under where we're gonna get blocked by the firewall. And just enough where 00:22:52.458-->00:22:57.458 we're gonna cause the service to be disrupted. So a quick case study here, it all started with 00:23:00.167-->00:23:04.417 this HTTP status four thirteen. Has anybody ever seen this status code before? It's kinda 00:23:04.417-->00:23:09.125 new for me. Um the description is the request entity is larger than the server's is willing or 00:23:09.125-->00:23:15.583 able to process. Like DING, that sounds really cool right? It literally gave up. So what we 00:23:15.583-->00:23:19.625 tried to do is figure out how to get rid of the four thirteen. We actually wanted to get like, a 00:23:19.625-->00:23:23.042 better status code. But we actually wanted the server to do the work and not quit right out 00:23:23.042-->00:23:26.833 of the gate. So I had to make the call more expensive. So once again I was kinda tweaking those 00:23:26.833-->00:23:31.125 knobs we discussed in the framework section of the talk. And I got to a spot where I was 00:23:31.125-->00:23:36.125 able to get to a relatively large response size and it was pretty darn latent. Now the next 00:23:38.417-->00:23:42.083 thing that we wanted to do was test it on a smaller scale. And to do that we used Repulsive 00:23:42.083-->00:23:47.583 Grizzly. So Repulsive Grizzly is a Skunkworks application DDoS framework that we are open 00:23:47.583-->00:23:52.375 sourcing today. I mentioned it's Skunkworks because uh you know it's kinda as is. It's 00:23:52.375-->00:23:55.292 definitely not a documented or feature rich as some of the other projects I've open 00:23:55.292-->00:24:00.542 sourced, but the idea being that I was hoping that the you know community kinda build on it and 00:24:00.542-->00:24:05.042 also I'm sure you've probably might have used other denial service tools in the past. The 00:24:05.042-->00:24:08.708 reason we wrote Repulsive Grizzly is we wanted to have a couple of special functions that 00:24:08.708-->00:24:12.667 would help us exploit application DDoS and microservices and we'll actually 00:24:12.667-->00:24:18.667 walk through a couple of those. It uses Eventlet for high concurrency so it's super fast. 00:24:18.667-->00:24:24.250 And it also leverages AWS SNS for logging. So SNS you can think of is kinda like a 00:24:24.250-->00:24:27.875 messaging service and so when we run these attacks and we actually scale them up, we have 00:24:27.875-->00:24:32.042 a place we can write log messages and sort of the health of our attack agents while we're 00:24:32.042-->00:24:36.917 running a test. It's pretty easy to configure too. See here's a mountain of cookies, aka, 00:24:36.917-->00:24:42.500 sessions. Delicious right um what we're trying to do here is bypass the WAF and so one of the 00:24:42.500-->00:24:46.583 techniques that Repulsive Grizzly leverages is it has the ability to Round Robin 00:24:46.583-->00:24:52.208 authentication objects so you might sign up to the site or maybe you know you've generated 00:24:52.208-->00:24:57.667 a bunch of session cookies for a particular application. You can use Repulsive Grizzly to sorta 00:24:57.667-->00:25:03.917 iterate through those and Round Robin them so you can sort of fly under the WAF if needed. So 00:25:03.917-->00:25:07.042 here's a single note test and it might a little bit hard to read in the back so I'll kinda walk 00:25:07.042-->00:25:10.958 you through what's goin on here. So we fired up this specific attack and we're looking at the 00:25:10.958-->00:25:15.542 status codes over here. So we've got some two hundreds and two hundre- oops five o fours and a 00:25:15.542-->00:25:21.000 ton of five o threes and as it goes on more- more five o threes more five o fours and I start 00:25:21.000-->00:25:24.667 getting pretty excited cause I realize at this point in time like I've caused quite a bit of 00:25:24.667-->00:25:28.750 unhealthiness and although there were some two hundreds that were comin through it wasn't very 00:25:28.750-->00:25:32.583 common so as I'm sitting there running the test and we have a few browsers open. I'm 00:25:32.583-->00:25:35.833 refreshing the page and in general I'm just getting site errors. Every once in a while a 00:25:35.833-->00:25:40.625 site would come back but in general it wasn't working very well. So the next step we 00:25:40.625-->00:25:45.417 decided was to develop Cloudy Kraken which is an orchestration framework um and Jer-Jeremy's 00:25:45.417-->00:25:49.125 the author of that and he'll kinda walk you through how he approached it. >>So um, Scott 00:25:49.125-->00:25:54.125 came up with uh an awesome new attack for um you know against the application, but in 00:25:56.625-->00:26:01.833 something like uh Netflix, it's a global service. There are lots and lots of WAFs there's denial 00:26:01.833-->00:26:06.875 service prevention mechanisms so while he can run those from one laptop, that's really not gonna 00:26:06.875-->00:26:11.083 cut it if we wanna try and attack the whole infrastructure. So what do we do? We automate 00:26:11.083-->00:26:15.708 it. So now we're gonna have a whole bunch of Cloudy Krakens running um and and a lot of 00:26:15.708-->00:26:21.542 Repulsive Grizzlys. So what is it? It's a RED team orchestration framework it's 00:26:21.542-->00:26:27.833 written uh in Python, runs on AWS. And some of the key features are um definitely you 00:26:27.833-->00:26:34.542 can get a fresh instance of global fleet every time you want to run the test and this really 00:26:34.542-->00:26:39.750 helps for getting fresh IP address. Uh fresh parts of the world, fresh cider blocks uh to 00:26:39.750-->00:26:44.667 really try and get around what you might normally see with a WAF or DDoS production. So you 00:26:44.667-->00:26:49.833 get lots of good global IP's because a common thing you can do uh for DDoS is to do a 00:26:49.833-->00:26:52.958 velocity base checking so you're watch and see how many one IP address hits and uh you can 00:26:52.958-->00:26:57.417 block based on that. And um another key point is um as you're doing all things attacks 00:26:57.417-->00:27:02.417 you're trying to uh try it out more um different kind of um attacks and configs is-it has 00:27:05.042-->00:27:10.042 all the uh code push and configuration automation built in. So if-when you have a new 00:27:10.042-->00:27:14.958 attack you wanna try out, you just go ahead and run the script again. It will rebuild global 00:27:14.958-->00:27:18.750 fleet and restart your attack. Uh since it is a global attack you wanna make sure that the 00:27:18.750-->00:27:22.958 timing is right so you can be effective and so it can be reproduced because I get is over 00:27:22.958-->00:27:26.667 time, you know it's a great attack tool but what you really wanna do is have it be a 00:27:26.667-->00:27:30.750 regression test. So that you can say every time I'm gonna run it for exactly five minutes at this 00:27:30.750-->00:27:34.833 time of day when I just did this new push of code to the backend and you can get-check and see 00:27:34.833-->00:27:39.042 how your infrastructure can handle it. So um getting all these instances around the world 00:27:39.042-->00:27:42.917 to start at exactly the same time and stop exactly at the same time is a key component. 00:27:42.917-->00:27:47.583 Alright and then lastly we are attacking, in some cases, production environments. So if 00:27:47.583-->00:27:51.667 we maybe make a mistake and attack the wrong one it's good to have an immediate kill switch 00:27:51.667-->00:27:58.458 to shut it all back down. So how many people have worked with AWS? That's a lot of people. So 00:27:58.458-->00:28:04.167 um generally with this uh is kinda the overview of how it works um. In AWS they have 00:28:04.167-->00:28:08.417 different regions around the world which are basically data centers around the world. So we 00:28:08.417-->00:28:13.583 can push out the instances to different regions. We have a single S three bucket a single 00:28:13.583-->00:28:18.917 Dynamo DB uh table that holds a configuration and the code that's actually gonna uh run 00:28:18.917-->00:28:23.667 against the servers in the test. And then like we said before we have the SNS is a Pub/Sub 00:28:23.667-->00:28:29.250 message system so you can send back all the status and it's-uh er status backboard-um dashboard 00:28:29.250-->00:28:35.875 in the backend. So this is kinda the general workflow of how Cloudy Kraken works. Um you put 00:28:35.875-->00:28:40.667 your uh attack code in a GitHub and then it'll update the code, push it out to S three, it'll 00:28:40.667-->00:28:46.542 push it out to Dynamo, it will reset everything and get ready to go. Next it'll actually build 00:28:46.542-->00:28:50.917 all of the environment, so again you get a fresh set of instances uh and and we definitely like to 00:28:50.917-->00:28:55.125 use instances and not docker and the big reason why is cause we can use that enhancement 00:28:55.125-->00:28:59.417 workdrive that you get on ADMC2 instances when they're big. Uh so it'll create all the 00:28:59.417-->00:29:04.958 networking um the the subnets and everything, get all uh IP addresses set up and then it'll 00:29:04.958-->00:29:09.625 launch instances. And then using Cloud in it it, it will config the machines. It will uh get all 00:29:09.625-->00:29:13.542 the machine to download the code, configure themselves and then wait to start the attack 00:29:13.542-->00:29:18.583 when the time hits. So as the attack is actually running uh it starts running all the data 00:29:18.583-->00:29:24.125 through SNS and a big part is at the end when we're uh done with testing we wanna make sure we 00:29:24.125-->00:29:28.125 get back all the log data from the systems so we can actually go back and analyze what the 00:29:28.125-->00:29:33.125 exact results were and uh output that we got. So now that we have the system built, uh we went 00:29:36.583-->00:29:41.667 ahead and ran the test. So we did test it against the production environment of uh a 00:29:41.667-->00:29:48.125 service and we did it using a multiregion and multiagent setup so in this case we were running 00:29:48.125-->00:29:53.250 in four different regions and we're running ten instances per region so for about forty 00:29:53.250-->00:29:59.458 instance overall, globally. And each one had about uh um I think it was two hundred and fifty 00:29:59.458-->00:30:06.083 threads each uh running. So we conducted two different five minute attacks. And then we had 00:30:06.083-->00:30:10.875 a chance to monitor its success and actually uh see how well it worked. So uh from the simple 00:30:10.875-->00:30:14.458 view of our status dashboard you'll see that we have at the bottom we have all the notes 00:30:14.458-->00:30:19.250 showing saying hey I'm online. Uh and then you'll see all the requests going through and what 00:30:19.250-->00:30:22.750 kinda status codes we get. So we-you can immediately see at the top which is kinda hard to 00:30:22.750-->00:30:26.708 see but those are all five o threes and five o fours. So uh a majority of the calls coming 00:30:26.708-->00:30:32.750 back as the test are running are all types of failures. So here's the results of the test. We had 00:30:32.750-->00:30:37.750 eighty percent failure rate. Um and on any sort of large UI or a large uh service like Netflix or 00:30:40.292-->00:30:45.458 Hulu or anything else, you're gonna have um problems if-if you know fifty percent of your calls 00:30:45.458-->00:30:49.542 are failing most of your UI will fail Or you might have parts of your UI which will rely on other 00:30:49.542-->00:30:53.792 parts so, once you get past a certain percentage you're gonna have no user experience and 00:30:53.792-->00:30:59.000 you're effectively offline. So during the attack you can see the first one ran great and we 00:30:59.000-->00:31:03.333 got really good results with that one. Uh the second test again we wrote some new code, 00:31:03.333-->00:31:07.083 pushed it out really quick, tried it again. Wasn't as effective um but this is kinda 00:31:07.083-->00:31:11.542 why we like to have that uh immediate ability to push out new code and retest and and have 00:31:11.542-->00:31:15.542 a high velocity of deployments just like we do with other services and microservices do 00:31:15.542-->00:31:19.750 the AV testing and pushing out new stuff. We're trying to replicate that. And the more dev 00:31:19.750-->00:31:24.333 setups type environment. Uh overall while this was running we had less than one percent of 00:31:24.333-->00:31:28.000 the traffic being blocked. So we effectively between all the cookies, between all the 00:31:28.000-->00:31:31.500 different parts of the world and all the IP addresses we were using we were able to get our 00:31:31.500-->00:31:36.792 attack to go through. And again overall at the time that we ran the test it would've cost about 00:31:36.792-->00:31:40.750 a dollar and seventy one cents to run the whole test to take out the those production service 00:31:40.750-->00:31:44.333 we were attacking. Uh these days, it could probably be a little bit cheaper with spot 00:31:44.333-->00:31:49.375 instances. Uh and overall, depending on the service you're using with AWS, you can actually 00:31:49.375-->00:31:54.958 fit it all into the free tier. So you can probably actually do it for free. [laughing] 00:31:54.958-->00:31:59.958 [clapping] Thank you >>So what failed? Um We had a couple things that I thought were 00:32:04.417-->00:32:09.250 pretty interesting worth discussing. Um we-we identified expensive API calls that we 00:32:09.250-->00:32:13.292 could invoke with non-member cookies and I'll explain what that is. Has anybody sure you've 00:32:13.292-->00:32:17.333 browsed a site you get like a J Session ID before you log in or like a PHP session ID or some 00:32:17.333-->00:32:21.750 sort of session identifier. We actually observed that we could take those and issue them 00:32:21.750-->00:32:25.875 against er against the API in certain circumstances and cau-and actually cause latent 00:32:25.875-->00:32:29.500 calls. So one of the first things I did was I was like I just wrote a slot name script 00:32:29.500-->00:32:34.750 and we dumped like five thousand cookies right and we kinda used those in a Round Robin fashion. 00:32:34.750-->00:32:38.792 That was-it was kind of an interesting finding. Um the expensive traffic resulted in 00:32:38.792-->00:32:43.917 many RPC's. It averaged to be about one call to the API gateway was seventy two hundred 00:32:43.917-->00:32:49.750 RPC calls between middle tier and backend services. And the WAF wasn't able to monitor those 00:32:49.750-->00:32:53.417 middle tiers RPC's it just wasn't configured to look at em. It was you know looking at the 00:32:53.417-->00:32:58.750 gateway but it wasn't actually looking at those middle tier and backend calls. So let's dive in 00:32:58.750-->00:33:03.542 a little bit on how this exactly worked and then we'll show a demonstration. So the first 00:33:03.542-->00:33:07.417 thing is we have our attack agents cycling through multiple session cookies and IP addresses 00:33:07.417-->00:33:13.292 right to bypass the WAF. Each request that we make is asking for seventy two hundred 00:33:13.292-->00:33:19.042 expensive calculations from multiple backends. The objects weren't in the Cache so we were 00:33:19.042-->00:33:22.917 getting Cache Miss'. So each time we had the Cache Miss they had to look-make a call to the 00:33:22.917-->00:33:28.250 service to look that information up. It took about fifteen seconds which returned this huge 00:33:28.250-->00:33:33.250 object score. And the queue kept taking longer and longer. We started noticing that fifteen 00:33:35.375-->00:33:38.292 seconds started to be like eighteen seconds, nineteen seconds and as that queue 00:33:38.292-->00:33:42.708 continued to fill up, uh the service got more and more unhealthy. The middle tier 00:33:42.708-->00:33:46.125 services, I mean sometimes they were returning two hundreds I mean they actually did return. 00:33:46.125-->00:33:50.833 Um but often they were turning some five by six status code five o three, five o four, 00:33:50.833-->00:33:54.875 sometimes just five hundreds. Sometimes it would return a two hundred but it would totally be 00:33:54.875-->00:33:59.833 an exception. Um so that was kinda interesting. And the API gateway had to start responding 00:33:59.833-->00:34:03.917 so it starts triggering breakers. It's the-you know the WAF's kicking in sometimes we're 00:34:03.917-->00:34:08.208 getting four o threes sometimes we're getting five o three, two hundreds, five o fours. But 00:34:08.208-->00:34:13.542 ultimately all these RPC calls were really really slamming the gateway so it knew it needed to 00:34:13.542-->00:34:17.500 start scaling up. But it couldn't really boot itself fast enough during the attack and 00:34:17.500-->00:34:22.958 ultimately the CPU just started to smash on the API gateway and we started getting these gateway 00:34:22.958-->00:34:27.917 timeouts. So at the-we reached a point where the gateway itself could no longer facilitate 00:34:27.917-->00:34:32.917 requests. And demo, alt tab. Okay so at this point in time we've already provisioned our 00:34:36.292-->00:34:41.583 attack environment and we're gonna run um a Kraken attack here. So the first thing we're 00:34:41.583-->00:34:46.792 gonna do is configure the attack uh the number of threads, the instances, what region or data 00:34:46.792-->00:34:51.292 center we want to run it from. Or regions, plural. And so for this specific example we'll run 00:34:51.292-->00:34:57.875 it from US west two. We're gonna run attack one. We'll run twenty threads per attack agent. We'll 00:34:57.875-->00:35:02.875 do seven agents and we'll run the test for two hundred and forty seconds. Cool. Alright so 00:35:04.958-->00:35:09.333 here's our staging environment. This is where we kinda did our testing and you'll see that the 00:35:09.333-->00:35:14.250 site's online. We'll go ahead and go back and pop on the Amazon council here to look and 00:35:14.250-->00:35:17.542 see how our agents are doing. We'll go ahead and hit the refresh button and notice that 00:35:17.542-->00:35:21.042 the agents are in pending states, so they are starting to boot up. And then right here you 00:35:21.042-->00:35:25.292 see the Grizzly tracker I mentioned before. Grizzly tracker is kind of what's 00:35:25.292-->00:35:29.542 listening to that central queue and kinda giving us what's going on with the status codes and the 00:35:29.542-->00:35:33.667 health of everything. Alright so once those agents come online and they're running, we'll go 00:35:33.667-->00:35:38.417 ahead a grab an IP address we'll SSH on. And I'll show you just what's kind of going on on those 00:35:38.417-->00:35:43.042 agents. So it's just booting up at this time. It's installing all the packages it needs. It's 00:35:43.042-->00:35:46.458 pulling down that Repulsive Grizzly attack framework, installing all of it's 00:35:46.458-->00:35:51.333 dependencies. And it's waiting to start the test. Alright so it-the agents are phoning home 00:35:51.333-->00:35:55.417 and they're starting the attack. We see the green coming in. And now the status codes start 00:35:55.417-->00:35:59.333 flowing and I'll pause it here. We'll see some five o threes, twenty six hundreds. So we know 00:35:59.333-->00:36:05.125 we've cause quite a bit of havoc. Um we'll continue to let it go, more five o threes are 00:36:05.125-->00:36:10.125 coming in. We'll pop back up to the site, we'll refresh. Is it healthy? Yes okay alright lets 00:36:13.792-->00:36:18.458 keep going keep going boom. We got a five o four origin re timeout that's pretty good 00:36:18.458-->00:36:23.458 right. So how do you defend against this and mitigate it. I-I think the first and most 00:36:28.583-->00:36:33.625 important step is really to understand what microservices impact your customers experience 00:36:33.625-->00:36:38.750 right like you need to know if you have specific services that if they become unstable, kind of 00:36:38.750-->00:36:43.000 result in a cascading system failure. And once you have a good understanding of what those 00:36:43.000-->00:36:48.000 are you need to put the proper CPU protections in place. A good example, to-a-a good reasonable 00:36:50.375-->00:36:54.333 uh mitigation is actually just to limit the batch and the object size, right, like if I 00:36:54.333-->00:36:58.792 can't make a request to your service that's absolutely ob-obnoxious and abnormal. You 00:36:58.792-->00:37:01.875 know if you're a service that usually returns ten objects and somebody's asking you for one 00:37:01.875-->00:37:05.167 million, you know you should probably have limits in place. Hard limits. And you should 00:37:05.167-->00:37:10.875 enforce those limits on both the client and the server. The rate limiter whenever possible or 00:37:10.875-->00:37:15.292 your web application firewall, it should monitor the middle tier signals. Or the cost of the 00:37:15.292-->00:37:19.458 request right. If we're only monitoring at the edge or at that gateway, we're missing the 00:37:19.458-->00:37:23.042 point. If I'm sending one request on that actually is resulting in seventy two hundred 00:37:23.042-->00:37:27.417 requests. We need to know that, and so we should have the visibility and the insight into 00:37:27.417-->00:37:32.042 those middle tier services, so that we can enforce the right sort of blocks and protections 00:37:32.042-->00:37:37.958 way before we end up in a cascading systems failure sort of world. The rate limiter 00:37:37.958-->00:37:43.792 should also monitor the volume of Cache Miss' and once again if your service has data in Cache. 00:37:43.792-->00:37:48.875 And most of the time you know those objects come from the Cache. It-it either- if you 00:37:48.875-->00:37:51.875 start seeing all these Cache Miss' that means one of two things, your Cache is 00:37:51.875-->00:37:56.875 misconfigured, or somebody's doing something nefarious right. You'll want to prioritize 00:37:59.250-->00:38:03.417 authenticated traffic over unauthenticated and-and as I kind of mentioned before we had 00:38:03.417-->00:38:06.917 these basically unauthenticated sessions and we we're able to use that to take down the 00:38:06.917-->00:38:12.333 service. Performing an authenticated denial service attack is a lot more expensive 00:38:12.333-->00:38:17.042 right. You actually have to get sessions and although there's ways to do that in general. Most 00:38:17.042-->00:38:22.500 likely it's going to be and authenticated attack. You'll want to configure reasonable 00:38:22.500-->00:38:28.000 library timeouts. Right, if you set your timeouts too aggressive um we might be able to trigger 00:38:28.000-->00:38:32.875 the circuit without a lot of work. If you trigger them-if you set the timeouts too lenient, 00:38:32.875-->00:38:36.833 you might be able to cause the services to become unhealthy before you've triggered a 00:38:36.833-->00:38:41.250 circuit breaker, so you have to be conscientious on what your library should be set to. 00:38:41.250-->00:38:48.125 Library timeouts, excuse me. And then finally triggering a fallback experience. And so you 00:38:48.125-->00:38:52.375 can kind of think about fallback experience once again as like your service is super unhealthy 00:38:52.375-->00:38:57.583 maybe you just want to at least return like some sort of generic some sort of experience to your 00:38:57.583-->00:39:03.292 customers so that way they don't just get some XML five o four error when they-when they 00:39:03.292-->00:39:08.292 perform the denial ser-or when the service is really unhealthy. So there's some future work in 00:39:10.750-->00:39:15.333 some areas I think we could explore a little bit more. Um automated identification of 00:39:15.333-->00:39:19.250 potential vulnerable endpoints. I eluded to this a little bit earlier but um it's kind if a 00:39:19.250-->00:39:23.833 manual process for me at this point but I imagine that with enough sampling, enough request 00:39:23.833-->00:39:28.417 sizing and enough sort of munging of the data, maybe we could find a way to identify 00:39:28.417-->00:39:32.667 those latent calls automatically. And then also auto tuning during and attack. 00:39:32.667-->00:39:37.750 As you imagine, as you're conducting a large scale application DDoS attack, things 00:39:37.750-->00:39:41.833 are going to change while the attack is going on the-the WAF's going to kick in. Services are 00:39:41.833-->00:39:47.167 going to go up and go down. Really we should be able to set up success and error criteria in 00:39:47.167-->00:39:51.958 a tool and let the-let the scanner sort of automatically tune how much work or how much 00:39:51.958-->00:39:57.583 requests it's sending when it's running the attack. And then finally I think there's really 00:39:57.583-->00:40:01.167 interesting opportunity for testing common open source microservice frameworks 00:40:01.167-->00:40:05.958 libraries and gateways. I-I would imagine that there's probably more to be explored in 00:40:05.958-->00:40:11.417 this space. And with that I'll say thanks. We are gonna be hangin out in the chillroom 00:40:11.417-->00:40:16.083 after the session which is where registration was, and here are links to pull the source code. 00:40:16.083-->00:40:21.083 Thanks. [clapping]