WEBVTT 00:00.000 --> 00:02.640 Hello, DEF CON Red Team Village. 00:02.640 --> 00:06.100 I want to thank you today for taking the time to listen to this talk. 00:06.100 --> 00:11.380 I've really been looking forward to it, and I'm excited to be able to share this information with you today. 00:11.380 --> 00:17.160 My talk is titled Combining Notebooks, Datasets, and Cloud for the Ultimate Automation Factory. 00:17.660 --> 00:30.160 And all through this presentation, I really want to challenge you to think about pushing the boundaries of the art of the possible, trying to change how we do things in our day-to-day lives, the manual processes, how can we be more efficient? 00:30.160 --> 00:36.000 How can we operationalize and streamline these activities so that we can be more productive in our lives? 00:36.000 --> 00:45.340 And also, as a Red Teamer, as a Blue Teamer, as a security practitioner, how can we better ourselves and our careers and really elevate ourselves to the next step? 00:45.740 --> 00:50.500 I know it's definitely sad that we're not all together in person at DEF CON. 00:50.500 --> 00:52.800 I know I had everything booked, ready to go. 00:52.800 --> 00:54.820 I was excited, looking forward to it. 00:54.820 --> 01:03.260 But I will say, having this virtual offering is something that's really changed the dynamic of information security and the landscape and the hacking and all of this. 01:03.260 --> 01:19.240 It's actually really set a playing field where a lot of people that didn't have the means or methods to attend Vegas or make it to DEF CON or Black Hat, now it kind of sets the bar where having it online, virtual, free, having direct access through Discord, 01:19.240 --> 01:31.260 through Slack, through different channels and mechanisms, to the industry experts, the people that I've looked up to for years who I've learned from, being able to share the stage with them alongside, it's a pretty awesome time. 01:31.260 --> 01:36.860 The other cool thing is, now I can sit in Discord, I can be ready for questions and answers as we're doing this talk. 01:36.860 --> 01:39.040 So feel free, interactively, to do this. 01:44.480 --> 01:52.740 We won't spend a lot of time in the About Me section, but by day, I'm a security architect, and my main focus is cloud security. 01:52.740 --> 01:57.860 So I get to see a lot of use cases around the new, latest, greatest emerging technologies in cloud. 01:57.860 --> 02:06.680 How are they being leveraged to help businesses and help drive innovation and success and actually decrease operating costs and accelerate delivery. 02:06.680 --> 02:12.200 So I'm taking a lot of those examples, and then I'm combining it with kind of my night job, which is security research. 02:12.200 --> 02:15.980 I mean, security is my, it's my job, but it's also my hobby. 02:15.980 --> 02:24.340 It's what I love to do anytime that I have spare time that's not either working or with family, I'm probably doing security research in some way, shape or form. 02:24.340 --> 02:26.520 So feel free to check out these projects. 02:26.520 --> 02:28.120 I have a blog on Medium. 02:28.440 --> 02:35.660 Any code, any examples, any demos, the slides, everything from this presentation will be available in GitHub. 02:35.660 --> 02:37.160 So you can start checking that out. 02:37.160 --> 02:43.220 If you want to follow along with anything, a lot of the content, the resources, the notebooks that we go through will be available there. 02:43.220 --> 02:48.720 So I encourage you pull it up while I'm talking, do that, check things out, and feel free to ask questions as we go along. 02:48.720 --> 02:51.240 But thanks again, and let's jump right in. 02:55.800 --> 02:57.960 So what are we actually going to cover? 02:57.960 --> 03:04.780 For the agenda today, there's really a couple of things, but I really want to focus on inputs and outputs. 03:04.780 --> 03:05.780 I think it's a mindset. 03:05.780 --> 03:08.780 We always talk about the hacking mindset, the security mindset. 03:08.960 --> 03:13.000 Well, in this case of this talk and the scope, I want you to think about inputs. 03:13.000 --> 03:14.140 What are the objectives? 03:14.140 --> 03:15.140 What are the goals? 03:15.140 --> 03:16.760 What do you put in? 03:16.820 --> 03:18.100 And then what are those outputs? 03:18.100 --> 03:23.400 How can we normalize these outputs so that way you don't have to try to remember how do you do 500 different scripts? 03:23.400 --> 03:25.140 How do you make these tools work? 03:25.140 --> 03:28.340 How do I manually parse through all these outputs and data? 03:28.360 --> 03:35.020 So throughout this, we're going to show kind of modular, repeatable technology, agnostic designs. 03:35.260 --> 03:38.820 And then how do you apply this from a cloud-focused technology? 03:38.820 --> 03:42.420 And we're going to spend our time in the AWS ecosystem. 03:42.420 --> 03:46.140 However, you really could translate this to any of the major cloud providers. 03:46.240 --> 03:48.360 And then also, we're going to provide some solutions. 03:48.360 --> 03:52.380 So how do you tie this all together so that way you can become more efficient? 03:52.380 --> 03:59.160 And I'm hoping that if you watch this talk, you apply these concepts, your key takeaways are I want you to make yourself better. 03:59.160 --> 04:01.160 I want to make the people around you better. 04:01.160 --> 04:04.740 And I'm hoping that overall, we can all work together and make the industry better. 04:08.000 --> 04:10.480 So this is the story of my life. 04:10.480 --> 04:14.620 Thank you, Jason, for sharing these tweets because the timing was perfect. 04:14.620 --> 04:23.880 And I started thinking in terms of, and this is in the scope of Bug Bounty, if you've ever worked on Bug Bounty, you see these different tweets of I just made $5,000. 04:23.880 --> 04:25.580 Or I made a $15,000 bug. 04:25.580 --> 04:27.100 Or I just made $400. 04:27.980 --> 04:33.460 It's kind of, that's how I always feel, is wow, let's just spend a couple hours, let's make a couple thousand bucks. 04:33.460 --> 04:34.620 But that's not the case. 04:34.620 --> 04:48.100 And I'm probably, I don't know if this resonates with any of you, but I'm the person that if you were to ever see me tweet I just made $2,000 on a Bug Bounty, that's because I just probably put in a hundred hours trying to research through reconnaissance, 04:48.100 --> 04:53.720 through tooling, through tests and failures and inputs and report writing just to make that. 04:53.720 --> 04:56.640 And that's what goes on, I think, often behind the scenes that we overlook. 04:57.040 --> 05:02.000 And one of the things is, too, is our most valuable commodity is time. 05:02.040 --> 05:20.420 My biggest challenge is finding the time to be able to do this, to really get in the weeds, to dig in and try to find these red team objectives to relearn, I'm always relearning things or figuring out or looking up or Googling or all the places where I'm trying to relearn all the things I've learned historically. 05:20.580 --> 05:21.700 How can we make it faster? 05:21.700 --> 05:23.360 How can we be more efficient? 05:23.400 --> 05:28.160 So when I have these two or three hours in an evening to spend, I can be focused. 05:28.160 --> 05:29.320 I can look at my targets. 05:29.320 --> 05:31.100 I know exactly what I want to do. 05:31.100 --> 05:38.680 I don't have to spend hours doing the reconnaissance research trying to just make selections on what my targets are because that takes up all my time. 05:38.680 --> 05:41.560 And then the time to value is really zero and net none. 05:41.580 --> 05:43.660 I'm going to focus on how do I be productive? 05:43.660 --> 05:44.860 How do I make money? 05:44.860 --> 05:46.140 And how do I find issues? 05:46.140 --> 05:48.940 And further, what I'm attempting to do. 05:48.940 --> 06:00.600 So that's what really drew me into how do I build this automated ecosystem and take an advantage of the new technologies and new capabilities out there so that I can be the best that I possibly can. 06:04.120 --> 06:13.180 So now let's jump into the architecture of the underlying design of the entire automated ecosystem that we're going to build and talk about through the remainder of this presentation. 06:13.200 --> 06:25.600 We'll have a bunch of demos and code snippets and things that we're going to kind of deep dive in and hoping that you can leverage these as accelerators so they can be agnostic to a specific tool, a specific process. 06:25.600 --> 06:40.460 But as you look at different things that you do and experience from a Red Team perspective or even just from a general information security process or practice that you do in your daily jobs, these are all components that can apply in a lot of different ways, 06:40.460 --> 06:41.780 which I think is really exciting. 06:41.780 --> 06:59.980 And that's a big part of why I feel like rather than just trying to deliver this custom tool that is just a script or something that you run and you routinely run it and it does the same thing every time, what I want to kind of teach is that thinking in terms of how do you build that underlying infrastructure that can support anything that can be thrown at you? 06:59.980 --> 07:11.880 How can you better yourself and understand rather than you kind of move up from knowing just how to run a tool to actually understanding what's going on behind the scenes, how do I piece this together and how do I make it my own? 07:11.880 --> 07:21.020 So I have this system that can be potentially better than others or bring you to the best in the industry or just solve challenges that you have on a day-to-day basis. 07:21.240 --> 07:25.760 So as we look at this architecture, it's really broken down into three key pieces. 07:25.760 --> 07:29.020 And on the left side is the user interaction. 07:29.140 --> 07:35.840 So these are the components of the cloud services that you're going to generally interact with, whether it's an input, whether it's an output. 07:36.060 --> 07:40.480 But in the design that I have that we're going to walk through, it's going to leverage these four pieces. 07:40.480 --> 07:48.000 And we're going to, as we go further, I'm going to give you a quick overview of every one of these components just to explain what it does, where it's used for. 07:48.040 --> 07:54.700 And in the back of your mind, continue to think, how can I reuse this for X function or what I'm trying to do or my daily job? 07:54.700 --> 07:58.620 And how can I convert my manual efforts that I do on a day-to-day basis? 07:58.620 --> 08:04.040 How can I save time, be more efficient, and automate this so that way I can make my life better? 08:04.960 --> 08:11.120 Basically, you can automate yourself and then you can focus more on the latest, greatest technology emerging. 08:11.120 --> 08:20.560 Push your program forward, mature in everything you're doing, and just continue to build and build on top so we're not relearning, we're not doing the same manual tasks over and over. 08:20.580 --> 08:30.680 And this ecosystem will really elevate that so that way you can do and you can build and it's modular and it's really an ecosystem of a lot of microservices in doing that. 08:30.800 --> 08:33.320 The middle layer is your processing and computation. 08:33.320 --> 08:37.960 So you have your user interaction that will pass everything over to your processing and computation. 08:37.960 --> 08:41.440 And this is a pretty big arsenal of different tools. 08:41.440 --> 08:58.780 You have, whether it's big data type, query analytics services, it could be eventually, we're not going to show any of these today, but if you get into machine learning or image recognition and thinking in terms of future opportunities of like you have screenshots of webpages, 08:58.780 --> 09:04.360 maybe you want to process a bunch of images of those really quickly, this would be your processing and computation layer. 09:04.460 --> 09:14.460 And then the last layer, I think is pretty key and it's really important to get this in the mindset as you kind of consider future opportunities, is that data storage layer. 09:14.540 --> 09:21.660 You look at today, a lot of the processes you may have, like if you're doing reconnaissance, you might run five or six different tools. 09:21.660 --> 09:28.380 In those inputs, you're going to submit, maybe it's a domain, maybe it's an ASN, maybe it could be a CIDR range. 09:28.380 --> 09:29.780 It could be various things. 09:29.780 --> 09:30.700 You input that. 09:30.700 --> 09:35.400 Generally, the outputs of your various tools and scripts, they're going to be slightly different. 09:35.400 --> 09:38.320 And that's where that source data bucket comes into play. 09:38.320 --> 09:42.600 So I expect, anticipate, let's pull in 100 tools eventually together. 09:42.600 --> 09:48.000 Let's all put that data, source data into that source data bucket that's just various formats and everything. 09:48.300 --> 09:51.580 And then what we want to do is we want to think in terms of how do we normalize this? 09:51.580 --> 09:53.420 What's the actual objective of this? 09:53.420 --> 10:08.020 Because in a lot of cases, I mean, and we're going to look from kind of a red team bug bounty perspective, but think broader as you think about your actual job, whether you're doing risk management, whether you're doing third party analysis, whether you are a security engineer, 10:08.020 --> 10:11.660 whether you're an architect, any role in this, think about how this can apply. 10:11.680 --> 10:13.240 But we look at our inputs and outputs. 10:13.240 --> 10:16.940 So a lot of our tools, they're either going to output things such as URLs. 10:16.940 --> 10:21.420 It might be parameters for a different web page. 10:21.420 --> 10:24.620 You could have the ASN numbers for network ranges. 10:24.620 --> 10:25.740 You could have CIDR ranges. 10:25.740 --> 10:27.080 You could have domain names. 10:27.080 --> 10:28.140 You could have subdomains. 10:28.140 --> 10:29.580 You could have company names. 10:29.580 --> 10:34.000 You could have certificate assignments, different inputs and outputs. 10:34.000 --> 10:38.040 So those are all of your various outputs that you're generally going to have across your tools. 10:38.340 --> 10:50.180 We need to make these common between that so we can further process this and make this data useful because I know one of the areas that I struggle with the most is parsing through this data of every tool and it just takes so much time. 10:50.220 --> 10:55.980 And that's something that we could, if you do it one time for a tool where you normalize the data, you never have to do it again. 10:56.100 --> 11:04.380 And the cooler part about this is we only need one person in the industry to do it for per tool or start to kind of standardize and build in. 11:04.480 --> 11:08.720 And that's where on my GitHub you'll see Project Straylight. 11:08.900 --> 11:17.040 And my goal is really to just have this running open network of here's accelerators we can do, here's code snippets that have helped me along the way. 11:17.040 --> 11:31.220 And it's maybe not so much a technology or tool that you install, but it's just a bunch of ideas and things that have helped me in my career that over the past 10 to 12 years that I've either had to relearn or do, I'm trying to aggregate it all into this ecosystem, 11:31.220 --> 11:51.340 this architecture that we see of how can I do what I've spent so many hours doing redundantly, manually, and how can I make it better so that way the newer generation, the next people that are going to be experts, leaders in this, can just build on top of what we already have so we're not continuing to rebuild and we can advance further and further and we can win in this industry. 11:51.340 --> 11:53.680 So that's a lot of the intent there. 11:53.680 --> 12:01.240 So as we look at the data storage layers, you basically go from your raw data, which could be the output of any tool, you get to more of a normalized version. 12:01.580 --> 12:05.300 And then you also get into your presentation layer. 12:05.300 --> 12:11.780 And that's when it's your reporting, it's your analysis, it's your metrics, it's the outputs that you want to show for the delivery of your work. 12:11.780 --> 12:14.520 And this could speak to any level depending on how you present it. 12:14.520 --> 12:23.900 So you could do an aggregate of here's all the attacks that we've had over the course of X days, summarized into charts, graphs, and that might be at more of a CISO level. 12:23.900 --> 12:30.620 You could have indicators of compromise or different signatures, different levels, which you would feed into an engineering team. 12:30.620 --> 12:32.340 That could be part of your presentation. 12:32.340 --> 12:47.200 But it's how do you normalize and how do you show this in kind of a formatted way so that way if you have to do monthly vulnerability reports or metrics or showing analysis of how you've further progressed or analyzed websites or targets and you want to compare what you've done, 12:47.200 --> 12:50.340 what you haven't done, that's where that presentation layer comes in. 12:50.440 --> 12:53.420 So we're going to jump in and I think we're going to do, let's do a demo. 12:53.420 --> 12:59.480 And let's kind of show, I'll show from end to end kind of one area where we can go across this of how this ecosystem works. 12:59.480 --> 13:02.280 And then after that, we'll dive in further. 13:02.280 --> 13:05.040 I'm going to show you the accelerators, the pieces, the components. 13:05.060 --> 13:07.860 Keep in mind that all of this is also available in GitHub. 13:07.860 --> 13:09.300 You can access it as we go along. 13:09.300 --> 13:10.040 You can check it out. 13:10.040 --> 13:11.200 Feel free to use it. 13:13.180 --> 13:16.140 Create issues in the GitHub if you find any things. 13:16.140 --> 13:17.360 Reach out to me with questions. 13:17.360 --> 13:23.840 But I really just want to continue to build and teach you how you can kind of think in terms of building your own automated ecosystem. 13:25.480 --> 13:29.640 As you're watching this too, you're probably thinking, why cloud? 13:29.640 --> 13:31.260 I have a really nice home lab. 13:31.260 --> 13:32.620 Do I really need it? 13:32.620 --> 13:39.020 There's three key areas beyond just the amount of what we just walked through in that architecture that you have available to you. 13:39.020 --> 13:43.760 But it's pretty exciting when you think about what cloud can mean to you. 13:43.760 --> 13:45.040 And if you hear... 13:45.040 --> 13:48.220 I mean, the businesses, industries, everybody is adopting it. 13:48.660 --> 13:49.760 That's nothing new. 13:49.760 --> 13:51.160 It's not anything new there. 13:51.160 --> 14:01.980 But I think there's three key parts of cloud that are really exciting as just an individual researcher where you don't need millions of dollars or tens of thousands of dollars of funding behind you to actually make progress. 14:02.040 --> 14:06.200 The first part is the democratized accessibility of datasets. 14:06.420 --> 14:10.240 There's some pretty massive and interesting datasets out there. 14:10.240 --> 14:18.920 The problem is that on our home labs, our home computers, our home servers, a lot of those don't have either the horsepower to accomplish what we need. 14:18.920 --> 14:29.280 We need more memory, more CPU, or it's not close enough to the data where we can't download petabytes of data or terabytes of data in a meaningful way to keep up and be able to analyze and do. 14:29.280 --> 14:32.520 And that's where you kind of run into issues of data gravity. 14:32.520 --> 14:38.040 And you need to build your compute actually closer to the data stores, the data repositories, because it's costly. 14:38.040 --> 14:39.700 It's expensive to move those. 14:39.700 --> 14:42.380 So how can we proceed with that? 14:44.120 --> 14:52.520 The other piece is that if you look across the world, we're facing right now, we're all in quarantine, we're social distancing. 14:52.540 --> 14:52.900 You have... 14:53.520 --> 15:01.720 So take a look at what's going on around the race to find vaccines and solutions to COVID and kind of the data around that in the pieces. 15:01.820 --> 15:10.100 The world is coming around cloud and the latest, greatest computational technologies to solve the world's biggest problems that impact everybody globally. 15:10.160 --> 15:19.480 So what if we can take those pieces that the top scientists in the world, the top in their industries are taking to solve the world's most challenging problems? 15:19.520 --> 15:26.020 I'm sure we can apply that to our daily bug bounty targets or our daily security processes that we need to improve. 15:26.020 --> 15:29.540 But in the same way, we don't have to spend tens of thousands of dollars for this. 15:29.540 --> 15:33.080 The other neat thing about it is it's financially viable. 15:33.180 --> 15:39.100 We can now stand up servers that have 128 gigabytes of memory. 15:39.100 --> 15:42.100 You can do unlimited storage. 15:42.100 --> 15:46.500 And the nice thing is you don't have to pay thousands of dollars to buy this from a capital standpoint. 15:46.500 --> 15:48.240 It's all operating expense. 15:48.240 --> 15:58.360 So as you look at it, if you need to run something that's extremely memory intensive and it's just one operation or a loop, you literally could stand up a massive server for just a couple minutes. 15:58.360 --> 16:03.380 It might cost you $10, $20 for maybe five or ten minutes of processing. 16:03.620 --> 16:14.640 But rather than needing to invest $80,000 in similar hardware equipment, you now can do this in your home office, in your home lab through the cloud capabilities charging. 16:14.640 --> 16:20.120 And that $20, if you think about it in terms of bug bounty investment, that's one P4. 16:20.120 --> 16:22.380 And you have that well covered. 16:22.380 --> 16:24.820 I mean, that's on the low end thinking in terms of that. 16:24.820 --> 16:39.760 But just looking at it from a business perspective, thinking in terms of how can I continue to invest in my capabilities, the technologies, the processes, so I can gain more time and I can make the time that I spend actually manually doing things, evaluating parts, 16:39.760 --> 16:42.160 how can I make it the most beneficial to myself. 16:47.700 --> 16:51.740 So now let's take a look at the various tooling that we can use. 16:51.740 --> 16:53.380 We've looked at the architecture diagram. 16:53.380 --> 17:02.780 I'd like to just give a quick overview of the different components, especially if you're not familiar with the AWS services that are listed on here, kind of what they do, why they're important and how they can apply to you. 17:02.860 --> 17:09.360 So if we go through the list, you saw from the user interaction space the Amazon API Gateway. 17:09.480 --> 17:12.900 So what this does is this is just an entry point into your ecosystem. 17:12.900 --> 17:23.200 And the neat thing about Amazon API Gateway is there's a lot of things you can do with it, but where I've really leveraged it is as a proxy into your Lambda function. 17:23.200 --> 17:32.020 So what you can do is you can do basically a GET request or a POST request through the API Gateway and pass that directly into an AWS Lambda function. 17:32.160 --> 17:44.680 And if you think in terms of what you've seen in the Jupyter notebooks and Python libraries and any functions you run where you pass something, you literally can pass it through a GET request through the API Gateway into AWS Lambda. 17:44.680 --> 17:58.560 So I kind of view that as top tier level of services where you've really normalized your content, you have exception handling, you know that you're ready to completely automate this, let's promote it into a Lambda function, pull it out of your Jupyter notebook, 17:58.560 --> 18:07.580 and then let's put it behind an API Gateway so you can really kick it off and call it ad hoc as you need and really start your ecosystem of automation around there. 18:08.340 --> 18:19.400 Another tool is Amazon SageMaker and it's really a larger framework and compute system where you can do more things like machine learning and training and all of that. 18:19.400 --> 18:22.740 But what I pretty much solely use it for are the Jupyter notebooks. 18:22.740 --> 18:42.140 So you can stand up the Jupyter notebooks, you can run it, you can adjust the size on it so if you need a lot more gigabytes or storage for a duration or a function, a lot of times I'll ramp up and use a larger instance of it for just a few minutes or just to do that process and then I'll turn it down to kind of the standard level. 18:42.140 --> 18:46.600 So Amazon SageMaker is really the tool mechanism for running the Jupyter notebooks. 18:46.900 --> 19:02.060 The static website hosting S3, that's an S3 bucket, you can actually set those up so that they can host web content, make them publicly available so you hit it, you can set an indexed HTML page, you can run it, and you can run websites out of S3 buckets which is really neat. 19:02.940 --> 19:11.860 The Amazon simple notification service, it's a way that you can send push messages to whether it's SMS text, whether it's emails, different things. 19:11.860 --> 19:14.600 So it's really just a notification service that we can do. 19:14.600 --> 19:25.700 In this case, I'll show a code snippet where we can actually send ourselves text so that way we know if there's anything that's going to take a little bit of time or you just want to get notified or have the final URL where you're going to access your data, 19:25.700 --> 19:27.560 you can get texted whenever it's ready. 19:27.560 --> 19:33.420 So you can kind of do bug bounty, reconnaissance, all of this on the fly as you build this system which is pretty cool. 19:35.140 --> 19:43.200 Now as we move into the processing and computation layers, AWS Lambda is really what's used to run code. 19:43.200 --> 19:57.220 So you can set it up, it runs a lot of different things, it's not just Python, but it's Node, and it runs on a serverless compute so really you put the code in there, it will stand up the compute resources and it only runs for a duration. 19:57.220 --> 20:04.700 I think the longest that you can run a Lambda is about 15 minutes so you're charged based on the duration that the computation is running. 20:04.780 --> 20:14.340 So if you think in terms of smaller Python functions that you have ready to automate and scale, that's where you can migrate them to the Lambdas and they can just always be ready to work off of triggers. 20:15.560 --> 20:28.480 The Amazon Elastic Block Storage, a lot of times, S3, there's a little bit of a caveat where S3 you get charged on get requests and put requests and writes, whereas EBS storage you pay for how much storage that you have assigned and allocated. 20:28.480 --> 20:39.680 So depending if I think I'm going to do millions of reads or writes, sometimes I'll do a lot of my testing against Elastic Block Storage and then once I have something that I know is ready to go robust and I want to leverage it in the greater ecosystem, 20:39.680 --> 20:50.020 I'll move it over to our data storage tiers while they're in the refined or the raw data levels to do more normalized and automated operations against. 20:50.860 --> 20:54.540 AWS Glue is a way that you can set up and you can crawl data sets. 20:54.540 --> 21:09.320 So Athena is, you picture that, is like you write your SQL queries and your query language and it will run it and distribute it against data sets and it can run against CSV files, it can run against relational data stores, unstructured data, and all those. 21:09.320 --> 21:14.420 So it's really powerful and if you haven't ever dug into it, AWS Athena, I highly recommend it. 21:14.420 --> 21:23.720 There's a lot of power in it and you can literally just sit there and use the console interface and pretty much ad hoc write queries against big data sets if you decide to as well. 21:23.960 --> 21:29.020 And then Glue is, I started talking, it's what indexes a lot of those data points around it. 21:29.020 --> 21:35.420 So you can create Glue indexes so you don't have to query the entire data set every single time and you can do those. 21:36.360 --> 21:49.340 AWS Step Functions, you won't see anything in the demos today leveraging this, but this is kind of a future capability where I want to leverage where you can actually, it's like setting up workflows and processes and you need X to happen and then once X completes, 21:49.340 --> 22:01.460 you can send the output to this one and do, right now, I'm just leveraging things off of triggers and alarms and Lambda files, but you can really start to use AWS Step Functions as orchestration, as a central core piece. 22:01.460 --> 22:03.840 So something I'm going to dig into more in the future. 22:04.620 --> 22:11.700 AWS Secrets Manager is a place that, so you can move all of your secrets, your API keys, out of your code, put them in Secrets Manager. 22:11.700 --> 22:13.280 It's a quick call out. 22:13.280 --> 22:18.080 You can load them, you can store them, and you can kind of store other parameters in there as you need as well. 22:18.080 --> 22:22.760 You can pull them back into your code so they don't sit as code, but you can use them as variables on the fly. 22:22.760 --> 22:24.200 So I highly recommend digging in there. 22:24.200 --> 22:27.120 There's some accelerators that you'll see that talk to those. 22:27.760 --> 22:37.380 And then on the right side are just all the S3 buckets and you can create multiple S3 buckets and have the different stores and it's just long term data storage for you. 22:40.180 --> 22:42.220 So let's jump in with a demo. 22:42.540 --> 22:51.480 This demo is going to show how we can query Rapid7's Project Sonar, which is essentially it's the 4DNS of the entire internet. 22:51.480 --> 22:54.160 And it's sitting in a dataset within S3. 22:54.160 --> 23:11.200 So rather than try to copy that entire dataset down into our own system, we're going to leverage a Jupyter Notebook and we're going to actually run a distributed query against that external dataset that's hosted in not our own S3 bucket, but the public datasets bucket. 23:11.380 --> 23:14.520 And we're going to get the results back and we're going to process it. 23:14.520 --> 23:16.740 And we're going to do it in just a matter of... 23:16.740 --> 23:26.440 it's probably going to take a couple of minutes as we walk through and talk through the code, but this is literally, I think generally the queries around the 4DNS for a domain wildcard is about 26 seconds. 23:26.660 --> 23:28.740 So let's walk through this. 23:29.540 --> 23:42.380 So on the screen here you can see my Jupyter Notebook and what we'll do is we'll walk through this too because I want you to think in terms of how can I reuse these components, how is this applicable well beyond just this sonar use case. 23:42.480 --> 23:43.720 So let's go ahead and start. 23:43.720 --> 23:47.100 We're going to go ahead and run this first section of code here. 23:47.100 --> 23:54.700 And this is what I talked about earlier too, which is nice is that if you're not an expert at code, you run it, you can break it down into cells, and you can run each cell. 23:54.700 --> 23:56.440 So we're going to run this first cell. 23:56.520 --> 24:00.280 What we just did was you can see this execution ID here. 24:00.300 --> 24:10.080 What we did is we set our domain to Microsoft.com so we're actually querying all the wildcards of the Microsoft domain 4DNS space. 24:10.080 --> 24:13.580 And keep in mind, we are not touching any of Microsoft's systems. 24:13.580 --> 24:19.440 This is purely utilizing the 4DNS dataset and we're getting the results from there. 24:19.800 --> 24:28.700 And then what we've done is we've set some of our variables around our bucket of where we want to store the results, what database we want to query against in Athena, which we've had some... 24:28.700 --> 24:33.120 the setup information is all contained within the GitHub as well so you can set that up. 24:33.120 --> 24:35.340 And then the table we want to query against. 24:35.580 --> 24:38.680 What we do next is we set up our query. 24:38.680 --> 24:53.260 So what this is doing is basically just saying select everything from wildcard microsoft.com and then we select the latest date of the dataset so that way we don't historically go through every single one iteration that we have. 24:53.880 --> 24:56.640 It issues the Athena query. 24:56.640 --> 24:58.400 It takes about 26 seconds to run. 24:58.400 --> 25:00.740 We have the Athena query ID here. 25:00.740 --> 25:04.560 So in this next component, I will actually jump back up. 25:04.560 --> 25:10.500 This run command that I'm highlighting, this is actually a way that you can call other notebooks, which is kind of neat. 25:10.500 --> 25:16.440 So as you think about in terms of you call functions within programming code, this will actually run and load an entire notebook. 25:16.440 --> 25:24.640 So what I've done is I try to take anything that's not tool specific and I put it in kind of a common notebook that you can set a lot of different things to run. 25:24.640 --> 25:35.480 So what this is actually doing is the query Athena function is running out of that other notebook because I've set it up well enough that you can pass in the results bucket. 25:35.480 --> 25:41.300 You just send it the query in the Athena database and it will kick it off so it doesn't matter what tool specific or anything. 25:41.300 --> 25:48.460 It will just run a query that you tell it to run which is why I moved it into that kind of standardized function area. 25:49.800 --> 25:54.040 So what we're going to do is we're going to run this next one and this is going to generate, pull the results back. 25:54.040 --> 25:55.780 So you can see that it's already succeeded. 25:55.780 --> 26:01.500 If we were to run this immediately it would just go in a loop and it would circle every five seconds and check for the results back. 26:01.640 --> 26:08.720 But it's pulled back 18,695 different rows and you can see the types of information we've just pulled back. 26:08.720 --> 26:22.280 So think you have 18,695 different rows basically for DNS entries from Microsoft.com in under 26 seconds with the IP addresses, the C names, A records, MX records, all of those. 26:23.000 --> 26:35.300 And then with that we're going to do this next thing is this is actually going to leverage the MaxMind database and we're going to start figuring out querying for the IP addresses and getting all that information back. 26:35.300 --> 26:41.160 So let's go ahead and run this and what this will do is it creates a data frame which leverages Pandas. 26:41.160 --> 26:46.760 So Pandas is a library within Python that you can run and I completely recommend it. 26:46.760 --> 26:51.680 It is one of the best things that I've ever found in terms of doing metrics and analysis and automation. 26:51.680 --> 26:56.120 But you can see it just processed 18,695 rows in just a few seconds. 26:56.120 --> 27:02.480 And now we have the latitudes, longitudes, countries, and localities all merged to all 18,000 entries. 27:03.140 --> 27:11.660 So what we're going to do now is if you say, hey, I just did all this work, I don't want a chance of losing it even though theoretically you could redo this over about 30 seconds. 27:11.740 --> 27:18.480 But if you want, you can use this command and you can save your data frame off to Excel so you can save it locally. 27:18.480 --> 27:25.700 So if you run this, it will just print and store those 18,000 records into an Excel spreadsheet which you can reload later on which is pretty handy to use. 27:25.700 --> 27:29.860 And if you want to save your artifacts and data points, you can always do that. 27:31.480 --> 27:35.660 So what we'll do now is we're going to run this next command and what this does is it just aggregates. 27:35.660 --> 27:36.800 And you can see how fast that was. 27:36.800 --> 27:45.720 It was almost instantaneous when I pressed the start button and it just grouped up all the different latitudes and longitudes so that we can get ready to plot them on a heat map. 27:46.040 --> 27:51.580 So we know 525 had this latitude, longitude, and it did all that work. 27:51.580 --> 27:52.280 And we can dig in. 27:52.280 --> 28:00.300 This also calls that the other notebook because it's centralized and it's not specific to the project sonar data set. 28:00.300 --> 28:03.480 So you can reuse and redo any of that. 28:03.660 --> 28:05.000 I am going to jump back up, though. 28:05.000 --> 28:07.040 I think this is something that's pretty neat. 28:07.040 --> 28:23.840 And if you think about in terms of how can I reuse some of this, this get location function that I have behind the scenes that just pulled in and merged all this together, what I did, if you're looking, I passed the entire data frame as a part of the function there and I passed the value column, 28:23.840 --> 28:25.480 which is the IP address column. 28:25.480 --> 28:36.180 So if you think about reusability, all I literally did was I just took, if you have an Excel spreadsheet of IP addresses or a CSV or anything like that, you load it into a data frame. 28:36.180 --> 28:37.940 You can pass that entire data frame. 28:37.940 --> 28:38.540 It doesn't matter what it is. 28:38.540 --> 28:39.740 It doesn't matter what the rest of it looks like. 28:39.740 --> 28:51.260 You pass the data frame name and the column that the IP addresses are in or obviously it will fail on if it's not an IP address as it goes through those because they're C names. 28:51.700 --> 28:53.660 And you do that and you can process it. 28:53.660 --> 28:57.660 So anything you have, that's completely reusable and that's just a few lines of code. 28:57.660 --> 28:58.200 It's in GitHub. 28:58.200 --> 29:06.900 You can pull it and I think in terms of just from a cyber operations, a SOC perspective, you could really get a lot of value out of some of that information. 29:06.900 --> 29:16.460 You could look at attackers, web denials, web traffic, who's hitting your web application firewalls and you can do heat maps and different things from an attacker perspective which is pretty cool. 29:17.660 --> 29:18.900 So let's go on down. 29:18.900 --> 29:22.980 We have that information and then we want to now, let's do this. 29:22.980 --> 29:26.140 So this is just a couple lines of code but it's really cool . 29:26.140 --> 29:29.520 So this uses the Google Maps API and you can see how fast that was. 29:29.520 --> 29:46.140 It just plotted the information we have up here for the latitudes and longitudes and created an interactive heat map so we can zoom in on and see what's going on and this shows the concentrations of Microsoft's forward DNS space across the globe. 29:46.140 --> 29:51.500 So you can look and as we zoom in, you can see the different regions they are included in. 29:52.260 --> 30:03.440 So let's, if we were just, let's say, let's tackle this is a big area here, you can zoom in and even though it shows Seattle space, you can go and see that it's not all just in Seattle but they have a few offices. 30:03.440 --> 30:06.540 It looks like in Redman, there's some Seattle and above. 30:06.540 --> 30:10.180 So pretty cool that it's all interactive and you can do that. 30:10.180 --> 30:13.980 And the other neat thing is you're thinking, well, I want to show this to management. 30:13.980 --> 30:17.000 How do I show my managers, my leadership about this? 30:17.020 --> 30:20.980 And I can't have them just opening a Jupyter notebook all the time and doing this. 30:20.980 --> 30:25.580 Well , one thing is you can click the download button and just download the map as an image. 30:25.580 --> 30:42.220 But the other pretty neat thing about all this, and if you think about the architecture of how I have that presentation layer, what this next command does is it takes this map and it actually uploads it to a static S3 bucket that's hosting a static S3 content as a website . 30:42.380 --> 30:53.320 So then I hit this, I uploaded the map file to the website and then I can navigate over to my bucket and if I hit load content right here , it should pull up that map. 30:54.040 --> 31:00.580 And now I can always send any of my leadership, any managers , anybody that wants to see anything , I can run this on an iterative basis while I'm there. 31:00.580 --> 31:03.340 It's on every 24 hours I run this process and do. 31:03.540 --> 31:09.860 And if you think in terms of visually what you want your attack surface to look like for your company, run this against your company every time. 31:09.860 --> 31:14.820 Have this dashboard always going and then you could even trigger on anomalies, you could do different things. 31:14.900 --> 31:20.880 And it's just pretty neat to always see here, here's our coverage from an internet presence that we have there. 31:20.880 --> 31:24.020 So I think there's a lot of value, a lot of opportunity to do in there. 31:24.020 --> 31:36.740 And just the fact that you can literally load it and save the interactive mapping part of it to an HTML file and then have people actually go up, browse to it, navigate it, that's pretty exciting. 31:41.770 --> 31:49.690 So for this next demo, we're going to use, this is actually one of my favorite ones and I'm really excited to continue the build on the capabilities of this. 31:49.690 --> 31:54.710 But this leverages a data set that's publicly available in S3 called Common Crawl. 31:54.710 --> 32:08.490 And what it is, it's a project that's been going on now for, I think it's actually been around for about eight years, where it crawls the internet, searches and indexes and actually downloads the HTML of files and stores them in S3 buckets. 32:08.490 --> 32:13.930 So it's just a massive repository and similar of basically web archives that you can search. 32:13.930 --> 32:17.730 So if you've used Wayback Machine and different components , this is very similar. 32:17.990 --> 32:20.970 And so what we're going to do is we're going to actually search it for a domain. 32:20.970 --> 32:31.110 We're going to pull back the files and it actually can eventually download the files and we can store them in an S3 bucket and then we can run a bunch of different tools around it. 32:31.110 --> 32:44.070 So if we want to do like find URLs or run HackCrawler or anything that's going to go through and does a good job of analyzing HTML, whether it's for vulnerabilities , whether it's for URLs, whether it's for parameters , we can do that against it. 32:44.070 --> 32:45.730 So let's walk through this now. 32:45.730 --> 32:50.050 I'll kind of show you the framework around it and there's a lot that we can build into this capability . 32:50.050 --> 33:03.490 The biggest challenge that I've had so far is just trying to do some of the asynchronous programming against it because a lot of times you have a large site where you're trying to do downloads and actually pulling the files, but as we get more efficient with this, 33:03.490 --> 33:06.550 I'm pretty excited about this and I think it has a lot of promise with it. 33:07.490 --> 33:10.030 So what we're going to do is we have all of our parameters set up. 33:10.030 --> 33:20.470 We're actually going to run this one against the smaller domain for demo purposes because if we have something that has 70,000 websites and pages, it's just not fast enough at this point to download this in a timely fashion. 33:20.470 --> 33:22.890 We would be sitting here for maybe 40 minutes or an hour. 33:22.890 --> 33:28.470 So in this case, we can always speed it up as we go, but we're going to run this against derbycon.com. 33:28.970 --> 33:31.570 So we've kicked off our Athena query. 33:31.570 --> 33:32.670 It's running it now. 33:32.670 --> 33:33.530 It's searching. 33:33.530 --> 33:41.030 It's looking for all of the webpages that this is indexed and cached and copied and saved within the Common Crawl dataset. 33:41.430 --> 33:54.390 So let's go ahead and kick off our run and what this is going to do is it's going to look and it's going to just wait for the query to stop running and then it will pull in and load the results for us into a database. 33:54.390 --> 33:56.910 So we'll let this run for a few minutes. 33:57.210 --> 33:59.410 All right, so it looks like we have our results here. 33:59.450 --> 34:03.510 So it pulled back 579 different websites. 34:03.510 --> 34:20.930 What we want to do is since it's been running over the course of eight years, there's a lot of different versions , iterations, which is actually pretty neat if you want to see how websites have evolved and look for changes or maybe there's a vulnerability or an issue or a comment in there that they moved out and you could see the difference between the different versions. 34:20.930 --> 34:24.870 But in this case, I'm interested more in just the unique users that we have we have. 34:24.870 --> 34:31.990 So we're going to run this next command which is going to sort them and it's going to drop all the duplicates and it's going to keep the latest version of the webpage. 34:32.010 --> 34:33.530 So let's go ahead and run this right now. 34:33.530 --> 34:34.810 And that was pretty instantaneous. 34:35.330 --> 34:37.510 And now we're down to 73 different rows. 34:37.510 --> 34:41.830 So there were a lot of different iterations, versions of the site cached. 34:42.110 --> 34:49.830 And then what this will do is we can run this next command, we can save it off to Excel so that way we can have our URL listing for future analysis, future loading. 34:51.050 --> 35:02.750 And then as we go on, what this will do is if you want to look, this will pull out and only list now our URLs which is what we care about a lot of times, especially if we're trying to scope, we want to do web crawls. 35:02.750 --> 35:08.130 And keep in mind that everything that we're showing today, everything is completely 100% passive. 35:08.130 --> 35:11.170 We are not touching any of these sites, these webpages or anything. 35:11.170 --> 35:15.430 So this is all examples of what you can do, 100% passive reconnaissance. 35:15.430 --> 35:18.990 So we will never touch a system of any of these domains that we've looked at. 35:18.990 --> 35:30.650 And then for next steps then you start, you can start based on scope and allowances and permissions and things like that to get more intrusive as we do further analysis, whether it's from a red teaming perspective or however. 35:30.850 --> 35:44.730 But what we have right now is just a listing of URLs and then what this next big section of kind of a lot of, a lot of different functions and code , I mean it's still for , for the value of what this does, it's, it's not too bad in terms of lines. 35:44.730 --> 35:48.550 But what this is going to do is it's going to go through now of the query results . 35:48.550 --> 35:52.110 So in this case we're dealing with what, like 73 records , so it's not that many . 35:52.110 --> 36:01.110 But if you were to run something, I think I ran, I think I ran one earlier this year against defcon.org and it came back with about 70,000 records. 36:01.110 --> 36:04.950 So some of these, there's a lot of data and content out there and it can take some time. 36:04.950 --> 36:09.910 And this is where, this is not that efficient, not that streamlined, a lot of improvement opportunity here. 36:09.950 --> 36:16.270 But it, it basically searches , it uses a library in Python called Beautiful Soup, which you can parse down HTML . 36:16.270 --> 36:18.050 It's great for HTML processing. 36:18.050 --> 36:21.970 It has some building capabilities to look for comments and URLs and links. 36:21.990 --> 36:23.390 So we're going to start setting this up. 36:23.390 --> 36:33.490 And what, I have three parameters that we can pass in, just mostly out of efficiency, is we can, we can say we want to search the files and generally we're going to always want to do that. 36:33.990 --> 36:35.290 Do we want to write the files? 36:35.290 --> 36:42.970 So what this does is this goes out and it retrieves, well, it's always going to go out and retrieve the files from the S3 buckets, but do we want to write them? 36:42.970 --> 36:47.690 So in this case, we're not going to write them for this demo purpose, but you could write it to an S3 bucket. 36:47.730 --> 36:51.110 Now you have your HTML files that you can run additional tools and things against. 36:51.110 --> 36:52.510 So it's, it's pretty valuable there. 36:53.150 --> 36:54.730 And then we can say how many records. 36:54.730 --> 37:04.370 So if we just want a subset for testing, if we don't want to pull down all 60,000 at once, we could, we can assign how many records and it will loop through and iterate over and download the number of records we say . 37:04.410 --> 37:08.170 And then we, we run this and this function will basically process these . 37:08.210 --> 37:17.830 It's going through, you can see I have a counter on it because a lot of times when you're just waiting and you get bored and don't know where it's at, so we can, you can always clean this up so it doesn't iterate every single count. 37:17.830 --> 37:28.070 But we've gone through the 73 HTML files we've down, we've processed them as byte strings, we've looked for URLs, the comments, and now we can actually save off the different sections. 37:28.070 --> 37:32.190 So we have, we've grabbed all the comments, we've grabbed all the titles, and we've grabbed all the links. 37:32.410 --> 37:36.870 So if I run this next one, this is actually going to show you the, the listing of links. 37:36.870 --> 37:47.830 And what I, what I like about this is that it actually shows you which website that the links came from as well, which I think is really valuable, especially with sometimes when you do crawls, you don't always know, where did I find this link? 37:47.830 --> 37:48.750 Where did this come from? 37:48.750 --> 37:55.390 And in this case, it does all of the mapping, so if you start building like a bigger mind map or something of it, you can, you can have some value there. 37:55.450 --> 38:05.250 And if we wanted to look at something instead, let's say we wanted to look at the, the comments across it, we can do that, and then we can just print it out. 38:05.250 --> 38:09.090 So this will just print out the top, the first 10 comments we have. 38:09.510 --> 38:11.230 Let's just see what this looks like. 38:12.490 --> 38:15.770 And then there's, there's your list of comments that are within the HTML. 38:15.770 --> 38:24.490 So as you get this, you can start then further looking through there of, you know, of, are there any secrets, are there passwords, you can do regex, you can, you can do different things in there, so it's, it's pretty neat. 38:25.670 --> 38:29.410 I guess that's the, that's the extent right now I think we're going to go through. 38:29.410 --> 38:30.910 I started looking into NetworkX. 38:30.910 --> 38:46.090 I think there's some promise there in terms of getting a larger map at scale, but I don't have a lot of that code done yet, but a lot I want to build on, especially with this functionality because I think there's so much value from even an automated perspective where now we can tie in additional tools that are really good at HTML processing. 38:46.090 --> 38:54.870 We can start to grab JavaScript libraries, codes, external references and all that and do, you can really start building a pretty big map of, of your target surface. 39:00.710 --> 39:02.450 Let's jump into another demo. 39:02.450 --> 39:14.810 This one is going to search the autonomous system numbers, which is part of generally your reconnaissance methodology of where you want to find out the different site or network ranges that are assigned to different companies. 39:14.810 --> 39:29.650 So what you typically do is you would enter in the company name or a keyword for that company and it would return the different site or ranges for that so then you can further do your reconnaissance doing whether you do NMAP scans or any type of searches against those IP ranges. 39:29.750 --> 39:38.670 So what this does is this uses the MaxMind databases as well that we talked about earlier with the geolocation data doing that, but this uses their ASN databases. 39:39.190 --> 39:49.730 And so part of this is we'll walk down through it all, but what it does is if you're looking here at this Jupyter Notebook now, we go and these first sections actually just prepare everything. 39:49.730 --> 39:56.990 So if you want to be able to set this up in an environment where it does it completely automated where it will download. 39:56.990 --> 40:09.930 We have the URLs here and you can set this up on your own just by running this notebook so that way you can continue to run tools against it and do and leverage this there because I really want to eliminate all the manual steps possible. 40:09.930 --> 40:14.250 So the last thing I want to do is have to log into their website, get the latest database and do. 40:14.250 --> 40:15.830 So this will walk you through it. 40:16.170 --> 40:22.590 One pretty neat thing about this is that recently they've changed to and you have to have a license key now. 40:22.590 --> 40:28.350 It's still free, but you have to log in, register in their portal, get a license secret to be able to download those databases . 40:28.350 --> 40:31.650 Now , when I share this code , I don't want my license key in there. 40:31.670 --> 40:38.670 That's another benefit of using the cloud and Amazon is that they have a service called Secrets Manager. 40:38.910 --> 40:44.690 And what you can do is actually you can call Secrets Manager from your Python code and pull out secrets in store. 40:44.690 --> 41:00.110 So as you start to get a big list of different API keys and as you talk about across different clouds the cloud, even previously we made a call for the Google Maps API, I pulled the API key that I used for that out of the Secrets Manager within AWS. 41:00.110 --> 41:03.990 So this function here is actually what does it. 41:03.990 --> 41:05.010 It's in the notebooks. 41:05.010 --> 41:07.210 It's in my central notebook because it's reusable. 41:07.350 --> 41:13.290 But you just pass in the secret name, the region name , it loads it into here as a variable so that way you don't have to have it in clear text. 41:13.290 --> 41:17.730 And it's a security best practice and it's definitely safe if you're putting your code into GitHub and things. 41:17.730 --> 41:23.910 So I think as security practitioners anyway, that's just something to get better at because we push our developers to do that. 41:23.910 --> 41:28.910 We push a lot of people that we talk to to do that, but it's hard and it takes extra time. 41:28.910 --> 41:36.250 So here's kind of just an accelerator that you can copy, paste, use it in your Python code, especially if you're working within a cloud ecosystem. 41:36.650 --> 41:43.770 An alternative to that is it's called Pickle and it's a library within Python 2 where you can actually save files locally . 41:43.770 --> 41:49.490 Now they're just binary files that can get loaded back into Pickle so they're not secured, encrypted, anything like that. 41:49.490 --> 41:58.670 But it can at least call it from your operating system rather than pulling it or storing it hard-coded into an environmental variable or within your code. 41:58.670 --> 42:01.630 So another option if you don't want to use Secrets Manager. 42:02.270 --> 42:04.570 So as we go through this, this all gets set up. 42:04.570 --> 42:14.130 I already have all this, but you can see if you put an exclamation mark before any commands, you can actually run Linux commands off of this and SageMaker's running in the back end of a Linux server. 42:14.130 --> 42:18.270 So you can do a lot of that too, which is pretty neat from a flexibility standpoint. 42:18.270 --> 42:24.550 So this manipulates all your data, does a WGET , pulls it all down, goes through that process, does a cleanup. 42:24.710 --> 42:29.490 I hate having files just sitting on my computer that I don't need or want . 42:29.490 --> 42:35.910 So whenever you unzip this , this will do all the cleanup for you, loads it into a directory, and then now you have your files. 42:36.310 --> 42:47.090 So what we're going to do is we're going to actually start on down here because what's neat about this is we can run all these searches. 42:47.090 --> 42:51.590 So we're do this after we have everything set up with just, what is this, like 10 lines of code. 42:51.590 --> 42:54.050 So it's actually pretty straightforward to do this. 42:54.050 --> 42:59.230 And you can start to automate it then by passing in variables, call this from other functions from a reconnaissance standpoint. 42:59.350 --> 43:04.610 I'm going to restart, reset our variables so we don't have any existing output. 43:04.610 --> 43:06.230 So you'll see this cleanup. 43:06.230 --> 43:12.250 Okay , so now what we're going to do is we're going to leverage Microsoft again as our example, and we're going to search their org for their numbers. 43:12.250 --> 43:14.490 So we're going to go ahead and run this. 43:15.570 --> 43:21.290 And what it's doing is it's actually calling out, pulling in the max mine, searching for it, and then it comes back. 43:21.290 --> 43:26.950 And it went through, it looks like the latest row that had Microsoft in was 430,000. 43:26.950 --> 43:32.370 So it went through a considerable amount of data for this, came back , and then it loaded it all into a data frame. 43:32.370 --> 43:37.330 So we have all of the entries for Microsoft's CIDR ranges into there. 43:37.450 --> 43:50.250 So now what we can do is we can write that to a CSV file, or we can start to parse and manipulate and pull it into the next lineup if maybe you want to now kick off NMAP scans automatically, you could leverage this data and do, because it's normalized. 43:50.910 --> 43:56.990 So this actually wrote it to my, back to my recon page that's that static S3 site. 43:56.990 --> 44:00.590 So now I can have a list of here's the output of the different CIDR ranges. 44:00.590 --> 44:03.610 So I can just kind of see what I'm doing on an overview of it. 44:03.610 --> 44:12.090 So it's kind of neat too that collectively you can kind of now have this dashboard that you can watch as you do your own reconnaissance and bug bounty or anything that you're doing. 44:12.090 --> 44:14.010 And it's kind of fun to see that. 44:14.570 --> 44:16.390 So that's, that's really that demo. 44:16.390 --> 44:19.070 And then it actually, we can look at the org too. 44:19.070 --> 44:20.970 So we see it's 8075. 44:20.970 --> 44:25.550 So if we want to search for that org number in case the name's different or anything, we just swap out. 44:25.550 --> 44:34.330 And it actually ran both, but I'm going to comment it out so you can just see the results and we run that against it and we should have the output of the org number pretty quickly too. 44:34.330 --> 44:40.090 So you can see the orgs and it just happens they all have Microsoft in them anyway, but that's just another option you can always do. 44:48.780 --> 44:49.540 All right. 44:49.540 --> 45:03.420 So let's take a look at two more quick demos and one of these and you'll, you'll pretty much hear from me over and over, especially if you worked with me of how much that I don't like to do manual effort in spreadsheets or CSV files and, and from a metrics perspective, 45:03.420 --> 45:05.080 how can I completely automate it? 45:05.080 --> 45:15.700 And the whole basis of this talk is around automating those manual mundane things that just take time away that we should always have almost in real time or at least near real time metrics and data points that we can have. 45:15.700 --> 45:21.380 So I, I hate using Excel spreadsheets and building graphs and charts off of those. 45:21.380 --> 45:34.180 Now I'm fine with updating them and keeping them as kind of how do you maintain different components, but one of the neat things about pandas is literally you can, you can import Excel spreadsheets with literally one single command. 45:34.540 --> 45:38.640 And so when, when you can do that, it's so easy to do. 45:38.640 --> 45:42.280 I figured, of course, we have to be thinking about security awareness. 45:42.280 --> 45:44.080 Security awareness should always be in the back of our mind. 45:44.080 --> 45:49.260 So when we're working with coworkers ourselves, I, so what I did was I took the step and I said, yeah, absolutely. 45:49.260 --> 45:50.560 We need a mask for that. 45:50.560 --> 46:04.220 So as we're working in quarantine, when we're on our Zoom calls with our peers , I created these masks that you can pick up, definitely support, pay the, pay the AWS bills for this, but remind your, remind your coworkers on your Zoom calls that they need to modernize , 46:04.220 --> 46:06.480 they need to codify and they need to automate . 46:06.480 --> 46:15.000 So I, I recommend pop these on, show that, rock these out as you're, as you're in your Zoom calls and just remind people that we need to we need to move faster. 46:15.000 --> 46:16.020 We need to be more efficient. 46:16.020 --> 46:18.680 We need to automate, codify and move forward. 46:18.680 --> 46:22.700 So as we go, as we go into this, let's, let's check this out. 46:22.700 --> 46:24.700 So we're going to, we're going to do this one. 46:24.700 --> 46:25.400 This is BSIM. 46:25.400 --> 46:27.840 So it's called building security and maturity model. 46:28.040 --> 46:30.900 It's one of the methodologies around application security. 46:30.900 --> 46:37.420 So if you're building an application security program, trying to measure how you're doing, this is one of the models that you can leverage and do. 46:37.420 --> 46:39.500 So all this does is really it imports it. 46:39.500 --> 46:42.780 And this is what I've used to measure programs against over the years. 46:42.780 --> 46:43.740 And I really like it. 46:43.740 --> 46:45.900 I'm a big advocate, big fan of BSIM. 46:46.300 --> 46:50.860 So what we can do is we can actually load this spreadsheet within just a moment. 46:50.860 --> 46:52.200 And this is just a spreadsheet too. 46:52.200 --> 46:58.340 So even though we're working , we've shown massive data sets of terabytes and gigabytes of data we've been processing. 46:58.340 --> 47:02.860 You can use this for just like single megabyte spreadsheets as well and still automate and do. 47:02.860 --> 47:03.820 So this loads it in. 47:03.940 --> 47:07.560 We can see , and this is kind of from a metric scorecard dashboard standpoint. 47:07.720 --> 47:08.800 And it's also in GitHub. 47:08.800 --> 47:12.060 So we can check this out outside of this presentation as well. 47:12.320 --> 47:16.680 And then let's say we want to keep dashboards just for our management, our leadership of the scores. 47:16.680 --> 47:23.360 You literally can just run this command as you update, have your team members update spreadsheets as you complete a project or a program. 47:23.400 --> 47:28.640 And then you can have your spider charts that show kind of where you're at in the maturity loop, where you want to be, where you're at. 47:28.640 --> 47:30.040 And these are kind of interactive. 47:30.040 --> 47:36.940 You can overlay these and they measure based on how BSIM actually measures of how high you are in the quadrants. 47:36.940 --> 47:40.680 So just one cool example that's pretty lightweight. 47:40.680 --> 47:45.500 Here's your output chart of you can see where you're currently at, where your target is, and how you're progressing. 47:45.500 --> 47:49.040 So you can kind of build on this and think about all the different metrics, spreadsheets you have. 47:49.040 --> 47:54.940 That's literally one line of code that you can pull in an Excel spreadsheet into a pandas data frame and manipulate. 47:55.060 --> 47:56.780 And then you never have to do this again . 47:56.780 --> 48:02.920 I mean, that took two seconds to run the latest, greatest metrics around an application security program measurement . 48:03.720 --> 48:08.760 Here's another example just with a CSV of the CVE library . 48:08.760 --> 48:11.620 So it's a data set where you can go actually to MITRE's website. 48:11.620 --> 48:15.740 You can download the historical in a CSV format, all of them. 48:15.740 --> 48:18.720 So I've already kind of taken the approach of that . 48:18.720 --> 48:25.160 We can download the file and we can actually just import it into a data frame. 48:25.160 --> 48:34.380 So these few lines of code here will walk through and it loads up 176,000 different lines of CVE code. 48:34.380 --> 48:34.700 So use. 48:36.180 --> 48:43.480 So if you ever wanted to do analytics around those, trying to look at group by how many were issued per year, you can get this. 48:43.480 --> 48:44.520 You can run it pretty quickly. 48:44.520 --> 48:48.620 So over the past 20 years, you can see how the counts have increased over time. 48:49.380 --> 48:52.740 You can look at which months are busiest with CVs. 48:52.740 --> 48:54.780 Why do you feel like you're always working over the holidays? 48:54.960 --> 49:01.740 You can kind of use this to compare and forecast where you're going to most likely have more patch management resourcing needed in different things. 49:01.740 --> 49:10.900 So kind of cool insights that you can share and kind of make it actually as data-driven background and back you up on why certain things are happening. 49:10.900 --> 49:17.560 And I think as we go and build programs and security, we should always think in terms of data-driven mindset. 49:18.760 --> 49:22.400 The other neat thing is you can load additional libraries. 49:22.400 --> 49:31.560 So this actually loads Bokeh, which is a charting tool, a graphing tool within Python, which is used pretty commonly across the scientific computing community as well. 49:31.720 --> 49:35.800 And you can also write these to interactive websites and things so you can show the dynamics. 49:35.800 --> 49:39.780 So you can start to chart these out, you can graph, you can save them, you can load them, you can do. 49:39.780 --> 49:45.160 So just a couple lines of code, repeatable, reusable, and you can build these that you can just use over and over again. 49:45.160 --> 49:49.700 So tons of efficiency gain, tons of opportunities to do so. 49:49.700 --> 49:52.700 I definitely recommend grasping that. 49:52.700 --> 50:01.380 Don't do manual work in spreadsheets anymore other than maybe just populating data or loading it from an API, but leverage it in code and automate your things. 50:06.730 --> 50:17.110 As we wrap up this presentation, I forgot to mention that as you build this automation, you have services running, it's really easy with the cloud to build your SNS notifications. 50:17.110 --> 50:25.150 So these are simple notification services and you can text yourself, give yourself reminders, let you know where the progress is in any of the processes that you're running. 50:25.150 --> 50:30.310 So definitely a handy snippet of code to really leverage and utilize to your advantage. 50:30.630 --> 50:48.970 The other part that I wanted to mention is we've looked a lot about kind of red teaming techniques and passive reconnaissance in doing this, but what I want you to think about is if you're in an information security program right now working, look at this from a holistic standpoint of how can you help out the other InfoSec domains that you work side by side along, 50:48.970 --> 50:59.170 whether you're in vulnerability risk management, whether you're in third party, whether you're in a cyber fusion center, you're doing security engineering or architecture. 50:59.170 --> 51:15.110 Think about those outputs and inputs and how can you better take the data that you're generating within your department and how can you normalize it and pass it on to make maybe the risk management team help them make better decisions, have better normalized data that they can process and do. 51:15.110 --> 51:27.070 So think about those gaps and I think that's somewhere that you can really accelerate in your career and it's a gap that we have of where we're really good at building out and ensuring individual silos within information security. 51:27.170 --> 51:40.350 But if you can start to show from a career perspective how you can branch and understand kind of what are the value statements , what are the objectives of these different departments within your program, that's where I think you can really accelerate in places that you can really, 51:40.350 --> 51:42.830 I guess, just build and extensify your career. 51:42.830 --> 51:52.370 So I encourage you to take these principles, these platforms, these topics that we've talked about and start applying them to your daily jobs and I think you'll see some great success. 51:52.370 --> 51:54.710 And again, I really appreciate the time today. 51:54.710 --> 51:57.270 I hope that you've taken something away out of this. 51:57.270 --> 51:58.650 Definitely jump into the GitHub. 51:58.650 --> 52:04.510 I'm going to be continuing to build out blog posts and code snippets and accelerators in there. 52:04.510 --> 52:10.170 So in the weeks and months, definitely push me, help drive me to keep that populated. 52:10.170 --> 52:14.690 But I encourage you to definitely reach out if you have any questions as you go. 52:14.690 --> 52:20.070 But I appreciate your time and enjoy the rest of the DEF CON and the Red Team Village. 52:20.070 --> 52:22.990 And thanks again and have a great day. 52:22.990 --> 52:23.350 DR. 52:23.350 --> 52:23.970 MICHAEL www .microsoft.com