So, my name is Jay Cran. I am the VP of Product at Bug Crowd. I think people have heard of Bug
Crowd before. I saw a few nodding heads, a few Bug Crowd t-shirts in the back, which is cool.
Basically, this is a tool I've been working on. It's open source. It's BSD. It's not affiliated
with Bug Crowd exactly, but it's something I've been working on and we think it's a good fit in
general. Yeah, good. Of course. Steady audio. Oh, am I floating in and out? All right, let's
try this. Anyway, we sort of have this philosophy about open source tools and this idea of spreading
to the community usable tools for bug bounty hunting and for testing and for, in general,
just doing OSN because it's a big part of security testing, too. So, this is a tool that I've been
working on for a while. I'll give you this super quick demo here. We'll jump into slides and then
we'll go back to videos that I've prepared. So, the idea... How many people were here for the
data split talk? Quite a few hands. Well,
so many people were here for the data split talk. So, the idea... How many people were here for the
data split talk? Quite a few people were here for the data split talk. So, the idea... How many
similar ideas between that talk and what you'll see here. It's cool to see what other folks are
doing in the space. It's cool to see how they think about the problem. There's a lot of open
data sources we can all use. It's about combining them and being able to tell a story with those
pieces of data. And so, you'll see very similar ideas here, but a little bit different in sort
of how I think about it. So, there's this idea of an interface, a GUI interface that
you're looking at right now. It's a web app and it's also an API. And so, you can use it
from this GUI or you can also pretty much do anything in the UI through the API.
And so, let's go ahead and actually I'll give you the super quick intro. Create a project.
Let's call it a test DEF CON project. And it creates a project and it drops you into this
module runner. And you can think about modules sort of like tasks, sort of like units of
gathering information. Just like a transform in Maltego world, it's very similar to that idea.
And then there's this idea of entities. And so, you know, we had a list of tasks. Searching GitHub,
scraping websites, doing DNS enumeration basically, and a bunch of different techniques.
What's kind of cool about this module runner is that if you hit this, it basically like changes
the form to ask you a bunch of questions based on that module. So, I mean this is designed for
technical users, but in general like it's meant to be somewhat usable by folks who maybe this is the
first time they're trying these sorts of things.
So, it asks you some questions. Somebody give me a domain. Anything.
What is it?
ibm.com
ibm.com. Let's do it. ibm.com. And let's, this thing's running on AWS server. So, let's like
say brute big and let's give it 10 threads against the DNS server. And then let's run it.
And then you see that little auto-enrich button at the bottom. You'll see that in a second.
That'll basically give you a bunch of questions. So, let's go ahead and do that.
So, I'm going to basically give you some things that I'll talk about in a second.
But you see it like starts to kind of enumerate it right there.
Oh, sweet. We're done copying. So, I'll take that.
All right. And so, it's actively enumerating while we're doing that.
And each one of those creates a little, another entity. So, like a DNS record here. And if you
click on that, you can see it. So, it's like a DNS record here. And if you click on that, you can
actually see, sorry, it's a little bit hard to see probably. But you can kind of see how, we don't
really have any details about this yet. But we do know that it's this IP address. We also know that
it's Akamai. Yeah. There you go. Cool. Can you guys see that in the back? Cool. Some nodding
heads. Awesome. So, we capture this information about this entity. So, we resolved it automatically.
We figured out that there's some other hosting. So, we're going to go ahead and do that. So, we're
going to go ahead and do that. So, we're going to go ahead and do that. So, we're going to go ahead and do that.
We know that we created it from that brute domain subforce. Sorry, DNS brute sub module. And we've
got some other entities here. And notice this little H. It's actually, there's some rules built
into it for stuff that's hidden in general, right? Like Akamai, we tend to like push away
because there's not much we can do with Akamai as testers. It's just a front end basically.
So, anyway, while that's running, it's filling out this entities page.
And I've got this in mobile mode. So, it may be showing up a little funny.
But you can see here, if you just keep scrolling down, you've got those entities. And then all
the aliases for them here. And so, you end up with this list of things that you've discovered.
And if you correlate those, you kind of end up seeing, oh, actually, that IP address is gw.ibm.com.
That's that. And you can now start to see load balancers and things like that. So,
you get this correlation that's happening.
Automatically. But they're each independent. And each entity can also have things run against it.
So, say we wanted to NMAP scan that. You can just go ahead and NMAP scan it.
And it'll do the right thing. So, it kicks off an NMAP scan. Okay. Let me just see here.
How about that?
Okay.
Okay.
Okay.
Got my slides now, which is great.
All right. Cool.
So, that's a super quick demo. That's a super quick intro to the concepts of it. Let me walk
you back for a second. And let's talk about what this thing really is. It's a framework. Much like
other frameworks we've heard today. Many similar concepts to Metasploit and to Maltego. It's built
in Ruby.
Those tasks are individual modules. And it's really oriented more toward discovering
organizational attacks. So, what I mean by that is you can punch in a top-level domain of a company
and it'll fill out everything it can figure out about that company. And I'll show that to you.
I've got a demo for that. It's written in mostly Ruby. Though there is a, you know, we link out to
NMAP and to mass scan and a few other tools where we haven't rewritten it in Ruby.
Okay.
It's also designed mainly for technical users. And it's useful to kind of think about
this thing that I'm showing you as an engine. Because it's also an API. And you can use that
to send information out of it or put information into it and enrich that information.
So, just architecturally, what you were looking at there was the GUI that sits on top of an API.
It has all those concepts of tasks, entities. I'll talk about strategies and handlers in a second.
That sits on top of Postgres and Sidekiq and Reddit.
So, it's pretty massively parallel. You can do lots of things at the same time.
If you've got a powerful box. And so, those concepts that I mentioned, entities, enrichment,
tasks, aliasing strategies. I'll just demo these things for you. Entities. I think everybody here
is pretty familiar with this concept of an entity. It's a thing, right? So, you think about Maltego,
right? Creating a thing. With Intrigue, we have these types over here on the right-hand side.
Some of those are going to look very familiar. DNS record.
IP address. Person. Physical location. Those are kind of things that we know about and they're
built into an ontology. And the only real requirement about these things is they have a
name, right? And a unique name. So, every entity is unique. A name and a type. And basically,
validation. You can validate lots of things about an entity, but the name is the most important
thing. And then there's this enrichment process that allows us to automatically look up a DNS
record, you know, to get the IP address associated with it. Or to find a website and know that, you
know, another website is the same website because we looked at the HTML and it's exactly the same.
So, just to demo that for you. I think I'm going to do it live. What do you guys think?
Yeah.
All right. Cool.
I like it. I like it. All right.
So, we basically kind of did this, but let me show you the thing that I want to point out here.
I have this module called, or this task called create entity.
Okay, cool, whatever.
We can create any of those entities I just showed you.
In this case, I'm going to create, let's create a DNS record.
And my stuff's more oriented to the network side.
I mean, like you saw with data split, it's more oriented to the people side and things like that.
This, in general, you can do a lot of the same things in these tools.
But it's about the flows that you build into them.
And so a lot of the flows are built around enumerating a network and the devices and the websites and things like that on a network today, right?
But there are lots of ways to extend on that.
Anyway, let's create intrigue.
Actually, let's do IBM again.
And let's not.
I'm going to leave this iterations as interactive for right now.
And I'll explain what that means in one second.
And I'm going to keep this auto-enrich box checked.
So, cool.
We created an entity.
Nothing spectacular there.
But if you do click on this, you'll notice it automatically enriched that.
And it automatically associated it with that IP address.
And so now, if there were multiple IP addresses, I'm going to give you an example of something like that.
I know Yahoo resolves to a lot.
They've got a lot of load balancers and things like that.
So, Yahoo.
I'll just create an entity, DNS record.
Yahoo.com.
Auto-enriching.
Run it.
Cool.
Click on that.
And you'll see, oh, cool.
It's actually all these things.
And there's a bunch of IPv6 stuff there, too.
Cool.
All right.
And then actually show me how that looks for everything.
Cool.
All right.
And again, remember, we can correlate this and actually show it's all the same thing.
Even though those are a bunch of different IPs, now you know they're basically all the same system.
Or there's a load balancer in front of them, right?
Cool.
Yeah, same thing.
All right.
Sweet.
So, that's creating an entity.
That's enriching an entity.
Any questions so far?
Yeah.
So, that's basically focusing on that single entity, right?
Yeah.
It's not like anywhere else.
Right.
Yeah.
But I'll show you that.
And I wanted to start with, like, something simple.
So, tasks, right?
We just ran a create entity task.
There's other tasks.
And, man, it's really hard to see that.
But I'll just give you an example.
By looking at the UI.
Sorry for putting green on black.
I thought it was DEF CON, so I was in this period.
Yes.
Cool.
Maybe that's a little easier to see.
Again.
So.
And these do largely what you'd expect, right?
Just.
Yeah.
Yeah, yeah, yeah.
And we're actively adding this up.
It's a framework, right?
You can contribute a task.
I'll show you the code in one second just so you get an idea of what it looks like.
Very similar ideas.
Some that are relatively new that are kind of interesting.
There's a lot of stuff being discovered right now with AWS S3 buckets being left open.
Nodding heads.
Yes.
Yeah.
People have seen that one.
Basically, it's really easy when you set up an S3 bucket to, like, set everyone permissions
or set authenticated user permissions, and you end up leaving stuff that you probably
ought not leave in the public, in the public.
For example, I was just...
The reason I added this, actually, I was messing around when...
I think it was...
It was Chris Vickery that found the 200 million voter database.
Did you guys see that?
Basically, 200 million accounts that got dumped by a provider of this sort of data.
And I was just floating around and happened to do a Google dork for something that I thought
might be similar.
I was trying to find it.
And found that there was another one by the state of Virginia that was also open, just
open in public.
And it turns out...
No, sorry.
It was North Carolina.
And North Carolina actually mandatorily requires you to gather.
This information and store it, right?
Which is pretty crazy.
They changed the law recently.
So these sort of databases are out there and floating around.
But that's a super digression.
Sorry.
Anyway, I built some modules to, like, help find those things.
And so let me give you an example of those one real quick.
So here's some buckets we can brew.
IBM, anybody else have buckets?
Sweden?
Sweden?
I don't know.
I haven't heard about Sweden losing all of its data.
We'll talk after.
Yes.
Holy crap.
All right.
Poor Swedes.
I feel for them.
Cool.
Do your thing.
Hit run task.
Does that actually work?
I think it's chugging right now.
Give me one sec.
All right.
This is a live demo.
So give me some slack.
You had mentioned box size earlier.
Box size.
Yeah, I would definitely go with, like, 8 gigs.
Like, it's Ruby, and then it's pretty massively parallel.
You can scale it pretty well.
So you can – let me show you a task, and then I'll show you sort of how that works.
No, no, no.
It's fine.
It's a good question.
The question basically was, like, how big of a box do you need to run this?
It's not actually that bad.
Crap, I can't show you because github.com.
Intrigue.
Intrigue core.
There we go.
So this is actually the code.
This is the easiest way for me to show this because it's not my system.
Here are the tasks, and we were looking at AWS Brute.
And so there's some metadata about a task, which basically lets you – that's how it manages all that ability to tell you which things are allowable or not in the module runner.
And then there's a run method, and that's pretty much it.
Yeah.
There we go.
So, I mean, the run method does the right thing.
In this case, it's a pretty dirty module because we're using Nokogiri to scrape HTML, but, I mean, you can use APIs to do these sorts of things, too.
Let me give you another example where it's a little more clean, like searching Shodan or Census, right?
Metadata.
This is going to look a lot like Metasploit in the way that it's set up, right?
There's this idea of a global config that it can pull from, and then it just, like, hits the API, right?
And grabs the results out and creates – this has this idea of creating an entity in this.
Anyway, coming back on track here.
As I mentioned, you can pull data from APIs.
You can scrape data.
Similar concept to transforms or modules and other frameworks, and then that run method really matters.
So I showed you DNS subdomain brute forcing, so I'm going to skip over that.
I mean, let's go back and actually check on it and see how it's doing, but otherwise, let's move on.
I think we did – I don't remember where we did that.
Actually, let's just run it again.
DNS subdomain brute force.
Choose a DNS record because that's the entity type we're going to use.
IBM.com.
And then do the right thing.
I just hit enter.
Sorry, I sped right through that.
It has a list built into it, but there are other sort of options.
And you can set threads.
So, like, if you have a bigger system, you can really thread this thing out.
Ah, right.
Let me get to that.
And so there's the demo of brute forcing.
Oh, also, one small point before we do that.
The brute force is pretty smart.
There's this idea of permutation.
So if it finds a dub-dub-dub.
It'll actually try a bunch of permutations on that.
And that's a pretty good way to find bugs.
Prod, stage, stuff that just gets left open.
So we talked about aliasing.
We talked about the idea that an IP address and a DNS record can really be the same thing.
And you want to preserve that relationship, but also you want to be able to split it apart
and look at them differently because that allows you to see load balancers and things
and really understand if there's stuff on Akamai
or if there's stuff on – yeah.
If there's stuff on different systems.
So that's really what aliasing does for us.
But let's just explore some data.
This is all well and good.
I think you get the concepts.
Let's go to DeskCon.
So we're just going to let this load.
Cool.
And you see that there's this searching.
Yeah.
And the ability to filter, show me all DNS records in this project, search.
Sorry for the UI here.
But that's every DNS record in this particular project.
And if you want to search like a name, Yahoo, or maybe the name NotYahoo,
give me everything that's NotYahoo in this that we found.
And so now you start to see all these different systems that we've been able to find.
But how do we find this?
What is the strategy?
What are these iteration things?
Why are they important?
That's really the question here.
So let me show you that.
Let's do IBM.
Let's create an entity.
And we're going to do this – and remember I said at the very beginning, this is organizationally
oriented, right?
You give it a top-level domain for a company, and it'll just kind of span out from there.
So we'll give it a DNS record of IBM.com.
And we're going to use the network enumeration strategy.
And what's a strategy?
Strategy basically says, when you get – I don't think I can step away from the podium
here.
But when you get a DNS record, run a Whois on it.
Or when you get an IP address, Nmap scan it.
And you can build these strategies of like allowing you to kind of build out your own
machine, if you will.
Of how you want to do it.
Now that's a very simple one.
And it basically is one single iteration of this.
But this recursion allows you to fan out.
So instead of setting that to interactive, where it's kind of man in the loop, I can
run modules one by one.
In this case, we're going to say, go ahead and run a bunch of modules, depending on what
you get.
So if you get an IP address, go ahead and scan it.
If you get a DNS record, go ahead and brute force it.
If you get something that looks like an AWS bucket, go ahead and try to brute it.
So we'll do six.
Because you don't want to scan the entire internet at once, right?
You want to limit it.
So there's this idea of iterations.
I try to be sensible in my iterations.
Anything more than eight finds the internet.
So we'll put it at six.
And we'll just run this.
And we'll leave enrichment on.
Enrichment is that idea of correlation between entities.
And it creates an entity.
And then it starts to fan out.
Cool.
Let's see if my demo works appropriately.
Actually, I may do a video for this one, guys, just because it's live.
Yeah, let's just do the video for this.
And I'll talk you through this.
So create a new project.
We're going to fan out, right?
We're going to create an entity.
And it's going to be a DNS record.
And a lot of this stuff is oriented around DNS today.
But it could be people.
It could be anything.
We're going to use Yahoo.
We're going to use network enumeration because we want to do the network side of this.
That's the strategy we're using.
And we'll do six iterations.
And we'll enrich.
And it runs.
I'm making this as big as I can.
I'm trying to talk you through it if you can't see it in the back.
And it's starting to fan out.
And do that look up.
You notice that it found net blocks there.
Those net blocks are associated with Yahoo.
We just grab all the information.
And we grab that from Whois.
Pretty basic.
And I notice in there it's yahooinc.com that that's registered to.
So I'm going to go start another fan out from Yahoo Inc.
Because I want those net blocks to be automatically enumerated.
And it's got a rule that basically says if the name of the entity, the top level entity is in there, go ahead and enumerate and build it out.
Cool.
And then you'll see this stuff starts to get scanned right away, too.
See that tasks run on this entity?
It actually kicked off a mass scan on port 80 across that net block.
Which is going to take a while to run, to be totally honest.
We're not going to have time to show you everything that comes out of that.
But see how all this stuff just fans out?
This is all based around the strategy.
You have to build a strategy that makes sense.
You don't want to scan the entire Internet at once.
I mean, you may.
We do.
But it's all based around the strategy.
It's not a good idea unless you're properly equipped.
And you can search.
I think I just showed searching there for all the mass scan stuff.
Notice that it creates IP addresses and network services.
It does try to look up and resolve and create URIs.
And URIs become, like, websites as well.
And I'll show you something cool with websites here in just a second.
I've got another demo.
Anyway, that's pretty much the end of that.
Let's fast forward a little bit.
And let's show you sort of how that looks once we have the data.
I keep telling you about the tool, but the data actually matters.
And see, we've got these statistics over here.
So it tells us basically what it was able to find.
If you click those, it'll take you right to here.
Cool.
Lots of IPv6.
And notice the Nmap scan results are being pulled in, too.
So I've got, like, again, hard to see in the back.
I apologize.
But I've got the OS and things like that being populated there.
And if you correlate it, it gets really interesting.
Because you can see load balance.
You can see load balancers very easily in front of these systems.
.
Yeah, really good question.
Even with low numbers of iterations, maybe two, how do you avoid, you know, scanning Akamai?
Really good question.
Really boils down to looking at the DNS name.
And if the DNS name says Akamai or Akamai Edge or, you know, one of a set of things.
.
Yeah.
Exactly.
So there's a prohibited list in this that's kind of built into it.
I'd like to expose it.
But I'm not sure exactly where to expose it yet.
I'll show it to you.
.
Correct.
I'm not sure who asked that.
But yes.
Yeah.
Right on.
Yeah.
Yeah.
And the idea of, like, when I get something, as long as I'm not at that depth, right, I'm not over my depth that I can't go further.
Go ahead and do everything that you know to do for that particular entity.
Right.
So.
Yeah.
Tasks.
Helpers.
And then I think there's a prohibited list in here.
.
This is the list of stuff that I will not do.
It basically comes down to, like, Akamai is a lot of that.
Azure.
.
You just don't want to go into Azure's infrastructure.
It's cool about infrastructure that sits on top of it and is DNSed appropriately for your company.
But beyond that, you don't want to go scan the Internet.
And this can be edited, obviously.
Anyway, let me show you another demo.
How am I doing on time?
Am I close?
Do I need to get kicked?
Okay.
Cool.
Thank you.
Thank you.
That's my man.
I appreciate that.
He's like, whatever, man.
I like it.
You just do you.
.
I mentioned it's an API.
You can also use it from the CLI.
I have a demo for that.
But this is a cool demo.
Not a lot of people know about this.
This is one of those, like, kind of fun ones that I like showing people.
Again, it's small back there.
I apologize.
I'll talk you through it.
We just opened up the URI spider module.
And we're giving it a website.
So I'm going to give it whitehouse.gov.
Because why not?
.
And we'll give it some threads.
Ten threads.
Five threads.
Because we're on a slow pace.
It's a slow connection.
Extracting DNS records, which is a pretty cool idea.
So anything that matches .gov will actually get extracted out as a DNS record.
Go ahead and give me email addresses.
Give me phone numbers.
Don't give me URIs.
Because that will give you a lot of crap.
And go ahead and run it.
And it starts right away.
So we found www, obviously.
Even I can't read this.
I apologize.
Go.whitehouse.gov.
Senate.gov.
We found a phone number.
.
We found an email address.
Karen Pence's email address.
Thanks, Karen.
Clinton Library, Nixon Library, blah, blah, blah.
So you get it.
It's basically scraping each page on the website, spidering, and then extracting out things that it knows about based on a rule matching engine.
And all this stuff is extensible.
It's pretty easy to write and to expand on it.
But here's a cool one.
That software package there.
We found Adobe PDF Maker 11 for Windows.
How do we get that?
We actually parse PDFs.
And pretty much any media file.
There's this cool library called Tikka.
And Tikka is super easy to embed.
And so it fires off Tikka for each PDF we find.
And we parse it.
And we give you all the information.
There's an EXIF that got parsed.
So we just kind of grab everything and parse it if we can.
And then store that data as JSON.
So all of this can be grabbed as JSON, by the way.
Like, literally, everything is available as a big JSON blob.
So you can do whatever you want with it.
That's pretty easy.
That's that demo.
And I'll show you the CLI quick.
And then I think I'm getting kicked off.
Yeah.
Hey.
We can continue in the hallway.
It's cool.
Seven minutes?
All the time in the world.
I love it.
This is just hitting that same API.
And I just listed out the modules.
You can run it as a command line tool.
It's a command line tool.
It's built in.
And I'm telling it to run.
And I'm telling it to, in the default project, go ahead and kick off a subdomain brute force
for whitehouse.gov.
And make sure I got my syntax right.
Cool.
It kicks it off.
It gives us the result ID, which we could pull as JSON and actually display it here.
But it'll take a long time to complete.
So the idea of web hooks, I'm building into it eventually, roadmap.
And you see, it's kicking it off in the UI, and everything just works in the UI, too.
So you can automate stuff via the CLI.
You can automate stuff via the API if you want to build it into a pipeline of some sort.
Anyway, that's that.
I think I'll take questions if there are any.
I'll leave it there.
I've got a question in the back.
.
Yeah.
Cool.
And how can you get involved?
Cool.
I just roped you in, by the way.
Thank you.
Yeah.
So actually, intrigue.io is the easiest way.
It'll get you to the right place.
And it'll help sort of steer you in the right direction.
I did put my slides up here.
If you're interested in those, I'll post the videos later.
Hit the news thing, and you end up here on a blog.
I guess I really like black and blue as a color.
I don't know why.
Getting started is probably where you want to go.
Getting started with Intrigue on Docker or getting started with Intrigue on AWS.
As I mentioned, there's quite a few dependencies in Postgres, Ruby.
So managing these things are a pain unless you set up.
Unless you're familiar with Ruby.
So try to Dockerize it or give you an AWS image that you can just play with.
Makes it easier.
Yeah.
So two questions.
First, how can you do this?
No, no, no.
Go ahead.
Sorry.
I just happened to Zoom.
You first.
So first, how can you do this without Cloudflare wanting to rape you and, like, blocking everything?
How can I do this without Cloudflare blocking me and doing other really bad things
to me?
Well, I mean, listen.
Every hosting provider has their rules about scanning.
Every hosting provider has their rules about what you can and cannot do.
Some are more friendly than others.
As long as you're talking to the AWS folks, they're really cool.
Their security team is pretty cool with the idea of research.
Linode is another that's really good with research.
And so as long as you choose a provider that's open to these things and you're not doing
anything terribly malicious, you're kind of surveying data, you can find people that are
good.
Yeah.
Yeah.
That was my second question, actually.
Nmap has two different modes.
Like, there are some that are actually offensive in nature and there are some that are more
passive and can cause damage.
Does this have anything in it that, when running against a target, would cause problems
where, like, something would crash, et cetera?
It's mostly meant for the external side.
At this point, I haven't done a lot of testing internally.
So the short answer is I'm not super aggressive with my scanning settings.
Like, there's a pretty good balance in there.
Yeah.
We're certainly not trying to knock systems over.
Yeah.
Yeah.
Yeah.
So I stayed away from social media for a long time.
I need to actually build it in.
The short answer is I'm going to say roadmap.
There's a little bit of stuff in there today.
Let me actually just pull it up and show you.
And I haven't tested this.
This is truly a live demo.
Phone number lookup.
Let's see.
What was I going to do?
Okay.
Let's create a person first.
Anybody volunteer?
Donald Trump.
Donald Trump.
Yeah.
I like it.
Donald Trump.
And we're not going to do any iterations.
Do that.
Click this.
And web.
I can do a web account check.
It's not that great.
I need to actually do more searching and scraping of social media.
We'll say roadmap on that.
All right.
All right.
All right.
But here, I'll show you.
You can search Bing.
Oh, also, I should show you the configuration for this stuff.
Oh, am I out?
I'm getting kicked out.
Sorry.
Last thing then.
Configuration is super easy.
It's all built in here.
You just click this thing.
Go sign up for the key.
Come back.
Plug it in.
Let's see which one that was.
That was data.gov.
I tried to send you to the right place so that you can just sign up for a key and make
it easy.
You don't have to have every key.
Models will work without them.
Try to fail gracefully.
That's it.
All right.
I'm kicked out.
Cool.
Awesome.