Hello everyone, my name is Apoorva Singh Gautam. I would like to thank Defcon Retain Village for having me here. I will talk about automating threat hunting on the dark web and the things that surrounds it. I presented this talk, I presented a short version of this talk at Grimcon and this is the extended version of it. So let's get started. I will switch off my webcam so that you guys can focus on the presentation here. Okay, so a little about me. My name is Apoorva Singh Gautam. I go by the handle ASGScorpion. I am a security researcher. I started into threat intel or hunting two years back and I've been loving it since then. I'm currently pursuing my master's in cyber security at Georgia Tech. Recently, during the summer, I was a research intern at UC Berkeley doing research in threat intelligence. Some of my hobbies are gaming. I pretty much love Rainbow Six Siege. I sometimes stream it also. I love hiking. I recently started into lockpicking and I've been enjoying it. I contribute to the security community. It's my passion contributing to the security community. I'm a senior teaching assistant at Siberia. I contribute at Station X also and at the local security meetups. And there are my socials if you want to contact me or hit me up. So what's today's agenda? So we will talk about introduction to dark web, what dark web is, how to access dark web, what Tor is, what is the difference between dark web, deep web, why you should perform hunting on the dark web. Before that, we will discuss what is threat hunting and why it is crucial to hunt on the dark web. We will discuss different methods to hunt on the dark web. Can the dark web hunting be automated? What's the pipeline or the architecture of hunting, automating the dark web hunting? Then we will discuss a little about threat intelligence lifecycle. That's how threat hunting on the dark web is analogous to threat intelligence lifecycle. What steps are there that corresponds to this? We will discuss a little about operational security, that's OPSEC, and why is it important to secure yourself when hunting on the dark web. And yeah, that's it. So introduction to dark web. I'm sure you must have seen this image a lot of times on the internet that shows difference between surface web, deep web, and dark web. So the surface web is the sites that are indexed by different search engines, that's Google, Bing, Yahoo, etc. And the majority of the portion of the internet is deep web. Now deep web is any site, any website that is not indexed by the search engine. So this can include your databases, your server instances, or any other different websites that you cannot search from the search engines. The third one is the dark web that we will talk mostly about today. And this is the part of the internet where you need some kind of software, special software to access the dark web. So this can include anything related to like drugs or weapons that's being sold on the dark web, or some kind of research, or books that's sold there. So dark web includes several forums and marketplaces where people sell different kind of things. So these type of things are sold there. So majority portion of the internet is deep web, and people confuse it between dark web and deep web. So as you can see, only 6% of the internet is dark web, and 80 to 90% is the deep web. Moving on, so how do you access the dark web? So there are several companies, several organizations that offer their own dark web systems, or you can call it, so the famous one is Tor, that's the onion router. The second is I2P, that's an invisible internet project. Zeronet is also becoming popular nowadays. So Tor has .onion domain, like the domain name ends with .onion, and I2P is .i2p. Talking more about Tor, what Tor is and how it works, is it's like a three-layer proxy system. If you can see on the image here, there are entry nodes, the entry node is where your traffic goes to, and then it goes through the middle layer, and the exit layer, and then goes to the destination. This way your identity is hidden, and also only the entry node is publicly listed, rest of the nodes are not publicly listed. So it hides your identity, it hides your IP address, and these nodes are volunteer-based systems. So Tor has about like 6,000 relays, and I don't know the number about I2P, but it is also becoming more popular nowadays. And the major thing about Tor is each node only knows the IP address of the next node or the previous node. So if we talk about here, the entry node doesn't know about the exit node or vice versa. So in this way, the location of the nodes are also protected. So there are many misconceptions about Tor, the dark web. The famous one is whenever we talk about Tor, people think about criminal or criminal things that goes on Tor. Yes, there is criminal side of Tor, but there is a good side of the Tor also, or the dark web. So it's really famous among whistleblowers or activists. There are many countries where free speech is limited. So they can use Tor, like people from those countries use Tor to express their speech or express what they think. Tor also has access to many old literature or researches, which is not available on the open web. And it's like safe haven for journalists, obviously. So there are many popular sites like Facebook and New York Times that has their counterpart on Tor, like their .NER website counterpart. So it's useful for whistleblowers or activists. The second thing is Tor is so many people think that it's illegal to access the dark web. So it's not illegal to access the dark web. It's illegal to indulge in these kinds of activities like purchasing drugs or purchasing any other illegal things on the dark web that is being sold there. But it's completely legal to access the dark web. And it's like the last thing which I would talk about is many people think Tor is like really big. But if you talk about the uptime or the availability of sites, there are very few onion domains on the dark web if you compare it with the clear web. So it's like the dark web is like very little part of the internet. So we will talk about the dark side of the dark web. I mean, the criminal side of the dark web. So there are many forums or marketplaces on the dark web. These are some of the relevant sites that's relevant to security researchers or people who want to access dark web for their organization's benefit. So this is like some of them are like a credit card market. So where different credit cards are being dumped or remote access. So these forums include remote access trojans or some kind of remote access tools or insider threats. So insider threats are it's like recent coming up a forum where insider like the people who are selling their company's secrets, they talk amongst themselves. So these are some of the relevant sites. Now coming to the cost. So how much it costs for some kind of for something to buy from the dark web. So as you can see, it's really easy to buy these things on the dark web and it. So like SSN, you can buy any SSN for $1 or fake FB friends, a fake FB with 15 friends, mobile malware, bank details, so exploit to zero days. So it's like these type of things are easy to get on the dark web. That's why security researchers and other people are focusing more on the dark web. You might have heard about recent news, like 500,000 Zoom accounts sold on the dark web or 267 million FB user profiles sold on the dark web. So there are many data breaches occurring day by day and they are being sold on the dark web. That's why researching on the dark web is really important. These are some of the product listings from the forums of the dark web. This is how the products are being listed. And you can see these are the average cost of accounts for different online services like bank services, what's the average cost of it or what's the average cost of service for video games. This is the average cost of tools that is being sold on the dark web. As you can see again, bank and financial average cost is $74. So you can buy a brute forcing tool for some bank at $74. Now coming to why you should hunt on the dark web. Before that, let's talk about what is threat hunting. Threat hunting is proactive searching for cyber threats. Proactive means before the attack happens. That's proactive search. You search for cyber threats from logs or indicators of compromise. That's IP addresses, emails, domains, etc. or textual data. That's what we are doing when we are searching the dark web. So it's basically hypothesis based because there's nothing concrete about the process. You take one use case and work on it and then you take another use case and work on it. And it goes iteratively in the same way. Many times there's use of machine learning or natural language processing that's NLP and advanced analytics process in this because you need to scan through the textual data if you are hunting on the dark web or if you are hunting on the clear web also but for textual data. So machine learning and advanced analytics are useful there. So why it's serious? Why threat hunting on the dark web is really important? What's so big about that? As I told you about again, there are many forums, marketplaces, dump shops. So what criminals or actors do? They learn new methods and techniques on the dark web. They monetize their skills. They trade their exploits or tools or even drugs and weapons and communication. So they communicate with each other and they share their ideas for new attacks. A security researcher or a person who is researching on the dark web, he can find a lot on, he can learn a lot while engaging within these communities. You can learn their techniques or TTPs, that's the attack techniques and procedures, how they think about the attack, how they plan an attack and it can identify. So if you do it correctly, it can identify attacks in the earlier stages, that's planning and recon stages and you can reduce the impacts that it causes. So suppose if your organization data is being sold on the dark web, there are different kinds of impacts that can cause to your organization. So some of the direct impacts are like personal information stolen or healthcare records stolen or even your company's trade secrets. And some of the indirect impacts are repetition of your organization, revenue loss, and nowadays the legal penalties that your data is lost and you have to cover the cost of the customers. So this is like, that's why this is really important to like to research on the dark web, to hunt threats on the dark web. On the same lines, these are the benefits of the threat hunting. So if you do it correctly, you can keep up with the latest trends of the attacks, you can get new TTPs, that's tactics, techniques and procedures, you can identify insider threats, you can discover data breaches. The main thing is you can prepare your SOCs and incident responders to deal with the attack because they will know before only what are the TTPs attackers are using. So they can reduce the damage and risk to the organization by acting quickly on that. So coming to the methods to hunt on the dark web. So we will discuss about some tools that's used to hunt on the dark web. And then we will discuss about the human element that can be used to hunt on the dark web. So talking about the first tool that's really, really important for this is Scrapy. So it's a web crawling framework. It's so famous, it's so important because it manages multithreading automatically. So you don't have to write, you don't have to spend too much time on the multithreading part, because it has already capabilities for multithreading using one or two lines of one or two lines of parameters. The second thing is Tor. Obviously, if you want to access the dark web, you need Tor. Onion scan is another tool that is used to search for onion websites. It can tell you if a website is up or not and the correlation between different websites on the dark web. Coming to Pryvoxy. So Pryvoxy is a web proxy. Before getting more into this. So when you access the dark web, you need some kind of proxy to access the dark web because your ISP, as I told you before, the entry nodes are publicly listed. So your ISP can have a blacklist to block the entry nodes. So you can't access the dark web or even if the ISP doesn't block it, he can see whether you are accessing the dark web or not. He cannot see what you are doing on there, but he can see whether you are accessing the dark web or not. So you might not want that. That's why you need some kind of proxy and a majority of people use SOCKS proxy. So basic difference between HTTP and SOCKS proxy is SOCKS proxy is a lower level proxy and it works on the SOCKS protocol. HTTP proxy only works on HTTP or HTTPS websites, but SOCKS proxy can work on other protocols also. And there are different tools to use SOCKS proxy like TSOCKS, Polypo and Pryvoxy. I've been using Pryvoxy and it has been so I don't have any problem with Pryvoxy. So and it's good. So another thing is SCP doesn't allow you to use directly SOCKS proxy because it doesn't support SOCKS proxy. So that's why you have to use these tools like Pryvoxy, TSOCKS or Polypo to route your SOCKS through Pryvoxy scripts. And there are other tools also like their search engines like Kilos or Recon where you can find different end-to-end domains. Apart from using SOCKS proxy, you can also use VPN with Tor for extra layer of protection and like encrypting your data. So getting more into SCP part, this image might seem a little confusing, but I will get into it like step by step. So this is like why SCP is really important and why SCP is so useful in hunting on the dark web. So if you can see there, so for explaining this, I will explain it in terms of Python code. So suppose everything you see here is a different Python program. So spider is a Python program, downloader is a Python program, middleware is a Python program and so on. So what spider does in spider? So in spider Python program, you give your onion domain on which you want to crawl the data or which you want to get the data. So it gives it to the engine. Suppose engine is just the program that manages every other Python programs. So it gives the onion domain to the engine. The engine gives it to the scheduler. So what scheduler does is here the multi-threading concept comes into the picture. Scheduler gets different domains and schedules it accordingly into multiple threads. So the onion domain goes into the scheduler, the scheduler gives it back to the engine and engine gives it to the middleware. So middleware program includes your proxy program and or login program. So what proxy program is that in proxy function. So if we are talking about middleware, that's a Python program, there's a proxy function into it. So proxy function is where you will note, you will put down your privoxy IP or Tor IP, so that the request response goes and comes through that proxy so that you can access the dark web. The login program, the login function is where you will put your user agents or cookies or so for accessing the dark web forums, you have to, so there nowadays, for all the forums, you have to have an account to access the dark web or to access that particular forum. So to access the forum, you need some kind of cookies or also there are many forums, many high level forums that implement captures. And as Google doesn't work on the dark web, these captures are like image-based capture or text-based capture that is easy to bypass. So you can use any machine learning capture bypassing service or any capture bypassing websites like death by capture or anti-capture to bypass the capture. So these are all codes you will write in this login function in middleware program. Now your request or your traffic goes through this middleware program to the downloader. What downloader does is it's a simple program to extract the HTML and save and give it back to the engine. So downloader extract the HTML and gives it back to the engine. Now engine gives it back to the spider. Now there's another function in spider that extracts the HTML entities that you want from the forums HTML, suppose forum name or document ID or the text-based data or author name who posted a particular content. So you get that and it is called items in Scrapy. So you get these items and it sends it to the item pipeline. Item pipeline is where your database is configured. So I use Elasticsearch, you can use any database whether SQL or NoSQL and it directly saves the items to the SQL. And so the important thing to note here is that Scrapy, so as I told you before multi-threading is automatically handled by Scrapy. The another thing is you don't have to give multiple onion domains to spider. So when the downloader gets the data, when the downloader gets the HTML page from the particular forum, you don't have to configure the code, there is a code to get all the onion domains on that particular HTML page. So the scheduler automatically schedules the other onion domains to go through the same process again and again. In this way, you don't have to give extra onion domains to the engine. And this is why Scrapy is really useful in crawling data from the dark web. So you can specify which domain to crawl and which domain to block. In this way, you can be safe from getting illegal data or getting illegal images. Moving on, now comes the human part. So we discussed about the tools, what tools can you use to hunt on the dark web. There is a human element also that's called human intelligence or HUMINT. So it's like it's a process of gathering intelligence through interpersonal contact, interpersonal contact rather by some kind of tools or technical process. That's why it's the most, it's most dangerous and difficult form because you are directly talking to the actor on the dark web, which is not safe. And it's not safe because you don't want your identity to be revealed to the actor or you don't want your organization's identity to be revealed to the actor. And it's important also because you can identify and respond to attacks much quickly. You can do post-attack investigation. So suppose your organization, like there's a data breach on your organization. If you want to confirm, so someone is selling this data on the dark web. And if you want to confirm whether they are selling the correct, whether they are lying about it or whether it is the truth. So you can activate your human intelligence or activate your the guy that is researching on the dark web to go and ask to the actors whether the data is correct or not. So that's post-attack investigation. You can also use it for new attacker vector, new attack vector discovery. So that's discovering new TTPs that the attackers are using or the attackers are discussing about. You can assume is as a high-tech equivalent of what an FBI agent does when he spends months or years working to infiltrate a criminal organization. That's why it's really hard to do it because you have to spend so much time on it. And that's why it's risky. And for this, you have to think like an actor, how they communicate within these communities, how they act within these communities. And the other thing is like it's the source from this is really valuable to your organization's safety. Now moving on, so we talked about tools, we talked about human intelligence, but now comes the pipeline or the architecture of how you can automate these threat hunting. So before that, I would suggest to set up a different system. You don't want your personal data to be on the system where you are doing threat hunting. So you can set up any lab or VM, whether physical or whether on cloud, just isolate the network and install relevant tools like KB, Privoxy, Tor. If you are using Elasticsearch or Kibana, then Elk and different Python libraries that would be necessary for your task. So this is the automated architecture that I have been using it. I will go it one by one. And I have this automate icon for the task that can be automated and for the task that I don't have, that's the only one I think is creepy setup and the design train NLP model. So it's hard to automate that part. So I will discuss it through. So first of all, you need to get the forums, forum links or the market links. So you can write a simple script to gather data from different search engines like I told you Recon and other search engines where you can get all the forum links. So that can be automated. So another thing is using SOCKS proxy. Like I told you, you have to use some kind of SOCKS proxy for this. So you can get SOCKS proxies IP. Again, you can write a simple script and get the SOCKS proxies that can be automated. Now comes the part of Scrappy setup. So Scrappy setup is is the so here you will write your login functions here. You will get your proxy setups and here you will like manage the settings of the Scrappy. Now, you can't automate this because when you get the onion links or different forum onion links, you have to go to the forums and sign in. Yes, you can automate the you can write scripts for logging in or signing in using different accounts. But I found it difficult to do this. That's why I just I have been using it manual. I've been doing it manually. So like creating different accounts like four to five accounts per forum and then noting down that into the Scrappy like the username, password and cookies into Scrappy. So for this, you have to do this. Also, you need different scrapers for different forums, because the architecture of forums is different for different forums. That's why you need to do this step manually, because you need to analyze, you need to first log into the forum, analyze it, and then write a different function for each forum for the HTML elements that you want to access. Coming to the crawler part. So what crawler is so the crawler parser analyzer, it's and the ELK part, it's all the part of the Scrappy that I discussed before. So these all are part of the Scrappy system. I've just written it differently. So you can understand what each part does. So what crawler does is again, pages from the forums, parser does it's parses the HTML pages, like getting the HTML elements like post, post content, author, etc. And the analyzer. So analyzer part is the part, you can write different function for this in Scrappy. So what analyzer does is so suppose you got the data from the from the dark web forum. Now you need to use some kind of techniques to evaluate the content that is relevant to your organization or relevant to your threat model. So we will discuss what threat modeling is in the later part of the presentation. So for now, just understand this, you can't focus on every other threat that's out there, you have to focus on threat that is relevant to your organization. So you need to do some kind of analysis to like get the data, get the relevant data from the dark web. So here comes the NLP model that I've been using. So you can design a trained NLP model in this way that it can just get you the content that is relevant to your organization. So you don't, suppose if you are a bank, you don't want to focus on tools or you don't want to focus on data breaches that's not relevant to your bank, you most likely would want to focus on the dump shops, that's where credit cards or debit cards are being dumped. So in this way, different organizations have different requirements, and you want to focus on those. Now, designing and training an NLP model can't be automated because you need some kind of content before content relevant to your threat model. And then you need to train your NLP model on that. It's like, so it's the same thing. It's somewhat like either you get the data first or you design your model first. So it's like egg and chicken problem. But nowadays, there are many NLP models like CDED LDA, where you can provide some kind of context before training the NLP model. So it's easy to do that. And then you store the data into ELK. So these all things can be automated. So coming to the part after getting this data, so what's the process after hunting? Now we'll discuss a little about Threat Intelligence Lifecycle. So what Threat Intelligence Lifecycle is, these are different steps that your organization takes to build a threat, like it starts from getting the data till presentation of the data. So how threat hunting on the dark web corresponds to this is, so there are like five phases, as you can see, direction, collection, processing, analysis, and dissemination. What we are doing is, we are doing direction phase from the human sources, like you can see from dark web social media forums. So in the direction phase, we identify dark web forums, we register on those forums, we acquire access on those forums. In the collection phase, you use Scrapy to establish access and collect raw data. Processing phase is also using Scrapy, so you parse raw HTML data, you extract topics and authors. And the analysis phase is where we use NLP and other machine learning models to infer relationship between these data. We get data that is relevant to our organization. We link data sources, we identify trends and hacks and leaks, etc. And dissemination phase is where we visualize the data in dashboards. If you are using Kibana or other kind of dashboards, we give out alerts and reports for our higher managers or the other people to see in our organization. So this is like crux of what threat hunting on the dark web maps to threat intelligence life cycle. Now, threat modeling. As I told you, I was going to talk about this in the coming presentation. So what threat modeling is, it's like getting your organization's critical assets and focusing on your organization's critical asset. So it's like understanding threats and how you can mitigate it when it happens to your organization particularly. So you understand what attacker wants, what different critical assets you have in your organization, what are different types of actors that can target you, whether they would be activists or insiders or some kind of criminal groups and know their capability. So here you choose your target on the dark web. So if you are a bank, you focus on credit card markets. If you are some other organization, if you want, you focus on inside the Fed markets or you focus on general markets. So in this way, you choose your target on the dark web. You prioritize risk as you can use pyramid of pain for that. So you prioritize risk and focus on IOCs that are relevant to your organization. Another thing is you don't just use one source to target, like there are many, many forums on the dark web. You don't just focus on one target, you focus on multiple targets. Apart from dark web, you focus on multiple clear websites also like pastebin or Twitter or nowadays on telegram also many these many actors are communicating. So you focus on the dark web also and the clear web also to get all the things you can for protecting your organization. So again, data collection processing, you collect data from the clear web and the dark web. So some of the sites are pastebin, Twitter, Reddit. On the dark web, it's forums, different forums and different marketplaces. You can do all this using this creepy crawler and parser that we discussed before. The analysis part in the threat intelligence model is you use, like I told you before, you use NLP machine learning or deep learning techniques. Some of them are like LDA, BERT, GPT to gather information related to your organization. You use social network analysis for analysis of different users on the dark web that post data related to your organization. There's clustering of products according to categories for the clustering thing. You classify different. So there's like binary classification, multi-class classification. So that's how you classify different products being sold on the dark web. So these all things come under analysis. I will touch a little on MITRE ATT&CK framework. So what MITRE ATT&CK is, it's a knowledge base of all the TTPs that was built using real world observations. So it contains different tactics, techniques, and procedures that attackers have used all these years. So you use ATT&CK matrix to map the intelligence you obtained to understand the TTPs better and to protect your organization more. So now coming to the operational security stuff. So hunting on the dark web or if you are doing human intelligence stuff on the dark web, you need to follow some set of processes so that you don't reveal your data or reveal your identity or your organizational identity as I talked before. So what is OPSEC? So OPSEC is the practice of hiding yourself online so that you don't reveal your real self or you don't reveal or compromise your own operations. It's derived from the US military, that's operational security. You need to hide your PIA, that's personal identifiable information. So you need to work on the dark web in such a way that you don't disclose your full name or driver's license or bank account or even a simple thing as email. This is what you need to protect and that's why security is really important. And it's also a hard thing to do because at the end of the day, we are all humans and we like to be seen as knowledgeable and we like to impress others. This all leads to gossip, gossiping, bragging and oversharing with others. That's why operational security is really hard. And most of the time people think of it as a process. So people think that, okay, I have to do human intelligence stuff. Now I have to follow operational security. It should not be like that. It should not be seen as a burden to perform or as another of your job tasks to perform. It should be a mindset like you should always think about operational security before doing human intelligence or before engaging with actors. So I will discuss some of the things that you can use to maintain opsec in your daily lifestyle. There are many other things. So the main thing you want is hiding your identity. So the first thing you can do or you should do is use a separate system like I talked about before. Also use a separate system where you don't store any personal information. Whether it be a lab or VM or some kind of system. The main thing is to use Tor with a proxy or Tor over VPN. The main thing is maintaining different personas on the dark web. It's like I told, it's an equivalent of an FBI agent going undercover. So he has some kind of persona. He has a backstory. You have to do that. You have to do the same thing. You should have different personas for different identities that you have on different forums. You should never mix it up. That's why you take extensive notes so that you don't mess up the personas. You should always watch what you say and you should always think before posting. Human intelligence is not a 95 job thing. You can't just talk to or you can't just communicate to actors during your job time because they will know that you are doing this as your job. In this way, you can be exposed. They can easily guess that you are a researcher and not a threat actor. It would be a tip off for them. That's why you have to do this work 24 by 7. It's not like you have to do this work on the weekends. You have to do this work after your work hours. That's why it's not a 95 job thing. You have to develop appropriate language skills because people don't talk formally like actors don't talk formally. You have to develop appropriate language skills or slang skills. Also, there are many forums like there are different Russian forums or German forums. You might need to develop that language skills like learning Russian, learning German. Another thing to note is changing time zones. If you are in US and you are accessing or you are engaging in a community on a Russian forum, you might want to change the time zone to Russia because again, it would be a tip off to the actors that you are a security researcher or that you are not a real actor. These are some of the things that should be noted before doing any intelligence stuff on the dark web. Now, that was it. So concluding all this, we discussed a little about the dark web, what dark web is, how to access dark web. We discussed about dark web forums and marketplaces, what different products are being sold there, what is the cost model around that. We discussed about threat hunting on the dark web, how you can hunt on the dark web. We discussed different tools and the main tool was scraping. That's the main framework that we are working on to hunt on the dark web. We discussed about human intelligence, how it can be used or how it should be used to support your tool-based data collection. We discussed about the pipeline or the architecture that can be used to automate the dark web hunting. We talked a little about a threat intelligence life cycle, how threat hunting on the dark web maps to the life cycle steps. And we talked about operational security and why it is so important and why it is so hard to do operational security. Again, some more points to notice, it's obviously threat hunting on the dark web is hard, but it's worth the effort. You don't get intelligence that you get from the dark web. You should always keep operational security in mind. Like I said before, you should look on more than one resource. You should look on forums. You could look on clear web forums also like pastebin and telegram as an example. And you should look on different other forums on the dark web also. It takes a lot of resources and a team effort. You can't do all these things alone. You should have a team for this. And we talked a little about usage of MITRE attack framework and how it can be useful to map your TTPs to that. These are some of the resources that I suggest you to read if you want to know more about the dark web stuff or dark web hunting stuff. The majority, the major ones are like a recorded feature insight, CrowdStrike, digital shadows. They release their blogs or white papers regularly. So read them and you'll understand what all things are there on the dark web. So yeah, that was it. I think, thank you so much. I hope you all liked my presentation and you can contact me on Twitter or LinkedIn if you have any doubt or if you want to discuss about this stuff more. I will hang on in discord to answer all your questions. So yeah, thank you.