WEBVTT 00:03.850 --> 00:06.870 Hello everyone, my name is Apoorva Singh Gautam. 00:06.870 --> 00:12.130 I would like to thank Defcon Retain Village for having me here. 00:12.470 --> 00:17.810 I will talk about automating threat hunting on the dark web and the things that surrounds it. 00:17.830 --> 00:26.170 I presented this talk, I presented a short version of this talk at Grimcon and this is the extended version of it. 00:26.170 --> 00:28.450 So let's get started. 00:28.450 --> 00:33.250 I will switch off my webcam so that you guys can focus on the presentation here. 00:36.180 --> 00:38.480 Okay, so a little about me. 00:38.480 --> 00:40.380 My name is Apoorva Singh Gautam. 00:40.380 --> 00:42.680 I go by the handle ASGScorpion. 00:42.680 --> 00:44.380 I am a security researcher. 00:44.380 --> 00:51.280 I started into threat intel or hunting two years back and I've been loving it since then. 00:51.280 --> 00:56.000 I'm currently pursuing my master's in cyber security at Georgia Tech. 00:56.060 --> 01:04.660 Recently, during the summer, I was a research intern at UC Berkeley doing research in threat intelligence. 01:04.660 --> 01:08.160 Some of my hobbies are gaming. 01:08.160 --> 01:11.260 I pretty much love Rainbow Six Siege. 01:11.260 --> 01:14.000 I sometimes stream it also. 01:14.000 --> 01:15.740 I love hiking. 01:15.740 --> 01:20.700 I recently started into lockpicking and I've been enjoying it. 01:20.700 --> 01:23.300 I contribute to the security community. 01:23.300 --> 01:27.060 It's my passion contributing to the security community. 01:27.060 --> 01:30.040 I'm a senior teaching assistant at Siberia. 01:30.040 --> 01:34.600 I contribute at Station X also and at the local security meetups. 01:35.040 --> 01:40.380 And there are my socials if you want to contact me or hit me up. 01:42.100 --> 01:44.280 So what's today's agenda? 01:44.280 --> 01:57.420 So we will talk about introduction to dark web, what dark web is, how to access dark web, what Tor is, what is the difference between dark web, deep web, why you should perform hunting on the dark web. 01:57.420 --> 02:03.140 Before that, we will discuss what is threat hunting and why it is crucial to hunt on the dark web. 02:03.140 --> 02:06.780 We will discuss different methods to hunt on the dark web. 02:07.080 --> 02:10.520 Can the dark web hunting be automated? 02:10.600 --> 02:15.840 What's the pipeline or the architecture of hunting, automating the dark web hunting? 02:16.420 --> 02:20.520 Then we will discuss a little about threat intelligence lifecycle. 02:20.660 --> 02:28.120 That's how threat hunting on the dark web is analogous to threat intelligence lifecycle. 02:28.420 --> 02:31.400 What steps are there that corresponds to this? 02:31.600 --> 02:39.760 We will discuss a little about operational security, that's OPSEC, and why is it important to secure yourself when hunting on the dark web. 02:40.040 --> 02:41.600 And yeah, that's it. 02:43.320 --> 02:45.040 So introduction to dark web. 02:45.560 --> 02:53.640 I'm sure you must have seen this image a lot of times on the internet that shows difference between surface web, deep web, and dark web. 02:53.740 --> 03:01.100 So the surface web is the sites that are indexed by different search engines, that's Google, Bing, Yahoo, etc. 03:01.620 --> 03:05.680 And the majority of the portion of the internet is deep web. 03:05.780 --> 03:11.400 Now deep web is any site, any website that is not indexed by the search engine. 03:11.400 --> 03:22.180 So this can include your databases, your server instances, or any other different websites that you cannot search from the search engines. 03:22.860 --> 03:27.220 The third one is the dark web that we will talk mostly about today. 03:27.340 --> 03:35.080 And this is the part of the internet where you need some kind of software, special software to access the dark web. 03:35.400 --> 03:49.000 So this can include anything related to like drugs or weapons that's being sold on the dark web, or some kind of research, or books that's sold there. 03:49.320 --> 03:55.840 So dark web includes several forums and marketplaces where people sell different kind of things. 03:55.840 --> 03:58.280 So these type of things are sold there. 04:00.140 --> 04:06.660 So majority portion of the internet is deep web, and people confuse it between dark web and deep web. 04:06.660 --> 04:13.100 So as you can see, only 6% of the internet is dark web, and 80 to 90% is the deep web. 04:16.090 --> 04:19.250 Moving on, so how do you access the dark web? 04:19.250 --> 04:31.430 So there are several companies, several organizations that offer their own dark web systems, or you can call it, so the famous one is Tor, that's the onion router. 04:32.890 --> 04:37.090 The second is I2P, that's an invisible internet project. 04:37.090 --> 04:40.490 Zeronet is also becoming popular nowadays. 04:41.130 --> 04:51.090 So Tor has .onion domain, like the domain name ends with .onion, and I2P is .i2p. 04:51.530 --> 04:58.610 Talking more about Tor, what Tor is and how it works, is it's like a three-layer proxy system. 04:58.610 --> 05:15.130 If you can see on the image here, there are entry nodes, the entry node is where your traffic goes to, and then it goes through the middle layer, and the exit layer, and then goes to the destination. 05:15.610 --> 05:26.370 This way your identity is hidden, and also only the entry node is publicly listed, rest of the nodes are not publicly listed. 05:26.830 --> 05:37.130 So it hides your identity, it hides your IP address, and these nodes are volunteer-based systems. 05:37.130 --> 05:48.530 So Tor has about like 6,000 relays, and I don't know the number about I2P, but it is also becoming more popular nowadays. 05:49.130 --> 06:02.410 And the major thing about Tor is each node only knows the IP address of the next node or the previous node. 06:02.410 --> 06:08.310 So if we talk about here, the entry node doesn't know about the exit node or vice versa. 06:08.310 --> 06:11.810 So in this way, the location of the nodes are also protected. 06:14.830 --> 06:19.770 So there are many misconceptions about Tor, the dark web. 06:19.810 --> 06:28.690 The famous one is whenever we talk about Tor, people think about criminal or criminal things that goes on Tor. 06:28.930 --> 06:35.910 Yes, there is criminal side of Tor, but there is a good side of the Tor also, or the dark web. 06:35.910 --> 06:41.330 So it's really famous among whistleblowers or activists. 06:42.070 --> 06:47.370 There are many countries where free speech is limited. 06:47.670 --> 06:56.590 So they can use Tor, like people from those countries use Tor to express their speech or express what they think. 06:57.170 --> 07:07.110 Tor also has access to many old literature or researches, which is not available on the open web. 07:07.810 --> 07:11.070 And it's like safe haven for journalists, obviously. 07:11.790 --> 07:19.970 So there are many popular sites like Facebook and New York Times that has their counterpart on Tor, like their .NER website counterpart. 07:20.050 --> 07:25.930 So it's useful for whistleblowers or activists. 07:26.290 --> 07:33.050 The second thing is Tor is so many people think that it's illegal to access the dark web. 07:33.350 --> 07:36.170 So it's not illegal to access the dark web. 07:36.170 --> 07:48.070 It's illegal to indulge in these kinds of activities like purchasing drugs or purchasing any other illegal things on the dark web that is being sold there. 07:48.070 --> 07:50.790 But it's completely legal to access the dark web. 07:52.710 --> 08:01.810 And it's like the last thing which I would talk about is many people think Tor is like really big. 08:01.810 --> 08:12.910 But if you talk about the uptime or the availability of sites, there are very few onion domains on the dark web if you compare it with the clear web. 08:12.930 --> 08:18.670 So it's like the dark web is like very little part of the internet. 08:22.040 --> 08:25.540 So we will talk about the dark side of the dark web. 08:25.540 --> 08:28.300 I mean, the criminal side of the dark web. 08:28.300 --> 08:32.160 So there are many forums or marketplaces on the dark web. 08:32.640 --> 08:43.420 These are some of the relevant sites that's relevant to security researchers or people who want to access dark web for their organization's benefit. 08:43.540 --> 08:47.240 So this is like some of them are like a credit card market. 08:47.240 --> 08:51.620 So where different credit cards are being dumped or remote access. 08:51.620 --> 08:59.380 So these forums include remote access trojans or some kind of remote access tools or insider threats. 08:59.380 --> 09:12.560 So insider threats are it's like recent coming up a forum where insider like the people who are selling their company's secrets, they talk amongst themselves. 09:12.560 --> 09:14.560 So these are some of the relevant sites. 09:15.660 --> 09:17.680 Now coming to the cost. 09:17.680 --> 09:24.520 So how much it costs for some kind of for something to buy from the dark web. 09:24.520 --> 09:29.040 So as you can see, it's really easy to buy these things on the dark web and it. 09:29.100 --> 09:40.320 So like SSN, you can buy any SSN for $1 or fake FB friends, a fake FB with 15 friends, mobile malware, bank details, so exploit to zero days. 09:40.320 --> 09:46.480 So it's like these type of things are easy to get on the dark web. 09:46.480 --> 09:50.760 That's why security researchers and other people are focusing more on the dark web. 09:50.760 --> 10:00.260 You might have heard about recent news, like 500,000 Zoom accounts sold on the dark web or 267 million FB user profiles sold on the dark web. 10:00.260 --> 10:06.900 So there are many data breaches occurring day by day and they are being sold on the dark web. 10:06.900 --> 10:11.160 That's why researching on the dark web is really important. 10:13.200 --> 10:17.760 These are some of the product listings from the forums of the dark web. 10:17.760 --> 10:20.540 This is how the products are being listed. 10:23.420 --> 10:34.860 And you can see these are the average cost of accounts for different online services like bank services, what's the average cost of it or what's the average cost of service for video games. 10:35.940 --> 10:40.180 This is the average cost of tools that is being sold on the dark web. 10:40.180 --> 10:43.840 As you can see again, bank and financial average cost is $74. 10:43.840 --> 10:49.180 So you can buy a brute forcing tool for some bank at $74. 10:51.520 --> 10:54.260 Now coming to why you should hunt on the dark web. 10:54.260 --> 10:57.780 Before that, let's talk about what is threat hunting. 10:57.940 --> 11:02.780 Threat hunting is proactive searching for cyber threats. 11:02.780 --> 11:06.400 Proactive means before the attack happens. 11:06.400 --> 11:08.000 That's proactive search. 11:08.140 --> 11:12.580 You search for cyber threats from logs or indicators of compromise. 11:12.580 --> 11:15.540 That's IP addresses, emails, domains, etc. 11:15.540 --> 11:16.600 or textual data. 11:16.600 --> 11:20.820 That's what we are doing when we are searching the dark web. 11:21.680 --> 11:27.460 So it's basically hypothesis based because there's nothing concrete about the process. 11:27.460 --> 11:32.180 You take one use case and work on it and then you take another use case and work on it. 11:32.180 --> 11:35.160 And it goes iteratively in the same way. 11:35.440 --> 11:54.740 Many times there's use of machine learning or natural language processing that's NLP and advanced analytics process in this because you need to scan through the textual data if you are hunting on the dark web or if you are hunting on the clear web also but for textual data. 11:54.780 --> 11:58.520 So machine learning and advanced analytics are useful there. 12:01.260 --> 12:02.940 So why it's serious? 12:02.940 --> 12:06.860 Why threat hunting on the dark web is really important? 12:06.860 --> 12:09.800 What's so big about that? 12:09.800 --> 12:15.800 As I told you about again, there are many forums, marketplaces, dump shops. 12:16.880 --> 12:19.880 So what criminals or actors do? 12:19.880 --> 12:22.340 They learn new methods and techniques on the dark web. 12:22.340 --> 12:24.300 They monetize their skills. 12:24.300 --> 12:30.140 They trade their exploits or tools or even drugs and weapons and communication. 12:30.140 --> 12:36.480 So they communicate with each other and they share their ideas for new attacks. 12:36.480 --> 12:48.800 A security researcher or a person who is researching on the dark web, he can find a lot on, he can learn a lot while engaging within these communities. 12:49.200 --> 13:02.060 You can learn their techniques or TTPs, that's the attack techniques and procedures, how they think about the attack, how they plan an attack and it can identify. 13:02.060 --> 13:12.840 So if you do it correctly, it can identify attacks in the earlier stages, that's planning and recon stages and you can reduce the impacts that it causes. 13:12.840 --> 13:22.560 So suppose if your organization data is being sold on the dark web, there are different kinds of impacts that can cause to your organization. 13:22.560 --> 13:31.340 So some of the direct impacts are like personal information stolen or healthcare records stolen or even your company's trade secrets. 13:31.360 --> 13:43.060 And some of the indirect impacts are repetition of your organization, revenue loss, and nowadays the legal penalties that your data is lost and you have to cover the cost of the customers. 13:44.780 --> 13:55.480 So this is like, that's why this is really important to like to research on the dark web, to hunt threats on the dark web. 13:57.560 --> 14:01.380 On the same lines, these are the benefits of the threat hunting. 14:01.380 --> 14:15.460 So if you do it correctly, you can keep up with the latest trends of the attacks, you can get new TTPs, that's tactics, techniques and procedures, you can identify insider threats, you can discover data breaches. 14:15.460 --> 14:27.640 The main thing is you can prepare your SOCs and incident responders to deal with the attack because they will know before only what are the TTPs attackers are using. 14:27.640 --> 14:32.600 So they can reduce the damage and risk to the organization by acting quickly on that. 14:35.800 --> 14:38.800 So coming to the methods to hunt on the dark web. 14:38.800 --> 14:42.740 So we will discuss about some tools that's used to hunt on the dark web. 14:42.740 --> 14:47.120 And then we will discuss about the human element that can be used to hunt on the dark web. 14:47.140 --> 14:53.880 So talking about the first tool that's really, really important for this is Scrapy. 14:53.880 --> 14:56.920 So it's a web crawling framework. 14:57.320 --> 15:03.940 It's so famous, it's so important because it manages multithreading automatically. 15:03.940 --> 15:16.500 So you don't have to write, you don't have to spend too much time on the multithreading part, because it has already capabilities for multithreading using one or two lines of one or two lines of parameters. 15:16.980 --> 15:18.900 The second thing is Tor. 15:18.900 --> 15:21.840 Obviously, if you want to access the dark web, you need Tor. 15:22.200 --> 15:28.140 Onion scan is another tool that is used to search for onion websites. 15:28.340 --> 15:35.600 It can tell you if a website is up or not and the correlation between different websites on the dark web. 15:36.380 --> 15:37.760 Coming to Pryvoxy. 15:37.760 --> 15:41.400 So Pryvoxy is a web proxy. 15:41.400 --> 15:44.180 Before getting more into this. 15:44.180 --> 15:57.240 So when you access the dark web, you need some kind of proxy to access the dark web because your ISP, as I told you before, the entry nodes are publicly listed. 15:57.240 --> 16:01.540 So your ISP can have a blacklist to block the entry nodes. 16:01.540 --> 16:11.320 So you can't access the dark web or even if the ISP doesn't block it , he can see whether you are accessing the dark web or not. 16:11.320 --> 16:16.700 He cannot see what you are doing on there, but he can see whether you are accessing the dark web or not. 16:16.700 --> 16:18.740 So you might not want that. 16:18.740 --> 16:23.940 That's why you need some kind of proxy and a majority of people use SOCKS proxy. 16:23.940 --> 16:32.960 So basic difference between HTTP and SOCKS proxy is SOCKS proxy is a lower level proxy and it works on the SOCKS protocol. 16:33.280 --> 16:42.540 HTTP proxy only works on HTTP or HTTPS websites, but SOCKS proxy can work on other protocols also. 16:43.420 --> 16:50.240 And there are different tools to use SOCKS proxy like TSOCKS, Polypo and Pryvoxy. 16:50.240 --> 16:57.080 I've been using Pryvoxy and it has been so I don't have any problem with Pryvoxy. 16:57.080 --> 16:58.960 So and it's good. 16:59.120 --> 17:09.520 So another thing is SCP doesn't allow you to use directly SOCKS proxy because it doesn't support SOCKS proxy. 17:09.520 --> 17:19.660 So that's why you have to use these tools like Pryvoxy, TSOCKS or Polypo to route your SOCKS through Pryvoxy scripts. 17:20.380 --> 17:28.860 And there are other tools also like their search engines like Kilos or Recon where you can find different end-to-end domains. 17:29.460 --> 17:41.020 Apart from using SOCKS proxy, you can also use VPN with Tor for extra layer of protection and like encrypting your data. 17:43.730 --> 17:55.050 So getting more into SCP part, this image might seem a little confusing, but I will get into it like step by step. 17:55.050 --> 18:03.250 So this is like why SCP is really important and why SCP is so useful in hunting on the dark web. 18:03.250 --> 18:11.230 So if you can see there, so for explaining this, I will explain it in terms of Python code. 18:11.230 --> 18:16.210 So suppose everything you see here is a different Python program. 18:16.210 --> 18:21.430 So spider is a Python program, downloader is a Python program, middleware is a Python program and so on. 18:22.210 --> 18:26.530 So what spider does in spider? 18:26.530 --> 18:35.670 So in spider Python program, you give your onion domain on which you want to crawl the data or which you want to get the data. 18:35.670 --> 18:38.470 So it gives it to the engine. 18:39.050 --> 18:44.770 Suppose engine is just the program that manages every other Python programs. 18:44.930 --> 18:49.870 So it gives the onion domain to the engine. 18:49.870 --> 18:51.550 The engine gives it to the scheduler. 18:51.550 --> 18:56.610 So what scheduler does is here the multi-threading concept comes into the picture. 18:56.750 --> 19:02.050 Scheduler gets different domains and schedules it accordingly into multiple threads. 19:02.090 --> 19:10.030 So the onion domain goes into the scheduler, the scheduler gives it back to the engine and engine gives it to the middleware. 19:10.030 --> 19:15.530 So middleware program includes your proxy program and or login program. 19:15.530 --> 19:19.050 So what proxy program is that in proxy function. 19:19.050 --> 19:25.410 So if we are talking about middleware, that's a Python program, there's a proxy function into it. 19:25.410 --> 19:39.210 So proxy function is where you will note, you will put down your privoxy IP or Tor IP, so that the request response goes and comes through that proxy so that you can access the dark web. 19:39.530 --> 20:01.690 The login program, the login function is where you will put your user agents or cookies or so for accessing the dark web forums, you have to, so there nowadays, for all the forums, you have to have an account to access the dark web or to access that particular forum. 20:01.690 --> 20:11.410 So to access the forum, you need some kind of cookies or also there are many forums, many high level forums that implement captures. 20:11.450 --> 20:19.210 And as Google doesn't work on the dark web, these captures are like image-based capture or text-based capture that is easy to bypass. 20:19.290 --> 20:30.950 So you can use any machine learning capture bypassing service or any capture bypassing websites like death by capture or anti-capture to bypass the capture. 20:30.950 --> 20:38.030 So these are all codes you will write in this login function in middleware program. 20:38.030 --> 20:43.310 Now your request or your traffic goes through this middleware program to the downloader. 20:43.310 --> 20:50.030 What downloader does is it's a simple program to extract the HTML and save and give it back to the engine. 20:50.030 --> 20:53.850 So downloader extract the HTML and gives it back to the engine. 20:54.110 --> 20:56.970 Now engine gives it back to the spider. 20:56.970 --> 21:16.370 Now there's another function in spider that extracts the HTML entities that you want from the forums HTML, suppose forum name or document ID or the text-based data or author name who posted a particular content. 21:16.370 --> 21:21.070 So you get that and it is called items in Scrapy. 21:21.070 --> 21:24.930 So you get these items and it sends it to the item pipeline. 21:24.930 --> 21:29.170 Item pipeline is where your database is configured. 21:29.170 --> 21:37.850 So I use Elasticsearch, you can use any database whether SQL or NoSQL and it directly saves the items to the SQL. 21:37.910 --> 21:49.890 And so the important thing to note here is that Scrapy, so as I told you before multi-threading is automatically handled by Scrapy. 21:49.890 --> 21:55.770 The another thing is you don't have to give multiple onion domains to spider. 21:55.810 --> 22:11.950 So when the downloader gets the data, when the downloader gets the HTML page from the particular forum, you don't have to configure the code, there is a code to get all the onion domains on that particular HTML page. 22:12.050 --> 22:18.590 So the scheduler automatically schedules the other onion domains to go through the same process again and again. 22:19.170 --> 22:24.550 In this way, you don't have to give extra onion domains to the engine. 22:25.110 --> 22:32.090 And this is why Scrapy is really useful in crawling data from the dark web. 22:33.250 --> 22:38.030 So you can specify which domain to crawl and which domain to block. 22:38.110 --> 22:43.950 In this way, you can be safe from getting illegal data or getting illegal images. 22:45.150 --> 22:49.150 Moving on, now comes the human part. 22:50.070 --> 22:54.790 So we discussed about the tools, what tools can you use to hunt on the dark web. 22:55.090 --> 23:00.430 There is a human element also that's called human intelligence or HUMINT. 23:00.610 --> 23:11.030 So it's like it's a process of gathering intelligence through interpersonal contact, interpersonal contact rather by some kind of tools or technical process. 23:11.210 --> 23:21.570 That's why it's the most, it's most dangerous and difficult form because you are directly talking to the actor on the dark web, which is not safe. 23:22.070 --> 23:35.130 And it's not safe because you don't want your identity to be revealed to the actor or you don't want your organization's identity to be revealed to the actor. 23:35.130 --> 23:43.610 And it's important also because you can identify and respond to attacks much quickly. 23:43.610 --> 23:46.270 You can do post-attack investigation. 23:46.270 --> 23:52.410 So suppose your organization, like there's a data breach on your organization. 23:52.830 --> 23:57.010 If you want to confirm, so someone is selling this data on the dark web. 23:57.010 --> 24:04.770 And if you want to confirm whether they are selling the correct, whether they are lying about it or whether it is the truth. 24:04.770 --> 24:17.470 So you can activate your human intelligence or activate your the guy that is researching on the dark web to go and ask to the actors whether the data is correct or not. 24:17.470 --> 24:19.530 So that's post-attack investigation. 24:19.590 --> 24:24.470 You can also use it for new attacker vector, new attack vector discovery. 24:24.470 --> 24:31.230 So that's discovering new TTPs that the attackers are using or the attackers are discussing about. 24:31.750 --> 24:43.870 You can assume is as a high-tech equivalent of what an FBI agent does when he spends months or years working to infiltrate a criminal organization. 24:43.870 --> 24:50.810 That's why it's really hard to do it because you have to spend so much time on it. 24:50.850 --> 24:53.330 And that's why it's risky. 24:53.490 --> 25:03.470 And for this, you have to think like an actor, how they communicate within these communities, how they act within these communities. 25:03.890 --> 25:12.050 And the other thing is like it's the source from this is really valuable to your organization's safety. 25:14.530 --> 25:26.350 Now moving on, so we talked about tools, we talked about human intelligence, but now comes the pipeline or the architecture of how you can automate these threat hunting. 25:26.350 --> 25:34.270 So before that, I would suggest to set up a different system. 25:34.370 --> 25:40.030 You don't want your personal data to be on the system where you are doing threat hunting. 25:40.030 --> 25:49.730 So you can set up any lab or VM, whether physical or whether on cloud, just isolate the network and install relevant tools like KB, Privoxy, Tor. 25:49.970 --> 25:59.330 If you are using Elasticsearch or Kibana, then Elk and different Python libraries that would be necessary for your task. 26:00.250 --> 26:06.450 So this is the automated architecture that I have been using it. 26:06.650 --> 26:09.590 I will go it one by one. 26:09.590 --> 26:23.590 And I have this automate icon for the task that can be automated and for the task that I don't have, that's the only one I think is creepy setup and the design train NLP model. 26:23.590 --> 26:25.830 So it's hard to automate that part. 26:25.830 --> 26:27.630 So I will discuss it through. 26:27.850 --> 26:34.810 So first of all, you need to get the forums, forum links or the market links. 26:34.810 --> 26:47.370 So you can write a simple script to gather data from different search engines like I told you Recon and other search engines where you can get all the forum links. 26:47.730 --> 26:49.550 So that can be automated. 26:49.830 --> 26:52.510 So another thing is using SOCKS proxy. 26:52.510 --> 26:55.810 Like I told you, you have to use some kind of SOCKS proxy for this. 26:55.810 --> 26:58.210 So you can get SOCKS proxies IP. 26:58.350 --> 27:04.030 Again, you can write a simple script and get the SOCKS proxies that can be automated. 27:04.790 --> 27:08.190 Now comes the part of Scrappy setup. 27:08.970 --> 27:16.830 So Scrappy setup is is the so here you will write your login functions here. 27:16.830 --> 27:25.330 You will get your proxy setups and here you will like manage the settings of the Scrappy. 27:25.750 --> 27:35.030 Now, you can't automate this because when you get the onion links or different forum onion links, you have to go to the forums and sign in. 27:35.050 --> 27:43.450 Yes, you can automate the you can write scripts for logging in or signing in using different accounts. 27:43.450 --> 27:47.210 But I found it difficult to do this. 27:47.210 --> 27:50.730 That's why I just I have been using it manual. 27:50.730 --> 27:52.130 I've been doing it manually. 27:52.130 --> 28:04.730 So like creating different accounts like four to five accounts per forum and then noting down that into the Scrappy like the username, password and cookies into Scrappy. 28:04.990 --> 28:07.150 So for this, you have to do this. 28:07.150 --> 28:16.150 Also, you need different scrapers for different forums, because the architecture of forums is different for different forums. 28:16.330 --> 28:34.150 That's why you need to do this step manually, because you need to analyze, you need to first log into the forum, analyze it, and then write a different function for each forum for the HTML elements that you want to access. 28:34.830 --> 28:36.650 Coming to the crawler part. 28:36.650 --> 28:46.530 So what crawler is so the crawler parser analyzer, it's and the ELK part, it's all the part of the Scrappy that I discussed before. 28:46.530 --> 28:49.950 So these all are part of the Scrappy system. 28:49.990 --> 28:51.770 I've just written it differently. 28:51.770 --> 28:54.250 So you can understand what each part does. 28:54.250 --> 29:06.590 So what crawler does is again, pages from the forums, parser does it's parses the HTML pages, like getting the HTML elements like post, post content, author, etc. 29:06.590 --> 29:07.890 And the analyzer. 29:07.890 --> 29:15.670 So analyzer part is the part, you can write different function for this in Scrappy. 29:15.670 --> 29:21.530 So what analyzer does is so suppose you got the data from the from the dark web forum. 29:21.530 --> 29:30.770 Now you need to use some kind of techniques to evaluate the content that is relevant to your organization or relevant to your threat model. 29:30.770 --> 29:35.930 So we will discuss what threat modeling is in the later part of the presentation. 29:36.050 --> 29:46.530 So for now, just understand this, you can't focus on every other threat that's out there, you have to focus on threat that is relevant to your organization. 29:46.570 --> 29:53.790 So you need to do some kind of analysis to like get the data, get the relevant data from the dark web. 29:53.790 --> 29:59.230 So here comes the NLP model that I've been using. 29:59.710 --> 30:09.770 So you can design a trained NLP model in this way that it can just get you the content that is relevant to your organization. 30:10.630 --> 30:28.990 So you don't, suppose if you are a bank, you don't want to focus on tools or you don't want to focus on data breaches that's not relevant to your bank, you most likely would want to focus on the dump shops, that's where credit cards or debit cards are being dumped. 30:29.270 --> 30:35.150 So in this way, different organizations have different requirements, and you want to focus on those. 30:35.150 --> 30:44.770 Now, designing and training an NLP model can't be automated because you need some kind of content before content relevant to your threat model. 30:44.830 --> 30:48.610 And then you need to train your NLP model on that. 30:49.550 --> 30:51.830 It's like, so it's the same thing. 30:51.830 --> 30:57.270 It's somewhat like either you get the data first or you design your model first. 30:57.270 --> 30:58.910 So it's like egg and chicken problem. 30:58.910 --> 31:12.410 But nowadays, there are many NLP models like CDED LDA, where you can provide some kind of context before training the NLP model. 31:12.910 --> 31:15.510 So it's easy to do that. 31:15.510 --> 31:18.070 And then you store the data into ELK. 31:18.070 --> 31:21.110 So these all things can be automated. 31:24.830 --> 31:32.670 So coming to the part after getting this data, so what's the process after hunting? 31:32.670 --> 31:36.450 Now we'll discuss a little about Threat Intelligence Lifecycle. 31:36.730 --> 31:52.110 So what Threat Intelligence Lifecycle is, these are different steps that your organization takes to build a threat, like it starts from getting the data till presentation of the data. 31:52.850 --> 32:05.970 So how threat hunting on the dark web corresponds to this is, so there are like five phases, as you can see, direction, collection, processing, analysis, and dissemination. 32:06.690 --> 32:14.010 What we are doing is, we are doing direction phase from the human sources, like you can see from dark web social media forums. 32:14.210 --> 32:23.770 So in the direction phase, we identify dark web forums, we register on those forums, we acquire access on those forums. 32:23.950 --> 32:28.610 In the collection phase, you use Scrapy to establish access and collect raw data. 32:29.190 --> 32:37.530 Processing phase is also using Scrapy, so you parse raw HTML data, you extract topics and authors. 32:37.590 --> 32:45.670 And the analysis phase is where we use NLP and other machine learning models to infer relationship between these data. 32:46.030 --> 32:49.510 We get data that is relevant to our organization. 32:49.610 --> 32:53.730 We link data sources, we identify trends and hacks and leaks, etc. 32:53.730 --> 32:57.410 And dissemination phase is where we visualize the data in dashboards. 32:57.410 --> 33:11.170 If you are using Kibana or other kind of dashboards, we give out alerts and reports for our higher managers or the other people to see in our organization. 33:11.430 --> 33:20.290 So this is like crux of what threat hunting on the dark web maps to threat intelligence life cycle. 33:21.730 --> 33:23.030 Now, threat modeling. 33:23.030 --> 33:27.610 As I told you, I was going to talk about this in the coming presentation. 33:27.610 --> 33:42.610 So what threat modeling is, it's like getting your organization's critical assets and focusing on your organization's critical asset. 33:42.610 --> 33:52.090 So it's like understanding threats and how you can mitigate it when it happens to your organization particularly. 33:52.090 --> 34:11.550 So you understand what attacker wants, what different critical assets you have in your organization, what are different types of actors that can target you, whether they would be activists or insiders or some kind of criminal groups and know their capability. 34:14.110 --> 34:17.210 So here you choose your target on the dark web. 34:20.090 --> 34:23.210 So if you are a bank, you focus on credit card markets. 34:23.210 --> 34:29.810 If you are some other organization, if you want, you focus on inside the Fed markets or you focus on general markets. 34:30.430 --> 34:33.750 So in this way, you choose your target on the dark web. 34:33.930 --> 34:38.450 You prioritize risk as you can use pyramid of pain for that. 34:38.450 --> 34:45.670 So you prioritize risk and focus on IOCs that are relevant to your organization. 34:46.810 --> 34:54.970 Another thing is you don't just use one source to target, like there are many, many forums on the dark web. 34:54.970 --> 34:59.630 You don't just focus on one target, you focus on multiple targets. 34:59.630 --> 35:13.350 Apart from dark web, you focus on multiple clear websites also like pastebin or Twitter or nowadays on telegram also many these many actors are communicating. 35:13.350 --> 35:22.790 So you focus on the dark web also and the clear web also to get all the things you can for protecting your organization. 35:23.990 --> 35:30.230 So again, data collection processing, you collect data from the clear web and the dark web. 35:30.450 --> 35:33.890 So some of the sites are pastebin, Twitter, Reddit. 35:34.330 --> 35:38.410 On the dark web, it's forums, different forums and different marketplaces. 35:38.490 --> 35:43.890 You can do all this using this creepy crawler and parser that we discussed before. 35:45.110 --> 35:55.350 The analysis part in the threat intelligence model is you use, like I told you before, you use NLP machine learning or deep learning techniques. 35:56.090 --> 36:03.810 Some of them are like LDA, BERT, GPT to gather information related to your organization. 36:04.010 --> 36:15.830 You use social network analysis for analysis of different users on the dark web that post data related to your organization. 36:16.270 --> 36:22.370 There's clustering of products according to categories for the clustering thing. 36:22.370 --> 36:24.110 You classify different. 36:24.230 --> 36:28.210 So there's like binary classification, multi-class classification. 36:28.210 --> 36:32.930 So that's how you classify different products being sold on the dark web. 36:32.970 --> 36:36.050 So these all things come under analysis. 36:36.070 --> 36:39.770 I will touch a little on MITRE ATT&CK framework. 36:39.770 --> 36:52.550 So what MITRE ATT&CK is, it's a knowledge base of all the TTPs that was built using real world observations. 36:52.690 --> 36:59.490 So it contains different tactics, techniques, and procedures that attackers have used all these years. 36:59.850 --> 37:10.290 So you use ATT&CK matrix to map the intelligence you obtained to understand the TTPs better and to protect your organization more. 37:13.500 --> 37:18.520 So now coming to the operational security stuff. 37:18.520 --> 37:38.220 So hunting on the dark web or if you are doing human intelligence stuff on the dark web, you need to follow some set of processes so that you don't reveal your data or reveal your identity or your organizational identity as I talked before. 37:39.100 --> 37:41.060 So what is OPSEC? 37:41.060 --> 37:54.140 So OPSEC is the practice of hiding yourself online so that you don't reveal your real self or you don't reveal or compromise your own operations. 37:54.140 --> 37:57.740 It's derived from the US military, that's operational security. 37:58.440 --> 38:08.520 You need to hide your PIA, that's personal identifiable information. 38:08.520 --> 38:20.640 So you need to work on the dark web in such a way that you don't disclose your full name or driver's license or bank account or even a simple thing as email. 38:21.140 --> 38:26.940 This is what you need to protect and that's why security is really important. 38:27.600 --> 38:38.960 And it's also a hard thing to do because at the end of the day, we are all humans and we like to be seen as knowledgeable and we like to impress others. 38:39.020 --> 38:44.800 This all leads to gossip, gossiping, bragging and oversharing with others. 38:44.800 --> 38:47.520 That's why operational security is really hard. 38:47.520 --> 38:51.500 And most of the time people think of it as a process. 38:51.500 --> 38:55.800 So people think that, okay, I have to do human intelligence stuff. 38:55.800 --> 38:57.900 Now I have to follow operational security. 38:57.900 --> 38:59.380 It should not be like that. 38:59.380 --> 39:07.800 It should not be seen as a burden to perform or as another of your job tasks to perform. 39:07.800 --> 39:16.880 It should be a mindset like you should always think about operational security before doing human intelligence or before engaging with actors. 39:17.820 --> 39:26.420 So I will discuss some of the things that you can use to maintain opsec in your daily lifestyle. 39:26.420 --> 39:28.520 There are many other things. 39:28.600 --> 39:32.820 So the main thing you want is hiding your identity. 39:32.820 --> 39:40.520 So the first thing you can do or you should do is use a separate system like I talked about before. 39:40.520 --> 39:45.160 Also use a separate system where you don't store any personal information. 39:46.400 --> 39:52.140 Whether it be a lab or VM or some kind of system. 39:52.260 --> 39:59.920 The main thing is to use Tor with a proxy or Tor over VPN. 40:01.620 --> 40:08.280 The main thing is maintaining different personas on the dark web. 40:08.280 --> 40:14.020 It's like I told, it's an equivalent of an FBI agent going undercover. 40:14.020 --> 40:15.920 So he has some kind of persona. 40:15.920 --> 40:17.060 He has a backstory. 40:17.060 --> 40:18.480 You have to do that. 40:18.480 --> 40:19.960 You have to do the same thing. 40:19.960 --> 40:25.300 You should have different personas for different identities that you have on different forums. 40:26.400 --> 40:29.320 You should never mix it up. 40:29.320 --> 40:36.380 That's why you take extensive notes so that you don't mess up the personas. 40:38.740 --> 40:45.240 You should always watch what you say and you should always think before posting. 40:47.240 --> 40:52.660 Human intelligence is not a 95 job thing. 40:52.660 --> 41:07.200 You can't just talk to or you can't just communicate to actors during your job time because they will know that you are doing this as your job. 41:07.200 --> 41:10.040 In this way, you can be exposed. 41:10.560 --> 41:17.220 They can easily guess that you are a researcher and not a threat actor. 41:18.720 --> 41:21.560 It would be a tip off for them. 41:21.560 --> 41:27.260 That's why you have to do this work 24 by 7. 41:27.260 --> 41:32.120 It's not like you have to do this work on the weekends. 41:32.120 --> 41:34.720 You have to do this work after your work hours. 41:34.720 --> 41:37.900 That's why it's not a 95 job thing. 41:38.120 --> 41:48.200 You have to develop appropriate language skills because people don't talk formally like actors don't talk formally. 41:48.200 --> 41:52.720 You have to develop appropriate language skills or slang skills. 41:52.720 --> 41:57.640 Also, there are many forums like there are different Russian forums or German forums. 41:57.640 --> 42:03.440 You might need to develop that language skills like learning Russian, learning German. 42:03.840 --> 42:07.180 Another thing to note is changing time zones. 42:07.180 --> 42:28.180 If you are in US and you are accessing or you are engaging in a community on a Russian forum, you might want to change the time zone to Russia because again, it would be a tip off to the actors that you are a security researcher or that you are not a real actor. 42:28.220 --> 42:33.900 These are some of the things that should be noted before doing any intelligence stuff on the dark web. 42:35.760 --> 42:38.240 Now, that was it. 42:38.240 --> 42:44.780 So concluding all this, we discussed a little about the dark web, what dark web is, how to access dark web. 42:44.820 --> 42:53.960 We discussed about dark web forums and marketplaces, what different products are being sold there, what is the cost model around that. 42:53.960 --> 42:58.340 We discussed about threat hunting on the dark web, how you can hunt on the dark web. 42:58.340 --> 43:03.360 We discussed different tools and the main tool was scraping. 43:03.360 --> 43:09.600 That's the main framework that we are working on to hunt on the dark web. 43:09.600 --> 43:20.580 We discussed about human intelligence, how it can be used or how it should be used to support your tool-based data collection. 43:20.800 --> 43:25.700 We discussed about the pipeline or the architecture that can be used to automate the dark web hunting. 43:26.300 --> 43:34.700 We talked a little about a threat intelligence life cycle, how threat hunting on the dark web maps to the life cycle steps. 43:34.720 --> 43:43.220 And we talked about operational security and why it is so important and why it is so hard to do operational security. 43:45.400 --> 43:54.360 Again, some more points to notice, it's obviously threat hunting on the dark web is hard, but it's worth the effort. 43:54.360 --> 43:58.020 You don't get intelligence that you get from the dark web. 43:58.580 --> 44:01.480 You should always keep operational security in mind. 44:02.640 --> 44:07.180 Like I said before, you should look on more than one resource. 44:07.180 --> 44:08.340 You should look on forums. 44:08.340 --> 44:14.900 You could look on clear web forums also like pastebin and telegram as an example. 44:15.300 --> 44:18.460 And you should look on different other forums on the dark web also. 44:18.800 --> 44:21.780 It takes a lot of resources and a team effort. 44:21.780 --> 44:24.360 You can't do all these things alone. 44:24.360 --> 44:26.900 You should have a team for this. 44:26.900 --> 44:38.220 And we talked a little about usage of MITRE attack framework and how it can be useful to map your TTPs to that. 44:39.880 --> 44:49.700 These are some of the resources that I suggest you to read if you want to know more about the dark web stuff or dark web hunting stuff. 44:50.040 --> 44:56.360 The majority, the major ones are like a recorded feature insight, CrowdStrike, digital shadows. 44:56.560 --> 44:59.920 They release their blogs or white papers regularly. 44:59.920 --> 45:07.140 So read them and you'll understand what all things are there on the dark web. 45:10.100 --> 45:12.800 So yeah, that was it. 45:12.800 --> 45:14.480 I think, thank you so much. 45:14.480 --> 45:24.900 I hope you all liked my presentation and you can contact me on Twitter or LinkedIn if you have any doubt or if you want to discuss about this stuff more. 45:24.900 --> 45:29.040 I will hang on in discord to answer all your questions. 45:29.040 --> 45:30.920 So yeah, thank you.