1 00:00:03,850 --> 00:00:06,870 Hello everyone, my name is Apoorva Singh Gautam. 2 00:00:06,870 --> 00:00:12,130 I would like to thank Defcon Retain Village for having me here. 3 00:00:12,470 --> 00:00:17,810 I will talk about automating threat hunting on the dark web and the things that surrounds it. 4 00:00:17,830 --> 00:00:26,170 I presented this talk, I presented a short version of this talk at Grimcon and this is the extended version of it. 5 00:00:26,170 --> 00:00:28,450 So let's get started. 6 00:00:28,450 --> 00:00:33,250 I will switch off my webcam so that you guys can focus on the presentation here. 7 00:00:36,180 --> 00:00:38,480 Okay, so a little about me. 8 00:00:38,480 --> 00:00:40,380 My name is Apoorva Singh Gautam. 9 00:00:40,380 --> 00:00:42,680 I go by the handle ASGScorpion. 10 00:00:42,680 --> 00:00:44,380 I am a security researcher. 11 00:00:44,380 --> 00:00:51,280 I started into threat intel or hunting two years back and I've been loving it since then. 12 00:00:51,280 --> 00:00:56,000 I'm currently pursuing my master's in cyber security at Georgia Tech. 13 00:00:56,060 --> 00:01:04,660 Recently, during the summer, I was a research intern at UC Berkeley doing research in threat intelligence. 14 00:01:04,660 --> 00:01:08,160 Some of my hobbies are gaming. 15 00:01:08,160 --> 00:01:11,260 I pretty much love Rainbow Six Siege. 16 00:01:11,260 --> 00:01:14,000 I sometimes stream it also. 17 00:01:14,000 --> 00:01:15,740 I love hiking. 18 00:01:15,740 --> 00:01:20,700 I recently started into lockpicking and I've been enjoying it. 19 00:01:20,700 --> 00:01:23,300 I contribute to the security community. 20 00:01:23,300 --> 00:01:27,060 It's my passion contributing to the security community. 21 00:01:27,060 --> 00:01:30,040 I'm a senior teaching assistant at Siberia. 22 00:01:30,040 --> 00:01:34,600 I contribute at Station X also and at the local security meetups. 23 00:01:35,040 --> 00:01:40,380 And there are my socials if you want to contact me or hit me up. 24 00:01:42,100 --> 00:01:44,280 So what's today's agenda? 25 00:01:44,280 --> 00:01:57,420 So we will talk about introduction to dark web, what dark web is, how to access dark web, what Tor is, what is the difference between dark web, deep web, why you should perform hunting on the dark web. 26 00:01:57,420 --> 00:02:03,140 Before that, we will discuss what is threat hunting and why it is crucial to hunt on the dark web. 27 00:02:03,140 --> 00:02:06,780 We will discuss different methods to hunt on the dark web. 28 00:02:07,080 --> 00:02:10,520 Can the dark web hunting be automated? 29 00:02:10,600 --> 00:02:15,840 What's the pipeline or the architecture of hunting, automating the dark web hunting? 30 00:02:16,420 --> 00:02:20,520 Then we will discuss a little about threat intelligence lifecycle. 31 00:02:20,660 --> 00:02:28,120 That's how threat hunting on the dark web is analogous to threat intelligence lifecycle. 32 00:02:28,420 --> 00:02:31,400 What steps are there that corresponds to this? 33 00:02:31,600 --> 00:02:39,760 We will discuss a little about operational security, that's OPSEC, and why is it important to secure yourself when hunting on the dark web. 34 00:02:40,040 --> 00:02:41,600 And yeah, that's it. 35 00:02:43,320 --> 00:02:45,040 So introduction to dark web. 36 00:02:45,560 --> 00:02:53,640 I'm sure you must have seen this image a lot of times on the internet that shows difference between surface web, deep web, and dark web. 37 00:02:53,740 --> 00:03:01,100 So the surface web is the sites that are indexed by different search engines, that's Google, Bing, Yahoo, etc. 38 00:03:01,620 --> 00:03:05,680 And the majority of the portion of the internet is deep web. 39 00:03:05,780 --> 00:03:11,400 Now deep web is any site, any website that is not indexed by the search engine. 40 00:03:11,400 --> 00:03:22,180 So this can include your databases, your server instances, or any other different websites that you cannot search from the search engines. 41 00:03:22,860 --> 00:03:27,220 The third one is the dark web that we will talk mostly about today. 42 00:03:27,340 --> 00:03:35,080 And this is the part of the internet where you need some kind of software, special software to access the dark web. 43 00:03:35,400 --> 00:03:49,000 So this can include anything related to like drugs or weapons that's being sold on the dark web, or some kind of research, or books that's sold there. 44 00:03:49,320 --> 00:03:55,840 So dark web includes several forums and marketplaces where people sell different kind of things. 45 00:03:55,840 --> 00:03:58,280 So these type of things are sold there. 46 00:04:00,140 --> 00:04:06,660 So majority portion of the internet is deep web, and people confuse it between dark web and deep web. 47 00:04:06,660 --> 00:04:13,100 So as you can see, only 6% of the internet is dark web, and 80 to 90% is the deep web. 48 00:04:16,090 --> 00:04:19,250 Moving on, so how do you access the dark web? 49 00:04:19,250 --> 00:04:31,430 So there are several companies, several organizations that offer their own dark web systems, or you can call it, so the famous one is Tor, that's the onion router. 50 00:04:32,890 --> 00:04:37,090 The second is I2P, that's an invisible internet project. 51 00:04:37,090 --> 00:04:40,490 Zeronet is also becoming popular nowadays. 52 00:04:41,130 --> 00:04:51,090 So Tor has .onion domain, like the domain name ends with .onion, and I2P is .i2p. 53 00:04:51,530 --> 00:04:58,610 Talking more about Tor, what Tor is and how it works, is it's like a three-layer proxy system. 54 00:04:58,610 --> 00:05:15,130 If you can see on the image here, there are entry nodes, the entry node is where your traffic goes to, and then it goes through the middle layer, and the exit layer, and then goes to the destination. 55 00:05:15,610 --> 00:05:26,370 This way your identity is hidden, and also only the entry node is publicly listed, rest of the nodes are not publicly listed. 56 00:05:26,830 --> 00:05:37,130 So it hides your identity, it hides your IP address, and these nodes are volunteer-based systems. 57 00:05:37,130 --> 00:05:48,530 So Tor has about like 6,000 relays, and I don't know the number about I2P, but it is also becoming more popular nowadays. 58 00:05:49,130 --> 00:06:02,410 And the major thing about Tor is each node only knows the IP address of the next node or the previous node. 59 00:06:02,410 --> 00:06:08,310 So if we talk about here, the entry node doesn't know about the exit node or vice versa. 60 00:06:08,310 --> 00:06:11,810 So in this way, the location of the nodes are also protected. 61 00:06:14,830 --> 00:06:19,770 So there are many misconceptions about Tor, the dark web. 62 00:06:19,810 --> 00:06:28,690 The famous one is whenever we talk about Tor, people think about criminal or criminal things that goes on Tor. 63 00:06:28,930 --> 00:06:35,910 Yes, there is criminal side of Tor, but there is a good side of the Tor also, or the dark web. 64 00:06:35,910 --> 00:06:41,330 So it's really famous among whistleblowers or activists. 65 00:06:42,070 --> 00:06:47,370 There are many countries where free speech is limited. 66 00:06:47,670 --> 00:06:56,590 So they can use Tor, like people from those countries use Tor to express their speech or express what they think. 67 00:06:57,170 --> 00:07:07,110 Tor also has access to many old literature or researches, which is not available on the open web. 68 00:07:07,810 --> 00:07:11,070 And it's like safe haven for journalists, obviously. 69 00:07:11,790 --> 00:07:19,970 So there are many popular sites like Facebook and New York Times that has their counterpart on Tor, like their .NER website counterpart. 70 00:07:20,050 --> 00:07:25,930 So it's useful for whistleblowers or activists. 71 00:07:26,290 --> 00:07:33,050 The second thing is Tor is so many people think that it's illegal to access the dark web. 72 00:07:33,350 --> 00:07:36,170 So it's not illegal to access the dark web. 73 00:07:36,170 --> 00:07:48,070 It's illegal to indulge in these kinds of activities like purchasing drugs or purchasing any other illegal things on the dark web that is being sold there. 74 00:07:48,070 --> 00:07:50,790 But it's completely legal to access the dark web. 75 00:07:52,710 --> 00:08:01,810 And it's like the last thing which I would talk about is many people think Tor is like really big. 76 00:08:01,810 --> 00:08:12,910 But if you talk about the uptime or the availability of sites, there are very few onion domains on the dark web if you compare it with the clear web. 77 00:08:12,930 --> 00:08:18,670 So it's like the dark web is like very little part of the internet. 78 00:08:22,040 --> 00:08:25,540 So we will talk about the dark side of the dark web. 79 00:08:25,540 --> 00:08:28,300 I mean, the criminal side of the dark web. 80 00:08:28,300 --> 00:08:32,160 So there are many forums or marketplaces on the dark web. 81 00:08:32,640 --> 00:08:43,420 These are some of the relevant sites that's relevant to security researchers or people who want to access dark web for their organization's benefit. 82 00:08:43,540 --> 00:08:47,240 So this is like some of them are like a credit card market. 83 00:08:47,240 --> 00:08:51,620 So where different credit cards are being dumped or remote access. 84 00:08:51,620 --> 00:08:59,380 So these forums include remote access trojans or some kind of remote access tools or insider threats. 85 00:08:59,380 --> 00:09:12,560 So insider threats are it's like recent coming up a forum where insider like the people who are selling their company's secrets, they talk amongst themselves. 86 00:09:12,560 --> 00:09:14,560 So these are some of the relevant sites. 87 00:09:15,660 --> 00:09:17,680 Now coming to the cost. 88 00:09:17,680 --> 00:09:24,520 So how much it costs for some kind of for something to buy from the dark web. 89 00:09:24,520 --> 00:09:29,040 So as you can see, it's really easy to buy these things on the dark web and it. 90 00:09:29,100 --> 00:09:40,320 So like SSN, you can buy any SSN for $1 or fake FB friends, a fake FB with 15 friends, mobile malware, bank details, so exploit to zero days. 91 00:09:40,320 --> 00:09:46,480 So it's like these type of things are easy to get on the dark web. 92 00:09:46,480 --> 00:09:50,760 That's why security researchers and other people are focusing more on the dark web. 93 00:09:50,760 --> 00:10:00,260 You might have heard about recent news, like 500,000 Zoom accounts sold on the dark web or 267 million FB user profiles sold on the dark web. 94 00:10:00,260 --> 00:10:06,900 So there are many data breaches occurring day by day and they are being sold on the dark web. 95 00:10:06,900 --> 00:10:11,160 That's why researching on the dark web is really important. 96 00:10:13,200 --> 00:10:17,760 These are some of the product listings from the forums of the dark web. 97 00:10:17,760 --> 00:10:20,540 This is how the products are being listed. 98 00:10:23,420 --> 00:10:34,860 And you can see these are the average cost of accounts for different online services like bank services, what's the average cost of it or what's the average cost of service for video games. 99 00:10:35,940 --> 00:10:40,180 This is the average cost of tools that is being sold on the dark web. 100 00:10:40,180 --> 00:10:43,840 As you can see again, bank and financial average cost is $74. 101 00:10:43,840 --> 00:10:49,180 So you can buy a brute forcing tool for some bank at $74. 102 00:10:51,520 --> 00:10:54,260 Now coming to why you should hunt on the dark web. 103 00:10:54,260 --> 00:10:57,780 Before that, let's talk about what is threat hunting. 104 00:10:57,940 --> 00:11:02,780 Threat hunting is proactive searching for cyber threats. 105 00:11:02,780 --> 00:11:06,400 Proactive means before the attack happens. 106 00:11:06,400 --> 00:11:08,000 That's proactive search. 107 00:11:08,140 --> 00:11:12,580 You search for cyber threats from logs or indicators of compromise. 108 00:11:12,580 --> 00:11:15,540 That's IP addresses, emails, domains, etc. 109 00:11:15,540 --> 00:11:16,600 or textual data. 110 00:11:16,600 --> 00:11:20,820 That's what we are doing when we are searching the dark web. 111 00:11:21,680 --> 00:11:27,460 So it's basically hypothesis based because there's nothing concrete about the process. 112 00:11:27,460 --> 00:11:32,180 You take one use case and work on it and then you take another use case and work on it. 113 00:11:32,180 --> 00:11:35,160 And it goes iteratively in the same way. 114 00:11:35,440 --> 00:11:54,740 Many times there's use of machine learning or natural language processing that's NLP and advanced analytics process in this because you need to scan through the textual data if you are hunting on the dark web or if you are hunting on the clear web also but for textual data. 115 00:11:54,780 --> 00:11:58,520 So machine learning and advanced analytics are useful there. 116 00:12:01,260 --> 00:12:02,940 So why it's serious? 117 00:12:02,940 --> 00:12:06,860 Why threat hunting on the dark web is really important? 118 00:12:06,860 --> 00:12:09,800 What's so big about that? 119 00:12:09,800 --> 00:12:15,800 As I told you about again, there are many forums, marketplaces, dump shops. 120 00:12:16,880 --> 00:12:19,880 So what criminals or actors do? 121 00:12:19,880 --> 00:12:22,340 They learn new methods and techniques on the dark web. 122 00:12:22,340 --> 00:12:24,300 They monetize their skills. 123 00:12:24,300 --> 00:12:30,140 They trade their exploits or tools or even drugs and weapons and communication. 124 00:12:30,140 --> 00:12:36,480 So they communicate with each other and they share their ideas for new attacks. 125 00:12:36,480 --> 00:12:48,800 A security researcher or a person who is researching on the dark web, he can find a lot on, he can learn a lot while engaging within these communities. 126 00:12:49,200 --> 00:13:02,060 You can learn their techniques or TTPs, that's the attack techniques and procedures, how they think about the attack, how they plan an attack and it can identify. 127 00:13:02,060 --> 00:13:12,840 So if you do it correctly, it can identify attacks in the earlier stages, that's planning and recon stages and you can reduce the impacts that it causes. 128 00:13:12,840 --> 00:13:22,560 So suppose if your organization data is being sold on the dark web, there are different kinds of impacts that can cause to your organization. 129 00:13:22,560 --> 00:13:31,340 So some of the direct impacts are like personal information stolen or healthcare records stolen or even your company's trade secrets. 130 00:13:31,360 --> 00:13:43,060 And some of the indirect impacts are repetition of your organization, revenue loss, and nowadays the legal penalties that your data is lost and you have to cover the cost of the customers. 131 00:13:44,780 --> 00:13:55,480 So this is like, that's why this is really important to like to research on the dark web, to hunt threats on the dark web. 132 00:13:57,560 --> 00:14:01,380 On the same lines, these are the benefits of the threat hunting. 133 00:14:01,380 --> 00:14:15,460 So if you do it correctly, you can keep up with the latest trends of the attacks, you can get new TTPs, that's tactics, techniques and procedures, you can identify insider threats, you can discover data breaches. 134 00:14:15,460 --> 00:14:27,640 The main thing is you can prepare your SOCs and incident responders to deal with the attack because they will know before only what are the TTPs attackers are using. 135 00:14:27,640 --> 00:14:32,600 So they can reduce the damage and risk to the organization by acting quickly on that. 136 00:14:35,800 --> 00:14:38,800 So coming to the methods to hunt on the dark web. 137 00:14:38,800 --> 00:14:42,740 So we will discuss about some tools that's used to hunt on the dark web. 138 00:14:42,740 --> 00:14:47,120 And then we will discuss about the human element that can be used to hunt on the dark web. 139 00:14:47,140 --> 00:14:53,880 So talking about the first tool that's really, really important for this is Scrapy. 140 00:14:53,880 --> 00:14:56,920 So it's a web crawling framework. 141 00:14:57,320 --> 00:15:03,940 It's so famous, it's so important because it manages multithreading automatically. 142 00:15:03,940 --> 00:15:16,500 So you don't have to write, you don't have to spend too much time on the multithreading part, because it has already capabilities for multithreading using one or two lines of one or two lines of parameters. 143 00:15:16,980 --> 00:15:18,900 The second thing is Tor. 144 00:15:18,900 --> 00:15:21,840 Obviously, if you want to access the dark web, you need Tor. 145 00:15:22,200 --> 00:15:28,140 Onion scan is another tool that is used to search for onion websites. 146 00:15:28,340 --> 00:15:35,600 It can tell you if a website is up or not and the correlation between different websites on the dark web. 147 00:15:36,380 --> 00:15:37,760 Coming to Pryvoxy. 148 00:15:37,760 --> 00:15:41,400 So Pryvoxy is a web proxy. 149 00:15:41,400 --> 00:15:44,180 Before getting more into this. 150 00:15:44,180 --> 00:15:57,240 So when you access the dark web, you need some kind of proxy to access the dark web because your ISP, as I told you before, the entry nodes are publicly listed. 151 00:15:57,240 --> 00:16:01,540 So your ISP can have a blacklist to block the entry nodes. 152 00:16:01,540 --> 00:16:11,320 So you can't access the dark web or even if the ISP doesn't block it , he can see whether you are accessing the dark web or not. 153 00:16:11,320 --> 00:16:16,700 He cannot see what you are doing on there, but he can see whether you are accessing the dark web or not. 154 00:16:16,700 --> 00:16:18,740 So you might not want that. 155 00:16:18,740 --> 00:16:23,940 That's why you need some kind of proxy and a majority of people use SOCKS proxy. 156 00:16:23,940 --> 00:16:32,960 So basic difference between HTTP and SOCKS proxy is SOCKS proxy is a lower level proxy and it works on the SOCKS protocol. 157 00:16:33,280 --> 00:16:42,540 HTTP proxy only works on HTTP or HTTPS websites, but SOCKS proxy can work on other protocols also. 158 00:16:43,420 --> 00:16:50,240 And there are different tools to use SOCKS proxy like TSOCKS, Polypo and Pryvoxy. 159 00:16:50,240 --> 00:16:57,080 I've been using Pryvoxy and it has been so I don't have any problem with Pryvoxy. 160 00:16:57,080 --> 00:16:58,960 So and it's good. 161 00:16:59,120 --> 00:17:09,520 So another thing is SCP doesn't allow you to use directly SOCKS proxy because it doesn't support SOCKS proxy. 162 00:17:09,520 --> 00:17:19,660 So that's why you have to use these tools like Pryvoxy, TSOCKS or Polypo to route your SOCKS through Pryvoxy scripts. 163 00:17:20,380 --> 00:17:28,860 And there are other tools also like their search engines like Kilos or Recon where you can find different end-to-end domains. 164 00:17:29,460 --> 00:17:41,020 Apart from using SOCKS proxy, you can also use VPN with Tor for extra layer of protection and like encrypting your data. 165 00:17:43,730 --> 00:17:55,050 So getting more into SCP part, this image might seem a little confusing, but I will get into it like step by step. 166 00:17:55,050 --> 00:18:03,250 So this is like why SCP is really important and why SCP is so useful in hunting on the dark web. 167 00:18:03,250 --> 00:18:11,230 So if you can see there, so for explaining this, I will explain it in terms of Python code. 168 00:18:11,230 --> 00:18:16,210 So suppose everything you see here is a different Python program. 169 00:18:16,210 --> 00:18:21,430 So spider is a Python program, downloader is a Python program, middleware is a Python program and so on. 170 00:18:22,210 --> 00:18:26,530 So what spider does in spider? 171 00:18:26,530 --> 00:18:35,670 So in spider Python program, you give your onion domain on which you want to crawl the data or which you want to get the data. 172 00:18:35,670 --> 00:18:38,470 So it gives it to the engine. 173 00:18:39,050 --> 00:18:44,770 Suppose engine is just the program that manages every other Python programs. 174 00:18:44,930 --> 00:18:49,870 So it gives the onion domain to the engine. 175 00:18:49,870 --> 00:18:51,550 The engine gives it to the scheduler. 176 00:18:51,550 --> 00:18:56,610 So what scheduler does is here the multi-threading concept comes into the picture. 177 00:18:56,750 --> 00:19:02,050 Scheduler gets different domains and schedules it accordingly into multiple threads. 178 00:19:02,090 --> 00:19:10,030 So the onion domain goes into the scheduler, the scheduler gives it back to the engine and engine gives it to the middleware. 179 00:19:10,030 --> 00:19:15,530 So middleware program includes your proxy program and or login program. 180 00:19:15,530 --> 00:19:19,050 So what proxy program is that in proxy function. 181 00:19:19,050 --> 00:19:25,410 So if we are talking about middleware, that's a Python program, there's a proxy function into it. 182 00:19:25,410 --> 00:19:39,210 So proxy function is where you will note, you will put down your privoxy IP or Tor IP, so that the request response goes and comes through that proxy so that you can access the dark web. 183 00:19:39,530 --> 00:20:01,690 The login program, the login function is where you will put your user agents or cookies or so for accessing the dark web forums, you have to, so there nowadays, for all the forums, you have to have an account to access the dark web or to access that particular forum. 184 00:20:01,690 --> 00:20:11,410 So to access the forum, you need some kind of cookies or also there are many forums, many high level forums that implement captures. 185 00:20:11,450 --> 00:20:19,210 And as Google doesn't work on the dark web, these captures are like image-based capture or text-based capture that is easy to bypass. 186 00:20:19,290 --> 00:20:30,950 So you can use any machine learning capture bypassing service or any capture bypassing websites like death by capture or anti-capture to bypass the capture. 187 00:20:30,950 --> 00:20:38,030 So these are all codes you will write in this login function in middleware program. 188 00:20:38,030 --> 00:20:43,310 Now your request or your traffic goes through this middleware program to the downloader. 189 00:20:43,310 --> 00:20:50,030 What downloader does is it's a simple program to extract the HTML and save and give it back to the engine. 190 00:20:50,030 --> 00:20:53,850 So downloader extract the HTML and gives it back to the engine. 191 00:20:54,110 --> 00:20:56,970 Now engine gives it back to the spider. 192 00:20:56,970 --> 00:21:16,370 Now there's another function in spider that extracts the HTML entities that you want from the forums HTML, suppose forum name or document ID or the text-based data or author name who posted a particular content. 193 00:21:16,370 --> 00:21:21,070 So you get that and it is called items in Scrapy. 194 00:21:21,070 --> 00:21:24,930 So you get these items and it sends it to the item pipeline. 195 00:21:24,930 --> 00:21:29,170 Item pipeline is where your database is configured. 196 00:21:29,170 --> 00:21:37,850 So I use Elasticsearch, you can use any database whether SQL or NoSQL and it directly saves the items to the SQL. 197 00:21:37,910 --> 00:21:49,890 And so the important thing to note here is that Scrapy, so as I told you before multi-threading is automatically handled by Scrapy. 198 00:21:49,890 --> 00:21:55,770 The another thing is you don't have to give multiple onion domains to spider. 199 00:21:55,810 --> 00:22:11,950 So when the downloader gets the data, when the downloader gets the HTML page from the particular forum, you don't have to configure the code, there is a code to get all the onion domains on that particular HTML page. 200 00:22:12,050 --> 00:22:18,590 So the scheduler automatically schedules the other onion domains to go through the same process again and again. 201 00:22:19,170 --> 00:22:24,550 In this way, you don't have to give extra onion domains to the engine. 202 00:22:25,110 --> 00:22:32,090 And this is why Scrapy is really useful in crawling data from the dark web. 203 00:22:33,250 --> 00:22:38,030 So you can specify which domain to crawl and which domain to block. 204 00:22:38,110 --> 00:22:43,950 In this way, you can be safe from getting illegal data or getting illegal images. 205 00:22:45,150 --> 00:22:49,150 Moving on, now comes the human part. 206 00:22:50,070 --> 00:22:54,790 So we discussed about the tools, what tools can you use to hunt on the dark web. 207 00:22:55,090 --> 00:23:00,430 There is a human element also that's called human intelligence or HUMINT. 208 00:23:00,610 --> 00:23:11,030 So it's like it's a process of gathering intelligence through interpersonal contact, interpersonal contact rather by some kind of tools or technical process. 209 00:23:11,210 --> 00:23:21,570 That's why it's the most, it's most dangerous and difficult form because you are directly talking to the actor on the dark web, which is not safe. 210 00:23:22,070 --> 00:23:35,130 And it's not safe because you don't want your identity to be revealed to the actor or you don't want your organization's identity to be revealed to the actor. 211 00:23:35,130 --> 00:23:43,610 And it's important also because you can identify and respond to attacks much quickly. 212 00:23:43,610 --> 00:23:46,270 You can do post-attack investigation. 213 00:23:46,270 --> 00:23:52,410 So suppose your organization, like there's a data breach on your organization. 214 00:23:52,830 --> 00:23:57,010 If you want to confirm, so someone is selling this data on the dark web. 215 00:23:57,010 --> 00:24:04,770 And if you want to confirm whether they are selling the correct, whether they are lying about it or whether it is the truth. 216 00:24:04,770 --> 00:24:17,470 So you can activate your human intelligence or activate your the guy that is researching on the dark web to go and ask to the actors whether the data is correct or not. 217 00:24:17,470 --> 00:24:19,530 So that's post-attack investigation. 218 00:24:19,590 --> 00:24:24,470 You can also use it for new attacker vector, new attack vector discovery. 219 00:24:24,470 --> 00:24:31,230 So that's discovering new TTPs that the attackers are using or the attackers are discussing about. 220 00:24:31,750 --> 00:24:43,870 You can assume is as a high-tech equivalent of what an FBI agent does when he spends months or years working to infiltrate a criminal organization. 221 00:24:43,870 --> 00:24:50,810 That's why it's really hard to do it because you have to spend so much time on it. 222 00:24:50,850 --> 00:24:53,330 And that's why it's risky. 223 00:24:53,490 --> 00:25:03,470 And for this, you have to think like an actor, how they communicate within these communities, how they act within these communities. 224 00:25:03,890 --> 00:25:12,050 And the other thing is like it's the source from this is really valuable to your organization's safety. 225 00:25:14,530 --> 00:25:26,350 Now moving on, so we talked about tools, we talked about human intelligence, but now comes the pipeline or the architecture of how you can automate these threat hunting. 226 00:25:26,350 --> 00:25:34,270 So before that, I would suggest to set up a different system. 227 00:25:34,370 --> 00:25:40,030 You don't want your personal data to be on the system where you are doing threat hunting. 228 00:25:40,030 --> 00:25:49,730 So you can set up any lab or VM, whether physical or whether on cloud, just isolate the network and install relevant tools like KB, Privoxy, Tor. 229 00:25:49,970 --> 00:25:59,330 If you are using Elasticsearch or Kibana, then Elk and different Python libraries that would be necessary for your task. 230 00:26:00,250 --> 00:26:06,450 So this is the automated architecture that I have been using it. 231 00:26:06,650 --> 00:26:09,590 I will go it one by one. 232 00:26:09,590 --> 00:26:23,590 And I have this automate icon for the task that can be automated and for the task that I don't have, that's the only one I think is creepy setup and the design train NLP model. 233 00:26:23,590 --> 00:26:25,830 So it's hard to automate that part. 234 00:26:25,830 --> 00:26:27,630 So I will discuss it through. 235 00:26:27,850 --> 00:26:34,810 So first of all, you need to get the forums, forum links or the market links. 236 00:26:34,810 --> 00:26:47,370 So you can write a simple script to gather data from different search engines like I told you Recon and other search engines where you can get all the forum links. 237 00:26:47,730 --> 00:26:49,550 So that can be automated. 238 00:26:49,830 --> 00:26:52,510 So another thing is using SOCKS proxy. 239 00:26:52,510 --> 00:26:55,810 Like I told you, you have to use some kind of SOCKS proxy for this. 240 00:26:55,810 --> 00:26:58,210 So you can get SOCKS proxies IP. 241 00:26:58,350 --> 00:27:04,030 Again, you can write a simple script and get the SOCKS proxies that can be automated. 242 00:27:04,790 --> 00:27:08,190 Now comes the part of Scrappy setup. 243 00:27:08,970 --> 00:27:16,830 So Scrappy setup is is the so here you will write your login functions here. 244 00:27:16,830 --> 00:27:25,330 You will get your proxy setups and here you will like manage the settings of the Scrappy. 245 00:27:25,750 --> 00:27:35,030 Now, you can't automate this because when you get the onion links or different forum onion links, you have to go to the forums and sign in. 246 00:27:35,050 --> 00:27:43,450 Yes, you can automate the you can write scripts for logging in or signing in using different accounts. 247 00:27:43,450 --> 00:27:47,210 But I found it difficult to do this. 248 00:27:47,210 --> 00:27:50,730 That's why I just I have been using it manual. 249 00:27:50,730 --> 00:27:52,130 I've been doing it manually. 250 00:27:52,130 --> 00:28:04,730 So like creating different accounts like four to five accounts per forum and then noting down that into the Scrappy like the username, password and cookies into Scrappy. 251 00:28:04,990 --> 00:28:07,150 So for this, you have to do this. 252 00:28:07,150 --> 00:28:16,150 Also, you need different scrapers for different forums, because the architecture of forums is different for different forums. 253 00:28:16,330 --> 00:28:34,150 That's why you need to do this step manually, because you need to analyze, you need to first log into the forum, analyze it, and then write a different function for each forum for the HTML elements that you want to access. 254 00:28:34,830 --> 00:28:36,650 Coming to the crawler part. 255 00:28:36,650 --> 00:28:46,530 So what crawler is so the crawler parser analyzer, it's and the ELK part, it's all the part of the Scrappy that I discussed before. 256 00:28:46,530 --> 00:28:49,950 So these all are part of the Scrappy system. 257 00:28:49,990 --> 00:28:51,770 I've just written it differently. 258 00:28:51,770 --> 00:28:54,250 So you can understand what each part does. 259 00:28:54,250 --> 00:29:06,590 So what crawler does is again, pages from the forums, parser does it's parses the HTML pages, like getting the HTML elements like post, post content, author, etc. 260 00:29:06,590 --> 00:29:07,890 And the analyzer. 261 00:29:07,890 --> 00:29:15,670 So analyzer part is the part, you can write different function for this in Scrappy. 262 00:29:15,670 --> 00:29:21,530 So what analyzer does is so suppose you got the data from the from the dark web forum. 263 00:29:21,530 --> 00:29:30,770 Now you need to use some kind of techniques to evaluate the content that is relevant to your organization or relevant to your threat model. 264 00:29:30,770 --> 00:29:35,930 So we will discuss what threat modeling is in the later part of the presentation. 265 00:29:36,050 --> 00:29:46,530 So for now, just understand this, you can't focus on every other threat that's out there, you have to focus on threat that is relevant to your organization. 266 00:29:46,570 --> 00:29:53,790 So you need to do some kind of analysis to like get the data, get the relevant data from the dark web. 267 00:29:53,790 --> 00:29:59,230 So here comes the NLP model that I've been using. 268 00:29:59,710 --> 00:30:09,770 So you can design a trained NLP model in this way that it can just get you the content that is relevant to your organization. 269 00:30:10,630 --> 00:30:28,990 So you don't, suppose if you are a bank, you don't want to focus on tools or you don't want to focus on data breaches that's not relevant to your bank, you most likely would want to focus on the dump shops, that's where credit cards or debit cards are being dumped. 270 00:30:29,270 --> 00:30:35,150 So in this way, different organizations have different requirements, and you want to focus on those. 271 00:30:35,150 --> 00:30:44,770 Now, designing and training an NLP model can't be automated because you need some kind of content before content relevant to your threat model. 272 00:30:44,830 --> 00:30:48,610 And then you need to train your NLP model on that. 273 00:30:49,550 --> 00:30:51,830 It's like, so it's the same thing. 274 00:30:51,830 --> 00:30:57,270 It's somewhat like either you get the data first or you design your model first. 275 00:30:57,270 --> 00:30:58,910 So it's like egg and chicken problem. 276 00:30:58,910 --> 00:31:12,410 But nowadays, there are many NLP models like CDED LDA, where you can provide some kind of context before training the NLP model. 277 00:31:12,910 --> 00:31:15,510 So it's easy to do that. 278 00:31:15,510 --> 00:31:18,070 And then you store the data into ELK. 279 00:31:18,070 --> 00:31:21,110 So these all things can be automated. 280 00:31:24,830 --> 00:31:32,670 So coming to the part after getting this data, so what's the process after hunting? 281 00:31:32,670 --> 00:31:36,450 Now we'll discuss a little about Threat Intelligence Lifecycle. 282 00:31:36,730 --> 00:31:52,110 So what Threat Intelligence Lifecycle is, these are different steps that your organization takes to build a threat, like it starts from getting the data till presentation of the data. 283 00:31:52,850 --> 00:32:05,970 So how threat hunting on the dark web corresponds to this is, so there are like five phases, as you can see, direction, collection, processing, analysis, and dissemination. 284 00:32:06,690 --> 00:32:14,010 What we are doing is, we are doing direction phase from the human sources, like you can see from dark web social media forums. 285 00:32:14,210 --> 00:32:23,770 So in the direction phase, we identify dark web forums, we register on those forums, we acquire access on those forums. 286 00:32:23,950 --> 00:32:28,610 In the collection phase, you use Scrapy to establish access and collect raw data. 287 00:32:29,190 --> 00:32:37,530 Processing phase is also using Scrapy, so you parse raw HTML data, you extract topics and authors. 288 00:32:37,590 --> 00:32:45,670 And the analysis phase is where we use NLP and other machine learning models to infer relationship between these data. 289 00:32:46,030 --> 00:32:49,510 We get data that is relevant to our organization. 290 00:32:49,610 --> 00:32:53,730 We link data sources, we identify trends and hacks and leaks, etc. 291 00:32:53,730 --> 00:32:57,410 And dissemination phase is where we visualize the data in dashboards. 292 00:32:57,410 --> 00:33:11,170 If you are using Kibana or other kind of dashboards, we give out alerts and reports for our higher managers or the other people to see in our organization. 293 00:33:11,430 --> 00:33:20,290 So this is like crux of what threat hunting on the dark web maps to threat intelligence life cycle. 294 00:33:21,730 --> 00:33:23,030 Now, threat modeling. 295 00:33:23,030 --> 00:33:27,610 As I told you, I was going to talk about this in the coming presentation. 296 00:33:27,610 --> 00:33:42,610 So what threat modeling is, it's like getting your organization's critical assets and focusing on your organization's critical asset. 297 00:33:42,610 --> 00:33:52,090 So it's like understanding threats and how you can mitigate it when it happens to your organization particularly. 298 00:33:52,090 --> 00:34:11,550 So you understand what attacker wants, what different critical assets you have in your organization, what are different types of actors that can target you, whether they would be activists or insiders or some kind of criminal groups and know their capability. 299 00:34:14,110 --> 00:34:17,210 So here you choose your target on the dark web. 300 00:34:20,090 --> 00:34:23,210 So if you are a bank, you focus on credit card markets. 301 00:34:23,210 --> 00:34:29,810 If you are some other organization, if you want, you focus on inside the Fed markets or you focus on general markets. 302 00:34:30,430 --> 00:34:33,750 So in this way, you choose your target on the dark web. 303 00:34:33,930 --> 00:34:38,450 You prioritize risk as you can use pyramid of pain for that. 304 00:34:38,450 --> 00:34:45,670 So you prioritize risk and focus on IOCs that are relevant to your organization. 305 00:34:46,810 --> 00:34:54,970 Another thing is you don't just use one source to target, like there are many, many forums on the dark web. 306 00:34:54,970 --> 00:34:59,630 You don't just focus on one target, you focus on multiple targets. 307 00:34:59,630 --> 00:35:13,350 Apart from dark web, you focus on multiple clear websites also like pastebin or Twitter or nowadays on telegram also many these many actors are communicating. 308 00:35:13,350 --> 00:35:22,790 So you focus on the dark web also and the clear web also to get all the things you can for protecting your organization. 309 00:35:23,990 --> 00:35:30,230 So again, data collection processing, you collect data from the clear web and the dark web. 310 00:35:30,450 --> 00:35:33,890 So some of the sites are pastebin, Twitter, Reddit. 311 00:35:34,330 --> 00:35:38,410 On the dark web, it's forums, different forums and different marketplaces. 312 00:35:38,490 --> 00:35:43,890 You can do all this using this creepy crawler and parser that we discussed before. 313 00:35:45,110 --> 00:35:55,350 The analysis part in the threat intelligence model is you use, like I told you before, you use NLP machine learning or deep learning techniques. 314 00:35:56,090 --> 00:36:03,810 Some of them are like LDA, BERT, GPT to gather information related to your organization. 315 00:36:04,010 --> 00:36:15,830 You use social network analysis for analysis of different users on the dark web that post data related to your organization. 316 00:36:16,270 --> 00:36:22,370 There's clustering of products according to categories for the clustering thing. 317 00:36:22,370 --> 00:36:24,110 You classify different. 318 00:36:24,230 --> 00:36:28,210 So there's like binary classification, multi-class classification. 319 00:36:28,210 --> 00:36:32,930 So that's how you classify different products being sold on the dark web. 320 00:36:32,970 --> 00:36:36,050 So these all things come under analysis. 321 00:36:36,070 --> 00:36:39,770 I will touch a little on MITRE ATT&CK framework. 322 00:36:39,770 --> 00:36:52,550 So what MITRE ATT&CK is, it's a knowledge base of all the TTPs that was built using real world observations. 323 00:36:52,690 --> 00:36:59,490 So it contains different tactics, techniques, and procedures that attackers have used all these years. 324 00:36:59,850 --> 00:37:10,290 So you use ATT&CK matrix to map the intelligence you obtained to understand the TTPs better and to protect your organization more. 325 00:37:13,500 --> 00:37:18,520 So now coming to the operational security stuff. 326 00:37:18,520 --> 00:37:38,220 So hunting on the dark web or if you are doing human intelligence stuff on the dark web, you need to follow some set of processes so that you don't reveal your data or reveal your identity or your organizational identity as I talked before. 327 00:37:39,100 --> 00:37:41,060 So what is OPSEC? 328 00:37:41,060 --> 00:37:54,140 So OPSEC is the practice of hiding yourself online so that you don't reveal your real self or you don't reveal or compromise your own operations. 329 00:37:54,140 --> 00:37:57,740 It's derived from the US military, that's operational security. 330 00:37:58,440 --> 00:38:08,520 You need to hide your PIA, that's personal identifiable information. 331 00:38:08,520 --> 00:38:20,640 So you need to work on the dark web in such a way that you don't disclose your full name or driver's license or bank account or even a simple thing as email. 332 00:38:21,140 --> 00:38:26,940 This is what you need to protect and that's why security is really important. 333 00:38:27,600 --> 00:38:38,960 And it's also a hard thing to do because at the end of the day, we are all humans and we like to be seen as knowledgeable and we like to impress others. 334 00:38:39,020 --> 00:38:44,800 This all leads to gossip, gossiping, bragging and oversharing with others. 335 00:38:44,800 --> 00:38:47,520 That's why operational security is really hard. 336 00:38:47,520 --> 00:38:51,500 And most of the time people think of it as a process. 337 00:38:51,500 --> 00:38:55,800 So people think that, okay, I have to do human intelligence stuff. 338 00:38:55,800 --> 00:38:57,900 Now I have to follow operational security. 339 00:38:57,900 --> 00:38:59,380 It should not be like that. 340 00:38:59,380 --> 00:39:07,800 It should not be seen as a burden to perform or as another of your job tasks to perform. 341 00:39:07,800 --> 00:39:16,880 It should be a mindset like you should always think about operational security before doing human intelligence or before engaging with actors. 342 00:39:17,820 --> 00:39:26,420 So I will discuss some of the things that you can use to maintain opsec in your daily lifestyle. 343 00:39:26,420 --> 00:39:28,520 There are many other things. 344 00:39:28,600 --> 00:39:32,820 So the main thing you want is hiding your identity. 345 00:39:32,820 --> 00:39:40,520 So the first thing you can do or you should do is use a separate system like I talked about before. 346 00:39:40,520 --> 00:39:45,160 Also use a separate system where you don't store any personal information. 347 00:39:46,400 --> 00:39:52,140 Whether it be a lab or VM or some kind of system. 348 00:39:52,260 --> 00:39:59,920 The main thing is to use Tor with a proxy or Tor over VPN. 349 00:40:01,620 --> 00:40:08,280 The main thing is maintaining different personas on the dark web. 350 00:40:08,280 --> 00:40:14,020 It's like I told, it's an equivalent of an FBI agent going undercover. 351 00:40:14,020 --> 00:40:15,920 So he has some kind of persona. 352 00:40:15,920 --> 00:40:17,060 He has a backstory. 353 00:40:17,060 --> 00:40:18,480 You have to do that. 354 00:40:18,480 --> 00:40:19,960 You have to do the same thing. 355 00:40:19,960 --> 00:40:25,300 You should have different personas for different identities that you have on different forums. 356 00:40:26,400 --> 00:40:29,320 You should never mix it up. 357 00:40:29,320 --> 00:40:36,380 That's why you take extensive notes so that you don't mess up the personas. 358 00:40:38,740 --> 00:40:45,240 You should always watch what you say and you should always think before posting. 359 00:40:47,240 --> 00:40:52,660 Human intelligence is not a 95 job thing. 360 00:40:52,660 --> 00:41:07,200 You can't just talk to or you can't just communicate to actors during your job time because they will know that you are doing this as your job. 361 00:41:07,200 --> 00:41:10,040 In this way, you can be exposed. 362 00:41:10,560 --> 00:41:17,220 They can easily guess that you are a researcher and not a threat actor. 363 00:41:18,720 --> 00:41:21,560 It would be a tip off for them. 364 00:41:21,560 --> 00:41:27,260 That's why you have to do this work 24 by 7. 365 00:41:27,260 --> 00:41:32,120 It's not like you have to do this work on the weekends. 366 00:41:32,120 --> 00:41:34,720 You have to do this work after your work hours. 367 00:41:34,720 --> 00:41:37,900 That's why it's not a 95 job thing. 368 00:41:38,120 --> 00:41:48,200 You have to develop appropriate language skills because people don't talk formally like actors don't talk formally. 369 00:41:48,200 --> 00:41:52,720 You have to develop appropriate language skills or slang skills. 370 00:41:52,720 --> 00:41:57,640 Also, there are many forums like there are different Russian forums or German forums. 371 00:41:57,640 --> 00:42:03,440 You might need to develop that language skills like learning Russian, learning German. 372 00:42:03,840 --> 00:42:07,180 Another thing to note is changing time zones. 373 00:42:07,180 --> 00:42:28,180 If you are in US and you are accessing or you are engaging in a community on a Russian forum, you might want to change the time zone to Russia because again, it would be a tip off to the actors that you are a security researcher or that you are not a real actor. 374 00:42:28,220 --> 00:42:33,900 These are some of the things that should be noted before doing any intelligence stuff on the dark web. 375 00:42:35,760 --> 00:42:38,240 Now, that was it. 376 00:42:38,240 --> 00:42:44,780 So concluding all this, we discussed a little about the dark web, what dark web is, how to access dark web. 377 00:42:44,820 --> 00:42:53,960 We discussed about dark web forums and marketplaces, what different products are being sold there, what is the cost model around that. 378 00:42:53,960 --> 00:42:58,340 We discussed about threat hunting on the dark web, how you can hunt on the dark web. 379 00:42:58,340 --> 00:43:03,360 We discussed different tools and the main tool was scraping. 380 00:43:03,360 --> 00:43:09,600 That's the main framework that we are working on to hunt on the dark web. 381 00:43:09,600 --> 00:43:20,580 We discussed about human intelligence, how it can be used or how it should be used to support your tool-based data collection. 382 00:43:20,800 --> 00:43:25,700 We discussed about the pipeline or the architecture that can be used to automate the dark web hunting. 383 00:43:26,300 --> 00:43:34,700 We talked a little about a threat intelligence life cycle, how threat hunting on the dark web maps to the life cycle steps. 384 00:43:34,720 --> 00:43:43,220 And we talked about operational security and why it is so important and why it is so hard to do operational security. 385 00:43:45,400 --> 00:43:54,360 Again, some more points to notice, it's obviously threat hunting on the dark web is hard, but it's worth the effort. 386 00:43:54,360 --> 00:43:58,020 You don't get intelligence that you get from the dark web. 387 00:43:58,580 --> 00:44:01,480 You should always keep operational security in mind. 388 00:44:02,640 --> 00:44:07,180 Like I said before, you should look on more than one resource. 389 00:44:07,180 --> 00:44:08,340 You should look on forums. 390 00:44:08,340 --> 00:44:14,900 You could look on clear web forums also like pastebin and telegram as an example. 391 00:44:15,300 --> 00:44:18,460 And you should look on different other forums on the dark web also. 392 00:44:18,800 --> 00:44:21,780 It takes a lot of resources and a team effort. 393 00:44:21,780 --> 00:44:24,360 You can't do all these things alone. 394 00:44:24,360 --> 00:44:26,900 You should have a team for this. 395 00:44:26,900 --> 00:44:38,220 And we talked a little about usage of MITRE attack framework and how it can be useful to map your TTPs to that. 396 00:44:39,880 --> 00:44:49,700 These are some of the resources that I suggest you to read if you want to know more about the dark web stuff or dark web hunting stuff. 397 00:44:50,040 --> 00:44:56,360 The majority, the major ones are like a recorded feature insight, CrowdStrike, digital shadows. 398 00:44:56,560 --> 00:44:59,920 They release their blogs or white papers regularly. 399 00:44:59,920 --> 00:45:07,140 So read them and you'll understand what all things are there on the dark web. 400 00:45:10,100 --> 00:45:12,800 So yeah, that was it. 401 00:45:12,800 --> 00:45:14,480 I think, thank you so much. 402 00:45:14,480 --> 00:45:24,900 I hope you all liked my presentation and you can contact me on Twitter or LinkedIn if you have any doubt or if you want to discuss about this stuff more. 403 00:45:24,900 --> 00:45:29,040 I will hang on in discord to answer all your questions. 404 00:45:29,040 --> 00:45:30,920 So yeah, thank you.