[00:00.000 --> 00:03.800] Okay, so the next speaker is Saifullah, who is going to be speaking about [00:03.800 --> 00:09.080] mastering open source intelligence, advanced techniques in the realm of big data. [00:09.080 --> 00:11.860] Oh, I'm reading, you can see on the screen already. [00:12.720 --> 00:18.940] About him, he's a founder for SwordSec, identify vulnerabilities in Apple, Twitter, so on, [00:18.940 --> 00:24.340] has delivered many talks in universities, teachers at conferences and whatnot. [00:24.340 --> 00:28.740] He's one of the most amazing person I know, so I would give him the stage. [00:28.740 --> 00:34.360] And let's give him a round of applause. I know you are... let's give it. [00:41.440 --> 00:51.080] Thank you. Hello again. Our topic is OSINT, you know, and I will show something advancing [00:51.080 --> 01:01.440] techniques in the big data. Our agenda is, I will talk about advanced techniques in OSINT, [01:01.440 --> 01:10.700] and big data, and its importance in open source intelligence. Google BigQuery for OSINT, [01:10.700 --> 01:16.120] Common Crowd, and advanced techniques case studies today. [01:20.400 --> 01:27.420] My name is Saifullah. I am software engineers who love cybersecurity. I have been researching [01:27.420 --> 01:34.620] cybersecurity since 2007. Found security problems, big companies like Google, Apple, [01:34.620 --> 01:42.300] Twitter, and my name is on the Google Hall of Fame page 10 years ago, and found a big issue that [01:42.300 --> 01:49.120] crashed on Apple iPhones from afar, and wrote about cryptocurrencies, mining devices, [01:49.120 --> 01:55.440] vulnerability, and science education talk about my article, and set up cybersecurity [01:55.440 --> 02:01.420] and intelligence talks at more than 20 universities and conferences in the world. [02:01.420 --> 02:09.660] Yes, we will talk about first the OSINT. Open source intelligence is the art of collecting [02:09.660 --> 02:17.460] information from publicly available source. This purpose spans both offensive strategies like [02:17.460 --> 02:24.180] identifying vulnerabilities, and defensive ones, such as secret assets, digital assets. [02:24.180 --> 02:31.660] It also goes beyond mere gathering. It involves analyzing data to understand potential threats [02:31.660 --> 02:38.300] or attacker patterns. In today's digital age, where information is abundant, [02:38.300 --> 02:42.840] mastering OSINT is crucial for a proactive approach to cybersecurity. [02:44.640 --> 02:53.340] Big data. Big data is also all about managing large amounts of information. It is marked by [02:53.340 --> 03:02.820] variety, velocity, volume, verification, and value. This means variety of different types of data, [03:02.820 --> 03:11.300] and velocity, speed, at which data comes in. Volume, amount of data, not pricing. Verification, [03:11.300 --> 03:19.760] making sure data is correct, and value is the usefulness of the data. Understanding these [03:19.760 --> 03:29.200] aspects helps in handling waste data effectively. And big data comes from various sources. We have [03:30.120 --> 03:38.320] most of sources, as you see, like the social media, GitHub, where code is shared, open web [03:38.320 --> 03:49.740] source, Google BigQuery public data set, public data stores like AWS or S3, and archive documents. [03:49.740 --> 03:59.380] These are critical points that will show something. And these sources offer a wealth of information that can be tapped into. [04:00.760 --> 04:08.200] Big data operates in a systematic way. Data collecting, firstly. After that, store and manage, [04:08.200 --> 04:16.380] and analyze, and deep learning to make sense of data. Finally, visualize. We present the data [04:16.380 --> 04:23.840] in an understandable way. This process helps us turn vast amounts of raw data into valuable insights. [04:26.580 --> 04:31.980] You can see, maybe you know these technologies. In this slide, we glance at some key tools [04:31.980 --> 04:38.620] and technologies that employ big data, from Hadoop and Spark for data storage and processing, [04:39.160 --> 04:45.220] to databases like Cassandra and MongoDB for scalable storage. These tools are vital in [04:45.220 --> 04:51.120] the OSINT landscape, enabling us to handle vast amounts of data efficiently. [04:54.080 --> 05:01.340] And here, we briefly highlight key big data tools and technologies, ranging from data warehousing, [05:02.000 --> 05:07.400] solutions like Google BigQuery, you know, and Snowflake to machine learning frameworks [05:07.400 --> 05:13.500] like TensorFlow. These are essential building blocks in analyzing vast data sets within the [05:13.500 --> 05:21.240] context of OSINT. In this slide, in the picture, we are exploring various components that interplay [05:21.240 --> 05:28.360] in the safety landscape. Big data analytics, utilizing vast amounts of data for safety insights. [05:28.860 --> 05:34.460] After, for example, maybe you can use it, network flows and anomalies detection. [05:34.460 --> 05:39.080] You can monitor network behavior and detected irregularities. [05:39.080 --> 05:49.000] And intelligence feeds and DNS captures, it is very critical points to gather intelligence or from [05:49.000 --> 05:57.080] public data set. And domain name system activities, you can monitor these domain name system activities [05:57.080 --> 06:06.320] and process data. These mean analyzing procedures and workflows. And very important things, [06:06.320 --> 06:10.880] web page text, crawling any web page from maze of data. [06:13.620 --> 06:20.340] Collection one, leaked password. Maybe you know and downloaded it. Clear text password [06:20.340 --> 06:26.120] leak represents a serious security concern. Throughout OSINT and big data analytics, [06:26.120 --> 06:33.600] we can track and analyze these leaks. Here's how regularly scan publicly database forums, [06:33.600 --> 06:41.380] dark net, or pass size for leaked credential. And use big data techniques to identify patterns [06:41.380 --> 06:49.500] such as common password or targeted organization. Or alert affected users or companies and contribute [06:49.500 --> 06:56.500] to threat intelligence. And educate on sacred project or practice, like using strong and unique [06:56.500 --> 07:04.940] passwords. This approach turn rave data leaks into actionable insights and change both individual [07:05.960 --> 07:16.480] or organizational security. Cisco umbrella top million domains, open source intelligence and [07:16.480 --> 07:23.240] big data can be leveraged for various analysis. For example, threat analysis. By monitoring these [07:23.240 --> 07:31.080] domains or these links, we can identify emerging threats like popular TLD, FQDN or domain like that [07:31.080 --> 07:39.400] or seasonal domain registration. And secret assessments by cross-referencing with known [07:39.400 --> 07:46.980] malicious domain or FQDN or link, we can pinpoint domains that may be part of phishing or malware [07:46.980 --> 07:54.320] campaigns. Competitive intelligence, analyzing the web presence and popularity of competitor [07:54.320 --> 08:01.480] domains within the top list or top one million domain list. The rich data set offers a starting [08:01.480 --> 08:10.260] point for diverse research from marketing insights to cyber security. Tranquil is another valuable [08:10.260 --> 08:19.420] resource to crawl domain list. This link is similar to Cisco's umbrella link but with distinct [08:19.420 --> 08:26.380] features. While both provide domain list, Tranquil combines multiple data source offering a more [08:26.380 --> 08:36.120] diversified view. And it is designed to minimize biases and provide a robust ranking of domains. [08:36.120 --> 08:41.000] Whether you are searching for market trends or analyzing some hackers or web popularity [08:41.000 --> 08:50.240] assessing security threats, Tranquil offers a unique perspective. At the meantime, there are [08:50.240 --> 08:56.860] over four million main domains in Tranquil and these updates every month or every week. Cloud [08:56.860 --> 09:05.000] Flare Radar offers a top one million domains list like that, but it is a valuable tool for tracking [09:05.000 --> 09:11.100] web trends and identifying popular technology or domains. Whether you are a market researcher or [09:11.100 --> 09:18.880] security professional, Cloud Flare Radar provides a snapshot of the internet's purpose. [09:20.080 --> 09:31.120] The darknet market archives show us hidden online markets from 2013 to 2015. These markets [09:31.120 --> 09:38.740] often sell illegal things and use secret websites. With open source intelligence and big data, [09:38.740 --> 09:45.280] we can study these markets, see what they sell, and learn how they work. [09:45.300 --> 09:52.440] This information helps people like police officers or defensive intelligence and researchers [09:52.440 --> 09:59.580] understand what is happening in the hidden parts of internet or underground internet. [09:59.580 --> 10:08.890] There is public data that you can download. The size of the file is over 1.6 terabytes. [10:09.740 --> 10:20.820] This is old data set. Maybe you can search, you will discover new data set in the market archives. [10:21.380 --> 10:31.080] GrabApp is a search tool that lets you explore open source code from GitHub repository. You can [10:31.080 --> 10:37.700] use regular expression. It's like a search engine for code, helping you find specific lines or [10:37.700 --> 10:44.680] patterns. In the world of OSINT and big data, GrabApp can be used to study how code is written [10:45.260 --> 10:53.000] or find common mistakes or even uncover sensitive information accidentally left in public code. [10:53.000 --> 10:59.560] Lesson two for both researchers and security expert. And I have written some simple [11:00.140 --> 11:10.040] regular expression tool and there are lots of credential AWS secret key. [11:11.640 --> 11:19.280] Google BigQuery and GitHub. Google BigQuery allows you to use regular expression tool [11:20.160 --> 11:26.820] to search engine and analyze large of data sets. In the context of OSINT and big data, [11:26.820 --> 11:31.860] this means you can quickly find patterns, any patterns, and match specific content [11:32.520 --> 11:40.020] or filter data in complex ways. As you can see, you can search any company naming GitHub data set. [11:40.960 --> 11:46.240] There are lots of data set in the Google BigQuery, like Stack Overflow, New York [11:46.240 --> 11:54.900] Fire Department, or the GitHub repository like that. It is a powerful tool for digging deep [11:54.900 --> 12:00.380] into data and uncovering insight that can be used for everything from market research to [12:01.000 --> 12:10.020] cybersecurity analysis. I have written simple regex tool to find company name in GitHub repository. [12:10.840 --> 12:22.980] You can see maybe there are over two terabyte data set. And after that you can search top [12:22.980 --> 12:29.020] question popular tech from programming language in Stack Overflow or GitHub repository. [12:31.220 --> 12:38.760] JavaScript most popular language in this data set. Question, answering, and repository. [12:38.760 --> 12:48.480] I have written simple SQL query. Also, you can search some private keys in GitHub repository. [12:49.220 --> 12:59.400] Maybe you watched in Live Overflow on YouTube. Live Overflow search Bitcoin private keys [12:59.400 --> 13:11.020] like these methods. And also you can search some private keys. As I mentioned, you will find some [13:11.020 --> 13:24.200] things like private keys or something. You can search anything. If you have [13:25.160 --> 13:31.400] some things or if you are using regular expression well. [13:31.840 --> 13:37.980] There are approximately over two terabyte data set for GitHub repository. [13:39.560 --> 13:49.760] But in SQL query, I have joined two data sets on GitHub. One of files, other content of GitHub [13:49.760 --> 13:58.900] repository. Paspin monitor. You know Paspin. But Paspin is not popular today. But you can monitor [13:58.900 --> 14:07.840] Pasp website now. Paspmon is a tool that monitors Paspin sites for specific content. In the real of [14:07.840 --> 14:14.180] OSINT and big data, it is valuable asset for tracking information leaks and finding exposed [14:14.180 --> 14:22.020] sensitive data or understanding trends in public data set or code sharing. It turns the [14:22.020 --> 14:28.780] vast sea of Paspin data into actionable insights for secret and market research. [14:30.040 --> 14:35.020] And also you can use regular expression too in the config file. [14:36.600 --> 14:47.340] Social media. It is important slide. My friend Paspin helped me this project. Thank you. You [14:47.340 --> 14:54.440] must have seen direct message like this as well from Twitter or Telegram or Discord channel. [14:55.160 --> 15:02.520] You can see. Some scammers sends to you direct message like that. So how can we catch these [15:02.520 --> 15:11.000] attackers or collect information from them? Let's start. Firstly, create some accounts on [15:11.000 --> 15:20.160] lots of social media accounts or like that, Discord, Telegram that you want to use as honeypot. [15:21.000 --> 15:29.860] After that, we will change privacy settings. After, follow these tweets and join some channels [15:29.860 --> 15:40.500] and group and username in Telegram, Discord and Twitter. You must select a low direct message [15:40.500 --> 15:52.020] from any members. Firstly, because of lots of hackers can see us. After that, you must select [15:52.260 --> 15:59.720] a low message from everyone on Twitter. And last one, you must toggle off of active and mute on [15:59.860 --> 16:11.160] Telegram. Let's crowd some hackers, scammers, sorry. First, to attract the attention of scammers, [16:11.160 --> 16:19.260] you can like retweet post from NFT and crypto groups only channels and follow those accounts [16:19.260 --> 16:25.500] using newly created Twitter, Discord and Telegram accounts to attract attention of scammers. [16:26.000 --> 16:35.900] And reload direct message from simple API services. You can analyze all content of messages from [16:35.900 --> 16:44.700] Twitter, Discord channel and Telegram message. You can analyze website title and you can analyze DNS [16:45.960 --> 16:55.620] services, sorry, DNS type of websites or FQDM and you can check crypto type and you can capture [16:55.620 --> 17:00.920] username and username ID to check another services. [17:03.040 --> 17:10.720] And Common Crowd project, open web story, web crowd data. I love it. Let's now take a look at [17:10.720 --> 17:18.160] Common Crowd. A valuable resource for anyone working with big data. Big companies like Facebook, [17:18.160 --> 17:27.520] OpenAI, Google, Tesla uses Common Crowd project. I have worked Common Crowd since [17:28.320 --> 17:35.780] five years ago. Yeah, five years ago. And we have worked lots of project in Common Crowd. [17:35.780 --> 17:42.660] And let's show next slide. And whether you are studying trends, doing market research [17:42.660 --> 17:48.480] or investing in cybersecurity threats, Common Crowd provides the data you need. [17:48.480 --> 17:56.340] Let's explore what we can do with it. In this screenshot of the May-June 2023 index, [17:56.340 --> 18:06.240] you see different types of data files. You can see VARC, VAT, VIT, Robotics and URL, etc. [18:06.240 --> 18:12.640] These files represent a rich collection of web data like webarchives.com. [18:13.240 --> 18:22.380] In terms of open source intelligence and big data, they provide the raw material for deep analysis. [18:22.380 --> 18:28.220] Whether it is tracking online trends, investigating cyber threats, or tracking [18:28.640 --> 18:35.900] some hackers or scammers, or conducting market research, these data opens up endless possibilities [18:35.900 --> 18:40.860] for understanding and navigating the digital landscape and world. [18:41.580 --> 18:46.700] Here we see VARC, VAT and VET files, key components in the web archiving. [18:47.440 --> 18:54.480] VARC captures the full request and response cycle. VAT contains metadata summarized [18:55.040 --> 19:03.980] in the center of VAT response file. And VET offers, right off the screen, plain text content. [19:04.540 --> 19:09.760] Together, they provide a comprehensive snapshot of web data for various analysis. [19:11.960 --> 19:17.600] We crawl VARC files as you see in the example. These are VARC links. A link has over one gigabit. [19:18.120 --> 19:23.590] And these files have 80,000 links. There are petabytes of datasets here to analyze. [19:24.620 --> 19:33.080] In this slide, we see some exciting examples of what can be done with Common Crawl. [19:33.080 --> 19:40.600] Imagine accessing 10 terabytes of metadata, 26 million domains, and 1.4 million SSL for just $10. [19:41.460 --> 19:47.260] This is really interesting. From analyzing pieces of web pages to exploring robots takes the [19:47.260 --> 19:54.280] files, Common Crawl offers a vast playground for researchers and analysts, cybersecurity analysts. [19:54.280 --> 19:59.840] It is an affordable and powerful resource for anyone looking to crawl into web data, public [19:59.840 --> 20:07.360] data. And after that, we explore the intriguing question. What is the most used JavaScript [20:07.360 --> 20:14.120] library? The public dataset. Billions and billions of links. By utilizing Apache Spark, [20:14.120 --> 20:19.440] we have analyzed all the script text across a vast collection of web pages. [20:21.200 --> 20:27.960] And you can crawl the email addresses, subdomain, and IP addresses that you want as in this example. [20:29.840 --> 20:35.580] We delve into the world of handling petabytes of website data using Apache Spark, maybe you know, [20:35.580 --> 20:43.460] and EMR. As detailed in the reference article, I like Mark, the author demonstrates how to process [20:43.460 --> 20:50.300] massive amounts of web data, such as Common Crawl dataset. To extract valuable insights [20:50.300 --> 20:56.120] from a big data perspective, this approach showcases the ability to analyze web trends, [20:56.120 --> 21:01.360] identify popular technology, like front-end or back-end technology. You can see how many web [21:01.360 --> 21:08.320] servers are in the Common Crawl dataset in this example. Passive search. We are highlighting the [21:08.320 --> 21:16.520] massive scope of data we have analyzed using Common Crawl December 2018. We dive into the [21:16.520 --> 21:21.360] technical aspect of our analysis utilizing Google Compute Engine. I leveraged Docker and Docker [21:21.360 --> 21:28.740] to create a robust and scalable environment. With 32 vCPUs and 280 gigabytes of RAM spread [21:28.740 --> 21:33.580] across 16 instances, I was well-equipped to handle a massive dataset. [21:34.900 --> 21:40.900] Last finished VCS script I used to look at a specific dataset. It is a simple script. [21:41.000 --> 21:47.020] And it checks for CoinHive in the URLs and gathers important details like the full web address. [21:47.020 --> 21:54.560] As you know, CoinHive was a crypto-miner JavaScript website, but now the domain [21:54.560 --> 22:03.760] redirects to Troy Hunt's blog. Troy Hunt registered this domain. In our final slide, [22:03.760 --> 22:10.380] we present the culmination of our analysis, a compressive list of URLs containing the CoinHive [22:10.380 --> 22:17.000] JavaScript by utilizing the power of Common Crawl dataset and the techniques we have explored. [22:17.020 --> 22:23.040] It is a concrete example of how OSINT and big data can be harnessed to pinpoint [22:23.040 --> 22:34.820] targeted information within the vast landscape of the web. Yes, you can find sub-domains of [22:34.820 --> 22:40.180] target websites, which is an important step in gathering information from Common Crawl. [22:40.180 --> 22:45.920] And we can also find out what technologies a website is using, like the server type, [22:45.920 --> 22:52.320] or what JavaScript deliveries they have. This shows how we can use big data to learn more about [22:52.320 --> 22:58.280] the target. And we explore another use of Common Crawl for open source intelligence, [22:58.280 --> 23:04.590] finding hidden directories and files, like robot.txt, slash admin, or config.json. [23:05.020 --> 23:08.720] This is a practical example of how big data can help us [23:08.720 --> 23:14.940] dig deeper and uncover hidden details on a target site or target domain. [23:16.360 --> 23:19.640] And you can check for data leaks. [23:21.280 --> 23:27.740] Internet search engine, you can, now that Census, Shodan, Zumai, Secret Service, Criminal IP, [23:27.740 --> 23:35.340] there are lots of search engines like that. And GreyNoise catches these websites and companies [23:35.340 --> 23:41.860] to protect your website or company. With GreyNoise, we can identify and block IP addresses [23:42.490 --> 23:49.680] of these actors. And in this screenshot, we are taking a closer look at GreyNoise capabilities, [23:49.680 --> 23:58.320] specifically targeting this Census actor. And you can block them. Thank you for listening. [23:59.300 --> 24:05.660] If you have any questions, let's meet right here. Thank you.