WEBVTT 00:03.150 --> 00:15.350 Hi, I'm Leo, the founder and CEO over at Graphistry, and I'm going to be talking today about Project Domino, which is not even at Graphistry, but a community open citizen data science effort against COVID misinformation. 00:15.870 --> 00:26.190 And before we go into it, I do want to say thank you to the DEF CON staff, AI Village staff, and the Project Domino volunteers, and also AI Village in particular, Sven. 00:26.330 --> 00:32.890 This is a crazy time for all of us, and so to be able to put this together, a big thank you for that. 00:32.890 --> 00:35.410 I'm honored to be giving this as a keynote. 00:35.650 --> 00:49.070 I'm hoping folks here will benefit from hearing what we've been learning, and also if you are impacted somehow by COVID and want a way to give back, this might actually be an opportunity for you as well. 00:49.090 --> 00:53.430 With that, I want to start with an example. 00:53.570 --> 00:54.790 This is about three months ago. 00:54.790 --> 00:57.930 It might feel like three years for some of us. 00:58.330 --> 01:05.130 And around the time when the Seattle breakout was happening, our friends were kind of dealing with the hospital situations over there. 01:05.570 --> 01:10.030 I think it was already like people were actually dying already in Seattle. 01:10.870 --> 01:19.390 Governor Kevin Stitt from Oklahoma posted this strange tweet, eating with my kids and all my fellow Oklahomans at the collective OKC. 01:19.390 --> 01:22.950 It's packed tonight, support local, Oklahoma proud, basically. 01:22.950 --> 01:28.890 And then he clarified next day in a press release, we encourage Oklahomans to go out to restaurants. 01:29.130 --> 01:33.230 So for us, with ties to the medical system, our head kind of was exploding. 01:33.230 --> 01:35.850 Like, what's up? 01:36.150 --> 01:47.070 Fast forward to a couple weeks ago, and unfortunately for Governor Kevin Stitt and his family, he tested positive for COVID. 01:47.070 --> 01:57.070 And unfortunately for his constituents who are relying on him, they have about a thousand per day infections at this point, because kind of what we feared would happen is exactly what happens. 01:57.190 --> 01:59.550 And now Oklahomans are dying. 01:59.810 --> 02:06.250 So it's... I do want to do kind of some disclosures and stuff in a bit. 02:06.250 --> 02:08.170 But to us, it's not about a left or right thing. 02:08.170 --> 02:11.790 It's like we're, you know, this is a medical health emergency. 02:12.130 --> 02:13.410 We need to do stuff. 02:14.110 --> 02:18.730 And that's kind of a good note, kind of where this project came from. 02:19.210 --> 02:24.750 We're kind of looking at kind of what can we do. 02:24.930 --> 02:34.410 And there's this interesting statement by the... from the World Health Organization saying, we're not just fighting an epidemic, we're fighting an infodemic. 02:34.650 --> 02:50.230 And kind of the reasoning that we kind of believe in here is until there's a medical cure, a pill, a vaccine, been tested, it's safe, kind of the only thing we really have is behavior change. 02:50.230 --> 03:02.590 So for that, unfortunately, you either need to get top-down policy enforcement or bottom-up kind of citizen voluntary compliance. 03:02.590 --> 03:05.490 And unfortunately, we have folks standing in the way of that. 03:05.490 --> 03:14.450 And, you know, for Seattle, for Oklahoma, for all of us at home, we're seeing kind of the effects of failures to win that fight. 03:15.390 --> 03:17.390 Some quick disclosures here. 03:17.390 --> 03:19.410 We are going to talk about real people. 03:19.690 --> 03:22.350 We've published... actually, we haven't directly published. 03:22.350 --> 03:23.730 We publish through partners. 03:23.730 --> 03:26.730 So we empower researchers to do through our tech. 03:26.730 --> 03:29.150 So they've published on some of this stuff before. 03:29.150 --> 03:36.390 And based on some of the results today, I expect to see some accounts going out and hopefully some takedowns. 03:37.230 --> 03:40.430 Likewise, this is a volunteer organization. 03:40.430 --> 03:48.590 So the views expressed by us, me, this talk, they're not necessarily representative of the employers of the volunteers. 03:48.950 --> 03:53.010 Likewise, you should expect some bias here. 03:53.010 --> 03:59.010 So a lot of us, actually, some people on the team were even affected by COVID and didn't have their lives impacted by that. 03:59.010 --> 04:00.030 Family members do. 04:00.030 --> 04:02.290 Their work and work life has been impacted. 04:02.410 --> 04:04.850 Misinformation is a broader topic. 04:04.850 --> 04:11.610 I'm personally in a place some folks may remember called the Soviet Union, where propaganda was a thing and surveillance was a thing. 04:11.610 --> 04:13.650 So conflicting thoughts on that. 04:14.270 --> 04:20.690 But I do want to point out from the team, I do especially want to call out some of the kind of folks on the leadership team. 04:20.690 --> 04:24.010 Sean Griffin's been involved in pandemic response. 04:24.010 --> 04:28.730 Actually, he helped both Obama and the Trump administration with pandemic planning. 04:29.490 --> 04:35.230 Anita was over at the NSF and looking at a lot of digital crime, blockchain, stuff like that. 04:35.230 --> 04:36.870 She's at UIUC now. 04:36.870 --> 04:44.410 Cody Webb has been this research powerhouse for our team and researching misinformation forever. 04:44.610 --> 04:49.410 I shared the names of some of the engineers helping on our backbone. 04:50.650 --> 04:59.150 The first example today will be actually Jeff Goldberg over at Social Forensics and David Knickerbocker, an independent researcher, has been helping us a bunch on. 04:59.750 --> 05:08.050 And I think, oh, yeah, and then tomorrow, if this is interesting to you, kind of the same type of stuff we're doing for COVID. 05:08.830 --> 05:20.070 Andrea's going to be talking over at the Biohacking Village about imagine you're a cancer patient and you go to Facebook and then you see all the same misinformation giving you false hope or misguiding you. 05:20.330 --> 05:21.950 So it's important stuff. 05:23.150 --> 05:23.790 Okay. 05:23.790 --> 05:26.710 So today, I'm going to talk about basically three things. 05:26.850 --> 05:32.750 First, I just want to take you down the rabbit hole through an example from anti-vaccination misinformation. 05:33.190 --> 05:42.130 Second, I want to then give you more of a security data science perspective and how we're actually going about it, trying to figure out scale and autoresponse and all that stuff. 05:42.150 --> 05:53.130 And then briefly, but incredibly important, why we're doing this as Open Citizen Data Science and why I'm actually pretty hopeful about where this is going long term. 05:53.130 --> 05:58.890 And maybe this could kind of give you a chance or give you ways to help out the community. 06:00.390 --> 06:01.510 Okay. 06:01.610 --> 06:03.530 So let's dive in. 06:04.010 --> 06:13.810 So we were doing an investigation and this name popped up, Sherry Tenpenny, and we decided to see where we go for the purposes of this talk. 06:14.830 --> 06:19.670 And for some folks, if you're from Australia, this may be a familiar name. 06:20.530 --> 06:24.490 She's Wikipedia famous or infamous at this point. 06:24.490 --> 06:39.330 So her 2015 lecture was canceled in Australia because apparently she is involved in a lot of vaccine misinformation and private organizations in Australia wanted no part of that. 06:39.370 --> 06:43.210 And so they didn't support her. 06:43.510 --> 06:52.680 For our purpose and purposes, I do want to point out there's kind of an ecosystem with different incentives here. 06:52.680 --> 06:54.780 And so if we look at somebody like Dr. 06:54.780 --> 07:00.480 Sherry, we see that she does teach courses, like hence the book tour. 07:00.480 --> 07:01.420 She sells books. 07:01.420 --> 07:02.900 She trains folks. 07:02.900 --> 07:06.740 She has about 50, 60,000 followers on Twitter, not shown. 07:06.740 --> 07:10.860 She also has a big presence on Facebook and also on newsletters. 07:10.860 --> 07:12.880 So stuff that you don't even see. 07:13.100 --> 07:14.380 And she gets paid to do it. 07:14.380 --> 07:18.520 Now it's kind of like her beliefs and her livelihood are the same thing. 07:18.520 --> 07:20.240 So she's very invested in this. 07:22.660 --> 07:28.420 But once we start looking at it, there are kind of a few things that looks a little strange. 07:28.740 --> 07:37.700 So she talks about Nazis in her profile, like that's the first thing, that's the first word she shows you. 07:38.420 --> 07:42.520 And so that kind of is kind of like a whistle to one side of the house. 07:42.660 --> 07:44.540 And then talks about abuses by the state. 07:44.540 --> 07:47.160 So that's kind of like a different kind of thing. 07:47.160 --> 07:52.000 And so it's kind of interesting to ask, like, who is the community around her? 07:52.180 --> 07:53.620 So let's take a look. 07:54.660 --> 07:59.280 So what I did here is loaded in busy Dr. 07:59.280 --> 07:59.840 T. 08:00.500 --> 08:11.680 And so we're looking at her, and what we see in colors around her are basically, she's only following a couple hundred people, but she has 50,000 people following her. 08:12.020 --> 08:26.680 And so when we look at a sample of that, what we see up top is a bunch of folks, essentially public, basically anti-vax, natural medicine, a lot of kind of medically-minded folks that you'd sort of expect to be here. 08:26.680 --> 08:30.440 But then the surprise is once we start seeing the other colors. 08:30.440 --> 08:38.720 And so the tools here just kind of automatically colored and laid it out and everything, like it's kind of just traditional graph analytics. 08:38.920 --> 08:44.360 So we see Donald Trump, we see Michelle Malkin, we see Candace. 08:45.900 --> 08:51.800 Basically, like in the dark green, we're seeing these public figures, kind of national conservative public figures. 08:52.640 --> 09:05.180 On the right, we're seeing, but a bit on the right, and then actually, especially down here, we're seeing a mixture of QAnon, which is a conspiracy group we're hearing about a lot nowadays. 09:05.600 --> 09:09.700 We're seeing a lot of MAGA and Trump supporters. 09:09.700 --> 09:12.200 So not super surprising given we had Trump. 09:13.120 --> 09:17.940 And so it's kind of interesting that we have on the top, we have like, who do you expect with anti-vax? 09:17.940 --> 09:21.040 But on the bottom, we have like the whole MAGA thing. 09:21.040 --> 09:23.340 We have conspiracy theorists from QAnon. 09:23.340 --> 09:26.860 They're actually even like UFO stuff going on, on the bottom. 09:27.080 --> 09:30.060 It's kind of like interesting to see that mix. 09:30.660 --> 09:32.840 So I want to point out another phenomenon here. 09:33.260 --> 09:37.780 When we're looking at each community, like the question is like, how are they interacting? 09:37.780 --> 09:39.880 Is it like they all really love our Dr. 09:39.880 --> 09:40.480 Busy T? 09:40.480 --> 09:42.420 Like she just wrote this wonderful book. 09:42.880 --> 09:49.220 What we actually see is like, for example, from the QAnon zone, they're reaching into the medical zone. 09:49.220 --> 09:51.080 So they're following folks there. 09:51.140 --> 10:00.420 Conversely, if we pick somebody big in the kind of this top region, sometimes it's localized, but other times you see it's reaching into the QAnon zone. 10:00.820 --> 10:05.560 Kind of no matter which cluster I do, you're going to see those reach outs. 10:07.120 --> 10:07.900 Excuse me. 10:07.900 --> 10:17.100 So that's actually kind of strange, where there's no reason to believe one conspiracy has anything to do with another. 10:17.640 --> 10:20.640 And so we're actually kind of almost like in this other world. 10:21.520 --> 10:25.700 And what we're seeing is the networks do cross over. 10:26.700 --> 10:30.500 And so the first thing is what we're seeing is the question is why? 10:30.500 --> 10:33.360 And that's basically to support influence. 10:34.460 --> 10:37.800 Sometimes that could be direct where you're just retweeting from each other's. 10:37.800 --> 10:40.140 Other times it might be to trick the Twitter algorithm. 10:40.140 --> 10:42.960 So for example, here is actually a Fox News reporter. 10:42.960 --> 10:44.460 So it's someone on TV. 10:45.560 --> 10:53.460 They have 40,000 followers, which is reasonable for a personality, but they're following 30,000 people back. 10:53.460 --> 10:58.780 So that probably suggests an inauthentic and automatic action of following back somehow. 10:59.040 --> 11:04.580 We don't know what order they happened in, but in general, that's prohibited by Twitter's terms of service. 11:04.580 --> 11:06.740 But they get clicks. 11:06.740 --> 11:11.460 And so even though it violates terms of service, there they are. 11:11.780 --> 11:15.940 And that's basically the welcome to the misinformation network. 11:17.360 --> 11:22.080 And I realized that I glossed over this, but the reason we care about Dr. 11:22.080 --> 11:34.440 T and that she's talking about this stuff is that she and her community are already pushing messages against a COVID vaccine, even though one hasn't even passed trials yet. 11:34.440 --> 11:37.800 She's against it before it's even been tested, just by principle. 11:38.140 --> 11:41.600 And so Twitter's been supporting her in doing that, essentially. 11:41.600 --> 11:43.060 And I'll get to that in a second. 11:44.380 --> 11:50.520 So I do want to take one more, I want to go a little bit deeper into our Dr. 11:50.520 --> 11:50.980 T. 11:51.460 --> 11:55.220 And so in particular, I want to understand the communities, like what they're doing. 11:55.280 --> 11:59.980 So here what we're going to do, and I want to see if there's kind of extra coordination structure. 11:59.980 --> 12:02.400 And so we're getting a bit more on the data science side. 12:02.460 --> 12:11.600 So what's going on here is we took the accounts, we actually took a look at about 100 million COVID-related tweets to see if there's kind of organized behavior within it. 12:11.780 --> 12:23.260 For each account, we just tried to look at for known sort of misinformation topics, what's sort of the signature, like, you know, how many times did it like activate on different topics. 12:23.260 --> 12:28.960 And for each account, we now have that sort of like misinformation fingerprint, essentially. 12:29.000 --> 12:31.800 And then we ran something called UMAP. 12:31.800 --> 12:36.400 I think it's a universal manifold approximation or something like that. 12:36.400 --> 12:40.480 But basically tries to smooth out and assume structure behind that. 12:40.760 --> 12:42.460 And then we took a look. 12:42.460 --> 12:44.840 So let's try it out. 12:45.620 --> 12:50.860 So yeah, so what's going on here is we're taking a look at those accounts. 12:50.860 --> 13:01.440 And we only are showing from all the accounts, we're looking at the 5000 or so that we actually saw active across multiple misinformation topics. 13:01.440 --> 13:08.960 And so these are kind of the really the ones that have kind of that behavior going across topics like are just like what's going on. 13:11.020 --> 13:13.080 And we ran UMAP. 13:13.080 --> 13:16.320 But then what we also did, you're seeing here, I'm drawing these little edges. 13:16.320 --> 13:20.760 We also ran a afterwards k-nearest neighbor and put them on screen. 13:20.940 --> 13:27.820 So that whenever we see accounts, kind of tweeting on similar topics and similar intensities. 13:28.060 --> 13:36.240 So it's really, for example, imagine two bots program to kind of talk about the same thing, we're controlled by the same person to kind of, you know, retweet someone else. 13:36.940 --> 13:38.900 That's when we're going to draw that edge. 13:39.180 --> 13:45.880 And so again, they're not necessarily following, they're actually just on certain topics, they're actually pushing those topics in the same way. 13:45.880 --> 13:47.860 So it's entirely content based. 13:48.400 --> 13:54.420 And what we're seeing here is some interesting structure pops out for across COVID Twitter. 13:55.080 --> 14:03.520 And then the second thing I did here is coloring, you're seeing some of these are a bit pinkish. 14:04.340 --> 14:06.620 So if we jump, let's take a look at this one. 14:07.000 --> 14:12.000 So for this kind of group of activity, we're seeing a lot of this q&a and stuff. 14:12.000 --> 14:18.440 And so what I'm doing here is if in those tweets talking about COVID, they also talk about q&a. 14:18.720 --> 14:22.480 And we have a couple words on that, we call that out too. 14:22.960 --> 14:28.340 And so what we're seeing is like, you know, for this general topic, a bunch of accounts never explicitly mentioned q&a. 14:28.340 --> 14:34.400 But it's definitely maybe it's like two coordinators, or who knows, or like something like that. 14:34.400 --> 14:40.500 But we're actually seeing a lot of q&a influence across these different organized groups. 14:40.940 --> 14:44.460 And actually, in particular, almost all of these have a bit of q&a. 14:44.720 --> 14:47.980 So you're seeing it's not all white, it's all a little off white. 14:48.000 --> 14:52.220 And then what's more, you're seeing some of them are you're seeing these pockets of heaviness. 14:52.300 --> 14:57.960 And we calculated out about 15% of these accounts spread throughout our q&a. 14:57.960 --> 15:02.020 And so at least for COVID Twitter, that was actually kind of interesting. 15:02.020 --> 15:03.520 And for us, that was a surprise. 15:03.800 --> 15:08.980 So that kind of put q&a on in terms of medical misinformation on our radar. 15:09.120 --> 15:10.600 So that's not good. 15:12.920 --> 15:15.600 Okay, so jumping ahead. 15:17.260 --> 15:20.460 So okay, so we're looking at our busy Dr. 15:20.460 --> 15:21.160 T. 15:22.100 --> 15:26.500 And clearly blatant activity going on here. 15:26.500 --> 15:38.780 The last thing I want to point out was, then we look on the right here, and Twitter is saying, hey, you should probably now that you're here, you should also check out this Judy Mikovits. 15:38.980 --> 15:40.820 This PhD sounds authentic, right? 15:40.820 --> 15:41.800 Sounds smart. 15:42.460 --> 15:53.120 You may have also known about this Judy as a video going around called Plandemic, which, you know, got pulled down as pretty bad misinformation. 15:53.860 --> 15:58.040 And so that was pretty shocking to see, like, I was like, oh, that name sounds familiar. 15:58.040 --> 15:59.360 So that was a little crazy. 15:59.360 --> 16:04.540 That was like the number one recommendation of Twitter as soon as you go to this web page is to learn about Plandemic. 16:04.880 --> 16:07.960 So and this is like, you know, way after the fact. 16:08.440 --> 16:25.880 And so kind of in summary, what we're seeing in this, in this basic, pretty basic, honestly, analysis, very blatant anti-vaccine misinformation, a lot of it about COVID in particular, we're seeing it's not just anti-vaxxers, but we're actually seeing a lot of mis, 16:25.880 --> 16:29.220 cross-community support by misinformers. 16:29.520 --> 16:31.080 It's pretty blatant. 16:31.080 --> 16:37.500 It's also it's pretty, pretty obvious how they abuse, like, the very few terms of service Twitter does have. 16:37.500 --> 16:39.580 So that's a little strange. 16:39.980 --> 16:52.900 It is a little disappointing to see Fox, the Republican Party, MAGA, and a lot of them kind of involved in this anti-medical misinformation. 16:52.900 --> 16:54.700 And again, it's so blatant. 16:55.040 --> 17:03.680 And then on the platform side, it's when we saw that overlap and like, oh, wait a minute, Twitter makes money off of some of this stuff. 17:04.740 --> 17:08.820 They're clearly looking the other way for the terms of service violation. 17:09.000 --> 17:12.260 And then like, what was the the the Plandemic promotion? 17:12.260 --> 17:14.280 Just that was just that blew my mind. 17:14.280 --> 17:18.920 So this is not, this is not a healthy situation. 17:19.580 --> 17:27.480 So from a data perspective, or actually security analytics perspective, I do want to step back. 17:27.960 --> 17:36.720 If we think about attackers, defenders, ultimately, the targets are individuals and organizations going on a radicalization journey. 17:36.720 --> 17:44.220 And so every time Twitter recommends you do another crazy thing that just takes you deep, deeper. 17:44.520 --> 17:53.340 Interesting extra level on the threat model here is that folks are essentially influenced based on their life conditions. 17:53.340 --> 17:57.280 So maybe they're going through a hard time, maybe they're elderly and don't really know what's going on. 17:57.280 --> 18:04.640 And also by their community, you know, if your if your pastor or somebody or your family friends tells you something, you're more inclined to believe it. 18:04.700 --> 18:12.060 And so being able to infect communities with this stuff is having mastery over that and platform level support, it's not good. 18:12.680 --> 18:17.060 For actors, it's a lot of what you'd expect. 18:17.560 --> 18:28.040 I think that the surprising thing for for threat actors here is we actually consider the, at this point, the, the platforms to be part of the threat. 18:28.320 --> 18:34.280 In particular, we we break down the platforms as the social network system. 18:34.280 --> 18:41.760 So Facebook owns a bunch of social network, Google owns a bunch of social networks, a few are just big on their own, right, like Reddit, Reddit and Twitter. 18:42.280 --> 18:45.040 And, you know, Twitter is a multi billion dollar profitable company. 18:45.380 --> 18:47.280 But here we are. 18:47.460 --> 18:49.760 A little less obvious to folks is messaging. 18:49.940 --> 18:59.220 And again, actually, a lot of the same companies like Google, Microsoft, Yahoo, own essentially the messaging channels, those are completely unmonitored. 18:59.220 --> 19:03.540 And so a lot of times the radicalization actually hops over to newsletters. 19:03.540 --> 19:05.800 So that's stuff basically dark data. 19:06.680 --> 19:14.060 And also the local news, I think like Fox and folks like that. 19:14.060 --> 19:22.620 A little surprising is there's kind of a monopoly effect where because you do have regional different ways of having monopolies on people's attention. 19:22.620 --> 19:24.960 That's all the reason these companies got so big. 19:25.380 --> 19:29.440 Same reason we're basically at the whimsy of these companies. 19:29.440 --> 19:35.580 And so when you kind of calculate it out, the 80% essentially is just about 13 of these groups. 19:36.140 --> 19:50.760 And so if we have a fickle Silicon Valley CEO, if we have employees who just go along for the ride, if we have advertisers keep feeding them money, you know, so that's kind of from the threat model perspective. 19:53.020 --> 19:58.800 Shifting gears a bit to more like kind of data like security, data engineering, data science. 19:59.080 --> 20:02.780 In order to do the data science, I did want to talk a little bit about this. 20:03.380 --> 20:08.620 We end up having to build a lot to work at kind of the speeds and scales we're targeting. 20:08.620 --> 20:15.340 So if you want to have an instant response pipeline, that means you have to kind of have streaming, you know, can you take hundreds or thousands of things a second. 20:15.340 --> 20:23.960 So our V1, you're kind of seeing the performance of that where you're architecting to a V2 so that we can actually go to arbitrary scale, just have essentially budget limited. 20:25.060 --> 20:27.540 You are noticing we're doing a lot of graph technology. 20:27.920 --> 20:39.180 And so basically that's a nice way to deal with a lot of heterogeneous data and a lot of like linkage of data, even actually in a machine learning world where instead of just direct links, you might have vectors, feature vectors. 20:39.700 --> 20:43.100 Two other interesting things about the stack I did want to point out here. 20:43.640 --> 20:53.260 One is we do do end-to-end GPU computing just to be able to, you know, you can actually, if you want to have a billion node graph in memory, you actually could do that stuff with, for example, a Rapids AI stack. 20:53.480 --> 20:58.300 Likewise, neural nets in this space, graph neural nets are getting interesting. 20:58.540 --> 21:02.800 We actually do more today on the traditional ones, but we're looking at those. 21:02.940 --> 21:06.160 And the last part, and I think most important, is kind of division of team. 21:06.580 --> 21:09.080 We have basically heroes for data engineers. 21:09.080 --> 21:15.980 And then a big part of this is figuring out ways to get out of the way, just given it is an open volunteer effort. 21:15.980 --> 21:18.060 And so we actually do a lot on the self-service. 21:18.060 --> 21:21.040 And that's a big theme for look about powering. 21:21.060 --> 21:29.120 So think notebooks, streamlet dashboards, graphistry, do visual analytics and automatic ML. 21:29.120 --> 21:34.680 Just increasing the ability of people just to go and handle the stuff. 21:37.040 --> 21:41.360 From the data science side, I kind of wanted a fun example here. 21:41.900 --> 21:47.820 For example, we had a news organization ask us about Obamagate. 21:47.820 --> 21:57.720 And so one of our analysts, Cody, did a good job of using hashtag analysis to connect the dots to a month back to some QAnon stuff. 21:57.840 --> 22:03.720 And it was kind of impressive because the Trump tweet made no sense at all, even when I read it today. 22:03.880 --> 22:05.720 All these kind of code words. 22:06.060 --> 22:14.940 But once we moved to more modern NLP techniques and go away from exact match of terms. 22:15.220 --> 22:19.360 So Cody had to look for Obamagate. 22:19.480 --> 22:24.460 But with NLP, we can do things like look for the word Obama and the word conspiracy. 22:24.460 --> 22:27.280 But not the word, but just the fuzzy topic. 22:27.840 --> 22:34.000 And when you have those things, then we can kind of more loosely kind of walk back. 22:35.080 --> 22:38.400 And so that means we can do kind of traditional analysis more easily. 22:38.400 --> 22:40.340 And so we've been building out a lot around that. 22:42.800 --> 22:47.580 I want to modeling from basically what everyone does now. 22:48.640 --> 22:56.000 And then the other thing I want to point out here is this is also opening up types of analysis that just were straight up, like, you couldn't even think about. 22:56.500 --> 23:07.420 So maybe one example could be for looking at masks in a community where you might ask, like, which messages are against mask use? 23:07.420 --> 23:11.960 That's kind of a strange... that's kind of a gets into semantics and meaning a bit. 23:11.960 --> 23:14.040 And that's hard to do just by keywords. 23:14.660 --> 23:16.760 And then could you summarize it? 23:16.760 --> 23:18.420 Could we actually have a demographic breakdown? 23:18.420 --> 23:21.800 There are just certain things that with traditional NLP, you couldn't have done. 23:21.800 --> 23:28.640 But over the years, over the last, like, five years, and actually even two years, it's just been really nuts what's been going on. 23:33.160 --> 23:37.780 Then kind of the last piece I'm going to share here is kind of just where we are, where we're going. 23:38.280 --> 23:48.200 You're kind of getting a sense that we can now bring in a researcher on a topic, and if we have the data, or we help them pull in the data, and kind of do this kind of analysis. 23:48.520 --> 23:57.100 Next level of empowerment we're looking at is, like, as we're building this out, is can we actually partner with less technical organizations, and if we have to kind of do this to protect their community? 23:57.100 --> 24:03.900 Imagine, like, a local civic leader talking about the cancer community with Andrea, with kind of at-risk patients. 24:03.900 --> 24:13.220 And there we're looking at, are there folks who are not even maliciously misinforming in their community, but people are just confused? 24:13.220 --> 24:14.900 Or are there people kind of targeting them? 24:14.900 --> 24:18.220 Can we start mapping out and building tools and stuff around that? 24:18.820 --> 24:23.760 So that's kind of the phase shift we're starting to go into, so that's pretty exciting. 24:23.820 --> 24:28.220 You're kind of seeing, as you build out that state of capability, all this stuff unlocks. 24:29.120 --> 24:34.480 And then, finally, I just kind of want to close out and tie together some of the stuff here. 24:34.800 --> 24:37.400 So the beginning, I was kind of sharing that investigation. 24:38.300 --> 24:44.100 Hopefully, what you got across is, like, yeah, the analysis is cool, and we're able to, like, do these giant analyses. 24:44.100 --> 24:53.900 But, like, ultimately, a lot of the important stuff is so in-your-face and so huge numbers that, like, again, that analogy of fishing with dynamite. 24:54.420 --> 25:07.480 And it's, you know, those 13 companies, they shouldn't be gifting, essentially, millions or billions of dollars in free targeted ads to these groups and targeted recommendations. 25:07.480 --> 25:10.620 It's kind of that stuff that I think is more systemic. 25:11.260 --> 25:18.900 I am disappointed to see kind of these otherwise legitimate groups appear to be corrupted by misinformation. 25:18.900 --> 25:23.400 So call out, essentially, we're seeing MAGA is just so ever-present here. 25:23.400 --> 25:26.280 We're seeing social media companies, regular media companies. 25:26.280 --> 25:33.000 So there are things you could do if you're an employee, if you are anywhere near the advertising spend of your company. 25:33.000 --> 25:34.420 All that stuff's important. 25:35.860 --> 25:41.420 And then kind of turning it around, that's all kind of social policy. 25:41.640 --> 25:45.760 A lot of folks tuning in are technologists of different stripe. 25:46.940 --> 25:55.340 And the bad news is we're seeing these monopolistic media companies are not able to self-regulate. 25:55.340 --> 25:59.740 Like, the left hand and the right hand are always at each other, and it's kind of brought us to where we are. 25:59.800 --> 26:09.200 Likewise, they've kind of gutted investigative and journalist teams just because their ad companies sucked all the money. 26:09.220 --> 26:14.740 And likewise, we're seeing private companies are sort of compromised as they're kind of trying to deal with revenue. 26:15.020 --> 26:17.020 So that's bad. 26:17.160 --> 26:21.560 What's good is we actually have an open source community. 26:21.920 --> 26:25.940 We, I mean, just like the technology community, that is huge. 26:26.040 --> 26:30.700 And now at the same time, data science is, you know, our Python that's almost like built into the DNA. 26:31.140 --> 26:35.800 So I see strong potential and gathering potential for that. 26:36.300 --> 26:38.480 I encourage that. 26:38.880 --> 26:42.400 You're seeing some of that happen here with Project Domino. 26:42.400 --> 26:50.180 And so on that point, if you are into this kind of stuff and you could help out with coding, head into our Slack. 26:50.560 --> 26:59.800 Likewise, as we launch in the coming months, these kind of more self-serve and tiers for folks sitting on top, reach out for partnering. 26:59.800 --> 27:17.500 And likewise, while we are looking at doing sustainable approaches to our growth, especially as sort of a bootstrap phase that we're in for the next year or so, we're definitely, you know, grants, whatever, that stuff does help with the core team do what they need to do. 27:17.500 --> 27:20.500 Likewise, you know, serverless compute time, anything like that. 27:21.380 --> 27:23.900 With that, thank you for your time. 27:23.900 --> 27:32.760 And hopefully this changed a little bit about either how misinformation is working in practice in the COVID era. 27:32.760 --> 27:39.900 And also, even more importantly, that we actually could do things and we actually detect it and then can actually build tools for it. 27:40.280 --> 27:41.280 Thank you.