WEBVTT 00:15.400 --> 00:23.600 Today I want to tell you about a project in the works to use data science to help verify elections. 00:24.270 --> 00:41.010 And although this project started in earnest only in October, we are going to have a publicly available version of it up and running at Verified Voting for the November election. 00:41.620 --> 00:50.110 But I first started thinking about this project, this project of using data to help verify elections back in 2005. 00:50.170 --> 01:06.490 And at the time I had what seemed to me a pretty simple idea, which was to take the number of ballots cast in each polling place and compare it to the number of voters who signed in in each polling place. 01:06.670 --> 01:11.450 And just to make sure that the one wasn't bigger than the other. 01:11.970 --> 01:21.310 And to use that as a check on elections and on whether an election was fair, whether you can verify. 01:21.310 --> 01:27.470 How do you verify that the election was run correctly? 01:28.170 --> 01:34.030 Well, I thought it was a pretty simple idea and it turned out that actually it was a very complicated idea. 01:34.470 --> 01:42.750 Because though the math of it is simple, the technology of it seems simple, it's just subtraction. 01:44.490 --> 01:48.590 You have to have the power to get it done. 01:48.690 --> 01:51.810 You have to have the power to get the data. 01:51.810 --> 02:01.310 You have to have the power to get the data in the right format and to get the data in a timeframe that it's useful. 02:01.310 --> 02:03.490 Some thanks are in order. 02:03.490 --> 02:14.370 I would like to thank the National Science Foundation for funding and in particular Jeremy Epstein for believing in the project. 02:14.370 --> 02:30.710 I would like to thank my team at Portland State, Eric Tsai and Raghu from Guadalupe, and also the staff and faculty at the Hatfield School at Portland State University. 02:30.710 --> 02:41.930 I'd like to thank Verified Voting and in particular Marian Schneider for supporting the project and having the vision to make it a reality even on a short notice. 02:42.110 --> 02:54.210 And finally, I'd like to thank the organizers of DEFCON for fulfilling a lifelong dream I never knew I had, namely to be active in the chat of my own presentation. 02:54.290 --> 02:57.030 So thanks to everyone. 02:57.030 --> 03:01.130 Let's start with a real story from a real election. 03:01.130 --> 03:08.430 In 2018, there were congressional elections all over the country, including in North Carolina . 03:08.430 --> 03:16.310 And in the ninth district, after election day, there were some questions. 03:16.310 --> 03:23.830 And there was one piece of simple data analysis in particular that had a big impact. 03:23.830 --> 03:27.690 It was a bar chart and it looked like this. 03:28.210 --> 03:39.290 What you can see in this chart is that on the absentee ballots, something was very different in Bladen County from all the other counties. 03:39.470 --> 03:45.650 Now, anomalies are quite common in elections, and there are all kinds of explanations. 03:46.130 --> 03:48.090 They never tell you what happened. 03:48.090 --> 03:50.630 They only tell you where you might look. 03:50.870 --> 03:56.390 And it's perfectly possible just from looking at the graph that there's a reasonable explanation. 03:56.390 --> 04:16.530 So for example, maybe the Republican candidate grew up in Bladen County, lived in Bladen County, and was a contributing citizen in Bladen County for a long time and had made a lot of relationships and had done better there for real, concrete, idiosyncratic reasons. 04:16.670 --> 04:23.950 But as people who knew the district in North Carolina looked at it more and more, they couldn't find an explanation like that. 04:23.950 --> 04:33.170 They couldn't find an explanation that made sense in terms of the differences that they knew were there between the different counties. 04:33.250 --> 04:41.750 And this data analysis was part of what raised the profile of the problem in this contest. 04:41.750 --> 04:47.050 And it eventually, an investigation was done. 04:47.070 --> 04:50.570 The Democratic candidate pushed for an investigation. 04:50.570 --> 05:19.150 And when the investigation was done, then they found that, in Bladen County, and they found enough through this investigation, they found enough compromised ballots that they decided, the North Carolina Board of Elections decided to not to certify the results and to have a do-over. 05:19.150 --> 05:22.070 Do-overs in elections are really rare. 05:22.070 --> 05:23.470 They are expensive. 05:23.470 --> 05:24.710 They're upsetting. 05:24.710 --> 05:33.710 And for a while, the citizens of North Carolina's ninth district did not have a representative in Congress, but it was the right thing to do. 05:33.890 --> 05:42.910 And that's not a happy story while it was happening, but sometimes that's what verifying an election looks like. 05:42.910 --> 05:49.010 Now, there are two really important details about this story. 05:49.090 --> 05:57.370 One detail is that the investigation happened before the Board of Elections made its decision about certification. 05:57.490 --> 06:05.990 If there had been some kind of investigation after certification, it probably wouldn't have made a difference. 06:05.990 --> 06:12.730 And that's because the way elections are, you really have to, at some point, decide who won and move on. 06:12.730 --> 06:14.070 Decide who won. 06:14.070 --> 06:15.150 Someone takes office. 06:15.150 --> 06:16.710 Someone takes power. 06:16.710 --> 06:23.890 So the critical time for elections is before the results are certified. 06:23.890 --> 06:35.670 The second thing about this story is that the investigation happened and was taken seriously by the Board of Elections because a candidate insisted on it. 06:35.670 --> 06:39.150 The losing candidate, in fact, insisted on it. 06:39.750 --> 06:47.750 And that's, as a technologist wanting to help use data to verify elections, that's a point we can't afford to forget. 06:47.750 --> 07:00.250 Because it's the candidates who have the power and the standing to sue in court or to press a Board of Elections to do an investigation. 07:00.250 --> 07:19.390 So as technologists, if we want to have a high impact, we want to somehow dovetail our interest in free and fair elections with the candidate's self-interest, with the candidate's interest in winning. 07:20.670 --> 07:25.930 I've done a lot of both partisan and nonpartisan work in elections. 07:25.930 --> 07:35.410 And when you're taking the nonpartisan side, it's really tempting to think of partisans as somehow 07:38.450 --> 07:45.430 evil or compromised or not as morally grand as nonpartisan. 07:45.430 --> 08:04.990 But in fact, in America, and probably I would guess all over the world, the real hard work of holding elections and election administration accountable is done by the people who have the most skin in the game. 08:04.990 --> 08:07.130 It's done by the partisans. 08:07.130 --> 08:12.300 It's done by the people who are competing to win the election. 08:12.690 --> 08:25.450 So the most powerful thing we can do as technologists in applying technology to this particular issue, how do you make sure that investigations that should happen do happen? 08:27.950 --> 08:43.050 We do it most effectively when we have in mind how can we help candidates insist on investigations and in fact help losing candidates insist on investigations when those investigations are justified. 08:43.050 --> 09:02.850 What we've built and what will be available live online after the November election, actually it will be available beforehand for folks to play around with, but will be there for the November election, is a system that allows a candidate, or really anybody, 09:02.850 --> 09:15.130 to go and look at visualizations of data of election results from the election before certification. 09:16.350 --> 09:34.730 And it allows them to look at anomalies, anomalies like the one in Bladen County if they exist, and some people might say that this is going to open up a big can of worms because there are lots of anomalies in elections. 09:34.730 --> 09:42.750 Elections in my experience are full of anomalies and in my experience most of them are completely, completely legitimate. 09:45.050 --> 09:50.630 So, are we encouraging mass hysteria? 09:51.690 --> 09:53.490 Here's how I view it. 09:53.490 --> 10:02.210 So, the only people, well the people with the most power to push for investigations really are the candidates. 10:02.510 --> 10:32.350 And while candidates tend to be in the candidate bubble, and I can tell you from personal experience, you always think you're going to win, you always think you did win, the staff of the campaign and the people who back the candidate both morally and politically and emotionally and with their dollars tend to have a more reasonable view of things. 10:32.350 --> 10:41.570 And candidates don't often insist for long on investigations into problems that aren't really problems. 10:41.570 --> 10:47.150 I really trust candidates to do that. 10:47.150 --> 11:02.330 And the point is that, and I am also willing to take a few candidates who maybe won't do that, I still think it's worthwhile because if the anomaly is for a real reason, an investigation should turn that up. 11:03.210 --> 11:08.290 So, my goal is to think about after the election. 11:08.290 --> 11:15.190 My goal is to increase the confidence in the election results after the election. 11:15.190 --> 11:24.270 And part of building that confidence is the idea that if there are investigations that should have happened, they happened. 11:25.050 --> 11:27.470 Here's how the system works. 11:28.290 --> 11:31.730 You choose a state. 11:32.290 --> 11:34.970 Let's say you choose North Carolina. 11:35.090 --> 11:39.270 And then you choose a contest type. 11:39.970 --> 11:46.050 So you could look at presidential, you could look at congressional, you could look at state house contests. 11:46.630 --> 11:55.230 And once you've done that, then you pick a particular contest or a group of contests. 11:55.230 --> 12:06.290 So you could pick all congressional contests or you could pick any particular congressional contest if you had chosen contest type congressional. 12:07.370 --> 12:15.170 Then what you get is some bar charts which the system has chosen to present to you. 12:15.170 --> 12:21.430 So the system is presenting anomalies, anomalies of interest. 12:22.230 --> 12:39.130 And if you ran this on the North Carolina 2018 data, the system would show you as one of the most anomalous charts and most interesting charts, the Bladen County anomaly. 12:39.410 --> 12:43.510 So if you look at the top chart, it's in a slightly different order. 12:43.510 --> 12:47.390 Bladen is on the left because it is the anomaly. 12:47.710 --> 13:01.910 And if you compare it to the original chart that was published by the political science professor and campaign consultant in North Carolina right after the election, you see that's the chart. 13:01.910 --> 13:14.450 It's absentee ballots, accepted absentee by mail ballots, and it's by county, and something anomalous happened in Bladen County. 13:15.480 --> 13:25.090 So that's a case where the anomaly really makes a difference in the outcome of the contest. 13:25.550 --> 13:29.510 And you can see that actually, that it's an important anomaly. 13:29.510 --> 13:37.660 If you look underneath the bar chart, you can see it says votes at stake 110, margin 900. 13:38.760 --> 13:46.580 This is a measure of how much impact this anomaly has on the margin. 13:46.580 --> 13:59.780 So if indeed it's an anomaly that's due to something that could be undone, then that tells you how valuable it might be to the losing candidate, who in this case was Dan McCready. 14:01.160 --> 14:12.600 Another anomaly that the system finds for North Carolina 08, sorry, in 2018, is the anomaly chart on the bottom. 14:12.600 --> 14:26.580 Here the anomaly is that there seem to be no provisional votes in Randolph County, whereas there are provisional votes in all the other counties. 14:26.580 --> 14:30.160 This is for the sixth congressional district. 14:32.000 --> 14:46.880 And it's not that important in terms of the overall votes at stake versus the margin, because you can look and see that the votes at stake, just 25 about, and the margin was about 37,000. 14:47.000 --> 14:49.560 So what's going on here? 14:49.560 --> 15:06.100 Well, it turned out that there were provisional votes in Randolph County in this contest, and the North Carolina Board of Elections had them, and they were part of the official results, and they were part of what you would see if you went to look them up online through their usual interface. 15:06.100 --> 15:12.520 They just hadn't made it into the data file that the Board of Elections provided for download. 15:12.880 --> 15:30.460 And I want to point out that while the Blayton County anomaly was known before we did our work, and, you know, you have every reason to question whether I reverse-engineered the system so it would pop out, this anomaly no one had noticed before, as far as I know. 15:31.740 --> 15:37.540 And when we pointed it out to the Board of Elections, they corrected the data file and posted a new one. 15:38.020 --> 15:47.120 So this shows that anomaly detection can help Boards of Elections with their process of continuous improvement. 15:47.120 --> 15:50.720 So that's a second application of this work. 15:51.180 --> 15:59.060 So that's the bar chart part of the system, where the system chooses anomalies to show you. 15:59.060 --> 16:06.360 There's another part of the system which allows you just to play around comparing various counts in scatter plots. 16:06.400 --> 16:09.780 So, for example, we're still in 2018. 16:09.800 --> 16:11.480 This is Florida. 16:11.920 --> 16:26.260 If you compare the United States Senate votes cast, total votes in the contest, to governor of Florida, and you compare by county, you get the following chart. 16:26.460 --> 16:35.860 And the interesting thing is that there is, you can see that all the counties line up, but one of them is a little bit off, and that county is Broward County. 16:35.920 --> 16:39.640 In that county, there was an undervote for U.S. 16:39.640 --> 16:42.960 Senate when people investigated. 16:42.960 --> 16:48.320 What they found was that the Broward County ballot was poorly designed. 16:48.320 --> 16:57.860 It had a design flaw which made it pretty easy for people to inadvertently miss the U.S. 16:57.860 --> 16:59.240 Senate contest. 16:59.380 --> 17:02.820 And it resulted in a significant undervote. 17:02.820 --> 17:05.200 And you can see that on this chart. 17:05.320 --> 17:10.980 I want to end by showing you my absolute favorite scatter plot chart of all time. 17:11.940 --> 17:24.560 This is from the Philadelphia 65th Ward by precinct in 2011, my election contest against Marge Tartaglione, the famous Marge Tartaglione. 17:25.500 --> 17:30.320 And you can see that there is a real outlier. 17:30.520 --> 17:32.160 I mean, that's a serious outlier. 17:32.160 --> 17:35.500 That's like a hundred votes worth outlier. 17:36.040 --> 17:50.160 So if I were Marge Tartaglione, and if the margin had been small, which it wasn't, I would be wanting to know what had happened in that precinct. 17:50.360 --> 17:58.700 And even if I just were a person who cared about elections and free and fair elections, I might want to know what happened in that precinct. 17:58.760 --> 18:04.280 And if someone had come to me and said, you know what, we saw this outlier and we're going to investigate it. 18:04.420 --> 18:06.780 And I hope you have an explanation. 18:07.340 --> 18:10.360 I would have said, yes, I have an explanation. 18:10.560 --> 18:19.920 That is the precinct where my daughter stood outside all day and said, vote for my mom, vote for my mom, vote for my mom. 18:20.720 --> 18:27.580 And this points to a third application of this system, which is to political science. 18:27.580 --> 18:30.740 What really makes a difference in voter behavior? 18:31.700 --> 18:39.760 One way to figure that out is to look at outliers and to get explanations for outliers. 18:40.400 --> 18:43.540 There's always something interesting behind an outlier. 18:43.580 --> 18:48.740 Let me say a little bit about the anomaly detection algorithm. 18:48.920 --> 19:12.760 So first of all, when I first built just the first working prototype of the system, I used just the simplest anomaly detection algorithm that fell to hand, which was to think of, let's say you're taking a particular contest by county, some certain set of ballots, 19:12.760 --> 19:25.100 and then for each county, if you have three candidates, then you have a vector in three space that is the vote totals for those three candidates. 19:25.320 --> 19:28.480 Or you can look at the percentage splits. 19:28.480 --> 19:31.800 You can have that be your vector in three space. 19:31.800 --> 19:38.820 In any case, if you take the vector for each county, you can think of them as points in three space. 19:38.840 --> 19:45.540 And then you can just, there's a very simple outlier detection thing called z-score. 19:46.020 --> 20:01.800 So you can just calculate for each point how far it is from all the other points, and then apply the z-score to figure out what's the outlier in distance from all the other points. 20:03.440 --> 20:13.640 So this is, you know, if you think about it in depth, this is, the z-score is the wrong thing to use because you don't have a normal distribution and stuff like that. 20:13.640 --> 20:25.620 But just throwing it together to see if it worked, I ran it on the North Carolina data, and out came the Bladen County anomaly as one of the most anomalous slices in North Carolina. 20:27.520 --> 20:30.300 So that's a pretty good proof of concept. 20:30.560 --> 20:38.580 Now we are doing something more sophisticated, and we're certainly interested in doing a variety of different things. 20:38.580 --> 20:48.700 So the tension here is between wanting to use percentages, because that levels the playing field between counties. 20:48.700 --> 20:54.880 Counties are of widely varying size in pretty much every state. 20:55.380 --> 20:59.400 Maybe not geographically, but in terms of population. 20:59.400 --> 21:04.620 And the same is true even of precincts often or any other subdivision you might use. 21:06.300 --> 21:15.600 So the argument for using percents is that it takes away that variability, which isn't what you want to look at anyway. 21:16.340 --> 21:23.940 So on the other hand, what really matters in the end is how many votes are involved. 21:23.940 --> 21:29.320 So that feels like an argument for using vote totals and not percentages. 21:29.340 --> 21:39.880 And the way we're splitting the difference there is that we are using percentages to identify the outlying point. 21:40.860 --> 22:06.320 And then once we have... if we have a case where we have a point that is an outlier, then we use the vote totals to calculate how many votes are at stake, meaning right now, what would the change in the margin be if you assume that that outlier count, 22:06.320 --> 22:12.200 if you alter it back to fit in better with the other counts? 22:13.520 --> 22:16.000 How much does that change the margin? 22:16.000 --> 22:18.880 By what percentage of the margin does it change? 22:19.000 --> 22:23.180 So what's the ratio of the votes at stake to the margin? 22:23.180 --> 22:36.860 If that ratio is one, that means that if this outlier really is not a true good vote count, and if it were changed, it might change the outcome of the contest. 22:36.860 --> 22:39.180 So that's clearly important. 22:39.240 --> 22:51.040 If that ratio was 1%, then that outlier, it might be interesting, but not from the point of view of would it change the outcome of the contest. 22:51.080 --> 22:57.820 So from the point of view of the losing candidate, that's not an interesting anomaly at all, if it's 1% of the margin. 22:57.820 --> 23:07.380 So we use this percentage of margin as a way of scoring which anomalies are likely to be of most interest. 23:07.500 --> 23:12.180 Let me say a word about what data will actually be available. 23:12.800 --> 23:39.160 So back in 2005, when I was thinking about how easy it would be to compare ballots cast to the number of voters checked in at the polling place, one of the most naive things was that I had no idea how hard it was to get election data, never mentioned to get it in time before certification. 23:40.800 --> 23:46.700 I had to actually sue the Commonwealth of Pennsylvania to get voter file data. 23:46.700 --> 23:55.540 I had to threaten to sue the Board of Elections in Philadelphia to get election results at precinct level in electronic form. 23:55.860 --> 24:08.220 And eventually I had to run for the Board of Elections myself, which by the way, I highly recommend, even though it was simultaneously the best and worst experience of my life serving. 24:08.220 --> 24:12.500 It was very difficult, but it was important and very satisfying. 24:12.500 --> 24:25.300 And the more technologists we have in office, in particular in offices that have some say over the conduct of elections, the better elections are going to be in this country. 24:26.620 --> 24:38.420 But what data is available, what format it's in, and how quickly you can get it, all of this varies wildly all over this country. 24:38.440 --> 24:45.260 And some states like North Carolina, Virginia do an awesome, awesome job of putting the data out there. 24:45.260 --> 24:48.300 And other states don't. 24:48.340 --> 24:53.280 And we will collect as fast as we can, as much as we can. 24:53.560 --> 25:03.200 And that's a place actually where we also can use help if people want to be part of the collecting that's awesome. 25:03.300 --> 25:36.220 While we're on the subject of collecting the data, you'll see if you go visit the code repository on GitHub, you'll see that a good amount of it, most of it in fact, is dedicated to munging the data, to taking the data from whatever format it arrives in and putting it into a common data format so that the analysis algorithms that we have will be applicable to all of the data that we have. 25:36.260 --> 25:41.140 That's something actually that I'm really proud of in the project already. 25:41.140 --> 25:47.740 It's never easy to take data from different formats and put it into a common format. 25:47.740 --> 25:52.800 This makes the process as straightforward as possible. 25:53.240 --> 26:07.460 So we are looking for people to contribute to the collection effort, but we are also looking for people to contribute to building the system. 26:07.460 --> 26:14.300 So I should say I should also thank the National Science Foundation, which funded the building of the back end. 26:14.440 --> 26:22.120 And it's the back end that you'll see on GitHub and that you should feel free to take and use however you like. 26:23.100 --> 26:32.940 Well, if you've stuck with this talk this long, then maybe you're even interested in thinking about contributing to the project. 26:32.940 --> 26:35.620 We would love that. 26:35.780 --> 26:46.760 We would love to have people build visualizations and analysis on top of the standardized data that we can now provide. 26:46.760 --> 27:15.360 We are looking for someone or someones to build tools that will pull data from APIs into our common data format and also take the data that we have ourselves and to put it out in the common data format that has been developed by the National Institute of Standards and Technology. 27:16.160 --> 27:23.440 We're also looking for people to help with documentation, just making sure our documentation is clear. 27:24.120 --> 27:29.440 We're looking for people who are interested in merging in other data sets. 27:29.440 --> 27:48.680 There is a lot of potential here to do analysis not based only on election results, but also on other election related data and even data that maybe doesn't seem at first glance to be election related, but I always think of the weather. 27:49.360 --> 27:58.400 It's one of these folk truths in elections that weather affects who comes out for elections. 27:58.400 --> 28:07.080 There will be data about COVID, there will be data about different ways that different jurisdictions have handled vote by mail. 28:07.080 --> 28:23.160 There's going to be all kinds of interesting data and we really would love to find people who want to merge in some of that data and build analysis on top of that combination of data. 28:23.220 --> 28:41.180 And finally, if you have experience building a successful open source community, it would be terrific to get your input and get your help in building this as a long live open source community. 28:41.240 --> 28:47.820 Because we're focused now on 2020, but there's a lot of potential for growth here.