Today I want to tell you about a project in the works to use data science to help verify elections.
And although this project started in earnest only in October, we are going to have a publicly
available version of it up and running at Verified Voting for the November election.
But I first started thinking about this project, this project of using data to help verify
elections back in 2005. And at the time I had what seemed to me a pretty simple idea,
which was to take the number of ballots cast in each polling place and compare it to the number
of voters who signed in in each polling place. And just to make sure that the one wasn't bigger
than the other. And to use that as a check on elections and on whether an election was fair,
whether you can verify. How do you verify that the election was run correctly?
Well, I thought it was a pretty simple idea and it turned out that actually it was a very
complicated idea. Because though the math of it is simple, the technology of it seems simple,
it's just subtraction. You have to have the power to get it done. You have to have the power to get
the data. You have to have the power to get the data in the right format and to get the data in
a timeframe that it's useful. Some thanks are in order. I would like to thank the National Science
Foundation for funding and in particular Jeremy Epstein for believing in the project.
I would like to thank my team at Portland State, Eric Tsai and Raghu from Guadalupe,
and also the staff and faculty at the Hatfield School at Portland State University. I'd like to
thank Verified Voting and in particular Marian Schneider for supporting the project and having
the vision to make it a reality even on a short notice. And finally, I'd like to thank the
organizers of DEFCON for fulfilling a lifelong dream I never knew I had, namely to be active in
the chat of my own presentation. So thanks to everyone. Let's start with a real story from a
real election. In 2018, there were congressional elections all over the country, including in North
Carolina. And in the ninth district, after election day, there were some questions. And
there was one piece of simple data analysis in particular that had a big impact. It was a bar
chart and it looked like this. What you can see in this chart is that on the absentee ballots,
something was very different in Bladen County from all the other counties. Now, anomalies are
quite common in elections, and there are all kinds of explanations. They never tell you what
happened. They only tell you where you might look. And it's perfectly possible just from looking at
the graph that there's a reasonable explanation. So for example, maybe the Republican candidate
grew up in Bladen County, lived in Bladen County, and was a contributing citizen in Bladen County
for a long time and had made a lot of relationships and had done better there for real,
concrete, idiosyncratic reasons. But as people who knew the district in North Carolina looked
at it more and more, they couldn't find an explanation like that. They couldn't find an
explanation that made sense in terms of the differences that they knew were there between
the different counties. And this data analysis was part of what raised the profile of the problem
in this contest. And it eventually, an investigation was done. The Democratic candidate
pushed for an investigation. And when the investigation was done, then they found that,
in Bladen County, and they found enough through this investigation,
they found enough compromised ballots that they decided, the North Carolina Board of Elections
decided to not to certify the results and to have a do-over. Do-overs in elections are really rare.
They are expensive. They're upsetting. And for a while, the citizens of North Carolina's ninth
district did not have a representative in Congress, but it was the right thing to do.
And that's not a happy story while it was happening, but sometimes that's what verifying
an election looks like. Now, there are two really important details about this story.
One detail is that the investigation happened before the Board of Elections made its decision
about certification. If there had been some kind of investigation after certification,
it probably wouldn't have made a difference. And that's because the way elections are,
you really have to, at some point, decide who won and move on. Decide who won. Someone takes
office. Someone takes power. So the critical time for elections is before the results are certified.
The second thing about this story is that the investigation happened and was taken seriously
by the Board of Elections because a candidate insisted on it. The losing candidate, in fact,
insisted on it. And that's, as a technologist wanting to help use data to verify elections,
that's a point we can't afford to forget. Because it's the candidates who have the power
and the standing to sue in court or to press a Board of Elections to do an investigation.
So as technologists, if we want to have a high impact, we want to somehow dovetail our interest
in free and fair elections with the candidate's self-interest, with the candidate's interest
in winning. I've done a lot of both partisan and nonpartisan work in elections. And when you're
taking the nonpartisan side, it's really tempting to think of partisans as somehow
evil or compromised or not as morally grand as nonpartisan. But in fact, in America,
and probably I would guess all over the world, the real hard work of holding elections and
election administration accountable is done by the people who have the most skin in the game.
It's done by the partisans. It's done by the people who are competing to win the election.
So the most powerful thing we can do as technologists in applying technology to
this particular issue, how do you make sure that investigations that should happen do happen?
We do it most effectively when we have in mind how can we help candidates insist
on investigations and in fact help losing candidates insist on investigations when those
investigations are justified. What we've built and what will be available live online after the
November election, actually it will be available beforehand for folks to play around with,
but will be there for the November election, is a system that allows a candidate, or really
anybody, to go and look at visualizations of data of election results from the election before
certification. And it allows them to look at anomalies, anomalies like the one in Bladen
County if they exist, and some people might say that this is going to open up a big can of worms
because there are lots of anomalies in elections. Elections in my experience are full of anomalies
and in my experience most of them are completely, completely legitimate.
So, are we encouraging mass hysteria? Here's how I view it. So, the only people,
well the people with the most power to push for investigations really are the candidates.
And while candidates tend to be in the candidate bubble, and I can tell you from personal
experience, you always think you're going to win, you always think you did win,
the staff of the campaign and the people who back the candidate both morally and politically
and emotionally and with their dollars tend to have a more reasonable view of things.
And candidates don't often insist for long on investigations into problems that
aren't really problems. I really trust candidates to do that. And the point is that,
and I am also willing to take a few candidates who maybe won't do that, I still think it's
worthwhile because if the anomaly is for a real reason, an investigation should turn that up.
So, my goal is to think about after the election. My goal is to increase the confidence in the
election results after the election. And part of building that confidence is the idea that
if there are investigations that should have happened, they happened. Here's how the system
works. You choose a state. Let's say you choose North Carolina. And then you choose a contest
type. So you could look at presidential, you could look at congressional, you could look at state
house contests. And once you've done that, then you pick a particular contest or a group of
contests. So you could pick all congressional contests or you could pick any particular
congressional contest if you had chosen contest type congressional. Then what you get is some bar
charts which the system has chosen to present to you. So the system is presenting anomalies,
anomalies of interest. And if you ran this on the North Carolina 2018 data, the system would
show you as one of the most anomalous charts and most interesting charts, the Bladen County
anomaly. So if you look at the top chart, it's in a slightly different order. Bladen is on the left
because it is the anomaly. And if you compare it to the original chart that was published
by the political science professor and campaign consultant in North Carolina right after the
election, you see that's the chart. It's absentee ballots, accepted absentee by mail ballots,
and it's by county, and something anomalous happened in Bladen County.
So that's a case where the anomaly really makes a difference in the outcome of the contest.
And you can see that actually, that it's an important anomaly. If you look
underneath the bar chart, you can see it says votes at stake 110, margin 900.
This is a measure of how much impact this anomaly has on the margin. So if indeed it's an anomaly
that's due to something that could be undone, then that tells you how valuable it might be
to the losing candidate, who in this case was Dan McCready. Another anomaly that the system finds
for North Carolina 08, sorry, in 2018, is the anomaly chart on the bottom. Here the anomaly is
that there seem to be no provisional votes in Randolph County, whereas there are provisional
votes in all the other counties. This is for the sixth congressional district.
And it's not that important in terms of the overall votes at stake versus the margin,
because you can look and see that the votes at stake, just 25 about, and the margin was about
37,000. So what's going on here? Well, it turned out that there were provisional votes in Randolph
County in this contest, and the North Carolina Board of Elections had them, and they were part
of the official results, and they were part of what you would see if you went to look them up
online through their usual interface. They just hadn't made it into the data file that the Board
of Elections provided for download. And I want to point out that while the Blayton County anomaly
was known before we did our work, and, you know, you have every reason to question whether I
reverse-engineered the system so it would pop out, this anomaly no one had noticed before,
as far as I know. And when we pointed it out to the Board of Elections, they corrected the data
file and posted a new one. So this shows that anomaly detection can help Boards of Elections
with their process of continuous improvement. So that's a second application of this work.
So that's the bar chart part of the system, where the system chooses anomalies to show you.
There's another part of the system which allows you just to play around comparing various counts
in scatter plots. So, for example, we're still in 2018. This is Florida. If you compare the
United States Senate votes cast, total votes in the contest, to governor of Florida,
and you compare by county, you get the following chart. And the interesting thing is that there is,
you can see that all the counties line up, but one of them is a little bit off,
and that county is Broward County. In that county, there was an undervote for U.S. Senate
when people investigated. What they found was that the Broward County ballot
was poorly designed. It had a design flaw which made it pretty easy for people to inadvertently
miss the U.S. Senate contest. And it resulted in a significant undervote. And you can see that on
this chart. I want to end by showing you my absolute favorite scatter plot chart of all time.
This is from the Philadelphia 65th Ward by precinct in 2011,
my election contest against Marge Tartaglione, the famous Marge Tartaglione.
And you can see that there is a real outlier. I mean, that's a serious outlier. That's like
a hundred votes worth outlier. So if I were Marge Tartaglione, and if the margin had been small,
which it wasn't, I would be wanting to know what had happened in that precinct. And even if I just
were a person who cared about elections and free and fair elections, I might want to know what
happened in that precinct. And if someone had come to me and said, you know what, we saw this outlier
and we're going to investigate it. And I hope you have an explanation. I would have said, yes, I have
an explanation. That is the precinct where my daughter stood outside all day and said, vote for
my mom, vote for my mom, vote for my mom. And this points to a third application of this system,
which is to political science. What really makes a difference in voter behavior?
One way to figure that out is to look at outliers and to get explanations for outliers.
There's always something interesting behind an outlier. Let me say a little bit about the anomaly
detection algorithm. So first of all, when I first built just the first working prototype
of the system, I used just the simplest anomaly detection algorithm that fell to hand, which was to
think of, let's say you're taking a particular contest by county, some certain set of ballots,
and then for each county, if you have three candidates, then you have a vector in three
space that is the vote totals for those three candidates. Or you can look at the percentage
splits. You can have that be your vector in three space. In any case, if you take the vector for each
county, you can think of them as points in three space. And then you can just, there's a very simple
outlier detection thing called z-score. So you can just calculate for each point how far it is from
all the other points, and then apply the z-score to figure out what's the outlier in distance from
all the other points. So this is, you know, if you think about it in depth, this is, the z-score is
the wrong thing to use because you don't have a normal distribution and stuff like that. But
just throwing it together to see if it worked, I ran it on the North Carolina data, and
out came the Bladen County anomaly as one of the most anomalous slices in North Carolina.
So that's a pretty good proof of concept. Now we are doing something more sophisticated,
and we're certainly interested in doing a variety of different things.
So the tension here is between wanting to use percentages, because that levels the playing
field between counties. Counties are of widely varying size in pretty much every state.
Maybe not geographically, but in terms of population. And the same is true even of
precincts often or any other subdivision you might use. So the argument for using percents
is that it takes away that variability, which isn't what you want to look at anyway.
So on the other hand, what really matters in the end is how many votes are involved.
So that feels like an argument for using vote totals and not percentages. And the way we're
splitting the difference there is that we are using percentages to identify the outlying point.
And then once we have... if we have a case where we have a point that is an outlier,
then we use the vote totals to calculate how many votes are at stake, meaning right now,
what would the change in the margin be if you assume that that outlier count, if you alter it
back to fit in better with the other counts? How much does that change the margin? By what
percentage of the margin does it change? So what's the ratio of the votes at stake to the margin?
If that ratio is one, that means that if this outlier really is not a true good vote count,
and if it were changed, it might change the outcome of the contest. So that's
clearly important. If that ratio was 1%, then that outlier, it might be interesting, but
not from the point of view of would it change the outcome of the contest. So from the point
of view of the losing candidate, that's not an interesting anomaly at all, if it's 1% of the
margin. So we use this percentage of margin as a way of scoring which anomalies are likely to be
of most interest. Let me say a word about what data will actually be available. So back in 2005,
when I was thinking about how easy it would be to compare ballots cast to the number of
voters checked in at the polling place, one of the most naive things was that I had no idea how hard
it was to get election data, never mentioned to get it in time before certification. I had to
actually sue the Commonwealth of Pennsylvania to get voter file data. I had to threaten to sue
the Board of Elections in Philadelphia to get election results at precinct level in electronic
form. And eventually I had to run for the Board of Elections myself, which by the way, I highly
recommend, even though it was simultaneously the best and worst experience of my life serving. It
was very difficult, but it was important and very satisfying. And the more technologists we have in
office, in particular in offices that have some say over the conduct of elections, the better
elections are going to be in this country. But what data is available, what format it's in,
and how quickly you can get it, all of this varies wildly all over this country. And some
states like North Carolina, Virginia do an awesome, awesome job of putting the data out there.
And other states don't. And we will collect as fast as we can, as much as we can. And
that's a place actually where we also can use help if people want to be part of the collecting
that's awesome. While we're on the subject of collecting the data, you'll see if you go visit
the code repository on GitHub, you'll see that a good amount of it, most of it in fact, is dedicated
to munging the data, to taking the data from whatever format it arrives in and putting it
into a common data format so that the analysis algorithms that we have will be applicable to
all of the data that we have. That's something actually that I'm really proud of in the project
already. It's never easy to take data from different formats and put it into a common
format. This makes the process as straightforward as possible. So we are looking for people to
contribute to the collection effort, but we are also looking for people to contribute to
building the system. So I should say I should also thank the National Science Foundation,
which funded the building of the back end. And it's the back end that you'll see on GitHub and
that you should feel free to take and use however you like. Well, if you've stuck with this talk
this long, then maybe you're even interested in thinking about contributing to the project. We
would love that. We would love to have people build visualizations and analysis on top of
the standardized data that we can now provide. We are looking for someone or someones to
build tools that will pull data from APIs into our common data format and also
take the data that we have ourselves and to put it out in the common data format
that has been developed by the National Institute of Standards and Technology.
We're also looking for people to help with documentation, just making sure our documentation
is clear. We're looking for people who are interested in merging in other data sets.
There is a lot of potential here to do analysis not based only on election results,
but also on other election related data and even data that maybe doesn't seem at first glance to be
election related, but I always think of the weather. It's one of these folk truths in
elections that weather affects who comes out for elections. There will be data about COVID,
there will be data about different ways that different jurisdictions have handled vote by mail.
There's going to be all kinds of interesting data and we really would love to find people
who want to merge in some of that data and build analysis on top of that combination of data.
And finally, if you have experience building a successful open source community,
it would be terrific to get your input and get your help in building this
as a long live open source community. Because we're focused now on 2020,
but there's a lot of potential for growth here.