start	end	text
15400	23600	Today I want to tell you about a project in the works to use data science to help verify elections.
24270	33090	And although this project started in earnest only in October, we are going to have a publicly
33090	41010	available version of it up and running at Verified Voting for the November election.
41620	47630	But I first started thinking about this project, this project of using data to help verify
47630	54970	elections back in 2005. And at the time I had what seemed to me a pretty simple idea,
54970	63370	which was to take the number of ballots cast in each polling place and compare it to the number
63370	70350	of voters who signed in in each polling place. And just to make sure that the one wasn't bigger
70350	79770	than the other. And to use that as a check on elections and on whether an election was fair,
79770	87470	whether you can verify. How do you verify that the election was run correctly?
88170	92410	Well, I thought it was a pretty simple idea and it turned out that actually it was a very
92410	100850	complicated idea. Because though the math of it is simple, the technology of it seems simple,
100850	110630	it's just subtraction. You have to have the power to get it done. You have to have the power to get
110630	118390	the data. You have to have the power to get the data in the right format and to get the data in
118830	128070	a timeframe that it's useful. Some thanks are in order. I would like to thank the National Science
128070	134370	Foundation for funding and in particular Jeremy Epstein for believing in the project.
134370	143770	I would like to thank my team at Portland State, Eric Tsai and Raghu from Guadalupe,
144410	151610	and also the staff and faculty at the Hatfield School at Portland State University. I'd like to
151610	158170	thank Verified Voting and in particular Marian Schneider for supporting the project and having
158170	164210	the vision to make it a reality even on a short notice. And finally, I'd like to thank the
164210	170990	organizers of DEFCON for fulfilling a lifelong dream I never knew I had, namely to be active in
170990	179650	the chat of my own presentation. So thanks to everyone. Let's start with a real story from a
179650	187810	real election. In 2018, there were congressional elections all over the country, including in North
187810	196990	Carolina. And in the ninth district, after election day, there were some questions. And
196990	204810	there was one piece of simple data analysis in particular that had a big impact. It was a bar
204810	214810	chart and it looked like this. What you can see in this chart is that on the absentee ballots,
214810	220790	something was very different in Bladen County from all the other counties. Now, anomalies are
220790	227510	quite common in elections, and there are all kinds of explanations. They never tell you what
227510	233930	happened. They only tell you where you might look. And it's perfectly possible just from looking at
233930	240150	the graph that there's a reasonable explanation. So for example, maybe the Republican candidate
240150	247450	grew up in Bladen County, lived in Bladen County, and was a contributing citizen in Bladen County
247450	253470	for a long time and had made a lot of relationships and had done better there for real,
253470	260010	concrete, idiosyncratic reasons. But as people who knew the district in North Carolina looked
260010	265090	at it more and more, they couldn't find an explanation like that. They couldn't find an
265090	271050	explanation that made sense in terms of the differences that they knew were there between
271050	279790	the different counties. And this data analysis was part of what raised the profile of the problem
279790	288230	in this contest. And it eventually, an investigation was done. The Democratic candidate
288230	298650	pushed for an investigation. And when the investigation was done, then they found that,
298650	306110	in Bladen County, and they found enough through this investigation,
306530	314150	they found enough compromised ballots that they decided, the North Carolina Board of Elections
314150	322070	decided to not to certify the results and to have a do-over. Do-overs in elections are really rare.
322070	328130	They are expensive. They're upsetting. And for a while, the citizens of North Carolina's ninth
328130	333710	district did not have a representative in Congress, but it was the right thing to do.
333890	340490	And that's not a happy story while it was happening, but sometimes that's what verifying
340490	349010	an election looks like. Now, there are two really important details about this story.
349090	355550	One detail is that the investigation happened before the Board of Elections made its decision
355550	362270	about certification. If there had been some kind of investigation after certification,
362270	368190	it probably wouldn't have made a difference. And that's because the way elections are,
368190	374650	you really have to, at some point, decide who won and move on. Decide who won. Someone takes
374650	383890	office. Someone takes power. So the critical time for elections is before the results are certified.
383890	390810	The second thing about this story is that the investigation happened and was taken seriously
390810	397550	by the Board of Elections because a candidate insisted on it. The losing candidate, in fact,
397550	405150	insisted on it. And that's, as a technologist wanting to help use data to verify elections,
405150	411070	that's a point we can't afford to forget. Because it's the candidates who have the power
411070	420250	and the standing to sue in court or to press a Board of Elections to do an investigation.
420250	430610	So as technologists, if we want to have a high impact, we want to somehow dovetail our interest
430610	436950	in free and fair elections with the candidate's self-interest, with the candidate's interest
436950	447570	in winning. I've done a lot of both partisan and nonpartisan work in elections. And when you're
447570	455410	taking the nonpartisan side, it's really tempting to think of partisans as somehow
458450	468610	evil or compromised or not as morally grand as nonpartisan. But in fact, in America,
468990	478950	and probably I would guess all over the world, the real hard work of holding elections and
478950	484990	election administration accountable is done by the people who have the most skin in the game.
484990	492300	It's done by the partisans. It's done by the people who are competing to win the election.
492690	499470	So the most powerful thing we can do as technologists in applying technology to
499470	505450	this particular issue, how do you make sure that investigations that should happen do happen?
507950	515110	We do it most effectively when we have in mind how can we help candidates insist
515110	521230	on investigations and in fact help losing candidates insist on investigations when those
521230	529070	investigations are justified. What we've built and what will be available live online after the
529070	534190	November election, actually it will be available beforehand for folks to play around with,
534190	541810	but will be there for the November election, is a system that allows a candidate, or really
541810	553410	anybody, to go and look at visualizations of data of election results from the election before
553410	562230	certification. And it allows them to look at anomalies, anomalies like the one in Bladen
562230	570210	County if they exist, and some people might say that this is going to open up a big can of worms
570210	577330	because there are lots of anomalies in elections. Elections in my experience are full of anomalies
577330	582750	and in my experience most of them are completely, completely legitimate.
585050	597350	So, are we encouraging mass hysteria? Here's how I view it. So, the only people,
597350	602210	well the people with the most power to push for investigations really are the candidates.
602510	608770	And while candidates tend to be in the candidate bubble, and I can tell you from personal
608770	615190	experience, you always think you're going to win, you always think you did win,
615770	623690	the staff of the campaign and the people who back the candidate both morally and politically
623690	632350	and emotionally and with their dollars tend to have a more reasonable view of things.
632350	639330	And candidates don't often insist for long on investigations into problems that
639330	648950	aren't really problems. I really trust candidates to do that. And the point is that,
648950	655590	and I am also willing to take a few candidates who maybe won't do that, I still think it's
655590	662330	worthwhile because if the anomaly is for a real reason, an investigation should turn that up.
663210	671970	So, my goal is to think about after the election. My goal is to increase the confidence in the
671970	679310	election results after the election. And part of building that confidence is the idea that
679310	686370	if there are investigations that should have happened, they happened. Here's how the system
686370	697870	works. You choose a state. Let's say you choose North Carolina. And then you choose a contest
697870	704150	type. So you could look at presidential, you could look at congressional, you could look at state
704150	714490	house contests. And once you've done that, then you pick a particular contest or a group of
714490	721110	contests. So you could pick all congressional contests or you could pick any particular
721110	731050	congressional contest if you had chosen contest type congressional. Then what you get is some bar
731050	739450	charts which the system has chosen to present to you. So the system is presenting anomalies,
739450	749450	anomalies of interest. And if you ran this on the North Carolina 2018 data, the system would
749450	757470	show you as one of the most anomalous charts and most interesting charts, the Bladen County
757470	764690	anomaly. So if you look at the top chart, it's in a slightly different order. Bladen is on the left
764690	771110	because it is the anomaly. And if you compare it to the original chart that was published
772030	778710	by the political science professor and campaign consultant in North Carolina right after the
778710	786690	election, you see that's the chart. It's absentee ballots, accepted absentee by mail ballots,
787850	794450	and it's by county, and something anomalous happened in Bladen County.
795480	805090	So that's a case where the anomaly really makes a difference in the outcome of the contest.
805550	810010	And you can see that actually, that it's an important anomaly. If you look
810530	817660	underneath the bar chart, you can see it says votes at stake 110, margin 900.
818760	828440	This is a measure of how much impact this anomaly has on the margin. So if indeed it's an anomaly
828440	836240	that's due to something that could be undone, then that tells you how valuable it might be
836240	843480	to the losing candidate, who in this case was Dan McCready. Another anomaly that the system finds
843480	854080	for North Carolina 08, sorry, in 2018, is the anomaly chart on the bottom. Here the anomaly is
854080	863260	that there seem to be no provisional votes in Randolph County, whereas there are provisional
863260	870160	votes in all the other counties. This is for the sixth congressional district.
872000	877840	And it's not that important in terms of the overall votes at stake versus the margin,
877840	885040	because you can look and see that the votes at stake, just 25 about, and the margin was about
885040	894500	37,000. So what's going on here? Well, it turned out that there were provisional votes in Randolph
894500	899120	County in this contest, and the North Carolina Board of Elections had them, and they were part
899120	903100	of the official results, and they were part of what you would see if you went to look them up
903100	910060	online through their usual interface. They just hadn't made it into the data file that the Board
910060	916100	of Elections provided for download. And I want to point out that while the Blayton County anomaly
916100	922220	was known before we did our work, and, you know, you have every reason to question whether I
922220	929160	reverse-engineered the system so it would pop out, this anomaly no one had noticed before,
929160	935300	as far as I know. And when we pointed it out to the Board of Elections, they corrected the data
935300	943960	file and posted a new one. So this shows that anomaly detection can help Boards of Elections
943960	950720	with their process of continuous improvement. So that's a second application of this work.
951180	959060	So that's the bar chart part of the system, where the system chooses anomalies to show you.
959060	964120	There's another part of the system which allows you just to play around comparing various counts
964120	973600	in scatter plots. So, for example, we're still in 2018. This is Florida. If you compare the
973600	981260	United States Senate votes cast, total votes in the contest, to governor of Florida,
981640	988860	and you compare by county, you get the following chart. And the interesting thing is that there is,
988860	993420	you can see that all the counties line up, but one of them is a little bit off,
993420	999840	and that county is Broward County. In that county, there was an undervote for U.S. Senate
1000300	1005780	when people investigated. What they found was that the Broward County ballot
1006380	1016240	was poorly designed. It had a design flaw which made it pretty easy for people to inadvertently
1016240	1023980	miss the U.S. Senate contest. And it resulted in a significant undervote. And you can see that on
1023980	1030980	this chart. I want to end by showing you my absolute favorite scatter plot chart of all time.
1031940	1038680	This is from the Philadelphia 65th Ward by precinct in 2011,
1038680	1044560	my election contest against Marge Tartaglione, the famous Marge Tartaglione.
1045500	1052660	And you can see that there is a real outlier. I mean, that's a serious outlier. That's like
1052820	1062680	a hundred votes worth outlier. So if I were Marge Tartaglione, and if the margin had been small,
1062680	1071240	which it wasn't, I would be wanting to know what had happened in that precinct. And even if I just
1071240	1077220	were a person who cared about elections and free and fair elections, I might want to know what
1077220	1082300	happened in that precinct. And if someone had come to me and said, you know what, we saw this outlier
1082300	1089180	and we're going to investigate it. And I hope you have an explanation. I would have said, yes, I have
1089180	1096520	an explanation. That is the precinct where my daughter stood outside all day and said, vote for
1096520	1104500	my mom, vote for my mom, vote for my mom. And this points to a third application of this system,
1104500	1110740	which is to political science. What really makes a difference in voter behavior?
1111700	1119760	One way to figure that out is to look at outliers and to get explanations for outliers.
1120400	1126240	There's always something interesting behind an outlier. Let me say a little bit about the anomaly
1126240	1133680	detection algorithm. So first of all, when I first built just the first working prototype
1133680	1143260	of the system, I used just the simplest anomaly detection algorithm that fell to hand, which was to
1145880	1152760	think of, let's say you're taking a particular contest by county, some certain set of ballots,
1152760	1160080	and then for each county, if you have three candidates, then you have a vector in three
1160080	1167680	space that is the vote totals for those three candidates. Or you can look at the percentage
1167680	1174320	splits. You can have that be your vector in three space. In any case, if you take the vector for each
1174320	1181500	county, you can think of them as points in three space. And then you can just, there's a very simple
1181500	1190560	outlier detection thing called z-score. So you can just calculate for each point how far it is from
1190560	1200400	all the other points, and then apply the z-score to figure out what's the outlier in distance from
1200400	1208440	all the other points. So this is, you know, if you think about it in depth, this is, the z-score is
1208440	1213780	the wrong thing to use because you don't have a normal distribution and stuff like that. But
1214440	1218620	just throwing it together to see if it worked, I ran it on the North Carolina data, and
1218620	1225620	out came the Bladen County anomaly as one of the most anomalous slices in North Carolina.
1227520	1233460	So that's a pretty good proof of concept. Now we are doing something more sophisticated,
1233460	1238580	and we're certainly interested in doing a variety of different things.
1238580	1246780	So the tension here is between wanting to use percentages, because that levels the playing
1246780	1254880	field between counties. Counties are of widely varying size in pretty much every state.
1255380	1260660	Maybe not geographically, but in terms of population. And the same is true even of
1260660	1270460	precincts often or any other subdivision you might use. So the argument for using percents
1270460	1275600	is that it takes away that variability, which isn't what you want to look at anyway.
1276340	1283940	So on the other hand, what really matters in the end is how many votes are involved.
1283940	1290580	So that feels like an argument for using vote totals and not percentages. And the way we're
1290580	1299880	splitting the difference there is that we are using percentages to identify the outlying point.
1300860	1306620	And then once we have... if we have a case where we have a point that is an outlier,
1307300	1318660	then we use the vote totals to calculate how many votes are at stake, meaning right now,
1319780	1327140	what would the change in the margin be if you assume that that outlier count, if you alter it
1327140	1336400	back to fit in better with the other counts? How much does that change the margin? By what
1336400	1343180	percentage of the margin does it change? So what's the ratio of the votes at stake to the margin?
1343180	1352320	If that ratio is one, that means that if this outlier really is not a true good vote count,
1352320	1357280	and if it were changed, it might change the outcome of the contest. So that's
1357280	1367040	clearly important. If that ratio was 1%, then that outlier, it might be interesting, but
1367040	1372400	not from the point of view of would it change the outcome of the contest. So from the point
1372400	1377400	of view of the losing candidate, that's not an interesting anomaly at all, if it's 1% of the
1377400	1385660	margin. So we use this percentage of margin as a way of scoring which anomalies are likely to be
1385660	1394620	of most interest. Let me say a word about what data will actually be available. So back in 2005,
1394620	1402980	when I was thinking about how easy it would be to compare ballots cast to the number of
1402980	1411840	voters checked in at the polling place, one of the most naive things was that I had no idea how hard
1411840	1421720	it was to get election data, never mentioned to get it in time before certification. I had to
1421720	1428580	actually sue the Commonwealth of Pennsylvania to get voter file data. I had to threaten to sue
1428580	1434240	the Board of Elections in Philadelphia to get election results at precinct level in electronic
1434240	1441200	form. And eventually I had to run for the Board of Elections myself, which by the way, I highly
1441200	1448380	recommend, even though it was simultaneously the best and worst experience of my life serving. It
1448380	1454860	was very difficult, but it was important and very satisfying. And the more technologists we have in
1454860	1463000	office, in particular in offices that have some say over the conduct of elections, the better
1463000	1471500	elections are going to be in this country. But what data is available, what format it's in,
1471500	1479020	and how quickly you can get it, all of this varies wildly all over this country. And some
1479020	1485260	states like North Carolina, Virginia do an awesome, awesome job of putting the data out there.
1485260	1494040	And other states don't. And we will collect as fast as we can, as much as we can. And
1495800	1501100	that's a place actually where we also can use help if people want to be part of the collecting
1501500	1508880	that's awesome. While we're on the subject of collecting the data, you'll see if you go visit
1508880	1516780	the code repository on GitHub, you'll see that a good amount of it, most of it in fact, is dedicated
1516780	1525080	to munging the data, to taking the data from whatever format it arrives in and putting it
1525080	1533800	into a common data format so that the analysis algorithms that we have will be applicable to
1533800	1540300	all of the data that we have. That's something actually that I'm really proud of in the project
1540300	1546940	already. It's never easy to take data from different formats and put it into a common
1546940	1556060	format. This makes the process as straightforward as possible. So we are looking for people to
1556060	1564780	contribute to the collection effort, but we are also looking for people to contribute to
1564780	1571320	building the system. So I should say I should also thank the National Science Foundation,
1571320	1578180	which funded the building of the back end. And it's the back end that you'll see on GitHub and
1578180	1585140	that you should feel free to take and use however you like. Well, if you've stuck with this talk
1585140	1593100	this long, then maybe you're even interested in thinking about contributing to the project. We
1593100	1601700	would love that. We would love to have people build visualizations and analysis on top of
1601700	1612120	the standardized data that we can now provide. We are looking for someone or someones to
1612980	1621640	build tools that will pull data from APIs into our common data format and also
1621640	1629120	take the data that we have ourselves and to put it out in the common data format
1629660	1635360	that has been developed by the National Institute of Standards and Technology.
1636160	1641960	We're also looking for people to help with documentation, just making sure our documentation
1641960	1649440	is clear. We're looking for people who are interested in merging in other data sets.
1649440	1657020	There is a lot of potential here to do analysis not based only on election results,
1657520	1664620	but also on other election related data and even data that maybe doesn't seem at first glance to be
1664620	1671840	election related, but I always think of the weather. It's one of these folk truths in
1671840	1681160	elections that weather affects who comes out for elections. There will be data about COVID,
1681160	1687080	there will be data about different ways that different jurisdictions have handled vote by mail.
1687080	1694240	There's going to be all kinds of interesting data and we really would love to find people
1694620	1703160	who want to merge in some of that data and build analysis on top of that combination of data.
1703220	1708940	And finally, if you have experience building a successful open source community,
1708940	1716340	it would be terrific to get your input and get your help in building this
1717060	1724320	as a long live open source community. Because we're focused now on 2020,
1724980	1727820	but there's a lot of potential for growth here.