start	end	text
10360	16800	So please join me in warmly welcoming Amanda to the Voting Village.
19450	23150	Thank you. Okay, everyone can hear me? We're good?
23150	27390	Okay, just start flailing your arms if you stop being able to hear me and I'll start yelling.
27790	33050	So yeah, my name is Amanda Glazer. I'm a PhD student at UC Berkeley in statistics.
33090	35270	And today I'm going to talk to you about risk-limiting audits.
35270	39010	So I'll give an overview, essentially, of how they work at a high level.
39010	45530	And then I'm going to talk about some recent work that we've done to make risk-limiting audits efficient for all contests.
45530	51650	So this has been a really big advancement because it's made it super feasible and efficient to audit every single contest in an election.
52290	56050	So I'll go through that at the end and also in a couple hours I'm going to have a demo.
56050	61370	So all of you will be able to see how an audit works and do it yourself with a deck of cards and some dice.
62530	69970	So this research is... and everything about this I'm going to present as joint work with Philip Stark, who unfortunately can't be here.
69970	73330	Am I echoing a little bit? Okay, good, okay.
73530	79190	Philip Stark, who couldn't be here today, he's my PhD advisor and has done tons of work with risk-limiting audits.
79650	82550	Introduced them some number of years ago.
82550	89770	And then another grad student, Jake Spertus, we worked on these methods to audit all contests.
89770	94690	And then the data I'm going to talk about today is from Orange County, so I'll talk a lot about them.
94770	100750	We're super grateful to all the staff in Orange County who have helped us with getting data from the 2020 and 2022 elections.
100750	105650	We were able to go through and simulate and audit on all the contests in those elections.
105650	107650	So I'll talk about that extensively.
110330	112190	Okay, so first, why audit?
112190	117850	I think probably this group of people doesn't need a ton of convincing as to why we should be conducting risk-limiting audits.
117850	120110	But I'll tell you just briefly.
120110	124430	So as you all know, computers are vulnerable to bugs, misconfiguration, hacking.
124650	126590	We've seen it here numerous times.
126590	128450	Things can go wrong in elections.
128850	133490	And even besides that, voters, poll workers, officials, everyone can malfunction.
133830	137590	Whether it's on purpose or not, things can go wrong.
137590	144450	And so generally we want a way to check, especially with something as important as our election outcomes, we want a way to check the results.
144450	150270	And we've seen here at DEF CON year after year just how vulnerable our systems are.
150270	154630	So here's from the report from a few years ago at DEF CON 25.
154810	156510	They said the results were sobering.
156510	162170	By the end of the conference, every piece of equipment in the voting village was effectively breached in some manner.
162190	170970	Participants with little prior knowledge and only limited tools and resources were quite capable of undermining the confidentiality, integrity, and availability of these systems.
170970	178890	So year after year here at this conference, we've seen every single one of these machines be able to be hacked, taken advantage of.
179790	184470	And again, we have the same sort of thing, a report a couple years later at DEF CON.
184550	188670	Same thing, over 100 machines, all of which were hacked into.
189150	197310	And so because we have these vulnerabilities coupled with things that are less potentially malicious, just human error, device malfunctioning,
197310	201850	we really need a way to check the election results, especially something so important.
203410	205690	So that's what I'm going to be talking about today.
206590	214490	The point of election audits is to provide voters with evidence that the reported winner in your election actually won the election.
214490	218110	So we can't take that for granted. We want a way to check that result.
218110	227570	So one thing that's super important here that I'm going to say over and over again in this talk is we have to have a voter-verifiable paper audit trail in order to do this.
227850	237070	So if we don't have paper from the election that we trust, that we know is kept secure, we could be auditing something where something already went wrong.
237070	241170	So if we're auditing paper where the results were flipped, doesn't do any good.
241250	244730	Because we could check all the paper and still get the wrong outcome.
244730	249470	So we have to know that this paper that we're checking, the ballots that we're checking, is correct.
249470	252390	If we went through and looked at all of them, we would get the right answer.
252390	255670	So I'm going to keep saying that because this doesn't work unless we have that.
255670	259730	The paper is kept secure. That's a huge prerequisite for all of this work.
260470	263550	So if we have that, then the last thing is we can check it.
263550	267330	And that's where we get those post-election audits, which I'm going to talk about today.
267330	273150	So this is essentially like a mathematically, statistically rigorous way to check election results.
275270	278190	So there's a few ways we could think about doing this.
278190	281370	So the first idea is we could do a full hand count.
281370	285270	So we could literally look at every single ballot card in an election.
285330	288170	Some people are shaking their heads because you know it's a lot of work.
288670	292070	And it's expensive, it's time intensive.
292070	297950	We could do it if we know that that paper was kept secure and we have access to it.
297950	302690	But it takes a lot of time, it's really expensive, it's hard to do regularly.
302690	304330	But that is one option.
305170	311290	The second option, which a lot of states do some form of this, is the statutory audits.
311290	314330	Which essentially you're looking at some fraction of the ballot.
314330	320510	So New York had a law where they would look at 3% of the machines in an election.
320510	323770	So not even 3% of the ballots, just 3% of the machines.
323770	327550	What you can think about what would go wrong there is if you think about it,
327550	329610	you can be like, okay, I'm going to target one machine.
329670	334410	There's a 97% chance they're not going to hit that one when they do this audit.
335010	339890	Or some states like California, they'll look at some fraction, like 1% of the ballots.
340070	344270	But another problem with those is a lot of times they don't tell you what to do once you look at the ballots.
344270	347510	So you look at 1% of the ballots, you look at 3% of the machines,
347510	350630	and you're like, okay, well, this doesn't quite match what I thought.
350630	353010	But there's not always clear next steps.
353090	355390	You don't know what to do. When you go to a full hand count,
355390	359930	how do you know if it's something that's outside of what it should be?
360130	364310	So these are super common, but they don't really give us the statistical guarantees
364310	367090	or any sort of more confidence that we would want.
367630	371470	So this third idea is Grist limiting audits, which is what I'm talking about today.
371470	374850	And so this is where we actually have statistical guarantees.
374850	379550	We know if the outcome is wrong, we have a certain percentage chance of correcting it,
379550	382310	and we're never going to reverse the correct outcome.
382450	385010	And so this gives us the type of guarantee we want.
385370	389210	It's much more efficient than full hand counts, but it will go to a full hand count
389210	392750	if what you're seeing is unusual, which is what you would want.
395270	398790	So essentially, you know, in one sentence, the goal of a Grist limiting audit
398790	402350	is to provide convincing evidence that the reported winner really won
402350	405590	or corrects the reported results if not.
405850	409150	And again, a Grist limiting audit, it's never going to correct an outcome
409150	412410	that's already correct if the reported outcome is true.
412410	416530	It's never going to overturn that, and it's going to have some pre-specified chance
416530	418350	of correcting a wrong outcome.
418950	421350	So that's what a Grist limiting audit does.
422570	425250	Okay, I told you I was going to say it a bunch of times.
425250	429190	I'm going to say it again. We have to have a trustworthy audit trail to do this,
429190	433650	otherwise it does not work. If the paper is wrong, even looking at all of it,
433650	435470	it's not going to give you the right answer.
435650	439970	So trustworthy means an accurate tabulation would show you who really won.
439970	443270	If you looked at all the paper, you have to know you're going to get the right results.
443270	446070	So you have to have that in order to do a Grist limiting audit.
447650	453390	You want it to be hand-marked paper ballots, voter verified, kept secure.
454190	457070	And then once we have that, we can check the reported results
457070	461190	by manually inspecting some or all of those paper ballots.
465240	469620	Okay, so once we have all that, we can check whether the reported winner really won,
469620	472820	which is conducting an RLA, or a Risk Limiting Audit.
473280	478600	And so the risk in Risk Limiting Audit is a chance of not correcting the outcome if it is wrong.
478600	480680	And so that's something you would specify in advance.
480680	484660	So you could say, I'm going to do a Risk Limiting Audit with a 5% risk limit,
484660	491400	and that means if the outcome's wrong, there's a 5% chance you wouldn't catch it, at most.
491420	496180	But there's a minimum 95% chance that you would correct that wrong outcome.
496180	499880	And you can specify that. You do specify that in advance.
499880	505140	So if you want it to be more careful with it, you can say, okay, we want a 1% risk,
505140	510140	versus you don't want that, you can say 10%, or whatever you want there.
511080	516440	They've done a variety of risks just in different simulations and different pilot audits,
516440	518160	so there's not a standard for that yet.
518160	523560	You see 5% come up a lot, but there's not a specific standard set for that yet.
524540	528840	So yeah, again, RLAs have a known minimum chance of correcting the outcome if it's wrong,
528840	530860	and never change the correct outcome.
531360	535740	And then the great thing about this is if the outcome is right, which hopefully happens a lot of the time,
535740	539120	you don't actually have to look at that many ballots. So it's pretty efficient.
539120	543720	You can know with high confidence that the outcome's right, and not actually have to look at that many ballots.
544040	549960	And so this is where statistics comes in, is that it allows us to use these statistical methods and tools,
550360	554860	which I'm not going to go into in great detail, but if you're interested, you can ask me about later.
555400	560460	And it uses these tools to guarantee that you have that large chance of correcting a wrong outcome,
560460	563400	and with that guarantee, it minimizes the workload.
563400	567080	So we want to look at as few ballots as possible to get that guarantee.
568960	571760	So like I was saying, you don't always have to look at a lot.
571760	576460	So for a ballot-pulling RLA, which is far from the most efficient type of RLA,
576460	581620	if you have a margin of 10%, it has an expected sample size of 470 ballots.
581620	586800	No matter how many ballots are in the elections, you could have a million ballots,
586800	589300	and it would still probably only have to look at a few hundred.
589500	592880	And again, this is one of the least efficient types of RLAs.
592880	599000	And we'll see an example with Orange County, where really you have to look at a pretty small fraction of the ballots for this to work.
599000	605960	So that's great. So we're looking at a small fraction, we have a statistical guarantee, we're in a good spot.
606220	613040	And if you want to play around with this yourself, this website has some tools where you can put in different margins, numbers of ballots,
613040	616100	and you can see for yourself what the sample size would be.
616100	622200	So this specific link is for ballot-pulling RLAs, but if you just not put in the last part of it,
622200	625880	you can see for other types of RLAs what the sample size would be.
629060	633260	Okay, so that's kind of the basics on what RLAs do.
633460	640720	So the next step from this that we were working on is thinking about, okay, we want to audit all contests ideally,
640720	643940	because we don't want to just be looking at a specific set of contests.
643940	646800	We don't want people to know which types of contests we'd be looking at,
646800	651180	because then they could know, oh, they're just going to look at these, so I'm going to change the results in this way.
651180	655100	We want to be looking at everything, so we know our outcome is correct.
655140	660100	And currently, no jurisdiction mandates RLAs of every contest in every election.
660680	666580	However, one of the recommendations in the 2018 National Academies of Science, Engineering, and Medicine report,
666580	673160	Securing the Vote, said that states should mandate risk-limiting audits prior to the certification of election results.
673320	678340	Like we were talking about with the current technology, this requires the use of paper ballots,
678340	683620	and that they should do this within a decade, and begin pilot programs, which we've seen in some places.
683800	688540	And then this last part, that they should be conducted for all federal and state election contests,
688540	690680	and for local contests where feasible.
690680	694900	So this is a recommendation that we do it for all contests.
694900	699260	And this is something that is definitely feasible, and I'm going to show you why today.
701020	703480	So the question is, how do we do this efficiently?
703480	707440	Because we don't want to be in a place where we're like, okay, we audit contest one,
707440	710760	we look at all these ballots, and then we do contest two, and we keep going.
710760	716660	We want to do one sample for a jurisdiction, and be able to audit all the contests at once,
716660	717980	pulling one sample.
718980	722860	So this is our big contribution of late, is a method that takes advantage,
722860	726080	specifically, of that overlap of contests on cards.
726220	730480	So the idea here is, if we know on a ballot card which contests are there,
730480	735180	then we can take advantage of that and know, okay, well, we pull this card,
735180	738660	we can look at all these contests and put that into our calculation,
738660	740580	and see where the risk goes after that.
740580	743100	And I'll show a bunch of examples of this.
745040	748400	So how we know this is something called card style,
748400	751580	and so that's a collection of contests on a ballot card.
751580	755400	So here's an example where we would see, for this first ballot card,
755920	761180	it's in card one, tray four, it's the 96th ballot card in that tray,
761620	765340	and we see it has the California governor's race on it,
765340	767500	but not the mayor of Irvine race.
768140	770160	And then there would be other contests, yes and no.
770160	774120	And then some of these ballot cards do have both the governor and mayor race.
774160	778620	And so this is the type of thing we need to be able to efficiently audit all these contests,
778620	782540	is we need to know, on these ballot cards, which contests they contain,
782540	785100	because that's going to allow us to target our samples.
785360	787180	And that's where it's going to get more efficient.
787180	789860	Because otherwise, if you don't know where the ballot cards are,
789860	792160	even looking at one contest is going to be hard,
792160	795380	because you're going to have to first find where those contests are.
799120	801460	So like I was saying, without this card-style data,
801460	804480	you just don't know which ballots to target to audit a contest.
804720	806820	So you could think about, you know, if you're in a county,
806820	810820	and you're looking at a smaller contest, like for school board, for example,
810820	813840	and you have all those ballots together for a county,
813840	817280	well, just finding those ballots first is going to take a lot of work.
817280	820200	And especially if the margin's small for that contest,
820200	822920	you could easily end up looking through all the ballots.
823240	824460	And so that's just not efficient.
824460	827960	It totally, you know, makes it much less feasible to conduct these.
828160	832360	So for example, if we have a contest that's only appearing on 1% of ballot cards,
832360	835260	then you would think, you know, you're only going to randomly sample it,
835260	837780	on average, one out of every 100 cards.
837780	840840	So this is going to up your workload really, really fast.
842240	846140	So what this card-style data does is it allows us to actually target cards
846140	847640	that contain the contest.
847640	852860	We're not looking at a population, essentially, of cards that don't contain the contest.
852860	856840	We can target it specifically there, and that's going to make the workload go way down.
857460	861660	And it's also going to allow us to take advantage of contest overlap on ballot cards,
861660	867340	so we can audit all the contests at once and take advantage of that overlap there.
868560	873420	So here's a couple examples of the types of overlap we might see just with two contests.
873420	875700	So this is kind of a simplified example.
875700	878640	So we have a big contest and a small contest.
878640	882160	So it could be where the big contest is something like the president,
882160	887000	and then the smaller contest is the mayor, which only, you know, a particular city is going to have.
887000	891480	And so the small contest is going to be on every ballot the big contest is on,
891480	896480	but big contest isn't necessarily going to always have that small contest on the ballot.
897500	901920	They could not overlap at all, so it could be two different mayor races for different cities,
901920	904700	and you just never are going to see them on a ballot card at the same time
904700	907060	if one city is bigger than the other.
907700	912980	And then we could have partial overlap for things like maybe school board and propositions,
912980	917440	where, you know, you're going to see them together on some people's ballots, but not all.
917500	922660	So there's a lot of different types of overlap, and we want to be aware of all of these and take advantage of it
922660	926960	so we know when we pull a card, we're using all the information on the card.
930700	935460	Okay, so what we do to do this is something called consistent sampling.
935460	938960	And so here's like a very simple example of how it works.
939260	944200	So you basically have all these ballot cards, in this case we have a big contest and small contest,
944200	949200	so just two contests again. Sometimes a big contest appears on a ballot card alone,
949200	953220	sometimes with a small contest, sometimes small contest is alone.
953340	957640	And so you can imagine we take all these ballot cards, in this example there's not that many,
957640	962000	we essentially shuffle them up, in this case we're giving them a random number between 0 and 1
962000	964940	and then ordering them, that's our way of shuffling them.
965580	969780	And then we know from the math behind the risk-limiting audit in this case
969780	974500	that we have to look at 5 cards containing B and 2 cards containing S.
975040	980280	And so we can go here and look, okay, 2 cards containing S, the cut line's here,
980280	985920	we get this ballot card and this ballot card, but the nice thing is that this we can use for both.
985920	989900	So we need 5 for B, and so the cut line's this solid line here,
989900	993160	and we take advantage of 2 that also have S on it.
993160	998180	So it allows us to basically, again, a lot of it's like kind of just keeping records,
998180	1001900	keeping track of what you're doing and being able to take advantage of that,
1002420	1007180	and use the overlap. So we're using the cards that contain both, we know they contain both,
1007180	1010980	we're just targeting the ballot cards that would contain either of them,
1010980	1013620	and we're creating a random sample from that.
1014140	1020060	And so this is called consistent sampling, and this is what we do in order to efficiently audit all the contests.
1021660	1025240	So I'm going to give an example from Orange County.
1025540	1029140	So Orange County, it's the third most populous county in California.
1029460	1033340	Here it is here, it's basically sandwiched between LA and San Diego.
1034480	1040980	A little over 3 million people as of the 2020 census, and close to 2 million active voters.
1041120	1047640	And so it's a pretty big area, and I think simulating and piloting audits in Orange County
1047640	1050600	is going to be analogous to a lot of U.S. states.
1051120	1054660	It actually has more registered voters than 24 U.S. states,
1054660	1061460	and it's the country's fifth largest election jurisdiction after LA, Maricopa, Harris, and San Diego.
1061560	1065520	So it's a really, really large election jurisdiction.
1065960	1072260	And it has 2,204 precincts and approximately 181 voting centers.
1072900	1079020	And so we were able to work with Orange County and get data from the November 2020 and 2022 general elections
1079640	1085080	to estimate how many cards that we would need to inspect to do an RLA with and without style information.
1085680	1088480	So we had worked out kind of the math and some toy examples,
1088480	1093020	but this was a larger scale simulation where we could actually see,
1093020	1097440	okay, how much difference does this make in something that's going to be on a larger scale,
1097440	1100120	you know, really comparable to a lot of U.S. states.
1101320	1102980	And so here's what we found.
1102980	1109440	So for 2020, we had about 1.5 million voters, a little over 3 million ballot cards.
1109440	1114500	So this was a two-page ballot, and there were 181 contests.
1114880	1119940	And not taking into account style, we would have had to do a full hand count to audit all the contests.
1119940	1124860	And the reason this is is there was, I believe, one contest that was a tie in this election,
1124860	1127040	and so you're going to have to look at all the ballots for that.
1127040	1129860	I think that intuitively makes sense, hopefully.
1130400	1134460	And it was a relatively small contest, and so because of that,
1134460	1137440	if you don't know where the contest is, you're going to have to look at everything,
1137440	1139560	because you need all the ballots from that contest.
1140940	1144560	So we're going to have to do a full hand count in a lot of cases with these small margins
1144560	1146840	when we don't know where the ballot cards are.
1147720	1153860	With style, auditing all the contests, their sample size goes down to 20,112.
1153860	1156880	So that's 0.6% of the ballot cards.
1156960	1159700	This is a massive reduction, and if you think about some of these laws
1159700	1164700	that are already looking at 1% or 3% of the ballots, this is less than that.
1164700	1168420	And we get a statistical guarantee here auditing all the contests.
1168420	1169840	So this is pretty major.
1171100	1174700	And then let's say we want to get rid of some of these smaller margin contests,
1174700	1178700	because we think, okay, probably they're going to do a full hand count on these contests anyway,
1178700	1183200	so let's just ignore them and see how that reduces our sample size.
1183200	1190040	And if we omit that one contest that was a tie, our sample size goes down to 15,964.
1190040	1191500	So half of a percent.
1191760	1196280	And if we omit the four contests that have margins less than half a percent,
1196280	1198940	our sample size goes down to under 10,000.
1199060	1201520	And we're only looking at 0.3% of the ballots.
1201920	1205380	So this is what I was saying before, that you really don't have to look at that many ballots
1206140	1208400	relative to the size of the election.
1208400	1213620	We have over 3 million ballot cards, and we're down to looking at 10,000, 20,000 of those
1213620	1217980	to make sure we have this rigorous guarantee in all contests.
1218940	1221900	And we see something similar with the 2022 data.
1221900	1224140	So here we had slightly less voters.
1224140	1231560	We had 994,227 voters cast close to 2 million ballot cards in 214 contests.
1231560	1233640	So slightly more contests here.
1233960	1236060	Again, you don't take into account style.
1236060	1237840	You're going to have to go to a full hand count.
1237840	1243680	There's a lot of little contests here with relatively few ballot cards, small margins.
1243680	1247020	You have to find those cards, so you have to go to a full hand count.
1248060	1253380	If we audit with that style data, taking into account all the overlap, all contests,
1253380	1256840	we're looking at 62,251 ballots.
1256840	1259680	So that's about 3% of the ballot cards.
1260300	1263320	And then again, if we omit some of these small margins because we think,
1263320	1265820	okay, they're going to go to a full hand count anyways,
1265820	1269200	we're down to 1% and almost half a percent.
1269640	1272480	So we reduced the sample size pretty dramatically.
1273680	1278440	And so this is great, because this is relatively little workload compared to, you know,
1278440	1279680	having to look through all the ballots,
1279680	1284900	and it suddenly makes it really feasible to conduct an audit of every single contest in an election.
1288560	1289980	So, yeah.
1289980	1292780	So the crux of this is that RLAs, hopefully I've convinced you,
1292780	1295780	are really important for ensuring elections are trustworthy.
1296320	1299320	The important thing about them is they have this known minimum chance
1299320	1303020	of correcting the outcome if it's wrong, and they're never going to change a cracked outcome.
1303560	1307440	And using the card style data, we can make these RLAs efficient and feasible.
1307440	1311080	So we saw with Orange County in the 2020 and 2022 general elections
1311080	1315440	that we would only have to inspect half a percentage or 3% of the ballot cards,
1315440	1320220	and that goes down even more if we take out some of those small margin contests.
1320740	1325020	And I'm going to be doing a demo with cards and dice, like I was saying earlier, at 3.30.
1325100	1329140	It's interactive, so we'll get into groups and everyone can see how an RLA works,
1329140	1331900	because we can all do it. It's a pretty cool process.
1331900	1335840	It's mainly a lot of accounting and looking for ballots and random sampling.
1335940	1339880	So this is something that every jurisdiction can be doing, that we should be doing.
1339880	1341900	It can give us more faith in the process.
1342040	1346160	That coupled with having those hand-marked, papered ballots that are kept secure.
1347580	1353220	But yeah, hopefully you all will see at 3.30, and I will take any questions now about RLAs,
1353220	1357880	this taking into account overlap, or anything else anyone wants to ask about.
1358020	1358600	Yes?
1362480	1366760	Yeah, and other people might be able to speak better about the security of the ballots in Maine,
1366760	1372160	but I mean, essentially, you want to know that people don't have access to them,
1373220	1377200	you can't lose ballots, no one's going to be able to mess with the ballots in any way.
1377200	1381260	You want to know, at the end of the day, the ballots that you're auditing,
1381260	1385060	if you were to do that full hand count, you would get the right outcome.
1385060	1387440	So that they're not tampered with in any way.
1387760	1390060	So the security of that is really important.
1390060	1393760	And there might be others that know a little bit more about on the ground what the security...
1406530	1411510	So, yeah, a lot of things, too, are done like a jurisdiction or county level.
1411510	1414150	So that's, I think, too, part of what makes elections a little chaotic,
1414150	1417390	because it's not like there's not one centralized system for this.
1417390	1421330	It's a lot of, you know, different jurisdictions doing different things.
1421330	1424890	And so it makes it a lot tougher.
1428860	1433940	So you audit the ballots that were received and processed.
1433940	1444740	And sometimes there are ballots that are returned as undeliverable because the address doesn't exist.
1444740	1453520	So there's no way that you would be able to clarify or determine if those ballots had found their way into being voted.
1453600	1461880	Yeah, so essentially a requirement of doing this is we need those hand marked paper ballots that we know are accurate.
1461880	1470600	So if there's any ballot that was, like, returned or ambiguous or that they chose not to include, you know, we couldn't do anything about that.
1470980	1478860	There are specific methods we can do to, like, if some of the records don't line up, like the card style data and the ballots we have,
1478860	1484420	we have ways where we can essentially, like, add in buffers to make it a little bit harder to, like, verify outcomes.
1484420	1488040	So if there's, like, some ambiguity, there's ways to deal with it statistically.
1488040	1490380	But we really do need, like, a secure...
1490380	1503680	No, I'm wondering about ballots that are undeliverable and make their way into the voting system and get counted.
1503680	1509500	Oh, they do get counted. Okay, so if they're counted and it's something we can go and look at,
1509500	1514580	then there's a couple different ways we could deal with it depending on, you know, what it looks like.
1514580	1524760	So we can assume a worst-case scenario there and essentially, you know, not let it add evidence to helping the audit stop.
1524760	1530840	So there's different kind of methods around that. If something is, yeah, undeliverable, there's any sort of ambiguity,
1530840	1533340	we can basically assume a worst-case scenario there.
1533340	1539400	Well, in Orange County, there's about 50,000 undeliverable ballots that are returned to the ROV.
1539400	1540720	Yeah.
1547950	1555710	Okay, so we're inspecting paper because I know sometimes RLAs can be done on ballot images too, right?
1555710	1556730	Yes.
1557030	1567910	So this requires preservation of the physical ordering of the ballots as they were counted through the machines?
1567910	1578230	So essentially, what you need is... that's a great question. So when we're talking about like card style here,
1578230	1582030	so we need to know where the ballot cards are and what contests appear on them.
1582030	1586690	So there's this question of, okay, well, how do we get that? Where does that come from?
1586750	1588990	And so there's a few different ways.
1588990	1595370	So one, if you're in an area where every precinct, for example, has the same ballot,
1595370	1601030	you might naturally have some of that information and you would still need information about the ordering to do the random sample,
1601030	1603610	but at least you would know what contests are on which ballots.
1604370	1610790	Outside of that, you can get information from that initial scan through of the ballots.
1610790	1615210	You can construct the style data from that.
1615210	1619670	And then there's different safeguards you can put into place to check that the tallies line up.
1619670	1624410	And if they don't, again, adding in some buffer to make things a little bit harder to verify it.
1624410	1631310	But we have to be able to physically locate the actual ballot that's represented by that piece of data.
1631310	1634950	Is that practical in a large environment?
1635550	1643210	Like, especially if you have, they have, Orange County said they have like 10,000, or no, how many precincts they have?
1643210	1647610	So Orange County, and again, this is one of the larger ones.
1647670	1649370	Yeah, so they have a lot of precincts.
1649370	1650730	And a lot of languages.
1650730	1654470	So they've got like a ton of ballot styles, right?
1654470	1658310	Yeah, I mean, so part of this, so there's kind of a couple sides of this.
1658310	1664250	So one on our side doing, you know, the statistics, we're thinking, okay, how can we make this as efficient as possible?
1664250	1668530	Meaning, you know, reducing the workload as much as possible while still getting those statistical guarantees.
1668530	1671250	Because it is effort to grab the paper.
1671250	1674710	And it's, you know, a massive organizational challenge as well.
1674710	1680210	So that's kind of the first thing is, how can much, can we reduce the sample size while still getting that statistical guarantee?
1680630	1688190	And then the other side of it is just, you know, making sure things, and this kind of falls more on specific, you know, jurisdictions,
1688190	1692290	to make sure things are organized in a way where it's as easy as possible to pull the paper.
1692390	1695470	And it is, you know, there's no way to get around some amount of work with this.
1695470	1700550	Because, you know, to verify a random sample of ballots, it is going to take some work.
1700790	1704150	But there's not a great way around that.
1704150	1705370	All right, thank you.
1708720	1712220	So, how do you guarantee a random sampling?
1712280	1721180	And the situation that goes through my mind is, in our county, we have a factory that lets out at 3.30 p.m. and everybody goes to the polls.
1721180	1721580	Yeah.
1721580	1730620	If we grab from that particular section, the people that work at that factory are very heavily leaning towards one party versus the other.
1730620	1730740	Yeah.
1730740	1733460	So wouldn't that potentially skew the results?
1733460	1739860	Yeah, so essentially, so you're not going to, like, look at, you know, the first chunk of ballots or a specific chunk.
1739860	1746980	We'll see in the demo, for anyone that stays, this is going to be, like, on a smaller scale with 65 cards or ballots.
1746980	1748340	We're going to roll dice.
1748340	1753120	And then so we're going to pick, like, okay, the 13th ballot, you know, the 14th ballot, the 25th.
1753120	1763160	And so essentially, you can use, like, a random number generator of some sort, you know, kind of a larger scale version of rolling dice to then randomly be picking these ballots.
1763160	1769660	And we have, like, different kind of code and software in place to do this and select a random sample of ballots.
1769660	1778640	So that was part of this work is we have all the code where it will tell you, you know, the expected sample size, which ballots you need to pull, randomly generate those.
1778640	1788380	So it doesn't matter, which you'll see in the demo, too, what order the ballots are in as long as you're randomly picking them is the big part.
1795950	1796890	Thank you.
1796890	1802070	I also have the dice, actually, Philip Staub gave me for the first audit here.
1802590	1803570	I had a question.
1803570	1810850	So when you have to draw ballot number 2700, and you know that the ballots are all organized in stacks of 500.
1810930	1811250	Yeah.
1811250	1816930	Is that okay to just say, like, okay, here are five stacks and then go to the last stack and get one out?
1816930	1821330	What if they're, like, off by plus minus one, the other stacks?
1822310	1824070	Yeah, that's a great question.
1824070	1828210	So this is another thing, too, we'll see in the demo, because we're going to make, like, six stacks.
1828210	1832730	And so a lot of this is, like, very careful organization and accounting work.
1832730	1840030	So you would have a ballot manifest, and it's going to tell you exactly which stack it's in, which number in this stack.
1840030	1842110	So you don't have to, like, go through and count every stack.
1842110	1849890	You should, you know, at one point that's done, and then you can know, okay, I need to go to this stack, this tray, this ballot within there.
1849890	1855270	So, yeah, a lot of this is, you can see, like, a lot can go wrong at a lot of steps.
1855270	1863990	But it's, like, very careful accounting and organizational work to make it as easy as possible to know, okay, I need to go here and get this number of ballot from there.
1871060	1875180	How often do they lay the ballots out that way for the results to be?
1875340	1876460	Do you understand what I'm asking?
1877200	1878160	Like, how often?
1878160	1881220	Like, you're saying they have the stack, the tray, and then the ballot within that.
1881220	1884640	Is that a normal way that they keep results if there's paper ballots?
1885080	1891940	Yeah, I think... I don't know for certain, like, the numbers of how often... pardon?
1891940	1893260	Only if they're doing an RLA.
1893780	1894560	Not if they're...
1894560	1895840	Only if they're doing an RLA.
1895840	1900260	Yeah, so, I mean, this becomes, like, really, really essential when you're doing an RLA, right?
1900260	1903560	Because then you need to know exactly where to go.
1903560	1905540	So, I mean, there's a mix.
1905540	1914720	But, you know, if you're doing an RLA, this sort of system of knowing, okay, cart, tray, position, and tray,
1914720	1918680	you have to have some sort of information like that, because you have to know where to go.
1918680	1921360	And, like, you know, the exact details of it won't vary.
1921360	1925200	But, yeah, to do an RLA, you'll have to know where to go.
1930620	1932120	Hello, thank you for the talk.
1932120	1938540	I was wondering, when you're, I guess you would say, pitching this or talking to other local election officials about this,
1938540	1940920	what kind of resistance do you get from people?
1942000	1945700	Yeah, so, yeah, it's pretty varied.
1945700	1951540	I mean, some people, of course, like the people in Orange County, great, you know, very, very helpful, you know,
1951540	1953900	want to do everything possible to implement this.
1953900	1956920	There's a variety of things. I mean, we can see this all over the news, too.
1956920	1959860	I mean, people don't like admitting that something could go wrong.
1959860	1963660	It's a lot of work, you know, what we were talking about, having to pull those ballots.
1963660	1970540	I mean, in some sense, there's not a way around the fact that you're going to have to pull 20,000 ballots or some number if you want to do it well.
1970740	1975140	So the workload, not wanting to admit there's anything wrong with their system.
1975240	1977720	I think that's a big thing, just like we see.
1977720	1981120	I think we see this a lot now, just like denial that anything could go wrong.
1981200	1983060	Everything's fine. Everything's secure.
1983480	1985660	So I think that's a big one.
1987560	1994000	Yeah, I mean, it's surprising, because I know, like, when I first learned about all this, I was like, wow, you know, this seems so obvious,
1994000	1999460	especially with something like our elections, where, you know, we keep repeatedly seeing issues come up that are very public.
1999580	2001100	We need something like this.
2001100	2006540	But, you know, a lot of people don't feel that way, obviously, because, you know, we don't have it everywhere yet.
2006540	2008580	But hopefully with time.
2011460	2016100	All right, please join me in thanking Anda for a great talk.