[00:00.000]Hi, I'm Philip Stark. Thank you for coming to this virtual talk. I'm going to talk about testing [00:06.320] [00:06.320]ballot marking devices. This is joint work with an undergraduate student, Ron Shi, who visited [00:11.600] [00:11.600]Berkeley for the last year and did some research with me. Why do we need to test ballot marking [00:18.600] [00:18.600]devices? Well, they can print votes that differ from what the voter saw on the screen or heard [00:23.380] [00:23.380]through the audio interface. The idea of voter verifiability is not really refined enough to [00:29.280] [00:29.280]capture the security properties that we need in a voting system. In particular, the ability to [00:35.500] [00:36.080]catch an error and spoil the ballot and request another opportunity to vote isn't enough to make [00:41.520] [00:41.520]ballot marking devices safe voting technology. For example, recent research by Bernard et al. [00:47.140] [00:47.140]showed that only 7% of voters notice errors that ballot marking devices have introduced [00:53.320] [00:53.320]into the printout. In effect, the security properties of paper are undermined by using [01:00.900] [01:00.900]ballot marking devices to mark the paper. There's a problem with the BMD security model in general. [01:07.200] [01:07.200]It basically makes voters responsible not only for their own errors, but also for the overall [01:11.920] [01:11.920]security of the system. But they don't give voters the tools they need to do that job. In particular, [01:17.060] [01:17.060]there's no way for a voter to present any other party, including an election official, with [01:21.600] [01:21.600]evidence that a BMD misbehaved. So if a voter complains to a local election official, there's [01:28.000] [01:28.000]no way for the election official to know whether the complaint reflects an actual malfunction, [01:32.520] [01:32.640]a voter error, or a cry of wolf trying to undermine the trust election. As a result of that error or [01:39.180] [01:39.180]malfeasance could change a lot of votes without raising any kind of alarm. A number of proponents [01:44.560] [01:44.560]of BMDs claim that they have benefits such as preventing overvotes, warning about undervotes, [01:50.760] [01:50.760]eliminating the possibility of ambiguous marks. I think that there's problems with those arguments. [01:56.420] [01:56.420]In particular, they assume that ballot marking devices function correctly. And there are many [02:00.780] [02:00.780]recent examples of failures on a wide scale, including in the state of Georgia, Northampton, [02:05.320] [02:05.320]Pennsylvania, and in Los Angeles. Precinct count optical scan systems can also protect against [02:11.040] [02:11.040]overvotes and undervotes. In fact, that's required under BBSG 1.0. So how can we figure out whether [02:17.820] [02:17.820]ballot marking devices actually worked adequately in a given election? We need to know that whatever [02:22.540] [02:22.540]errors occurred weren't numerous enough to change the outcome of any contest in the election. [02:28.860] [02:28.900]Three different approaches have been proposed to testing ballot marking devices. [02:32.580] [02:32.620]One is pre-election logic and accuracy testing, where you look at a machine before election day, [02:37.940] [02:37.940]run some test patterns through it, and verify that it prints the right thing. Another approach is [02:42.480] [02:42.480]passive testing, where you look at something like the spoiled ballot rate and try to detect [02:46.320] [02:46.320]anomalously high rates of spoiled ballots as a possible signal that the machines are misbehaving [02:51.260] [02:51.260]and voters are catching it. And the third approach is parallel or live testing, where [02:55.840] [02:55.840]testers periodically throughout the day, on election day, or during early voting period, [03:01.920] [03:01.920]will mark some ballots but not cast them, and verify that what's marked on the printout matches [03:07.920] [03:07.920]what their intent was. And the point of our research is to show that none of these, in fact, [03:12.620] [03:12.620]work in practice. Now, how much testing do we really need to do? That depends on how big a [03:18.540] [03:18.540]problem would make a material difference, and I have argued a long time that a sensible threshold [03:24.300] [03:24.300]for materiality is enough to change the reported winner of one or more contests. That is, we'd like [03:29.040] [03:29.040]to have high confidence that whatever errors occurred, they didn't alter who won. Many contests [03:35.000] [03:35.000]in the U.S. are decided by less than one percent, even statewide contests. For example, in 2016 [03:41.160] [03:41.160]presidential election, the margin in statewide contests in Michigan, Rhode Island, Pennsylvania, [03:48.160] [03:48.160]and Wisconsin were all under one percent, with Michigan being as low as 0.22 percent. [03:53.280] [03:54.100]Now, I'm going to frame this as a two-person adversarial game and think about what strategies [03:59.380] [03:59.380]are available to the two players. So the evildoer is Mallory, who's trying to alter the outcome of [04:04.740] [04:04.740]one or more contests in an election. Mallory doesn't want to be detected. The point of this [04:09.140] [04:09.140]isn't to cast fear, uncertainty, and doubt. It's to get away with altering an election outcome. [04:13.740] [04:14.100]Mallory knows how the ballot marking devices will be tested in general because that will [04:20.060] [04:20.060]be a matter of public record. That is an action taken by the local election official. [04:24.240] [04:25.800]So Mallory knows the state history of the machine. Mallory knows how voters are [04:33.800] [04:33.800]interacting with it and knows what votes have been cast earlier in the day, how long each voting [04:37.780] [04:37.780]session took, and so on. And Mallory has a good model of voter behavior because Mallory can [04:43.540] [04:43.540]basically install spyware on voting machines and keep track of how voters interact with those [04:47.580] [04:47.580]machines in previous elections and on into the future. In contrast, Pat is our tester. Pat is [04:53.820] [04:53.820]trying to make sure that any ballot marking device problem that alters one or more outcomes will be [04:58.480] [04:58.480]detected. In contrast to Mallory, Pat has to obey the law, has to protect voter privacy. Pat doesn't [05:04.940] [05:04.940]know which contest Mallory will attack nor the strategy Mallory is going to use to attack them. [05:09.340] [05:09.340]So this is a very asymmetric problem. All right, so because the threshold for materiality depends [05:15.640] [05:15.640]on the number of votes that it takes to alter an election outcome, it's important to keep track of [05:21.580] [05:21.580]how big or how small elections are in the United States. The median turnout by county in 2017, [05:30.300] [05:30.300]not 3017, oh sorry, in the 3017 U.S. counties that there are in 2018 was a little under 3,000 voters. [05:39.340] [05:39.740]There are fewer than 43,000 voters in more than two-thirds of U.S. jurisdictions. [05:45.620] [05:45.980]And in 73 percent of states, more than 50 percent of counties have fewer than 30,000 active voters. [05:53.560] [05:53.560]That is, the median size of turnout in a county is 30,000 voters or fewer. In 92 percent of states, [06:04.360] [06:04.360]that number is 100,000. That is, more than 50 percent of counties have fewer than 100,000 [06:09.760] [06:09.760]active voters. In 2019, only 317 U.S. cities had populations of 100,000 or more [06:18.580] [06:18.580]out of more than 19,000 incorporated places. So if about 80 percent of the population is a voting age [06:25.680] [06:25.680]and turnout is about 55 percent, which is roughly what it's been historically, then contests for [06:31.560] [06:31.560]elected officials in something like 98 percent of incorporated places involve fewer than 44,000 [06:37.620] [06:37.620]voters. So we need to think about ways of testing things on contests that involve fewer than 44,000 [06:43.980] [06:43.980]voters, and many contests will involve even, you know, fewer than 3,000 voters. The 2019 [06:50.320] [06:51.900]median population of U.S. incorporated areas is about 725, so about 50 percent of incorporated [06:58.460] [06:58.460]places have a turnout of less than 320 voters. All right, this is just to give an idea how much [07:06.140] [07:06.140]of the country has a median turnout, had a median turnout in 2018 of less than 30,000 voters. It's [07:13.220] [07:13.220]most of the country by area. So what's Mallory's strategy space? How can Mallory figure out which [07:19.400] [07:19.400]transactions or what votes to try to alter? Mallory basically can pick based on a very large [07:26.920] [07:26.920]number of state variables in the ballot marking device, the time of day, how long the wait was [07:32.880] [07:32.880]between voters, how many people have voted on the machine already, how does this particular voter [07:38.580] [07:38.580]interact with the machine, including the selections, what contests the voter ignores, [07:42.820] [07:42.820]how many times the voter revises selections, how long the voter reviews things, whether the voter [07:48.120] [07:48.120]looks at every page of candidates in a contest, how long the voter reviews selections, inactivity [07:55.180] [07:55.180]warnings, BMD settings, font sizes, languages, whether the voter uses the audio interface, [08:01.240] [08:01.240]the sip and pop interface, all of these things are available to a maldoer, to Mallory, to try [08:06.000] [08:06.000]hack the election. Now here are some examples of just how many different possible voting [08:12.300] [08:12.300]transactions there are. I'm giving two columns of numbers. The more realistic number is pretty [08:18.120] [08:18.120]realistic in the United States. Many jurisdictions have ballots that contain 20 or more contests, [08:26.040] [08:26.040]but we're going to use three as basically a lower bound. And similarly, you can look at different [08:30.780] [08:30.780]variables that Mallory could use to target these things, from the number of candidates per contest, [08:34.980] [08:34.980]time of day, number of people that voted, time for selection, the settings that the voter uses, [08:42.680] [08:42.680]the contrast and saturation of the screen, font size, audio use, tempo, volume, and so on. So [08:48.560] [08:48.560]conservatively, there's on the order of six million different combinations of settings [08:52.920] [08:53.540]that are likely to have some reasonable probability of being used. More realistically, [08:59.820] [08:59.820]there's something over 10 to the 47th, a truly staggering number of possible voting transactions. [09:05.560] [09:05.560]There's no way to probe even a microscopic fraction of those using testing, either [09:11.040] [09:11.040]pre-election logic accuracy or live testing. So what can Pat do? That's what Mallory can do. [09:18.500] [09:18.500]Pat can monitor voter behavior in a non-invasive, non-privacy invading way, in particular can look [09:24.620] [09:24.620]for ballot rates, and Pat can try to catch a malfunction by using the BND before, during, [09:30.460] [09:30.460]or after an election. That is, doing logic and accuracy testing, or live testing, or post-mortem. [09:35.960] [09:35.960]So Pat really does have to test at random in some way. If Pat tests in a way that's [09:41.760] [09:41.760]predictable, such as once an hour, or pulls only one machine aside and tests it, or tests only some [09:47.640] [09:47.640]combinations of votes, only interacts with the machine in some particular way, then because [09:53.060] [09:53.060]Mallory knows what Pat's strategy is, Mallory can just avoid changing those transactions and hide. [09:59.960] [10:01.220]Similarly, Pat can't just set aside machines on election day for live testing. Pat needs to test [10:05.800] [10:05.800]the machines that are actually in use, or malware could detect that the machine is being used in a [10:10.300] [10:10.300]way that is not typical of voters. And moreover, because there are so many possible combinations [10:16.900] [10:16.900]of settings, combinations of ways of transacting a vote, uniform random sampling is doomed. You [10:23.180] [10:23.180]really do need to sample more often from those transactions that voters are going to use more [10:28.840] [10:28.840]often in order to have a reasonable chance of sampling at least once from any set of transactions [10:33.840] [10:33.840]that contain enough votes to alter the outcome of one or more contests in the election. [10:38.020] [10:38.540]So ideal sampling would mimic voter behavior. It would basically sample from what voters actually [10:43.040] [10:43.040]do. So we're going to look at exactly that. Suppose we could mimic voters perfectly, how many [10:49.960] [10:50.520]transactions would we actually need to use as tests in order to have a good chance of detecting [10:54.420] [10:54.420]outcome-changing errors or alterations. So it's important to know that in a jurisdiction-wide [11:01.440] [11:01.440]contest, changing the votes on 1% of transactions can typically change the margin by 2%, but if [11:07.980] [11:07.980]there's undervotes, you can change it by even more than that. And if the contest is only on a fraction [11:12.860] [11:12.860]of ballots cast in the election, then you don't need to change even that large of a percentage to [11:16.960] [11:16.960]change a margin by a much larger percentage. For instance, if you have a contest that's only on, [11:22.400] [11:22.400]that only 1 in 10 voters is eligible to vote in, and the undervote rate is 30%, then changing the [11:28.040] [11:28.040]votes on 1% of transactions could change the margin in that contest by 29%. That's a lot of leverage. [11:33.940] [11:34.940]So passive testing relies on voters noticing errors and spoiling their ballots. Now in order to [11:41.940] [11:41.940]know how large a spoilage rate is enough to sound an alarm, we have to have a good idea of how often [11:47.620] [11:47.620]voters spoil ballots when the machines are functioning correctly, and then we have to know [11:52.480] [11:52.480]how often they will notice errors if errors happen, and whether they will report [11:58.180] [11:58.180]those errors and thereby, you know, request a new ballot and trigger an alarm. [12:03.600] [12:03.660]Problem is that kind of training data is unlikely to be available in part because you can't step [12:08.780] [12:08.780]on the same election twice. There are all kinds of differences from election to election that are [12:13.280] [12:13.280]likely to change the spoiled ballot rate, including complexity of the ballot, ballot layout, [12:17.260] [12:18.240]complexity of the social choice functions, and so on. So how do we set a threshold for one to [12:23.500] [12:23.500]sound an alarm if we're using passive auditing, passive testing? It's going to depend in part [12:29.140] [12:29.140]on the number of transactions Mallory alters, which votes are affected, which contests are affected, [12:33.920] [12:33.920]and so on. And Pat is not going to know any of these things. Pat needs to test in a way that [12:37.640] [12:37.640]is going to be sensitive enough to changing the outcome of any contest whatsoever. [12:42.920] [12:43.600]So let's make some really optimistic assumptions and work through the numbers and figure out just [12:48.020] [12:48.020]how much testing Pat would have to do or how many voters would have to be voting in the particular [12:52.140] [12:52.140]contest so that a change in their spoiled ballot rate would be noticed. But let's assume in [12:57.220] [12:57.220]particular that the spoiled ballot rate follows a spoiled ballots follow a Poisson distribution [13:01.880] [13:01.880]with a known rate if there's no hacking and a different known rate if there is hacking. Now [13:06.020] [13:06.020]there's no reason to assume that except it's a common model for things. This is really just a [13:10.480] [13:10.480]thought experiment. It's not intended to be a realistic model of how voters detect and [13:17.520] [13:17.520]spoil ballots. We're going to look at contest margins of one to five percent and false positive [13:22.840] [13:22.840]and false negative rates of five percent and one percent. So a false positive rate is saying that [13:27.320] [13:27.440]there's a problem when there really isn't one. A false negative rate is failing to notice that [13:31.560] [13:31.560]there's a problem when in fact one or more outcomes have been altered. Now here's kind of [13:36.240] [13:36.240]how it plays out. This is for five percent rate of false negatives and false positives. If you look [13:41.900] [13:41.900]at this, going across the top is the base rate of spoiled ballots when things are clean and then [13:49.520] [13:49.520]as you go from row to row you're really looking at what is the rate of errors in the printouts [13:58.580] [13:58.580]that would be required to reverse a margin of size one percent, two percent, three percent, [14:03.880] [14:03.880]on to five percent. The detection rate we're assuming is either seven percent or twenty-five [14:08.420] [14:08.420]percent. Seven percent is consistent with what Bernard et al. found in their study of actual [14:13.980] [14:13.980]voters in an experiment that wasn't an actual election. So if you look at this to have a five [14:20.580] [14:20.580]percent false positive and false negative rate, you'd need on the order of half a million ballots [14:26.540] [14:26.540]or more for a realistic rate of voters detecting errors and spoiling their ballots to protect [14:33.380] [14:33.380]against altering a contest with a margin of one percent. That number goes down as the margin gets [14:38.540] [14:38.540]wider, but as I've already argued there are a very large number of contests that are decided by one [14:43.420] [14:43.420]percent or less, important contests. If we make a more stringent threshold of requiring only a one [14:49.500] [14:49.500]percent rate of false negatives or false positives, then we would need on the order of a million [14:53.520] [14:53.520]voters or more in the contest in order to be able to detect an alteration to one percent to [14:58.680] [14:59.340]to enough ballots to alter to reverse a margin of one percent. So let's think about how big this [15:06.400] [15:06.400]number half a million or a million is in the context of actual elections. And so let's I'm [15:11.220] [15:11.220]going to use California as an example. 41 of California's 58 counties had fewer than 100,000 [15:16.880] [15:16.880]voters in the 2018 midterm election, so passive auditing would not have worked for any of those. [15:22.300] [15:22.300]33 had fewer than 100,000 voters in the 2016 presidential election, so again passive auditing [15:28.020] [15:28.020]would not have given you an acceptably low false positive and false negative rate, even under these [15:32.700] [15:32.700]optimistic assumptions that everything follows a Poisson distribution with a known rate. [15:37.600] [15:38.340]So passive testing couldn't have protected contests with margins of three percent or smaller [15:43.040] [15:43.040]in those jurisdictions that have a hundred thousand or fewer voters. In many California [15:49.020] [15:49.020]counties turn out so small that there would be no way to detect problems through spoilage rates [15:53.420] [15:53.420]without having an unacceptably high rate of false alarms would be that invalidating elections left [15:58.920] [15:58.920]and right. Okay, so that analysis assumed that votes were being changed more or less at random, [16:09.080] [16:09.080]that every voter had some chance of having his or her votes altered, but in fact Mallory has [16:15.140] [16:15.140]access to information to use to target the attack against voters who are less likely to notice [16:20.500] [16:20.500]problems or less likely to spoil a ballot if there is a problem. In particular, Mallory could target [16:30.000] [16:30.000]voters with visual impairments, voters who are blind, that could, if such voters, current ballot [16:38.380] [16:38.380]marking devices don't provide such voters a technology to check whether the printout actually [16:43.320] [16:43.320]matches the voters intentions or what the voter was told on the audio output or on the screen in [16:50.760] [16:50.760]you know in a larger font size. So if two percent of voters have a visual impairment that would [16:55.560] [16:55.560]prevent them from checking the printout directly themselves, then Mallory could change the outcomes [17:00.240] [17:00.240]of jurisdiction-wide contests that have margins of four percent or more without increasing the [17:04.220] [17:04.220]spoiled ballot rate at all, because those voters would have no opportunity to notice that there [17:08.760] [17:08.760]was a problem in the printout. Voters that have some kind of motor impairments that make it [17:16.460] [17:16.460]difficult for them to, that limited dexterity that it makes it difficult for them to handle a piece [17:21.840] [17:21.840]of paper, some ballot marking devices have an accessibility feature that allows a voter to [17:27.480] [17:28.280]cast the ballot without actually handling it. In some cases those features don't even print the [17:33.900] [17:33.900]until the voter has said I want you to cast this for me after you print it. Now because that doesn't [17:41.220] [17:41.220]even give the voter the opportunity to look at the piece of paper, the ballot marking device can cheat [17:46.660] [17:47.280]with impunity on ballots like that. So if there are enough voters who are using this autocast [17:52.360] [17:52.360]feature, then relatively wide margins can be altered without any possibility of detection. [17:59.700] [18:00.780]Languages other than English voters, if a voter is looking at a ballot on screen in one language [18:07.200] [18:07.200]but then printing it out in English, the voter might be less likely to check the printout if [18:13.400] [18:13.400]that's if the voter is not so comfortable in English. Moreover, if a voter who is a, you know, [18:21.020] [18:21.020]clearly a native foreign language speaker reports a problem with a ballot, it might be the case that [18:27.420] [18:27.420]some poll workers would be less likely to believe that the voter actually detected a [18:34.620] [18:34.620]problem with the device and rather more likely to believe that the voter made a mistake. [18:40.760] [18:40.760]There's all kinds of things that Mallory can monitor to try to target the attack by looking [18:45.220] [18:45.220]at how much attention the voter is paying, whether the voter is in a hurry, whether the [18:48.620] [18:48.620]voter is reviewing selections and so on. All right, so there are other problems with passive [18:56.720] [18:56.720]testing. Among them, it becomes a really easy way to raise a thud attack through uncertainty [19:02.320] [19:02.320]and doubt by simply asking voters to spoil their ballots more often than casting doubt on the [19:07.080] [19:07.080]election. All right, so let's look at these oracle bounds now. Suppose that instead of relying on [19:13.440] [19:13.440]voters to run the tests, we have to, we're going to try something like parallel testing, active [19:19.540] [19:19.540]testing, or logic and accuracy testing, where we input test patterns and look at what comes out. [19:25.480] [19:25.480]So if we had perfect knowledge of voter behavior, that is if basically Pat could pick voters at [19:33.800] [19:33.800]random and look over their shoulders while they vote and see whether the printout matched what [19:38.380] [19:38.380]the vote was presented on the screen or in the audio output, that's kind of a best case scenario [19:44.500] [19:44.500]that doesn't involve Pat having to figure out the distribution of voter transactions with [19:50.460] [19:50.460]the BMD. So even under those circumstances, [19:54.560] [19:57.500]it takes a fair number of votes in order to, you have to spy on a fair number of voters, [20:05.480] [20:05.480]have to look over the shoulder of a fair number of voters to have a reasonable chance of [20:08.560] [20:09.260]noticing a problem in the election. So for instance, if Mallory alters 15 transactions [20:16.040] [20:16.040]in a contest that has a little under 3,000 voters, which was the 2018 median jurisdiction [20:21.000] [20:21.820]turnout, that could change the outcome of the contest by one percent or more, but Pat would [20:28.860] [20:28.860]need to look over the shoulder of at least 540 voters, about 18 percent of the capacity of the [20:34.380] [20:34.380]machine. That would involve testing each ballot marking device several times an hour, but once [20:39.260] [20:39.260]an hour would not be enough. If you're limiting things to once an hour for 13 hours a day, [20:44.840] [20:44.840]to have a 95 percent chance of catching a problem, you have to have over 6,500 voters in [20:52.440] [20:52.440]the contest, which is almost triple the median turnout in jurisdictions in the U.S. and 20 times [20:56.660] [20:56.660]the median number of active voters in incorporated areas. In reality, Pat can't shoulder serve, [21:02.320] [21:02.320]Pat needs to make a model of voter behavior, and that's got to be calibrated to data. [21:06.580] [21:06.680]That's going to require monitoring voters in extreme detail, all of the details I mentioned [21:12.220] [21:12.220]before in relating to voting transactions. That would compromise voter privacy completely [21:17.440] [21:17.870]and probably be illegal. Nonetheless, let's imagine that Pat had a budget of running an [21:24.240] [21:24.240]infinite number of tests using a particular model. How many voters would Pat need to observe in order [21:32.700] [21:32.700]to get a model that was accurate enough to detect an alteration to some fraction of the votes? [21:37.920] [21:37.920]To have 99 percent confidence of detecting a change to a half percent of the votes, [21:43.180] [21:43.180]even if Pat could conduct an infinite number of tests in order to model behavior well enough to [21:48.100] [21:48.100]detect things at that confidence level, would involve monitoring three and three-quarter [21:54.420] [21:54.420]million voters in excruciating detail. Of course, their behavior in one election might not match [21:59.680] [21:59.680]their behavior in another election, and many jurisdictions that you need to be monitoring [22:04.680] [22:04.680]aren't big enough. They just simply don't have that many voters. As you look at larger margins [22:10.200] [22:10.200]and lower confidence levels, that bound goes down. But even from 95 percent confidence of [22:15.420] [22:15.420]detecting alterations to five percent of the votes, you'd have to observe more than a million [22:21.100] [22:21.100]voters. These are very, very conservative bounds. If you had a test limit of 2,000 tests to conduct, [22:27.420] [22:27.420]then the number of voters you would need to observe, you need to have an even more accurate [22:31.840] [22:31.840]model, you would have to observe even more voters. So the message here is you really have to look at [22:36.460] [22:36.460]monitor a million voters in excruciating detail, at least in order to have a reasonable chance of [22:41.660] [22:41.660]picking up outcome-changing errors. All right, so the situation is really not very good. I mean, [22:47.200] [22:47.200]it's actually worse than this. Even if you were able to do that much testing, if you find a [22:51.760] [22:51.760]problem, the only remedy is a new election. You have no idea which transactions were altered, [22:56.160] [22:56.160]what the right outcome should be. Margins aren't known before the testing happens. If it turns out [23:01.520] [23:01.520]that the margin is smaller than you allocated testing for while the election was going on, [23:06.980] [23:06.980]there's no way to go back and fix that. The tests themselves have uncertainty, and that means [23:13.100] [23:13.100]that you really need to factor that in in deciding who really won. Is there really strong [23:17.420] [23:17.420]enough evidence that someone really won? If there's only a 95% chance you would have seen a problem [23:22.640] [23:22.640]that big. Moreover, this is going to require new systems, extra hardware, additional staff, and [23:29.960] [23:29.960]additional training. This is a very expensive proposition, even if it could possibly be [23:34.260] [23:34.260]mounted. So our conclusion is that also, as I mentioned before, while BMDs are widely touted [23:41.980] [23:41.980]as helping some groups of voters, in fact BMDs pose an ideal vector for disenfranchising those [23:48.060] [23:48.060]very same groups of voters because they will interact with the machines in specific ways. [23:53.020] [23:53.700]In short, there doesn't really seem to be a way to rescue the trustworthiness of elections if [23:59.040] [23:59.040]you're casting most of the votes on ballot marking devices, or a substantial fraction [24:02.500] [24:02.500]of votes on ballot marking devices. And prudent election administration would involve minimizing [24:07.120] [24:07.120]the number of voters who use ballot marking devices, really reserving them for their [24:11.860] [24:11.860]accessibility benefits, where those accessibility benefits really do help particular groups of [24:16.400] [24:16.400]voters. Thank you very much for your attention. [24:18.800]