So our next section is about common misconceptions and false parallels about voting technology. And our presenters are David Jefferson, who I introduced before. That's the chairman of Verified Voting. And Joe Hall, who's the chief technologist and director for Internet Architecture Project at the Center for Democracy and Technology. So let's give him a round of applause. Thank you, Jake. Okay, what I'm going to talk about is not about particular voting systems or the code or vulnerabilities or anything like that. I'm going to talk about the requirements for voting and how they compare with the requirements for other applications, online applications in particular. The point I'm going to be making, the bottom line, is that from a technical point of view, the security, privacy, transparency, reliability, and other properties that are going to be used for voting, the requirements of voting as an application are more complex, harder to satisfy, than any of the other security problems you're likely to have encountered. I'm going to make a detailed comparison between voting and e-commerce as the prime example. So let me move on here. First, I want to just show up this generic slide about the laws and regulations. There are a large number of technical requirements that have to be satisfied simultaneously to have a satisfactory public election. Now, I'm not going to read all this slide. I'm just going to read the headings here, voter authentication issues, double vote prevention, ballot selection, meaning the person gets the correct ballot, accurately capturing the voter's choices, the integrity of the balloting process, the accuracy of the tally of those ballots, the two different kinds of privacy, which I will come back to in a moment, mixed-mode ballots, software certification, software integrity, the availability and reliability of the voting process, all forms of security issues, transparency, manageability, and the mission criticality. Now, I'm going to highlight only a couple of those. I'm not going to read all of them. But I wanted to first of all talk about ballot integrity that, of course, that no ballots are lost, no ballots are corrupted in the process, and no phony ballots are inserted into the process in the course of an election. I want to highlight privacy and what I call privacy two, two different notions of privacy, that is, simultaneously, requirements for a public election. The first notion is that it should be impossible to associate a cast ballot with the person who cast it, at least up to the size of a precinct. So the most you should be able to say is that you cast one of the 150 ballots that are in the ballot box at that precinct this day. So that's as close as we ought to be able to identify you, the size of a precinct. And nobody should be able to identify any closer than that. Not a court, not an election official, nobody. That's a strong privacy requirement. The second privacy requirement is a little more subtle. Even if the voter wants to tell you or tell the world how he voted, he can do so. He's free to declare how he voted, but he's not free to prove it. He is not allowed to have any proof. And the protection that is afforded by that is a voter who can't prove how he voted is less subject to coercion than a voter who has proof, and so the coercer could demand that proof as to how he voted, and a voter who can't prove how he voted can't sell his vote. He can't sell one vote, and you can't create a market for vote buying and selling. So this second privacy criterion is different from the first one, but both are critical to... to strong election integrity. Now, I mentioned reliability and security and transparency. The point about reliability is that all those previous properties have to hold, even in the case of partial failures of the voting system, partial technical failures. If a voting machine goes down, or if the local area network or the voting precinct goes down, or the poll book goes down, or something like that, you have to be able to finish the election anyway. It has to be robust against all kinds of partial system failures. Of course, nothing will be robust against, you know, a regional power failure, perhaps, but at least partial system failures. And security talks about all of the other good properties that the election has to have have to... have to hold. And security talks about all of the other good properties that the election has to have have to hold, even in the case of a limited, at least, conspiracy of election officials, programmers, voters, third parties, to undermine the election, okay? Even if they're trying to do so. We need to be able to conduct an election and guarantee the integrity, even in the face of that. Okay. Now let me move on. So I'm going to... I said I was going to talk to you about how voting is dispensable. It's distinctly different from other security problems. And the comparison I'm going to make is online commerce, let's say, buying things from Amazon, versus online voting. You know, voting from your home computer. Now, we don't... We don't have a lot of online voting in the United States. We have some. Online voting is an extremely dangerous idea, but I'm not here to rail against it from that point of view. I'm talking about requirements here. And people have a tendency to think in terms that if I can shop safely online, why can't I vote safely online? After all, the two are very similar. They both involve a simple online transaction. You navigate to the correct website. You identify yourself. Once you're identified, you make your choices, your purchases, or your votes. When you've looked over your purchases and your votes, and you're happy, you press that, you know, buy button or that vote button, and you're done. It's a... So superficially, from just an online transaction point of view, they sound very similar. And what I'm about to try to convince you of now is that they are not remotely similar, except in that superficial sense. So on the left side there are the categories. These are the categories of differences that I'm going to be talking about, and the middle column is shopping, and the right-hand column would be voting. For shopping, there's no eligibility requirement of who's allowed to buy from Amazon. You can be a kid. You can be a non-citizen. You can be a felon. We don't care. Amazon does not care. They don't have an eligibility requirement. But we do have strong eligibility requirements for who is allowed to vote in a U.S. election, and we have to ascertain them. We do that in all states at voter registration time, and in a growing number of states now. We also do it, voter ID states, at voting time for, at least for in-person voters, if not for absentee voters. So there's this eligibility issue that distinguishes the two. Then there's the double vote transaction. Everybody knows that you can't vote twice in the same election. But Amazon doesn't care if you buy two pairs or four pairs of socks in the same transaction. Or if you come back and buy more things ten times in the same day. There is no analog to the notion of double vote prohibition in the e-commerce world. Again, it's more an additional requirement or structural difference more difficult in the voting world. Proxy transactions. A proxy transaction is when you authorize explicitly or implicitly somebody else to use your Amazon account. Your kid, your spouse, or you give somebody your credit card number, or you give them your password. Amazon doesn't care if you do that. That's perfectly legal. They're okay with that. They don't even know. However, in the voting world, proxy voting is not allowed. You cannot authorize somebody to vote for you under any circumstances. In any of the states. Period. So, the consequence of those first three things. The first three differences between shopping and voting is that it appears in the issue of voter identification and authentication. In the case of Amazon and e-commerce, they are satisfied with weak authentication of who the buyer is. The true identity of the buyer is not really required. When you sign up for an Amazon account, they don't ask for your driver's license number. They don't ask for your passport. They don't ask for your picture. They don't really care who you are. What they care about is that when you later do a transaction, that the transaction is probably good and the credit card is definitely valid at the moment of the transaction. That's what they care about. But the identity of exactly who's buying, you or your spouse or your kid, they don't really care. I mean, they'd like it for a marketing information, but it's not a barrier to the purchase if they don't actually know who you are. But in voting, because we do have to know that you're eligible to vote, we can't allow you to vote twice, and we can't allow you to vote for somebody else, we need strong voter identification and authentication to guarantee those properties. And it's even harder online, if we're talking about online voting, because we are worried about large-scale attacks through weaknesses in authentication. You need strong authentication of ballots. Sorry, of voters. Additional distinctions between shopping and voting. Privacy. The privacy rules for shopping are, well, both parties, the buyer and the seller, know everything about all parts of the transaction. They know what you're buying, how much it costs, how to return it or what happens if the transaction somehow goes bad. And, you actually do get a receipt in some sense for the transaction, which you can show to a third party and you can prove to a court, for example, if the transaction goes bad, that you have the details and you have the proof of it. But in voting, who casts what vote is an absolute secret, as I said earlier. And a voter knows how she voted, but she must not be able to prove it to a third party. It's a different, much more subtle privacy requirement. The privacy requirements for shopping are policy requirements, business and contractual arrangements that the seller has with the buyer. They are encased in law in voting. Integrity. Integrity means that the transaction is actually correctly done. So, in the case of shopping, Amazon will not go out of business if 1% of all of its transactions are fraudulent. And it's a good thing because they are. But in the voting world, we cannot tolerate a 1%. We can't tolerate a 1 tenth of 1% error. We really need systems that are architected and designed and intended and as far as humanly possible, 100% accurate recording of every transaction. Especially in this era of closer and closer elections. Nonetheless, errors occur anyway. So let's talk about error detectability. In the commerce world, failures and frauds are going to be detectable eventually because of double entry bookkeeping and the presence of receipts in both directions and a paper trail. If nothing else, some account is going to go to zero someday and you're going to detect it. But in the voting world, modified vote transactions may be completely undetectable. And privacy violations, the requirement that votes be private in the two ways that I described, are almost always undetectable. So we have this, one of the burdens that we have in architecting election systems is the fact that many errors are fundamentally undetectable. Because the voting officials, they don't know how you intended to vote. And if you claim that they recorded your vote incorrectly, you can't prove it to them as a third party. So it's just really not fundamentally possible in many cases to actually detect that an error happened at all. Even if you did detect it, let's talk about recoverability. In the case of the shopping world, there are all kinds of ways to recover from bad transactions. Many times they're reversible with returns or refunds, other ameliorations like insurance and tax deductions and just price increases to recover the losses. Those are all kinds of ways to deal with the risk of faulty or fraudulent transactions in the commerce world. But even if you detected in the voting world that there was a fraudulent vote cast, you wouldn't be able to do anything about it. Because if I know there's a fraudulent vote cast, I don't know what the vote should have been. I don't know who cast it. I can't bring them to justice. The recoverability issue is far more troubling, both detection and recoverability, in the elections world than it is in the commerce world. And finally, additional, this is my last slide here, availability. Amazon needs to be up almost all the time. But if they were down for an hour a year, they wouldn't, you know, they wouldn't fail. Or if they were down for 10 hours a year, they wouldn't fail. They would fire their IT team maybe, but they would not suffer catastrophic losses. However, voting on election day must be up. Period. Because whereas Amazon would suffer a certain loss of business or most likely just only delayed business, there are statutory requirements for the date that election day is. And you can't postpone it. And you can't ask, well, voter, come back tomorrow. We're fixing our systems today. That's just not an option. So the damage from downtime in a voting system on election day is basically unrecoverable. The transparency requirements are different, too. In the shopping world, buyers have only the right to examine and question the details of their own transactions. Period. Nobody else's. But in the voting world, voters and candidates and the press and everybody else, at least in principle, has a right to examine and observe and audit every detail about the election. Period. From soup to nuts up to as long as they preserve the privacy properties, you know, no identification of which voter cast which ballot. But everything else is supposed to be transparent, at least in principle. In practice it's not, but I'm talking in principle here. The motivations of attackers are completely different, too. People who attack Amazon transactions are usually motivated by money. They're trying to steal your personal data. They're trying to steal your credit card number. But in the voting world, the desire is almost always political. They're trying to fix the outcome of the election or they're trying to disrupt the election. They may also be making money doing that if they are fixing the election on behalf of somebody else who's paying them. And finally, the consequences of a successful attack are completely different in the voting and e-commerce world. Somebody loses money, usually the merchant, not the shopper, with a fraudulent or faulty transaction in the shopping world. But in the voting world, the wrong people may be elected, the wrong initiatives that should have passed fail or should have failed pass. And the legitimacy of democratic government is undermined and our national security is imperiled. So the bottom line here is voting, security, privacy, transparency, availability, reliability, all in every dimension more difficult than the security problems that you may have encountered in other application domains. Joe. Oh, we're about to switch. That's okay. We, you would think we would have solved the projector problems by now in the, in the world. You're close. You got it. There you go. Thank you, sir. And it's such a joy to share the stage with people like David and Candace and Matt Blaze after this. There, we've been working on this stuff for, for a long time and they are mentors and role models for, for me and for a ton of other people. And I'll stop gushing. All right. Hi, my name's Joe. You may have seen me earlier today. I'm from the Center for Democracy and Technology in Washington, DC. I should mention I am hiring an election technology fellow. So if you know something about cybersecurity and election technology, please come talk to me. I'll be here all weekend. So what I'm going to do is talk about a number of what I would consider common misconceptions. Things you hear people shout. Why don't we just do this? Or why don't we just do that? Or aha, I have a solution to this extremely complicated problem that you've never thought of about before. Don't dare talk about blockchain and Bitcoin or cryptocurrencies to me. You don't put anything on the blockchain you expect to be secret for 80 years. Just okay? Like that's it. You know, we're using public key cryptography for this crap. So that's not on here, but I wanted to rant a little bit about that. So I'm going to go through five misconceptions. I'm going to go pretty quickly, just because you may, one, have beef with my interpretation, or two, have your own misconception that you'd like to raise and I'd love to just get into that and talk about that stuff because that's where the action is. You don't want to hear me talk too much. So just use hand counted paper ballots is often what you hear people say. Why can't we just do it like how Canada does or how England does it? And that's because they have a parliamentary system where you make exactly one choice on the ballot. We have extremely complicated ballots. The Netherlands has pretty complicated ballots, but then again, it's, anyway, we have very complicated ballots. All, you know, in California, there are mosquito abatement district contests. It's almost a dog catcher on your ballot, which is great because you want people to be able to elect things that they need to elect. Maybe we make too many decisions in the voting box, but that's more of a political statement. We also try our best to enfranchise as many people as possible. Many, many other classes of people do we want to try and allow the ability to vote independently and privately by themselves. So people with disabilities, voters, people with vision impairment, it's very difficult. People with motor disabilities is also difficult. If you have compound disabilities, so if you have hearing disabilities and a sight disability, that's extremely hard. There have been actually a couple of voting systems, maybe just one, that targeted that particular kind of compound disability. There are Native American languages that have no written form, that even though these people are literate and speak, they would prefer to vote in the language that they speak at home, and we try our best to enfranchise them. L.A. County, for example, I think has 11 languages, things like Laotian. It's not even included in the Voting Rights Act, which is the thing that tends to tell you what kind of languages you have to support given traditionally disenfranchised populations. But what we want to do is make sure, if you, as my dad would say, don't dream in English, we want to make sure you can vote in the language that you dream in. And third, it's just faster and easier and it can be much more usable to use a touchscreen for many people. Paper, when designed correctly, is awesome. But imagine trying to put 11 languages on a single piece of paper. If you can do that, man, you can make a lot of money. But that's not to say that there's no place for paper ballots at all. For example, the voter verified paper audit trail. Optical scan ballots are exactly what we're talking about. That's rooting trust and reliability and indelibility in a physical object. A colleague of ours, Doug Jones, prefers etched aluminum, but we haven't found a way to do that in a, as long as it would be cheap and quick and all these other things, it would last for hundreds of years. Which is maybe not such a good thing, I'm not sure. We also do recounts. These are the heavily lawyered up, we're going to count every single ballot and see if we come up with a different result and then, stuff that we worked on in my post-doc at Princeton and that Phillip Stark has really sort of led from Berkeley are these things called risk limiting audits and these are statistical methods of counting a random sample of ballots, comparing it to the corresponding data structure. So you're looking at here's the thing the person marked, here's the data structure that interprets what the person marked. And those are statistical tests designed to make sure that if you find enough error that would cause the outcome to be different, that you would actually have announced somebody else, you escalate until you've actually counted all the ballots, right? Or more practically in some places, you may have to rerun an election, which is easier for smaller elections, almost impossible legally constitutionally for something like the presidential election. Damn it. That means I'm taking too much time on each slide, guys. Anyway, so that's one. Hacking is a new thing in elections. So if you've been in this whole track all day, you may hear some of this over and over again. Oh, damn it. Cool. Awesome. Thank you, Goon. Cycle, cycle, cycle. I'll just talk. So there's been a considerable amount of hacking of various kinds. Hacking is a new thing. It's a new thing. It's a new thing. It's a new thing. It's a new thing. Hacking is just a horrible word. It's such a big umbrella word, right? You can almost describe any kind of interference. Or meddling is the word of the now for elections. You can just sort of say, hey, they did something. We're not sure what they did, but they did something. Or in some cases, we know what they did. In fact, the U.S. has a pretty substantial history, and I have a lot of notes on this that I can share with you. If you have any friends that are Chileanos or Chileanos, in Chile, we did a whole bunch of horrible things to them, using mostly money in elections. In Italy, to make sure the communists didn't get elected, we did a bunch of things there. In the Philippines, those weren't like actual direct interventions in the machinery of their democracies, so to speak, but they were ways of using influence and power to directly change the outcome of elections. The coolest story is the Ukraine 2014 election, which you may have heard of. And if you haven't, there's a really cool short paper I can point you at. A hacker group affiliated with the Russian government, supposedly, broke into the Ukrainian central election administrator's tabulation server, changed the results about an hour before they were going to be announced. The Ukrainians found out, rolled everything back, and announced the real results, but Russian television actually reported the result that the hackers hacked into the machine. And so that's the only time we've ever seen something that is probably as close to confirmation of an actual hack that we've ever seen. The other things are not nearly as sexy. So for example, there's a bunch of denial of service attacks, and while those are very powerful, that's like everyone's hammer, right? It's always going to be there. It's always going to be hard to protect against, increasingly so. There was a case, I think in Illinois, I'm having a hard time tracking this down, but I will find it, where a poll worker ran an optical, I think it was a couple poll workers, ran an optical scan ballot through the optical scanner like 200 extra times, trying to inflate the count for one particular contest. But it's really easy to just count the number of ballots in the bin and recognize, oh geez, there's 200 pieces of paper missing, or this is off, and just recount the stuff. So that was caught, and they were prosecuted. And then this is one that David Jefferson told me about, that there was an insider attack on the Canvas database in the 1994 presidential election in South Africa, the first election for Mandela. I didn't even know about that one, so that's really awesome. I mean, it's good to know about. So hacking has happened before. And we've covered this a little bit, you know, this strict anonymity requirement. People often say, well, that's where your problem is. If you didn't have this strict anonymity requirement, you wouldn't have all these problems. So, for example, in the United Kingdom, when you go vote, you sign your name right next to a number that's printed on your ballot. So there's an actual mapping between your signature and the number on your ballot. But that mapping is kept as essentially a state secret that you have to meet very high bar in an actual election fraud court case, and able to unmask that. So it's a little bit more complicated than that. Masking. Unmask that mapping. Anyway, so this gets actually kind of interesting. You may not have realized this, but in the very beginning of at least the United States, and obviously this is a more global room than just the U.S., but you had to be a white male landowner in order to vote, right? And so you had to own a certain amount of land, not just any land. It had to be a certain size. You had to be a man, and you had to be a white man at that. You had to actually raise your hand or announce your vote, viva voce, out loud. That's awesome in terms of transparency. You could actually sit there and record. Everyone could independently record it and come up with their own results. But what that means is the coercion and the forms of undue influence, like vote buying, is still right there. You know, a father can say to the son, hey, you're not going to inherit that land that you want so bad if you don't show me exactly how you voted before you cast a ballot. And I should mention, there's only one state in the U.S. that allows you to show your ballot to someone before you cast a ballot. And that's a state in the U.S. that allows you to show your ballot to someone before you cast it. And it's West Virginia. For some reason, in the Constitution, it has this thing about open voting, which allows you to show your ballot to someone. So if you were going to buy a ballot, it's still illegal to purchase votes in West Virginia. But if you were going to do it, that would be the mechanism to do it. You're not going to get a lot of votes, I don't think. And so before about 1900, Election Day was a payday for most American voters. You would go to the polls. There would be a bunch of people with very bright and distinctive color of paper that were partisan ballots that listed for one party all the people. And you would sit there and just play them off each other until you got enough money. And they could go and watch you, deposit it in, and then give you the second half of what they would want to pay you. And it was about three months' worth of pay. We adopt this thing called the Australian ballot. Most states, between 1890 and about 1905, adopted the Australian ballot. This is an anonymous ballot printed by the government on a uniform stock with everyone's name on it. And it was a ballot that was printed the same size on the same ballot. When that happened, voter participation rates just plummeted because it was no longer a payday, and now it was your actual civic duty. You had to actually care. You had to give a shit and go and vote to do that stuff. And it's also really important to be able to make sure that the votes are anonymous so that you don't have, in days like now where you don't, you wouldn't go to a polling place necessarily and do that, but there could be, you know, a coordination mechanisms involving actual cryptocurrencies to pay you, things like Zcash, right? The thing that is the more anonymous version of Bitcoin. And now, you know, and this is something else that David Jefferson pointed out to me, this is such a tense political climate we live in that you can imagine that no, you know, never has the anonymity of your vote been more important. You could be someone in a party who holds an unpopular view all of a sudden, and they may not be the person who's supposed to be may take retribution on you if they found you voted a way that they didn't want you to. Okay. So here's ones that are maybe more interesting to the technical folks in the room. You'll hear people say our voting system is highly distributed, so it's hard to attack. I should say that monoculture and the absence of monoculture, it's a weak guard at best. As people have said, there are often similarities across systems that can be exploited even if they're very different platforms. But it's only really a benefit, this distribution, to contests that span more than one jurisdiction. And what do I mean by that? I mean that if you have your proverbial Tony Soprano, I guess Tony Soprano was never in a proverb, so whatever. Have your fictitious Tony Soprano, if he could fix a local waste management bond in his favor for 100 grand, say a $5 million waste management bond or something like that, not a very big number. But that means the threat model there is for that one jurisdiction, that one county, that one town, they're going to have to start thinking about stuff like that and planning for the nasties that are around your town trying to influence your employees and otherwise get into the systems. And even then, many states are dominated by very populous single or one or two or three jurisdictions, right? And because voting equipment has a hard time scaling, it's hard to get into the systems. And so I think it's important to think about that. It's more than likely that those jurisdictions are going to be running the same voting software or at least not very different voting machines and software. And so those statewide races can be influenced by attacking one or two local jurisdictions. And then when you have a lot of jurisdictions, you can sort of pick the easiest ones to attack. So maybe you could attack the big jurisdictions, but it turns out that they have really good cybersecurity practices and regular physical security practices and whatever. Plato's security practices. If they have all those things set, you might be like, well, geez, I can go attack a piece of lower hanging fruit. I may have to change more votes. I may have to disrupt more records. Like something I care a lot about, you know, your voter registrator databases are online, but they're often the canonical versions are in staged, you know, air gap networks that are updated with deltas every few days and stuff like that. But the data on voter registration systems is insane. And so I think it's important to think about that. And so I think that's a really good point. I think that's a really good point. And I think that's a really good point. And I think that's a really good point. There's a lot of things that are staged for electronic poll books. These are the computers we're using instead of spiral brown notebooks anymore in polling places. You can imagine that staged data is sitting somewhere probably not very well secured, but we hope so before it's put on those poll books. That's a natural place to do a very disruptive attack where you single out even a given party or just randomly select people in a place that have a certain kind of political affiliation. Finally, the systems are not connected to the Internet. So you may have heard this a lot, too. But the system is not connected to the Internet. It's not connected to any kind of political affiliation. So I think that's a really good point. And I love to point out what Dan Wallach has said in the past. It's like, yeah, we had viruses before the Internet. We had these things called floppy disks. And they would be transmission vectors for malware. And there's just so many ways to cross air gaps these days. You know, like I was in my earlier talk today, I talked about air gap networks having rogue Wi-Fi attached to them for whatever reason to help people out because people don't recognize that that may be a security vulnerability. But it helps them do that. And so I think that's a really good their work better. And so, you know, insider attacks is another really good point to make here is that insiders have the ability to cross the air gap if you get the right insider. And, you know, there's the machines in the voting village, which I highly recommend you go check out. The AccuVote TS, which is still being used in Georgia, was one where Ed Felton and his team designed a virus that you could put on during the primary on one machine. And then between the primary and the general election, it would propagate to all the other equipment so that you'd have the hack ready for the general election. So while these requirements are really hard and it means our security models have to be a little different, the attackers are going to be just as clever to get around those kinds of things. And I think that's it. So thank you. There's my GQ signal if you want to talk about any of this or if you want to potentially come work for me. That would be wonderful. We'd love to take questions. Just yell and I'll repeat it. There's a lot of things where it seems like these problems are solvable. Maybe some of it's a statutory problem. But, I mean, vote by mail. Yeah, someone can see my ballot, someone can see my mail. But it's hard to deal with these things in scale, isn't it? Well, Washington until recently was a singular case because of the, well, in Oregon. I don't have the same confidence in vote by mail that you do. Okay? Number one is authentication of the voter is very weak. Okay? The best you have is a signature check, which is often not really even done or is only done with software aid. Usually you don't have a strong way of authenticating who the voter is. So in most states in the United States, when there is vote fraud, if it's not committed by election officials, it's committed through mail-in voting. So mail-in voting is definitely weaker. Yes, you say, you seem to shrug your shoulders when you said that your vote could be lost in the mail, as though that didn't matter. The fact is you as the voter may not know that, or you may not learn it until late, until too late. And yes, maybe you can go and vote in person if you discover on time that your vote didn't make it. But the point is you may have voted by mail because you were out of state or not in a position to go. I mean, for many people, that's what absentee voting is all about in every other state. Besides, besides Washington and Oregon and Colorado. So I guess I'm not as cavalier about the strengths of mail-in voting as you are. I don't think it's a good idea for it to be universal as it is in Washington and Oregon. And I, I don't think it strengthens democracy to have that be the only mode of voting. Yeah, I would just comment on that. In Washington, you can actually track your vote from when it was received. When it was counted. You can actually verify whether or not it happened. And you get a full month, two months to vote. So there's plenty of time to address all of these problems. Yes, but most people don't vote a month or two months in advance. They vote in the last month or the last couple of days. Anyway, let me, let's, let's move on. Well, and so just to make a quick comment, I vote by mail in a place that has universal vote by mail. I really appreciate it because I, I, I am so busy that I, that's, but I, I think the computer science community at least, that works on elections, recognizes that, I think I'm speaking for many people at once that I don't have the liberty to, but I'm going to, and say that, that I think we have the opposite view, that vote by mail tends to relax a lot of these requirements. Like, for example, I know a guy, I know a guy. In San Francisco who gives his lawyer his ballot with his signature on it and $200 and has his lawyer vote for him, which is like, it's vote, it's weird. It's, it's not vote buying. It's not coercion. It's more like you'll make better decisions than I will because I trust you, Mr. Lawyer and stuff like that. And so it allows a bunch of things that are sort of wrinkles in this stronger model of things. And the trick there is the, is the models that sort of fit in place in terms of the, in place of universal vote by mail. Universal vote by mail are, are, are, are newer things like vote centers, right? Where you have, instead of small polling places all over the place, you have big polling places that you can go to any one of them. The problem with stuff like that is if you have optical scan ballots, you have to have enough copies of every ballot style, which can, in a primary election with a bunch of parties, can just, can grow geometrically. And so... Print on demand. Print on demand, for sure, works very well. Some states that's very hard, right? Because you've got to certify the... The actual printer and the people doing the printer as being ballot printing entities, right? But that means we can change the laws. But print on demand is, is a way to do that. You don't get some of the features you get from actual bonded printers in print on demand. But anyway. And there are things... I'm sorry, I'll shut up. Larry? . . So the question was, at what point does the convenience versus security, what point does turnout get affected by the tension between convenience and security? So if you make things too secure and not convenient enough, are you lowering turnout? That's a great question. I'm not aware of research on that. Paul Gronke and folks like that who, who, who... Who do a lot of voter sentiment analysis may have data on that. There's probably some natural experiments, but I don't know. It's something we have to plan well for and make sure that elections are as usable as possible and easy as possible, but no easier, right? Sir? . . So let me try and repeat the question and let me know if I got it wrong. Is there a, any theoretical model that preserves anonymity in voting where you can... Say it again? . Treat each vote as an atomic transaction, sort of like acid, right? . Are you talking about remote voting or are you talking about... . So there are formal methods here and there are people who, who model polling places. I'm not sure if that exactly would answer your question, but come find me and I can at least point you to some of those papers. And if it's not there, maybe you can do it. But I have a feeling that, that the polling place models people use, especially in queuing theory, which is, you know, how, how often, how often do people arrive at a polling place to optimize the distribution of voting machines? They have to deal with that stuff all the time. . I would, I would just say that voting at a precinct is an atomic transaction if it, from a, say, computing point of view. I mean, from beginning to the end of the transaction, you're not interfered with as a voter. And if I understand you correctly, yes, a voting transaction is intended to be atomic. . Maybe that sounds like, sounds like I didn't satisfy you. . Come talk to us. Candace? . . . . . . . . . . . . . . . . . . . . . . . . . . somehow. In some places it is counted by hand in very tiny places. But many places they use optical scanners, sometimes central count optical scanners, which I don't think we have any here. They're really curious machines. They're very high throughput, expensive, $100,000. You can think of like Pitney Bowes mail handler kind of things to very quickly go through these systems. And in fact, in the Ohio study from 2007, we installed Linux on the ES and S1 just to prove that it was a general purpose computing platform. And they do have vulnerabilities. And often, you know, the things like the tracking capabilities where you with your vote by mail ballot have something that allows you to check and see that your vote by mail ballot has been received, has then been counted and accepted, that's so crucial because often you can see something happen there like, hey, it was rejected for some reason. And then you have it. In many places you have a period of time where you have to wait for it to be corrected. You have a period of time to correct that stuff. What you may not know, and this happens with regular optical scan, too, is if that scanner correctly interpreted the marks you made on that ballot, right? And this is where the newer computer vision systems are a lot better at doing things that older systems couldn't do. For example, if you have an oval and someone circles the oval instead of filling it in or putting an X in it or something like that, older systems had a hard time catching this. It sounds like from what I'm hearing from people that newer systems detect marginalized marks and other things like that. And so, you know, it's a good thing that we're able to better count those. But it's not perfect. And that's why you need to have regular audits and actually compare some of these marginal marks and figure out if they would have affected the race. And Candace, you said something about correctability. Could you repeat the question? Correctability on a macro scale versus that of a limited evaluation. I understand what you're saying about correctability as an individual, though, is not possible. But what about in actual results, or in actual tallies of the results, that that could be a little bit more expensive? Well, so that's a complex question. So most of the time, there isn't a second process after the Canvas to check the initial Canvas results at all. Now, you've heard it several times today, people have recommended that we have risk-limiting audits of paper ballots. And the central idea there is that you would take a random sample of the paper ballots and check them by hand. And there's a statistical criterion to indicate whether or not this confirms the results from the original machine Canvas, or whether it disconfirms them. Okay. And the thought is, though the law has not followed this yet, is that if the risk-limiting audit afterwards determines that the results were originally incorrectly tabulated, either for procedural error or software error or malicious code or something like that, then the results of the risk-limiting audit would legally replace those of the machine counts. And then you would presumably, that would immediately presumably trigger an investigation, a forensic investigation, of the voting machines, and exactly why did they not match the results of human counts of these ballots. Right now, this law and these procedures are not in place, and we would, you know, it's one of the things that we are looking to try to institute in every state in the union as soon as we can. And for, you know, I think that's a good question. I think that's a good question. I think that's a good question. If anyone out there is a real math nerd, have I got something for you. Walter Mabane, who is a political scientist and a total math nerd, has a really cool election forensics course that all the materials are available online, and all he does is work with totals, with vote totals, at the precinct level or whatever level, and uses various kinds of statistical tests to spot tampering. And he's used these in many, many a foreign country. Things like Benford's Law, which is, I won't get into it, but look it up, it's fun, it's basically the distribution of the first number in a distribution of numbers, so like, the one in 1045, or something like that. And then there's a second order, Benford's Law, which is the second one. Anyway, these all have distributions that you can spot when someone is actually messing with stuff. But what can you do afterwards? You know, you're not going to actually, like, renormalize the results, like a census kind of a thing where you say, oh, we miscounted something, but we know how we miscounted and we're going to do that. Man, campaign lawyers will never let you get away with that. I did want to add one other thing, which is that the idea of doing forensic examinations of voting machines or the back-end systems after an election, even if something is suspected as going wrong, there are no procedures, there is no law to permit, let alone require, such investigations. It's just not possible under the law today. In fact, it would be totally prohibited. So this is a frontier that I think we have to change in the future. Yeah, I think we've got time for one more. Sir? Do we actually know the quality or magnitude of election temporary right now? Absolutely not. I mean, yeah. I don't know. There are a few things we can say. I mean, I mentioned earlier that most of the... Most of the cases of detected fraud, you know, detected and prosecuted, have involved either election officials or vote-by-mail. Now, that doesn't tell you how many cases there are. It just says almost all of those that we have are of that kind. The notion of voter ID that's been introduced in many states has been touted as a protection against vote fraud, but I think most of the people in this room know that that's totally ineffective, except against... I think most of the people in this room know that that's totally ineffective, except against one kind of fraud, which never happens, which is a voter walking in and impersonating another voter in that same precinct. It just doesn't happen. Thank you, everyone. Thank you. Thank you. Thank you.