1 1 2 3 4 5 6 7 8 *************************************** 9 U.S. ELECTION ASSISTANCE COMMISSION 10 ROUNDTABLE DISCUSSION 11 DECEMBER 11, 2007 12 *************************************** 13 14 15 UNITED STATES ELECTION ASSISTANCE 16 COMMISSION ROUNDTABLE DISCUSSION was taken 17 before April C. Balcombe and Michelle 18 Robertson, Certified Shorthand Reporters in and 19 for the State of Texas, reported by 20 computerized stenotype machine at the Omni 21 Austin Hotel Downtown, 700 San Jacinto 22 Boulevard, Austin, Texas 78701, on December 11, 23 2007 commencing at the hour of 1:00 p.m. 24 25 TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 2 1 P R O C E E D I N G S 2 >> KING: Well, good afternoon everybody. 3 We will get started here in just a moment. I 4 think all the panelists are here. We have a 5 few housekeeping items, if you would, the 6 normal cell phones, Blackberries, anything that 7 beeps, squawks, if you would either put it on 8 silent or off would be appreciated. 9 The agenda for this afternoon is a panel 10 discussion on five issues related to security 11 in the draft version of the 2007 VVSG, and we 12 have brought together a panel today of people 13 with both experience and interest in security 14 issues related to voting systems. And Matt, if 15 you would, if you would flip to the next slide. 16 Back up one. As I was preparing for my 17 comments today, I was doing my normal Googling, 18 looking for precedent. Always interested in 19 historical perspectives on voting systems and 20 technology in general, and I came across a 21 couple that I thought might help at least me 22 get my perspective on this. And I think in 23 fairness to Charles Dual, that is attributed to 24 him, but I don't think he ever really said 25 that. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 3 1 In any event, Matt, if you would go to the 2 agenda, this is an ambitious agenda. It is 3 what we are going to attempt to hold to today. 4 We have got five questions that we have been 5 asked to address, and we are going try get a 6 break into the middle. If we get fatigued we 7 may break sooner, and we may break more often. 8 But what I would like to start with is 9 asking the members of the panel to introduce 10 themselves, and then we will have opening 11 remarks from Brian Hancock from the E.A.C. 12 So Ron, if we can start with you and just 13 work around the table. 14 >> I am Ron Rivest. I am on the computer 15 science department at MIT. 16 >> I am Peter Ryan. I am a professor off 17 computer science at New Castle University in 18 the UK. 19 >> My name Daniel Castro. I am a senior 20 analyst with the Information Technology and 21 Innovation Foundation. 22 >> I am Alec Yasinsac with SAIT Lab and 23 computer science department at Florida State 24 University. 25 >> Okay. Chris Thomas, Director of TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 4 1 Elections, state of Michigan. And Chair of the 2 U.S. Election Assistance Commission Board of 3 Advisors. 4 >> Sarah Johnson, Executive Director of 5 Kentucky Board of Elections and the Chair of 6 U.S. Election Systems Standards Board Executive 7 Board. 8 >> Brian Hancock, I am Director of Testing 9 and Certification for the U.S. Election 10 Assistance Commission. 11 >> I am John Wack from the National 12 Institute of Standards and Technology. 13 >> Good afternoon. My name is Costis 14 Toregas. I have two hats here. One is a 15 computer science scientist in George Washington 16 University, and I am also on the staff of 17 Montgomery County, Maryland, advising the 18 county council on information technology 19 matters. 20 >> Michael Shamos. I am a professor in 21 the School of Computer Science at Carnegie 22 Mellon University, and I think I am here 23 because I have done over 120 voting system 24 exams for six states since 1980. 25 >> Hello, I am Juan Gilbert from Auburn TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 5 1 University, professor in computer science and 2 software engineering. 3 >> I am Merle King. I am Executive 4 Director of the Center for Election Systems at 5 Kennesaw State University, and I will be 6 moderating this panel discussion today. 7 If you look at the agenda, you will notice 8 we have got five questions spread out over five 9 hours and if you have figuring the overhead, we 10 will spend 30 to 40 minutes per question. I am 11 hoping that everybody will kind of 12 self-regulate themselves, and we will share the 13 microphone appropriately. 14 When you do get ready to speak, if you 15 would, either put your tent card up vertically 16 or in some way get my attention, and we will 17 try to move around the room as quickly as we 18 can. But before we begin, I would like to ask 19 Brian Hancock to give some introductory 20 remarks. 21 >> HANCOCK: Thank you, Merle, appreciate 22 it. And thank all of you for agreeing to 23 participate with us here in this very important 24 meeting. We know you have busy schedules, and 25 we know that many of you have come from quite a TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 6 1 long distance in order to join us. 2 I should note that this is a first in a 3 series of round table discussions that the 4 E.A.C. intends to hold on the TDJC 5 recommendations. We will have similar sessions 6 with test labs, accessibility and usability 7 professionals, election officials, activists 8 and manufacturers. Roundtable discussions will 9 each continue to educate and inform the E.A.C. 10 on specific areas of the document under 11 consideration. 12 I also need to acknowledge Matt Masterson 13 over there of the E.A.C. who did the majority 14 of the work to make this roundtable discussion 15 a reality. Matt is also the primary E.A.C. 16 staff person available and working on a 17 day-to-day basis dealing with next iteration of 18 the VVSG. Before I turn the round table over 19 to the capable hands of our moderator Merle 20 King, I would say a few words about why we are 21 here holding this discussion. 22 Beyond the obvious, that is to get input 23 on the TGDC recommendations for the next 24 iteration of the VVSG, I think we need to look 25 at what we are trying to accomplish in a macro TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 7 1 sense. General goals of course are to develop 2 an implement standards to make voting systems 3 as reliable, secure, accurate, and accessible 4 as possible. 5 These rather simple words when spoken are 6 rather simple words when spoken but as all of 7 you know better than I do, extremely difficult 8 to produce. By undertaking the development and 9 adoption of a new iteration of the VVSG, the 10 E.A.C. and its partners are charting the course 11 for the development of voting systems in the 12 United States for the foreseeable future. This 13 is a truly awesome responsibility and one which 14 we must undertake with full commitment and 15 proper due diligence. 16 Given the scrutiny that this process 17 specifically and the electoral process more 18 generally is under, failure in this endeavor is 19 certainly not an option. This is why we have 20 invited you here today. The NIST and the TGDC 21 work on these recommendations while extensive 22 and very good is only the beginning of the 23 process of development, review, and adoption. 24 It is not the end. The real work for the 25 E.A.C. election officials, academics, and the TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 8 1 American public begins now. 2 The questions we pose to the panelists as 3 conversation starters related to fundamental 4 aspects of the TGDC recommendations. The 5 questions were asked so we can begin open and 6 reason discussion on the direction we're 7 charting for our voting future. 8 To open our round table, we ask the 9 question related to the development of a 10 detailed risk assessment framework for voting 11 systems. 12 I personally feel that the development of 13 risk assessment should be the cornerstone of 14 framing our public debate about the VVSG. 15 Others in the academic community agree with 16 this assessment. Professor Douglas Jones of 17 the University of Iowa has stated that, I 18 quote, if we can quantify the costs of threats 19 and defensive measures we'll be able to rank 20 threats in order of their likelihood and 21 defensive measures in order of their 22 importance. 23 Professor Jones acknowledges this will be 24 a difficult task but an essential task not only 25 so we can judge the adequacy of our voting TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 9 1 system standards but also the adequacy of our 2 recommendations for best practices and the 3 adequacy of state laws and administrative 4 procedures. 5 I thank you, once again, for joining us 6 today and look forward to our discussions this 7 afternoon. 8 >> KING: Thank you, Brian. If you can, 9 Matt, move to the first question. I hope 10 members of the panel as well as the audience 11 can see the question as it is displayed. 12 But reflecting back on Brian's comments 13 about the need for a risk assessment, I thought 14 when I first looked at voting systems, that 15 analogous comparisons were really 16 inappropriate. 17 I would hear people say that voting 18 systems should be like ATM machines or voting 19 systems should be like lottery systems, and not 20 only did I think that an analysis of those 21 environments would not aluminate voting 22 systems, I thought that may be harm done in the 23 comparisons, because voting systems are 24 somewhat unique. 25 My opinion of that changed somewhat, in TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 10 1 part, based on presentation of a gentleman from 2 the Nevada Gaming Commissions IT Security 3 Group, and it opened my eyes that maybe some 4 other people are working with models that we 5 can learn from and would be appropriate. 6 So, the question about risk assessment -- 7 and I am going to throw some items out for our 8 panel to kind of help form a starting point for 9 this discussion -- is some questions. And one 10 is: Have -- is there a consensus of the 11 definition of a scope of a voting system? 12 And for those of us who do systems 13 analysis, scoping the system is always 14 critical -- what is inside, what is outside of 15 the system. 16 Are there users? Are there managers? Who 17 are the stakeholders involved in the systems? 18 Have we defined the risk, and more importantly, 19 have we kept up with metrics by which to 20 measure risk? 21 Risk is the possibility of an event 22 occurring that has a negative impact on an 23 organization, so we think about risk in terms 24 of the likelihood of occurrence and the impact 25 of that risk. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 11 1 Have we distinguished between inherent 2 risks -- those are risks in a system for which 3 there is no control -- and residual risk, risks 4 that still remain once controlled are designed 5 into a system? 6 Do our risk models presume a capability to 7 audit all risk out of a system? And if you 8 work with auditors or if you have done auditing 9 work, we talk about the cost of auditing risk 10 out of a system as you approach the 98, 99, 100 11 percent levels of certainty, the costs of 12 auditing the risk approaches infinity. 13 And then another item that I really don't 14 see talked about in risk assessment of voting 15 systems, but we talk about it all the time in 16 financial auditing, is the notion of reasonable 17 assurance. Do we have a metric? Is that in 18 our lexicon voting systems of auditing to the 19 point of reasonable assurance? 20 In auditing the credibility of the 21 auditors in developing the risk assessment 22 models are critical. In financial auditing, we 23 see the auditors have credentials, they have 24 experience sets, they have training, they have 25 a code of ethics. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 12 1 And the IIA, the Institute of Internal 2 Auditors, one of the requirements to be an 3 auditor is: Shall engage only in those 4 services for which they have the necessary 5 knowledge, skills, and experience. So even 6 talking about who should be developing the risk 7 assessment models can be a problem. 8 In looking at how do we assign weights to 9 risk, how do we prioritize risks, in financial 10 systems we look at are the risks associated 11 with core functions within the organization? 12 Are they associated with ancillary? 13 Are risks greater for -- for continuity 14 plans; that is, contingency plans that enable 15 an election center to keep functioning; or are 16 the risks greater for disaster recovery plans? 17 The extent of the system, the change, are 18 you looking at re-engineering the system or 19 making trivial changes to the report generation 20 functions of the system? 21 So there are many models in the financial 22 world that enable auditors to identify risk, to 23 develop models of risk, and very importantly, 24 to reach consensus of those models for the 25 stakeholders. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 13 1 In the corporate model of risk assessment, 2 particularly since the Sarbane Oxley 3 legislation, the identification of who is 4 accountable for risk assessment in the model, 5 and then, finally, I think there was a question 6 about what are the allowable levels of risk. 7 And I am a pilot, so I look at the NTSB 8 reports frequently, and there have been six 9 incidents at the Austin, Texas airport in the 10 past couple of years. And I got a ticket to 11 fly out of there tomorrow, so I guess, by 12 assumption, that is an allowable level of risk. 13 But we often don't talk in elections, we 14 have a tendency to swing very quickly to the 15 absolutes, and we don't talk about what is 16 reasonable assurance, what is allowable risk. 17 So with what as my questions for the 18 panelists to kind of open the discussion, I 19 think Ron also had some remarks that he wanted 20 to open with. 21 >> RIVEST: Thanks, Merle. I prepared 22 some remarks that address both risk assessment 23 and software independence. I think it is 24 related. 25 I guess I should state first I am on the TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 14 1 TGDC, but my remarks today are my own. They 2 don't reflect the TGDC in any formal sense of 3 the E.A.C. or anybody else, so I am happy to 4 answer questions based on my experience in the 5 TGDC and so on. 6 The questions were raised, what are the 7 risks in voting systems, how to assess them, 8 what are acceptable and unacceptable risks. 9 And I think the best guide we have to that 10 so far is the Brennan Center Report on the 11 machinery of democracy. They did a careful 12 study, looked at hundreds of different risks to 13 a voting system, and tried to evaluate their 14 severity. 15 They identified software is the most risky 16 component in voting systems. The voting system 17 may contain malicious code or code that is 18 erroneous. It can change votes and cause an 19 incorrect election outcome, perhaps 20 undetectably. 21 Of course people are the source of most 22 problems, and their metric, I thought, was a 23 very interesting one, which is how many people 24 are required to cause, say, an incorrect 25 election outcome or a significant change in the TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 15 1 election outcome to happen. 2 So if you're looking at what are you 3 risking election, you're risking that we get 4 some of the incorrect election outcome, if we 5 don't get that right, then our democracy isn't 6 working properly. 7 And another risk, an associated risk, is 8 that the outcome is right but it is not 9 believed. You don't have sufficient evidence 10 to convince the loser or the populous or the 11 voters that it is the correct outcome. 12 So the TGDC proposed this notion of 13 software independence as a way of mitigating 14 some of the risks associated with software. I 15 want to talk a bit about it. It is perhaps the 16 most significant aspect of the draft of the 17 VVSG. 18 It is there to mitigate the risk of 19 software problems causing incorrect election 20 outcomes. So "software independence" means 21 basically -- quoting here -- "that an 22 undetected bug or error or malicious code in 23 the software can cause an indetectable change 24 in the election outcome." 25 Maybe it is more helpful to look at it the TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 16 1 other way around. Software independence 2 means that you are placing your complete trust 3 in the counting of the votes by the software or 4 accepting any risk that there may be incorrect 5 or even malicious software in the counting of 6 the votes. So what is wrong with software 7 independence? We heard lots of testimony in 8 the TGDC that indicate software is correct and 9 not susceptible to manipulation is beyond the 10 state of the art. 11 This is obvious to people using the 12 computers today. Gene Spafford wasn't able to 13 make it here. He had some nice words that he 14 put in his written testimony saying it really 15 isn't possible to tell whether software or 16 other technology contains what he called hidden 17 functionality. So is it possible that this 18 thing might do something sometimes that is not 19 what you expect? That is just not part of what 20 we know how to do. 21 And if you look at it from an actually how 22 you would assess it, it is really hard. A 23 voting system may contain 10,000 pages of code 24 and any one page changing an I to a J somewhere 25 may cause the program to misbehave at some TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 17 1 point which may not be revealed in testing but 2 may happen in an election. This is 3 particularly hard when the relevant software 4 may not even be available to software systems. 5 May depend on operating systems or drivers for 6 which the source code is not even available for 7 review by the tester. So it really is a tough 8 problem. Be nice if it weren't so, but it 9 really is hard. 10 So our voting systems certification will 11 probably never mean that the software in the 12 voting system is bug free and always gives the 13 correct result. Guaranteed -- we just are not 14 able to give that level of assurance these 15 days. In addition, we have the problems of 16 assuring that the software on the voting system 17 is indeed the software that was tested in the 18 first place. 19 California, I thought it was interesting, 20 they have been top to bottom reviewed, revealed 21 how easy it is on the existing voting system 22 for a virus to propagate from one machine to 23 another, changing the software and every 24 machine that the voting memory card went to. 25 So to mitigate these software independence TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 18 1 requires that the election outcome not be 2 totally dependent on records produced by 3 software. The voter must have the ability to 4 directly verify his or her choices on a paper 5 ballot, and these paper ballots must be usable 6 as a check on the electronic counts, such as 7 with a post election statistical audit. Any 8 time you are trying mitigate risk, you fall 9 into one of two major paradigms. You are 10 either trying to prevent the problem, which is 11 what you doing when you are trying to prove the 12 software correct or detect the problem and 13 recover from it and the paper ballots and the 14 post election audit are really in that 15 paradigm. Software independence fits in the 16 detect and recover paradigm. 17 So the major concern is undetectable 18 changes in the election outcome. You have got 19 to have a -- you don't want to be hood winked 20 and not even know about it. I would like to 21 view software dependence as very much like 22 being a seat belt in a car. Cars may have 23 undetected faults in the braking system or 24 elsewhere causing you to end up in a ditch, and 25 undesired outcome. Seatbelts prevent you from TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 19 1 the undesired outcome, you know, going through 2 the windshield or whatever. Of course, you 3 don't need to wear the seatbelts. You don't 4 need to do a post election statistical audit 5 but you should. 6 There is lots of issues with software 7 independence, I mean, voters may not examine 8 all their ballots. You need to consider that 9 and realize that even if significant fraction 10 of voters do look at their voter verified 11 records that you do it in confidence that 12 attempts by software to cheat the voters can be 13 detected. And you realize there are 14 advantageous when the voters actually read the 15 ballots themselves in that regard. It is also 16 the case that a post election statistical audit 17 itself may not look at all the ballots. It is 18 after all, just a statistical sample. But it 19 turns out to be, if you do the math, 20 surprisingly cheap. I mean, if you have a five 21 percent margin of victory, 20 percent recount 22 will actually get you up to over to 90 percent 23 confidence that you will have found fraud 24 sufficient enough to change the election 25 outcome. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 20 1 Of course, an audit is not required by the 2 draft of VVSG. It can't be since the draft 3 VVSG is just about equipment testing and not 4 about procedures. But a well-designed voting 5 system based on the principle of software 6 independence can yield election outcomes having 7 a high degree of confidence that they are 8 correct, that they correspond to what the 9 voters' choices are. The risk of incorrect 10 election outcome is mitigated by the use of 11 these voter verified independent records and 12 the post election audits. 13 Back to the original question about 14 acceptable and unacceptable levels of risk, I 15 personally think it is unacceptable for a 16 voting system to have the property of a 17 undetected software bug or error or malicious 18 code can undetectable change the election 19 outcome. That is something we don't need to 20 accept and shouldn't. 21 I also think it is unacceptable for any 22 single person to change the electronic outcome, 23 you know, even before the post election audit. 24 And OVET can check for these kinds of things. 25 There are reasons for doing things like TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 21 1 the OVET that went into vulnerability testing 2 that are outside of the integrity of the 3 election. There are issues of voter privacy 4 for example. Can someone figure out how you 5 voted from the records created by the -- that 6 you can't detect just by looking at paper 7 ballots. You really need to look at the system 8 and internal of that to tell whether voter 9 privacy is being well protected. 10 So there is lots of more to say about 11 different kinds of risk. There is, you know, 12 risk that we will miss out on new developments 13 and technology and so on. We will be talking 14 about that later. I think I have talked long 15 enough. 16 >> KING: Thank you, Ron. With that, then 17 I would like to open up the discussion to other 18 panels and Mike. 19 >> SHAMOS: I think I would find it useful 20 if we did a little more question defining 21 before we got heavily into the topic. So how 22 do you evaluate what is an allowable level of 23 risk? That sounds like that it begs for 24 numerical answer. That we are willing to 25 accept one in thousand or one in million TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 22 1 elections going bad. And I am not sure that is 2 what the question -- that is what the question 3 really is. So I would like to start with a 4 discussion of what we mean by an allowable 5 level of risk and how it would every be 6 assessed. 7 >> KING: Okay. Do you have thoughts on 8 that? 9 >> SHAMOS: Of course. I am not proposing 10 an answer to the question. I can tell you 11 about some experiences that we have had though. 12 I was on the project serve review committee in 13 2004 for a system for internet voting for 14 Americans overseas, and one of the exercises 15 that we attempted to go through, which we 16 ultimately discarded as foolish, was an attempt 17 to place probabilities on various potential 18 actions. So what is the chance that an 19 intruder will try to effect the outcome of this 20 election? I don't think it matters whether the 21 chance is one percent or 100 percent. You want 22 to prevent them from doing it. 23 If it is an issue that we are doing risk 24 assessment so we can decide where to spend the 25 money and spend a lot of money preventing the TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 23 1 risks that we think are very likely to occur 2 and then not spend very much money on the risks 3 that we don't think that are likely to occur, 4 that will increase the probability of the ones 5 that we don't think are likely because once 6 people figure out what we haven't protected, 7 they will turn their attention to that. 8 And I think the whole exercise of risk 9 modeling, the fundamental folly of it was 10 brought home to me in the recent mortgage 11 crisis. There was a company called Campbell 12 and Company, which was an 11 billion-dollar 13 hedge fund that lost 12 percent of its value in 14 one day because of the failure of the mortgage 15 market. 16 And when the CEO of Campbell was 17 interviewed by the Wall Street Journal as to 18 how this could possibly happen, his answer now 19 often quoted, you can see many hits on the 20 internet on this. He said, "Our risk models 21 failed." And so you can develop the greatest 22 model that you think you can develop, but if it 23 is inaccurate for some reason, it is going to 24 only give you false assurance that you have 25 really guarded against something. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 24 1 So instead of -- what I do favor is an 2 exercise in risk assessment where you attempt 3 to enumerate as many of the possible risks as 4 you can, not necessarily signing probabilities 5 to them. But then devise remediations for them 6 and attempt to place a value or cost on making 7 those remediations and then see what the whole 8 picture is of weighing the necessity of 9 evaluating intentions which is what you are 10 doing assessing when you evaluate the 11 probability of something occurring. So I 12 didn't answer the question, but just to get 13 things started. 14 >> TOREGAS: Two additional thoughts on 15 the risk question. Do you, Mr. Moderator, 16 encourage us to think about the scope that we 17 feel comfortable with. And I think if I 18 thought of the American voter, he or she is not 19 so concerned about only the machines risk 20 profile but the risk profile of the entire 21 process of elections. And I know that the VVSG 22 is looking at a machine, but perhaps we might 23 think about how we could daisy chain risk 24 models all the way up and down from end to end 25 of the entire voting process and the entire TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 25 1 election process so we could give that voter 2 the confidence not only in the machine but also 3 in the entire process. 4 The second concern that I have, and again 5 you mentioned it, that sometimes in a risk 6 analysis, we say 97 percent, we can fund 7 98 percent. We begin to run out of money, 8 99 percent, we are getting pretty close to 9 infinite and so on. 10 And the question therefore begs an issue 11 that we discussed in another forum about a week 12 ago, and that is the beginnings of some kind of 13 a cost benefit analysis which would help us 14 place the risk evaluation in the context of 15 components other than probabilities alone, and 16 you have mentioned one, which might be expense, 17 cost, how much are you willing to pay to 18 guarantee an extra nine on your string of nines 19 after the 99 gets done? And so those are the 20 two thoughts I wanted to propose that we spend 21 sometime on, cost benefit analysis, and the 22 notion that the scope of the risk assessment 23 ultimately has to be an end-to-end process of 24 the election process itself so therefore as we 25 are looking into the VVSG at the machine TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 26 1 itself, how can we make it so that the risk 2 model we discussed and the risk model someone 3 is sent off to create can in fact be related to 4 additional risk models up and down that chain. 5 Thank you. 6 >> KING: John. 7 >> WACK: A bunch of points. Everybody 8 said very interesting things. 9 One thing that I want to point out is 10 that while a risk assessment, I think, is very 11 well justified, very important, it shouldn't be 12 used too much to arrive at decisions as to -- 13 for example, you know, is software independence 14 justified or not. 15 In my experience in this process, a number 16 of people have said -- have come across with 17 reasoning of the form of various things such as 18 software independence and whether we will have 19 to continue to using paper records is going to 20 be expensive, do the risks, do the threats, 21 really justify this, prove it. 22 I want to point out, in the answers to 23 these sorts of questions, there are a number of 24 things that we have to keep in mind. One is 25 that, as everybody knows, voting system is TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 27 1 complex information technology equipment. 2 And in fact, you can even argue in some 3 ways, because it is used infrequently and when 4 it is brought up, it has to run correctly, 5 things of that sort. It might even be more 6 complex than when we generally think of -- when 7 we think of desktop systems in large 8 organizations and complex networks. 9 There has already been a fair amount of 10 risk assessment work in this particular area. 11 Where I work in the government we have been in 12 charge of writing guidance for other agencies 13 on how they ought to manage their networks and 14 manage their information technology. 15 And pretty much what is in the 16 recommendations right now starts pretty much 17 with the guidance that we have issued thus far. 18 We have taken advantage of risk assessments in 19 that particular area and have pretty much said, 20 they apply to information technology, we think 21 very much they apply to voting systems as well. 22 So I would not like to see a risk assessment 23 kind of start at the very beginnings and not 24 make use of that existing work. 25 The other thing that I would like to say TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 28 1 is that, in an assessment of risks, I think it 2 is important to not look solely at what sorts 3 of threats are out there and the likelihood 4 that they'll be exploited or what sorts of 5 vulnerabilities, but also look at the usability 6 of things. 7 One of the other things that has come 8 across in doing risk assessments in helping to 9 manage systems more securely is to look at 10 their usability. Are they easy to manage? If 11 they aren't easy to manage, if the controls are 12 difficult to use, they won't be used. 13 If paper records are produced that contain 14 all of the information you need for very 15 precise audits, but the paper is thin, it tears 16 easily, it jams in inexpensive printers, it is 17 less likely that audits will occur. 18 So the overall usability of the systems, 19 the ease of managing them, the ease of using 20 some of the audit capabilities that are out 21 there today also have to be factored in, in 22 this overall assessment, I think, to basically, 23 ultimately arrive at what I believe the goal 24 is, which is: Shall we go ahead with certain 25 things that have been proposed, such as TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 29 1 software independence. 2 So maybe I am a little bit all over the 3 place here in my comments, but I guess mainly 4 what I want to say is, I think many things have 5 to be looked at in this assessment overall, and 6 I would like to just caution that we take 7 advantage of what is out there already, in 8 terms of managing information technology, and 9 also look very carefully at how well the 10 controls that are out there today are being 11 implemented and whether they are sufficiently 12 usable. 13 Sometimes making things easier to use for 14 people makes them far more secure than 15 otherwise. That's really all I had to say. 16 >> KING: Okay. Alec? 17 >> YASINSAC: Yes, thanks. 18 Dr. Rivest mentioned the Brennan Center 19 study, and that is an excellent study. In 20 fact, there are a couple of excellent studies 21 from there. And there are several sources, as 22 John mentioned, of threat models out there. 23 The California top to bottom review did an 24 excellent threat model, and there have been 25 several done at Berkeley and other places. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 30 1 But from what I've seen, there has not 2 really been anyone to try to put a structure on 3 these things. The Brennan Center study is a 4 perfect example. It is a wonderful study, very 5 comprehensive, but it is all rules of thumb, 6 it's all expert opinion, and there is very 7 little quantification laid down to it. 8 For example, there is no structure laid 9 over those threats that would allow somebody to 10 reason about classes of threats beyond the 11 specific instance that Dr. Rivest gave of the 12 ability of one or two -- a small number of 13 people to have an impact. That was the 14 exclusive focus of that particular study. 15 So how much impact a single line of code 16 would have on the ability to conduct a safe 17 election, there is really no way to reason 18 about that with the models we have. 19 And so in order to even conduct an 20 effective study -- and we have done it Florida 21 state, now six of these studies -- and in order 22 to conduct an effective study, you have to have 23 a much more precise, much more systematic 24 classification system to be able to detail what 25 the threats are so that the threats can be TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 31 1 analyzed in the context of the system that is 2 being evaluated, and those tools just aren't 3 there right now. 4 >> KING: Thank you. I wanted to come 5 back to something that Mike said about the 6 mitigation of risks, and I am curious, from our 7 two election officials that are on our panel -- 8 because obviously, any deficiencies in risk 9 mitigation cascade down and accumulate at the 10 implementation level. 11 I am curious whether you have any thought 12 on the role of risk mitigation at the 13 implementation level, as opposed to higher up 14 in the development cycle and the design. I 15 don't want to put you on the spot, but -- 16 >> JOHNSON: I think it is extremely 17 important to have it at that level, too. I 18 guess we're sort of -- like the individuals 19 have already said, a cost benefit analysis is 20 something that is very important to us, because 21 we obviously have to, you know, afford to 22 purchase them and keep them up, and also what 23 John is saying the usability and trying to 24 marry those two, keeping all of that in mind, 25 when you're developing models, the assessments, TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 32 1 and the classification system is very important 2 to us, as election officials, because we have 3 to take that system and obviously go back to 4 our local, county individuals and then our, you 5 know, precinct officers, who are predominately 6 elderly, to be able to use these machines, much 7 less the voter. 8 We see the forest, and it is a pretty big, 9 thick, dense forest right now. 10 >> THOMAS: I would concur with Sarah. 11 What we have up there are methods of avoiding 12 risk that come from older systems, and we're 13 now bringing new systems in, so there is this 14 process of trying to adapt the older checks and 15 balancing and the other assessments. 16 The higher up that is done, I think that 17 is a good approach, but when it ultimately 18 comes down to the implementation, which is with 19 all of these folks -- but that's the next level 20 that is really going to integrate these new 21 systems into what our current laws, procedures, 22 and whatnot have to avoid the risks. 23 >> KING: Alec, I wonder if you have a 24 comment on that, because I think Chris has just 25 added another dimension to your observation; TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 33 1 and that is, the legacy threat models may no 2 longer be appropriate to the new technologies 3 that we're rolling at. 4 >> YASINSAC: Well, I think that is 5 absolutely true. It really boils down to the 6 notion that led to software independence, is 7 that changing -- electronic changes can be 8 very, very easy in large scale. Changing paper 9 things in large scale generally is very, very 10 hard. 11 Conversely, I am not sure that the models 12 for protecting paper have moved forward to 13 match the precision that is now demanded in our 14 elections and the ability to report quickly and 15 report accurately and the conflict that is 16 created between the reporting of the first 17 count as an electronic count that may not 18 include all of the ballots, that may not have 19 the precise construction of all of the input of 20 the public, and then producing a second count 21 that has additional information that may 22 conflict with the first count. 23 So just the damage to public confidence 24 that that notion has is something that, I 25 think, hasn't been addressed, and it is TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 34 1 directly because of the emergence of the 2 electronic aspect of voting. 3 >> KING: Yes? 4 >> TOREGUS: Just to piggyback on that, on 5 Section 8 of the VVSG, there is a beautiful 6 diagram, a process diagram, of what happens 7 during an election, and stimulated by what 8 Christopher -- if I might call you that -- 9 mentioned. I wonder if the people who put 10 together that flowchart were able to somehow 11 put it side by side with what people might call 12 the legacy process and make sure that the kinds 13 of computer relationships and the computer 14 standards that are now being promulgated are 15 being based on a theoretical concept or on a 16 concept that actually matches the reality on 17 the ground. 18 I am not speaking about whether the 19 reality on the ground is good or not. I am 20 just saying that if it is not matching, then 21 we're going to incur significant change 22 management cause of every organization that 23 conducts elections. 24 And whereas we can easily -- let's say 25 that -- easily change something that the TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 35 1 machine does, when we talk about thousands and 2 thousands of institutions, we have to be a 3 little bit more cautious. 4 So to look back to my question, when 5 the -- I think it was 8.1.2, when the procedure 6 diagram was put together, was that an idealized 7 procedure, or does that reflect the current 8 state of practice in the field? I would like 9 to know that. 10 >> KING: Okay. I think we're going to 11 find out. 12 John? 13 >> WACK: I don't know if you will not. 14 First of all, thank you for calling it 15 "beautiful," "pretty," I don't remember, but if 16 you like them, I have some artwork myself that 17 I could interest you in. 18 The models were put together by 19 researchers at NIST working with members of the 20 TGDC, but also looking at previous models, 21 diagrams and talking with numerous election 22 officials who have been involved in the 23 development of previous versions, and they 24 simply form what is thought of as kind of the 25 general flow of operations, general flow of TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 36 1 what happens in various aspects of voting, 2 pre-election and so on, so forth. 3 And it is there sort of as a framework for 4 the requirements. In other words, you know, 5 the requirements are based upon the general 6 practice, you know, as shown by these flow 7 diagrams. 8 Are they up to date with existing 9 practices? We believe they are. But I should 10 caution that they are also fairly general. 11 They don't get into too many specifics. 12 They're intended to be kind of what you would 13 normally expect to happen, regardless of what 14 state or what county. 15 And then I also wanted to piggyback myself 16 on Alec's point, too, about just emphasizing 17 again, that, for example, if software 18 independence results in -- well, the VVSG, or 19 the recommendations, I should call them -- 20 refer to it as an independent verifiable 21 record, but if it is pushing paper, that's fine 22 and dandy, but an assessment of all of this 23 needs to ensure that paper itself is 24 sufficiently usable to be used as an auditing 25 tool. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 37 1 I think that that may have got lost a 2 little bit or maybe not addressed as well as 3 perhaps it needs to be. We've heard a lot of 4 feedback from people in the election community. 5 And essentially if it's -- if an approach 6 like this gets used generally, it has to be 7 extremely usable, and more work definitely 8 could be done in that area. 9 >> KING: Okay. Juan? 10 >> GILBERT: Yes, Alec's comments prompted 11 me along a certain line of thinking. I am not 12 a risk assessment expert, but it brought up 13 some questions when you mentioned the Brennan 14 study and some studies you have done. 15 One of the things, when we're talking 16 about software independence, is there such a 17 thing as risk assessment independent, such a 18 way that the risk is independent of the 19 specific voting equipment? 20 So listening to the conversations we've 21 had so far, it appears that -- it sounds as if 22 the actual assessments are highly dependent on 23 the actual equipment that is being used. 24 So one of the things that came to my mind 25 is this idea of classifying these threats with TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 38 1 a different metric. 2 So I had a question for you, Alec. Is 3 there a metric such as the number of ballots 4 impacted by a threat to be used to classify 5 threats? So if you had a metric such as that, 6 to me it appears that would be independent of 7 any system. 8 So if you had catastrophic, moderate, low, 9 none classifications, and you had a number of 10 ballots that were impacted by each level, if 11 you had such a model, would that be independent 12 of any system? That's my question. And does 13 such a model exist? That is my question. 14 >> YASINSAC: As I said, I am not aware of 15 one that exists, and I did put one together for 16 this panel, but I didn't get it in time to be 17 distributed. 18 And one of the attributes of the model 19 that I constructed was the impact, and I did 20 not go so extensive as you, but exclusively 21 mentioned that it was wholesale or retail. 22 Either it was a large-scale attack or a 23 small-scale attack of a few ballots. 24 So that is an attribute that I think is 25 essential to a threat model. And there are a TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 39 1 number of them, as I have been working on this, 2 actually, for a good period of time, and 3 interacting with a lot of different people, 4 there were many attributes that popped out as 5 being obvious to me that they fit in. 6 Where that kind of model obviously gets 7 difficult is when you get further down the tree 8 and you need to know what level of detail is 9 appropriate to not exclude some equipment or 10 some types of systems, and then to take the 11 discussion pertinent to the threats on the 12 scale that you need to be able to discuss, 13 whether that scale is for cost, for risk 14 voidance at the point of the elections 15 officials, or for consideration of developing 16 new and innovative solutions for voting 17 systems, how do you design that threat model. 18 So it has to be -- but I think there are a 19 group of attributes that are widely acceptable 20 and that we could identify if we were to spend 21 some time on it. 22 >> KING: Okay. I wanted to add, too, I 23 had the opportunity to see a presentation by 24 the state election director of Louisiana on 25 their contingency planning model, and that's an TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 40 1 interesting model to look at in terms of 2 realistic assessment of what can go wrong in an 3 election in a scale that is really staggering. 4 So perhaps in addition to the models that 5 have been discussed, there are some things that 6 are in circulation out there already that can 7 be brought in. 8 What I would like to do, if there are no 9 more comments on this -- I am sorry. Peter? 10 >> RYAN: Yes, I would like to follow up 11 on Brian's comments about the measures, because 12 we actually toyed with a very similar sort of 13 measure back in Utah that was slightly a 14 different measure of impact on the number of 15 votes that can be changed divided by the number 16 of people who would have to collude in order to 17 do that. 18 That seems to be an actual measure in some 19 respects. It seems to work quite well in a 20 class of systems, but we quickly realized if 21 you try to push it to the extreme, it starts to 22 break up. 23 The classic example is, if you start this, 24 to think about the graphic schemes. There, the 25 idea is to achieve software independence and so TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 41 1 on. Perhaps the risk boils down to, is there 2 some undetected flaw in the mathematical 3 arguments, the logic, and crypto, and so on and 4 so forth. 5 So there you get experts that look in the 6 system and judge [Indiscernible] and so then, 7 the number of people who, in some sense, might 8 collude in making a mistake of the system, is 9 an entirely different kind of class of people 10 than the class of people who might work with 11 election officials. 12 So you try to -- I usually try to apply 13 this measure across the entire spectrum of 14 system, and I don't think it works for that 15 kind of reasoning. 16 For example, if you are ignoring it -- the 17 kind of person who would be involved, I hardly 18 think that Ron would find a flaw in the system 19 and keep it to himself or something. The odds 20 of that happening are astronomically lower than 21 the a number of election officials at a polling 22 station colluding to undermine the outcome of a 23 polling station. So these kind of measures are 24 interesting, but certainly, if you want go 25 across the entire spectrum the system, I guess TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 42 1 it gets a bit dodgy. 2 >> RIVEST: I think the Brennan Center, 3 some aspects, I think they were talking about 4 election officials would certainly be expected 5 much higher integrity than a typical voter, you 6 know, where if you are doing something like 7 chain voting where you just have to persuade a 8 voter to turn over their vote, that would be 9 persuading an election official to violate 10 their oath, so that would be a much, much 11 higher barrier. And so I think that that kind 12 of consideration is very relevant, yeah. 13 >> KING: Yes, Costis. 14 >> TOREGAS: One last thing on 1B or 1A, 15 you asked how do you value what is an allowable 16 level of risk? It seems to me that we also 17 have to stop and pause and think who says 18 allowable? And ultimately, from my experience 19 of decades of working with government 20 officials, ultimately the people that have to 21 decide what is an allowable level of risk are 22 not the so-called experts, computer scientists 23 or even the industry, but it is the people who 24 are actually going to buy the machines and make 25 sure that elections run well. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 43 1 So we have to start thinking about how do 2 we communicate very complex issues of risk 3 assessment and risk management to people who on 4 a daily basis have to make very complex 5 decisions? Somewhere along this line we have 6 to start thinking about communication to other 7 sectors. And that is something that I am very 8 much an advocate of, instead of having sectoral 9 dialogues like we have a computer scientist and 10 then the advocate groups, I think we have to 11 learn to begin to discuss things across those 12 groups because without doing that, it is very 13 difficult to identify questions that I have 14 just raised such as who defines. 15 >> KING: Costis, do you think that is 16 also related to something I think we have heard 17 from two panelist about the need for an 18 expanded lexicon and a taxonomy so that we 19 can -- when we do share these ideas, it is an 20 accurate sharing of the ideas? Okay. Mike. 21 >> SHAMOS: In response to the question of 22 who defines, initially it is the state 23 legislatures. And typically when they have 24 acted to define the allowable level of risk, 25 they have defined it as zero. There are TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 44 1 numerous requirements in statutes in all the 2 states that refer to things like absolute 3 accuracy and prevent every person from 4 interfering or tampering with the equipment. 5 So they are phrased in absolutes. 6 For federal -- elections for federal 7 offices, it would be the Congress. There is 8 already a definition of an acceptable error 9 rate, and it will end up being left to the VVSG 10 to say what an allowable level of risk is 11 because the states will inevitably accept that 12 unless so it is out of kilter with their 13 expectation or what is publicly acceptable. 14 If you go up to somebody in the street and 15 you say, what is an allowable level of risk 16 that somebody could steal an election? They 17 would say zero. We just can't have it. That 18 is completely unrealistic of them to say that. 19 But it is certainly the desire. 20 We have had a lot of useful discussion 21 about this question. Unfortunately, I have the 22 feeling that we didn't answer it. So I am not 23 sure that what we have done is of great utility 24 to the E.A.C. 25 >> KING: If you would and I have TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 45 1 another -- I have several here on the table. 2 Mike, I think that is going to be recurring 3 dilemma throughout today. But what the E.A.C. 4 has asked of us and the Standards Board is do 5 the best that we can, and I think along Costis' 6 suggestion is get that dialogue going. And I 7 did not see the order in which things went up 8 here, but Chris I saw yours first, so I call on 9 you. 10 >> THOMAS: Just a quick comment. 11 Addressing what the E.A.C. can do. And one, I 12 would give the E.A.C. credit and Brian Hancock 13 in particular for his Denver conference that he 14 had earlier this year that did exactly that. 15 Bringing together the various elements. 16 And the other thing I would point out is 17 that the E.A.C. is moving forward with 18 management guidelines. And to some extent, 19 management guidelines are analyzing risk. In 20 other words, how really you modify the legacy 21 system and how you operate elections with these 22 new systems. 23 And this comes back to beg the question, 24 who makes the risk analysis? And obviously it 25 is not any one sector. It needs everybody to TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 46 1 be involved in that. And as Michael indicated, 2 ultimately legislatures are the ones that weigh 3 in there, and I think they would find some 4 decent guidance coming from a federal level 5 with input from the wider community and making 6 this risk analysis. As opposed to just them 7 making the analysis, which as Michael 8 indicated, they are rarely going to do anything 9 more than zero. 10 >> KING: Okay. Thank you. Daniel. 11 >> CASTRO: In response to, you know, 12 question that we have been talking about about 13 what is an allowable level of risk. I would 14 argue that one thing is that is what states and 15 Congress is willing to pay for. I think when 16 you ask the people on the street, yes, they 17 want zero and legislators want zero risk. But 18 as a public policy question, this comes down to 19 how much are you willing to spend? And this is 20 what Costis was talking about. 21 I do want to go back to another point that 22 we have talking about in terms of risk 23 analysis, and it seems like there are some 24 consensus at least that we -- there is no 25 defender of risk analysis that has been done TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 47 1 yet and more work here. 2 When you are doing a risk analysis, there 3 is a number of steps but among those you have 4 prioritizing risk, identifying policies to 5 respond to those risks and then choosing the 6 best implementation for those policies by doing 7 a cost benefit analysis. 8 It seems like, and I don't want to get 9 ahead of where the discussion is going, but if 10 we are saying that software independence is the 11 policy we want to choose, it seems like that is 12 getting ahead of ourselves because we haven't 13 completed the risk analysis yet and we haven't 14 completed the cost benefit analysis to decide 15 if that is the policy that we should accept. 16 >> KING: Okay. Thank you. Ron. 17 >> RIVEST: Yeah, I wanted to respond to a 18 suggestion that Mike Shamos just made that the 19 VVSG may be the place where the allowable level 20 of risk is determined. And then in some sense, 21 you know, what the VVSG does is sort of set up 22 the proposal or maybe not VVSG, would set up a 23 framework of what can be certified. But the 24 actual risk that is run by any election 25 jurisdiction, really depends on the procedures TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 48 1 as much as anything. What it does is provides 2 a range of acceptable levels of risk. 3 The question really is, does the equipment 4 that would be certified support, you know, a 5 level of risk controlling the risk in a way 6 that would satisfy the potential customers, the 7 election officials and the voting public? You 8 know, for example, I mean you may have election 9 equipment which, you know, provides the option 10 for chain voting, for the voter, if they can 11 take the ballot outside the polling place and 12 exchange it, that is a say procedural question. 13 Similarly, there may be an option on the 14 voting equipment which allows for a set up 15 validation so you can check the software on the 16 system is really the software that is supposed 17 to be there. But those are procedural choices 18 that the jurisdiction has to find as to how to 19 enforce voter movement or either to exercise -- 20 the question in some sense is does the VVSG, 21 you know, provide a range of choices to the 22 election officials that allow them to get 23 within their comfort zone for a level risk. 24 >> KING: Well, thank you. I think that 25 is an excellent start in terms of the scope and TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 49 1 the level of participation that we are looking 2 for in these questions. And in order to keep 3 on the schedule, I would like to remind the 4 panel at the end of today, towards five 5 o'clock, there is going to be an opportunity to 6 each panel member to come back and address 7 issues that either have become better formed in 8 their mind during the panel discussion or that 9 they really want to make sure or emphasize, and 10 I am trying to also take notes to help with 11 some summation of those points as a way of 12 prompting our memories when we get to that 13 point in it. 14 But at this time, I would like to move to 15 the second question. Matt, if we could. The 16 2005 VVSG states one of the goals for the next 17 iteration of the VVSG was to create performance 18 standards that promote innovation rather than 19 design-oriented standards that limit design 20 choices for potential manufacturers. 21 And Juan, I think you have agreed to open 22 up the discussion on that topic. 23 >> GILBERT: As you can see, there are two 24 direct questions underneath that initial 25 introduction and getting right to those TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 50 1 questions, you know, where does this document 2 meet or exceed this goal? Where does the 3 document fall short? When I read the VVSG, I 4 think what I liked about it very much was the 5 performance standards are clearly defined with 6 respect to, for example, evaluation, actual 7 metric that need to be measured and calculated 8 and the associated benchmarks. 9 Now, I think these are things that are 10 good for the VVSG or any standards guidelines 11 to specify these kinds of things. So I applaud 12 the VVSG for putting those in there. Now, I 13 know some people will question the values for 14 those benchmarks and say, are those valid and 15 how broadly can they be generalized? And I 16 have a comment on that in a second. 17 As I address the second question up there, 18 is the concept of software independence as 19 defined by the TGDC recommendations too 20 technologically restrictive? If so how would 21 you change it to be more expansive to include 22 more innovation? So these are two questions we 23 will address. 24 Now, I think the language in the VVSG is 25 clear on software independence. In fact, I TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 51 1 could take it and give it to a nonsoftware 2 person and say, here, read these two sentences 3 that define software independence and they say, 4 oh, okay, I got it. To me, that is a test to 5 say the language is clear. And so that is, I 6 think that is an accomplishment in the VVSG. 7 Now, although I consider this to be an 8 excellent start with clear language, the VVSG 9 also says that it makes a statement that paper 10 schemes are software independent as an example. 11 And I think there are issues and questions 12 about making such a claim, does that exclude 13 other things or is that a strong recommendation 14 to use paper? Which brings on my next point, 15 which is, getting to innovation. Looking at 16 the VVSG in my opinion, I think functional 17 requirements versus hardware and software 18 requirements are pinnacle and most important. 19 And I think the VVSG has these functional 20 requirements in there. I think that promotes 21 innovation. There is going to be debate about 22 the different benchmarks and metrics to say how 23 did we come to those and do those across? But 24 I think the definition in particular for the 25 innovation class gives you the opportunity to TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 52 1 challenge some of those if you do have 2 innovation. 3 After a few innovations are considered, I 4 think we will discover how well VVSG 5 accommodates those. But I don't think we can 6 get into this mode of what I call analysis 7 paralysis, where we overanalyze something to 8 the extent where we never get anywhere and 9 never even get a case to test the VVSG. 10 So what I like to do is cut my comment 11 short. I could go on with some additional 12 things, but I would like to get into answering 13 those questions up there, the two bullet 14 points. And I think that is a good way to lead 15 the discussion. I turn it back to Merle. 16 >> KING: Thank you, Juan. Daniel. 17 >> CASTRO: Yeah, I just wanted to start 18 off by -- I mean, the question up here is I 19 think a good question. And it goes to the 20 point should we have functional standards or 21 design standards? But just starting off and I 22 will put this out there. I would guess there 23 might be some disagreement on it. 24 I do think the software independence 25 although it is -- I say this in my written TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 53 1 statements, it uses functional standard 2 language but ultimately it is really a design 3 standard. When you look at it, the end result 4 of it is it is forcing voting systems to only 5 use software independent or any voting system 6 that is not using software. To me, that is a 7 design standard. So when we are looking at 8 this, yes, I would say that is technologically 9 too restrictive. Whether or not that is good 10 or bad for security, we can discuss that. But 11 is it a design standard or functional standard? 12 I would say it is definitely a design standard. 13 >> KING: Okay. Thank you. Mike. 14 >> SHAMOS: So I think there is a 15 significant block of folks on the panel today 16 who have issues with software independence as a 17 concept. I am one of them. And what I want to 18 do is raise the what I think are from my point 19 of view the major issues. I think software 20 independence is unarguably wonderful, warm, and 21 maternal. We certainly don't want to say that 22 it is okay for software on its own to influence 23 the outcome of an election. And I don't think 24 though that by rejecting software independence 25 we are saying that it is okay for software to TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 54 1 do that. 2 So it starts out with a kind of -- with a 3 motherhood feel to it that seems hard to argue 4 against. It doesn't go far enough. What 5 software independence says is that the software 6 itself can cause an undetected change in the 7 outcome of an election, but it doesn't take 8 into consideration other things that might 9 cause an undetected change in the outcome of 10 the election. 11 What software independence says is you 12 can't rely only on the software. You are going 13 to rely on something else. And yet there is no 14 discussion of the something else and its 15 integrity, and if the something else is 16 handling of traditional pieces of paper in the 17 traditional way that have always been used to 18 influence the outcome of American elections, 19 then software independence isn't so good. 20 Because what it is doing is turning over the 21 responsibility to system that have known flaws, 22 and so I think software independence, if there 23 were another statement that said what is the 24 other thing that we are going to rely on if we 25 are not going to rely on the software, and we TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 55 1 could evaluate that, then we would get 2 somewhere. 3 There has never been a comparative study 4 of the security of D.R.E. systems with the 5 security of paper-based systems. It has never 6 been done. And there seems to be a belief that 7 once you put something on paper, it either 8 becomes sacrosanct, is unalterable, or over the 9 centuries we have developed these excellent 10 paper handling methods that guarantee that no 11 human being can influence the outcome of an 12 election. 13 How about if we had a standard of human 14 independence, which is so to say that no 15 undetected human can cause an undetected change 16 in the outcome of the election. You wouldn't 17 argue that either. That is a motherhood kind 18 of thing to say. We know probably can't 19 achieve that either. And I think that what we 20 are doing by requiring software independence is 21 frankly I think it is a subrose way of simply 22 mandating paper trails. 23 Because the VVSG itself even says that the 24 only known systems that achieve software 25 independence are paper trail systems, and it TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 56 1 admits begrudgingly that possibly somebody 2 might come up with something else in the future 3 but they are relegated to the innovation class 4 for which we have no standards because we have 5 never seen such a thing before because we don't 6 think it can be done. And I think the effect 7 that has on potential developers of software 8 systems is extremely negative. 9 Also, I haven't been able to find another 10 field of endeavor, maybe somebody can enlighten 11 me in which software independence is a 12 requirement. Not in the handling of nuclear 13 weapons, not in the handling of passenger 14 airliners that's hold 750 people. Not in the 15 handling of trains and other devices in which 16 human life is at risk. We don't require it 17 there. And the reason is we don't know how to 18 make software independent systems. There is no 19 book. There is no manual. There is no set of 20 standards that tell you how do you make a 21 system that is completely software independent. 22 What are the other things that you are 23 going to shove the responsible on and how do 24 you make those good? And so to show up with a 25 set of standards that say, ahh, we have this TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 57 1 magic thing called software independence and 2 you have got to have it. We can't tell you how 3 to do it, but you have got to have that. And 4 oh, by the way, pieces of paper do it. That is 5 not hanging together for me. 6 >> KING: Alec. 7 >> YASINSAC: Stole a bunch Of my 8 comments. 9 >> SHAMOS: I am sorry. I should have let 10 you put yours up first. 11 >> YASINSAC: I think there is no doubt 12 and in fact, I think most of the folks that 13 propose software independence would agree that 14 the goal here -- and in fact it was in 15 Dr. Biford's statement that he just doesn't 16 feel like a voting system can be devised that 17 doesn't have a paper trail. And I think that 18 is a fairly common opinion among a lot of the 19 folks, in fact maybe everybody universally that 20 supports the software independence viewpoint. 21 And it is understandable. 22 I certainly understand why it makes a 23 difference, as I mentioned before. It is 24 clearly harder to change lots and lots and lots 25 of paper ballots in a single stroke of the pen TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 58 1 or single movement of the keyboard or to 2 generate. We can argue the merits of that at 3 length and ad nauseam. But what I would like 4 to suggest and ask is part of the VVSG and a 5 part of the consideration of moving forward, 6 instead of arguing about whether it is a paper 7 trail or an amendment or not, is to back up and 8 try to describe what the properties are that we 9 want this paper ballot to have. 10 Does it have to be human readable? Does 11 it have to be immutable and in what sense does 12 it have to be immutable? Does it have to be 13 storable? Does it have to withstand heat? 14 Does it have to be something that can be seen 15 or something that can be read or interpreted in 16 the disabled community without any help or 17 without devices? What is it about this paper 18 trail that makes it essential to the voting 19 process? 20 Because none of that that I can -- I 21 couldn't find any of that in the VVSG myself. 22 There was a discussion, I believe, in chapter 23 four. But the definitions for example of IVVR 24 and VVPT and VVR in the document were really 25 circular and were of little to no help. The TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 59 1 discussion of the testing gave a better insight 2 to the notion that software independence 3 requires something that people can see. 4 Something that they can feel. What are the 5 driving properties of that medium? 6 >> KING: Okay. Thank you. John, I think 7 you were next. 8 >> WACK: Let's see. My role is with 9 regard to this question really is just to talk 10 about what is in the VVSG, and I wanted to 11 point out that I think the TGDC has the 12 resolution for software independence in 13 December of 2006. 14 And so the immediate question after that 15 was let's make sure that those systems that use 16 paper records do so in a way that is as usable 17 as possible or as usable as it is possible to 18 make them in the amount of time we had 19 available. So the requirements for example for 20 voter verified paper trail systems were in my 21 opinion improved a good bit, especially in the 22 way that paper is dealt with. The reliability 23 requirements were approved a great deal to 24 ensure that failure rates of printers and 25 accordian jams of paper and things of that sort TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 60 1 should not happen. 2 And then ultimately in the testing, there 3 is a large scale volume testing approach. 4 Where in essence there is going to be a mock 5 election as part of the testing campaign. And 6 systems will be tested pretty much from the 7 beginning to the end and end testing. And that 8 in itself, in my opinion will put to rest many 9 problems with voting systems and many problems 10 with use of paper records. 11 So I do believe that while S.I. states a 12 goal, that it is backed up with more attention 13 paid to making sure that paper records are 14 usable. Could more work be done in that area? 15 Definitely so. I think that is definitely 16 true. Could IVVR be examined more closely and 17 more of a performance aspect be given to the 18 requirements. That is definitely true. I 19 think though that what is in there essentially 20 constitutes good starts in those areas. And I 21 would like to just make sure that people look 22 at the area first and form conclusions 23 afterward. 24 >> KING: John, if can I ask a follow-up 25 question, and then I will go to Peter and Ron. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 61 1 In one of the projects that preceded the NIST 2 management of the VVSG was the IEEE P1583 3 project. And on that project, we struggled 4 with the notion of functional specifications 5 versus design specifications, curious if those 6 kind of questions came up early in the NIST 7 project and how those were resolved. 8 >> WACK: Well, they did, and they are 9 terms of art to a certain extent, and the 10 conclusions I, myself, drew and some others 11 were that you could go down a rathole pretty 12 quickly arguing about what is a design 13 requirement, a functional requirement, what is 14 a performance requirement. 15 And design requirements, for example, 16 which supposedly limit the design to a specific 17 approach, are not necessarily bad. Performance 18 requirements are not necessarily good. 19 Performance requirements, where they make 20 sense, are good. 21 For example, in the VVSG, there are 22 performance requirements for usability of 23 interfaces, and the performance aspect really 24 is: We don't care what it looks like, to a 25 large extent. We don't care how big it is, how TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 62 1 small it is, what color it is. We just want to 2 make sure people vote accurately when they use 3 it. So you're writing to a certain performance 4 level of accuracy. 5 In some other areas, security is a good 6 example. It is not a good idea to strictly 7 say, it must be secure. You know, you have to 8 get -- well, it is probably a complicated 9 discussion, but there are a number of ways in 10 which security controls can be implemented 11 incorrectly, and so to a certain extent, there 12 has to be some fairly specific design-related 13 requirements. 14 I would say, though, in general, we wanted 15 to go more towards performance-based 16 requirements, obviously because it just makes 17 it easier for vendors to come up with good 18 solutions. 19 So I think that was pretty much what 20 people wanted to do. In areas where that 21 didn't occur, perhaps that can be approved upon 22 in the second draft. But in some areas, I 23 think it was justifiable to stick with design 24 requirements. 25 I did mention functional requirements. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 63 1 Functional requirements are how a system -- 2 what a system ought do, and they're kind of 3 in-between design and performance. And again, 4 I'll reiterate, you go down a rathole pretty 5 quickly with a lot of these terms. 6 But in general, yes, we wanted to go 7 towards performance as much as possible. 8 >> KING: Thank you, John. 9 Peter? 10 >> RYAN: Yes. Well, there is quite a few 11 issues here which need clarification. I can't 12 speak to the details on how things are worded 13 in the document. I can speak from my own point 14 of view. 15 To me, certainly software independence 16 doesn't mean paper audit trail, certainly, in 17 the sense of a VVPAT, human readable, 18 necessarily. That is the first comment I need 19 to make. If that's what the document seems to 20 imply, that would be an unfortunate phrasing of 21 the document. 22 The other point is that, certainly when I 23 came across the term first, I guess from Ron, I 24 took it as meaning it is not the be end, the 25 end all. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 64 1 To answer Mike's comment, yes, I fully 2 agree if you make the system, it's 3 dependability is independent of the software, 4 but you need to also investigate a lot of other 5 things. And to a lot of us, myself included, 6 one of the goals we're going through is to make 7 the particular integrity of the system 8 independent of essentially all components of 9 the system -- including human beings, hardware, 10 and so on and so forth. 11 So I viewed software independence as a 12 sort of first step, I guess largely on the 13 grounds -- as I think Ron pointed out -- is to 14 make sure it is the most critical, most 15 vulnerable piece of system, for example, D.R.E. 16 systems. 17 So that was my view, that was the first 18 step, but surely, we need to go beyond that. 19 And my final point, I think, is that you can go 20 beyond that. 21 And there seem to be certain comments here 22 that systems don't exist or seem unconceivable 23 that could achieve these goals of software 24 independence and perhaps independence of humans 25 and officials and so forth. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 65 1 And I think we have a group of concepts, 2 several of us, that say that is actually not 3 true. Now is not the time to talk in detail 4 about them. But I believe such systems do 5 exist, at least in theory. 6 My question is about whether if, in 7 practice, they're pretty reliable and we have 8 to argue about the cost of employing them, and 9 whether they are credible in the eyes of the 10 electorate at large, the stakeholders, and 11 issues like that. 12 But in principle, I think there are 13 systems. We have a group of concepts that 14 something like this can exist. I'll leave it 15 at that. 16 >> KING: Ron? 17 >> RIVEST: Thanks. I also wanted to 18 respond to some of the points that Mike Shamos 19 raised with respect to software independence. 20 You brought the airplane analogy, which is 21 one that is commonly brought up, and I think it 22 is an interesting one to sort through and to 23 think about. 24 It is actually an interesting one for lots 25 of reasons. An airplane is actually the prime TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 66 1 example of multiple redundant systems and 2 backup systems, which is really what we're 3 talking about with software independence here, 4 is having another way of being able to achieve 5 a vote count. 6 They've got multiple hydraulic systems. 7 If one of them goes down, you have another one 8 there. The pilot has a separate oxygen tank 9 and oxygen mask, in case there should be 10 decompression, you know, so that he can fly the 11 plane even in spite the failure of a 12 compression system. 13 There is even talk about the plane being 14 able to land itself if it should be overcome by 15 a terrorist and such. So there is lots of 16 mechanisms for protecting against failure of 17 any one given component. 18 But even more so, the airplane analogy 19 doesn't go far enough because, when you think 20 about it hard, voting systems have a higher 21 calling, if you will, than an airplane does. 22 An airplane, when it functions correctly, gets 23 the passengers from point A to point B. When 24 you are there, you know you're there, and you 25 know you're in the right town. Although when I TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 67 1 landed in Austin last night, the taxi driver 2 said, Do you know where you are? I said, No. 3 He said, You are in London. This is London, 4 England weather. But you know where you are 5 when you get to the airport. 6 So the correctness of the operation of an 7 airplane is sort of self-evident when you land 8 at the airport. The correctness of the voting 9 system is not so self-evident. 10 A winner is announced, and you wonder, is 11 that really the correct winner. It is not as 12 obvious as being at the airport in Austin, 13 Texas. 14 So I think a voting system has to meet a 15 higher calling. It has to be not only correct, 16 but demonstrably correct. You have to be able 17 to demonstrate to everyone, the voters and the 18 losers, that you got the right answer, in a way 19 that, merely pointing at the software and 20 saying, this software is certified and we think 21 the software in the machines is the same 22 software that is certified and there has been 23 no viruses, etcetera, probably does not suffice 24 for a lot of people. 25 So the difference between being correct TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 68 1 and demonstrably correct I think is part of the 2 point. It certainly is possible and maybe not 3 even hard to write D.R.E.'s that are correct. 4 I think we realized that early on in the work 5 at the TGDC. 6 The hard part with the voting system that 7 is all software is to make the results 8 demonstrably correct. How do you persuade 9 somebody that it is actually giving you the 10 right answers? And that is a near impossible 11 task. 12 >> KING: Thank you, Ron. 13 Juan, you next, and then back to Mike. 14 I was struck by something you said, Ron, 15 about voting systems having a higher calling, 16 and I think all of us in the room agree to 17 that, but I am not sure where the charter is 18 for that. 19 Because we talked earlier today about, in 20 risk assessment, that normally is something 21 that is done off of a collection of priorities 22 established by the organization, and I don't 23 disagree with what you said. 24 But as we start to look back for the 25 artifacts that prove that or attest to that, TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 69 1 that could be very helpful in establishing some 2 of our risk models. 3 Juan? 4 >> GILBERT: I think the gentleman -- 5 >> KING: The gentleman from Auburn is -- 6 >> SHAMOS: Yes, he wants the last word. 7 So as far as airplanes, yes, I agree, that 8 the analogy is certainly not perfect. 9 But I was struck by something you said, 10 which I liked, and I am going to throw it back 11 to you to confirm that you really said it, 12 which is that redundant systems on the airplane 13 are sufficient. 14 If that is true, then if we had redundant 15 software systems in a voting machine and the 16 two were really independent of one another, 17 they were different code bases, came from 18 different places, etcetera, it seems to me that 19 ought to be able to satisfy the requirement of 20 software independence; that is, this other 21 thing that we're talking about, that is going 22 to take the reliance on software A could be 23 software B. I don't think you're going to say 24 yes to that. 25 >> RIVEST: Can I answer that, please? TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 70 1 >> KING: If it is a short answer. 2 >> RIVEST: It is not a short answer. 3 >> KING: Answer, anyway. 4 >> RIVEST: So the -- that's a path which 5 the TGDC explored at length. I mean, you're 6 following down the footsteps of thinking that 7 the TGDC went when it was involving the VVSG. 8 It seemed very attractive at first, to 9 take that analogy of redundant systems in 10 looking at redundant software systems to try to 11 provide that confidence that you want. 12 In the end, there are no marketplace 13 examples of that, first of all. And in the 14 end, it seemed that that was actually an 15 illusion. You were chasing after a mirage most 16 likely. 17 It may turn out to be workable in the end, 18 but I don't think it is likely to happen real 19 soon for the following reason: Because 20 independence is an illusion, I think, when you 21 try to build such a system. 22 How do you build such a system? You have 23 vendor A producing part A. Vendor B producing 24 part B. The election officials need to choose 25 it. Well, they need to choose some system TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 71 1 integrator to put those parts together. The 2 system integrator is looking for the code of 3 part A and part B. The independence is lost at 4 the point you're putting it together. 5 And so I think the idea that you can 6 actually combine disparate systems in a way 7 that preserves what you want. The 8 independence, is extremely tough to do in 9 practice, because you would have middlemen 10 stepping in right away to provide lack of 11 independence, to provide a single point of 12 accountability for the election officials 13 should something go wrong, and then they will 14 be able to play with either system and change 15 the outcome. 16 Second of all, you have the issue of being 17 demonstrably correct as well, which you still 18 have two software systems which are 19 complicated, and then trying to argue that 20 they're both correct. 21 It is very common with inversion 22 programming to see programmers making the same 23 errors. It happens all the time when people do 24 studies. So it is not any way a guarantee that 25 you're getting the independence you want. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 72 1 And finally, to answer the question, where 2 does the charter come from for demonstrably 3 correct, I think the legitimacy of the 4 government, the elected officials, the voters 5 want to see depends on the confidence of the 6 election. That is where the charter comes 7 from, is being able to demonstrate in a 8 convincing way to the voters and to the losers 9 that the outcome is correct. 10 >> SHAMOS: Okay. Thank you. I had 11 yielded temporarily. I want to finish up my 12 comments now. 13 >> KING: Go ahead, Mike. 14 >> SHAMOS: I don't know if it would be a 15 surprise to anybody or not, but there is no 16 D.R.E. with VVPAT that exhibits software 17 independence as it's defined in the VVSG. 18 And if those systems were submitted to a 19 voting system testing laboratory today, and the 20 testing laboratory did its job, it would not 21 pass them. 22 And the reason for that is that all of the 23 VVPAT systems -- I am not talking Op Scan -- 24 all of the VVPAT systems, the VVPAT itself is 25 created by software. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 73 1 And in all of these systems, there is some 2 code or some unreadable device that the human 3 cannot interpret that tells whether or not his 4 ballot is valid. And so if he can't tell that 5 a valid ballot has been printed out, then it is 6 not independent of software. 7 Furthermore, the idea that there is -- 8 we're after this demonstrability, which I would 9 love to achieve, but how can it be that having 10 pieces of paper in boxes and rolls, which are 11 handled by many humans and counted erroneously 12 by any humans provide any kind of 13 demonstrability that the totals in the election 14 were correct. They haven't since the 1850's. 15 I can't imagine what has happened in the year 16 2007 to solve that problem suddenly. 17 I agree that, to a certain extent, Op Scan 18 ballots achieve software independence to a 19 certain degree, at least to the extent that the 20 ballot is not created by a computer and, 21 therefore, is independent of software. I'll 22 agree with that, but it still has all of the 23 other problems. 24 So what I think, if we're talking about 25 ghosts, chasing software independence as a TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 74 1 necessary aspect of a system for voting but not 2 a necessary aspect of a system for anything 3 else that we do, it sets an unnecessarily high 4 standard that rules out all kinds of wonderful 5 systems, including systems that have been fully 6 operational for 25 years, all of a sudden they 7 become unacceptable, because suddenly, we can't 8 demonstrate that they work. 9 I don't agree that we can't demonstrate 10 that they work. We may not be demonstrating it 11 to a certainty. But certainly, looking at 12 appropriate testing protocols for things, it 13 gets us to a sufficient level of certainty that 14 the system has been working. 15 It always has. It does in everything else 16 that we use in life. And so I don't understand 17 why we have to produce 100 percent proofs of 18 correctness for everything that is going on in 19 an election. 20 >> KING: Thank you, Mike. 21 Peter? 22 >> RYAN: I would like to pick up again on 23 this sort of avionics analogy, because first of 24 all, to add to Ron's comment, the avionics 25 failure is a manifest; whereas, there is no TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 75 1 God's eye view of what the outcome of an 2 election could be. We can't just do an 3 extrinsic check to see that the outcome is 4 correct. That is one fundamental difference. 5 The other difference is how we can 6 recover. 7 >> AUDIENCE MEMBER: Microphone. 8 >> RYAN: Sorry. Is the issue about how 9 you recover if you detect errors. If avionics 10 goes critically wrong, you don't recover, and 11 the plane crashes. 12 With the kind of software independence 13 systems that some of us are investigating with 14 voting systems, if you detect an error, you can 15 potentially recover from it if you've got a 16 suitable recovery mechanism. So I think that 17 is another fundamental difference. 18 And so another reason why you need triple 19 redundancy and so forth in avionics, is that 20 you're going to have to go on the possibility 21 that if things go wrong, you may not have a 22 good recovery mechanism. 23 And by definition, if we're going to have 24 proper avionics, the system is going to have to 25 be software dependent; whereas, I think we TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 76 1 have -- as I mentioned earlier -- proof of 2 concept demonstrations with voting systems, 3 that we can have software independent systems, 4 so there are all bunch of systems in which I 5 think the avionics analogy doesn't really hold 6 water. 7 Yes, I'll oppose that. 8 >> SHAMOS: Well, if I can respond for a 9 second. Give me 15 seconds. 10 >> KING: Go ahead. 11 >> SHAMOS: I actually agree that in 12 avionic systems, that failures manifest. Yes, 13 it is true. It is often if a plane crashes in 14 the place where we can see it, then we see the 15 wreckage. 16 But very often, the NTSB is completely 17 unable, after years of study, to figure out 18 exactly what it was that caused the plane to go 19 down, whether it was a software problem or 20 something else. 21 By the way, I am not arguing against 22 redundancy or cryptographic systems. I was a 23 big supporter of both here. I am with you on 24 all of these things. What I don't understand 25 is the requirement of software independence. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 77 1 If you can achieve it, it is a good thing, 2 but requiring it of all systems is what seems 3 to me to be unnecessarily restrictive. 4 >> KING: Thank you. Ron and then Peter 5 and then we will wrap up. 6 >> RIVEST: I was just puzzled by again 7 Mike's comment that D.R.E.s with VVPATs are not 8 software dependent. I am wondering if you can 9 sort of carefully delineate how an undetected 10 able software bug could cause an undetectable 11 change in the election outcome. 12 >> SHAMOS: It is easy. I will take a 13 continuous rule VVPAT. The continuous rule of 14 VVPAT. Every ballot image -- excuse me, cast 15 vote record. Every cast vote record, there is 16 a code printed with it so that an association 17 can be made between that ballot image and the 18 ballot image ostensibly stored in electronic 19 memory so they can be reconciled. So what I 20 do, if I have access to the software and I can 21 be an intruder, an undetected intruder, what I 22 do is printout a beautiful VVPAT for the voter 23 that has identically his choices on the VVPAT. 24 But the code that I print is an invalid code. 25 Then when he leaves the ballot, when he leaves TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 78 1 the booth, I then printout another VVPAT that 2 has the votes my software wants to it have and 3 yet prints a code that says this is a valid 4 ballot. I mark it as having been spoiled, for 5 example. 6 >> RIVEST: I would argue that that is not 7 a voter verified system. You have codes that 8 unreadable by the voter. 9 >> SHAMOS: They are no voter verified 10 systems because they all have unreadable codes 11 on them. 12 >> RIVEST: You can write valid and 13 invalid in English and then have the voter -- 14 >> SHAMOS: Yeah, you can do that, but 15 there are no systems in which everything on the 16 ballot that can be used to invalidate the 17 ballot is visible and readable to the human 18 being. 19 >> RIVEST: I agree -- I am in agreement 20 that system that does not have all of the 21 important information that could validate be 22 voter verifiable would not be software 23 independent, that's correct. So systems. 24 >> KING: Let the record show that we have 25 an agreement. Thank you. Let's see, I don't TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 79 1 know who was up first, Alec or John? Oh, I am 2 sorry. Sarah. 3 >> JOHNSON: I have an easy question for 4 Peter. Specifically you had mentioned that you 5 have got some systems, proof of concept systems 6 that you are looking at, and obviously a lot of 7 this discussion is centering with D.R.E. That 8 is what we are talking a lot about. There are 9 thousands used and continuing to be used as a 10 speak in elections. Do you have the system 11 that you are looking at in England or other 12 entities, do you have D.R.E.s that meet the 13 type of software, the two different softwares 14 in one system that Michael was talking about? 15 >> RYAN: I think the first response is 16 the kind of system I primarily look at is not 17 really a D.R.E. system in that sense. It is 18 cryptographic, but it doesn't involve the voter 19 interacting with the D.R.E. touch screen or 20 anything like that. So I don't know, does that 21 answer your question? 22 >> JOHNSON: The question was a general 23 question in general because we have got, do you 24 have a system that can do this? We have got, 25 no, you can't get it done. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 80 1 My question was in general about your 2 proof of concepts system that you have 3 mentioned. Without going into a lot of detail, 4 are they anything that we have seen today? Are 5 they Op Scan? Are they D.R.E.s or are they 6 something that is a hybrid that is something 7 totally different? 8 >> RYAN: They are not D.R.E.s as a said. 9 Yes, they are not. Keep it simple like that. 10 I guess best described it completely 11 differently. They are essentially 12 cryptographic. They do involve paper trail of 13 sorts in the sense that the voter does -- can 14 in principle take away, well, I think Ron calls 15 it protected ballot form which holds their vote 16 in encrypted form on the form. And they can 17 later check that that gets into a tabulation 18 process because a bit of mathematics going on 19 there. 20 So there is a sort of paper trail in some 21 sense but it is not, as it stands, a human 22 readable at least at the point that it becomes 23 the protected ballot. So human readable at the 24 time that the voter casts the vote in ways I 25 could explain later. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 81 1 So again we come back to this issue that 2 there seems to be some confusion as to whether 3 software independence is synonymous with, well, 4 VVPAT. Seems to be the implication. In my 5 mind that is clearly not true, in the sense 6 that I understand VVPAT. So yeah, but it is 7 still quite theoretical. We have prototypes of 8 it, and we have run a trial small scale student 9 election trial with it last year. We will run 10 more next year so the thing has been tested 11 out. Does that answer? 12 >> KING: Okay. As we get ready to wind 13 up this topic, I have got Alec, John, and the 14 Juan. And if you could hold your comments to 15 may be a minute. 16 >> YASINSAC: Less than that. I had 17 proposed initially that we talk about the 18 properties that this persistent ballot need to 19 have, the VVR, the IVVR. My contention is if 20 you defined those properties and you defined 21 them very explicitly in terms of what you 22 needed to have to be able have a verifiable 23 system, paper wouldn't meet it. 24 That is my contention, and I think that 25 has been borne out of why we are where we are. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 82 1 But since John is next, I will mention 2 this as well, that software independence leans 3 heavily on voter verification. Evidence is 4 pretty good that voters don't verify their 5 verifiable record even when they mark it 6 themselves. It is debatable about how careful 7 the average voter is in making their own 8 marking. 9 So in lieu of that information, if voters 10 don't verify the VVPAT and voters aren't across 11 the board careful in their marking of the 12 individual ballot then where does that leave 13 software independence? 14 >> KING: Thank you. John. 15 >> WACK: Just as a quick response, I 16 don't know the percentage of voters who vote on 17 VVPAT systems. I believe Op Scan in the last 18 general election was 49 percent. And so I 19 would just say there is a tendency to associate 20 VVPAT with software independence, and I would 21 sort of try to step back with from that. The 22 VVPAT systems that were implemented to begin 23 with I don't think were the best design and 24 focusing on those tends to cloud the issue a 25 little bit. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 83 1 The other thing I wanted to point out that 2 was something to Mike and Ron is that the TGDC 3 hotly debated whether there could be anything 4 on the paper record that essentially the voter 5 could not verify or might be a secret vote that 6 could be somehow or another changed in the 7 electronic memory so that it would invalidate 8 the paper record. VVSG 2005 it had a 9 requirement that said that the paper record 10 ought to have a code or shall have a code on it 11 that can link to its electronic counterpart. 12 Some states require that. 13 The current recommendations basically said 14 that has to be provided as an option. You have 15 to be able to turn it off because some states 16 regard that as a violation of privacy. There 17 was also a lot of debate about whether there 18 out to be any bar codes on paper record because 19 basically a voter can't verify a bar code. 20 And the decision was that all paper 21 records shall be produced in a way that they 22 can be Optically Scanned which does not require 23 a bar code. However, bar codes are allowed to 24 be used most likely for the purposes of adding 25 additional content to the paper record, for TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 84 1 example, if the paper record is going to be 2 scanned and read back to a blind voter, perhaps 3 the bar code could contain pronunciation answer 4 keys to candidates names and things of sort. 5 But anyway, the requirement for linking the 6 paper record to the electronic record was 7 changed from VVSG 2005 from a I think should be 8 provided as an option. 9 >> KING: Okay. Thank you and finally, 10 Juan. 11 >> GILBERT: My comments are to summarize 12 kind of these questions. I guess listening to 13 Michael and Ron, one of the things that I 14 clearly observe, and I like to bring this up 15 and bring Brian in on my summary, which is, Ron 16 said to his knowledge there is no software 17 independence systems that use software as a 18 back up or redundant observer. I think that 19 there are examples of that, you know, there is 20 one that we have worked on that uses video as a 21 verification mechanism. And a novel way. 22 Other people tried it but they didn't use 23 it a certain way. We hear Peter talks about 24 one way to do software independence as well. 25 As in the spirit of the VVSG and in the spirit TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 85 1 of the things we are discussing, we constantly 2 talk about if you have something, we need to 3 test it. We need the standard to test by. I 4 think way have this innovation class which is 5 outlined, but this is where Brian comes in. I 6 think we should have some things to test 7 innovation class itself. 8 There clearly are some things out there 9 that people have experimented with, that people 10 are trying, and I think these things will be 11 excellent candidates to give the innovation 12 class a trial. Looking at the definition of 13 software independence, and straight from the 14 VVSG, it clearly says that the voting systems 15 software is not capable. So that begs the 16 question is where is the boundary of the voting 17 system software? So if I have one piece of 18 software that in my mind, that is the voting 19 system that is doing the tallying, that is 20 presenting the ballot, doing those things. 21 If I have an observer, that is a piece of 22 software. Now the argument becomes is that 23 part of the voting system? Or is it an 24 independent observer? So these are all 25 questions I think that have to be answered, and TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 86 1 I think taking -- and this is just my 2 recommendation, taking a few of these 3 innovations that claim to be software 4 independent and trying the innovation class, 5 let's see if the innovation class will come out 6 by trying in, quote, unquote, innovations and 7 see if they meet these definitions. And so I 8 end there. 9 >> KING: Okay. Thank you. I think I am 10 going to exercise my privilege as moderator and 11 we are going to take a break a little earlier 12 than planned. I am starting to see fatigue or 13 low blood sugar set in. But let's -- 14 >> RYAN: Or jet lag. 15 >> KING: Or jet lag. Yes, Peter came a 16 long way. Let's take a 15-minute break now, 17 and when we come back, Alec, we will start with 18 question number three and your lead in to that 19 question. Thank you. 20 (Break) 21 >> KING: I think we are ready to 22 reconvene if the folks in the lobby could come 23 in and find a seat. We are missing two members 24 of the panel. Brian, did you see Sarah and 25 Chris? TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 87 1 >> HANCOCK: Yes, let me see if I can go 2 round them up real quick. Merle, also they are 3 working on cooling the room off a little bit 4 since it is warm. 5 >> Alec mentioned that he revised his 6 documents. I am just wondering if those 7 documents are out on the table, his revised 8 versions? 9 >> YASINSAC: They are early versions. 10 >> KING: Carrie, you raise a good point. 11 Maybe at the end of today, if anybody has 12 updates to any documents they have submitted, 13 they can identify that. Matt just informed me 14 the other panelists are on their way, but they 15 have given us permission to start without them. 16 We are on to the third question in the set 17 of questions that we were asked to address. I 18 believe that question is displayed on the 19 screen behind us. "Do methodologies exist to 20 test voting system software so it can be 21 reliably demonstrated to operate correctly? 22 And then what added security benefits are 23 created by S.I. that are not met by the testing 24 process?" And Alec has volunteered to open 25 with some guiding remarks on that. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 88 1 >> YASINSAC: I would actually like to 2 take this two directions. The first is to 3 address the first part of that directly, do 4 methodologies exist to test voting system? I 5 think the answer to that fairly much an 6 unequivocal no, there aren't ways to test the 7 system to verify that it is absolutely correct 8 and has proven that year after year after year. 9 On the other hand, software engineering 10 has matured to the level that we are able to 11 provide or the software engineering is able to 12 provide high quality software based on mature 13 software engineering processes, implementation 14 of best practices, keeping of data that allow 15 processes to be modified, to be able to repeat 16 success and not repeat failure. 17 And I personally believe that this is the 18 biggest omission, and it is not addressed at 19 all in the VVSG. It doesn't mention any 20 reference to the advances in software 21 engineering to be able to leverage process 22 maturity, be able to try and achieve the level 23 of quality that would be incorporated into the 24 voting systems that we need. And I will state 25 strongly that any software that is involved in TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 89 1 a electronic voting system needs to be high 2 assurance. It needs be engineered with rigor 3 and it needs to be a software that has a lot of 4 attention given to it to make it happen. 5 The second place that I would like to take 6 this is actually to address something that Ron 7 mentioned a few minutes ago about redundancy. 8 It is actually a little bit of comparing apples 9 and oranges of S.I. and testing processes. 10 Because software independence really is not a 11 testing approach. It is not a development 12 approach. 13 It really is an architecture and that 14 architecture that provides the ability to have 15 redundant mechanisms that can verify one 16 another. And in this case, it is the mechanism 17 that is the main parachute we call it, or I 18 call it, is the electronic system and the back 19 up parachute is the paper trail. And so that 20 structure is inherently better in some ways 21 than having an independent mechanism by itself 22 that you try to engineer to a very, very high 23 level of sophistication or very, very high 24 level of assurance. 25 So what software independence has done is TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 90 1 provide this structure that has a bit more 2 inherent security. Now, it is not without 3 problems, and that redundancy specifically -- 4 you have to think about the failure modes that 5 those redundant mechanisms may take. 6 For example, in the case of the paper 7 failure, in software independent systems, there 8 is no redundancy. If the paper system itself 9 is wrong at the beginning or is manipulated 10 before the election, so that we, for example, 11 reverse candidates on some ballots or if the 12 paper is manipulated during the voting process, 13 or if the paper is manipulated after the voting 14 process, there is no redundant mechanism, in 15 general, that can generally detect that 16 failure. 17 It is the redundancy of software 18 independence, is exclusively the other 19 direction. It will allow you to detect faults 20 or flaws that are on the electronic side, but 21 it doesn't really give you the redundancy that 22 you would like to have on the paper side. 23 So on the flip side, if you do have 24 electronic failure, then your failure mode 25 allows you to detect some problems that you may TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 91 1 have in the election, and as have been pointed 2 out, you may or you may not be able to trace 3 that failure back to its cause. 4 And as was pointed out, if you can't trace 5 it back to its cause, then in some sense, at 6 least, the election fails, because one of the 7 goals of the election system is not just that 8 it be accurate but that it be demonstrably 9 accurate. 10 So if you have a failure, that you can 11 detect that failure after the fact, and even -- 12 I would contend, that even if you can make a 13 strong argument about why the result was wrong, 14 that first result being wrong is reasonable 15 doubt to the candidate that lost and the folks 16 that supported the candidate that lost. 17 So what software independence -- one of 18 the things that it really does programmatically 19 inject into the election system is the notion 20 that that first count, if it is not perfect, 21 that is okay, because we can fix it with a 22 paper trail. 23 I think that is the wrong approach to take 24 for election systems. If we have a paper 25 trail -- and I am not against the paper trail. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 92 1 The point is, if we're going to use it, we need 2 to use it during the election period to have a 3 balance of mechanisms that -- that redundantly 4 verify one another. 5 So that when we produce that election 6 result on election night, it is verified, and 7 it's correct, and it takes overwhelming 8 evidence after the election is reported that 9 there was fraud or whatever it would be to 10 overturn that result, and that should be the 11 focus, and that doesn't appear to me to be the 12 focus of software independence. Yes, I believe 13 that -- yeah. 14 I did also mention that the part of the 15 redundant mechanism of software independence is 16 audits, and audits, as was mentioned, they 17 can't be required in the VVSG, although 18 responsible election officials will do them. 19 I am not certain that it is possible for 20 them to be conducted with enough rigger that 21 this redundant mechanism is going to be as 22 effective, as many people believe it will be 23 done. So it certainly raises questions about 24 the comprehensive nature of the redundant 25 mechanisms. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 93 1 And I, for one, believe that we should 2 have redundant mechanisms involved to be able 3 to verify these systems. I am just not sure 4 that software independence provides it and it 5 goes far enough from my perspective. So that 6 is my opening comments. 7 >> KING: Okay. Thank you, Alec. 8 John? 9 >> WACK: If I can respond quickly in 10 agreeing with many of the things you said. I 11 recalled that there was debate during the 12 development of the requirements for IVVR as to 13 whether they were perhaps too design specific, 14 did they -- did they basically mandate a system 15 that you're going to do the auditing post 16 election. You couldn't actually do it during 17 the day so that, by the end of the day, the 18 records could be -- the IVVR records could also 19 be consulted and you could end up with 20 verifiable results at the end of the day. 21 And this, as well as some other concepts, 22 were debated and I think the general consensus 23 of the TGDC was that we couldn't -- they 24 couldn't write requirements yet to that -- to 25 those sorts of approaches, and so that's, in a TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 94 1 sense, how the innovation class came about. 2 And so I -- I guess basically I am just 3 saying that was considered, but then we had to 4 get back to the real problem of writing a 5 standard that is specific that doesn't 6 constrain approaches too much, that -- you 7 know, that people can actually write tests. 8 And there was a constant tension, in a 9 sense, between what we would like, you know, 10 versus what we know how to do. Some of these 11 approaches were considered, but again, that is 12 where the innovation class came in at that 13 point. 14 >> KING: Alec, I had a question I wanted 15 to follow on, looking back at my notes. 16 You said that if paper was to be used -- 17 and I am paraphrasing, so correct me, that it 18 should be used during the election. Do you 19 recall your comment on that? 20 >> YASINSAC: Yes. 21 >> KING: If you could, amplify that a 22 little bit. 23 >> YASINSAC: Well, it is the notion of a 24 true parallel test. That notion being that, 25 while you're conducting the election, while the TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 95 1 election is ongoing, you select random machines 2 and you pull them offline. 3 And you use the paper trail -- you print 4 the paper or if it is a paper printing machine 5 or if it's a ballot marking device, you take it 6 offline and you run it in essentially a 7 realtime test of what the machine is doing. 8 And then you analyze the results of that test 9 of the paper against the electronic record that 10 is created. 11 And you can detect -- in many cases, 12 statistically, as Dr. Rivest says, you can 13 detect, if there is an inherent fault in the 14 software or potentially if there is malicious 15 fault by using parallel tests during the voting 16 day, as opposed to waiting until after the 17 election is over to conduct audits to test the 18 machines that have been used. 19 >> KING: Okay. Ron? 20 >> RIVEST: I just wanted to follow up 21 with a question to Alec. 22 I like your concern for high-quality 23 software engineering. I think that would be 24 wonderful to try to improve the current state 25 of voting systems by getting better quality TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 96 1 software engineering. 2 I am just wondering if you can say a 3 little more about how you do that and how do 4 you set up a certification system. When does a 5 system fail because it has got inadequate 6 software engineering standards somehow? 7 >> YASINSAC: The approach that I am 8 advocating here is not to fail the product. 9 What I am suggesting here is that we evaluate 10 the process, which, again, is the way it 11 appears to me that the industry is going, that 12 you have a maturity level of your processes 13 that is analyzed, via a standards body that 14 some that are already in existence, to be able 15 to determine how your organization produces 16 software. 17 If they exercise best practices, what the 18 history of their development is. And we'll 19 get, I guess, into OEVT later, but the OEVT can 20 be added in here to be a cross-check in the 21 VVSG process to validate and verify, I guess 22 you can say, the level -- the maturity level 23 that has been assigned to an organization. 24 And the OEVT will be able to give you a 25 good indication of whether that organization TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 97 1 has quality software processes underlying the 2 product that they produced. 3 It won't tell you exactly, but it gives 4 you a pretty good idea. Again, saying this 5 from having just done several of these things, 6 you can tell a whole lot about the product -- 7 the process by looking at the product. 8 If I might, while I have my microphone on, 9 come back. I heard the discussion about not 10 knowing how to write requirements for redundant 11 mechanisms. I know it is difficult and I know 12 it is not probably in the literature now on how 13 to do that. But I would contend that we also 14 just don't know very much about how to write 15 requirements for software independent systems, 16 as I have raised before. In the VVSG as it 17 stands now, there is not a mechanism in place 18 that I can see that would allow me to take a 19 system and be able run it against a processor, 20 an algorithm, or an analysis and determine if 21 it is -- if I present a system and say it is 22 VVPAT, how does it become verified by the ITA? 23 That it is actually software independent. 24 I think it is an objective process at this 25 point. My point is, if the notion is we don't TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 98 1 have a good way to write requirements for 2 independent mechanisms, that wasn't -- didn't 3 appear to be anything that stopped the 4 requirement to go to software independence. 5 >> KING: Thank you. Peter. 6 >> RYAN: Place the mic. Is that better? 7 So one important point which I think is maybe 8 implicit in some of what we have been saying 9 is, as you all gathered I am a fan of software 10 independence, in the sense we have explained. 11 That doesn't mean that trying to guarantee high 12 quality of the software that runs the election 13 isn't an issue. Clearly we must be able to 14 guarantee that the election will run smoothly 15 on the day and techniques for robustness are 16 still important. 17 We run it -- potentially there is a 18 paradox to the shear transparency of the system 19 that I have been trying to and software 20 independent systems may act against it because 21 the whole point is to try and detect any error 22 or corruption that occurs and be able to 23 correct it and so on. But of course the fact 24 that it is done in a transparent public way may 25 paradoxically undermine the trustworthiness of TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 99 1 the system. That is something we have to be 2 very careful about. 3 So I want to stress this point that whilst 4 I think as a design principle, the architecture 5 of these systems, software independence is 6 crucial, but that doesn't mean we ignore the 7 quality and robustness of the software. So I 8 think that is one important thing. I can jump 9 to your other point about how we might I guess 10 specify the property of software independence. 11 That actually strikes me that that is one of 12 the easier things to do here at least in 13 theory. And I think some of us could sit 14 around fairly quickly come up with a full 15 definition of what that means. 16 We compare to that some of the other tasks 17 and challenges we talked about earlier about 18 how do you set up a threat model and guarantee 19 its completeness and so on and so forth. There 20 are really major research problems and wide 21 open issues in the security committee for 22 decades and not ones we will solve in a hurry. 23 But that one I think is actually one that is 24 definitely doable. 25 >> KING: Okay. Thank you. John and then TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 100 1 Costis. 2 >> WACK: Just quickly, excuse me, 3 responding to Alec. But in general, one of the 4 problems with the VVSG is 600 pages and 5 sometimes difficult 106, even if you have an 6 active part in developing it. But I contend 7 that it does basically quantify what software 8 independence is and make it pretty clear 9 whether the system you are bringing for testing 10 is software independent or not. There is a 11 very important chapter in there called the 12 conformance clause and it is not a clause. It 13 is an actual big chapter. But it pretty well 14 defines the sorts of systems that can be 15 software independent as those that use 16 independent voter verifiable records. 17 And currently, those would be Op Scan or 18 VVPAT. It pretty wall lays that out. There is 19 a complicated, it looks like an Alexander 20 Caulder mobile of bubbles called the device -- 21 the class structure, the device class structure 22 and it codifies this and it gets back to 23 another point and that is that one of the 24 biggest improvements that the VVSG represents 25 is that it is made some advances on being TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 101 1 precise basically. 2 And one of big issues in its development 3 was making sure that test labs and vendors 4 agreed on what the system requirements are and 5 how it ought to be tested. So that forced 6 essentially writing to what people well 7 understood and to a certain extent, that is, 8 you know, that has probably caused some issues 9 in that, you know, writing more 10 performance-based requirements where things 11 would be more difficult to test and would 12 require more interpretation down the road. 13 Might have been desirable in terms of making 14 VVSG -- what am I saying here? Making it 15 easier for newer technologies to conform. But 16 it is the enemy of precision and that has been 17 one of the big problems in the past. 18 >> KING: Costis. 19 >> TOREGAS: I wanted to bring back one 20 notion that we touched on earlier in the 21 conversation. That is the human element in the 22 systems approach. And I have been reading and 23 rereading that sentence, methodology exist to 24 test voting system software, so we can reliably 25 demonstrate to operate correctly. In my own TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 102 1 experience in complex software deployments have 2 covered only about 20 percent or so of the 3 investments made and the anxiety and risk 4 profiles have to do with hardware and software. 5 A full 80 percent have to do with the 6 organizational aspects, the human dimensions. 7 And I know that it is kind of easy to 8 sweep under the rug and say we will get to that 9 later, but let's fix the machine part first. 10 But I constantly worry about being able to 11 verify the correct operation system software at 12 such a high level because all of us understand 13 and enjoy the discussion and the collegiality 14 that comes with software engineering and 15 evaluations and so on. 16 But ultimately in some precinct, some 17 polling station, somebody will be pushing a 18 button or not pushing a button or reading some 19 kind of an error message or not reading an 20 error message. And I guess what I would like 21 to raise is, is there a role for a prestigious 22 document like a VVSG to address that human 23 component or at least put some bounds around it 24 so we know if we spend so much effort trying to 25 get to the inth degree or to the tenth decimal TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 103 1 point of accuracy and precision on the software 2 reliability, if you will, that we somehow 3 forget that there is a huge component that has 4 to do with human dimensions that may perhaps 5 throw our concerns out the window altogether? 6 I don't have an answer for that. But I know I 7 would love to have some kind of an annex to the 8 VVSG that addresses the human components to 9 answer that question about correct operation. 10 >> KING: That is an interesting 11 observation because it certainly gets talked 12 about. But whether it gets reduced into the 13 document in a formal way, that is a very 14 interesting observation. Mike. 15 >> SHAMOS: So I will propose to answer 16 the questions that are on the slide. Do 17 methodologies exist to test voting system 18 software so it be reliably demonstrated to 19 operate correctly? Sure. Provided that you do 20 it during the election. Parallel testing 21 achieves that, properly deployed and 22 administered. Some more interesting question 23 is can you tell in advance that the voting 24 system software is going to operate correctly? 25 And everything depends on the meaning of the TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 104 1 word reliable. If reliably means can it be 2 absolutely demonstrated to be so? No. If 3 reliably means to a sufficient level of 4 assurance to be able to be used in an election? 5 I think the answer is yes, although others 6 differ with that. 7 What added security benefits are created 8 by S.I. that are not met by the testing 9 process? The answer is there may be profound 10 benefits, or may be no benefit at all. It 11 depends on what the alternative to the software 12 is. If the verification and auditing mechanism 13 is more reliable than the software could have 14 been, then you certainly have achieved a 15 benefit. If the alternative is less reliable 16 than the software was, then you have achieved 17 no benefit and in fact you have made things 18 worse by relying on something that was less 19 secure than the original. And as to the issue 20 of is it easy to determine whether a system is 21 software independent or not, I am going to 22 venture to guess that it is touring undecidable 23 to determine whether a system is software 24 independent. 25 There are certainly cases in which you can TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 105 1 demonstrate that there is software 2 independence. For example, in a case where no 3 piece of software ever touches a document 4 ballot, it is marked completely by human and 5 tabulated completely by humans, that is 6 software independent. On the other hand where 7 you have document ballots of the nature of a 8 VVPAT where there are markings on the ballot 9 that are created by a piece of software, there 10 are some things the software might create that 11 might make the system not software independent 12 and yet on the other hand there may be ways of 13 having software write something on a ballot in 14 such a way that it really is software 15 independent and I despair it being able to 16 prove that. In any case, I still come back to 17 the point that we have to look at what is the 18 other thing that we are relying on if it isn't 19 the software? 20 >> KING: Thank you, Mike. Sarah and then 21 Daniel. 22 >> HANCOCK: Actually it is me. Because I 23 don't have a name tag, I am using Sarah's. 24 Sorry about that. 25 I just wanted to tag onto what Costis TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 106 1 brought up about the management side of things 2 and VAC agrees with that 100 percent. And we 3 are developing a set of management guidelines 4 as a companion document to the VVSG and that 5 was certainly the intent, they would be used 6 together. We are currently in the second year 7 of a three year initial development phase of 8 these management guidelines and certainly even 9 after this initial development stage we will 10 continue to look at the best practices out 11 there to continue to increase and make this 12 document more usable for election officials to 13 get exactly to the goal that you stated. 14 >> KING: Good. Daniel. 15 >> CASTRO: I wanted to address a few 16 points. One in terms of the idea of how can 17 you reliably test if the software dependent 18 system operates reliably? It seems within the 19 VVSG there is already an assumption that there 20 is a way to do that. And that is because when 21 you talk about VVPATs one of the issues is how 22 do you create an accessible VVPAT? Within the 23 VVSG it is that you can use another software 24 dependent system which would read back the 25 VVPAT in audio version for the accessible user. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 107 1 Now, how do you know the best word to 2 correctly, obviously that is not software 3 independent and there really is no way to do 4 that in a software independent way. If you 5 want to say the voting is fair and uniform for 6 all voters, then you have to be making the 7 statement that, of course, that 8 software-dependent component of the system is, 9 in fact, as reliable as the rest of the voting 10 system, so it seems like within the VVSG, there 11 is already that hidden assumption that it is 12 possible to have a software dependent system 13 that is reliably tested. 14 I think there is another disconnect in the 15 VVSG, and that is the idea of having open-ended 16 vulnerability testing of integrity of the 17 software. 18 Even though you're requiring software 19 independence, there is many reasons to have 20 open-ended vulnerability testing. I think that 21 is a good idea overall. You want to look at 22 privacy and availability issues and other 23 issues that may arise. 24 If you're talking about testing for 25 integrity, it seems like, why are you doing TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 108 1 that if you have the software independence 2 within the system. 3 And, finally, I want to amplify a point I 4 think we brought up a few times, and that is, 5 as we talked about, many times voters are not 6 verifying the VVPAT. 7 So by the definition of software 8 independence, which is the voting system shall 9 be software independent, that is, an undetected 10 error of fault in the voting system software 11 should not be capable of causing undetectable 12 change in the election results. 13 So by that definition, if there is an 14 error or a fault in the voting system software, 15 and voters are not checking their VVPATs to 16 make sure it is correct, then that is violating 17 the idea of software independence. 18 >> KING: Thank you, Peter. 19 Ron? 20 >> RIVEST: I wanted to speak to these 21 questions of the definition of software 22 independence. You've raised some good 23 questions about the limits of the definition 24 and when it applies and what counts as software 25 independent. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 109 1 In part, the question is, what is the most 2 useful definition of software independence, and 3 you need to have a precise definition for 4 testability, of course, too. So with the two 5 issues that were raised -- one is the voters 6 not checking. 7 So if a voter does not check his printed 8 record that he's got if it is printed out, say, 9 or whatever, it remains as software 10 independent, because it is detectable. 11 "Detectable" meaning capable of being 12 detected. If the voter is not looking at it, 13 of course, then he is not detecting it, but it 14 is still detectable. 15 So it's just the fact that the voters 16 don't look at -- the output wouldn't violate 17 the definition of S.I. So you can't put a 18 standard in place that requires the voters to 19 do certain things and so on. It is only about 20 the equipment, so you just have to say it 21 supports the detections of those kinds of 22 errors. 23 So S.I. does not require that the voters 24 look at the ballots, and of course, it can't. 25 You can't pass a system or not depending on TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 110 1 what voters are going to do. 2 And with respect to voters with 3 disabilities, you get into a very 4 interesting -- if you have a blind voter, of 5 course a blind voter can't look at their 6 ballot. And so then the question is: Is this 7 going to be SI or not? 8 The intent is that a system should not be 9 able to tell if a voter who is voting is blind 10 or not. So if a voter is using an interface 11 that is designed for blind voters, but if a 12 poll worker goes and votes as if he were a 13 blind voter and sees the printout himself, then 14 the error is detectable, and that would qualify 15 as S.I. 16 If the voting system knows for sure 17 whether the voter is blind or not, then, in 18 fact, it may not be S.I. because the voting 19 system could change the votes of only the blind 20 voters. But if the voting system can't tell if 21 the voter is blind or not, then you can have a 22 system, which is S.I. 23 We're getting into the fine points of the 24 definitions here. They are interesting 25 questions. But the question basically: Is TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 111 1 there evidence produced by the system that 2 would allow an attempt to cheat, attempt to 3 defraud somebody, or disenfranchise somebody? 4 The example that Michael had: Are those 5 going to be detectable in principle, and 6 whether they're actually detected or not is a 7 separate question. 8 >> KING: Peter and Juan. 9 >> RYAN: First of all, I wanted to agree 10 with Mike that, yes, clearly, you have to look 11 at the rest of the system to see whether 12 dependency is thrown if you move it away from 13 the software. And people are doing that. 14 We are looking -- taking this sort of 15 system, a wider view of the system. So that is 16 something which is taken on board and people 17 are working on that. 18 One other comment I wanted to make about 19 this one about the assumption or having to 20 depend on voters performing various checks. 21 That is one thing that people recognize that 22 could be an issue, and so we have looked at 23 other alternatives, so you can supplement, for 24 example, voter checks by other kind of checks. 25 So one of the ideas that has been TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 112 1 suggested is to have sort of a verified 2 encrypted paper trail, a sort of additional 3 copies of these protected ballots that is kept, 4 say, locally, so you can have independent 5 auditing authorities also making checks at the 6 cross-points between that and, for example, 7 what is published on the web bulletin board 8 that goes into the tabulation, or having voter 9 assistance organizations, they -- who can vote, 10 for example, pass their ballot, their 11 receipt -- protected receipt to their local 12 representative who could do the check for them. 13 So there are ways you can supplement the 14 voter, the dependence on the voters by other 15 mechanisms if you're concerned about that. 16 So I think that is my point, yes, and 17 there are serious issues, but people are taking 18 those on board. 19 >> KING: Juan? 20 >> GILBERT: I think there is definitely 21 other ways to do this. One thing would be a 22 multimodal approach. So getting at the idea 23 that you have one system or one interface that 24 everyone votes on independent of ability, in 25 that case, what a sighted person does would be TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 113 1 similar or the same as one who is blind, if you 2 have an interface where you can accommodate 3 multiple modalities on the same system. 4 Now, the system can't treat one group any 5 different because you don't know who is going 6 to vote how. So a sighted person could vote as 7 a blind person. 8 And these kinds of approaches, I think, 9 that gets to something, I think Daniel was 10 bringing up, which is you actually do have a 11 sense of being able to test and get a 12 verification that is equal across for everyone. 13 That should be our goal, and we are not 14 here to talk about usability. I understand you 15 have a different panel to talk about that. 16 But I think it is kind of -- the two go 17 together from a usable security perspective. I 18 mean, the ultimate secure system is one that no 19 one can use. 20 I can secure it in such a way that no one 21 can use it, I can guarantee it is secure, but 22 you lose usability, and I can make it so freely 23 usable and accessible, that it has no security. 24 So I think, in a sense, those two go 25 hand-in-hand, and those two have to be TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 114 1 considered simultaneously. Otherwise, you're 2 going to shorthand one or the other. 3 >> KING: Thank you. 4 I have a question for Chris and Sarah, and 5 it is going back to the first one up there. 6 If you could consider from either a state 7 jurisdiction or a county jurisdiction 8 perspective that ties into a comment that Mike 9 made earlier about mitigation of risk occurring 10 at different levels. 11 So the question that I have is: If, in 12 the final version of the VVSG, whatever 13 software testing methodologies are implemented, 14 do you see -- among your peers or among your 15 own colleagues in your jurisdiction, do you see 16 the development of methodologies at the state 17 or at the local level to perform testing of 18 systems that may pick up any residual risks 19 that are in the systems once they're passed 20 through the federal certification? 21 >> THOMAS: Well, there has been 22 discussions of doing post-election auditing, of 23 course, as one way, and the issue of whether 24 the voter verifies their ballot or not, I mean, 25 really all of this plays out in a political TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 115 1 world. 2 And the political world generally knows 3 when an election -- when election results don't 4 conform to a norm that is expected, and that's 5 when I see the VVPATs coming into real play, is 6 at the recount. 7 That is where you're going to be comparing 8 what the machine says -- the software says 9 compared to what all of the little pieces of 10 paper said, as difficult and onerous as that 11 may be. So that is an element. 12 Now, state certification of systems after 13 the federal system is kind of all over the map. 14 There is not -- I don't see a lot coming out of 15 states that I am aware of. You see what 16 California has been up to, obviously. 17 But that is doing much more than once the 18 system passes, other than to see that it 19 conforms to the actual state statutes and that 20 it conforms whatever may be unique or not 21 tested at the federal level. 22 Obviously, there is the accuracy tests 23 that are done. There is pre- and post-accuracy 24 tests, and then there is auditing. And that's 25 sort of -- the auditing is the next frontier. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 116 1 Michigan has got one of these pew grants 2 to work with, Maryland and, I believe, Utah, to 3 come up with some documentation on 4 post-election auditing. But beyond that, I 5 don't see a lot coming out of the states. 6 >> KING: Thank you. Sarah? 7 >> JOHNSON: I agree with my counterpart. 8 I don't see a lot coming out of states, not 9 because there isn't a desire or recognition of 10 how important this issue is. Not at all for 11 that reason. The reason being in most state 12 agencies that do elections and state boards, in 13 my case that do certification, the expertise 14 isn't there. We don't have the expertise. We 15 don't necessarily have access to the scientists 16 and to all the expertise. 17 And I am sure Alec can talk about the 18 money -- the project that it took for Florida 19 to do their test. California spent over a 20 million dollars, etcetera, etcetera. You have 21 got the timing issue and the money issues that 22 all do factor into it. 23 I don't see a movement that states are 24 going to start doing more testing or 25 reinventing the wheel. I think there is TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 117 1 already a reliance on the federal testing 2 system. We have all learned a lot about what 3 that testing system was or isn't, so to speak, 4 but I just don't see the move because of the 5 expertise and the money. And we rely on our 6 state legislators who view or don't view this 7 as an important issue to do beyond what the 8 federal does. 9 >> KING: I have a follow-up question, 10 Ron, and then I will turn to you. John, in the 11 discussions at NIST, is what Sarah talked about 12 factored into the scope decision on the VVSG, 13 the fact that if many states are simply 14 accepting federal certification as the defacto 15 state certification that that places an 16 additional level of expectation on the federal 17 VVSG? 18 >> WACK: That -- I don't know how best to 19 answer that. 20 >> THOMAS: The answer is yes. 21 >> WACK: To a certain extent, I would say 22 that the big area where I heard discussions was 23 really more in additional testing that states 24 are doing on top of federal certification 25 testing. And that some states were actually TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 118 1 doing some fairly expensive expert security 2 reviews and additional banging away and in 3 finding some real problems, finding some fairly 4 significant problems and issues that would lead 5 them to wonder why the systems passed the 6 certification test to begin with. 7 And that this was expensive and if the 8 VVSG could essentially set up a system so that 9 the conformance testing to the VVSG would in 10 itself sufficiently rigorous, states may not 11 have to do additional testing. Or may not have 12 to do additional testing to the extent that 13 some states were doing. So that it might be 14 ultimately less expensive to have a more 15 rigorous VVSG and a more rigorous performance 16 testing process that, ultimately, you know, 17 those savings would be passed onto states. I 18 don't know if that really answers your 19 question. 20 >> KING: John, does rigorous there imply 21 depth, more breadth or both? 22 >> WACK: Well, I would say that both. 23 Breadth in terms of volume testing. Previous 24 versions of the standards allowed bypassing 25 certain parts of the system in testing. You TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 119 1 could hook a test harness up to the system and 2 bypass the user interface and not have to test 3 that to the same extent. 4 Some of the accuracy tests were specific 5 accuracy tests, and the new approach is really 6 to in essence, like I said earlier, conduct the 7 mock election and pretty much have live test 8 subjects banging away on systems for a period 9 of time, starting with the user interface and 10 ending up with making sure that you are getting 11 usable records out of the voting system. So 12 that is the breadth. 13 The depth, to a certain extent is handled 14 in the usability performance testing for 15 accuracy. And open-ended vulnerability 16 testing, I have sometimes wished that it had 17 been called an expert security review because I 18 think the open-ended part of it leads people to 19 believe that it is, in a very expensive 20 open-ended process that will never end. But it 21 is an expert security review that in essence 22 allows people to dive down into the system if 23 they need to find out if there are any issues. 24 So I guess my answer is, I think to the extent 25 that people could, they try to make it wide and TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 120 1 deep. 2 >> KING: Okay. Thank you. Ron. 3 >> RIVEST: I think John actually made 4 most of the points I am about to make. But I 5 just wanted to emphasize them a bit. I think 6 one of nice things about the new VVSG is the 7 volume testing that is going on. It really is 8 running a mock election as John said, and I 9 think that is a major help in trying to weed 10 out systems that are unreliable when you have 11 got frequent printer jams or other problems. I 12 think that just running a heavy duty mock 13 election with real equipment with real voters a 14 major improvement. 15 And I think it is modeled, back to the 16 question of tension between the state and 17 federal -- or not tensions but cooperation 18 between those two, it really is modeled after 19 the California volume testing. So the states 20 are leading the way in some of these things I 21 guess is where the federal government is, 22 adopted and picking it up and saying volume 23 testing is a good thing to be doing here. 24 >> TOREGAS: If can I ask a clarifying 25 question. Have the legal people opined about TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 121 1 liability, shared liability, if state or 2 municipal government, county government assumes 3 that a test that was done at the federal level 4 would ferret out all of these inconsistencies 5 and in fact was proven not to? What happens? 6 I am sure there is an easy answer to that but. 7 >> KING: I will defer that to counselor 8 Shamos. 9 >> SHAMOS: There is not an easy answer. 10 What happens is we had an example of this in 11 Pennsylvania where a system that had been 12 certified was suddenly decertified when certain 13 flaws were pointed out by citizens. A 14 re-examination was conducted and was 15 decertified. That was easy to do. Then the 16 counties that had purchased the system showed 17 up at the door of the secretary of the 18 commonwealth and said, we need money to buy 19 another system that you said was certifiable in 20 place of the one that you originally said was 21 certifiable. 22 And in that particular case the number of 23 counties in which the system was used was small 24 enough that the secretary was willing to pay 25 the freight. But if half the counties in the TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 122 1 state had used that system, we would still be 2 in litigation over it. 3 It is absolutely unclear whether there 4 is -- could be any liability on the secretary 5 of state for carrying out secretary of state's 6 duties. Could there be any liability on for 7 example the consultant to the secretary of 8 state who actually performed the examination? 9 I shudder to think. 10 >> KING: Alec. 11 >> YASINSAC: I just wanted to follow-up. 12 I think there is a bit of a misconception about 13 the OEVT and what has happened in the states. 14 To my knowledge, no state does OEVT testing as 15 part of their certification process. 16 The TTVR in California was all systems 17 that had been in use and the question was 18 decertification process, not the certification 19 process where the plan was laid out ahead of 20 time. I think this is a critical point because 21 in the future it was -- it is actually fairly 22 easy to find flaws when nobody knows you are 23 going to be looking for flaws. 24 Once this notion is applied and once if it 25 goes the way it would -- would be most TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 123 1 beneficial in my belief, then the process would 2 mature so that the OEVTs are guaranteed to find 3 things. I mean, it would be hard to put 4 together a team almost with the systems that 5 have been out there. They won't find faults in 6 the process, the TTVR, the Florida studies, 7 there have been things that have found flaws 8 because the systems weren't engineered to be 9 able to defend against that type of analysis. 10 It was done after the fact and so it was 11 very easy to find those faults. So the next 12 thing we have to look at is what happens when 13 you do an OEVT and don't find faults? Then 14 that is the big question there. But it is not 15 exactly an implementation on what the states 16 have done. It is kind of a reversal that says 17 we need to do it up front and hopefully that 18 will impact the process. 19 >> KING: Ron. 20 >> RIVEST: I was referring to the volume 21 testing California does, not the top to top 22 bottom review. You are absolutely right that 23 that was post facto. The volume testing I 24 believe in California is done before 25 certification in California. But I may be TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 124 1 mistaken on that. 2 >> YASINSAC: I am not sure. 3 >> TOREGAS: One more question. One more 4 quick question. Again, to signify my 5 ignorance. Where is the industry in all of 6 this in terms of demonstration of reliability 7 and so on? 8 I presume that there is a certain finite 9 number of companies that are involved in this 10 business, and I, again, presume that the VVSG 11 is ridden with an eye towards the industry 12 itself. Can somebody kind give me an idea of 13 where the industry is in terms of their current 14 ability to demonstrate correct operation and 15 their ability to operate correctly under the 16 proposed VVSG? 17 >> KING: Well, there are no vendors 18 present at the table here. 19 I don't know, Brian, is that something you 20 want to comment on in term of the vendors' 21 roles in the VVSG? 22 >> HANCOCK: Well, the only comment that I 23 will make is the structure of the TGDC was laid 24 out in the Help America Vote Act, as we all 25 know. And as TGDC is constructed, there is no TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 125 1 vendor involvement at the table, although 2 certainly in 2005, we received numerous 3 comments from the vendor community, and we 4 certainly expect to receive at least as many, 5 if not more comments this time around. 6 >> KING: Yes, I think cost is something 7 that you mentioned earlier, and we've had some 8 side conversations about, and that is: Who are 9 the stakeholders in this process? 10 And obviously the vendors are a 11 stakeholder. But then the follow-on question 12 is: What is their role? What is their 13 responsibilities? What is their 14 accountability? What is the state of the 15 practice? 16 Those are all related to that essential 17 question of who are the stakeholders. 18 >> TOREGUS: I guess the reason I ask it 19 is not because I am speaking for the vendor 20 community. I am not, because I don't belong to 21 the vendor community. 22 But if I had a 25-year-old son -- I have a 23 20-year-old son, so I am getting there -- if I 24 had a 25-year-old son. And he said, Dad, 25 should I go into this business? Is this a good TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 126 1 business for me? Is this an entrepreneurial 2 business to get involved in? 3 And having heard this discussion and 4 having read some of the materials, I would say, 5 boy, that is a real rough area to promote to my 6 25-year-old son. 7 If I am the secretary for economic 8 development in the state or at the federal 9 level, and I want to promote and to strengthen 10 industry, it's part of the harsh balancing act 11 between standard setting organization that 12 tries to develop kind of aggressive and correct 13 things, but at the same time, be able to find 14 the actual product in the marketplace. 15 And unless the states and localities of 16 the federal government is prepared to 17 manufacture these things, we also have to give 18 ear to that. I hear your discussion that, 19 historically speaking, they have not been given 20 a place on the table. 21 But ultimately, there has to be some kind 22 of an intervention, some kind of listening ear, 23 and I would suggest that would be useful again, 24 part of the expanding communication for this 25 discussion. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 127 1 >> KING: And well said. 2 John? 3 >> WACK: I had one other comment, and 4 that is, the question up there, "Do 5 methodologies exist to test voting system 6 software so it can be reliably demonstrated to 7 operate correctly," was -- that question was 8 pondered a lot. 9 Actually, one of the people working with 10 NIST in the TGDC went down the route of 11 essentially wanting to develop systems in such 12 a manner that they can be proved through some 13 sort of a fault analysis to be correct, and 14 they, you know, possibly may not require any 15 sort of independent record. This analysis 16 would be sufficient. 17 And the little I know about this area -- I 18 know it is pretty foolhardy to talk about 19 something you don't really know a whole lot 20 about in public. I think peter Ryan knows 21 probably more about what I have to say than I 22 certainly do. 23 But if you are going to basically develop 24 software that you can test to reliably 25 demonstrate to be correct, you have to develop TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 128 1 it in a way, in a very specific way, such that 2 it can actually be tested. So you probably 3 have to use very formal methods, which the DOD 4 and I am sure the airline manufacturers use, to 5 develop software in very specific ways in which 6 it can be more easily tested. 7 And ultimately -- you know, the E.A.C. 8 ultimately has to end up with a document that 9 testers can very clearly read, and it can't be 10 fuzzy testing. It has to be fairly specific 11 testing. It has to know what passes and what 12 doesn't. 13 So the reason I say that basically is we 14 could have gone down that approach with the 15 VVSG, possibly, and really upped the way in 16 which software was developed and specified it 17 and possibly could have made it easier and more 18 accurately tested. 19 However, that would have come at great 20 expense, quite a bit of an expense. It would 21 have taken much longer to develop the VVSG, and 22 it may not have been something that the vendors 23 would have wanted to accept, because they would 24 have had to change their software development 25 practices in a big way, and it would be much TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 129 1 more expensive for a vendor to get into the 2 business than currently. 3 So, basically I bring this up just to say 4 that, yes, I believe there are methodologies 5 out there to test software to reliably 6 demonstrate whether it is correct, but it means 7 that it has to be developed in specific ways to 8 make that possible, and that will be very 9 expensive. 10 >> KING: John, thank you. 11 I wanted to make a comment that ties 12 together John and Alec's observation about the 13 possible challenges of applying open-ended 14 testing to products that were not designed to 15 sustain that kind of scrutiny. 16 And John said we need to be looking at how 17 the VVSG and its implementation impacts vendors 18 coming into the market space. And I think for 19 those of us who rely upon these vendors, I 20 think there is an additional issue of having 21 those vendors stay in the market space. 22 And one of the concerns that is expressed 23 often by election officials is: What are the 24 contingency plans if vendors decide that they 25 are going to move their resources into other TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 130 1 markets, that they may be more competitive in? 2 So I do recognize that the folks that are 3 crafting the draft of the VVSG do think about 4 the vendors' role and often think about it in a 5 way of just trying to understand the vendors' 6 role, and sometimes the vendor doesn't fully 7 understand the vendors' role, it seems, so a 8 very complicated problem. 9 I would like to move on now, if we could, 10 to the next question on the list, question 11 number 4: "What are the relative merits of the 12 various types of direct, that is by the voter, 13 and indirect by automated system independent 14 verification techniques?" 15 And we didn't have a volunteer for there, 16 so I will make a few introductory comments, and 17 I'll turn it over to the other members of the 18 panel. 19 When I thought about the direct 20 verification, there is a lot of benefit to 21 having the voter directly involved in the 22 verification of the ballot. 23 One is that the verification techniques 24 can be very intuitive, may require a small 25 amount of training and/or easily understood by TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 131 1 the voters, that is, what they're verifying. 2 The second is that they can be realtime, 3 and in transaction processing systems, we often 4 talk about the inherent challenges of 5 pre-editing and post-editing. 6 And that is, when you can pre-edit data, 7 that is, get the right data, correct data into 8 the system, it optimizes the processing and 9 minimizes the need for exception handling. 10 On the other hand, if the edit is kind of 11 post-processing, that involves not only methods 12 of handling the air conditions of the 13 anomalies, but then you have to have a method 14 of verifying that the verification has, in 15 fact, occurred and any changes have been done 16 properly. 17 So some questions that I wrote down is: 18 Can correct verification approaches be enhanced 19 by better ballot design? And again, I think in 20 part that is a usability question and an 21 accessibility question. 22 But, I think, also, when we ask voters to 23 verify what they have done -- we have heard on 24 this panel and other discussions -- that voters 25 often do not verify. They don't consult the TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 132 1 VVPAT, so is that possibly -- instead of 2 sometimes it is assumed to be an issue of voter 3 disinterest, could that be issues related to 4 ballot design? Could it be issues related to 5 navigational schemes within the system, related 6 to voter education initiatives? 7 One of the things that we -- we often talk 8 about, is one of the holy grails in voter 9 verification, is the notion of the secret 10 ballot. 11 And my experience is that there are a lot 12 of different ways that we define the secret 13 ballot. Some of it is a technological 14 definition. Some of it is certainly a legal 15 statutory definition. 16 But do we have consensus on what we mean 17 by secrecy of the ballot? And even when we 18 talk about the ballot, verifying -- 19 verification of the voter's intent, 20 verification of the voter's choices, 21 verification of the voter's tabulated choices, 22 there is so many different dimensions to that 23 that again, a theme that I have heard here 24 today is that perhaps a lexicon in models would 25 be helpful in reaching consensus. The indirect TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 133 1 methods of voter verification obviously 2 introduce a different system, whether that 3 system is manual or automated or a hybrid of 4 that. And when we introduce an additional 5 system then we have all of the uncertainty that 6 a new system creates not only its inherent 7 functionality but its interfaces to the 8 existing system become source of risk. 9 And my final question that I have to help 10 the panel kind of form their responses is going 11 back to something that Mike said and I said 12 earlier and it deals with reasonable assurance. 13 When we are talking about verification, whether 14 direct or indirect, what is our target? Is it 15 our target that absolute zero uncertainty or 16 are we moving toward discussions of reasonable 17 assurance, reasonable test and what might those 18 be? So with those comments to kind of help us 19 form some questions, I will turn it over to the 20 panel. Mike. 21 >> SHAMOS: I will plead again for some -- 22 a definitional beginning of exactly what we 23 mean by verification. So I think what most 24 voters believe is going on with their 25 verification is not what is actually going on. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 134 1 There is a common phrase that is used but 2 voters want to be sure that their ballot was 3 counted as cast. That is the phrase. Counted 4 as cast. There is no system currently that 5 provides that kind of verification, except the 6 cryptographic ones. I am talking about the 7 commercially available systems do not provide 8 that. What they do provide is a verification 9 that the system correctly captured the voters 10 intent. Because if the system is able to 11 capture it and spit it back out to the voters 12 so the voter can check and see that all the 13 selections are the ones the voter wanted to 14 make, that indeed demonstrates to the voter 15 that she was heard correctly. 16 As to what happens downstream, the voter 17 does not have assurance of that. In fact, we 18 want to make sure that the vote is counted 19 correctly. We want to make sure that whatever 20 record is made of the vote is sufficiently 21 permanent, that it still exists not only at the 22 end of the election but at the time of any 23 audit or recount. And we want to make sure and 24 this is virtually never spoken by anybody, we 25 want to make sure that no unauthorized voters TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 135 1 voted in this election because it doesn't do me 2 any good if I know that my vote was counted if 3 ten people vote for every living person, then 4 it doesn't matter whether your vote was counted 5 correctly. 6 And further more, all of this stuff needs 7 to be able to verified by the voter after the 8 election is over. And typically, what the 9 VVPAT provides is the first form of 10 verification, which is the system heard me 11 correctly. So in answer question four, I am 12 interested in exactly what kind of verification 13 are we talking about? And my preference would 14 be for end-to-end verification. But there are 15 no commercial systems to provide that. 16 >> KING: If I can ask just the following 17 question. Mike, do you think there are legal 18 barriers to that end-to-end verification? 19 >> SHAMOS: No. 20 >> KING: Okay. 21 >> SHAMOS: I think what you are saying 22 is, if the voter can satisfy herself that her 23 vote was counted then it seems to stand to 24 reason that she ought to be able to prove that 25 to her neighbor that her vote was this and TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 136 1 counted that way. 2 But there are cryptographic systems that 3 permit solo verification without the ability to 4 prove to someone else how the voter voted. So 5 was that the concern? The secrecy? Also I 6 agree we need a definition of secrecy. Secrecy 7 seems to mean two things. To some people it 8 means that no one else can find out how the 9 voter voted without the voter's permission. 10 And to me, what secrecy means is not only can 11 no one else find out, but the voter cannot 12 demonstrate to another person how she voted. 13 And I think those are often intermixed, and we 14 ought to keep them straight, secrecy one and 15 secrecy two or something like that. 16 >> KING: Thank you. Ron and then Peter. 17 >> RIVEST: Thanks. I think Mike started 18 off on a good direction here. It is often 19 helpful to have clear definitions as to what is 20 being verified and what we are talking about 21 and I agree with everything he said on those 22 points. Just to try to clarify some of the 23 potential distinctions one could make, when I 24 talk about things that might be verified in 25 voting I tend to make a following three-way set TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 137 1 of steps that need to be verified which are 2 pretty much -- first is cast is intended. Is 3 the vote captured correct? Did the vote get 4 correctly captured. That can be direct or 5 indirect as we have talked about. So cast is 6 intended. 7 And the second step which is often skipped 8 as collected as cast. So usually there is a 9 process of collecting the records before they 10 are counted so are they properly collected? Is 11 there -- and some of the end-to-end systems 12 have a formalization of that process where in 13 fact all of the ballots that were cast 14 encrypted form to the website so you can 15 actually see the collection and you can verify 16 that a particular ballot is present in that 17 collection. So verifying that they are 18 collected as cast is a verifiable step in some 19 of these systems. 20 And then finally, you have the step 21 counted as collected. So you want to verify 22 that the tabulation is correct. That is 23 interesting if the ballots are encrypted of 24 course, but there are ways of getting around 25 that. So count cast as intended, collected as TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 138 1 cast, counted as collected I think is an 2 interesting and useful set of distinctions to 3 make. The issue about no dead voters is 4 probably the one of collected as cast. I guess 5 you have got things in the collection which 6 correspond to things that weren't cast and so 7 have to go there. I think the best way to do 8 that is to post a list of the voters who 9 allegedly voted and have other citizens be able 10 to check that and say, you know, this person 11 didn't vote because they are dead or something. 12 But that is a hard one. 13 >> KING: Okay. Thank you. Peter. 14 >> RYAN: Well, I agree very much with 15 what Mike said and what Ron just said. I want 16 to follow-up your comment about the secrecy. 17 That is a very good point and people have 18 recognized that and there are at least three 19 different flavors which correspond I think very 20 precisely with what you just said. Privacy in 21 the naive sense, a passive adversary, and there 22 is an even more sophisticated one called 23 cohesion resistance so that has been taken up 24 and people have come up with precise 25 definitions of these different TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 139 1 characterizations of security. So that is a 2 good point. 3 Yeah, I would just like to echo what Ron 4 said and the way I would like to describe some 5 of those cryptographic systems as they are in 6 the sense three stages in which the counting, 7 collecting, and counting of votes that can go 8 wrong. First in the encryption or encoding 9 step that things can go wrong and then sort of 10 transmission into the tabulation phase, and 11 then the actual tabulation and encryption 12 phase. So you have to look at all three of 13 those and make sure that there are ways to do 14 checks and balances to make sure that any 15 malfunctions at any of those stages can be 16 detected. So that is in essence what is going 17 on at a very high level. 18 >> KING: And Mike, if I am not mistaken, 19 there are jurisdictions where the vote of the 20 dead can be counted depending on when they cast 21 it in early voting. So you may need to expand 22 that explanation. I wanted to pose another 23 question to the panel that is related to this. 24 And I think it is a good question because it 25 doesn't have an obvious answer. And that is, TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 140 1 can mandates produce technology? And this has 2 come up on a couple of different discussions. 3 >> SHAMOS: Mandates plus money. 4 >> KING: And if it can produce 5 technologies, are they appropriate 6 technologies? And by that I am always looking 7 for the application of the law of unintended 8 consequence. So as we look at the VVSG and 9 perhaps other things that kind of swirl around 10 federal issues in election technology, be 11 interested in the panel's thoughts about the 12 efficacy of mandating technology and evolution 13 through statute or through incentives plus 14 statute and whether there are any potential 15 unintended consequence of that. And I can't 16 tell if these are up for this question or 17 for -- okay. Mike, go ahead. 18 >> SHAMOS: Unintended consequence. So 19 when VRE machines first came into use in the 20 early 1980s, almost all states that allowed 21 them had a requirement that they had to 22 maintain a paper record of vote, individual 23 cast vote records. And the vendors recognized 24 at that time that it was impossible to maintain 25 that record in sequential form where it -- or TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 141 1 would be possible to tie a particular cast vote 2 record back to a particular voter, so various 3 randomization techniques were used, and they 4 were used consistently in all D.R.E. machines 5 until after the year 2000, when the call came 6 for voter verified paper records and the 7 vendors saw that they had a sales opportunity, 8 they threw away absolutely everything they had 9 always understood about the need for 10 randomization and immediately came out with 11 inexpensive, sequential paper rolls, which they 12 knew they couldn't use, but nonetheless, it was 13 a solution to a mandate, it was a solution to a 14 problem, and look what has happened. And now, 15 they're all over the place. 16 And so the unintended consequence is that 17 some people have woken up and said, wait a 18 minute, this violates voter privacy, we can't 19 ever actually let anybody look at that 20 sequential paper trail, or it will violate 21 voter privacy. 22 So it is stored away in a safe, and nobody 23 ever gets to use it. They don't use it for 24 audits. They don't use it for anything. I 25 think that it is a highly unintended TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 142 1 consequence, but it is because the vendors were 2 forced to do something very quickly to respond 3 to a perceived market need. 4 And everybody threw away the cautions of 5 the past and allowed them to do it, and then 6 actually bought the systems and then actually 7 used the systems. 8 >> KING: All right. Thank you, Mike. 9 Any other discussion on this question? 10 Yes? 11 >> THOMAS: I just agree with Mike. I 12 think that as this political process moves down 13 the road in terms of defining what these 14 systems do is, watch out what you're going to 15 get, and I think also watch out for undermining 16 further any confidence that people are going to 17 have. 18 The more this goes down the road, the more 19 discussions that are out there and kind of all 20 of the boogie man stuff that goes on with it. 21 It is doing an incredible job of undermining 22 things. 23 >> KING: Costis? 24 >> TOREGUS: A reaction you provoked with 25 your question, and then something else. In TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 143 1 other sectors when we think about new 2 technology, there is usually an opportunity to 3 do what the social engineers would call 4 "business process re-engineering." 5 In other words, look at the process as it 6 existed before the new technology comes on 7 board, and then have a chance to change the 8 process because of what the technology enables 9 you to do. 10 It appears to me that in this particular 11 sector, election technology sector, we're 12 trying to kind of have our cake and eat it, 13 too. We think about innovation, maybe some 14 changes, but it is usually at the fringes. 15 I would be hard-pressed to remember a 16 paper I read or a product that I saw in the 17 marketplace that makes fundamental changes to 18 the process itself. 19 And of course, we could quickly say, well, 20 the process is sacrosanct. This isn't the law. 21 This isn't how we do it. And yet, in many 22 other areas of government, we have found ways 23 to modify the process and take advantage of the 24 existing technology. 25 For me, at least, I find the technology of TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 144 1 elections has not yet matured enough, perhaps, 2 to be allowed to enter this realm of thinking 3 of business process re-engineering, and perhaps 4 that is something to be considered in the fifth 5 question, we talk about the innovation class. 6 The other comment that I have is with 7 these kind of evaluations and verification and 8 so on. After a certain point, we have to also 9 begin to think about how the voter thinks. 10 And I have to be honest with you as a 11 voter myself, I would much rather go to an 12 election place, cast my vote, and have the 13 confidence and the expectation that the system 14 that receives my vote, the public 15 administration system behind that process is 16 strong enough, transparent enough, and 17 professional enough to take care of everything 18 else. It is a hope. 19 But in a sense, here we are, trying to 20 manufacture the technology component to do 21 something that perhaps we ought to be looking 22 at strengthening our public administration 23 system that could then run circles around what 24 a simple machine can or cannot do. That is a 25 bigger, a higher level of expectation that TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 145 1 perhaps some other group needs to be thinking 2 about. 3 >> KING: Yeah, I am going to take this 4 opportunity to plug the management guidelines 5 effort of the E.A.C., which I think that's a 6 fairly good description of what the goals of 7 that process is. 8 Dan? 9 >> CASTRO: I just wanted to add on to 10 what Mike said earlier about can mandates 11 produce technology, and, yes, but the funding 12 doesn't matter. 13 And I think it is interesting, because 14 with voting technology, the way we're setting 15 the standards here is very different than what 16 you see in a lot of other industries. 17 For example, in the auto industry, when 18 they're setting fuel efficiency standards, that 19 is set for cars that will be developed in the 20 future. It is not imposing any kind of 21 standards on the existing technology. I think 22 it is very important when you talk about 23 setting standards that don't have any -- any 24 funding tied to it. 25 I also think it is important, before you TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 146 1 set those standards -- maybe there is a need 2 for it -- but when you decide that there is a 3 need, you have to be deciding that based on a 4 clear risk analysis, which is what we 5 originally talked about. You have to say this 6 standard is this much measurably better than 7 what we had before to justify that kind of 8 cost. 9 >> KING: That is a good point. 10 When I was formulating the question on -- 11 you know, can you guide the evolution of 12 technology with mandates, I did think about the 13 fuel efficiency and the '73 Pinto and '74 Vega, 14 and I do have concerns about how effective it 15 is to mandate the development of technologies. 16 I think what I would like to do at this 17 point is, I am looking at the -- at the body 18 language of the panelists, and I think it is 19 time for another break. 20 And I would like this 15-minute break to 21 be a little shorter than the last 15-minute 22 break, because we do -- we still have another 23 question, and then we have some summarization 24 to go through, but I do think people need to 25 stand and stretch, particularly up here up TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 147 1 front. 2 So if we could, let's take a hard 15 3 and -- 4 >> AUDIENCE MEMBER: What time do we have 5 to be back? 6 >> KING: Good point. I have 4:25. Let's 7 be back at 4:40. 8 (Recess from 4:25 to 4:40 p.m.) 9 >> KING: If we can take our seats please. 10 Before we start in on the fifth and final 11 question, I wanted to kind of recap the agenda 12 and where we need to go and what we want to try 13 to finish up in the next hour or so. After we 14 have addressed question five, we have left 15 enough time in the schedule to come back and 16 really maybe tease apart some of the points we 17 looked at earlier that perhaps subsequent 18 discussions of questions illuminated something 19 that we want to go back through and review. 20 And then before we leave today, we have 21 got an opportunity for every panelist to make a 22 closing statement. And my intent there was 23 usually when I am giving a lecture, there is 24 two or three things that I want to make sure 25 that the students walk away from and often I TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 148 1 refer to those as just the take away. So there 2 will be an opportunity at the very end for each 3 of to us pass onto our colleagues on the panel, 4 pass along to the E.A.C. Standards Board and to 5 pass on to the E.A.C. commissioner the things 6 that we really would hope that will be 7 persistent from this meeting and merit further 8 attention and perhaps further discussion at 9 subsequent meetings. 10 So I am going to -- Peter, I am hoping 11 that your biological clock is revving up around 12 now. Mine is starting to -- well, that you are 13 on UK time. No. I am sorry, it is very late, 14 isn't it? But Peter has volunteered, and we do 15 have question five up there. He has 16 volunteered to lead us into opening discussion 17 of that question. So if you would. Oh, you 18 thought you had? Then I accept your 19 withdrawal. I am sorry. We had discussed this 20 earlier. So I will do my best to get things 21 started but jump in and help me. 22 The question is how can innovative systems 23 be evaluated for purposes of certification? 24 And some follow-up questions about how do other 25 industries deal with testing and certification TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 149 1 of innovative products? And this is I think 2 directed towards the concept of an innovation 3 class as an alternative way of introducing 4 systems into the VVSG certification process. 5 Do we create unintended back doors for the 6 certification process and again looking ahead 7 at potential unintended consequence? And then 8 can a set of limited standards be created in 9 order to make the path toward certification of 10 innovative systems more clear? 11 And I think in looking back at the 2005 12 VVSG, I had an opportunity to work with the 13 E.A.C. on the collecting and cataloging of 14 comments. And I can remember distinctly there 15 was a day about 45 days out from the end of 16 comments where I was asked, you know, is there 17 much activity? And the answer was no, it was 18 just some people are responding. And then as 19 people really thought through the implications 20 of what was in that document, it accelerated. 21 And then as a function of that acceleration, 22 other people saw the comments that were being 23 posted and that inspired additional lines of 24 thinking. 25 So I know that the authors and the TGDC TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 150 1 and the Standards Board are all concerned that 2 as this document goes forward, that due 3 diligence has been done on thinking through 4 unintended consequence. We have got always the 5 mandate that we have to live with these 6 documents for whatever their life is. And so I 7 think with that as an introduction about how is 8 innovation managed perhaps in other industries? 9 Are there models that can be looked at? And 10 does the innovation class proposal create a 11 potential back door to the certification 12 process? And then finally, is there potential 13 limited set of standards that could accommodate 14 the innovation class approach? 15 >> SHAMOS: I have now forfeited my right 16 to talk. 17 >> KING: I have been waiting all day for 18 that to happen. Recognize my -- go ahead. 19 >> SHAMOS: So I am not a fan of the 20 innovation class or the name innovation class. 21 So in fact I think it is a proof that possibly 22 too much of the VVSG is design oriented rather 23 than functional oriented. Because if it were 24 truly functionally oriented then anybody could 25 propose a system for certification, and it TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 151 1 would be examined to see if it performed the 2 necessary function. 3 If one of functions is software 4 independence, fine. If one of functions is 5 that it must provide verifiability at certain 6 levels in the voting process then fine. But I 7 think what is going on is the VVSG has anointed 8 VVPAT systems as the ones that satisfy the 9 requirements. And if you are not a VVPAT, then 10 it is your obligation to show up hat in hand 11 and prove to the examiners that your system 12 indeed satisfies these functions. 13 And I don't think there is any reason at 14 all to make that -- to make that distinction. 15 It is conceivable that somebody could come up 16 with a system that didn't fit into the VVSG at 17 all. Hypothetically, suppose I had a reaping 18 machine that was able -- I could point it at 19 your head and determine from that how you 20 wanted to vote. Such a system, frightening 21 though it might be, if it were submitted for 22 certification I don't believe it could be 23 tested to the 2007 VVSG which doesn't even 24 contemplate such a thing. 25 Okay. But yet, the systems that we are TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 152 1 talking about, for example, the prime three 2 system at Auburn, is perfectly consistent with 3 the kinds of voting systems that we have seen 4 before and doesn't need to be sent to the back 5 of bus in the innovation class. 6 As far as whether a certification process 7 for innovative systems is going to be a back 8 door, I think it is quite the other way around. 9 I think that the -- that what happens in these 10 so called innovative systems -- and I am not 11 sure what you mean by innovative. I think it 12 was an effort to sugar coat what was really 13 going on. But let's take it at face value and 14 if these innovative systems were truly 15 innovative, that is they incorporate new ideas 16 and new ways of achieving security, new ways of 17 achieving assistive voting, new ways of 18 achieving verifiability, then it certainly 19 isn't going be a back door that will allow 20 these fundamentally bogus and substandard 21 systems to somehow get into the elevated class 22 of the VVPAT. It will not happen that way at 23 all. 24 My belief is that these systems will end 25 up being better, and it is not a back door. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 153 1 And in fact, the VVPATs that have already taken 2 the back door. And so can a set of limited 3 standards be created in order to make the path 4 toward certification more clear? I don't think 5 you need a set of limited standards? I 6 wouldn't suggest that the standards ought to be 7 limited. 8 We have certain functional requirements 9 that we believe need to be met. A system that 10 meets them gets certified and a system that 11 doesn't, doesn't. That is my view of it. 12 >> KING: Thank you, Pete. 13 >> RYAN: I will have a try and make some 14 comments. The more I hear, the more I am glad 15 I didn't try to lead on this. Because I 16 realize there are all kinds of hidden agendas 17 and things which I didn't appreciate at the 18 outset. 19 Well, let me speak to the first question, 20 how do you evaluate these individual systems? 21 Strikes me that that's profoundly difficult 22 problem. I think it goes back to the 23 discussion we were having this morning about 24 how do you set up your threat models and so on 25 and so forth. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 154 1 So I think I would just suggest -- that 2 strikes me as a very difficult one. I don't 3 know if you are expecting a real answer. The 4 only thing that comes to my mind is that 5 actually is going to require a substantial 6 amount of research to come up with systematic 7 ways in developing threat models, which will -- 8 in a sense, it comes back to your question 9 earlier. 10 I think the threat models and so forth are 11 dependent on the systems. If you come up with 12 completely new kinds of technologies, you 13 introduce totally new classes of threat and so 14 forth. So I think this is a profoundly 15 difficult problem, will be my first comment. 16 Yes, I think I'll leave it for that. And 17 hand it over to someone else for a minute. 18 >> KING: I'll respond. I think that 19 is -- it is a challenge, as we're trying to, if 20 you will, kind of wrap our minds around the 21 risk assessment threat assessment models, for 22 lack of a better term, for conventional systems 23 moving through the VVSG process, what will be 24 the threshold of uniqueness in an innovation 25 system that will require the research of a TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 155 1 different set of models perhaps to validate it. 2 So from -- from my perspective, as I'm 3 looking at -- not just the current VVSG but the 4 past and future ones -- is trying to understand 5 the overhead that is embedded into the 6 application of the VVSG and how that plays out 7 in terms of time-to-market decisions, 8 cost-to-market decisions, and cost to the 9 jurisdiction. 10 And the one thing that we know is that 11 they're always unintended consequences, so 12 trying to think through concurrently, not only 13 the rigger of the proposed VVSG, but also the 14 implications of the innovation class model, I 15 think, has added a level of complexity, at 16 least in my own analysis. 17 John first, and then Costis. 18 >> WACK: I thought I would just talk a 19 little bit about what is in the VVSG, and as 20 best I understand it, why it turned out the way 21 it was. 22 At a certain point, in writing 23 requirements for new systems, there was a 24 decision made by members of the TGDC that we 25 couldn't write requirements that were specific TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 156 1 enough, good enough, for some emerging 2 technologies, such as cryptographic systems, so 3 there was a desire since these systems, in 4 essence, looked good, to somehow or the other 5 create a pathway to getting them certified, 6 even though there aren't currently requirements 7 in the VVSG. 8 So that's how the innovation class came 9 about. And then people struggled for a long 10 time about, okay, how do we still write 11 specific requirements for taking systems that 12 we don't have specific requirements for and get 13 them through the certification process? 14 And eventually people started to look at 15 it as a standards maintenance issue, which 16 means we've got a problem here, and that is, 17 how do we update the VVSG in a sense so that 18 it -- new requirements for new types of systems 19 can be added to it? And what can we write 20 currently that is specific, that labs, again, 21 can use? Because labs need very specific 22 requirements that they can test to. So the 23 innovation class was born in a sense. 24 And to a large degree, a lot of those 25 questions up there are really directed toward TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 157 1 how can the E.A.C. do this? Because the VVSG 2 doesn't report to do this. The VVSG has some 3 requirements in there that essentially say 4 innovative systems should meet all of the other 5 requirements in the VVSG. 6 But the E.A.C., in a sense, was tasked by 7 the TGDC -- or was recommended by the TGDC to 8 come up with a system for actually looking at 9 these new technologies and figuring out how to 10 get them certified and tested. 11 So, to a certain extent, I think it is -- 12 I hate to put it this way, but it is the 13 E.A.C.'s problem, at least that is the way the 14 TGDC kind of drafted the VVSG. And they do 15 need help. It is a very difficult problem. 16 I think we all felt that how could we 17 actually do this within the constraints of a 18 standard and felt that we could not, that it 19 had to be a separate process from the standard 20 that might feedback into it at some point. 21 >> KING: Thank you. 22 Costis. 23 >> TOREGUS: Two or three thoughts here. 24 First of all, I have to agree with my colleague 25 to the left, that if the VVSG was, indeed, TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 158 1 performance based, we wouldn't need a 2 subordinate limited standard group. 3 And, in fact, from the discussion that we 4 had last week at the IPI roundtable on this 5 very topic, the question was raised about a 6 600-page document. And in my mind, performance 7 standards, you don't need 600 pages to describe 8 performance standards. You can describe them 9 fairly adequately in a small number pages and 10 then challenge innovation to occur. 11 Some people would almost have you think 12 that innovation and certification are an 13 oxymoron put together, that innovation somehow 14 goes beyond traditional certification, and then 15 certification catches up to it, and you 16 certainly don't want to slow down that process 17 of innovation. 18 But given that you've got an innovation 19 class out there, I just wanted to put the other 20 side of the coin. 21 In my own state of Maryland, we have a 22 $1.6 billion budget deficit coming up. In my 23 own County of Montgomery, we have $400 million 24 of a budget deficit. 25 If somebody even understood there is talk TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 159 1 about changing the machines we use for voting 2 to get it one more time, I think people would 3 just freeze. They just wouldn't be able to 4 rationalize it. 5 So instead of an innovation class, maybe 6 we also ought to think about some kind of a, 7 let's say, transformation class. 8 Given the machines that we have today, 9 what is it that we can do to the systems or on 10 the machines, the software inside the machines, 11 so in fact, when they become more parallel to 12 what the VVSG would have us think. 13 I think the notion that we're designing 14 from scratch systems has to be balanced with 15 the notion of where are we budgetarily in the 16 states and in the counties. We can't just say 17 we imagine a new system to come into play, 18 especially since we just had a new system come 19 into play. 20 So in addition to the innovative systems, 21 I would like to put in a good word for how do 22 we transfer the existing systems we have to 23 become VVSG compliant so in some kind of 24 transformative systems, rather than only 25 innovation systems. It might be an idea that TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 160 1 we could have some fun talking about. 2 >> KING: Okay. Juan? 3 >> GILBERT: In the innovation class, it 4 talks about -- well, one of the things that I 5 think is important is when you submit for the 6 innovation class -- I guess no one has done 7 this, so I am assuming you have to submit 8 material stating why you're innovative. 9 What is important to me is who is going to 10 review that. It goes back to what I said 11 earlier, about this balance of usability and 12 security. 13 I think it is critical that whoever is 14 reviewing innovation, it has to be a diverse 15 board or a review committee. You can't have 16 all security people on that committee making 17 decisions because they like a certain 18 technology that is secure to them, and then no 19 one in the world can use it without a college 20 degree in mathematics. 21 So I think it is important that -- I like 22 the idea of this innovation class, and I think 23 certain things, if we're going to have an 24 innovation class and certify a system, we may, 25 going back to what he was talking about, is TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 161 1 maybe certify aspects or methods or components 2 of systems that could be integrated. 3 So if you're going to -- for example, a 4 multi-modal interface, there is clear 5 usability, performance benchmarks in the VVSG. 6 If you can meet those, then could you 7 potentially certify that as an innovation, 8 because it is clearly something that is 9 innovative that could be attached to existing 10 systems, or do you have to certify the whole 11 system? 12 And that -- I don't know the answer to 13 that, but I am throwing it out there, is that 14 do you provide component certification, or is 15 there a whole system? 16 In either case, I think it has to be 17 carefully decided on who that review committee 18 is composed of and their backgrounds such that 19 we get an optimal decision. 20 >> KING: Okay. Thank you. 21 Brian? 22 >> HANCOCK: Thanks, Merle. I want to 23 piggyback what Costis said and what Juan was 24 talking about. 25 You know, there are a lots of things we TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 162 1 don't know about what is going to happen with 2 the next iteration of the VVSG, but I think 3 there is a couple of things that we certainly 4 do know and I think probably all of us here can 5 agree on. Looking at the document as it is, I 6 think there are things that everyone needs to 7 be aware of, especially our policy makers, 8 legislators and people that are going to be 9 making decisions at that level. 10 First of all, whatever this comes out 11 looking like, the testing is going to be 12 exponentially more expensive than it is 13 currently. I don't think there is any way 14 around that, given the way it is right now. 15 Second of all, pretty obvious that no system 16 out there, and I would include Op Scan systems 17 could meet this document as it is currently 18 written. Those are two things that we know and 19 I think we need to throw out there for 20 everybody to think about. Okay. Daniel. 21 >> CASTRO: I just wanted to expound a 22 little bit on what Juan said. I think that is 23 a very important point because innovations have 24 historically been good for voting systems and 25 voting technology, but the idea is that TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 163 1 innovations can be small and large and can be 2 very incremental. But the cumulative effect 3 often that is what we are really after 4 sometimes. 5 It is not clear from the current 6 innovation class standard which I think the 7 idea of an innovation class is good in the 8 sense that it recognizing, that the VVSG is 9 recognizing the importance of innovation but 10 bad in the execution of it. 11 But I don't think it gives a clear path to 12 certification, especially for small 13 innovations. You may have a very small 14 innovation and a lot of people question, is it 15 worth going through a very complicated 16 innovation class certification process? But 17 that small innovation may still be very useful. 18 It may be a minor cost savings, but these many 19 minor cost savings would add up. 20 The second point I want to make is that -- 21 I think we talked about this a little bit -- is 22 that the innovation class is currently defined 23 as really a subcomponent of software 24 independence. That is the other method or 25 means by which a vendor can get software TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 164 1 independence. But again, when we talk about 2 functional standards versus design standards, 3 like I have said before, I think software 4 independence is really design standard. If you 5 want to have true innovation, you have to make 6 innovation class purely functional. It should 7 be outside of any kind of design standard. In 8 this case, it should be outside of the software 9 independence class. 10 >> KING: I have a comment that really 11 tries to tie together three different groups of 12 people who are indirectly mentioned in both the 13 conversation today but also in the VVSG. 14 I kind of put this under the heading of 15 the unintended consequence. And it is the 16 individuals involved in open-ended 17 vulnerability test, the individuals involved in 18 reviewing innovation class proposals, and the 19 individuals involved in risk assessment of 20 systems. And what is implied is that there 21 will be individuals who possess some skill and 22 some knowledge and perhaps some experience for 23 that. But I think what we have heard from Juan 24 and heard whoever made the comment on the OEVT 25 that the success of those components of the TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 165 1 VVSG will be in large part dependent upon the 2 abilities and skill sets of those people. And 3 I am not proposing that that be something that 4 we resolve here today. 5 But I pointed out as for most of us who 6 are involved in managing organizations, 7 staffing is the absolute hardest thing that we 8 do, identifying people who have the skill sets, 9 the time, and the availability. And so when I 10 read through the VVSG that is one of things 11 that struck me is who are these people? Where 12 do we find them? How do we train them? How do 13 we certify them? How do we supervise them? 14 How do we evaluate them? And again under that 15 heading of unintended consequence, that would 16 be instructive to better understand in terms of 17 assessing the doability of the innovation class 18 and also the open-ended vulnerability testing. 19 >> GILBERT: Yeah, we need a functional 20 spec for the selection of the individuals. 21 >> KING: Well, I bring it up because the 22 intersection of people who understand voting 23 systems and security, etcetera, it is still a 24 relatively small number of people. And I think 25 we need to pay attention to not only the TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 166 1 unfunded mandates but in this case almost the 2 human resource mandate that is imbedded into 3 the VVSG. John. 4 >> WACK: Well, I will not be working in 5 this forever and I do need a job after this 6 one. Probably the rest of us we could form a 7 corporation here. But I agree completely with 8 you. And at the same time though I do have to 9 mention that right now NIST has this national 10 voluntary laboratory system program and part of 11 that program really is about making sure that 12 labs, whatever they test, have individuals with 13 sufficient knowledge and experience and things 14 of that sort. 15 I could see that the criteria for voting 16 system labs will need to be updated somewhat. 17 There will probably have to be more work done 18 with that particular program to make sure that 19 the right level of experience does get in 20 there. 21 >> KING: Sorry. Ron. 22 >> RIVEST: A couple of things with regard 23 to innovation. I think one thing that hasn't 24 been said and maybe is just obvious is the 25 importance of supporting innovation in this TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 167 1 area. It is an area where we all realize that 2 we are still not where we want to be with 3 voting systems. I think academics are 4 realizing that the voting systems are an 5 interesting problem area to work in. 6 We are seeing more and more academics 7 trying to put their students to working, trying 8 to come up with better voting systems and 9 improve it. We are seeing innovation in part 10 because of the attention getting and also 11 because it is a hard problem. Voting is really 12 one of very toughest problems I have ever seen 13 in the security area. The requirements are 14 contradictory. People get challenged by that. 15 So we are seeing people come out of 16 academia or out of the wood work, whatever, and 17 working on this as you might not otherwise 18 expect. So innovation is happening in this 19 area. The question is how to integrate that 20 into the certification process. 21 Juan said something I would like to 22 support as well. Is it something on the TGDC 23 we were not able to do is which was to support 24 certification components of voting systems. 25 And I think that has the potential, it is a TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 168 1 hard issue how do that well, but trying to 2 support certification of the components could 3 open the door to a lot of interesting 4 innovation. If you have an Op Scan system 5 which was just the scanner part and couldn't 6 talk through a standardized protocol to a 7 tabulation system in itself was a separately 8 certified piece and interfaced standards that 9 were supported, I think it would open a door to 10 a lot more innovation. But I think we are a 11 ways from that kind of vision yet and making 12 component certification work. 13 I think when we are doing new things, and 14 I think some of the cryptographic systems are 15 very interesting new proposals, and how to 16 think about certification. I think what John 17 is saying is right, too. 18 We can have a more open process, a 19 somewhat higher standard kind of thing. You 20 can make it more public and might even require 21 source code to be public or do other things, 22 have a lot of public hearings on innovative 23 ideas, so that it is a different kind of 24 process in a more transparent way. And this 25 may help increase confidence. You're not TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 169 1 getting unintended consequences, as Merle is 2 concerned about. That's it. 3 >> KING: Juan? 4 >> GILBERT: I would just like to make a 5 comment about this, going back to something 6 Merle said a while ago in some meeting we had. 7 When you think about what we're doing, 8 we're talking about election science, and there 9 is really no discipline -- can anybody go and 10 major in election science? 11 I mean, how do you learn how do this kind 12 of stuff? So from my perspective, and Merle 13 knows I always go on the bandwagon about this, 14 we're talking about election science, and it is 15 not rocket science. It is harder. 16 At least rocket science is a defined 17 field. At least you can get training in it. 18 At least there is something there. 19 Here, what do we have? We have 20 disciplines that have traditionally been in 21 silos, you know, separate disciplines that are 22 coming together trying to do something that has 23 never been done before, and we really don't 24 have true training on how to do this. So we're 25 kind of making it up as we go along in one TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 170 1 sense, but I think we are kind of hard on 2 ourselves, too. 3 So I think the VVSG -- and maybe this 4 could have been a summary -- but I think the 5 VVSG is a step in the right direction, and 6 we're definitely critical of it. 7 But, again, keep in mind that there really 8 isn't a discipline or a book for dummies on how 9 to do this stuff, so... 10 >> KING: Juan makes a good point. I just 11 saw where NASA has postponed the shuttle launch 12 for a month because of a failed fuel cell 13 sensor. 14 And if we have bad fuel cell sensors on 15 our election equipment, we are still going to 16 have an election on election day, so maybe it 17 is a lot more difficult than rocket science. 18 Peter? 19 >> RYAN: Yes, a comment that Ron just 20 made that prompted in my mind, that moving to, 21 say, cryptographic systems, that potentially 22 changes the certification game, because it 23 almost then becomes a much more public process. 24 It's much more transparent. What you are 25 trying to evaluate there are the TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 171 1 crypto-algorithms, the protocols, and I guess, 2 to some extent, the procedures around them and 3 so forth, rather than trying to certify a lump 4 of code. 5 And that is, in a sense, precisely the 6 point of making the thing software independent, 7 is that it doesn't depend on certifying a lump 8 of code. So in some sense, you're changing the 9 certification game there, and potentially 10 making it perhaps easier and cheaper, perhaps. 11 Just to come back, you made some comment, 12 I think, that the innovation class is likely to 13 make things exponentially more expensive to 14 certify, or something to that effect. 15 >> Just the VVSG [indiscernible] 16 >> KING: Okay. Alec? 17 >> YASINSAC: I think one of the things we 18 haven't really addressed. Peter, you just 19 raised a perfect issue. And while it may be 20 more transparent in terms of everything is out 21 there in the open, I think what it would mean 22 is very, very far fewer people would be able to 23 have any real idea of what was going on. 24 I mean, if you look at the number of 25 programmers there are in this country that TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 172 1 really understand how to write C plus plus, 2 that is a small percentage of the population, 3 but if you look at the percentage of people who 4 could understand most of these algorithms, it 5 is extremely narrow and extremely small, which 6 is one of the large challenges of that 7 solution -- of those set of solutions, I should 8 say. 9 >> KING: Ron? 10 >> RIVEST: I would like to respond to 11 that issue. It is one that is raised often 12 with cryptographic systems. And cryptography 13 is the use of mathematics for integrity and 14 secrecy. It is a technology that has been 15 around for quite a while. It's taught at lots 16 of universities. 17 And in some ways, it's much simpler than 18 software. So if your choice is understanding a 19 few equations and understanding 10,000 pages of 20 code, you know, I think the choice is clear as 21 to which one is more digestible. In fact, the 22 code isn't even available to look at here. 23 So I think it is a bit of a false issue 24 here. When you think about how people develop 25 confidence in systems. Some understand it TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 173 1 themselves, and I think ones that are publicly 2 documented with technical articles that you can 3 read and digest that will be popularized for 4 this stuff, too, will help with that. 5 But not everybody understands all of the 6 technical details of any system. You certainly 7 don't understand how an airplane works except 8 in the grossest terms. Certainly people use 9 D.R.E.'s and D.R.E.'s with VVPATs don't 10 understand how the software works on that, 11 either. 12 So I think you have to be fair when you 13 are talking about what it takes to 14 understand -- there has to be some social 15 process, whereby we as a society get 16 comfortable with the systems we're using. 17 And that is going to involve experts at 18 all levels of applying their expertise, 19 communicating with people who are good at 20 explaining things, having everybody -- making 21 the system as public, open, and transparent as 22 possible and getting everybody a level of 23 comfort with it. 24 And I don't think that cryptographic 25 systems are any different in meeting that kind TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 174 1 of a process than our D.R.E.'s or something 2 like that. We have been through that process 3 and we have seen that they have some problems, 4 so getting digested in that way. 5 So I think we have a process in place for 6 taking complex technology and making it 7 something that people are comfortable with. 8 All of the people that have the expertise 9 have looked at this and they feel okay with it. 10 Many people say, okay, I'll go along with that. 11 It doesn't need to be the case that 12 everybody needs to understand every aspect. 13 Even a paper ballot system, not everybody 14 understands how a paper ballot system works. 15 They may not understand what a provisional 16 ballot is. They may not understand what the 17 counting procedure is like or how they're 18 transported to the county office. 19 There is lots of details that people just 20 assume get taken care of. You have to think 21 about a broader scale what it takes for 22 technology to be accepted in a social manner. 23 It is not easy. I am not saying that 24 anybody should take a cryptographic proposal 25 and accept it blindly, because there may be TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 175 1 mistakes there, too, or other problems to be 2 dealt with, but just to have a larger 3 perspective on what the process is, take 4 technology, and integrate it into our lives. 5 >> YASINSAC: And if I may. 6 >> KING: Go ahead. 7 >> YASINSAC: I agree that the general 8 population is probably not a ton of difference, 9 but the question -- the point was, the ability 10 to have the transparency and the openness. It 11 is like, you know, if the code were all written 12 in Russian, I couldn't tell anything about it. 13 If it is written in cryptography, that 14 just makes it a much smaller set of people that 15 are able to understand that technology and 16 analyze it in detail, in my view. 17 Of course, which brings up a point, that, 18 you know, when I was in the marines, it was 19 every marine was a rifleman. In school, every 20 student, I believe, should be a cryptographer, 21 so that's what I am pushing for. 22 >> KING: I am going to think about that. 23 I have a question for my colleagues in the 24 elections community. 25 The term "innovation" is pregnant with TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 176 1 meaning, and I think most election people have 2 had about all of the innovation they can 3 stomach. 4 In fact, the word "stability", 5 "predictability," "stasis," all have a more 6 positive connotation in the election community. 7 So from your perspective, could you 8 comment on the need for the VVSG to accommodate 9 innovation and how that factors into your 10 vision for your unit's use of technology in the 11 foreseeable future? 12 >> THOMAS: I think we would all like to 13 see a better mousetrap, okay. I don't think 14 there is any election official out there who 15 would say, what we've got right now is perfect. 16 But they will say, in many instances, what we 17 have right now, we know how to operate and make 18 it work. 19 And innovation recently hasn't worked so 20 well. And the -- the cases are obvious out 21 there in terms of problems that people have 22 had. In a broader political realm, not just 23 partisan politics, but election technology is 24 being rolled out in a very different 25 environment than it ever has in the past, and TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 177 1 much of which doesn't have a lot to do with the 2 Florida situation in the 2000 election. 3 So given all of that, I guess we would 4 prefer that you not call it innovation, that 5 you find some other phraseology, and I also go 6 back to the issue about components. 7 And I find this as an irritant to me, 8 because quite frankly, we're developing things 9 with our voter registration system that are 10 going to intersect with the voting system, and 11 it is going to gather candidates from across 12 the state and give local officials the ability 13 to roll that into the existing software. 14 Now, I cannot be dependent on some vendor, 15 perhaps against their financial interest, to 16 take that forward and infinitesimal, and this 17 system doesn't really give me an option. 18 It gives me an option to either not to 19 advance or to take what is put on the chopping 20 block and be happy with it, and we are going to 21 move ahead regardless. 22 And so I find that to be a shortcoming, 23 that there is not some type of a component 24 certification. And quite frankly, I view the 25 auto market as a component. It is an electric TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 178 1 pencil. 2 So, yes, we would like to see it move 3 forward. We would like to see innovations. I 4 don't know who the first one is that is going 5 to step up to buy this next system that is 6 cryptography; somebody will. 7 But I think what is happening now -- and 8 you're seeing it as jurisdictions move to 9 optical scan -- is that there is -- it is a 10 fallback. You're going back to something you 11 know how to operate. 12 The voter gets it. I mean they do 13 understand much like the lever machines, I pull 14 a lever down, a little wheel turns. I did have 15 one voter once say they know how I voted just 16 by turning those wheels backwards. But that 17 was only one out of several million. So I do 18 think there is a point at which this does get 19 beyond the comprehension of the voter in terms 20 of what is this thing that I am now being asked 21 to trust to take that leap of faith and there 22 is a lot of competing interest and competing 23 messages that talk about whether it is good or 24 bad or you should trust it or they can figure 25 out how you wrote it. And that wasn't present TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 179 1 many years ago. 2 That is new. It is there. That is 3 community that obviously is at the table that 4 everyone has to deal with. But I don't 5 think -- I am not saying people are satisfied 6 with the status quo. I don't think election 7 officials are. Clearly they are not. 8 But I will just leave this as my summary. 9 Where these new guidelines go is what is the 10 unintended consequence is that there will not 11 be a system that anyone can afford to buy, that 12 is going to be produced. You know, we bought 13 systems recently with federal money. We spent 14 $65 million for both of our components, and we 15 are told that these things are going to last 16 about seven years. And there is no $65 million 17 unless Congress is coming back to the trough 18 for us that is going to available in seven 19 years to go out and buy another one of these 20 systems. 21 So I do see a lot of things coming 22 together. So if the innovation is cheap, it 23 will have great merit as part of it. But if it 24 is going to be another one of these huge cost 25 increases to state and local government, you TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 180 1 are going to see a lot more Optical Scan out 2 there. With the, you know, the issues at hand, 3 we are Op Scan state wide. We can make it 4 work. I would never tell you it is perfect by 5 any means, but we can make it work. 6 >> KING: Sarah. 7 >> JOHNSON: Just to add a little humor to 8 the meeting. We are all so serious here. This 9 is my own personal opinion here. I tend to 10 feel like election officials are the tennis 11 ball in a tennis match. We are the ones being 12 batted back and forth across the net, back and 13 forth activist groups, you know, Congress, our 14 own state legislators and where we are the ball 15 being batted back and forth. And we just want 16 people to leave us alone. For a little while. 17 Five minutes. So that we can shake our head 18 and go, woo, okay. Or have some of Kentucky's 19 finest beverage. How about that? 20 Actually, we two agree and I agree, and I 21 think the Standards Board agree that we don't 22 like the term innovations especially as wide 23 open as it is. And we will be commenting on 24 that in a couple of days. 25 I agree with Chris, we don't specifically TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 181 1 like the voting machines that we have now. 2 There are things we would like to fix from a 3 procedural standpoint, the usability 4 standpoint, etcetera. But as Chris said, we 5 just spent $21 million buying new systems to 6 meet the federal requirements, and we don't 7 have that kind of money sitting around to 8 instantly replace these. So it is a concern 9 for all of us. And any time you are talking 10 about voting equipment, as some of the other 11 panelists have mentioned, it might be from a 12 technology side the best thing in the world. 13 It might be the best thing that can you come 14 up. 15 It might be brilliant and we might even 16 agree with you if we understood everything 17 about it, that it is brilliant. But we have to 18 then know how to use it ourselves, bring it 19 back home, teach our county clerks as I said 20 earlier, teach the precinct officers, mind you 21 who are still reeling from the fact of going to 22 these newer audio ballots and kind of more 23 computerized, the average election official is 24 in their 70s and they are still trying to 25 understand that knob thing that you have to TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 182 1 turn on those voting machines, as they call it. 2 Let alone the voter. So we have a lot of 3 things that factor into that. All of that 4 dependent upon Congress or the state 5 legislature for the funding for that issue. So 6 while we appreciate innovation and never want 7 to stifle it, if whatever it ends up being, it 8 has got to be a piece of a much, much larger 9 puzzle. 10 >> KING: Thank you. Peter. 11 >> RYAN: Yes, I just had a couple of 12 quick comments. One -- first one is I don't 13 think innovation technologies in this field 14 don't have to be expensive. I think I would 15 suggest the one that I have been working on can 16 run on quite conventional off-the-shelf kit so 17 in that sense it could be quite inexpensive. 18 I want to switch to another thing and 19 question really to you guys. The certification 20 process in the U.S., who pays for it at the 21 moment? Is it paid for by the vendors or? 22 Vendors. And that is likely to stay like that? 23 Because I saw -- I wonder where there might be 24 a merit in considering some sort of hybrid 25 thing where may be -- I don't know the vendors TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 183 1 perhaps pay for initial evaluation and 2 feasibility and to help the innovation may be 3 then, it could be subsidized a bit by federal 4 funds or something. That might be a thought. 5 Just one other thing I wanted to say, 6 certainly in the UK for evaluating secure 7 systems in general, not voting systems 8 specifically, there are methods for doing that, 9 so there are so called clefts, CSB licensed 10 evaluation facilities, you have probably heard 11 of them. So CSG is sort of an analog of NCIS I 12 guess it used to be called. Center for 13 computer security, or something. Well, they 14 proclaim themselves as the national authority 15 on information assurance systems. They license 16 companies like logic to provide these 17 evaluation facilities. So companies come to 18 them, and the person who certifies the 19 certifiers is CSG in the context. 20 I don't know if you have any analogous 21 model in the U.S. or if it is worth checking 22 into. And of course behind all this is the 23 framework for evaluation and I know some people 24 wondered to what extent the common criteria 25 will be applicable to voting technology. I am TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 184 1 not sure that question has ever really been 2 answered. My own suspicion is not entirely but 3 maybe there is some useful ideas that can be 4 carried across. So just a few comments. 5 >> KING: Thank you. Daniel. 6 >> CASTRO: Thanks. To respond to your 7 question about who pays for the certification. 8 I mean the vendors pay. I argue it is really 9 the tax payers who are ultimately paying. That 10 is the matter in the federal government can pay 11 for it, the vendor can pay for it and the 12 federal government can pay for it when they 13 give the money to the states who will buy the 14 voting systems. The other point I just wanted 15 to respond just because I work for the 16 Information Technology and Innovation 17 Foundation so I hate that innovation is going 18 to get batted around. 19 So the definitions of innovation that most 20 people use is new and better. So I think there 21 is a lot of time where we talk about something 22 that is new but not necessarily better. So I 23 argue that is not innovation. And in that 24 sense, let's not call that innovation. Let's 25 call it new technology which is more costly. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 185 1 >> RYAN: Cheaper too. 2 >> CASTRO: And more cheaper. 3 >> KING: I would like to make a comment 4 about Daniel's observation about who pays for 5 testing. From the vendors' point of view that 6 is all cost recovering and then the question 7 becomes from whom do they recover the cost? 8 And it can be from the innovators and so if 9 there is a small cadre of jurisdictions who 10 rally behind a modification, then they will 11 bear the cost of that testing over whatever 12 time arise in the vendor chooses to spread it. 13 So it is a very difficult decision I am 14 sure for the vendors to make about how far they 15 spread the cost and to whom. But I think you 16 are correct that ultimately it is the 17 jurisdictions and the taxpayers in those 18 jurisdictions who pay for the cost of the 19 testing. 20 >> KING: Costis. 21 >> TOREGAS: Again, drawing from other 22 sectors, the whole notion of the new technology 23 coming to government is very well-known and has 24 been studied extensively. And the key question 25 to ask is who drives it into the government? TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 186 1 In the election technology field, if I am not 2 mistaken, it is driven by legislative might, 3 there is a law that is passed that says this is 4 the way we will do it. And that has unintended 5 consequences for sure. 6 In order to kind of bring back some kind 7 of a more rational approach to it, I continue 8 to think that some kind of a cost benefit 9 analysis of either the standard or irrespective 10 of the standard, the whole process of 11 innovation in election technology would be 12 extremely helpful. I think the vast majority 13 of elected officials who end up having to vote 14 for the budgets, whether state legislators or 15 county commissioners, they have to vote to 16 appropriate money or use money that actually 17 goes back to the taxpayer argument. Have to 18 begin 106 the principles behind the innovation, 19 and you are absolutely right, innovation 20 shouldn't drive you towards better 21 circumstances. 22 But the question is, according to whose 23 standard? And most of the times, the 24 technology is driven by the user component. 25 This is an unusual one where the user is not TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 187 1 necessarily in the driver's seat. If you stop 2 and think about enterprise research planning 3 systems, you stop to think about CRM systems, 4 you stop to think about inventory systems, 5 those are traditionally driven by the user 6 community because they are the ones that reach 7 into their wallet, pay, and they know what they 8 want. In this particular circumstance, we have 9 a variance from that innovation transfer 10 mechanism so it might be useful to think about 11 how to handle innovation when it is being 12 driven not by the user, the end user. 13 >> KING: Peter, is your sign up? 14 >> RYAN: Sorry. 15 >> KING: Well, now my mic is back on. 16 Are there any more comments on this 17 question number five? 18 Okay. What I would like to do next is 19 kind of begin the wrap up of this -- of this 20 panel discussion. 21 And what we have talked about is that each 22 panelist would be given an opportunity to kind 23 of make a summarizing statement, if there is 24 things that you feel are important for your 25 colleagues on the panel or members of the TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 188 1 audience or the Standards Board folks to take 2 away from this. This is the opportunity to do 3 that. 4 As I am looking through my notes today, we 5 have covered a lot of ground. We have talked 6 about a lot of different things from the 7 perspective of the testing labs, from the 8 perspective of NIST, from the academic point of 9 view, from the election official point of view, 10 so I think we have done a good job in terms of 11 the scope of what we have covered. 12 What I would like to do -- and, Ron, since 13 we started with you today at the opening, I 14 would like to give you the last word in the 15 closing. I think that is fair. 16 So I would like to start in this direction 17 and kind of work around the table and, again, 18 think about what you heard today. Think about 19 the message that you brought about the things 20 that are important for continued discussion and 21 continued reflection. 22 And then I would like to make a few 23 closing comments, and then we're going to let 24 the E.A.C. close up the session today. 25 It has been a long day and I am -- I don't TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 189 1 think I can go much beyond 6 o'clock here. So, 2 Juan, if we can start with you, and then we'll 3 move around the table. 4 >> GILBERT: I am very brief. I think the 5 VVSG is good. I like the innovation class. I 6 think we need to test it. I think we need to 7 put some systems through the innovation class, 8 and let's see what happens. The innovation 9 class, in itself, is innovative. Let's 10 evaluate it. 11 I would like to reiterate my comment, that 12 election science is not rocket science. It is 13 harder. I also would like to say that you have 14 to have multiple disciplines at the table for 15 this because it is an interdisciplinary 16 approach to a very hard problem. 17 And we have got to be careful about who is 18 doing the evaluation. We need a diverse team 19 that brings different things to the table to 20 ensure that we cover all aspects necessary to 21 get the best system or process in place for our 22 citizens. And I end there. 23 >> KING: Thank you, Juan. 24 Mike? 25 >> SHAMOS: Yes. There is an elephant in TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 190 1 the room, he was introduced by Brian, but I am 2 going to flesh him out, and that has to do with 3 the cost of building a system that will comply 4 with the 2007 VVSG and the cost of testing a 5 system to determine whether it satisfies the 6 2007 VVSG. 7 In 1980 Pennsylvania passed its electronic 8 voting statute that laid out 17 mandatory 9 requirements that any voting system had to 10 satisfy. That list of 17 has not been amended 11 since then, and we're now 27 years later. 12 That set of 17 requirements was sufficient 13 for punch-card systems, Optical Scan systems, 14 D.R.E.'s without paper trails, D.R.E.'s with 15 paper trails, and some of the systems that, 16 today, we're classifying as innovative. 17 Those 17 requirements can be written in 18 two pages. I have them in front of me right 19 here. I am not even going to pretend that 20 they're adequate. 21 However, we now look at the 2007 VVSG at 22 600 pages. That's 300 times more requirements 23 than have been satisfactory for the state of 24 Pennsylvania, the sixth largest state in the 25 country, for 27 years. We haven't had a TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 191 1 serious challenge to the security of an 2 election in Pennsylvania during that time. 3 In those days, the vendors were able to 4 get a certification exam by paying $450. That 5 was too cheap. The price was set in 1929, and 6 wasn't modified since then. 7 I have no idea how much it is going to 8 cost a vendor to go through a 2007 VVSG 9 certification, and I think, at some point, 10 we're going to have to face that question 11 directly. 12 There is going to have to be a 13 subcommittee of somebody who figures out what 14 it is going to cost and who is going to pay for 15 it. That is, I think, the major unspoken thing 16 from today. 17 >> KING: Thank you, Mike. 18 Costis. 19 >> TOREGUS: I'll try to be equally brief. 20 I, too, was very nervous, when I heard 21 Bryan's two comments. They're shattering to 22 me. That means we need to take stock in the 23 VVSG and make some changes. We cannot continue 24 on the path given the two statements that were 25 made. Something has to happen differently. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 192 1 My contribution to the discussion has to 2 do with two different things. One is, I 3 believe that something again -- perhaps another 4 component of the E.A.C.'s handling is a strong 5 communication strategy, especially towards 6 elected officials, the both fronts of election 7 technologies, for simple things like risk 8 assessment and simple things you can't have 100 9 percent guarantee. 10 There are ways to begin to change the 11 mindset of people. You cannot change the 12 mindset of people without trying, and I think 13 all too often, we accept the fact that someone 14 in the state legislature or someone in the 15 county council will not understand that you 16 can't have 100 percent of something, and I 17 think a strong communication strategy can do 18 that. 19 The second point is going back to your 20 comment about vocabulary -- building vocabulary 21 that we can begin to communicate with. Beyond 22 the vocabulary, we need to have simultaneous 23 dialogues of the stakeholders. 24 It is not healthy to have stakeholders in 25 this room and stakeholders in the next room and TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 193 1 the next room. The mathematicians will tell us 2 if we optimize subparts and put them together, 3 that is a suboptimal solution. 4 In order to find the optimal solution, we 5 have to do it with a simultaneous discussion, 6 and perhaps that is beginning to happen, and I 7 look forward to hearing more and to participate 8 more in this. 9 And then the final concept is one of cost. 10 I think in today's world, cost will drive quite 11 a lot of things. I stopped believing in Santa 12 Claus. Perhaps Santa Claus comes from time to 13 time, and we're all too happy to have Santa 14 Clause come. But the reality is that we have 15 what we have. 16 So, therefore, incremental changes and 17 making sure that we can cross-connect and 18 understand from each other's best practices can 19 be very, very helpful, so I think the VVSG 20 could be thought about in two parts. 21 One is, somebody can rip off and apply to 22 existing systems and another part that will 23 give us our future vision 10, 20 years out. 24 Thank you. 25 >> KING: Thank you, Costis. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 194 1 >> John? 2 >> WACK: Well, it has been interesting. 3 I think the VVSG is -- the recommendations has 4 really touched off a lot of emotions in a lot 5 of different audiences. And to a certain 6 extent, that's good. To a certain extent, it 7 works against the fair evaluation of it. 8 I think that the VVSG needs to be 9 considered as, first of all, a beta version. 10 It's going to get two more rounds of review, 11 and, therefore, I think that it needs to be 12 read thoroughly and understood and comments 13 made specifically on that. 14 The other thing, I think is important, is 15 that it be considered as a tool. It's a tool 16 that was made by the TGDC. It is a hand plane. 17 It's a power saw, whatever. It is a tool 18 designed to do something. 19 And there are many aspects to it. There 20 is an awful lot of usability on it and stuff, a 21 lot of reliability stuff. There's a lot of 22 security stuff. It does need to be 23 considered -- at least I think the authors 24 intended it to be considered as a whole, if you 25 were to remove software independence, you might TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 195 1 need to then all of a sudden take a look at 2 what is in the reliability area, decide that 3 needs to be revamped quite a bit. It needs to 4 be looked upon as a standard for future voting 5 systems, not today's voting system. Today 6 voting systems work with VVSG 2005. It is not 7 intended to, you know, codify probably any 8 particular type of voting system today, 9 especially VVPAT. 10 In no way do I think the authors of VVSG 11 wanted to promote VVPAT. It is a big standard. 12 Don't let the 600 pages scare you. They -- the 13 number of pages could have been reduced 14 significantly but was made to be 600 largely so 15 it would be more understandable and readable. 16 Brian, I do have to take a little bit of 17 issue with your exponential increase in 18 testing. Exponentially increasing cost in 19 testing is a huge increase in testing and 20 expense, and I don't know that that really will 21 be true. Certainly it will be more expensive 22 but that isn't the VVSG's fault. I think 23 that -- I believe the number of states that 24 have gone towards using all paper is around 38 25 and there are more to follow. And people are TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 196 1 just more concerned about the quality of voting 2 systems today, and the increase in testing 3 would occur kind of regardless I think. 4 With that, I just would like to close and 5 say I think it is worth examining closely and 6 making sure that you just don't latch to 7 specific areas and say, you know, that is not 8 good enough or that is good. I think it needs 9 to be examined in its whole, not just the 10 security aspects that we mainly discussed 11 today. Thank you. 12 >> KING: John, thank you. Brian, I will 13 skip you and hold you for the last. Sarah. 14 >> JOHNSON: One thing I would like to say 15 thanks. I am representing the Standards Board 16 here and a lot of issues that are our board has 17 come up with potential draft resolutions have 18 been discussed today, so it is a little 19 gratifying from our end as election officials 20 although we are certainly not your equals in 21 the technical field. And likewise in our 22 election field, it is a two-way street. It is 23 really gratifying here that a lot of our issues 24 were echoed and certainly discussed here. So 25 there are a lot of things I will be taking back TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 197 1 to my group as we meet starting tomorrow to 2 formulate our formal resolution/comments on the 3 VVSG. And I do appreciate that on that behalf. 4 Just kind of in reference to some other 5 things, I mean, it is true the VVSG is a tool 6 and it does have a specific purpose. But we 7 all know that through life, tools get used for 8 thing that they weren't necessarily intended to 9 be used for. For example, in college, I found 10 that a wooden meat tenderizer, my little 11 utensil thing works great as a hammer. It 12 wasn't intended for that, but it works when you 13 need to hang a picture up. 14 So I think that that taken in the context 15 of where we are now, the political realities 16 that we as election officials deal with, that 17 the E.A.C. will deal with, does shine somewhat 18 of a different light on perhaps the VVSG. It 19 does say they are voluntary. Big V, Voluntary 20 Voting Systems Guidelines. 21 But in reality, once you take something 22 that is voluntary and it also does depend a 23 little on your state law, sometimes they don't 24 become voluntary. They may say they are 25 voluntary, but in reality, they are mandates or TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 198 1 standards that all state's at some point are 2 held to that. And the other issue I would 3 point out is it is true that this particular 4 VVSG draft or any iteration, whatever ends up, 5 is supposed to be for future voting systems. 6 But again that political reality will make it a 7 little more difficult to stick with that future 8 voting systems because once they are in effect, 9 they very well can be looked at and compared to 10 current voting systems. 11 So we do have to deal with elections under 12 current voting systems even though this 13 100-pound elephant of called the VVSG for this 14 new class, whatever you do call them, 15 technologywise and we all have to live with 16 that. That is something we are struggling with 17 as election officials and federal and state 18 politicians will often struggle with is the 19 concept that these are voluntary and that this 20 is for some future system that doesn't exist 21 when in reality we are dealing with something 22 that does exist. 23 >> KING: Thank you, Sarah. Chris. 24 >> THOMAS: I would just very briefly 25 agree with Sarah's comments. As a community, TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 199 1 we are very much involved or agree with what 2 you are bringing to the table. This is 3 expanding the horizons of this community pretty 4 extensively as over the past 30 years. What 5 has happened in the last few years has been a 6 great leap. And we are still trying to sort 7 all that out. 8 So we don't want to leave the impression 9 that we are trying to keep people out of our 10 business. We are not. We understand that we 11 have got to operate as an inclusive part of 12 government and of elections community and we 13 are willing to do that. But we are looking to 14 see where the dust is going to settle. 15 The fear of, like Brian saying, that even 16 Optical Scan wouldn't survive under these 17 standards is a little mind boggling. So you 18 know, we will look at this, and I appreciate 19 all your comments. I will definitely taking 20 many of them back to the Standards Board -- 21 Board of Advisors, excuse me, tomorrow. And 22 there are a number of folks that are in the 23 audience here that are also getting the benefit 24 of it that will be deliberating on these over 25 the next couple of days. So thank you. TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 200 1 >> KING: Thank you, Chris. Alec. 2 >> YASINSAC: Right. There are two quick 3 points that I would like to just reemphasize as 4 take aways. And I believe I said the first one 5 a couple of times. All software and electronic 6 voting systems ought to be high assurance and 7 engineered rigorously. I believe process 8 maturity is something that the vendors will 9 be -- would be willing to take on and could 10 take on at their own expense and make it part 11 of their business process. And then the OEVT 12 could accomplish the purpose of back door 13 verification of the process viability through 14 the assessment of the product. And 15 unfortunately, I don't think the VVSG leveraged 16 and picked up this -- what I think would be a 17 tremendous improvement. 18 The second take away I would like to leave 19 is that one of things I think software 20 independence pushes us further and further 21 toward is the auditing on the back end, the 22 first count not being as important. And there 23 is an old adage that says, we got time to do 24 it, do it over. But we don't have time to do 25 it right the first time. I think we can't do TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 201 1 that in voting systems. We have to get the 2 first count right and that is what it is all 3 about. So I hope you will take that away from 4 today. 5 Thank you very much. Thanks for allowing 6 me to be here. I am honored and pleased. 7 Thanks. 8 >> KING: Thank you, Alec. Daniel. 9 >> CASTRO: I just wanted to respond to or 10 highlight some of the things that other people 11 said. I would like to extend on the first 12 thing Ron said earlier, that the goal of the 13 VVSG is to get demonstratively correct voting 14 systems. I think we are talking about the 15 VVSG, I mean we have said this before, it is 16 about voting system guidelines but really what 17 we care about in the end is about the elections 18 we want demonstratively correct election 19 systems. Not just voting systems. 20 And I think we need to consider that when 21 we are working with standards, it is not enough 22 to say it is the voting system right but it is 23 the entire election right, just remembering 24 that. And when we talk about any aspect, not 25 just security, also cost when you are looking TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 202 1 at cost. It is not just about how much does 2 the voting system cost but how much does it 3 cost to operate and certify and whether the 4 costs are involved in the process is there. 5 And then I also want to mention two things 6 that Juan said. First of all, Juan had 7 mentioned that the innovation class is 8 innovative. Maybe so is software independence 9 but again I want to go back to that definition 10 of innovation. It is definitely new, not 11 necessarily better but that is something that I 12 think we have been talking about. 13 Another thing that Juan mentioned is that 14 usability goes hand in hand with security. I 15 think it is very important that we keep in mind 16 that there is many different values here: 17 Security, usability, reliable, accessibility, 18 cost, and the impact on innovation. And all of 19 these are values that the VVSG and the E.A.C. 20 when they are look at the VVSG should be 21 seeking to improve. I think we can all agree 22 that some future voting systems will be 23 significantly better than what we have today. 24 And the goal is how -- when we are looking at 25 VVSG, the VVSG should be asking how do we get TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 203 1 there and how do we get there sooner? Rather 2 than later. 3 >> KING: Okay. Thank you, Daniel. 4 Peter. 5 >> RYAN: Okay. Well, first of all, I 6 would like also to say thank you very much for 7 the invitation to be here. I am honored and 8 privileged, and I have really enjoyed it. I 9 hope I have contributed something to it. And I 10 am very pleased to see the way you are sort of 11 conducting this in an open transparent way. 12 There is that transparent word again. I hope 13 perhaps something similar will happen in the UK 14 in Europe. I am not aware of anything. I 15 think you are ahead of game in that sense. I 16 guess you have been ahead of the game in voting 17 technology for over a century now. 18 The UK generally speaking we are just 19 using old paper and stubby pencils. So am 20 pleased to witness this and to some extent 21 participate in it. 22 I would also point out that of course this 23 is not just a U.S. issue, and we are facing to 24 some extent, perhaps a lesser extent but to 25 some extent the same sort of issues over in TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 204 1 Europe and so on. So whether there is any 2 possible synergy there that can be exploited, I 3 don't know. But it is worth thinking about 4 maybe. 5 I would also like to echo Juan's comments 6 about and it is highly disciplinary and 7 sociotechnical and so I think people are now 8 beginning to appreciate the extent to which 9 that that is true, and we really do need to 10 look at it in the large -- and as you said, 11 look at the interests of the various 12 stakeholders, and speaking for myself, I would 13 really welcome the opportunity to be able to 14 talk to more stakeholders, election officials, 15 and politicians, as well as voters and so on 16 and so forth. 17 And I am also so very pleased to see this 18 innovation category class that you're 19 introducing, and whether it's framed quite 20 right now at the moment is difficult, but I 21 think it is great you're trying to be open to 22 innovation. 23 I certainly think, speaking of some 24 self-interest here, I guess, but I think there 25 are innovative technologies out there which TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 205 1 hold out great promise and that is really being 2 innovative and potentially better in many 3 senses for providing higher security, and 4 usability, reliability, and so forth. And 5 potentially, that's even at lower costs. 6 But of course, all of that has to be 7 weighed up, and there are still, I think, major 8 research issues there, which is why I guess I 9 am sort of slightly hesitant about -- there is 10 always this problem with new technology: At 11 what point do you standardize and what do you 12 standardize and how do you standardize and so 13 on? And you are obviously facing that issue. 14 I guess I was slightly confused by some of 15 the distinctions between sort of functional 16 design and performance standards. I am not 17 quite sure I quite understand the distinction 18 between these. I suspect actually that the 19 distinctions get blurred at times. 20 One of the things I have become 21 increasingly conscious of over the years, when 22 I was a young, naive security guy, I thought 23 that you started off, you put out your security 24 policies, high-level abstract policies, and 25 then you sort of trickled down through the TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 206 1 system, and you draw out your threats and so 2 on. 3 But when you speak to people in the field, 4 they say the policies are typically driven by 5 the threats. So in fact, it doesn't work in 6 this sort of top-down way. 7 And also, there is also a tendency to sort 8 of confuse properties with mechanisms, which is 9 another thing which happens a lot in security. 10 I suspect the same sort of thing may be 11 happening to some extent here. 12 So I think a great deal of thought has to 13 go into how to frame things at this high level 14 so you get the right degree of constraint but 15 not overdoing it. That is going to be very 16 tricky, and I think it's going to take time and 17 feedback and so on and so forth. So I think 18 that's pretty much what I wanted to say. 19 I just wanted to throw in, I think you 20 were saying it's nice have catch phrases you 21 can latch on to. One which I heard from Josh 22 Fenelobe (phonetic) who claims that it is not 23 original to him is that you should verify the 24 election, not the system, which I always rather 25 liked, in the sense that is the spirit of these TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 207 1 end-to-end systems. 2 But of course, also echoing the fact that 3 you want a fair degree of confidence that the 4 election is going to run smoothly and correctly 5 before you launch into it. But you don't want 6 to have to rely on the evaluation of code and 7 so forth to have the assurance of accuracy in 8 the end. 9 So, again, Ron sort of commented that it 10 has to be perceived to be accurate as well as 11 vital, so I like that catch phrase. I think 12 that's all I wanted to say. 13 >> KING: Thank you, Peter. 14 Ron? 15 >> RIVEST: Thank you. Thanks for giving 16 me the last word here. 17 I would like to echo, thanks to the 18 organizers for inviting me here. It is a 19 pleasure and a privilege to participate, and it 20 has been a great panel, actually. 21 I think this process is, actually -- the 22 E.A.C. and everybody involved in putting this 23 process together deserves to be commended. It 24 really is, I think serving the country well and 25 getting the right issues on the table and TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 208 1 getting debated. 2 And having the draft VVSG out for public 3 comment like this, I think, is exactly the 4 right thing and the comments that are being 5 debated here are just right on target. 6 These issues aren't simple, and they 7 deserve everybody putting their heads together 8 to figure out what is the right approach. 9 The VVSG, I just wanted to emphasize the 10 three things that are the highlights for me of 11 the new proposal. Software independence, of 12 course, is one, and I think that deserves a lot 13 of discussion. 14 I think it is the right way to go. If you 15 have concerns about it, I think the right 16 question -- I would encourage you to ask -- if 17 you would like to have systems that are outside 18 of the software independence sphere certified 19 somehow, how do you draft standards for them? 20 That is one of the things that we really 21 had the toughest part with trying to figure out 22 in a situation where you've got a voting system 23 where the count really depends on the software. 24 How can you, in an economic way, try to 25 get certification of such a software system out TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 209 1 there? And I think, really, we got stumped 2 with that. We couldn't do it, and I think 3 software independence seems to be the cheaper, 4 more trustworthy alternative, and trying to 5 certify systems that depend so heavily on 6 software seems to be just not something that 7 this industry can reasonably support. So S.I. 8 is one. 9 The OEVT is the second -- to me, a major 10 interesting component. I like John's proposal 11 for renaming of that, as expert security 12 review. I think that is maybe a better name 13 for that, maybe we can call it that. 14 And the innovation class, I like new and 15 better as well, a better mousetrap, or 16 variations of class, I don't know. Those are 17 the three aspects, I think, I'll highlight. 18 There have been a lot of issues brought up 19 here, which brought to mind interesting echos 20 of the whole debate within the TGDC, because 21 there are a lot of things that are explicitly 22 outside of what the scope of what the TGDC had 23 to consider. 24 The drafting of the VVSG, the proposed 25 VVSG is merely an equipment certification TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 210 1 standard. It is not more than that, and there 2 is so many important issues in voting systems 3 that are outside of that that I think would 4 have been wonderful if they could have been 5 addressed in something like the VVSG, but they 6 are just outside of this scope. 7 Let me list some of those, and I think 8 these are issues which I hope the E.A.C. will 9 carry forward in some manner or the other to 10 try to see that they get studied and addressed 11 and approved somehow. 12 The whole economic picture is one that has 13 been brought up. We have the funding of 14 testing and so on, is certainly is one that I 15 think would be nice to have worked up or a 16 review of how that's done. Are there 17 innovative ideas in just sort of the funding 18 structure there. 19 The whole idea of procedures -- election 20 procedures, that was outside of the scope of 21 the VVSG, and they certainly impinge on 22 security and so on. I know the E.A.C. has 23 already been active in that area, and that is 24 great. 25 The whole issue of standards for TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 211 1 interoperability and component certification is 2 an area which was outside of the scope of this 3 VVSG, and I think it deserves thought, and it 4 would be nice to see some movement forward on 5 that. 6 The issue of transition between current 7 existing systems and new systems is one that I 8 know the E.A.C. has been working hard at, and 9 election officials are concerned about, as 10 Sarah said, and I think that is also outside of 11 the VVSG, per se. 12 And the other issue that Alec raised 13 earlier about the whole software development 14 process is one that, again, is outside of what 15 the VVSG could handle and was scoped to handle, 16 chartered to handle. And I think it was 17 another gray area for work. 18 So there is lots of ways for improving 19 voting that is certainly outside of what the 20 VVSG can attempt to do. And I think as we look 21 at the VVSG and send in comments and I think it 22 is important to keep in mind what the scope of 23 the VVSG is supposed to be and what is outside 24 of the scope. 25 And for things that are outside of the TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 212 1 scope, don't drop those issues, but they are 2 probably going to be reflected in a revised 3 VVSG. 4 That's all. Thanks. 5 >> KING: Good. Thank you, Ron. I am 6 about to end on time, which is a phenomenon in 7 and of itself. The first thing that I want to 8 say is thank you to everybody on the panel 9 today. This has been an extraordinary 10 classroom. I think I have learned more today. 11 I heard some old voices more clearly, and I 12 heard some new voices who brought some new 13 things to the table and this was especially 14 satisfying for me. 15 I did write down many things but in 16 reflecting on how long the VVSG should be, 17 whether 600 pages is enough to explain 18 something that complicated or two pages, but I 19 like a three-line voting standard that says 20 cast has intended, collect as cast, and count 21 as collected. And I don't think I ever heard a 22 more succinct expression of the business that 23 we are about and I thank you for that, Ron. 24 In my closing remarks, I hope that all of 25 us keep in mind that the work of everybody on TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 213 1 this project, the NIST personnel, the TGDC, all 2 of the academicians who have made this now a 3 part of the research is all critical, but it 4 all funnels down to a relatively small number 5 of people, the E.A.C., itself and the Standards 6 Board who must become accountable for this 7 document. And I think we need to keep in mind 8 the stress that that places on those 9 individuals to look at something this 10 complicated and make sense and look at the 11 future of what they are doing. So we wish you 12 the very best in your meetings tomorrow and to 13 the E.A.C. commissioners, we wish you the best 14 as you work through this process. I would like 15 to ask everybody if they would to thank the 16 panelists for their work here today. 17 (Applause) 18 >> KING: Some people came very long 19 distances. Peter in particular. Many of us 20 dropped what we were doing at the end of our 21 semesters to get here. And so I do thank 22 everybody for that. And Brian, with that, I 23 will turn it over to you. 24 >> HANCOCK: Thank you, Merle. Let me 25 just reiterate the thanks that you just echoed TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 214 1 to the panelists on behalf of our E.A.C. 2 commissioners and at least three of the four 3 are here in the audience and our executive 4 director. I really do thank you for coming. 5 We appreciate your openness and candor, your 6 willingness to be here. You know, we need you. 7 We will continue to need you. And I hope that 8 we can feel free to call on all of you as we go 9 down this road to the final adoption of this 10 document. Thank you once again. 11 >> KING: With that, this meeting is 12 adjourned. Thank you. 13 Meeting adjourned at 6:00 p.m.) 14 15 16 17 18 19 20 21 22 23 24 25 TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210 215 1 STATE OF TEXAS COUNTY OF TRAVIS 2 REPORTERS' CERTIFICATE 3 UNITED STATES ELECTION ASSISTANCE COMMISSION STANDARDS BOARD MEETING 4 DECEMBER 11, 2007 5 We, the undersigned Certified Shorthand 6 Reporters in and for the State of Texas, 7 certify that the above and foregoing 8 proceedings were taken down, as stated in the 9 caption, and reduced to typewriting under ou 10 direction, and that the foregoing represent a 11 true, correct, and complete transcript of the 12 said proceedings. 13 SUBSCRIBED AND SWORN TO under our hand and 14 seal of office on this the 23rd day of 15 December, 2007. 16 ________________________________ April C. Balcombe, CSR, CRR, CCP 17 Texas CSR 5752 Expiration: 12/31/09 18 TEXAS CLOSED CAPTIONING 310 East 34th Street 19 Austin, Texas 78705 1-512-480-0210 20 21 _______________________________ 22 Michelle Robertson, CSR, RPR Texas CSR 4268 23 Expiration: 12/31/09 TEXAS CLOSED CAPTIONING 24 310 East 34th Street Austin, Texas 78705 25 1-512-480-0210 TEXAS CLOSED CAPTIONING 310 East 34th Street, Austin, Texas 78705 (512)480-0210