“Exploring Alternative Methods and Goals for Federal Testing and Certification”
Panel Discussion

NIST/EAC Future of Voting Systems Symposium

February 27, 2013

Disclaimer

Challenges for Testing and Certification
In the future, we may not be able to:
•	separate the consequence of poorly performing election systems from well performing voting systems
•	distinguish between poorly performed election administration and poorly performing voting and election systems 

Implications for testing
Emerging election systems provide inputs to, and utilize outputs of the voting system
The performance of these systems impact the operational accuracy and reliability of the voting system – and vice versa
Our testing strategies lack symmetry – we may be over-testing voting systems and under-testing election systems and the competencies of election workers
Systems
•	A system is a collection of components (including subsystems) that transform inputs into outputs.
•	Systems utilize feedback loops to monitor system states and adjust performance
•	Systems maintain interfaces with other systems

System Components
•	Hardware
•	Software
•	Data
•	People
•	Procedures

Voting Systems
•	Vote Capture
•	Vote Tabulation

Election Systems
Systems used to collect, store, compute,  analyze, report, and disseminate data related to an election.  Includes voter registration, digital pollbooks, ballot delivery and retrieval, election night reporting, ballot marking systems, voting systems, social media systems, candidate qualifying systems, etc.
Emerging Systems
Emerging Systems
Emerging Systems
Emerging Systems
Mature Systems

System Convergence
Core Competencies of Election Officials 
and Election Workers
What is Certification/Testing?
•	A checkbox on RFP for system acquisition
•	Indemnification
•	Attestation that the required attributes of the system exist and function as intended
•	Revelation of the architecture, behaviors, capabilities and limitations of the system that informs its proper use.
Great Expectations
In the management of systems, we get what we inspect; not what we expect.
Many expected that after the implementation of the VVSG and a testing/certification program, we would get better elections.  
We got better voting systems.