Search CSRC

***Disclaimer: Items in the following lists are provided for research purposes, and do not imply endorsement by NIST.*** U.S. Government Activities / Initiatives Related Standards / Best Practices C-SCRM Research / References Involved Standards Organizations / Associations   U.S. Government Activities / Initiatives Committee on National Security Systems Directive (CNSSD) 505 - "...provides the guidance for organizations that own, operate, or maintain [National Security Systems (NSS)] to address supply chain risk and implement and sustain SCRM capabilities".  Comprehensive National...

Comments Received in Response to: Federal Register Notice (June 28, 2013) Computer Security Incident Coordination (CSIC): Providing Timely Cyber Incident Response   Date (2013) Comment Received From Aug. 14 Carbon Black, (Michael Viscuso, CEO) Aug. 14 CERT Coordination Center at the Software Engineering Institute at Carnegie Mellon University (Ryan Meeuf, CERT Coordination Center, Carnegie Mellon Univ.) Aug. 14 C.I.G.N.E.T. (Vishwas Rudramurthy) Aug. 14 Internet Identity (IID) (Chris Richardson, Senior Manager, Federal...

April 27, 2010: NIST SP 800-22rev1a (dated April 2010), A Statistical Test Suite for the Validation of Random Number Generators and Pseudo Random Number Generators for Cryptographic Applications, that describes the test suite.   Download the NIST Statistical Test Suite. July 9, 2014: This update has a few minor corrections to the source code. The first change corrects the non-overlapping template test to make it correctly skip bits when a sequence matches.  The second change is to correct the π values in the overlapping template test. Software Revision History August 11, 2010:...

This information is provided for historical purposes.  Papers Statistical Testing of Random Number Generators; Proceedings of the 22nd National Information Systems Security Conference, October 1999. Presentations Empirical Statistical Testing of RNGs, 1999 RSA Data Security Conference, San Jose, CA, 1/99. Statistical Testing of RNGs, ANSI X9F1 Meeting, Institute for Defense Analyses, Alexandria, VA, 4/99. Statistical Testing of Random Number Generators, The 22nd National Information Systems Security Conference, Crystal City, VA, 10/99.

Research tools to support combinatorial testing. No license is required and there are no restrictions on distribution or use. All software is provided free of charge and will remain free in the future. NIST is an agency of the US Government, so this software is public domain. You are free to include it and redistribute it in commercial products if desired. ACTS is in Java, and will run on any platform with the latest Java release (Java free download is at java.com).  There are many users running current Windows, Mac OS, and Linux systems.  ACTS Basic version download:  Java .jar file. No...

A:  All or nearly all failures involve only 1 to 6 factors The key insight underlying combinatorial testing’s effectiveness resulted from a series of studies by NIST from 1999 to 2004. NIST research showed that most software bugs and failures are caused by one or two parameters, with progressively fewer by three or more. That is, they were only revealed when multiple conditions were true.  For example, a 2-way interaction fault could be "altitude = 0 AND volume < 2.2". So testing all 2-way combinations of parameter values could detect this problem. A method called "pairwise testing" has been...

Combinatorial testing is being applied successfully in nearly every industry, and is especially valuable for assurance of high-risk software with safety or security concerns.  Combinatorial testing is referred to as effectively exhaustive, or pseudo-exhaustive, because it can be as effective as fully exhaustive testing, while reducing test set size by 20X to more than 100X. Case studies below are from many types of applications, including aerospace, automotive, autonomous systems, cybersecurity, financial systems, video games, industrial controls, telecommunications, web applications, and...

Self-driving cars and autonomous systems of all types are notoriously difficult challenges for software assurance.  Both traditional testing and formal methods are even harder to apply for autonomous systems than in ordinary cases. The key problem is that these systems must be able to function correctly in a vast space of possible input conditions.  For example, autonomous vehicles must deal with lighting, rain, fog, pedestrians, animals, other vehicles, road markings, signs, etc.  Combinatorial methods are uniquely well suited to analysis and testing for this enormous input space, because by...

The tools distributed here are used extensively in testing for security vulnerabilities.   Survey article: Simos, D. E., Kuhn, R., Voyiatzis, A. G., & Kacker, R. (2016). Combinatorial Methods in Security Testing. IEEE Computer, 49(10), 80-83. Introduces CT-based approaches for security testing and presents our case studies and experiences so far. The success of the presented research program motivates further intensive research on the field of combinatorial security testing. In particular, security testing for the Internet of Things (IoT) is an area where these approaches may prove...

Autonomous systems are increasingly seen in safety-critical domains, such as self-driving vehicles and autonomous aircraft.  Unfortunately, methods developed for ultra-reliable software, such as avionics, depend on measures of structural coverage that do not apply to neural networks or other black-box functions often used in machine learning.   This problem is recognized and teams are seeking solutions in aviation and other fields. As one notes, "How do we determine that the data gathered to train an AI system is suitably representative of the real world?[1]"  This key question is currently...

The field of formal methods covers a broad range of mathematically-based techniques for specifying and verifying properties of software and systems.  Formal methods can be very effective for certain classes of problems, but they have gained a reputation for enormous expense.  One of the greatest opportunities for cost-effective use of these methods is the union of formal methods with testing. When a formal specification can be used in generating expected test results, the cost of developing the specification can be offset by a great reduction in the otherwise high cost of producing a test...

SEQUENCE COVERING The sequence covering array construct described below was introduced in: D.R. Kuhn, J.M. Higdon, J.F. Lawrence, R.N. Kacker and Y. Lei, "Combinatorial Methods for Event Sequence Testing",   First International Workshop on Combinatorial Testing, in Proceedings of the IEEE Fifth International Conference on Software, Testing, Verification and Validation (ICST 2012), Montreal, Quebec, Canada, April 17-21, 2012, pp. 601-609.   Preprint The notion of sequence covering has been extended to ordered combination coverage, with a theorem on necessary and sufficient conditions for...

Combinatorial methods improve security assurance in two ways: Reducing vulnerabilities - Multiple studies show that about two-thirds of security vulnerabilities result from ordinary coding errors that can be exploited (for example, lack of input validation).  By identifying errors more efficiently, combinatorial testing can reduce vulnerabilities as well.  Specialized security testing - We have been able to achieve huge improvements in fault detection for cryptographic software, hardware Trojan horse and malware, web server security, access control systems, and others.   Below are some...

One of the central problems in testing is determining what output should be expected for a given set of inputs.  This question is known as the "test oracle problem".  When a test oracle must be determined manually, testing becomes extremely expensive and time consuming.  Thus a component of any test process must be efficient solutions to the oracle problem. This section discusses a variety of approaches to solving the oracle problem and reducing test cost.  Automated Test Generation Oracle-free Testing  

Latest research: Combinatorial Frequency Differencing. NIST Cybersecurity Whitepaper.- Describes measures of the frequency of combination coverage and difference between Class and Non-class elements in machine learning classification problems.  Illustrates application of these methods for identifying weaknesses in physical unclonable function implementations.  Combinatorial Coverage Difference Measurement. NIST Cybersecurity Whitepaper.- Introduces a variety of measures that can be applied to understanding differences in combination coverage. Also see our User Manual for the coverage...

This research grew out of our 2001 paper on failures in medical device software, which found that the failures were triggered by only 1 to 4 variables interacting. Surprisingly, although "pairwise" testing had been popular for many years, no one had looked at the actual distribution of failures by number of interacting factors. We continued this work and published other papers finding that all, or nearly all, software failures involve interactions among a small number of variables, no more than 6, in thousands of failure reports. Below are some of  our research areas. If you'd like to find out...

Papers Covering Array Library Seminars & Talks & Tutorial Combinatorial Methods For Modeling & Simulation Workshop Papers DOs and DON'Ts of testing

Through virtual events and email list, the Forum provides our members: a venue to exchange information, share ideas and best practices, resources, and knowledge; an ongoing opportunity to leverage the work done in other organizations to reduce possible duplication of effort; and access to a community and network of cybersecurity and privacy professionals across the U.S. federal, state, and local government and at higher education organizations handling federal information.  Virtual Events Refer to the Events Page for upcoming Forum meetings and registration information.   Forum...

FY 2026  ISPAB  BOARD MEMBERS  Steven Lipner, Former Chairperson Executive Director SAFECode Term Expired May 2026 Edna Conway CEO & Founder EMC Advisors Term Expires January 2030 Anne Dames Distinguished Engineer International Business Machines (IBM) Term Expires November 2028 Michael Duffy Federal Chief Information Security Officer (CISO), Acting Office of Management and Budget With continuing association with the Cybersecurity and Infrastructure Security Agency (CISA) and the Department of Homeland Security (DHS) Term Expires January 2028 Bill English Chief Information Officer (CIO) /...

Upcoming ISPAB Meetings: No meetings are scheduled at this time.  Updates will be provided as they become available.   Stay in the Know! Sign up for new meeting notifications or reach out to the ISPAB team directly via email. ARCHIVE:  Meetings Held in 2026 January 21-22, 2026 (Hybrid) Federal Register Notice Announcing Meeting Agenda Meeting Minutes Additional Details on the January 2026 Event Page  Meetings Held in 2025 July 16-17, 2025 (Virtual) Federal Register Notice Announcing Meeting Agenda Meeting Minutes Link to July 2025 Event...

Topics of Interest In accordance with 15 U.S.C. 278g-4, the duties of the Information Security and Privacy Advisory Board are to identify emerging managerial, technical, administrative, and physical safeguard issues relative to information security and privacy. The focus of the Board's work for FY 2015-2016 includes the following areas: Quantum (physics, pre-shared keys, quantum key distribution, block chains) Cybersecurity Office of Management and Budget OMB Circular A-130 Revised Cyber-marathon CyberStats Measuring outcomes for cybersecurity Cybersecurity protections in...

ISPAB Charter: ISPAB Charter for 2026-2028. Annual Reports: ISPAB Annual Report for Fiscal Year 2025 ISPAB Annual Report for Fiscal Year 2024 ISPAB Annual Report for Fiscal Year 2023 ISPAB Annual Report for Fiscal Year 2022 ISPAB Annual Report for Fiscal Year 2021 ISPAB Annual Report for Fiscal Year 2020 ISPAB Annual Report for Fiscal Year 2019 Annual reports for 1995 - 2018 are found on the GSA web page at: Federal Advisory Committee Act (FACA) . When you reach the site, please select “The Annual Report of the President on Federal Advisory Committees – 1972-1998.”...

Sample Certificates and CRL from RFC 5280 certificate/CRL Corresponding section of RFC5280 RSA self-signed certificate C.1 RSA Self-Signed Certificate Section C.1 contains an annotated hex dump of a "self-signed" certificate issued by a CA whose distinguished name is cn=Example CA,dc=example,dc=com. The certificate contains an RSA public key, and is signed by the corresponding RSA private key. End Entity Certificate Using RSA C.2 End Entity Certificate Using RSA Section C.2 contains an annotated hex dump of an end...

Version 1.07 enabling tools for PKI client software developers This page contains conformance tests for relying parties that validate X.509 certification paths.  Each test consists of a set of X.509 certificates and CRLs.  The tests are fully described in the Conformance Testing of Relying Party Client Certificate Path Processing Logic document.  The goal for the first release of these tests was to address the X.509 features used in the DoD Class 3 PKI.  While this test suite remains available for use, it has been superseded by the Public Key Interoperability Test Suite (PKITS), which...

July 15, 2024 NIST Releases SP 800-73-5 and SP 800-78-5 including comment dispositions for SP 800-73-5 (Part 1, Part 2, and Part 3) and SP 800-78-5.   September 27, 2023 Personal Identity Verification (PIV) Interfaces, Cryptographic Algorithms, and Key Sizes: Drafts of SP 800-73-5 and SP 800-78-5 Available for Public Comment In January 2022, NIST revised Federal Information Processing Standard (FIPS) 201, which establishes standards for the use of Personal Identity Verification (PIV) Credentials – including the credentials on PIV Cards. NIST Special Publication (SP) 800-73-5: Parts 1–3 and...