Assured Autonomy and Explainable AI Papers
Our conference and journal papers on assured autonomy and explainable AI. We try to include links to the full papers, but for those not yet linked, please contact us for a copy: [email protected].
Papers
2025
Lanus, E., Lee, B., Chandrasekaran, J., Freeman, L. J., Raunak, M. S., Kacker, R. N., & Kuhn, D. R. (2025, March). Data frequency coverage impact on ai performance. In 2025 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 258-267). IEEE.
Wei, Q., Sikder, F., Feng, H., Lei, Y., Kacker, R., & Kuhn, R. (2025). SmartExecutor: Coverage-Driven Symbolic Execution Guided via State Prioritization and Function Selection. Distributed Ledger Technologies: Research and Practice, 4(1), 1-29.
Dahal, A., Shree, S., Lei, Y., Kacker, R. N., & Kuhn, D. R. (2025, March). Fairness Testing of Machine Learning Models using Combinatorial Testing in Latent Space. In 2025 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 268-277). IEEE.
Kuhn, D. R., Raunak, M. S., & Kacker, R. N. (2025). Measuring and Visualizing Dataset Coverage for Machine Learning. Computer, 58(4), 18-26.
Raunak, M. S., Kuhn, D. R., & Kacker, R. N. (2025). Ensuring Reliability Through Combinatorial Sequence Coverage. IEEE Reliability Magazine, 2(2), 49-57.
Olsen, M., Raunak, M. S., Kuhn, D. R., Van Lierop, H., Badorf, F., & Durso, F. (2025, March). A Combinatorial Approach to Reduce Machine Learning Dataset Size. In 2025 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 248-257). IEEE.
2024
Khadka, K., Shree, S., Lei, Y., Kacker, R. N., & Kuhn, D. R. (2024, May). Assessing the degree of feature interactions that determine a model prediction. In 2024 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 180-187). IEEE.
Raunak, M. S., Kuhn, D. R., Kacker, R. N., & Lei, J. Y. (2024). Combinatorial testing for building reliable systems. IEEE Reliability Magazine, 1(1), 15-19.
Shree, S., Khadka, K., Lei, Y., Kacker, R. N., & Kuhn, D. R. (2024, October). Constructing Surrogate Models in Machine Learning Using Combinatorial Testing and Active Learning. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (pp. 1645-1654).
Chandrasekaran, J., Lanus, E., Cody, T., Freeman, L. J., Kacker, R. N., Raunak, M. S., & Kuhn, D. R. (2024). Leveraging combinatorial coverage in the machine learning product lifecycle. Computer, 57(07), 16-26.
Raunak, M. S., Kuhn, D. R., Kacker, R. N., & Lei, Y. (2024). Ensuring reliability through combinatorial coverage measures. IEEE Reliability Magazine, 1(2), 20-26.
Shree, S., Lei, Y., Kacker, R. N., & Kuhn, D. R. (2024, July). Proxima: A Proxy Model-Based Approach to Influence Analysis. In 2024 IEEE International Conference on Artificial Intelligence Testing (AITest) (pp. 64-72). IEEE.
Kuhn, D. R., Raunak, M. S., Kacker, R. N., Chandrasekaran, J., Lanus, E., Cody, T., & Freeman, L. (2024). Assured Autonomy Through Combinatorial Methods. Computer, 57(5), 86-90.
Kuhn, D. R. (2024). Challenges of Assured Autonomy. IEEE Transactions on Reliability, 73(1), 83-84.
Badorf, F., van Lierop, H., Olsen, M., Kuhn, D. R., & Raunak, M. S. (2024). Using combinatorial frequency approaches to determine suitability of machine learning datasets. Tech. Rep. LOY251008.
2023
Chandrasekaran, J., Lanus, E., Cody, T., Freeman, L.J., Kacker, R., Raunak, M., Kuhn, D.R. From Scoping to Re-engineering: Leveraging Combinatorial Coverage in ML Product Lifecycle (submitted).
Olsen, M., Raunak, M. S., & Kuhn, D. R. (2023, June). Predicting ABM Results with Covering Arrays and Random Forests. In International Conference on Computational Science (pp. 237-252). Cham: Springer Nature Switzerland.
Kuhn, D. R., Raunak, M. S., & Kacker, R. N. (2023, April). Ordered t-way combinations for testing state-based systems. In 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 246-254). IEEE.
Khadka, K., Chandrasekaran, J., Lei, Y., Kacker, R. N., & Kuhn, D. R. (2023, April). Synthetic Data Generation Using Combinatorial Testing and Variational Autoencoder. In 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 228-236). IEEE.
2022
Durso, F., Raunak, M. S., Kuhn, R., & Kacker, R. (2022, October). Analyzing Failures in Artificial Intelligent Learning Systems (FAILS). In 2022 IEEE 29th Annual Software Technology Conference (STC) (pp. 7-8). IEEE.
Freeman, L., Batarseh, F. A., Kuhn, D. R., Raunak, M. S., & Kacker, R. N. (2022). The path to a consensus on artificial intelligence assurance. Computer, 55(3), 82-86.
Kuhn, D. R., Raunak, M. S., Prado, C., Patil, V. C., & Kacker, R. N. (2022, April). Combination Frequency Differencing for Identifying Design Weaknesses in Physical Unclonable Functions. In 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 110-117). IEEE.
Kuhn, D. R., Raunak, M. S., & Kacker, R. N. (2022). Ordered t-way Combinations for Testing State-based Systems. NIST CSWP 26, June 13, 2022.
Laplante, P., & Kuhn, R. (2022, October). AI Assurance for the Public–Trust but Verify, Continuously. In 2022 IEEE 29th Annual Software Technology Conference (STC) (pp. 174-180). IEEE.
Patel, A. R., Chandrasekaran, J., Lei, Y., Kacker, R. N., & Kuhn, D. R. (2022, April). A combinatorial approach to fairness testing of machine learning models. In 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 94-101). IEEE.
Shree, S., Chandrasekaran, J., Lei, Y., Kacker, R. N., & Kuhn, D. R. (2022, August). DeltaExplainer: A Software Debugging Approach to Generating Counterfactual Explanations. In 2022 IEEE International Conference On Artificial Intelligence Testing (AITest) (pp. 103-110). IEEE.
Wagner, M., Leithner, M., Simos, D. E., Kuhn, R., & Kacker, R. (2022, April). Developing multithreaded techniques and improved constraint handling for the tool CAgen. In 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 87-93). IEEE.
2021
Chandrasekaran, J., Lei, Y., Kacker, R., & Kuhn, D. R. (2021, April). A Combinatorial Approach to Explaining Image Classifiers. In 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 35-43). IEEE.
Chandrasekaran, J., Lei, Y., Kacker, R., & Kuhn, D. R. (2021, April). A Combinatorial Approach to Testing Deep Neural Network-based Autonomous Driving Systems. In 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 57-66). IEEE.
Kampel, L., Simos, D. E., Kuhn, D. R., & Kacker, R. N. (2021). An exploration of combinatorial testing-based approaches to fault localization for explainable AI. Annals of Mathematics and Artificial Intelligence, 1-14.
Kuhn, D. R., Raunak, M. S., & Kacker, R. N. (2021). Combinatorial Coverage Difference Measurement. NIST Cybersecurity Whitepaper.
Kuhn, D. R., Raunak, M. S., & Kacker, R. N. (2021). Combinatorial Frequency Differencing. NIST Cybersecurity Whitepaper.
Lanus, E., Freeman, L. J., Kuhn, D. R., & Kacker, R. N. (2021, April). Combinatorial Testing Metrics for Machine Learning. In 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 81-84). IEEE.
Raunak, M. S., & Kuhn, R. (2021). Explainable Artificial Intelligence and Machine Learning. IEEE Computer, 54(10), 25-27.
Toohey, J. R., Raunak, M. S., & Binkley, D. (2021). From Neuron Coverage to Steering Angle: Testing Autonomous Vehicles Effectively. Computer, 54(8), 77-85.
Wu, J. C., & Kacker, R. N. (2021). Standard Errors and Significance Testing in Data Analysis for Testing Classifiers.
2020
Chandrasekaran, J., Feng, H., Lei, Y., Kacker, R., & Kuhn, D. R. (2020, August). Effectiveness of dataset reduction in testing machine learning algorithms. In 2020 IEEE International Conference On Artificial Intelligence Testing (AITest) (pp. 133-140). IEEE.
DR Kuhn, R Kacker, Y Lei, D Simos, "Combinatorial Methods for Explainable AI", Intl Workshop on Combinatorial Testing, Porto, Portugal, March 23-27, 2020.
Kuhn, R., Kacker, R. N., Lei, Y., & Simos, D. (2020). Input Space Coverage Matters. Computer, 53(1), 37-44.
2019
R. Kuhn, R. Kacker, An Application of Combinatorial Methods for Explainability in Artificial Intelligence and Machine Learning. NIST Cybersecurity Whitepaper, May 22, 2019.
2018 and earlier
DR Kuhn, D Yaga, R Kacker, Y Lei, V Hu, Pseudo-Exhaustive Verification of Rule Based Systems, 30th Intl Conf on Software Engineering and Knowledge Engineering, July 2018.
D.R. Kuhn, I. Dominguez, R.N. Kacker and Y. Lei. "Combinatorial Coverage Measurement Concepts and Applications", 2nd Intl Workshop on Combinatorial Testing, Luxembourg, IWCT2013, IEEE, Mar. 2013.
Presentations
- R. Kuhn, R. Kacker, Explainable AI, NIST presentation. PDF Explainable AI PPT Explainable AI
- D R Kuhn, R Kacker, Risk, Assurance, and Explainability for Autonomous Systems, Advancements in Test and Evaluation of Autonomous Systems (ATEAS) workshop, Dayton, OH, Oct, 2019. NIST presentation.
- R. Kuhn, Assured Autonomy - Problems and Possible Solutions, Hill AFB, Aug, 2021.
