Encrypted Neural Networks Without Overflows

Abstract: In this talk, we will discuss a typically underestimated issue in most fully-homomorphic encryption (FHE)-based machine learning applications. In particular, the Cheon-Kim-Kim-Song (CKKS) scheme enables private inference by evaluating neural networks on encrypted real data, but it only supports addition, multiplication, and rotation operations. On the other hand, neural networks are heavily based on activation functions, which are intrinsically nonlinear. Turning a neural network into a CKKS circuit thus requires approximating all activation functions by polynomials within a certain interval: traditionally, these intervals are sampled (and heuristically widened) activation ranges over a dataset. During the presentation, we will show that CKKS circuits built this way are still vulnerable to overflow events, where legitimate inputs exceed the tolerances of the circuit and cause corrupted outputs. In particular, we propose an efficient algorithm to find such inputs in the (visual) vicinity of benign inputs and demonstrate that it can, e.g., lead to corrupted outputs for 47% of CIFAR-10 images processed by a CKKS circuit. We will show that such overflows can also happen without any malicious intervention: by simply changing the test dataset of a MNIST network to USPS (a dataset that targets the same digit recognition task), we observed overflows in more than 5% of cases. To completely eliminate overflows, we propose a formal verification technique that computes certified bounds on the ranges of all neurons in the network. Our overflow-free solution is compatible with most CKKS-based frameworks, as it allows us to simply substitute heuristically constructed polynomials with rigorously designed ones.

Joint work: Philipp Kern, Alberto Leporati, Edoardo Manino, Lorenzo Rovida, Carsten Sinz, Samuel Teuber.

Suggested readings

• Encrypted Neural Networks without Overflows
• Certified Error Analysis of Homomorphically Encrypted Neural Networks