When is blockchain a problem for privacy? Immutability can be a problem because private information stored in a blockchain cannot be deleted. Laws and regulations may require that users be allowed to remove private information at their request. Thus there is a need for redactable blockchain and redactable distributed ledger technology.
When is blockchain a problem for security? Immutability can be a problem because security sensitive information stored in a blockchain cannot be deleted. Security policies may require deleting data that is security sensitive when it is no longer needed, especially for biometric data.
We have designed and implemented in Hyperledger Fabric, a new form of distributed ledger technology (DLT), a data block matrix, which provides the integrity assurance of blockchain but allows for controlled revision or deletion of data. This is an essential property for using DLT in applications that must support privacy requirements for deletion of private data at a user's request. Blockmatrix extends the range of application for blockchain solutions by solving the conflict between privacy regulations and blockchain, and allowing exception management. This brings blockchain capabilities to Identity and Access Management.
Hyperledger Fabric drop-in component for data block matrix is now available.
More information on the code and distribution can be found at: https://csrc.nist.gov/projects/redactable-distributed-ledger.
Blockchains provide a strong mechanism to ensure that data blocks have not been altered, but this feature conflicts with many privacy requirements, such as those in GDPR, which allow users to have private data deleted at their request. The immutability property makes a blockchain solution impractical for many such privacy rules, leading to the need for an editable or redactable blockchain.
The blockchain immutability property solves the problem of double spending in cryptocurrencies. But conventional blockchains are hard to use in many distributed system applications, without the ability to modify or delete data. The added trust of distributed ledgers is a valuable feature, providing greatly simplified auditability and verification of actions among multiple parties in applications such as supply chain and others, but the inability to modify or delete data is often still needed.
Rethinking Distributed Ledger Technology (IEEE Computer) explains the need for a redactable blockchain and distributed ledgers.
New journal paper:
Roberts, J. D., Defranco, J. F., & Kuhn, D. R. (2023). Data Block Matrix and Hyperledger Implementation: Extending Distributed Ledger Technology for Privacy Requirements. ACM Distributed Ledger Technologies: Research and Practice, 2(2), 1-11. https://doi.acm.org?doi=3585539
Presentations:
NIST's Secure Data-Sharing Platform for Clinical Data, incorporating Hyperledger data block matrix, at Healthcare Information and Management Systems Society (HIMSS 2023)
Redactable Distributed Ledger: a Clinical Trial Use Case, 9th IEEE World Forum on Internet of Things (IEEE WFIoT2023)
Redactable Distributed Ledger (pdf) - IEEE Global Emerging Technology, 2022
Algorithm and data structure:
Kuhn, D. R. (2022). A Data Structure for Integrity Protection with Erasure Capability, NIST CSWP 25
(final version of 2018 paper, updated to include distribution and availability information)
Potential application areas include anywhere that a blockchain could be used, but a need for privacy also exists.
- Next Generation Database Access Control (NDAC) (open source)
- We won! Technology Maturation Accelerator Program - funding to further develop "NIST inventions that have commercialization potential, put the spotlight on them, and help maneuver them from the labs to the commercial market." The first effort in this project will be to integrate blockmatrix functions into Hyperledger Fabric, making it possible to use Hyperledger in a broader range of applications.
- European Telecommunications Standards Institute, Introduction to Permissioned Distributed Ledger (PDL), Jan 24, 2022 (see also summary announcement)
- European Telecommunications Standards Institute, IPv6 Security, Cybersecurity, Blockchain ETSI GR IP6 031 V1.1.1 (Tech. Rpt.)
- European Telecommunications Standards Institute, standards work item: IPv6 and Cloud using DataBlockMatrix for Food Supply Chain Tracking and Tracing IPv6-based DataBlockMatrix
Blockmatrix has significant applications in digital logistics, and is being used in Cities2030, a large EU project to increase efficiency and safety in cities and regional food supply chains. It is also being used to protect data in medical records, as part of Next Gen Database Access Control, with multiple licenses issued.
We are also studying possibilities for use of blockmatrix in central bank digital currency (CBDC) applications. Privacy is a central issue for CBDC, and blockmatrix may offer solutions. Its integration into Hyperledger Fabric could make it useful in applications supporting the proposed European Central Bank digital currency. See also this presentation to the European Telecommunications Standards Institute, from Sept. 14, 2021.
Technology Transfer
The data block matrix is patented but easily licensed for industry use (US Patent #11,175,826).
- Federal Laboratory Consortium for Technology Transfer - Available technology - Data Block Matrix
- patent and licensing information - NIST Technology Partnerships Office
- patent highlighted on NIST Invent calendar
- presentation to European Telecommunications Standards Institute, 14 Sept 2021, "Blockmatrix Data Structure and Hyperledger Implementation"
- presentation (mp4) with audio narration for IEEE 5G & Blockchain Summit, Sept 23, 2021, "Blockchain vs. Blockmatrix and Hyperledger Implementation"
The most widely recognized form of DLT is the blockchain structure, which provides the basis for cryptocurrencies and a variety of other applications. Most currently available distributed ledger designs using blockchain provide certain properties:
We compare these properties with the needs of more typical applications of distributed data storage and retrieval in Table 1. Note that six of the nine blockchain properties designed for cryptocurrency are at odds with the requirements of many other applications.
Cryptocurrency |
Finance, supply chain, etc. |
Pseudo-Anonymity
|
ID required for contracts or regulation
|
Public access
|
Controlled access
|
Small transaction size
|
Large documents, images
|
Immutable records
|
Changes and deletions often required
|
Proof of work
|
Flexible consensus models
|
Block ordering
|
Timestamps often required
|
Decentralization
|
Same in many applications
|
Replication
|
Same in many applications
|
Integrity protection
|
Same in many applications
|
Table 1. Comparing characteristics of DLT applications
The mismatch between blockchain properties and many application needs has led to a number of problems in applying blockchain designs to data management problems. For example, Bitcoin is designed to provide some degree of anonymity in transactions (i.e., only public identifiers, not real-world identities are used), but the law may prohibit anonymity for many types of transactions and require participants to be identified for tax or other purposes. Laws such as the European Union General Data Protection Regulation (GDPR), that require the ability to delete privacy relevant information, may limit the type of information that can be stored in a blockchain.
For system engineers, the price of distributed trust is often added complexity. The design choices that were made to incorporate anonymity and prevent double spending in blockchains often lead to seemingly unnecessary complications when applied to areas beyond cryptocurrency. For example, immutability has resulted in designs where alterable records must be kept off of the blockchain, with only pointers to them stored in the blockchain itself. Alternatively, some designs involve encrypting data on the blockchain, then destroying the encryption key to “delete” the data. Neither of these options may be desirable for many applications, as the first option leads to unnecessary complications, and the second risks the data being decrypted in the future, when data must be protected for decades. These are serious design issues for supporting privacy requirements such as those of GDPR, resulting in proposals such as an “editable blockchain” using new forms of hashing. For cryptocurrency, a consensus algorithm is needed to guarantee record ordering in the absence of a central time authority (i.e., transactions are ordered based on group consensus, rather than time of entry into a system), and this ordering is used to prevent double spending. Designs for access control using blockchain may involve tokenizing permissions, then passing these to users, and spending down the value to remove a permission from a user. All of these strategies are needed to take advantage of blockchain’s trust properties, but blockchains would probably not be used if a more conventional database could provide the desired distributed trust.
At first glance, blockchain solutions for applications such as supply chain, financial settlement, and others may appear to offer nothing more than added complexity in comparison with a conventional database. However, when more than one organization is involved, the decentralized trust of blockchains and other distributed ledgers can be a tremendous advantage. For example, consider regulated industries where audit is a part of doing business. Every node on the system can have a full set of records detailing the movement of assets. Any shared database can keep track of asset movement, but DLT adds trust by maintaining current, integrity-protected records at every organization, making it easy to audit the process. Thus, the financial industry views full traceability and simplified reconciliation of transactions among the key advantages of DLT. We can view DLT as adding a layer of distributed trust to the problem of data storage and retrieval.
Blockchain's hash-based integrity verification provides trust, at the cost of an inability to delete or update records, leading to design complications that would not arise with conventional database management systems. Similarly, the sequencing guarantees of blockchain consensus protocols are needed for cryptocurrency in the absence of a universal timestamp. Moreover, actions within the distributed ledger must be connected with other actions in the real world, through accurate timestamps. We are developing a new architecture that provides the trust features of blockchains, with characteristics that allow for simpler designs and greater practicality in conventional data management problems. This alternative can lead to new approaches to incorporating trust into distributed systems applications. The data blockmatrix data structure provides key capabilities:
Blockchain – provides integrity, immutability
Data block matrix – provides integrity, erasure
Team
Josh Roberts, NIST
Joanna DeFranco, NIST and Penn State University
Sylvain Chantreau, NIST
Cedric Bell, NIST
Rick Kuhn, NIST
Jeff Voas, NIST
David Ferraiolo, NIST
Dylan Yaga, NIST
Additional resources