Our Erasure Coding Definition
Erasure coding is a methodology of providing data resilience, where data is broken into fragments or “shards”, then expanded and encoded with redundant data pieces (shards) and then distributed across a set of different data stores, whether dispersed geographically, or across different nodes within the same Datacentre.
An example of Storage Vendor Technologies that Redpalm works with that use Erasure Coding are Cleversafe (using geographical dispersion) and Nutanix (using node-based dispersion).
The benefit of erasure coding is that when data becomes corrupted, it is able to be reconstructed by using information about the data that’s stored elsewhere within in the array or infrastructure. Erasure codes can be used in place of traditional RAID to reduce the overhead and time to restore/reconstruct data. One potential drawback of erasure coding is that it can be more CPU-intensive, which can translate into increased latency, however modern technologies are often able to compensate for this.
Erasure Coding is particularly advantageous where there are large volumes of data or where any application or system needs to tolerate failures, such as disk array systems, data grids, distributed storage environments, object stores and archival storage. One common use case for erasure coding is object-based cloud storage, often referred to as simply “Object Storage”.
Erasure coding works by creating a mathematical function to describe a set of numbers that can be checked for accuracy and recovered if one is lost. This mathematical method is known as polynomial interpolation or oversampling. In mathematical terms, the protection offered can be represented in simple form by the using the following equation: n = k + m. “k” being the original amount of data or symbols and “m” being the redundant symbols that provide protection against failures. “n” is the total number of symbols created after the erasure coding process. For example, in a 10 of 16 configuration, six extra symbols (m) are added to the 10 base symbols (k). The resulting 16 data shards (n) would be spread across 16 drives, nodes or geographic locations. The original file could therefore be reconstructed from 10 fragments or shards.
This method was actually developed more than 50 years ago. In one of the earliest forms of this approach known as Reed-Solomon, data was reconstructed using any combination of “k” symbols, or pieces of data, even if “m” symbols were unavailable, where in a 10 of 16 configuration, six drives, nodes or geographic locations could be unavailable, and the original file could still be recovered.