A Solution to the Network Challenges of Data Recovery in Erasure-coded Distributed Storage Systems: A Study on the Facebook Warehouse Cluster

By K. V. Rashmi, Nihar B. Shah, Dikang Gu, Hairong Kuang, Dhruba Borthakur, Kannan Ramchandran. USENIX Workshop on Hot Topics in Storage and File Systems, 2013

329 citations recorded by the source corpus.

Erasure codes, such as Reed-Solomon (RS) codes, are being increasingly employed in data centers to combat the cost of reliably storing large amounts of data. Although these codes provide optimal storage efficiency, they require significantly high network and disk usage during recovery of missing data. In this paper, we first present a study on the impact of recovery operations of erasure-coded data on the data-center network, based on measurements from Facebook's warehouse cluster in production. To the best of our knowledge, this is the first study of its kind available in the literature. Our study reveals that recovery of RS-coded data results in a significant increase in network traffic, more than a hundred terabytes per day, in a cluster storing multiple petabytes of RS-coded data. To address this issue, we present a new storage code using our recently proposed "Piggybacking" framework, that reduces the network and disk usage during recovery by 30% in theory, while also being storage optimal and supporting arbitrary design parameters. The implementation of the proposed code in the Hadoop Distributed File System (HDFS) is underway. We use the measurements from the warehouse cluster to show that the proposed code would lead to a reduction of close to fifty terabytes of cross-rack traffic per day.

Subfields: Computer Networks and Communications

View Paper: https://www.semanticscholar.org/paper/5e63e47cb3386b032ec43a92ce5980466228c761

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

Citations in the corpus

Locally Repairable Codes (2012, unknown): “.”
One extra bit of download ensures perfectly private information retrieval (2014, background): “This solution is clearly impractical due to the massive amount of download required, but is the only feasible solution if there is only a single storage node storing the data [5].”
Locality and Availability in Distributed Storage (2014, background): “Some of these results have found their way into practice: codes with small locality were recently deployed in Azure production clusters [20], while others have been tested in Facebook clusters [6], [21].”
Speeding Up Distributed Machine Learning Using Codes (2015, background): “Recently, codes have been extensively used for the storage layer to provide reliability by exploiting storage redundancy [4, 7, 9].”