Data warehousing and analytics infrastructure at facebook

By Ashish Thusoo, Zheng Shao, Suresh Anthony, Dhruba Borthakur, Namit Jain, Joydeep Sen Sarma, R. Murthy, H. Liu. SIGMOD Conference, 2010

464 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/03c7355211f80f7e415fdeb544520776a3d4b0d1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

Effective Straggler Mitigation: Attack of the Clones (2013, unknown): “Increas- USENIX Association 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI '13) 189 ing replication, however, is difficult as clusters already face a dearth of storage space [19] [20].”
RCFile: A fast and space-efficient data placement structure in MapReduce-based warehouse systems (2011, background): “In this section, we will introduce three existing data placement structures, and discuss why they may not be suitable to a Hadoop-based data warehouse system.”
Practical Lessons from Predicting Clicks on Ads at Facebook (2014, methodology): “The online joiner outputs a real-time training data stream to an infrastructure called Scribe [10].”
Rethinking erasure codes for cloud file systems: minimizing I/O for recovery and degraded reads (2012, background): “For these reasons, Facebook is evaluating RAID-6 and erasure codes in their cloud infrastructure [47].”
Gaia: Geo-Distributed Machine Learning Approaching LAN Speeds (2017, methodology): “Our experiments on our prototypes of Gaia running across 11 Amazon EC2 global regions and on a cluster that emulates EC2 WAN bandwidth show that Gaia provides 1.8–53.5× speedup over two state-of-the-art distributed ML systems, and is within 0.94–1.40× of the speed of running the same ML algorithm…”
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale (2020, background): “Readers access model training data in parallel from remote storage Hive, Facebook’s exabyte-scale data warehouse [51].”
CopyCatch: stopping group attacks by spotting lockstep behavior in social networks (2013, background): “More information about Facebook’s infrastructure can be found in [26].”
Data-centric Artificial Intelligence: A Survey (2023, unknown): “Many big tech companies have built infrastructures to organize, understand, and debug data for building AI systems [7, 15, 230, 233].”
On the use of MapReduce for imbalanced big data using Random Forest (2014, methodology): “In 2010, Facebook had 21 Peta Bytes of internal warehouse data with 12 TB new data added every day and 800 TB compressed data scanned daily [51].”
Random Forests for Big Data (2015, background): “Im-50 proved subsampling strategies can also be designed, like the core-set strategy used for clustering problems in [14], which extracts a relevant small set of points to perform an approximate clustering efficiently.”
The Anatomy of the Facebook Social Graph (2011, methodology): “Unless otherwise noted, calculations were performed on a Hadoop cluster with 2,250 machines, using the Hadoop/Hive data analysis framework developed at Facebook [45, 46].”
Big Data computing and clouds: Trends and future directions (2013, background): “[125] to develop an analytics platform to process Facebook’s large data sets.”