Finding a Needle in Haystack: Facebook's Photo Storage

By D. Beaver, Sanjeev Kumar, Harry C. Li, J. Sobel, Peter Vajgel. USENIX Symposium on Operating Systems Design and Implementation, 2010

522 citations recorded by the source corpus.

Subfields: Computer Networks and Communications

View Paper: https://www.semanticscholar.org/paper/a6fef68aa30429ee15223b6be658a4536ab19128

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

Citations in the corpus

Less pain, most of the gain: incrementally deployable ICN (2013, background): “Why does Zipf matter? Anecdotal evidence suggests that in the presence of Zipf workloads, having multiple caching layers or cooperative caching provide limited improvements [6, 52].”
Toward Scalable Systems for Big Data Analytics: A Technology Tutorial (2014, background): “Facebook implemented Haystack [133] to store a massive amount of small-file photos.”
Better never than late: meeting deadlines in datacenter networks (2011, background): “This partition-aggregate structure is shown in Figure 3, and applies to many web applications like search [4], social networks [6], and recommendation systems [10].”
WiscKey (2016, background): “…Sanjay Ghemawat and Jeff Dean 2011], e-commerce [DeCandia et al. 2007], data deduplication [Anand et al. 2010; Debnath et al. 2010], photo stores [Beaver et al. 2010], cloud data [Lai et al. 2015], social networking [Armstrong et al. 2013; Dong 2015; Sumbaly et al. 2012], online gaming [Debnath…”
SILT: a memory-efficient, high-performance key-value store (2011, background): “In systems with lossy-compressible data, e.g., picture stores [7, 26], data can be adaptively compressed to ﬁt in a ﬁxed-sized slot.”
Context-Aware Computing, Learning, and Big Data in Internet of Things: A Survey (2018, methodology): “There are different ﬁle systems in big data, such as Hadoop distributed ﬁle system [188], Cosmos (Microsoft development) [189], Haystack (Facebook development) [190], and Google File System (GFS) [191].”
Big Data: A Survey (2014, background): “Facebook utilizes Haystack [84] to store the large amount of small-sized photos.”
f4: Facebook's Warm BLOB Storage System (2014, background): “…and Implementation (OSDI ’14) 383 f4 currently stores over 65PBs of logical BLOBs and reduces their effective-replication-factor from 3.6 to either 2.8 or 2.1. f4 provides low latency; is resilient to disk, host, rack, and datacenter failures; and provides sufficient throughput for warm BLOBs.”
Towards Making Systems Forget with Machine Unlearning (2015, background): “From a privacy perspective, users who become concerned with new privacy risks of a system often want the system to forget their data and lineage.”
P3: Toward Privacy-Preserving Photo Sharing (2013, unknown): “On the flip side, photo sharing services have raised privacy concerns such as leakage of photos to unauthorized viewers and the use of algorithmic recognition technologies by providers.”
Workload analysis of a large-scale key-value store (2012, unknown): “In the work describing Facebook’s photo storage system [8], the authors presented statistics of I/O requests for the photos, which exhibit clear diurnal patterns, consistent with our observations in this paper (Sec.”
Big data related technologies, challenges and future prospects (2015, background): “Databases other than text files may sometimes be used to store log information to improve the query efficiency of the massive log store [8,9].”