The Hadoop Distributed File System

By Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler. IEEE Conference on Mass Storage Systems and Technologies, 2010

5,389 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/8ce4c0ee315d86f32ec7354ccdf8d8996e8ee270

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

Splitwise: Efficient Generative LLM Inference Using Phase Splitting (2023, background): “Distributed dataflow systems orchestrate large-scale computational graphs and aim to provide general-purpose programmability [34], [46], [75], [82].”
CloudCV: Large-Scale Distributed Computer Vision as a Cloud Service (2015, background): “We envision a system where a program running on CloudCV simply “calls” for a feature; if it is cached, the features are immediately loaded from distributed storage ( HDFS [44]); if it is not cached, then the feature extraction code is run seamlessly in the background and the results are cached for f”
The Snowflake Elastic Data Warehouse (2016, methodology): “The next choice was then between using S3 or developing our own storage service based on HDFS or similar [46].”
Hopper: Decentralized Speculation-aware Cluster Scheduling at Scale (2015, background): “We now give an overview of the implementation of Hopper in decentralized and centralized settings.”
Multimodal Learning for Web Information Extraction (2017, methodology): “To scale up the system, we group web pages into blocks (each having around 40, 000 web pages), and compute both stages in a distributed manner (based on Apache Spark MapReduce [28]), where data blocks are stored in a distributed file system (based on Hadoop HDFS [24]) providing around 2.”
Trinity: a distributed graph engine on a memory cloud (2013, result): “To support fault-tolerant data persistence, these memory trunks are also backed up in a shared distributed file system called TFS (Trinity File System), which is similar to HDFS [10].”
Fast crash recovery in RAMCloud (2011, background): “There is considerable evidence that this is not t he case in datacenters [23, 8]; for example, it is not unusual for ent ir racks to become inaccessible at once.”
GraphSC: Parallel Secure Computation Made Easy (2015, background): “Section IV and Figure 5 later in the paper will elaborate on the details of our models and terminology.”
On the Optimality of Averaging in Distributed Statistical Learning (2014, methodology): “Several platforms such as Map-Reduce [Dean and Ghemawat, 2008], Hadoop [Shvachko et al., 2010], and Spark [Zaharia et al.”
Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks (2014, background): “Distributed Storage Systems In big data analytics, distributed file systems [15, 45, 49], e.”
Distributed under Creative Commons Cc-by 4.0 Mlitb: Machine Learning in the Browser (background): “The usage of web browsers as compute nodes provides the capability of running sophisticated ML algorithms without the expense and technical difficulty of using custom grid or super-computing facilities (e.g., Hadoop cloud computing Shvachko et al. (2010)).”
Multi-tenant GPU Clusters for Deep Learning Workloads: Analysis and Implications (2018, result): “This is usually due to either transient error in HDFS or HDFS name node recovery.”
Faasm: Lightweight Isolation for Efficient Stateful Serverless Computing (2020, background): “Spark [79] and Hadoop [62] support stateful distributed computation.”
The Missing Piece in Complex Analytics: Low Latency, Scalable Model Management and Serving with Velox (2014, methodology): “For the training phase, we might compute W and X periodically (e.g., daily) using a large-scale cluster compute framework like Spark or GraphLab [9] based on a snapshot of the ratings logs stored in a distributed ﬁlesystem like HDFS [16].”