Using RDMA efficiently for key-value services

By Anuj Kalia, Michael Kaminsky, D. Andersen. Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication, 2014

578 citations recorded by the source corpus.

This paper describes the design and implementation of HERD, a key-value system designed to make the best use of an RDMA network. Unlike prior RDMA-based key-value systems, HERD focuses its design on reducing network round trips while using efficient RDMA primitives; the result is substantially lower latency, and throughput that saturates modern, commodity RDMA hardware. HERD has two unconventional decisions: First, it does not use RDMA reads, despite the allure of operations that bypass the remote CPU entirely. Second, it uses a mix of RDMA and messaging verbs, despite the conventional wisdom that the messaging primitives are slow. A HERD client writes its request into the server's memory; the server computes the reply. This design uses a single round trip for all requests and supports up to 26 million key-value operations per second with 5μs average latency. Notably, for small key-value items, our full system throughput is similar to native RDMA read throughput and is over 2X higher than recent RDMA-based key-value systems. We believe that HERD further serves as an effective template for the construction of RDMA-based datacenter services.

Subfields: Computer Networks and Communications, Hardware and Architecture, Information Systems

View Paper: https://www.semanticscholar.org/paper/29a1148d75878671dc3663bf480e33d7bd91597d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Hardware and Architecture

In Information Systems

Citations in the corpus

Datacenter RPCs can be General and Fast (2018, methodology): “Co-design with RDMA is popular, with numerous examples from key-value stores [44, 36, 24, 61, 60], state machine replication [52], and transaction processing systems [25, 61, 20].”
Design Guidelines for High Performance RDMA Systems (2016, methodology): “Section 4.3 applies the guidelines to improve the CPU efficiency and throughput of the HERD key-value cache [20] by 83% and 35% respectively .”
NetCache: Balancing Key-Value Stores with Fast In-Network Caching (2017, background): “Given the high-throughput and low-latency requirements of large-scale Internet services, key-value storage systems are shifting to in-memory designs [1, 15, 16, 22, 23, 26, 27, 29, 30, 34, 36, 38, 42, 43].”
No compromises: distributed transactions with consistency, availability, and performance (2015, methodology): “We conservatively set the leases to 10ms in all our experiments and have not observed any false positives during their execution.”
Shenango: Achieving High CPU Efficiency for Latency-sensitive Datacenter Workloads (2019, background): “Examples include MICA [46], IX [18], Arrakis [59], mTCP [35], Sandstorm [50], FaRM [25], HERD [39], RAMCloud [57], SoftNIC [31], ZygOS [61], Shinjuku [38], and eRPC [40].”
The RAMCloud Storage System (2015, background): “2014], an operating system that supports kernel bypass for many I/O operations; and Herd [Kalia et al. 2014], a key-value cache that uses RDMA to improve latency and throughput.”
A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters (2020, background): “Conventional one-sided RDMA operations (WRITE and READ) require at least two roundtrips: getting the remote address, and writing (reading) the value to (from) that address [39, 40, 50, 70].”
Fast in-memory transaction processing using RDMA and HTM (2015, methodology): “To bridge HTM (which essentially uses OCC) and 2PL, DrTM implements the exclusive and shared locks using one-sided RDMA operations, which are cache-coherent with local accesses and thus provide strong consistency with HTM.”