Safe and effective fine-grained TCP retransmissions for datacenter communication

By V. Vasudevan, Amar Phanishayee, Hiral Shah, E. Krevat, D. Andersen, G. R. Ganger, Garth A. Gibson, Brian Mueller. Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication, 2009

499 citations recorded by the source corpus.

This paper presents a practical solution to a problem facing high-fan-in, high-bandwidth synchronized TCP workloads in datacenter Ethernets---the TCP incast problem. In these networks, receivers can experience a drastic reduction in application throughput when simultaneously requesting data from many servers using TCP. Inbound data overfills small switch buffers, leading to TCP timeouts lasting hundreds of milliseconds. For many datacenter workloads that have a barrier synchronization requirement (e.g., filesystem reads and parallel data-intensive queries), throughput is reduced by up to 90%. For latency-sensitive applications, TCP timeouts in the datacenter impose delays of hundreds of milliseconds in networks with round-trip-times in microseconds. Our practical solution uses high-resolution timers to enable microsecond-granularity TCP timeouts. We demonstrate that this technique is effective in avoiding TCP incast collapse in simulation and in real-world experiments. We show that eliminating the minimum retransmission timeout bound is safe for all environments, including the wide-area.

Subfields: Computer Networks and Communications

View Paper: https://www.semanticscholar.org/paper/6b6868ebaa63352615c29a319d63bc2cf5e44956

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

Citations in the corpus

Improving datacenter performance and robustness with multipath TCP (2011, background): “Recent research has shown there are benefits from tuning TCP for data center use, such as by reducing the minimum retransmit timeout[16], but the problem TCP solves remains unchanged.”
The Impact of Virtualization on Network Performance of Amazon EC2 Data Center (2010, background): “The minimum TCP retransmission timeout is normally set to 200 ms in today’s Linux kernel [14].”
Reproducible network experiments using container-based emulation (2012, methodology): “For example, the Incast paper reports results for one hardware queue size and link rate, but it was not clear when to expect the same effect with slower links, or how to set the queue size [43].”
DeTail: reducing the flow completion time tail in datacenter networks (2012, methodology): “For FH, we use a TCP RTOmin of 10ms, as suggested by prior work [12, 31].”
IX: A Protected Dataplane Operating System for High Throughput and Low Latency (2014, background): “We support extremely high-resolution timeouts, as low as 16 μs, which has been shown to improve performance during TCP incast congestion [64].”
Less Is More: Trading a Little Bandwidth for Ultra-Low Latency in the Data Center (2012, background): “Second, it is important to quantify the buffering requirements for incast-like communication patterns [50] with our approach.”
Re-architecting datacenter networks and stacks for low latency and high performance (2017, methodology): “Even if we use aggressively small timers of Vasudevan et al.[38], MPTCP (and any tail-loss TCP variant) is crippled by synchronized losses leading to large and unpredictable FCTs. DCTCP uses ECN to get early feedback, as well as large shared buffer switches to absorb initial bursts, so it does signi”
Language-Directed Hardware Design for Network Performance Monitoring (2017, background): “Effective performance monitoring of large networks is crucial to quickly localize problems like high queueing latency [12], TCP incast [61], and load imbalance across network links [27].”
pFabric: minimal near-optimal datacenter transport (2013, methodology): “Note that all schemes use a small minRTO which has been shown to greatly mitigate the Incast problem [19] (DCTCP additionally benefits from aggressive ECN marking [3]).”
CONGA: distributed congestion-aware load balancing for datacenters (2015, methodology): “We then revisit the design decisions shown in Figure 1 in more detail from the top down.”
Data center TCP (DCTCP) (2010, methodology): “Recently, researchers have shown [32, 13] that lowering the RTO min to 1ms, and us-ing high-resolution retransmission timers alleviates the impact of incast-induced timeouts.”
ICTCP: Incast Congestion Control for TCP in data center networks (2010, methodology): “Previous solutions, focused on either reducing the waiting time for packet loss recovery by faster retransmissions [15], or controlling switch buﬀer occupation to avoid overﬂow by using ECN and modi-ﬁed TCP at both sender and receiver sides[2].”
RDMA over Commodity Ethernet at Scale (2016, background): “They either tune the retransmission timer (as in [35]), or control the TCP receiving window ([39]), or tune the ECN parameter ([38]).”
DRILL: Micro Load Balancing for Low-latency Data Center Networks (2017, background): “hosts run an incast application similar to [69], and 10% of them send simultaneous requests for 10KB flows to 10% of the other hosts (all randomly selected).”
Finishing flows quickly with preemptive scheduling (2012, methodology): “Unless otherwise stated, we use single-rooted tree, a commonly used data center topology for evaluating transport protocols [3, 19, 20, 22].”
A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters (2020, background): “is well-known for creating incast and packet loss in TCP/IP network [66].”
Better never than late: meeting deadlines in datacenter networks (2011, background): “This has been observed in past work [8,23] and is also present our experiments.”