Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's Datacenter Network

By Arjun Singh, J. Ong, Amit Agarwal, Glen Anderson, Ashby Armistead, Roy Bannon, Seb Boving, Gaurav Desai, Bob Felderman, Paulie Germano and 9 more. 2015

971 citations recorded by the source corpus.

We present our approach for overcoming the cost, operational complexity, and limited scale endemic to datacenter networks a decade ago. Three themes unify the five generations of datacenter networks detailed in this paper. First, multi-stage Clos topologies built from commodity switch silicon can support cost-effective deployment of building-scale networks. Second, much of the general, but complex, decentralized network routing and management protocols supporting arbitrary deployment scenarios were overkill for single-operator, pre-planned datacenter networks. We built a centralized control mechanism based on a global configuration pushed to all datacenter switches. Third, modular hardware design coupled with simple, robust software allowed our design to also support inter-cluster and wide-area networks. Our datacenter networks run at dozens of sites across the planet, scaling in capacity by 100x over ten years to more than 1Pbps of bisection bandwidth.

Subfields: Computer Networks and Communications, Information Systems, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In ML Systems & Hardware

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “Further the hosts between the two pods are connected via the Google datacenter network (Singh et al., 2015).”
Attojoule Optoelectronics for Low-Energy Information Processing and Communications (2016, background): “Already in 2012, a network connecting servers inside just one data center had a capacity of >1 Pb/s (1015 b/s) [29]; such data center network traffic largely does not count the communication of information inside the racks of servers or within the servers themselves, which can only be”
SilkRoad: Making Stateful Layer-4 Load Balancing Fast and Cheap Using Switching ASICs (2017, background): “For SLB we use the state-of-the-art performance 12 Mpps for 52-byte packets using 8 cores reported by Google [20].”
Swift: Delay is Simple and Effective for Congestion Control in the Datacenter (2020, background): “In practice, congestion can happen in the fabric which can be over-subscribed [46].”
RDMA over Commodity Ethernet at Scale (2016, methodology): “Our data center network is an Ethernet-based multi-layer Clos network [1, 3, 19, 31] as shown in Figure 1.”
MAERI: Enabling Flexible Dataflow Mapping over DNN Accelerators via Reconfigurable Interconnects (2018, background): “A fat-tree supports 2x bandwidth at every level from the leaves up to the root, and is a classic topology for providing non-blocking bandwidth, and is used extensively in datacenter networks [38].”