Pathways: Asynchronous Distributed Dataflow for ML

By P. Barham, Aakanksha Chowdhery, Joseph Dean, Sanjay Ghemawat, S. Hand, D. Hurt, Michael Isard, Hyeontaek Lim, Ruoming Pang, Sudip Roy and 6 more. 2022

159 citations recorded by the source corpus.

We present the design of a new large scale orchestration layer for accelerators. Our system, Pathways, is explicitly designed to enable exploration of new systems and ML research ideas, while retaining state of the art performance for current models. Pathways uses a sharded dataflow graph of asynchronous operators that consume and produce futures, and efficiently gang-schedules heterogeneous parallel computations on thousands of accelerators while coordinating data transfers over their dedicated interconnects. Pathways makes use of a novel asynchronous distributed dataflow design that lets the control plane execute in parallel despite dependencies in the data plane. This design, with careful engineering, allows Pathways to adopt a single-controller model that makes it easier to express complex new parallelism patterns. We demonstrate that Pathways can achieve performance parity (~100% accelerator utilization) with state-of-the-art systems when running SPMD computations over 2048 TPUs, while also delivering throughput comparable to the SPMD case for Transformer models that are pipelined across 16 stages, or sharded across two islands of accelerators connected over a data center network.

Subfields: Computer Networks and Communications, Hardware and Architecture, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Hardware and Architecture

In Efficiency & Compression

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “• Efficient scaling – We demonstrate the first large-scale use of Pathways (Barham et al., 2022) – a new ML system which enables training a single model across thousands or tens of thousands of accelerator chips in a highly efficient manner.”
PaLM 2 Technical Report (2023, neutral): “Software: Pathways (Barham et al., 2022), Paxml (aka Pax) (Pax, 2022), Saxml (aka Sax) (Sax, 2022), JAX (Bradbury et al.”
nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models (2022, background): “, 2019)) can reduce communication latency, an inherent variance of GPU performance (caused by various random factors such as fabrication process variation and operating system conditions) is another performance bottleneck (Roy et al., 2022).”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, unknown): “The original study [40] introduced PaLM, a densely-activated decoder-only transformer language model, which was trained using Pathways [212], a large-scale ML accelerator orchestration system that enables efficient training across TPU pods.”
InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, background): “Though a line of vision foundation models is proposed [7, 13–21], video understanding and the corresponding tasks are less explored compared with image ones, mainly used for validating that the visual features from these models are also beneﬁcial to spatiotemporal representations.”
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models (2023, background): “Since MoEs can feature rather different computational profiles from standard dense models, there is also significant research on optimizing inference and training systems (Barham et al., 2022; Gale et al., 2023; Hwang et al., 2023).”
This paper is included in the Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation. (unknown): “TeraPipe [33] discovers a new pipeline parallelism dimension for transformer-based LMs. Google’s Pathway system [7] is a concurrent work of Alpa.”