Sequence Parallelism: Long Sequence Training from System Perspective

By Shenggui Li, Fuzhao Xue, Yongbin Li, Yang You. 2021

185 citations recorded by the source corpus.

Transformer achieves promising results on various tasks. However, self-attention suffers from quadratic memory requirements with respect to the sequence length. Existing work focuses on reducing time and space complexity from an algorithm perspective. In this work, we propose sequence parallelism, a memory-efficient parallelism to solve this issue from system perspective instead. Our approach is compatible with most existing parallelisms (e.g., data, pipeline, and tensor parallelism), which means our sequence parallelism makes 4D parallelism possible. More importantly, we no longer require a single device to hold the whole sequence. Besides, using efficient attention with linear complexity, our sequence parallelism enables us to train transformer with infinite long sequence. Specifically, we split the input sequence into multiple chunks and feed each chunk into its corresponding device (i.e., GPU). To compute the attention output, we integrated ring-style communication with self-attention calculation and proposed Ring Self-Attention (RSA). Experiments show that sequence parallelism performs well when scaling with batch size and sequence length. Compared with tensor parallelism, our approach achieved 13.7\times and 3.0\times maximum batch size and sequence length respectively when scaling up to 64 NVIDIA P100 GPUs. With efficient attention, sequence can handle sequence with over 114K tokens, which is over 27\times longer than existing efficient attention works holding the whole sequence on a single device.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Reducing Activation Recomputation in Large Transformer Models (2022, background): “Sequence parallelism as suggested in [9] where activations are partitioned along sequence dimensions throughout the network can alleviate this problem.”
Long Context Transfer from Language to Vision (2024, methodology): “Efforts have also been made on data curation [22, 79, 5] and system optimization [40, 45, 27] during long context training.”
A Survey of Large Language Models (2023, unknown): “Besides, Colossal-AI has also implemented tensor parallelism for higher-dimensional tensors [182–184] and proposed sequence parallelism [185] especially for sequence data, which can further decompose the attention operation of the Transformer model.”
LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, methodology): “Previous work, Ring-Style sequence parallelism (Li et al., 2021; 2023a; Liu et al., 2023a; Zhu, 2023) ignores the heterogeneous networking feature on GPUs and utilizes P2P communication in both inter-node and intra-node settings.”
Yi: Open Foundation Models by 01.AI (2024, methodology): “We continue pretrain the full-attention model using sequence parallelism [43] and distributed attention.”
Ring Attention with Blockwise Transformers for Near-Infinite Context (2023, methodology): “While ring communication has found applications in other parallel computing scenarios [2, 16, 14, 34], our work stands out as the first work to show that it can be applied to self-attention as used in Transformers and to make it fit efficiently into Transformer training and inference without adding…”
Gated Linear Attention Transformers with Hardware-Efficient Training (2023, unknown): “5 Empirical Study”