LongT5: Efficient Text-To-Text Transformer for Long Sequences

By Mandy Guo, J. Ainslie, David C. Uthus, Santiago Ontañón, Jianmo Ni, Yun-Hsuan Sung, Yinfei Yang. 2021

414 citations recorded by the source corpus.

Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the performance of Transformer-based neural models. In this paper, we present a new model, called LongT5, with which we explore the effects of scaling both the input length and model size at the same time. Specifically, we integrated attention ideas from long-input transformers (ETC), and adopted pre-training strategies from summarization pre-training (PEGASUS) into the scalable T5 architecture. The result is a new attention mechanism we call {\em Transient Global} (TGlobal), which mimics ETC's local/global attention mechanism, but without requiring additional side-inputs. We are able to achieve state-of-the-art results on several summarization tasks and outperform the original T5 models on question answering tasks.

Subfields: Artificial Intelligence, Large Language Models, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Model Architectures

In Efficiency & Compression

Citations in the corpus

FNet: Mixing Tokens with Fourier Transforms (2021, methodology): “…not require generation so we leave FNet decoders and encoder-decoder setups to future work; although we do remark that the FNet encoder could be used as a drop in replacement in a Transformer as other works have successfully demonstrated; see, for example, (Zaheer et al., 2020; Guo et al., 2021).”
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024, positive): “Scaling attention to longer context will unlock new capabilities (modeling and reasoning over multiple long documents [24, 43, 49] and ﬁles in large codebases [30, 47]), new modalities (high-resolution images [10], audio [23], video [25]), and new applications (user interaction with long history…”
RWKV: Reinventing RNNs for the Transformer Era (2023, background): “, 2022), including sparse attention (Beltagy et al., 2020; Kitaev et al., 2020; Guo et al., 2022), approximating the full attention matrix (Wang et al.”
Unifying Language Learning Paradigms (2022, methodology): “We later realized, while preparing to put this paper up on arxiv that there have been new results on Scrolls benchmark using a model (Guo et al., 2021) using 16k sequence lengths as opposed to ours (2k) where we kept it at 2k once we had obtained sota.”
FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference (2022, background): “LongT5 (Guo et al., 2022) routes long-range attention through a small number of global tokens.”
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval (2024, neutral): “LongT5 XL (Guo et al., 2022) 53.1 CoLT5 XL (Ainslie et al., 2023) 53.9 RAPTOR + GPT-4 55.7 Comparison to State-of-the-art Systems Building upon our controlled comparisons, we examine RAPTOR’s performance relative to other state-of-the-art models.”
A Comprehensive Overview of Large Language Models (2023, neutral): “Moreover, BLIP-2 [201] proposes a two-stage strategy to pre-train a Querying Transformer (Q-Former) for the alignment between vision and language modalities: in the first stage, vision-language representation learning is bootstrapped from a frozen visual encoder; and in the second stage, a frozen…”
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (2023, methodology): “There is a push towards devising efficient attention mechanisms, such as linear nested attention [80], Transient Global attention [44], CoLT5 [4], and Synthesizer [111].”
Prompt Injection: Parameterization of Fixed Inputs (2022, neutral): “There has been a surge of recent works addressing this problem [6, 38, 1, 13, 40, 8].”
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, methodology): “This also implies that our proposed HTML-denoising can scale to larger-size models consistently.”