TorchScale: Transformers at Scale

By Shuming Ma, Hongyu Wang, Shaohan Huang, Wenhui Wang, Zewen Chi, Li Dong, Alon Benhaim, Barun Patra, Vishrav Chaudhary, Xia Song and 1 more. 2022

12 citations recorded by the source corpus.

Large Transformers have achieved state-of-the-art performance across many tasks. Most open-source libraries on scaling Transformers focus on improving training or inference with better parallelization. In this work, we present TorchScale, an open-source toolkit that allows researchers and developers to scale up Transformers efficiently and effectively. TorchScale has the implementation of several modeling techniques, which can improve modeling generality and capability, as well as training stability and efficiency. Experimental results on language modeling and neural machine translation demonstrate that TorchScale can successfully scale Transformers to different sizes without tears. The library is available at https://aka.ms/torchscale.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, Model Architectures, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Furu Wei
2. Li Dong
3. Shaohan Huang
4. Shuming Ma
5. Wenhui Wang
6. Saksham Singhal
7. Zewen Chi
8. Xia Song

In Computer Vision and Pattern Recognition

1. Furu Wei
2. Li Dong
3. Wenhui Wang
4. Shaohan Huang
5. Shuming Ma
6. Saksham Singhal
7. Zewen Chi
8. Qiang Liu

In Hardware and Architecture

1. Vishrav Chaudhary
2. Li Dong
3. Saksham Singhal
4. Shuming Ma
5. Shaohan Huang
6. Zewen Chi
7. Furu Wei
8. Xia Song

In Model Architectures

1. Furu Wei
2. Wenhui Wang
3. Li Dong
4. Shuming Ma
5. Shaohan Huang
6. Vishrav Chaudhary
7. Lei Cui
8. Yaru Hao

In Scaling & Learning Theory

1. Li Dong
2. Yaru Hao
3. Qiang Liu
4. Furu Wei

Citations in the corpus

• Language Is Not All You Need: Aligning Perception with Language Models (2023, methodology): “In the future, we would like to scale up K OSMOS -1 in terms of model size [13, 14, 60], and integrate the speech [12] capability into K OSMOS -1.”
• Retentive Network: A Successor to Transformer for Large Language Models (2023, unknown): “The implementation is based on TorchScale (Ma et al., 2022).”
• A Length-Extrapolatable Transformer (2022, background): “For causal sequence modeling, the recurrent-style designs (Dai et al., 2019; Hutchins et al., 2022; Ma et al., 2022b) are also competitive.”
• LongNet: Scaling Transformers to 1, 000, 000, 000 Tokens (2023, methodology): “All experiments are conducted based on the torchscale [MWH + 22] codebase.”