AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model

By Saleh Soltan, Shankar Ananthakrishnan, Jack G. M. FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andrew Rosenbaum, Anna Rumshisky and 6 more. 2022

91 citations recorded by the source corpus.

In this work, we demonstrate that multilingual large-scale sequence-to-sequence (seq2seq) models, pre-trained on a mixture of denoising and Causal Language Modeling (CLM) tasks, are more efficient few-shot learners than decoder-only models on various tasks. In particular, we train a 20 billion parameter multilingual seq2seq model called Alexa Teacher Model (AlexaTM 20B) and show that it achieves state-of-the-art (SOTA) performance on 1-shot summarization tasks, outperforming a much larger 540B PaLM decoder model. AlexaTM 20B also achieves SOTA in 1-shot machine translation, especially for low-resource languages, across almost all language pairs supported by the model (Arabic, English, French, German, Hindi, Italian, Japanese, Marathi, Portuguese, Spanish, Tamil, and Telugu) on Flores-101 dataset. We also show in zero-shot setting, AlexaTM 20B outperforms GPT3 (175B) on SuperGLUE and SQuADv2 datasets and provides SOTA performance on multilingual tasks such as XNLI, XCOPA, Paws-X, and XWinograd. Overall, our results present a compelling case for seq2seq models as a powerful alternative to decoder-only models for Large-scale Language Model (LLM) training.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

In Large Language Models

In Natural Language Processing

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “Additionally, we compare to XGLM (7.5B) 1-shot results (Lin et al., 2021) and 32-shot AlexaTM results (Soltan et al., 2022).”
A Comprehensive Overview of Large Language Models (2023, unknown): “We identify critical information in Table I and II such as architecture, training strategies, and pipelines that improve LLMs’ performance or other abilities acquired because of changes mentioned in section III.”
A Survey of Large Language Models (2023, unknown): “4T tokens - X PaLM [56] Apr-2022 540 - - - 780B tokens - 6144 TPU v4 - X X AlexaTM [98] Aug-2022 20 - - - 1.”
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (2023, methodology): “Encoder-Decoder or Encoder-only ELMo [80], BERT [28], RoBERTa [65], DistilBERT [90], BioBERT [57], XLM [54], Xlnet [119], ALBERT [55], ELECTRA [24], T5 [84], XLM-E [20], ST-MoE [133], AlexaTM [95] Training: Masked Language Models Model type: Discriminative (BERT-style) Pretrain task: Predict masked ”
When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards (2024, background): “The only information conveyed by the examples is related to formatting (Soltan et al., 2022).”
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM (2023, unknown): “A TM (Soltan et al., 2022).”
Scaling Data-Constrained Language Models (2023, methodology): “Mixing of data is employed for the pre-training data of multilingual LLMs, where text data from different languages is combined [23, 126, 100, 74].”
CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023, methodology): “Inspired by recent advances in NLP research [Tay et al., 2022, Soltan et al., 2022], we explore to combine span denoising with CLM tasks to improve the model with better causal generation capability [Le et al., 2022].”