Few-shot Learning with Multilingual Language Models

By Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du and 11 more. 2021

281 citations recorded by the source corpus.

Subfields: Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

1. Luke S. Zettlemoyer
2. Furu Wei
3. Li Dong
4. Michael Lewis
5. Yi Tay
6. Timo Schick
7. Swaroop Mishra
8. Chitta Baral

Citations in the corpus

• Toolformer: Language Models Can Teach Themselves to Use Tools (2023, background): “…2022) and the related tendency to hallucinate facts (Maynez et al., 2020; Ji et al., 2022), difficulties in understanding low-resource languages (Lin et al., 2021), a lack of mathematical skills to perform precise calculations (Patel et al., 2021) and an unawareness of the progression of time…”
• PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts (2022, background): “Since then, PromptSource and P3 have been extended for research on multi-lingual prompting (Lin et al., 2021) and priming, i.e., in-context few-shot learning (Min et al., 2021).”
• What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, background): “As such, there has been an explosion of work on developing LLMs and training techniques that produce strong zero-shot generalization [Brown et al., 2020, Wang and Komatsuzaki, 2021, Du et al., 2021, Lin et al., 2021, Rae et al., 2021, Hoffmann et al., 2022, Chowdhery et al., 2022].”
• What Language Model to Train if You Have One Million GPU Hours? (2022, methodology): “We train a multilingual model to evaluate the effectiveness and potential impacts of this practice.”
• Language Models are Multilingual Chain-of-Thought Reasoners (2022, background): “Such an approach may be useful as English is often used as the source language for cross-lingual transfer (Hu et al., 2020) and has been found effective when used as the prompt language (Zhao & Schütze, 2021; Winata et al., 2021; Lin et al., 2021b).”
• AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model (2022, methodology): “As can be seen, AlexaTM 20B outperforms the supervised M2M-124 615M model from Goyal et al. (2022) and XGLM 7.5B, which is a multilingual decoder-only model (Lin et al., 2021), across almost all pairs.”
• Leveraging QA Datasets to Improve Generative Data Augmentation (2022, background): “Lin et al. (2021) train multilingual autoregressive language models to enable few-shot learning in multiple languages.”
• In-BoXBART: Get Instructions into Biomedical Multi-Task Learning (2022, methodology): “Weller et al. (2020) and Ye and Ren (2021a) for improving task instructions for large LMs. Recently, many approaches have been proposed to improve model performance using instructions (Wu et al., 2021, 2022; Lin et al., 2021; Kuznia et al., 2022).”
• BioTABQA: Instruction Learning for Biomedical Table Question Answering (2022, background): “In particular, Q exhibits some symptoms/signs and asks about what potential disease it is, e.g., “I have joint pain and swelling on my face, what’s wrong with me”.”
• QAmeleon: Multilingual QA with Only 5 Examples (2022, background): “Prompting in multilingual settings has achieved the best performance using English prompts and target language exemplars (Winata et al., 2021; Lin et al., 2022; Shi et al., 2022).”
• Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, background): “However, they are still useful for expanding the language coverage of the training data and can help bridge the resource gap for languages with limited training data (Urbizu et al., 2023; Lin et al., 2021).”
• DeepNet: Scaling Transformers to 1,000 Layers (2022, background): “The capacity has substantially increased from millions of parameters [2], [3] to billions [4], [5], [6], [7], [8], [9], [10], [11], and even trillions [12], [13].”
• In-context Examples Selection for Machine Translation (2022, methodology): “For example, Lin et al. (2021) demonstrate that a 7.5B multilingual generative model, XGLM, outperforms a supervised sequence-to-sequence baseline in 45 translation directions on the FLORES-101 machine translation benchmark (Goyal et al., 2022) using just 32 randomly sampled translation examples as ”
• Dataset Debt in Biomedical Language Modeling (2022, unknown): “%) or were dead links (5%).”
• Scaling Data-Constrained Language Models (2023, methodology): “Up-sampling is often employed when pre-training data is partly limited, such as data from a high-quality domain like Wikipedia or text in a rare language for training multilingual LLMs [60, 82].”
• Using natural language prompts for machine translation (2022, unknown): “The space of prompting multilingual models, on the other hand, remains underexplored, only having recently been studied by Lin et al. (2021).”
• How Many Data Samples is an Additional Instruction Worth? (2022, unknown): “In addition, many approaches have been proposed to improve model performance us-ing instructions (Wu et al., 2022; Lin et al., 2021; Wang et al., 2022b; Luo et al., 2022; Kuznia et al., 2022; Patel et al., 2022; Mishra and Nouri, 2022).”