Unifying Language Learning Paradigms

By Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier García, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, Donald Metzler. 2022

139 citations recorded by the source corpus.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, background): “The fourth is UL2 20B (Tay et al., 2022), and the fifth is Codex (Chen et al.”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “More surprisingly, the gains become more signiﬁcant when the language model’s scale increases, e.g., we see +3%-6% absolute accuracy improvement over UL2-20B but +9%-23% for LaMDA-137B and GPT-3.”
Emergent Abilities of Large Language Models (2022, positive): “For example, the zero-shot SuperGLUE performance of UL2 20B (Tay et al., 2022a) is comparable to GPT-3 175B despite UL2 being significantly smaller (6·1022 versus 3.”
Scaling Instruction-Finetuned Language Models (2022, methodology): “Finally, we instruction-ﬁnetune U-PaLM , which is a 540B PaLM model initialized from PaLM-540B and then pretrained with an UL2 objective for 20k additional steps (Tay et al., 2022a,b).”
StarCoder: may the source be with you! (2023, methodology): “Even in cases when models and training data are both released permissively (Ra � el et al., 2020; Tay et al., 2022), external researchers typically do not have an opportunity to participate in guiding the development of industry-produced models.”
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation (2022, background): “More broadly, given that 1) infilling is an important functionality for real-world programming and 2) there is a growing trend in pre-training with the right context (Aghajanyan et al., 2022; Fried et al., 2022; Bavarian et al., 2022; Tay et al., 2022), we expect more future pre-trained models to pe”
Efficient Training of Language Models to Fill in the Middle (2022, neutral): “A more systematic study, similar to [Wang et al., 2022, Tay et al., 2022] but focused on free-form infilling generation, can clarify this further.”
Language Models are General-Purpose Interfaces (2022, background): “It partially conﬁrms that non-causal modeling is conducive to ﬁnetuning (Wang et al., 2022b; Tay et al., 2022; Artetxe et al., 2022).”
Teaching Small Language Models to Reason (2022, neutral): “Here, LLMs are models with at least tens of billions of parameters, such as PaLM 540B (Chowdhery et al., 2022), GPT-3 175B (Brown et al., 2020), or UL2 20B (Tay et al., 2022).”
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? (2022, background): “When it comes to scaling large models, we show that novel inductive biases can be indeed quite risky which might explain why most state-of-the-art large language models (Rae et al., 2021; Chowdhery et al., 2022; Tay et al., 2022) are based on relatively vanilla architectures.”
Is ChatGPT a General-Purpose Natural Language Processing Task Solver? (2023, background): “, 2022], mixture of denoisers [Tay et al., 2022], and majority voting on multiple predictions (self-consistency) [Wang et al.”
Scaling Data-Constrained Language Models (2023, unknown): “UL2 We experimented with the UL2 objective [95, 96] for a causal model but did not find it to outperform regular causal language modeling on our evaluation tasks.”
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2023, methodology): “We have the following observations: • PRP variants based on FLAN-UL2 with 20B parameters can achieve best results on all metrics on TREC-DL2020, and are only second to the blackbox, commercial gpt-4 based solution on NDCG@5 and NDCG@10 on TREC-DL2019, which has an estimated 50X larger model size.”
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (2023, methodology): “We pre-train it with large-scale HTML corpus curated from CommonCrawl on a mixture of long-span denoising objectives (Raffel et al., 2020; Tay et al., 2022), and finetune it for each downstream task.”
CodeT5+: Open Code Large Language Models for Code Understanding and Generation (2023, methodology): “Inspired by recent advances in NLP research [Tay et al., 2022, Soltan et al., 2022], we explore to combine span denoising with CLM tasks to improve the model with better causal generation capability [Le et al., 2022].”