LaMDA: Language Models for Dialog Applications

By R. Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, A. Jin, Taylor Bos, Leslie W. Baker, Yu Du and 47 more. 2022

1,891 citations recorded by the source corpus.

We present LaMDA: Language Models for Dialog Applications. LaMDA is a family of Transformer-based neural language models specialized for dialog, which have up to 137B parameters and are pre-trained on 1.56T words of public dialog data and web text. While model scaling alone can improve quality, it shows less improvements on safety and factual grounding. We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding. The first challenge, safety, involves ensuring that the model's responses are consistent with a set of human values, such as preventing harmful suggestions and unfair bias. We quantify safety using a metric based on an illustrative set of human values, and we find that filtering candidate responses using a LaMDA classifier fine-tuned with a small amount of crowdworker-annotated data offers a promising approach to improving model safety. The second challenge, factual grounding, involves enabling the model to consult external knowledge sources, such as an information retrieval system, a language translator, and a calculator. We quantify factuality using a groundedness metric, and we find that our approach enables the model to generate responses grounded in known sources, rather than responses that merely sound plausible. Finally, we explore the use of LaMDA in the domains of education and content recommendations, and analyze their helpfulness and role consistency.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “…are GLaM (Du et al., 2021), Gopher (Rae et al., 2021), Chinchilla (Hoffmann et al., 2022), Megatron–Turing NLG (Smith et al., 2022), and LaMDA (Thoppilan et al., 2022), all of which achieved few-shot state-of-the-art results on a significant number of tasks at the time of their release.1…”
Training language models to follow instructions with human feedback (2022, methodology): “…intervals. used for many recent large LMs—predicting the next token on a webpage from the internet—is different from the objective “follow the user’s instructions helpfully and safely” (Radford et al., 2019; Brown et al., 2020; Fedus et al., 2021; Rae et al., 2021; Thoppilan et al., 2022).”
Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, methodology): “One potential method for improving the quality of decoding could involve generating multiple reasoning paths and scoring each of them with a verifier, though this requires training the verifier (Cobbe et al., 2021; Shen et al., 2021; Thoppilan et al., 2022).”
Scaling Up Models and Data with t5x and seqio (2022, neutral): “We also provide a model conﬁguration (without any checkpoints) for a decoder-only architecture that is compatible with LaMDA (Thoppilan et al., 2022).”
Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, methodology): “…have found that even when the desired answer is ﬁxed, introducing diversity in the reasoning processes can be highly beneﬁcial; therefore we leverage sampling, as commonly used for open-ended text generation (Radford et al., 2019; Brown et al., 2020; Thoppilan et al., 2022), to achieve this goal.”
Show Your Work: Scratchpads for Intermediate Computation with Language Models (2021, neutral): “For all experiments, we start with the LaMDA large language model (Thoppilan et al., 2022).”
Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “In our experiments, we use LaMDA-PT, a dense left-to-right, decoder-only transformer language model of 137B parameters (Thoppilan et al., 2022).”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “While OPT-IML models outperform baseline OPT on an extensive set of evaluations (Section 5), nevertheless, they are susceptible to the various risks associated with using large language models relating to factual correctness (Thoppilan et al., 2022; Brown et al., 2020a; Chowdhery et al., 2022), gene”
Emergent Abilities of Large Language Models (2022, neutral): “Better than random and not correlated with scale: boolean expressions, crash blossom, dynamic counting, entailed polarity hindi, epistemic reasoning, factuality of summary, fantasy reasoning, gender sensitivity chinese, gender sensitivity english, high low game, identify math theorems, intersect geo”
OPT: Open Pre-trained Transformer Language Models (2022, background): “We conclude that OPT-175B has a strong ability to maintain a consistent persona across conversations, a behavior also highlighted in LaMDA (Thoppilan et al., 2022).”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, neutral): “0 — Unreleased LaMDA [10, 60, 21] 137 None None ??? — 14.”