PaLM 2 Technical Report

By Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Z. Chen and 117 more. 2023

1,524 citations recorded by the source corpus.

We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture of objectives. Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM. This improved efficiency enables broader deployment while also allowing the model to respond faster, for a more natural pace of interaction. PaLM 2 demonstrates robust reasoning capabilities exemplified by large improvements over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable performance on a suite of responsible AI evaluations, and enables inference-time control over toxicity without additional overhead or impact on other capabilities. Overall, PaLM 2 achieves state-of-the-art performance across a diverse set of tasks and capabilities. When discussing the PaLM 2 family, it is important to distinguish between pre-trained models (of various sizes), fine-tuned variants of these models, and the user-facing products that use these models. In particular, user-facing products typically include additional pre- and post-processing steps. Additionally, the underlying models may evolve over time. Therefore, one should not expect the performance of user-facing products to exactly match the results reported in this report.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “With the recent advances of LLMs, LLM-based assistants start to exhibit artificial general intelligence across diverse tasks, from writing and chatting to coding [5, 26, 1, 32].”
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “AudioPaLM [73] further integrated the codec tokens into the LLM PaLM 2 [74], demonstrating promising results on speech recognition and translation tasks.”
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, background): “AudioPaLM [Rubenstein et al., 2023] fused the codec tokens into the LLM PaLM 2 [Anil et al., 2023], and demonstrated promising results on speech recognition and translation tasks.”
PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (2023, background): “Large language models (LLMs) have attracted increasing attention in the field of artificial intelligence [1, 2, 3, 4, 5, 6], with various applications from question answering [7, 8], machine translation [9, 10] to content creation [11, 12].”
WizardCoder: Empowering Code Large Language Models with Evol-Instruct (2023, methodology): “These include OpenAI’s GPT3&4 [1, 2], Google’s PaLM [3, 4], and Bard3, DeepMind’s Chinchilla [5], and Gopher [6], as well as Anthropic’s Claude4.”
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023, background): “Despite the recent advances in this field, a noticeable gap exists between closed-source and open-source LLMs—closed-source models like GPT-4 (OpenAI, 2023), PaLM-2 (Anil et al., 2023), and Claude 2 (Bai et al., 2022) dominate popular mathematical reasoning benchmarks such as GSM8K (Cobbe et al., 20”
Scaling Relationship on Learning Mathematical Reasoning with Large Language Models (2023, background): “Large language models (LLMs) (Anil et al., 2023; Touvron et al., 2023b; OpenAI, 2023) have shown considerable abilities in various math reasoning tasks (Saxton et al.”
Position: Levels of AGI for Operationalizing Progress on the Path to AGI (2023, neutral): “In the next section, we introduce a level-based ontology of AGI.”
"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models (2023, neutral): “PaLM2 [13].”
Large language models in healthcare and medical domain: A review (2023, methodology): “[76] The BERT-D2 model achieved an F1 score of 81.97% on the DDI Extraction 2013 corpus.”
Large Language Models for Mathematical Reasoning: Progresses and Challenges (2024, background): “5, PaLM-2 (Anil et al., 2023), and LLaMa (Touvron et al., 2023a)), and found a significant drop in accuracy for backward reasoning compared to forward reasoning.”
Formalizing and Benchmarking Prompt Injection Attacks and Defenses (2023, methodology): “Examples of LLMs include GPT-4 [34], LLaMA [6], Vicuna [18], and PaLM 2 [12].”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “…an effective and powerful platform for a wide range of downstream tasks: large language models can enable not only fluent text generation [1, 2, 3] but emergent problem-solving [4, 5, 6] and creative generation of prose [7, 2] and code [8], while vision-language models enable open-vocabulary…”
Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts (2023, neutral): “(Anil et al., 2023) 32.9 67.1 0.0 67.9 32.1 0.0”
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models (2023, background): “All evaluations follow the settings from Google et al. (2023), unless specified otherwise.”
Learning Interactive Real-World Simulators (2023, background): “Language models trained on internet text succeed at text-based tasks [1, 43] but not physical tasks, which requires perception and control.”