Galactica: A Large Language Model for Science

By Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, A. Hartshorn, A. Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, Robert Stojnic. 2022

1,065 citations recorded by the source corpus.

Information overload is a major obstacle to scientific progress. The explosive growth in scientific literature and data has made it ever harder to discover useful insights in a large mass of information. Today scientific knowledge is accessed through search engines, but they are unable to organize scientific knowledge alone. In this paper we introduce Galactica: a large language model that can store, combine and reason about scientific knowledge. We train on a large scientific corpus of papers, reference material, knowledge bases and many other sources. We outperform existing models on a range of scientific tasks. On technical knowledge probes such as LaTeX equations, Galactica outperforms the latest GPT-3 by 68.2% versus 49.0%. Galactica also performs well on reasoning, outperforming Chinchilla on mathematical MMLU by 41.3% to 35.7%, and PaLM 540B on MATH with a score of 20.4% versus 8.8%. It also sets a new state-of-the-art on downstream tasks such as PubMedQA and MedMCQA dev of 77.6% and 52.9%. And despite not being trained on a general corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. We believe these results demonstrate the potential for language models as a new interface for science. We open source the model for the benefit of the scientific community.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Information Systems and Management, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Information Systems and Management

In Large Language Models

In Natural Language Processing

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “Following the scaling laws of Kaplan et al. (2020), several Large Language Models with more than 100B parameters have been proposed, from GPT-3 (Brown et al., 2020) to Gopher (Rae et al., 2022) or specialized models, e.g. Galactica, for science(Taylor et al., 2022).”
Llemma: An Open Language Model For Mathematics (2023, background): “However, if we seek to maximize performance within one domain, such as medicine (Singhal et al., 2022; 2023), finance (Wu et al., 2023), or science (Taylor et al., 2022), a domain-specific language model may offer superior capabilities for a given computational cost, or lower computational cost for…”
C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models (2023, background): “While traditional NLP benchmarks are becoming obsolete, these new ones are extensively used in recent research to assess the latest LLMs (Taylor et al., 2022; Chowdhery et al., 2022; Hoffmann et al., 2022; Touvron et al., 2023; OpenAI, 2023).”
MAmmoTH2: Scaling Instructions from the Web (2024, neutral): “MINERVA [Lewkowycz et al., 2022] and Galactica [Taylor et al., 2022] were pioneers in continued training of LLMs to adapt to scientific domains for math and science reasoning.”
Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training (2023, neutral): “…of larger and higher-quality general or domain-specific data (Touvron et al., 2023a; Zhou et al., 2023; Gunasekar et al., 2023; Feng et al., 2023; Taylor et al., 2022), more sophisticated prompt design (Wei et al., 2022; Zhou et al., 2022; Creswell et al., 2022), or better training algorithms…”
ReFT: Reasoning with Reinforced Fine-Tuning (2024, methodology): “Though online self-training achieves some improvements with Galac-tica, it is still far behind ReFT on average.”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, background): “[252] R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic, “Galactica: A large language model for science,” arXiv preprint arXiv:2211.09085, 2022.”
ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports (2022, methodology): “This requires quantitative studies on the quality of simplified radiology reports generated with ChatGPT and similar LLMs.”
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health (2023, background): “5 [31], ChatGPT [1], GPT-4 [29], Flan-PaLM [8], and Galactica [25] for biomedical applications are being extensively evaluated.”
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (2024, background): “While many sources have been considered for pretraining data including text from books [2, 3, 17, 4], Wikipedia [2, 3, 17, 4], and research papers [2, 4, 18], a highly common choice is to use web text, i.e. text scraped from webpages on the public internet [19, 20].”
Large Language Models for Software Engineering: A Systematic Literature Review (2023, background): “Researchers have observed that scaling up the model sizes significantly enhances their capacity, leading to remarkable performance improvements when the parameter scale surpasses a certain threshold [95, 246, 260].”
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (2024, background): “Large Language Models (LLMs) have demonstrated strong performance across a wide range of tasks (Brown et al., 2020; Taylor et al., 2022; Yuan et al., 2023; Chuang et al., 2024).”
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, positive): “…AI field has also seen a rise in open-source LLMs such as Mistral (Jiang et al., 2023), Llama Series (Touvron et al., 2023a;b; Dubey et al., 2024; Taylor et al., 2022), DeepSeek (Bi et al., 2024; Shao et al., 2024), Qwen (Bai et al., 2023; Yang et al., 2024) etc. Notably, Llama serves as a…”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, positive): “3) Galactica: Aiming to solve the problem of information overload in the scientific field, Galactica was proposed to store, combine, and reason about scientific knowledge, including Healthcare.”