BloombergGPT: A Large Language Model for Finance

By Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Stuart Rosenberg, Gideon Mann. 2023

1,395 citations recorded by the source corpus.

The use of NLP in the realm of financial technology is broad and complex, with applications ranging from sentiment analysis and named entity recognition to question answering. Large Language Models (LLMs) have been shown to be effective on a variety of tasks; however, no LLM specialized for the financial domain has been reported in literature. In this work, we present BloombergGPT, a 50 billion parameter language model that is trained on a wide range of financial data. We construct a 363 billion token dataset based on Bloomberg's extensive data sources, perhaps the largest domain-specific dataset yet, augmented with 345 billion tokens from general purpose datasets. We validate BloombergGPT on standard LLM benchmarks, open financial benchmarks, and a suite of internal benchmarks that most accurately reflect our intended usage. Our mixed dataset training leads to a model that outperforms existing models on financial tasks by significant margins without sacrificing performance on general LLM benchmarks. Additionally, we explain our modeling choices, training process, and evaluation methodology. We release Training Chronicles (Appendix C) detailing our experience in training BloombergGPT.

Subfields: Artificial Intelligence, Management Science and Operations Research, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Large Language Models

Citations in the corpus

Llemma: An Open Language Model For Mathematics (2023, background): “, 2022; 2023), finance (Wu et al., 2023), or science (Taylor et al.”
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (2023, methodology): “that are optimized for specific tasks such as code generation, BloombergGPT [117] for the financial domain.”
How Can Recommender Systems Benefit from Large Language Models: A Survey (2023, neutral): “Wang et al. [ 2022 ] train a general-purpose recommendation model based on items with mixture-of-modality features, which are encoded by language or vision foundation models.”
ChipNeMo: Domain-Adapted LLMs for Chip Design (2023, background): “One approach is to train a domain specific foundation model from scratch, e.g., BloombergGPT [10] for finance, BioMedLLM [11] for biomed, and Galactica [39] for science.”
VerilogEval: Evaluating Large Language Models for Verilog Code Generation (2023, background): “Question: Explain the high-level functionality of the Verilog module. assign {out[0],out[1],out[2],out[3],out[4], out[5],out[6],out[7] User Prompt: Question: Explain the high-level functionality of the Verilog module.”
SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023, neutral): “…nature of such jailbreaks, which are often difficult to detect or mitigate [11, 12], pose a significant barrier to the widespread deployment of LLMs, given that the text generated by these models may influence educational policy [13], medical diagnoses [14, 15], and business decisions [16].”
Jailbreaking Black Box Large Language Models in Twenty Queries (2023, neutral): “Although still at its infancy, the field of study surrounding large language models (LLMs) has shown significant promise in advancing numerous fields, including code generation [28], business analytics [38], and medicine [32].”
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “, 2017), helping analysts with logical queries about statistical data (Wu et al., 2023; Yang et al., 2023a), and assisting in theorem proving and scientific discovery in advanced research fields (Taylor et al.”
FinGPT: Open-Source Financial Large Language Models (2023, neutral): “BloombergGPT [ Wu et al. , 2023 ] , though remarkable in its finance-specific capabilities, comes with an intensive computational requirement.”
Large Language Models in Finance: A Survey (2023, background): “• The fine-tuned finance LLMs outperform BloombergGPT[55] in most finance tasks reported in the papers.”
Large Language Models for Education: A survey and outlook (2024, neutral): “Traditional algorithms, constrained by their reliance on fixed-form responses, have limited the applicability of study assistance tools in real-world educational settings.”
Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, result): “Unlike previous ML algorithms, these large-scale models with a huge amount of parameters have shown versatile comprehension and reasoning capabilities in various fields such as health care [7], law [8], finance [9], education, and so on [10].”
Recommender Systems in the Era of Large Language Models (LLMs) (2023, neutral): “Here are the top five candidate movies : Here is the movie rating history of a user: 8.0 9.2 9.8 7.5 Based on the above rating history of this user, please rate a movie named John Wick: Chapter 4 with a range of 1-10 points.”
Large Language Models for Information Retrieval: A Survey (2023, neutral): “The term “document” will henceforth refer to any text-based content subject to retrieve, including both long articles and short passages.”