On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model

By Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, Hyoungseok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, W. Park, Jung-Woo Ha and 1 more. 2022

112 citations recorded by the source corpus.

Many recent studies on large-scale language models have reported successful in-context zero- and few-shot learning ability. However, the in-depth analysis of when in-context learning occurs is still lacking. For example, it is unknown how in-context learning performance changes as the training corpus varies. Here, we investigate the effects of the source and size of the pretraining corpus on in-context learning in HyperCLOVA, a Korean-centric GPT-3 model. From our in-depth investigation, we introduce the following observations: (1) in-context learning performance heavily depends on the corpus domain source, and the size of the pretraining corpus does not necessarily determine the emergence of in-context learning, (2) in-context learning ability can emerge when a language model is trained on a combination of multiple corpora, even when each corpus does not result in in-context learning on its own, (3) pretraining with a corpus related to a downstream task does not always guarantee the competitive in-context learning performance of the downstream task, especially in the few-shot setting, and (4) the relationship between language modeling (measured in perplexity) and in-context learning does not always correlate: e.g., low perplexity does not always imply high in-context few-shot learning performance.

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “Findings presented in Shin et al. (2022) demonstrate how the pre-training corpus can impact few-shot performance, while Razeghi et al. (2022) investigates how models are able to perform numerical reasoning from in a few-shot setting.”
A Survey of Large Language Models (2023, unknown): “Recent studies either conduct extensive experiments for investigating the effect of emergent abilities and the contributing factors to such abilities [220, 237, 421], or explain some specific abilities with existing theoretical frameworks [60, 231].”
A Survey for In-context Learning (2023, background): “Pretraining corpus domain (Shin et al., 2022a) Pretraining corpus combination (Shin et al., 2022a) Number of model parameters (Wei et al., 2022b; Brown et al., 2020) Number of pretraining steps (Wei et al., 2022b) Inference Label space exposure (Min et al., 2022c) Demonstration input distribution…”
Scaling Data-Constrained Language Models (2023, background): “[94] Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung-Woo Ha, et al.”
Training Trajectories of Language Models Across Scales (2022, background): “On the contrary, Shin et al. (2022) claims the opposite relationship for in-context learning performance and perplexity when training language models with different corpora, but they only test four downstream tasks on a few model checkpoints.”