Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations

By Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, Kang Min Yoo, Taeuk Kim. 2022

153 citations recorded by the source corpus.

Despite recent explosion of interests in in-context learning, the underlying mechanism and the precise impact of the quality of demonstrations remain elusive.Intuitively, ground-truth labels should have as much impact in in-context learning (ICL) as supervised learning, but recent work reported that the input-label correspondence is significantly less important than previously thought.Intrigued by this counter-intuitive observation, we re-examine the importance of ground-truth labels in in-context learning.With the introduction of two novel metrics, namely Label-Correctness Sensitivity and Ground-truth Label Effect Ratio (GLER), we were able to conduct quantifiable analysis on the impact of ground-truth label demonstrations.Through extensive analyses, we find that the correct input-label mappings can have varying impacts on the downstream in-context learning performances, depending on the experimental configuration.Through additional studies, we identify key components, such as the verbosity of prompt templates and the language model size, as the controlling factor to achieve more noise-resilient ICL.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, background): “Since the ﬁrst version of our paper, Kim et al. (2022) showed that using negated labels substantially lowers the performance in classiﬁcation.”
A Survey for In-context Learning (2023, background): “In terms of the effect of input-label mapping, Kim et al. (2022b) drew an opposite conclusion that correct input-label mapping does impact the ICL performance, depending on speciﬁc experimental settings.”
When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards (2024, unknown): “We find that changing the in-formativeness We also experiment with removing subject information from instructions (Figure A.3, Tables”
A Survey on In-context Learning (2022, background): “Putting multiple corpora together may give rise to emergent ICL ability, pretraining on corpora related to the downstream tasks does not always improve the ICL performance, and models with lower perplexity do not always perform better in the ICL scenarios.”
Large Language Models Can Be Easily Distracted by Irrelevant Context (2023, background): “A line of work studies the model performance with incorrect prompting ex-emplars, i.e., the example problems are paired with wrong answers (Min et al., 2022; Kim et al., 2022).”
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters (2022, methodology): “Zhang et al. (2023) proposes a method for automatically constructing the in-context exemplars for CoT.”