A Survey on In-context Learning

By Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, Zhifang Sui. Conference on Empirical Methods in Natural Language Processing, 2022

1,101 citations recorded by the source corpus.

With the increasing capabilities of large language models (LLMs), in-context learning (ICL) has emerged as a new paradigm for natural language processing (NLP), where LLMs make predictions based on contexts augmented with a few examples. It has been a significant trend to explore ICL to evaluate and extrapolate the ability of LLMs. In this paper, we aim to survey and summarize the progress and challenges of ICL. We first present a formal definition of ICL and clarify its correlation to related studies. Then, we organize and discuss advanced techniques, including training strategies, prompt designing strategies, and related analysis. Additionally, we explore various ICL application scenarios, such as data engineering and knowledge updating. Finally, we address the challenges of ICL and suggest potential directions for further research. We hope that our work can encourage more research on uncovering how ICL works and improving ICL.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/30c0cdc414f68211d5d0514df027cec22e005174

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation (2024, neutral): “Few-Shot Learning: In few-shot learning, the model is provided with a small number of task-specific examples at inference time.”
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2024, neutral): “Length Bias , another prevalent challenge, skews evaluations by disproportionately favoring longer or shorter responses regardless of quality (Zheng et al., 2023; Koo et al., 2023a).”
Towards Making the Most of ChatGPT for Machine Translation (2023, background): “Compared with traditional machine translation systems, ChatGPT can incorporate additional information, like human interactions, through the input prompts (Dong et al., 2023).”
A Survey on Large Language Models for Code Generation (2024, neutral): “For bias in multi-PL, the results showed various LLMs’ performance in generating code in different PLs. Base models achieved the highest Pass@1 rate in Python for outperforming C++ and Java by 5.7% and 11.3%, respectively.”
In-Context Retrieval-Augmented Language Models (2023, background): “More importantly, they tend to produce factual inac-curacies and errors (Lin et al., 2022; Maynez et al., 2020; Huang et al., 2020).”
RAG vs. GraphRAG: A Systematic Evaluation and Key Insights (2025, unknown): “On the MultiHop-RAG benchmark, the gains from inference-time strategies are more pronounced.”