A Survey for In-context Learning

By Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, Zhifang Sui. arXiv.org, 2023

401 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/8aa98fbfb6f1e979dead13ce24075503fe47658e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (2024, background): “Recent years have witnessed the emergence of various prompting techniques for large language models (LLMs), such as Chain-of-Thought (COT) (Wei et al., 2022), In-context Learning (ICL) (Dong et al., 2023), and Retrieval Augmented Generation (RAG) (Lewis et al., 2020).”
Language is All a Graph Needs (2023, background): “In prompt engineering (Li and Liang, 2021; Lester et al., 2021; Shin et al., 2020) or in-context learning (Dong et al., 2022), pretrained models are usually frozen.”
In-Context Retrieval-Augmented Language Models (2023, methodology): “Inspired by the success of In-Context Learning (Brown et al., 2020; Dong et al., 2023), In-Context RALM refers to the following specific, simple method of concatenating”
A Survey on Model Compression for Large Language Models (2023, unknown): “This approach targets whole sets of weights at once, offering the advantage of reducing model complexity and memory usage while maintaining the overall LLM structure intact.”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, unknown): “Existing studies have demonstrated that LLMs possess a strong capability for in-context learning (Dong et al., 2022), which enables them to extract and utilize valuable information from context knowledge to rectify nonfactual claims they previously generated.”
Large Language Models are not Fair Evaluators (2023, background): “However, it is unclear how reliable LLMs are as evaluators, as they are known to be sensitive to textual instructions and inputs (Dong et al., 2022; Turpin et al., 2023; Bowman, 2023).”
Harms from Increasingly Agentic Algorithmic Systems (2023, background): “…and Gebru 2018; Ehsan et al. 2022; Kasy and Abebe 2021; steven t. piantadosi [@spiantado] 2022; Wolfe and Caliskan 2022], the generation of toxic language [Abid et al. 2021; Gehman et al. 2020], and informational harms [Carroll et al. 2022; Jiang et al. 2019; Li et al. 2022; Weidinger et al. 2022].”
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression (2023, methodology): “The commonly used technologies such as In-Context Learning (ICL) (Min et al., 2022; Dong et al., 2023), Retrieval Augment Generation (RAG) (Lewis et al., 2020; Asai et al., 2024), and Multi-turn Agent (Shen et al., 2024; Park et al., 2023; Wu et al., 2023a) are driving prompts to be increasingly…”
Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation (2023, background): “In the case of Text-to-SQL, the objective is to generate queries that match the given questions, thus LLMs are supposed to learn the mapping from questions to SQL queries.”
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (2023, methodology): “The ICL paradigm is widely adopted with LLMs [18, 34, 78].”
Design Principles for Generative AI Applications (2024, background): “One emerging skill revolves around crafting efective natural language prompts, known as in-context learning [20, 40, 189] or prompt engineering [185, 193].”
Can We Edit Factual Knowledge by In-Context Learning? (2023, methodology): “Demonstration Organization Previous studies (Liu et al., 2022; Rubin et al., 2022; Lu et al., 2022) suggest that demonstration organization including Demonstration Selection and Demonstration Ordering (Dong et al., 2023) is also crucial for ICL.”
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024, methodology): “In step two, we sample tasks from Task Pool as in-context examples (Dong et al., 2022) and prompt GPT-4 Turbo to generate approximately 100 new tasks (20 iterations).”
In-Context Learning Creates Task Vectors (2023, background): “This capability, referred to as In-Context Learning (ICL), has been used extensively, yielding impressive empirical results (Brown et al., 2020; Liu et al., 2023; Dong et al., 2022).”
A Comprehensive Overview of Large Language Models (2023, unknown): “Here in masked language modeling, the attention is not bidirectional but unidirectional, attending either left-to-right or right-to-left context.”
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023, methodology): “Intuitively, in-context learning works by learning through analogies drawn from the given demonstration examples (Dong et al., 2023).”