Long-context LLMs Struggle with Long In-context Learning

By Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, Wenhu Chen. Trans. Mach. Learn. Res., 2024

375 citations recorded by the source corpus.

Large Language Models (LLMs) have made significant strides in handling long sequences. Some models like Gemini could even to be capable of dealing with millions of tokens. However, their performance evaluation has largely been confined to metrics like perplexity and synthetic tasks, which may not fully capture their true abilities in more challenging, real-world scenarios. We introduce a benchmark (LongICLBench) for long in-context learning in extreme-label classification using six datasets with 28 to 174 classes and input lengths from 2K to 50K tokens. Our benchmark requires LLMs to comprehend the entire input to recognize the massive label spaces to make correct predictions. We evaluate on 15 long-context LLMs and find that they perform well on less challenging classification tasks with smaller label space and shorter demonstrations. However, they struggle with more challenging task like Discovery with 174 labels, suggesting a gap in their ability to process long, context-rich sequences. Further analysis reveals a bias towards labels presented later in the sequence and a need for improved reasoning over multiple pieces of information. Our study reveals that long context understanding and reasoning is still a challenging task for the existing LLMs. We believe LongICLBench could serve as a more realistic evaluation for the future long-context LLMs.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/2717e5c7384ec12cfd6cf9c34897c6adad3230ed

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Damai Dai
2. Xu Sun
3. Lei Li
4. Zhiyong Wu
5. Qingxiu Dong
6. Ce Zheng
7. Zhifang Sui
8. Baobao Chang

In Information Systems

1. Xu Sun
2. Lei Li
3. Baobao Chang
4. Jingjing Xu

Citations in the corpus

• A Survey on In-context Learning (2022, background): “Domain-specific investigation is required to fully leverage the potential of in-context learning in various modalities.”
• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (2024, neutral): “…unlock a multitude of complex real-world applications, such as repository-level code understanding [BSK + 24, JYW + 23, POC + 23], long-document question-answering [CPG + 23, LZD + 24], self-play reasoning [Ope24], extreme-label in-context learning [LZD + 24], and long-horizon agent tasks [Wen23].”
• AutoSurvey: Large Language Models Can Automatically Write Surveys (2024, unknown): “An example survey focused on Emotion Recognition using LLMs is in Appendix F. window limitations : LLMs encounter inherent restrictions in output length due to limited processing windows [13–17].”