The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning

By Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Raghavi Chandu, Chandra Bhagavatula, Yejin Choi. arXiv.org, 2023

325 citations recorded by the source corpus.

The alignment tuning process of large language models (LLMs) typically involves instruction learning through supervised fine-tuning (SFT) and preference tuning via reinforcement learning from human feedback (RLHF). A recent study, LIMA (Zhou et al. 2023), shows that using merely 1K examples for SFT can achieve significant alignment performance as well, suggesting that the effect of alignment tuning might be"superficial."This raises questions about how exactly the alignment tuning transforms a base LLM. We analyze the effect of alignment tuning by examining the token distribution shift between base LLMs and their aligned counterpart. Our findings reveal that base LLMs and their alignment-tuned versions perform nearly identically in decoding on the majority of token positions. Most distribution shifts occur with stylistic tokens. These direct evidence strongly supports the Superficial Alignment Hypothesis suggested by LIMA. Based on these findings, we rethink the alignment of LLMs by posing the research question: how effectively can we align base LLMs without SFT or RLHF? To address this, we introduce a simple, tuning-free alignment method, URIAL. URIAL achieves effective alignment purely through in-context learning (ICL) with base LLMs, requiring as few as three constant stylistic examples and a system prompt. We conduct a fine-grained and interpretable evaluation on a diverse set of examples, named JUST-EVAL-INSTRUCT. Results demonstrate that base LLMs with URIAL can match or even surpass the performance of LLMs aligned with SFT or SFT+RLHF. We show that the gap between tuning-free and tuning-based alignment methods can be significantly reduced through strategic prompting and ICL. Our findings on the superficial nature of alignment tuning and results with URIAL suggest that deeper analysis and theoretical understanding of alignment is crucial to future LLM research.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/600d9287efc4703bdb99ce39b5e8b37da0baa6f6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Safety Alignment Should Be Made More Than Just a Few Tokens Deep (2024, unknown): “Some works have also noted asymmetries in the representation power and utility of different tokens (Zhang & Wu, 2024; Lin et al., 2023; He et al., 2024).”
WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild (2024, methodology): “In addition, we ask LLM judges generate structured explanations that enable human verification for further calibration, inspired by Just-Eval [17].”
Search-o1: Agentic Search-Enhanced Large Reasoning Models (2025, unknown): “…documents based on both the current search query and previous reasoning steps, and then produces refined information that seamlessly integrates with the prior reasoning chain. forgetting general capabilities (Lin et al., 2024; Dong or expanding training data (Henighan et al Zeng et al., 2024).”
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, methodology): “We use URIAL [35] to elicit responses from the base model.”
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2024, unknown): “One straightforward approach to obtain this over-all score is to calculate the average of the aspect-specific scores (Lin et al., 2023; Lin and Chen, 2023b).”