Alignment & RLHF

Instruction tuning, RLHF, preference optimization, helpful/harmless assistants.

A topic in the Science corpus, drawn from 99 papers.

Trusted voices in Alignment & RLHF

Ordered by the trust Alignment & RLHF's own network concentrates on each researcher, most first.

  1. Christian P. Robert — trusted by 56 researchers here
  2. Dario Amodei — trusted by 69 researchers here
  3. Paul Christiano — trusted by 71 researchers here
  4. Jan Leike — trusted by 34 researchers here
  5. Shane Legg — trusted by 30 researchers here
  6. Tom B. Brown — trusted by 31 researchers here
  7. Nick Bostrom — trusted by 19 researchers here
  8. Angelica Chen — trusted by 23 researchers here
  9. Miljan Martic — trusted by 23 researchers here
  10. Jeff Wu — trusted by 22 researchers here
  11. Stuart Armstrong — trusted by 14 researchers here
  12. W. B. Knox — trusted by 17 researchers here

Most-cited papers in Alignment & RLHF

Ordered by the recorded citation count from the source corpus; a citation is not an endorsement.

  1. Training language models to follow instructions with human feedback (2022, Neural Information Processing Systems) — 22,660 citations
  2. Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, arXiv.org) — 17,670 citations
  3. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, Neural Information Processing Systems) — 9,702 citations
  4. Deep Reinforcement Learning from Human Preferences (2017, Neural Information Processing Systems) — 5,646 citations
  5. Scaling Instruction-Finetuned Language Models (2022, Journal of machine learning research) — 4,301 citations
  6. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (2022, arXiv.org) — 4,254 citations
  7. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning (2023) — 3,705 citations
  8. Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, Annual Meeting of the Association for Computational Linguistics) — 3,444 citations
  9. Constitutional AI: Harmlessness from AI Feedback (2022, arXiv.org) — 3,377 citations
  10. Learning to summarize from human feedback (2020) — 3,335 citations
  11. Fine-Tuning Language Models from Human Preferences (2019) — 2,638 citations
  12. LIMA: Less Is More for Alignment (2023) — 1,362 citations
  13. Towards a Unified View of Parameter-Efficient Transfer Learning (2021) — 1,223 citations
  14. A General Language Assistant as a Laboratory for Alignment (2021) — 1,175 citations
  15. Scaling Laws for Reward Model Overoptimization (2022) — 1,126 citations
  16. WizardCoder: Empowering Code Large Language Models with Evol-Instruct (2023) — 1,012 citations
  17. Aligning AI With Shared Human Values (2020) — 970 citations
  18. The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023) — 953 citations
  19. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023) — 921 citations
  20. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022) — 837 citations
  21. Instruction Tuning with GPT-4 (2023) — 837 citations
  22. Otter: A Multi-Modal Model With In-Context Instruction Tuning (2023) — 692 citations
  23. Improving alignment of dialogue agents via targeted human judgements (2022) — 676 citations
  24. Scalable agent alignment via reward modeling: a research direction (2018, arXiv.org) — 615 citations
  25. SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization (2019) — 613 citations
  26. RRHF: Rank Responses to Align Language Models with Human Feedback without tears (2023) — 545 citations
  27. How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources (2023) — 510 citations
  28. Reward learning from human preferences and demonstrations in Atari (2018) — 490 citations
  29. Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor (2022) — 487 citations
  30. Policy Shaping: Integrating Human Feedback with Reinforcement Learning (2013) — 484 citations
  31. Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision (2023) — 457 citations
  32. PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (2023) — 400 citations
  33. Two Cheers for Rebooting AI: Building Artificial Intelligence We Can Trust (2020) — 364 citations
  34. The alignment problem from a deep learning perspective (2022) — 333 citations
  35. MIMIC-IT: Multi-Modal In-Context Instruction Tuning (2023) — 310 citations
  36. Pretraining Language Models with Human Preferences (2023) — 302 citations
  37. A Long Way to Go: Investigating Length Correlations in RLHF (2023) — 284 citations
  38. Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive (2024) — 257 citations
  39. Combining manual feedback with subsequent MDP reward signals for reinforcement learning (2010) — 242 citations
  40. Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (2016) — 222 citations
  41. The Capacity for Moral Self-Correction in Large Language Models (2023) — 214 citations
  42. Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training (2019) — 212 citations
  43. Reward-rational (implicit) choice: A unifying formalism for reward learning (2020) — 208 citations
  44. Memory-assisted prompt editing to improve GPT-3 after deployment (2022) — 194 citations
  45. ImageBind-LLM: Multi-modality Instruction Tuning (2023) — 181 citations
  46. Supervising strong learners by amplifying weak experts (2018) — 178 citations
  47. Corrigibility (2015) — 177 citations
  48. Preference-based reinforcement learning: a formal framework and a policy iteration algorithm (2012) — 177 citations
  49. Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021) — 176 citations
  50. SVIT: Scaling up Visual Instruction Tuning (2023) — 176 citations

Most active researchers

By papers in this topic: a volume measure, not trust.