Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits

By Ruibo Liu, Chenyan Jia, Ge Zhang, Ziyu Zhuang, Tony X. Liu, Soroush Vosoughi. 2023

44 citations recorded by the source corpus.

We present Second Thought, a new learning paradigm that enables language models (LMs) to re-align with human values. By modeling the chain-of-edits between value-unaligned and value-aligned text, with LM fine-tuning and additional refinement through reinforcement learning, Second Thought not only achieves superior performance in three value alignment benchmark datasets but also shows strong human-value transfer learning ability in few-shot scenarios. The generated editing steps also offer better interpretability and ease for interactive error correction. Extensive human evaluations further confirm its effectiveness.

Subfields: Natural Language Processing, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

1. Minlie Huang
2. Qun Liu
3. Zhexin Zhang
4. Hao Sun
5. Liangyou Li
6. Jiawen Deng
7. Jiale Cheng
8. Xin Jiang

Citations in the corpus

• Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “Thus, an edition module takes effect after the generation to fix some problems in some works (Liu et al., 2023b; Lo-gacheva et al., 2022).”
• Aligning Large Language Models with Human: A Survey (2023, methodology): “Motivated by these findings, Liu et al. (2022b) recommend training LMs to produce edit operations between source (i.e., low-quality responses) and target (i.e., high-quality responses) sequences, which are subsequently integrated into a dynamic programming framework.”