Director: Generator-Classifiers For Supervised Language Modeling

By Kushal Arora, Kurt Shuster, Sainbayar Sukhbaatar, Jason Aaron Edward Weston. 2022

51 citations recorded by the source corpus.

Current language models achieve low perplexity but their resulting generations still suffer from toxic responses, repetitiveness, and contradictions. The standard language modeling setup fails to address these issues. In this paper, we introduce a new architecture, Director, that consists of a unified generator-classifier with both a language modeling and a classification head for each output token. Training is conducted jointly using both standard language modeling data, and data labeled with desirable and undesirable sequences. Experiments in several settings show that the model has competitive training and decoding speed compared to standard language models while yielding superior results, avoiding undesirable behaviors while maintaining generation quality. It also outperforms existing model guiding approaches in terms of both accuracy and efficiency. Our code is made publicly available.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, methodology): “Using D IRECTOR to combine multiple feedback signals is also effective for the reply corrector .”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “Figure 7: DIRECTOR (Arora et al., 2022) employs a language model head and a classifier head at every step during left-right generation, predicting the next token by combining the two probabilities.”
Using In-Context Learning to Improve Dialogue Safety (2023, methodology): “Other work has investigated using classifiers to guide generation (Dathathri et al., 2020; Arora et al., 2022).”
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback (2022, methodology): “(Arora et al., 2022), when learning from binary feedback, works better than reranking or reward-based learning.”
DialGuide: Aligning Dialogue Model Behavior with Developer Guidelines (2022, background): “Other approaches include strategies for responding to problematic contexts, such as steering away from toxicity (Baheti et al., 2021; Arora et al., 2022), using apologies (Ung et al., 2022), and non-sequiturs (Xu et al., 2021b).”