Learning Natural Language Generation with Truncated Reinforcement Learning
By Alice Martin, Guillaume Quispe, Charles Ollion, Sylvain Le Corff, Florian Strub, Olivier Pietquin. 2022
9 citations recorded by the source corpus.
This paper introduces TRUncated ReinForcement Learning for Language (TrufLL), an original approach to train conditional languagemodels without a supervised learning phase, by only using reinforcement learning (RL). As RL methods unsuccessfully scale to large action spaces, we dynamically truncate the vocabulary space using a generic language model. TrufLL thus enables to train a language agent by solely interacting with its environment without any task-specific prior knowledge; it is only guided with a task-agnostic language model. Interestingly, this approach avoids the dependency to labelled datasets and inherently reduces pretrained policy flaws such as language or exposure biases. We evaluate TrufLL on two visual question generation tasks, for which we report positive results over performance and language metrics, which we then corroborate with a human evaluation. To our knowledge, it is the first approach that successfully learns a language generation policy without pre-training, using only reinforcement learning.
Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Hannaneh Hajishirzi
- Yejin Choi
- Jack Hessel
- Prithviraj Ammanabrolu
- Mohit Bansal
- Luke S. Zettlemoyer
- Rafet Sifa
- Christian Bauckhage
In Natural Language Processing
- Hannaneh Hajishirzi
- Yejin Choi
- Jack Hessel
- Prithviraj Ammanabrolu
- Mohit Bansal
- Luke S. Zettlemoyer
- Kianté Brantley
- Christian Bauckhage
In Reinforcement Learning
- Hannaneh Hajishirzi
- Yejin Choi
- Kianté Brantley
- Christian Bauckhage
- Prithviraj Ammanabrolu
- Rafet Sifa
- Wanqiao Xu
- Rajkumar Ramamurthy
Citations in the corpus
- Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “Bahdanau et al. (2016) proposed an actor-critic algorithm to address the variance/large action space problems when using REINFORCE for language generation; follow-up works such as KG-A2C (Ammanabrolu & Hausknecht, 2020), TrufLL (Martin et al., 2022), AE-DQN (Zahavy et al., 2018), and GALAD (Ammanabr”
- Reinforcement Learning for Language Models (2023, methodology): “Prior studies utilize sampling ideas [17, 18] and penalization of less likely actions in a critic [19, 20] to address the large action space problem.”