Batch Policy Gradient Methods for Improving Neural Conversation Models

By Kirthevasan Kandasamy, Yoram Bachrach, Ryota Tomioka, Daniel Tarlow, David Carter. 2017

37 citations recorded by the source corpus.

We study reinforcement learning of chatbots with recurrent neural network architectures when the rewards are noisy and expensive to obtain. For instance, a chatbot used in automated customer service support can be scored by quality assurance agents, but this process can be expensive, time consuming and noisy. Previous reinforcement learning work for natural language processing uses on-policy updates and/or is designed for on-line learning settings. We demonstrate empirically that such strategies are not appropriate for this setting and develop an off-policy batch policy gradient method (BPG). We demonstrate the efficacy of our method via a series of synthetic experiments and an Amazon Mechanical Turk experiment on a restaurant recommendations dataset.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Stefan Riezler
2. Julia Kreutzer
3. Carolin (Haas) Lawrence
4. Eric P. Xing
5. Zhiting Hu
6. Han Guo
7. Bowen Tan
8. Zhengzhong Liu

In Natural Language Processing

1. Stefan Riezler
2. Julia Kreutzer
3. Carolin (Haas) Lawrence
4. Zhiting Hu
5. Eric P. Xing
6. Han Guo
7. Zhengzhong Liu
8. Bowen Tan

In Reinforcement Learning

1. Julia Kreutzer
2. Stefan Riezler
3. Carolin (Haas) Lawrence
4. Zhengzhong Liu
5. Bowen Tan
6. Eric P. Xing
7. Charles Peter Elkan
8. Oleg Rokhlenko

Citations in the corpus

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, neutral): “(2011), and restaurant recommendation (Kandasamy et al., 2017).”
• End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient (2017, methodology): “Off-policy policy gradient is estimated by [8]:”
• Text Generation with Efficient (Soft) Q-Learning (2021, methodology): “Previous work has used either importance weighted PG [42, 68, 25] or Q -learning based algorithms [16, 23, 38].”
• Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks (2020, methodology): “logs, which can be utilized with offline RL methods (Kandasamy et al., 2017; Zhou et al., 2017; Hancock et al., 2019).”