End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient

By Li Zhou, Kevin Small, Oleg Rokhlenko, Charles Peter Elkan. 2017

43 citations recorded by the source corpus.

Learning a goal-oriented dialog policy is generally performed offline with supervised learning algorithms or online with reinforcement learning (RL). Additionally, as companies accumulate massive quantities of dialog transcripts between customers and trained human agents, encoder-decoder methods have gained popularity as agent utterances can be directly treated as supervision without the need for utterance-level annotations. However, one potential drawback of such approaches is that they myopically generate the next agent utterance without regard for dialog-level considerations. To resolve this concern, this paper describes an offline RL method for learning from unannotated corpora that can optimize a goal-oriented policy at both the utterance and dialog level. We introduce a novel reward function and use both on-policy and off-policy policy gradient to learn a policy offline without requiring online user interaction or an explicit state space definition.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, background): “…al., 2016; Nguyen et al., 2017; Kiegeland & Kreutzer, 2021), summarization (Stiennon et al., 2020; Paulus et al., 2017), dialogue (Li et al., 2016; Zhou et al., 2017; Jaques et al., 2020), image captioning (Rennie et al., 2017), question generation (Pang & He, 2021), text-games (Narasimhan et…”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “In prior work, offline RL approaches have been applied in the areas of dialogue and language interfaces, where datasets consist of logged interactions, such as agent-customer transcripts (Zhou et al., 2017).”
MOReL : Model-Based Offline Reinforcement Learning (2020, background): “[12], and has applications in healthcare [33, 34, 35], recommendation systems [36, 37, 38, 39], dialogue systems [40, 19, 41], and autonomous driving [42].”
Text Generation with Efficient (Soft) Q-Learning (2021, methodology): “Previous work has used either importance weighted PG [42, 68, 25] or Q -learning based algorithms [16, 23, 38].”
Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks (2020, background): “Similarly, industrial chatbots can easily collect vast amounts of interaction logs, which can be utilized with offline RL methods (Kandasamy et al., 2017; Zhou et al., 2017; Hancock et al., 2019).”