Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation

By Xinting Huang, Jianzhong Qi, Yu Sun, Rui Zhang. 2020

21 citations recorded by the source corpus.

Dialogue policy optimization often obtains feedback until task completion in task-oriented dialogue systems. This is insufficient for training intermediate dialogue turns since supervision signals (or rewards) are only provided at the end of dialogues. To address this issue, reward learning has been introduced to learn from state-action pairs of an optimal policy to provide turn-by-turn rewards. This approach requires complete state-action annotations of human-to-human dialogues (i.e., expert demonstrations), which is labor intensive. To overcome this limitation, we propose a novel reward learning approach for semi-supervised policy learning. The proposed approach learns a dynamics model as the reward function which models dialogue progress (i.e., state-action sequences) based on expert demonstrations, either with or without annotations. The dynamics model computes rewards by predicting whether the dialogue progress is consistent with expert demonstrations. We further propose to learn action embeddings for a better generalization of the reward function. The proposed approach outperforms competitive policy learning baselines on MultiWOZ, a benchmark multi-domain dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Li Shen
2. Mingchen Gao
3. Tom Young
4. Zhenyi Wang
5. Qiuling Suo
6. Le Fang
7. Tiehang Duan
8. Irwin King

In Computer Vision and Pattern Recognition

1. Li Shen
2. Mingchen Gao
3. Zhenyi Wang
4. Qiuling Suo
5. Le Fang
6. Tiehang Duan
7. Tom Young
8. Irwin King

Citations in the corpus

• Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions (2022, background): “Similar scenarios occur when Amazon extends the market for the personalized conversation AI product, Alexa, to different countries [19,29].”
• Recent advances in deep learning based dialogue systems: a systematic survey (2021, methodology): “Huang et al. (2020a) evaluated the dialogue systems by combining the utterance-level contextualized representation and topic-level graph representation.”
• A Survey on Deep Semi-Supervised Learning (2021, background): “Dialogue policy learning MultiWOZ [111] [112] Dialogue generation Cornell Movie Dialogs Corpus [113], Ubuntu Dialogue Corpus [114] [115]”