Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning

By Jason Williams, Kavosh Asadi, Geoffry Zweig. 2017

339 citations recorded by the source corpus.

End-to-end learning of recurrent neural networks (RNNs) is an attractive solution for dialog systems; however, current techniques are data-intensive and require thousands of dialogs to learn simple behaviors. We introduce Hybrid Code Networks (HCNs), which combine an RNN with domain-specific knowledge encoded as software and system action templates. Compared to existing end-to-end approaches, HCNs considerably reduce the amount of training data required, while retaining the key benefit of inferring a latent representation of dialog state. In addition, HCNs can be optimized with supervised learning, reinforcement learning, or a mixture of both. HCNs attain state-of-the-art performance on the bAbI dialog dataset (Bordes and Weston, 2016), and outperform two commercially deployed customer-facing dialog systems at our company.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Reinforcement Learning

Citations in the corpus

Keep CALM and Explore: Language Models for Action Generation in Text-based Games (2020, background): “More broadly, the concept of generating candidates and re-ranking has been studied in other interactive lanugage tasks such as dialogue (Zhao and Eskenazi, 2016; Williams et al., 2017; Song et al., 2016; Chen et al., 2017) and communication games (Lazaridou et al., 2020).”
Neural Approaches to Conversational AI (2018, neutral): “It may be replaced by other models, such as a Bayesian version described in the next subsection for efficient exploration, and recurrent networks (Zhao and Eskénazi, 2016; Williams et al., 2017) that can more easily capture information from conversational histories than expert-designed dialogue stat”
Incremental Learning from Scratch for Task-Oriented Dialogue Systems (2019, methodology): “Data-driven methods, such as reinforcement learning (Williams et al., 2017; Zhao and Eskenazi, 2016; Li et al., 2017) and supervised learning (Wen et al., 2016; Eric and Manning, 2017; Bordes et al., 2016), have been applied to optimize dialogue systems automatically.”
End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient (2017, methodology): “Recently, several supervised learning algorithms have been proposed to learn goal-oriented dialog policies [3, 28].”
Toward Continual Learning for Conversational Agents (2017, background): “There have been prior studies that partly address the data-intensiveness problem outside the neural model (Wen et al., 2016b; Williams et al., 2017; Zhao et al., 2017; Eshghi et al., 2017; Wen et al., 2016a; Zhao et al., 2017).”
Key-Value Retrieval Networks for Task-Oriented Dialogue (2017, background): “While the competitions were designed to test systems for state tracking, recent work has chosen to repurpose this data by only using the transcripts of dialogues without state annotation for developing systems (Bordes and Weston, 2016; Williams et al., 2017).”
Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems (2018, background): “End-to-end task-oriented dialog systems train a single model directly on text transcripts of di-alogs (Wen et al., 2017; Serban et al., 2016; Williams et al., 2017; Zhao et al., 2017; Seo et al., 2017; Serban et al., 2017).”
Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems (2018, background): “Williams et al. (2017) proposed a hybrid code network for task-oriented dialogue that can be trained with supervised and reinforcement learning.”
Advancing the State of the Art in Open Domain Dialog Systems through the Alexa Prize (2018, background): “Alquist (Pichl et al., 2018) experimented with statistical dialogue management systems, utilizing hybrid code networks (HCN) (Williams et al., 2017) modified for the open domain use case, by taking in as input the conversational context, along with dialogue acts and system actions, and outputted…”
Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management (2017, background): “A number of authors have proposed training a standard neural-network policy in two stages (Fatemi et al., 2016; Su et al., 2016a; Williams et al., 2017).”
Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems (2019, background): “The applications within NLP has been comparatively limited, e.g., Shu et al. (2016, 2017b) for opinion mining, Shu et al. (2017a) for document classiﬁcation, and Lee (2017) for hybrid code networks (Williams et al., 2017).”
Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models (2019, methodology): “RL techniques are then used to find the optimal dialog policy (Gasic and Young, 2014; Su et al., 2017; Williams et al., 2017).”
Adversarial Learning of Task-Oriented Neural Dialog Models (2018, background): “Williams et al. (2017) designed a hybrid supervised and reinforcement learning end-to-end dialog agent.”
Sentiment Adaptive End-to-End Dialog Systems (2018, methodology): “We kept the same experiment setting in Williams et al. (2017), e.g. last action taken was also used as a feature, along with word embeddings (Mikolov et al., 2013) and bag-of-words; LSTM with 128 hidden-units and AdaDelta optimizer (Zeiler, 2012) were used to train the model.”
Explainable Outfit Recommendation with Joint Outfit Matching and Comment Generation (2018, background): “Text generation involves a wide variety of tasks and studies, such as text summarization [23], [24], machine translation [25], [26], dialogue systems [27], [28], and image captioning [29], [30].”
Multilingual Autoregressive Entity Linking (2021, neutral): “…domains, spanning Question Answering (De Cao et al., 2019; Nie et al., 2019; Asai et al., 2020), Dialogue (Bordes et al., 2016; Wen et al., 2017; Williams et al., 2017; Chen et al., 2017; Curry et al., 2018), Biomedical systems (Leaman and Gonzalez, 2008; Zheng et al., 2015), to name just a…”
End-to-End Learning of Task-Oriented Dialogs (2018, background): “Williams et al. (Williams et al., 2017) proposed a hybrid code network for task-oriented dialog that can be trained with supervised and reinforcement learning (RL).”
OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs (2019, background): “…systems such as conversational recommendations (Salem et al., 2014; Bordes et al., 2017; Sun and Zhang, 2018; Dalton, 2018), information querying (Williams et al., 2017; de Vries et al., 2018; Reddy et al., 2018), etc., with datasets collected mostly through bootstrapped simulations (Bordes et…”