A Hierarchical Recurrent Encoder-Decoder for Generative Context-Aware Query Suggestion

By Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, Jian-Yun Nie. 2015

573 citations recorded by the source corpus.

Users may strive to formulate an adequate textual query for their information need. Search engines assist the users by presenting query suggestions. To preserve the original search intent, suggestions should be context-aware and account for the previous queries issued by the user. Achieving context awareness is challenging due to data sparsity. We present a novel hierarchical recurrent encoder-decoder architecture that makes possible to account for sequences of previous queries of arbitrary lengths. As a result, our suggestions are sensitive to the order of queries in the context while avoiding data sparsity. Additionally, our model can suggest for rare, or long-tail, queries. The produced suggestions are synthetic and are sampled one word at a time, using computationally cheap decoding techniques. This is in contrast to current synthetic suggestion models relying upon machine learning pipelines and hand-engineered feature sets. Results show that our model outperforms existing context-aware approaches in a next query prediction setting. In addition to query suggestion, our architecture is general enough to be used in a variety of other applications.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Generative Models, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Generative Models

In Signal Processing

Citations in the corpus

Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models (2015, methodology): “…variants of our HRED model, and compare against several alternatives, including basic n -gram models (Goodman 2001), a standard (non-hierarchical) RNN trained on the concatenation of the utterances in each triple, and a context-sensitive model (DCGM-I) recently proposed by Sordoni et al. (2015b).”
Neural Approaches to Conversational AI (2018, positive): “ by capturing longer context (Yao et al., 2015; Serban et al., 2016, 2017; Xing et al., 2018). One popular approach is the Hierarchical Recurrent Encoder-Decoder (HRED) model, originally proposed in (Sordoni et al., 2015a) for query suggestion and applied to response generation in (Serban et al., 20”
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents (2019, background): “…further inquiries in the capabilities of these neural network architectures and developments (Serban et al., 2016; Miao, Yu, and Blunsom, 2015; Sordoni et al., 2015; Serban et al., 2017; Li, Monroe, and Jurafsky, 2016; Li et al., 2017) indicated that they were limited which made communicating…”
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues (2016, methodology): “This helps compare how well each model can integrate the dialogue context into its response, since it has previously been argued that for long contexts hierarchical RNNs models fare better (Sordoni et al. 2015a).”
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses (2017, methodology): “The hidden state allows for RNNs to better model sequential data, such as language.”
Hierarchical Multiscale Recurrent Neural Networks (2016, methodology): “However, considering the fact that non-stationarity is prevalent in temporal data, and that many entities of abstraction such as words and sentences are in variable length, we claim that it is important for an RNN to dynamically adapt its timescales to the particulars of the input entities of…”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, methodology): “…dialogue system architectures based on neural networks have shown promising results on several dialogue tasks (Ritter et al., 2011; Vinyals and Le, 2015; Lowe et al., 2015a; Sordoni et al., 2015b; Shang et al., 2015; Li et al., 2015; Serban et al., 2016; Serban et al., 2017b,a; Dodge et al., 2015).”
SampleRNN: An Unconditional End-to-End Neural Audio Generation Model (2016, background): “In contrast to WaveNets, we mitigate the problem of long-term dependency with hierarchical structure and using stateful RNNs, i.e. we will always propagate hidden states to the next training sequence although the gradient of the loss will not take into account the samples in previous training sequen”
Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning (2015, background): “[29] pro-pose a hierarchical recurrent neural network for context-aware query suggestion in a search engine.”
DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset (2017, methodology): “More recent work concentrates on generation-based approaches, which are mainly based on the sequence-to-sequence encoder-decoder architecture (Sordoni et al., 2015; Serban et al., 2016b).”
Attention with Intention for a Neural Network Conversation Model (2015, background): “Similarly as observed in [11], the model produces natural responses to user inputs.”
Learning Generic Sentence Representations Using Convolutional Neural Networks (2016, methodology): “our hierarchical CNN-LSTM model to extract meaningful sentence representations. However, our model can also be considered as a hierarchical language model for paragraphs. Similar work can be found in [20, 21, 22]. However, [20, 21] uses a LSTM for the sentence encoder, while [22] uses a bag-of-words”
Prefix-to-SQL: Text-to-SQL Generation from Incomplete User Questions (2021, background): “(3) Following query auto-completion works in information retrieval (Sordoni et al., 2015; Bhatia et al., 2011), we consider the question prefix to be ar X iv :2 10 9.”
Unsupervised Learning of Sentence Representations using Convolutional Neural Networks (2016, methodology): “For reference, the CNN-LSTM composite model was trained for roughly two weeks.”
Evaluation of session-based recommendation algorithms (2018, background): “2013] or distributional embeddings [Baeza-Yates et al. 2015; Djuric et al. 2014; Grbovic et al. 2015; Reddy et al. 2016; Tagami et al. 2015; Vasile et al. 2016; Zheleva et al. 2010].”
Dialogue Transformers (2019, background): “Recurrent Neural Networks A common choice in recent years has been to use a recurrent neural network (RNN) to process the sequence of previous dialogue turns, both for open domain [5, 6] and task-oriented systems [7].”
A Survey on Conversational Recommender Systems (2020, background): “e difficulties expressing their preferences. In the field of information retrieval, many approaches to query suggestion were proposed to assist users in expressing their information needs, see, e.g., [129] for a more recent example. Limited work however exists so far to apply such ideas to the conte”