An Information Retrieval Approach to Short Text Conversation

By Zongcheng Ji, Zhengdong Lu, Hang Li. 2014

260 citations recorded by the source corpus.

Human computer conversation is regarded as one of the most difficult problems in artificial intelligence. In this paper, we address one of its key sub-problems, referred to as short text conversation, in which given a message from human, the computer returns a reasonable response to the message. We leverage the vast amount of short conversation data available on social media to study the issue. We propose formalizing short text conversation as a search problem at the first step, and employing state-of-the-art information retrieval (IR) techniques to carry out the task. We investigate the significance as well as the limitation of the IR approach. Our experiments demonstrate that the retrieval-based model can make the system behave rather "intelligently", when combined with a huge repository of conversation data from social media.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Retrieve and Refine: Improved Sequence Generation Models For Dialogue (2018, background): “Retrieval models (Ji et al., 2014) do not have this problem, but instead either produce engaging responses or else completely erroneous ones which they cannot Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI 978-1-948087-75-9 tune to…”
Retrieve, Rerank and Rewrite: Soft Template Based Neural Summarization (2018, methodology): “Inspired by retrieve-based conversation systems (Ji et al., 2014), we assume the golden summaries of the similar sentences can provide a reference point to guide the input sentence summarization process.”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, background): “[54] for response selection from a large-scale post-response repository.”
A Survey on Dialogue Systems: Recent Advances and New Frontiers (2017, background): “Generative models are able to generate more proper responses that could have never appeared in the corpus, while retrieval-based models enjoy the advantage of informative and fluent responses[23], because they select a proper response for the current conversation from a repository with response sele”
Sequential Match Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots (2016, background): “…selects a proper response from an index based on matching be-tween the response and an input message with or without context (Hu et al., 2014; Ji et al., 2014; Wang et al., 2015; Yan et al., 2016; Wu et al., 2016; Zhou et al., 2016), while the latter employs statistical machine translation…”
Natural Language Processing Advancements By Deep Learning: A Survey (2020, methodology): “In general, this problem has been formulated as a search problem and uses IR techniques for task completion [171].”
PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval (2021, background): “…distance ( i.e., dissimilarity) for visualization: the longer the distance is, the less similar it is. et al., 2020b), information retrieval (Luan et al., 2021; Khattab and Zaharia, 2020), dialogue (Ji et al., 2014; Henderson et al., 2017) and entity linking (Gillick et al., 2019; Wu et al., 2020).”
Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems (2018, background): “tracted significant attention in the information retrieval (IR) community. Existing approaches to building conversational systems include generation-based methods [23, 26] and retrieval-based methods [9, 40, 42]. Compared with generation-based methods, retrieval-based methods have the advantages of ”
A Deep Look into Neural Ranking Models for Information Retrieval (2019, methodology): “tion answering, task completion, and social chat (i.e., chit-chat) [71]. In general, AC could be formulated either as an IR problem that aims to rank/select a proper response from a dialog repository [72] or a generation problem that aims to generate an appropriate response with respect to the input”
Mechanism-Aware Neural Machine for Dialogue Response Generation (2017, methodology): “Previous rule-based (Williams and Young 2007; Misu et al. 2012; Young et al. 2013) and retrieval-based (Ji, Lu, and Li 2014) conversational models requires manual efforts in rule developing and feature engineering, or can only response the posts in pre-existing cases, thus are difﬁcult to be…”
Skeleton-to-Response: Dialogue Generation Guided by Retrieval Memory (2018, methodology): “The IR-based models (Ji, Lu, and Li 2014; Hu et al. 2014) directly copy an existing response from a training corpus when receiving a response request.”
Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network (2018, methodology): “Most modern chatbots are data-driven, either in a fashion of information-retrieval (Ji et al., 2014; Banchs and Li, 2012; Nio et al., 2014; Ameixa et al., 2014) or sequence-generation (Ritter et al., 2011).”
Multi-view Response Selection for Human-Computer Conversation (2016, background): “Responding to this challenge, two promising solutions have been proposed: 1) retrieval-based model which selects a response from a large corpus (Ji et al., 2014; Yan et al., 2016; Yan et al., ).”
RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking (2021, background): “Such a two-stage procedure is particularly useful in a variety of natural language processing tasks, including question answering (Mao et al., 2021; Xiong et al., 2020b), dialogue system (Ji et al., 2014; Henderson et al., 2017) and entity linking (Gillick et al., 2019; Wu et al., 2020).”
Retrieval-guided Dialogue Response Generation via a Matching-to-Generation Framework (2019, methodology): “This problem is avoided in traditional retrieval systems (Ji et al., 2014; Hu et al., 2014) by preceding the selection of informative and engaging responses.”
Enhance word representation for out-of-vocabulary on Ubuntu dialogue corpus (2018, methodology): “Existing methods can be divided into two categories: retrieval-based methods (Wang et al., 2013; Ji et al., 2014; Yan et al., 2016b) and generation based methods (Vinyals & Le, 2015).”