Modeling Interestingness with Deep Neural Networks

By Jianfeng Gao, Patrick Pantel, Michael Gamon, Xiaodong He, Li Deng. 2014

180 citations recorded by the source corpus.

This paper presents a deep semantic similarity model (DSSM), a special type of deep neural networks designed for text analysis, for recommending target documents to be of interest to a user based on a source document that she is reading. We observe, identify, and detect naturally occurring signals of interestingness in click transitions on the Web between source and target documents, which we collect from commercial Web browser logs. The DSSM is trained on millions of Web transitions, and maps source-target document pairs to feature vectors in a latent space in such a way that the distance between source documents and their corresponding interesting targets in that space is minimized. The effectiveness of the DSSM is demonstrated using two interestingness tasks: automatic highlighting and contextual entity search. The results on large-scale, real-world datasets show that the semantics of documents are important for modeling interest-ingness and that the DSSM leads to significant quality improvement on both tasks, outperforming not only the classic document models that do not use semantics but also state-of-the-art topic models.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval (2015, methodology): “…multi-task framework is extensible to other meth-ods for sentence/document representations, such as those based on convolutional networks (Kalchbren-ner et al., 2014; Shen et al., 2014; Gao et al., 2014b), parse tree structure (Irsoy and Cardie, 2014), and run-time inference (Le and Mikolov, 2014).”
A Deep Relevance Matching Model for Ad-hoc Retrieval (2016, methodology): “We point out three major diﬀerences between these two matching problems which may lead to signiﬁc-antly diﬀerent architecture design for the deep matching models.”
Towards Coherent and Cohesive Long-form Text Generation (2018, methodology): “Our design of these two discriminators is inspired by the Deep Structured Semantic Model (DSSM) which was originally developed to measure the semantic similarity between two texts (Huang et al., 2013; Gao et al., 2014; Palangi et al., 2016; Xu et al., 2018).”
A Neural Network Approach to Context-Sensitive Generation of Conversational Responses (2015, methodology): “Continuous representations of words and phrases estimated by neural network models have been applied on a variety of tasks ranging from Information Retrieval (IR) (Huang et al., 2013; Shen et al., 2014), Online Recommendation (Gao et al., 2014b), Machine Translation (MT) (Auli et al., 2013; Cho et a”
Neural Approaches to Conversational AI (2018, neutral): “, 2014), a document pair in recommendation (Gao et al., 2014b), a question-answer pair in QA (Yih et al.”
Hierarchical Attention Networks for Document Classification (2016, background): “Attention serves two beneﬁts: not only does it often result in better performance, but it also provides insight into which words and sentences contribute to the classiﬁcation decision which can be of value in applications and analysis (Shen et al., 2014; Gao et al., 2014).”
The Design and Implementation of XiaoIce, an Empathetic Social Chatbot (2018, background): “By contrast, in the research community, there is a growing interest in developing fully data-driven, end-to-end (E2E) systems for social chatbot (chitchat) scenarios, as reviewed in Chapter 5 of Gao et al. [2019]. The difference is mainly due to different design goals of social chatbots. Traditional”
Effective Use of Word Order for Text Categorization with Convolutional Neural Networks (2014, methodology): “…on token-level applications (e.g., POS tagging) by Collobert et al. (2011), CNN has been used in systems for entity search, sentence modeling, word embedding learning, product feature mining, and so on (Xu and Sarikaya, 2013; Gao et al., 2014; Shen et al., 2014; Kalchbrenner et a r X i v : 1412 .”
Convolutional Neural Network for Paraphrase Identification (2015, background): “(B1) “Mary gave birth to a son in 2000.””
Semantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base (2015, methodology): “In this work, we adapt a convolutional neural network (CNN) framework (Shen et al., 2014b; Shen et al., 2014a; Gao et al., 2014) to this matching problem.”
Embedding Entities and Relations for Learning and Inference in Knowledge Bases (2014, methodology): “This general framework for relationship modeling also applies to the recent deep-structured semantic model (Huang et al., 2013; Shen et al., 2014a;b; Gao et al., 2014; Yih et al., 2014), which learns the relevance or a single relation between a pair of word sequences.”
Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval (2015, positive): “This work has been motivated by the earlier successes of deep learning methods in speech [41]–[45] and in semantic modelling [3], [10], [46], [47], and it adds further evidence for the effectiveness of these methods.”
A Primer on Neural Network Models for Natural Language Processing (2015, background): “Gao et al (Gao et al., 2014) suggest using as core features not only the word form itself but also a unique feature (hence a unique embedding vector) for each of the letter-trigrams in the word.”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, methodology): “DSSM is further augmented by introducing Convolutional layers [34, 49, 122, 125] and recurrent layers with Long Short-Term Memory (LSTM) units [95].”
Learning to Match using Local and Distributed Representations of Text for Web Search (2016, background): “Outside of document ranking, learning text representations for the target task has been explored in the context of other IR scenarios, including query classification [21], query auto-completion [26], next query prediction [25, 38], and entity extraction [9].”
Semi-supervised Question Retrieval with Gated Convolutions (2015, methodology): “, 1998) have also been successfully applied to various NLP tasks (Kalchbrenner et al., 2014; Kim, 2014; Kim et al., 2015; Zhang et al., 2015; Gao et al., 2014).”
Learning Bidirectional Intent Embeddings by Convolutional Deep Structured Semantic Models for Spoken Language Understanding (2015, background): “The convolutional and max-pooling layers are able to capture prominent words of the word sequences [15, 18].”
Applying deep learning to answer selection: A study and an open task (2015, background): “[15] proposed a CNN based network which maps source-target document pairs to embedding vectors such that the distance between source documents and their corresponding interesting targets is minimized.”