Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval

By Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, Rabab Kreidieh Ward. 2015

861 citations recorded by the source corpus.

This paper develops a model that addresses sentence embedding, a hot topic in current natural language processing research, using recurrent neural networks (RNN) with Long Short-Term Memory (LSTM) cells. The proposed LSTM-RNN model sequentially takes each word in a sentence, extracts its information, and embeds it into a semantic vector. Due to its ability to capture long term memory, the LSTM-RNN accumulates increasingly richer information as it goes through the sentence, and when it reaches the last word, the hidden layer of the network provides a semantic representation of the whole sentence. In this paper, the LSTM-RNN is trained in a weakly supervised manner on user click-through data logged by a commercial web search engine. Visualization and analysis are performed to understand how the embedding process works. The model is found to automatically attenuate the unimportant words and detect the salient keywords in the sentence. Furthermore, these detected keywords are found to automatically activate different cells of the LSTM-RNN, where words belonging to a similar topic activate the same cell. As a semantic representation of the sentence, the embedding vector can be used in many different applications. These automatic keyword detection and topic allocation abilities enabled by the LSTM-RNN allow the network to perform document retrieval, a difficult language processing task, where the similarity between the query and documents can be measured by the distance between their corresponding sentence embedding vectors computed by the LSTM-RNN. On a web search task, the LSTM-RNN embedding is shown to significantly outperform several existing state of the art methods. We emphasize that the proposed model generates sentence embedding vectors that are specially useful for web document retrieval tasks. A comparison with a well known general sentence embedding method, the Paragraph Vector, is performed. The results show that the proposed method in this paper significantly outperforms Paragraph Vector method for web document retrieval task.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

A Structured Self-attentive Sentence Embedding (2017, background): “…networks (Kalchbren-ner et al., 2014; dos Santos & Gatti, 2014; Kim, 2014) as an intermediate step in creating sentence representations to solve a wide variety of tasks including classiﬁcation and ranking (Yin & Sch¨utze, 2015; Palangi et al., 2016; Tan et al., 2016; Feng et al., 2015).”
Distilling Knowledge from Reader to Retriever for Question Answering (2020, methodology): “This technique was later extended by using convolutional neural networks (Shen et al., 2014) and recurrent neural networks (Palangi et al., 2016).”
Unsupervised Dense Information Retrieval with Contrastive Learning (2021, neutral): “, 2014) or recurrent neural network (Palangi et al., 2016).”
Towards Coherent and Cohesive Long-form Text Generation (2018, methodology): “Our design of these two discriminators is inspired by the Deep Structured Semantic Model (DSSM) which was originally developed to measure the semantic similarity between two texts (Huang et al., 2013; Gao et al., 2014; Palangi et al., 2016; Xu et al., 2018).”
A deep learning framework for financial time series using stacked autoencoders and long-short term memory (2017, methodology): “Currently, there is no rule of thumb to select the number of delays and hidden layers [21, 22].”
End-to-End Retrieval in Continuous Space (2018, neutral): “However, models for learning text representations have found application in IR by re-ranking the top candidates proposed by a discrete retrieval system (Huang et al., 2013; Shen et al., 2014; Palangi et al., 2016; Dos Santos et al., 2015; Lei et al., 2016).”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, methodology): “y and document independently before computing their relevance. DSSM is further augmented by introducing Convolutional layers [27, 78, 81] and recurrent layers with Long Short-Term Memory (LSTM) units [62]. To effectively incorporate dialog history, Yan et al. [110] reformulated input query X, and co”
Multimodal Intelligence: Representation Learning, Information Fusion, and Applications (2019, background): “neural network structures in pseudo-Siamese network settings [65]–[67], [78]–[81].”
Question-Answering with Grammatically-Interpretable Representations (2017, methodology): “…layer using GloVe vectors (Pennington, Socher, and Manning 2014), a phrase embedding layer using bidirectional LSTMs for sentence embedding (Palangi et al. 2016), an attention flow layer using a special attention mechanism, a modeling layer using LSTMs, and an output layer that generates…”
Promptagator: Few-shot Dense Retrieval From 8 Examples (2022, background): “Representation based models (Palangi et al., 2016; Gillick et al., 2018; Karpukhin et al., 2020) encode a query and passage independently into a common dense space, and scores their relevance based on vector dot-product or cosine similarity.”
Tracing Fake-News Footprints: Characterizing Social Media Messages by How They Propagate (2018, methodology): “RNNs have been successfully applied in a myriad of domains for modeling sequential data [10], such as information retrieval [28], sentiment analysis [33] and machine translation [6].”
How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence (2020, methodology): “(3) Semantic matching models in sentence level, ABCNN (Yin et al., 2016), and document level, SMASH-RNN (Jiang et al., 2019).”
Deep Learning of Grammatically-Interpretable Representations Through Question-Answering (2017, methodology): “BiDAF is constructed from 6 layers: a character embedding layer using CNNs, a word embedding layer using GLOVE [9] vectors, a phrase embedding layer using bidirectional LSTMs for sentence embedding [10], an attention flow layer using a special attention mechanism, a modeling layer using LSTMs, and a”
Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey (2019, background): “With the development of high computational devices, deep neural networks, in recent years have gained tremendous attention in many Artificial Intelligence (AI) communities such as Computer Vision [65, 127], Natural Language Processing [16, 66], Web Mining [102, 150] and Game theory [120].”
Seq2Slate: Re-ranking and Slate Optimization with RNNs (2018, methodology): “…in recent years deep models have been used for information retrieval, focusing on embedding queries, documents and query-document pairs [Huang et al., 2013, Guo et al., 2016, Palangi et al., 2016, Wang and Klabjan, 2017, Pang et al., 2017] (see also recent survey by Mitra and Craswell [2017]).”
Neural information retrieval: at the end of the early years (2017, methodology): “2 Document ranking Learn Learn to match [79, 89, 114, 152, 156, 157, 175, 176, 207] Learn to predict [1, 58] Learn to generate [116]”