A Language Modeling Approach to Information Retrieval

By Jay M. Ponte, W. Bruce Croft. 1998

In today's world, there is no shortage of information. However, for a specific information need, only a small subset of all of the available information will be useful. The field of information retrieval (IR) is the study of methods to provide users with that small subset of information relevant to their needs and to do so in a timely fashion. Information sources can take many forms, but this thesis will focus on text based information systems and investigate problems germane to the retrieval of written natural language documents. Central to these problems is the notion of "topic." In other words, what are documents about? However, topics depend on the semantics of documents and retrieval systems are not endowed with knowledge of the semantics of natural language. The approach taken in this thesis will be to make use of probabilistic language models to investigate text based information retrieval and related problems. One such problem is the prediction of topic shifts in text, the topic segmentation problem. It will be shown that probabilistic methods can be used to predict topic changes in the context of the task of new event detection. Two complementary sets of features are studied individually and then combined into a single language model. The language modeling approach allows this problem to be approached in a principled way without complex semantic modeling. Next, the problem of document retrieval in response to a user query will be investigated. Models of document indexing and document retrieval have been extensively studied over the past three decades. The integration of these two classes of models has been the goal of several researchers but it is a very difficult problem. Much of the reason for this is that the indexing component requires inferences as to the semantics of documents. Instead, an approach to retrieval based on probabilistic language modeling will be presented. Models are estimated for each document individually. The approach to modeling is non-parametric and integrates the entire retrieval process into a single model. One advantage of this approach is that collection statistics, which are used heuristically for the assignment of concept probabilities in other probabilistic models, are used directly in the estimation of language model probabilities in this approach. The language modeling approach has been implemented and tested empirically and performs very well on standard test collections and query sets. In order to improve retrieval effectiveness, IR systems use additional techniques such as relevance feedback, unsupervised query expansion and structured queries. These and other techniques are discussed in terms of the language modeling approach and empirical results are given for several of the techniques developed. These results provide further proof of concept for the use of language models for retrieval tasks.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, background): “For example, the quality of the core task in the natural language processing domain, language modeling [35, 4], was significantly improved by using Highway layers [21, 48] or tying input and output embeddings [36].”
Relevance-based Word Embedding (2017, methodology): “To estimate the relevance model distributions in the training set, we considered top 10 retrieved documents in a target collection in response to each query using the Galago 3 implementation of the query likelihood retrieval model [29] with Dirichlet prior smoothing ( µ = 1500) [45].”
Neural information retrieval: at the end of the early years (2017, background): “…use in language modeling for automatic speech recognition (ASR) and machine translation (MT) might be loosely related to language modeling for IR (Ponte and Croft 1998), state-of-the-art performance provided by neural language models trained on vast amounts of data could not be readily applied…”
Generalizing Translation Models in the Probabilistic Relevance Framework (2016, background): “Nevertheless, the “basic” models based on the Probabilistic Retrieval (PR) Framework [30], and language modelling [27] have maintained a respectable command of the research and practice of IR.”
Improving Passage Retrieval with Zero-Shot Question Generation (2022, background): “As language models continue to improve rapidly (Rae et al., 2021; Chowdhery et al., 2022), the performance of UPR may see corresponding gains over time.”
Latent entity space: a novel retrieval approach for entity-bearing queries (2015, background): “This is known as the language modeling (LM) approach (Ponte and Croft 1998). n ( w , q ) denotes the number of occurrences of w in q .”
Connecting the Dots: Event Graph Schema Induction with Path Language Modeling (2020, unknown): “For example, in Figure 1 We train the path language model on two tasks: learning an auto-regressive language model (Ponte and Croft, 1998; Dai and Le, 2015; Peters et al., 2018; Rad-ford et al.; Yang et al., 2019) to predict an edge or a node, given previous edges and nodes in a path, and a…”
Leveraging Passage-level Cumulative Gain for Document Ranking (2020, background): “BM25 [34] and language models [32, 42]) or supervised (e.”
Towards Conversational Search and Recommendation: System Ask, User Respond (2018, methodology): “We take the following representative product search or recommendation methods as baselines: • QL : The query likelihood model by Ponte and Croft [25], which is a language modeling approach by ranking documents with the log-likelihood of the query in the document’s language models.”
Information Retrieval: A Biomedical and Health Perspective (2020, background): “One researcher found that the best reviews came from faculty of junior academic status [176], while another showed the best reviews came from younger faculty working at top academic institutions or who were known to the editors [177].”
Pseudo-Relevance Feedback Based on Matrix Factorization (2016, methodology): “In this subsection, we explain how to choose a proper weight function and how to update the query model for the language modeling framework [28].”
Word Embedding based Generalized Language Model for Information Retrieval (2015, methodology): “These posterior probabilities are estimated for each document d during indexing time with the help of the prior probability (P (q|d)) according to the Bayes rule [8, 6, 10].”
Applications of Topic Models (2017, methodology): “The document language models in information retrieval [Ponte and Croft, 1998] attempt to model the query generation process based on the document models.”
A Study of Neural Matching Models for Cross-lingual IR (2020, methodology): “TbT-QT takes only the top-1 translation of a query term and uses the query likelihood model [11] for retrieval.”
Vital nodes identification in complex networks (2016, methodology): “The initial scores R i and F (0) ↵ are calculated using a text relevance function f , such as the vector space model [307] and the statistical language model [308,309].”