On the Effect of Low-Frequency Terms on Neural-IR Models

By Sebastian Hofstätter, Navid Rekabsaz, Carsten Eickhoff, Allan Hanbury. 2019

41 citations recorded by the source corpus.

Low-frequency terms are a recurring challenge for information retrieval models, especially neural IR frameworks struggle with adequately capturing infrequently observed words. While these terms are often removed from neural models - mainly as a concession to efficiency demands - they traditionally play an important role in the performance of IR models. In this paper, we analyze the effects of low-frequency terms on the performance and robustness of neural IR models. We conduct controlled experiments on three recent neural IR models, trained on a large-scale passage retrieval collection. We evaluate the neural IR models with various vocabulary sizes for their respective word embeddings, considering different levels of constraints on the available GPU memory. We observe that despite the significant benefits of using larger vocabularies, the performance gap between the vocabularies can be, to a great extent, mitigated by extensive tuning of a related parameter: the number of documents to re-rank. We further investigate the use of subword-token embedding models, and in particular FastText, for neural IR models. Our experiments show that using FastText brings slight improvements to the overall performance of the neural IR models in comparison to models trained on the full vocabulary, while the improvement becomes much more pronounced for queries containing low-frequency terms.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

Citations in the corpus

ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (2020, neutral): “In this work, we compare against KNRM [5, 36], Duet [21, 23], ConvKNRM [5], and fastText+ConvKNRM [11].”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “Figure 18: The effectiveness/efficiency tradeoff of the TK model compared to BERT and other preBERT interaction-based neural ranking models on the MS MARCO passage retrieval test collection, taken from Hofstätter et al. [2020].”
Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval (2019, methodology): “FastText+Conv-KNRM [14] was the best non-BERT model on the leaderboard.”
Interpretable & Time-Budget-Constrained Contextualization for Re-Ranking (2020, background): “A key success factor are fine-tuned word representations, covering most of the indexed vocabulary [13].”
Local Self-Attention over Long Text for Efficient Document Retrieval (2020, unknown): “Neural models have shown successful results in a number of IR tasks [9, 10, 17].”
Selective Weak Supervision for Neural Information Retrieval (2020, methodology): “Conv-KNRM, which captures soft matches between query-document n-grams using kernels [5], significantly outperforms feature-based methods in the Chinese Sogou-T search log [23, 53, 54], the Bing click log [5], and MS MARCO which includes millions of expert relevance label [17, 37].”
Let's measure run time! Extending the IR replicability infrastructure to include performance aspects (2019, methodology): “We keep the defaults for the model configurations from their respective papers, except for MatchPyramid where we follow the 5-layer configuration from [6].”
A Modern Perspective on Query Likelihood with Deep Generative Retrieval Models (2021, unknown): “In fact, as observed in previous studies [13], OOVs correspond to infrequent words that – due to their rarity – contain crucial information for retrieval.”
From doc2query to docTTTTTquery (2019, methodology): “[5], which is, from what we can tell, the best non-ensemble, non-BERT method from the leaderboard with an associated paper; and DeepCT [2], a recently-introduced BERT-based document expansion method.”
TU Wien @ TREC Deep Learning '19 - Simple Contextualization for Re-ranking (2019, background): “We assume this is due to the ability of the contextualization to overcome OOV and infrequent terms, which have been shown to negatively impact simpler neural models [5].”