Understanding inverse document frequency: on theoretical arguments for IDF

By Stephen E. Robertson. 2004

1,682 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Learning to Match using Local and Distributed Representations of Text for Web Search (2016, background): “red, so when ranking for the query ‘Australia’ then only the term frequency of ‘Australia’ is considered, even though counting terms like ‘Sydney’ and ‘koala’ can be good positive evidence. Robertson [33] pointed out that under the probabilistic model of IR there is, in fact, no good justiﬁcation fo”
Non-Parametric Adaptation for Neural Machine Translation (2019, methodology): “Our baseline strategy relies on a sentence level similarity score, similar to those used for standard information retrieval tasks (Robertson, 2004).”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, background): “However, CIDEr achieves human consensus using term frequency-inverse document frequency (TF-IDF) [121].”
A Survey of App Store Analysis for Software Engineering (2017, methodology): “They trained the classifier using a support vector machine for each of 50 classes, and used the BM25 weighting scheme [190] on the features.”
A visual analytics approach to understanding cycling behaviour (2012, neutral): “Travelling overhead we hear the mechanical whirr of the rebel ship and a series of directional laser bolts. theatres and it had a profound impact on both aesthetic practices and exhibition technologies in the years immediately following (Kerins, 2011, p. 32).”
IDTAXA: a novel approach for accurate taxonomic classification of microbiome sequences (2018, methodology): “We employ a text mining approach to weigh k-mer matches based on their inverse document frequency (IDF) [22, 23].”
Random-Walk Term Weighting for Improved Text Classification (2007, background): “idf has been one of the most extensively studied weighting schemes [21, 17].”
Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learning (2018, positive): “Our first baseline, Retrieval, builds on the BM25 algorithm [48], which is the state-of-the-art in ranking models employing term-matching.”
Analysis of the Paragraph Vector Model for Information Retrieval (2016, neutral): “is usually considered to be a better scheme for term weighting than TF-ICF [20].”
Identifying Opportunities for Valuable Encounters: Toward Context-Aware Social Matching Systems (2015, methodology): “The idea is related to earlier discussed concepts that have been applied in data mining and by traditional recommender systems, such as the TF-IDF measure [Robertson 2004; Salton and McGill 1986] and Inverse User Frequency Transformation [Symeonidis et al. 2007] to identify relevant information.”
CIDEr: Consensus-based image description evaluation (2014, methodology): “To encode this intuition, we perform a Term Frequency Inverse Document Frequency (TF-IDF) weighting for each n-gram [36].”
Balanced Mini-batch Sampling for SGD Using Determinantal Point Processes (2017, methodology): “To measure similarities between documents, we represent each document with a vector x of the tf-idf [37] scores of each word in the document.”
Comparing BERT against traditional machine learning text classification (2020, background): “On the other hand, given a term t i , we denote by tf i the frequency of the term t i in the document under consideration [20].”
An Updated Duet Model for Passage Re-ranking (2019, methodology): “In traditional IR models, the inverse document frequency (IDF) [Robertson, 2004] of a query term provides an effective mechanism for weighting the query terms by their discriminative power.”