A statistical interpretation of term specificity and its application in retrieval
By Karen Spärck Jones. 2021
5,376 citations recorded by the source corpus.
The exhaustivity of document descriptions and the specificity of index terms are usually regarded as independent. It is suggested that specificity should be interpreted statistically, as a function of term use rather than of term meaning. The effects on retrieval of variations in term specificity are examined, experiments with three test collections showing in particular that frequently‐occurring terms are required for good overall performance. It is argued that terms should be weighted according to collection frequency, so that matches on less frequent, more specific, terms are of greater value than matches on frequent terms. Results for the test collections show that considerable improvements in performance are obtained with this very simple procedure.
Subfields: Artificial Intelligence, Computer Networks and Communications, Information Retrieval & Search
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Graham Neubig
- Yejin Choi
- Yoshua Bengio
- Noah A. Smith
In Computer Networks and Communications
- Grgoire Montavon
- Zachary Chase Lipton
- Jaime Teevan
- K. Müller
- Michael J. Cafarella
- Satish Narayanasamy
- Chun-Hung Hsiao
- Yuting Chen
In Information Retrieval & Search
- Sebastian Riedel
- Gautier Izacard
- Edouard Grave
- Wen-tau Yih
- Fabio Petroni
- Jamie Callan
- Patrick Lewis
- Zhiyuan Liu
Citations in the corpus
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “Historically, this information retrieval step was implemented using term-matching methods, such as TF-IDF or BM25 (Jones, 1972; Robertson et al., 1995).”
- Character-level Convolutional Networks for Text Classification (2015, methodology): “For the TFIDF (term-frequency inverse-document-frequency) [14] version, we use the counts as the term-frequency.”
- Distilling Knowledge from Reader to Retriever for Question Answering (2020, methodology): “Traditionally, information retrieval systems were based on hand-crafted sparse representations of text documents, such as TF-IDF or BM25 (Jones, 1972; Robertson et al., 1995).”
- Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “One of the most used weighing scheme is known as TF-IDF, and is based on inverse document frequency, or term specificity (Jones, 1972).”
- A Memory Efficient Baseline for Open Domain Question Answering (2020, methodology): “Traditional information retrieval is based on sparse vector representations, with hand crafted term weighting (Jones, 1972; Robertson et al., 1995).”
- Sparse, Dense, and Attentional Representations for Text Retrieval (2020, methodology): “Extensions to TF-IDF (Spärck Jones, 1972) and BM25 (Robertson et al., 2009) are discussed in Section 3.1.”
- From Frequency to Meaning: Vector Space Models of Semantics (2010, background): “The most popular way to formalize this idea for term–document matrices is the tf-idf (term frequency × inverse document frequency) family of weighting functions (Spärck Jones, 1972).”
- LexRank: Graph-based Lexical Centrality as Salience in Text Summarization (2004, methodology): “A commonly used measure to assess the importance of the words in a sentence is the inverse document frequency, or idf, which is dened by the formula ( Sparck-Jones, 1972 ): idfi = log N ni (1)”
- "What is relevant in a text document?": An interpretable machine learning approach (2016, methodology): “For several text applications, machine learning (ML) models based on global word statistics like TFIDF [1,2] or linear classifiers are known to perform remarkably well, e.g. for unsupervised keyword extraction [3] or document classification [4].”
- UNAL-NLP: Combining Soft Cardinality Features for Semantic Textual Similarity, Relatedness and Entailment (2014, methodology): “Then each word was reduced to its stem using the Porter’s algorithm (Porter, 1980) and a idf weight (Jones, 2004) was associated to each stem (wai weights in eq.”
- BARTScore: Evaluating Generated Text as Text Generation (2021, methodology): “Reference Length We break down each test set into four buckets based on the reference length, which are [15 , 25) , [25 , 35) , [35 , 45) , [45 , 54] and compute the Kendall’s Tau average correlation of different metrics across all language pairs within each bucket.”
- Tool learning with large language models: a survey (2024, neutral): “If the quantity of tools is limited or upon receiving the tools retrieved during the tool retrieval phase, the LLMs need to select the appropriate tools based on the tool descriptions and the sub-question, which is known as LLM-based tool selection.”
- Zero-training Sentence Embedding via Orthogonal Basis (2018, background): “Similar to the idea of inverse document frequency (IDF) (Sparck Jones, 1972), a word that is commonly present in the corpus is likely to be a stop word, thus its corpus-wise uniqueness is small.”