SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval

By Yang Bai, Xiaoguang Li, Gang Wang, Chaoliang Zhang, Lifeng Shang, Jun Xu, Zhaowei Wang, Fangshan Wang, Qun Liu. 2020

123 citations recorded by the source corpus.

Term-based sparse representations dominate the first-stage text retrieval in industrial applications, due to its advantage in efficiency, interpretability, and exact term matching. In this paper, we study the problem of transferring the deep knowledge of the pre-trained language model (PLM) to Term-based Sparse representations, aiming to improve the representation capacity of bag-of-words(BoW) method for semantic-level matching, while still keeping its advantages. Specifically, we propose a novel framework SparTerm to directly learn sparse text representations in the full vocabulary space. The proposed SparTerm comprises an importance predictor to predict the importance for each term in the vocabulary, and a gating controller to control the term activation. These two modules cooperatively ensure the sparsity and flexibility of the final text representation, which unifies the term-weighting and expansion in the same framework. Evaluated on MSMARCO dataset, SparTerm significantly outperforms traditional sparse methods and achieves state of the art ranking performance among all the PLM-based sparse models.

Subfields: Artificial Intelligence, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

Citations in the corpus

Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, background): “…on term-weighting schemes, with many recent works proposing sophisticated, contextualized weights for both queries, and documents(Dai and Callan, 2019; Gao et al., 2021; Lin and Ma, 2021; Mallia et al., 2021; Dai and Callan, 2020; Bai et al., 2020; Zhao et al., 2021; Formal et al., 2021b,a).”
SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking (2021, methodology): “We compare to the following sparse approaches (1) BM25 (2) DeepCT [4] (3) doc2query-T5 [18] (4) and SparTerm [1], as well as state-of-the-art dense approaches ANCE [25] and TCT-ColBERT [15].”
Learning Passage Impacts for Inverted Indexes (2021, methodology): “As another way of expanding documents, the very recent SparTerm [1] method predicts an importance score for every term in the vocabulary and uses a gating mechanism to only keep a sparse subset of those, using them to learn an end-to-end score for relevant and non-relevant documents.”
SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval (2021, methodology): “However, EPIC and SPARTA (document) representations are not sparse enough by construction – unless resorting on top-𝑘 pooling – contrary to SparTerm, for which fast retrieval is thus possible.”
Answer Complex Questions: Path Ranker Is All You Need (2021, background): “The experimental results in two modes have shown that EPR can enhance top-1 (best path for answer extraction) PEM by at least 6% so that it can improve the order of the best path to a higher level.”
LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum (2026, unknown): “The performance of LACONIC stems from the integration of powerful”
Granite Embedding Models (2025, unknown): “…could be useful for exact matching of terms and could be efficiently ingested as an inverted index as shown by Zamani et Sparse models are popularly trained to predict the importance of each vocabulary term based on the logits of masked-language model head (Bai et al., 2020; Formal et al., 2021b).”
A Survey of Model Architectures in Information Retrieval (2025, unknown): “At a higher level, LSR can be viewed as a way to learn token importance or “impact” scores from data (Dai and Callan, 2019b; Bai et al., 2020; Mallia et al., 2021), in contrast to static formulas like BM25.”