Bridging the lexical chasm: statistical approaches to answer-finding

By Adam L. Berger, R. Caruana, David A. Cohn, Dayne Freitag, Vibhu Mittal. 2000

361 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Retrieval & Search

Citations in the corpus

Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “These approaches, based on near-exact matches between tokens of the queries and documents, suffer from the lexical gap and do not generalize well (Berger et al., 2000).”
Clickthrough-based translation models for web search: from word models to phrase models (2010, background): “where z is a vector of factors that mix to produce an observation [6].”
Finding similar questions in large question and answer archives (2005, methodology): “[2] is to use statistical techniques developed in information retrieval and natural language processing.”
Relevance-Based Language Models (2001, background): “Furthermore, explicit models of relevance are better suited to other information organization tasks, such as summarization [6], question answering [4], topic detection and tracking (TDT) [25, 12] and text segmentation [3].”
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (2021, background): “Retriever-based Lexical approaches suffer from the lexical gap [5].”
Using TF-IDF to Determine Word Relevance in Document Queries (2003, background): “…D, a word w, and an individual document d є D, we calculate w d = f w, d * log (|D|/f w, D ) (2), where f w, d equals the number of times w appears in d, |D| is the size of the corpus, and f w, D equals the number of documents in which w appears in D (Salton & Buckley, 1988, Berger, et al, 2000).”
Semantic Matching in Search (2014, methodology): “[21] apply machine translation model to the task of finding answers to questions in question answering.”
Learning strategies for story comprehension: a reinforcement learning approach (2005, background): “For example, [Berger et al., 2000] induces questions-answer correspondence from FAQs and call center data.”
Towards Unsupervised Dense Information Retrieval with Contrastive Learning (2021, background): “These approaches, based on exact matches between tokens of the queries and documents, suffer from the lexical gap and do not generalize well (Berger et al., 2000).”
Design and realization of a modular architecture for textual entailment (2013, background): “At the same time, T and H1 have a relatively small lexical overlap, a phenomenon named the ‘lexical chasm’ (Berger et al. 2000).”
Query Drift Compensation: Enabling Compatibility in Continual Learning of Retrieval Embedding Models (2025, unknown): “…language models, the use of neural retrieval models (Karpukhin et al., 2020; Liang et al., 2020; Khattab & Zaharia, 2020; Luan et al., 2021; Muennighoff, 2022) has become more common than traditional lexical approaches (Robertson et al., 2009), which have the lexical gap (Berger et al., 2000).”