Learning Dense Representations of Phrases at Scale

By Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, Danqi Chen. 2020

130 citations recorded by the source corpus.

Open-domain question answering can be reformulated as a phrase retrieval problem, without the need for processing documents on-demand during inference (Seo et al., 2019). However, current phrase retrieval models heavily depend on sparse representations and still underperform retriever-reader approaches. In this work, we show for the first time that we can learn dense representations of phrases alone that achieve much stronger performance in open-domain QA. We present an effective method to learn phrase representations from the supervision of reading comprehension tasks, coupled with novel negative sampling methods. We also propose a query-side fine-tuning strategy, which can support transfer learning and reduce the discrepancy between training and inference. On five popular open-domain QA datasets, our model DensePhrases improves over previous phrase retrieval models by 15%-25% absolute accuracy and matches the performance of state-of-the-art retriever-reader models. Our model is easy to parallelize due to pure dense representations and processes more than 10 questions per second on CPUs. Finally, we directly use our pre-indexed dense phrase representations for two slot filling tasks, showing the promise of utilizing DensePhrases as a dense knowledge base for downstream tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Few-shot Learning with Retrieval Augmented Language Models (2022, background): “Guu et al. (2020) also explore a query-side ﬁnetuning at ﬁnetuning time to avoid index refreshes, which is also explored in the context of phrase-based retrieval by Lee et al. (2021b).”
Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, background): “However, even on Wikipedia-based benchmarks, titles on their own are not well-suited for retrieval at passage-level, given they can only identify an article (that might contain several passages).”
Learning to Retrieve Passages without Supervision (2021, background): “Lee et al. (2021a) demonstrated strong end-to-end ODQA results with this approach, and Lee et al. (2021b) showed that it is also effective for passage retrieval.”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, background): “In NLP, a similar contrastive learning objective has been also explored in different contexts (Henderson et al., 2017; Gillick et al., 2019; Karpukhin et al., 2020; Lee et al., 2020).”
Question Answering Infused Pre-training of General-Purpose Contextualized Representations (2021, methodology): “Especially related is Lee et al. (2021), which also uses question generation and a crossencoder teacher model to improve phrase-indexed QA.”
Retrieval-Augmented Generation for Large Language Models: A Survey (2023, background): “Retrieval units vary from tokens (e.g., kNN-LM [ Khandelwal et al. , 2019 ] ) to phrases (e.g., NPM, COG [ Lee et al. , 2020, Lan et al. , 2022 ] ) and document paragraphs, with finer granularities offering precision at the cost of increased retrieval complexity.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “DensePhrases learns dense phrase representations by data augmentation and knowledge distillation, and further employs enhanced negative training (considering both in-batch and pre-batch negatives) and query-side fune-tuning.”
Phrase Retrieval Learns Passage Retrieval, Too (2021, methodology): “By evaluating DensePhrases (Lee et al., 2021) on popular QA datasets, we observe that it achieves competitive or even better passage retrieval accuracy compared to DPR, without any re-training or modiﬁcation to the original model (Table 1).”
Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction (2022, background): “[24] learn phrase (multi-word) representations for QA collections.”
PromptCap: Prompt-Guided Task-Aware Image Captioning (2022, methodology): “To incorporate external knowledge into answering VQA questions, we leverage a retriever [29] to first retrieve relevant texts from Wikipedia given the concatenation of the question and PromptCap as a query.”
A Unified Encoder-Decoder Framework with Entity Memory (2022, background): “overheads in indexing, retrieving, and reading a large number of extra documents (Lee et al., 2021; de Jong et al., 2022).”
Simple Entity-Centric Questions Challenge Dense Retrievers (2021, background): “, 2021), or fine-grained phrase retrieval (Lee et al., 2021).”
Text Embeddings by Weakly-Supervised Contrastive Pre-training (2022, methodology): “Negative Sampling We explore two alternative methods to enlarge the number of negatives: Pre-batch negatives [33] reuse embeddings from previous batches as additional negatives, while MoCo [25] introduces a momentum encoder and uses a FIFO queue to store negatives.”
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (2021, background): “DensePhrases (Lee et al., 2021a) is a system for Open-QA that relies on a multi-vector encoding of passages, though its search is conducted at the level of individual vectors and not aggregated with late interaction.”