Relevance-guided Supervision for OpenQA with ColBERT

By Omar Khattab, Christopher Potts, Matei A. Zaharia. 2020

113 citations recorded by the source corpus.

Abstract Systems for Open-Domain Question Answering (OpenQA) generally depend on a retriever for finding candidate passages in a large corpus and a reader for extracting answers from those passages. In much recent work, the retriever is a learned component that uses coarse-grained vector representations of questions and passages. We argue that this modeling choice is insufficiently expressive for dealing with the complexity of natural language questions. To address this, we define ColBERT-QA, which adapts the scalable neural retrieval model ColBERT to OpenQA. ColBERT creates fine-grained interactions between questions and passages. We propose an efficient weak supervision strategy that iteratively uses ColBERT to create its own training data. This greatly improves OpenQA retrieval on Natural Questions, SQuAD, and TriviaQA, and the resulting system attains state-of-the-art extractive OpenQA performance on all three datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Distilling Knowledge from Reader to Retriever for Question Answering (2020, methodology): “Recently, methods based on dense vectors and machine learning have shown promising re-sults (Karpukhin et al., 2020; Khattab et al., 2020).”
Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, methodology): “looked into using the attention in the reader to build a retriever but used manually crafted rules to reduce the token scores to document scores [13, 17, 30].”
Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “Similarly, Khattab et al. (2020) proposes the ColBERT model, which keeps a vector representation for each term of the queries and documents.”
You Only Need One Model for Open-domain Question Answering (2021, methodology): “While it would be optimal to retrieve passages based on cross-attention be-tween the query and all the passages (Khattab and Zaharia, 2020), it is computationally intractable.”
Hindsight: Posterior-guided training of retrievers for improved open-ended generation (2021, methodology): “Recently, Khattab et al. (2021) has shown that ColBERT establishes state of the art retrieval results on open-domain QA benchmarks.”
R2-D2: A Modular Baseline for Open-Domain Question Answering (2021, background): “The progress can be attributed to advances in neural retrieval methods (Karpukhin et al., 2020; Izacard and Grave, 2020; Khattab et al., 2020; Luan et al., 2021; Xiong et al., 2020, inter alia) that benefit from smarter negative sampling strategies or a better trade-off between complex…”
Retrieval Augmentation Reduces Hallucination in Conversation (2021, methodology): “ColBERT Khattab and Zaharia (2020) propose ColBERT as a method of computing contextualized late-stage interaction between the context and candidate representations to improve ranking capabilities, and indeed the method is extended to downstream generative QA models in Khattab et al. (2020).”
A Memory Efficient Baseline for Open Domain Question Answering (2020, background): “Furthemore, similarly to Khattab et al. (2020), we pad questions and passages to fixed-length sequences without using mask when applying the embedder function.”
Multi-Task Retrieval for Knowledge-Intensive Tasks (2021, methodology): “One further approach is ColBERT-QA (Khattab et al., 2020), which additionally uses a data augmentation strategy closely related to our own approach described in § 3.”
Retrieve, Rerank, Read, then Iterate: Answering Open-Domain Questions of Arbitrary Complexity from Text (2020, background): “Firstly, some previous work assumes the access to a well-tuned parameterized retrieval system that helps navigate the large amount of text (Das et al., 2019; Lee et al., 2019; Asai et al., 2020; Guu et al., 2020; Khattab et al., 2020; Xiong et al., 2020).”
Towards Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “Only ColBERT and docT5query, which is based on BM25, obtain better results than BM25.”
Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering (2021, methodology): “According to the different ways of encoding the question and document as well as of scoring their similarity, dense retrievers in existing OpenQA systems can be roughly divided into three types: Representation-based Retriever [16], [29], [36], [72], Interaction-based Retriever [15], [31], and Repres”
Generate rather than Retrieve: Large Language Models are Strong Context Generators (2022, background): “Second, the representations of questions and documents are typically obtained independently in modern two-tower dense retrieval models (Karpukhin et al., 2020), leading to only shallow interactions captured between them (Khattab et al., 2021).”
Rethinking the Objectives of Extractive Question Answering (2020, background): “Using the proposed approach, we were able to reach significant improvements through the wide spectrum of datasets, including +1.28 EM on Adversarial SQuAD and +2.07 EM on NaturalQuestions for BERT-base.”
Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP (2022, methodology): “For instance, Khattab et al. (2021a) introduces a pipeline for multi-hop reasoning that is trained with weak supervision, extending work by Lee et al. (2019) and Khattab et al. (2021b).”
X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval (2022, methodology): “1, Mean-Max strategies [25, 26, 47, 59] ignore the importance of different frames and words.”
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems (2023, background): “Retrieval-augmented generation (RAG; Guu et al. 2020; Lewis et al. 2020; Khattab et al. 2021; Izac-ard et al. 2022) is now a common strategy for bol-stering LLMs by combining them with retrieval systems.”
Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval (2021, methodology): “We build a strong many-hop baseline model that combines recent results from the open-domain question answering [17] and multi-hop [23] literatures.”
Extending Context Window of Large Language Models via Positional Interpolation (2023, background): “One line of work extends LLMs by augmenting it with retrieval modules which fetch related documents and include the retrieval results into the input context of an LLM (Karpukhin et al., 2020; Guu et al., 2020; Izacard et al., 2022; Jiang et al., 2022; Khattab et al., 2021; Santhanam et al., 2022).”