Contextualized Sparse Representations for Real-Time Open-Domain Question Answering

By Jinhyuk Lee, Minjoon Seo, Hannaneh Hajishirzi, Jaewoo Kang. 2019

31 citations recorded by the source corpus.

Open-domain question answering can be formulated as a phrase retrieval problem, in which we can expect huge scalability and speed benefit but often suffer from low accuracy due to the limitation of existing phrase representation models. In this paper, we aim to improve the quality of each phrase embedding by augmenting it with a contextualized sparse representation (Sparc). Unlike previous sparse vectors that are term-frequency-based (e.g., tf-idf) or directly learned (only few thousand dimensions), we leverage rectified self-attention to indirectly learn sparse vectors in n-gram vocabulary space. By augmenting the previous phrase retrieval model (Seo et al., 2019) with Sparc, we show 4%+ improvement in CuratedTREC and SQuAD-Open. Our CuratedTREC score is even better than the best known retrieve & read model with at least 45x faster inference speed.

Subfields: Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Learning Dense Representations of Phrases at Scale (2020, methodology): “Second, in our architecture, we use a base model (SpanBERT-base) for a smaller dimension of token representations ( d = 768 ) and does not use any sparse representations including tf-idf or contex-tualized sparse representations (Lee et al., 2020).”
Question Answering Infused Pre-training of General-Purpose Contextualized Representations (2021, methodology): “Bi-encoder QA has been used for efficient opendomain QA via phrase retrieval (Seo et al., 2018, 2019; Lee et al., 2020, 2021), but its lower accuracy compared to cross-encoder QA has previously been viewed as a drawback.”
Phrase Retrieval Learns Passage Retrieval, Too (2021, methodology): “While previous work (Seo et al., 2019; Lee et al., 2020) relied on a combination of dense and sparse vectors, Lee et al. (2021) demonstrate that dense representations of phrases alone are sufﬁcient to close the performance gap with retriever-reader systems.”
SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval (2020, background): “This paper is perhaps the most re-lated to the sparse representations from (Lee et al., 2020).”