Simple Applications of BERT for Ad Hoc Document Retrieval

By Wei Yang, Haotian Zhang, Jimmy J. Lin. 2019

201 citations recorded by the source corpus.

Following recent successes in applying BERT to question answering, we explore simple applications to ad hoc document retrieval. This required confronting the challenge posed by documents that are typically longer than the length of input BERT was designed to handle. We address this issue by applying inference on sentences individually, and then aggregating sentence scores to produce document scores. Experiments on TREC microblog and newswire test collections show that our approach is simple yet effective, as we report the highest average precision on these datasets by neural approaches that we are aware of.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

Citations in the corpus

How Context Affects Language Models' Factual Predictions (2020, methodology): “Pre-trained language models have also been ﬁne-tuned and adapted to be used as information retrieval systems [Yang et al., 2019b, Yilmaz et al., 2019, Seo et al., 2019].”
BERTScore: Evaluating Text Generation with BERT (2019, background): “…inference, sentiment analysis, paraphrase detection, question answering, named entity recognition (Devlin et al., 2019), text summarization (Liu, 2019), contextual emotion detection (Huang et al., 2019), citation recommendation (Jeong et al., 2019), and document retrieval (Yang et al., 2019).”
Sparse, Dense, and Attentional Representations for Text Retrieval (2020, methodology): “Due to the computational cost, crossattentional BERT is applied only in reranking as in prior work (Nogueira and Cho, 2019; Yang et al., 2019).”
Revisiting Few-sample BERT Fine-tuning (2020, background): “1 Introduction Fine-tuning self-supervised pre-trained models has significantly boosted state-of-the-art performance on Natural Language Processing (NLP) tasks [13, 28, 43, 51, 56].”
Multi-Stage Document Ranking with BERT (2019, methodology): “As demonstrated by Lin (2019), in a limited data regime, it is not entirely clear that neural techniques actually perform better than well-tuned “classic” IR techniques; subsequent work by Yang et al. (2019a) show that the gains are modest at best.”
Efficient Document Re-Ranking for Transformers by Precomputing Term Representations (2020, methodology): “ons, while being simpler, e.g., no paragraph segmentation required, as is needed by FirstP/MaxP/SumP[4],oralternativetrainingdatasetsandsentence segmentation, as required by the system of Yang et al. [48]. Vanilla BERT encodes as much of the document as possible (adhering to the transformer maximum ”
Applying BERT to Document Retrieval with Birch (2019, methodology): “Yang et al. (2019b) also combined Anserini and BERT to demonstrate large improvements in open-domain question answering directly on Wikipedia.”
The neural hype, justified! (2019, background): “Over the past year, a number of studies [23, 5, 11, 16, 14, 25] have independently established this empirical result.”
A Deep Look into Neural Ranking Models for Information Retrieval (2019, unknown): “Some BERT-based model [89] takes attention as the interaction function to learn the interaction vector (i.”
Delaying Interaction Layers in Transformer-based Encoders for Efficient Open Domain Question Answering (2020, methodology): “To assess the interest of delayed interaction, we run experiments on the reference Open Domain Question Answering challenge OpenSQuAD as Yang et al. (2019a), starting by the associated extractive Question Answering sub-task SQuAD v1.”
AUEB at BioASQ 7: Document and Snippet Retrieval (2019, unknown): “Most approaches train shallow task-specific layers on top of bert [30, 20], much as in our bert based document retrieval model (Section 2.”
Cross-Lingual Relevance Transfer for Document Retrieval (2019, background): “Interest in these models have also extended to search-related tasks such as retrieval-based question answering (Yang et al., 2019a), passage ranking (Nogueira and Cho, 2019), and document ranking (Yang et al., 2019b; MacAvaney et al., 2019; Yilmaz et al., 2019).”
Cross-Domain Sentence Modeling for Relevance Transfer with BERT (2019, methodology): “[70] are the first to successfully apply BERT to “ad hoc” document retrieval.”
Reranking for Efficient Transformer-based Answer Selection (2020, background): “Yang et al. [23] applied it to Ad Hoc Document Retrieval, obtaining significant improvement.”
Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval (2019, methodology): “For Robust04, we follow the ﬁve-fold cross-validation settings in Lin (2018) and Yang et al. (2019a); for Core17 and Core18 we similarly apply ﬁve-fold cross validation.”
The Cascade Transformer: an Application for Efficient Answer Sentence Selection (2020, methodology): “, 2019), ad-hoc document retrieval (Yang et al., 2019b; MacAvaney et al., 2019), and semantic understanding (Liu et al.”
Zero-shot Neural Retrieval via Domain-targeted Synthetic Query Generation (2020, background): “As a result, neural retrieval models achieve new state-of-the-art performance on several information retrieval (IR) benchmarks (Guo et al., 2016; Pang et al., 2016; Hui et al., 2017; Dai et al., 2018; MacAvaney et al., 2019; Yang et al., 2019a; Gillick et al., 2018; Yang et al., 2019b).”