Deeper Text Understanding for IR with Contextual Neural Language Modeling

By Zhuyun Dai, Jamie Callan. 2019

509 citations recorded by the source corpus.

Neural networks provide new possibilities to automatically learn complex language patterns and query-document relations. Neural IR models have achieved promising results in learning query-document relevance patterns, but few explorations have been done on understanding the text content of a query or a document. This paper studies leveraging a recently-proposed contextual neural language model, BERT, to provide deeper text understanding for IR. Experimental results demonstrate that the contextual text representations from BERT are more effective than traditional word embeddings. Compared to bag-of-words retrieval models, the contextual language model can better leverage language structures, bringing large improvements on queries written in natural languages. Combining the text understanding ability with search knowledge leads to an enhanced pre-trained BERT model that can benefit related search tasks where training data are limited.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, methodology): “lied on all term pairs, while BERT-Siamese is representation-based. Previous research found interaction-based models more effective as they capture the relevance match between all querydocument terms [32, 34, 35, 36]. However, the effectiveness of interaction-based models is only available at the re”
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (2020, background): “Nogueira et al. [25], MacAvaney et al. [19], and Dai et al. [4] investigate incorporating such LMs (mainly BERT, but also ELMo [29]) on different ranking datasets.”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, neutral): “We cover a wide range of techniques, grouped into two categories: transformer models that perform reranking in multi-stage ranking architectures and learned dense representations that perform ranking directly.”
Document Ranking with a Pretrained Sequence-to-Sequence Model (2020, background): “, 2019), BERT– MaxP (Dai and Callan, 2019), Birch (Yilmaz et al.”
Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval (2019, methodology): “BERT has received a lot of attention for IR, mainly focused on using it as a black-box re-ranking model to predict query-document relevance scores [8, 22].”
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2020, background): “Recently, researchers have utilized deep learning to improve traditional passage retrievers, including document expansions (Nogueira et al., 2019c), question expansions (Mao et al., 2020) and term weight estimation (Dai and Callan, 2019).”
Context-Aware Document Term Weighting for Ad-Hoc Search (2020, methodology): “Although much recent work has applied contextualized neural language models like BERT [14] to IR tasks, prior work mainly focused on using BERT as black-box re-ranker [9, 27, 28, 31].”
Efficient Document Re-Ranking for Transformers by Precomputing Term Representations (2020, methodology): “Dai and Callan [4] found that the additional context given by natural language queries (e.g., topic descriptions) can improve document ranking performance, when compared with keyword-based queries.”
Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-Shot Learning (2019, background): “2 Some have observed that the context provided by query descriptions are valuable for neural ranking, particularly when using contextualized language models [9].”
Fine-Tuning LLaMA for Multi-Stage Text Retrieval (2023, unknown): “We see that both our retriever and reranker outperform existing methods. repLLaMA achieves an MRR@100 score that is about 3 points higher than CoCondenser, while rank-LLaMA exceeds (to our knowledge) the current state-of-the-art document reranker, MORES+ [8], by 1 point in MRR@100.”
Fine-grained Fact Verification with Kernel Graph Attention Network (2019, background): “The kernel extracts matching patterns which provide a variety of relevance match signals and shows strong performance in various ad-hoc retrieval dataset (Dai and Callan, 2019).”
Long Document Ranking with Query-Directed Sparse Transformer (2020, methodology): “And recently the state-of-the-art in many text ranking tasks has been taken by BERT or other pretrained language models (Devlin et al., 2019; Nogueira et al., 2019; Nogueira and Cho, 2019; Dai and Callan, 2019; Yang et al., 2019; Craswell et al., 2020), when sufficient relevance labels are available”
Rethinking Query Expansion for BERT Reranking (2020, methodology): “It has been previously shown that applying a BERT reranker using descriptive queries written in natural language provide impressive performance gains over using classic keywords queries [2].”
Capreolus: A Toolkit for End-to-End Neural Ad Hoc Retrieval (2020, result): “However, recent experiments with BERT-based rerankers [1, 9, 19] show impressive improvements on Robust04, even with limited data.”
SLEDGE: A Simple Yet Effective Baseline for Coronavirus Scientific Knowledge Search (2020, background): “Others later observed effectiveness at document ranking tasks, showing that these models can handle natural-language questions better than prior approaches (Dai and Callan, 2019b) and that they can be incorporated into prior neutral ranking techniques (MacAvaney et al., 2019).”
Deep Reinforced Query Reformulation for Information Retrieval (2020, methodology): “Indeed, recent context-aware neural ranking techniques such as BERT have been shown to be effective over question-like queries [12].”
EARL: Speedup Transformer-based Rankers with Pre-computed Representation (2020, methodology): “The experimental setup on ClueWeb09-B followedDai and Callan [6].”
COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List (2021, methodology): “the-art in text ranking with huge performance leap (Nogueira and Cho, 2019; Dai and Callan, 2019b).”
Early Exiting BERT for Efficient Document Ranking (2020, background): “…recent years, large-scale pre-trained language models, especially those based on the transformer architecture (Vaswani et al., 2017), have been applied to IR tasks and have pushed the state of the art even further (Nogueira and Cho, 2019; Dai and Callan, 2019; Yilmaz et al., 2019; Li et al., 2020).”