Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models

By Wei Yang, Kuang Lu, Peilin Yang, Jimmy J. Lin. 2019

152 citations recorded by the source corpus.

Is neural IR mostly hype? In a recent SIGIR Forum article, Lin expressed skepticism that neural ranking models were actually improving ad hoc retrieval effectiveness in limited data scenarios. He provided anecdotal evidence that authors of neural IR papers demonstrate "wins" by comparing against weak baselines. This paper provides a rigorous evaluation of those claims in two ways: First, we conducted a meta-analysis of papers that have reported experimental results on the TREC Robust04 test collection. We do not find evidence of an upward trend in effectiveness over time. In fact, the best reported results are from a decade ago and no recent neural approach comes close. Second, we applied five recent neural models to rerank the strong baselines that Lin used to make his arguments. A significant improvement was observed for one of the models, demonstrating additivity in gains. While there appears to be merit to neural IR approaches, at least some of the gains reported in the literature appear illusory.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Document Ranking with a Pretrained Sequence-to-Sequence Model (2020, background): “Note that the latter is considered to be a strong baseline, even for pre-BERT neural ranking models (Yang et al., 2019).”
The neural hype, justified! (2019, background): “Whereas before I presented empirical evidence, later expanded upon in Yang et al. [22], that (pre-BERT) neural models, with limited training data, struggle to even beat well-tuned “bag of words” query expansion techniques, I now believe that pretrained transformer models are unequivocally more…”
Selective Weak Supervision for Neural Information Retrieval (2020, methodology): “Similar ambivalent effectiveness on Neu-IR systems when relevance training data are limited has been observed in multiple previous studies [4, 5, 46, 47].”
Training Curricula for Open Domain Answer Re-Ranking (2020, methodology): “Contrary to recent critiques of the actual effectiveness of neural ranking architectures [21, 39], recent work with transformer-based contextualized language models (e.”
A Little Bit Is Worse Than None: Ranking with Limited Training Data (2020, methodology): “On Robust04, we use 5-fold cross-validation with three folds for training, one for validation, and the other for evaluation, matching the splits in (Yang et al., 2019).”
Overview of the TREC 2020 Deep Learning Track (2021, background): “…into one of three categories: • nnlm: if the run employs large scale pre-trained neural language models, such as BERT [Devlin et al., 2018] or XLNet [Yang et al., 2019b] • nn: if the run employs some form of neural network based approach— e.g. , Duet [Mitra et al., 2017, Mitra and Craswell, 2019]…”
A Metric Learning Reality Check (2020, methodology): “Papers of this type have been written for machine learning [30], image classification [4], neural network pruning [2], information retrieval [68], recommender systems [10], and generative adversarial networks [33].”
Cross-Domain Sentence Modeling for Relevance Transfer with BERT (2019, methodology): “[68] recently conducted a thorough meta-analysis of over 100 papers that report results on the TREC Robust 2004 Track.”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, methodology): “• In information retrieval, where a non-neural method from 2004 is superior to all neural approaches developed through 2019 [164].”
SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking (2021, background): “While BOW models remain strong baselines [27], they suffer from the long standing vocabulary mismatch problem, where relevant documents might not contain terms that appear in the query.”
Keyphrase Generation for Scientific Document Retrieval (2020, methodology): “We further apply a pseudo-relevance feedback method, known as RM3 (Abdul-Jaleel et al., 2004), on top of the models to achieve strong, near state-of-the-art retrieval results (Lin, 2019; Yang et al., 2019).”
MarkedBERT: Integrating Traditional IR Cues in Pre-trained Language Models for Passage Retrieval (2020, result): “In spite of all the work invested in the development of neural ranking models, a recent study conducted over a 100 papers by [16] showed that most models failed against strong nonneural baselines.”
Information Retrieval: A Biomedical and Health Perspective (2020, background): “laid out principles for responsible reuse of RCT data [206].”
Deep learning in clinical natural language processing: a methodical review (2019, background): “Further, it is clear from other subfields involving ML for clinical tasks that deep learning is not always successful.(113,114) In this we mean that oftentimes it fails to outperform basic models, such as logistic regression.”
Overview of the TREC 2019 deep learning track (2020, methodology): “With small data, there has been some discussion about whether deep learning methods really outperform strong traditional IR baselines [Yang et al., 2019a].”
Applying BERT to Document Retrieval with Birch (2019, methodology): “Yang et al. (2019b) also combined Anserini and BERT to demonstrate large improvements in open-domain question answering directly on Wikipedia.”