The Neural Hype and Comparisons Against Weak Baselines

By Jimmy J. Lin. 2019

154 citations recorded by the source corpus.

Recently, the machine learning community paused in a moment of self-reflection. In a widelydiscussed paper at ICLR 2018, Sculley et al. [13] wrote: "We observe that the rate of empirical advancement may not have been matched by consistent increase in the level of empirical rigor across the field as a whole." Their primary complaint is the development of a "research and publication culture that emphasizes wins" (emphasis in original), which typically means "demonstrating that a new method beats previous methods on a given task or benchmark". An apt description might be "leaderboard chasing"-and for many vision and NLP tasks, this isn't a metaphor. There are literally centralized leaderboards1 that track incremental progress, down to the fifth decimal point, some persisting over years, accumulating dozens of entries. Sculley et al. remind us that "the goal of science is not wins, but knowledge". The structure of the scientific enterprise today (pressure to publish, pace of progress, etc.) means that "winning" and "doing good science" are often not fully aligned. To wit, they cite a number of papers showing that recent advances in neural networks could very well be attributed to mundane issues like better hyperparameter optimization. Many results can't be reproduced, and some observed improvements might just be noise.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Latent Retrieval for Weakly Supervised Open Domain Question Answering (2019, background): “Language Models While unsupervised neural retrieval is notoriously difficult to improve over traditional IR (Lin, 2019), we include them as baselines for comparison.”
Passage Re-ranking with BERT (2019, critical): “This, at least in part, prevented the neural ranking models from being successful when compared to more classical IR techniques (Lin, 2019).”
End-to-End Open-Domain Question Answering with BERTserini (2019, background): “Recently, Lin (2018) showed that a well-tuned Anserini implementation of a query expansion model proposed over a decade ago still beats two recent neural models for document ranking.”
Multi-Stage Document Ranking with BERT (2019, background): “As demonstrated by Lin (2019), in a limited data regime, it is not entirely clear that neural techniques actually perform better than well-tuned “classic” IR techniques; subsequent work by Yang et al. (2019a) show that the gains are modest at best.”
Simple Applications of BERT for Ad Hoc Document Retrieval (2019, methodology): “Researchers have proposed many neural ranking models (Mitra and Craswell, 2019), but there has recently been some skepticism about whether they have truly advanced the state of the art (Lin, 2018), at least in the absence of large amounts of log data only available to a few organizations.”
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2020, background): “However, it has been shown (Lin, 2019) that these neural ranking models do not yield substantial improvement over traditional methods due to the lack of large-scale labeled retrieval data.”
Cross-Domain Sentence Modeling for Relevance Transfer with BERT (2019, methodology): “For Robust04, we follow the five-fold cross-validation settings in Lin [34] over 250 topics.”
Complementing Lexical Retrieval with Semantic Residual Embedding (2020, methodology): “CLEAR vs. Classic Lexical Retrieval BM25 and BM25+RM3 are among the most widely-used ﬁrst-stage retrieval models in state-of-the-at search engines (Lin, 2019).”
Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models (2019, background): “In a recent SIGIR Forum opinion piece, Lin [11] criticized the state of information retrieval research, making two main points.”
The neural hype, justified! (2019, background): “One year ago, in the SIGIR Forum issue of December 2018, I ranted about the “neural hype” [9].”
MS MARCO: Benchmarking Ranking Models in the Large-Data Regime (2021, methodology): “methods against each other, as well as against well-tuned traditional IR methods, which led to concerns [41] in the IR community as to whether “real progress” was being made.”
Keyphrase Generation for Scientific Document Retrieval (2020, methodology): “Overall, BM25 + RM3 achieves the best retrieval e ﬀ ectiveness, conﬁrming previous ﬁndings on ad-hoc retrieval in limited data scenarios (Lin, 2019).”
Investigating the Successes and Failures of BERT for Passage Re-Ranking (2019, background): “In information retrieval (IR), the lack of publicly available large-scale datasets for many tasks, such as ad-hoc retrieval, has restricted observing substantial improvements over traditional methods [6, 13].”
Training Curricula for Open Domain Answer Re-Ranking (2020, result): “Contrary to recent critiques of the actual effectiveness of neural ranking architectures [21, 39], recent work with transformer-based contextualized language models (e.g., BERT [9]) on document and answer ranking have shown clear ranking superiority over prior baselines [23, 28, 29].”
The Simplest Thing That Can Possibly Work: Pseudo-Relevance Feedback Using Text Classification (2019, methodology): “In a similar vein, I’ve recently expressed skepticism about whether neural ranking models actually improve over existing models, at least absent large amounts of behavioral training data [13].”
Shallow pooling for sparse labels (2021, background): “As recently as 2018 it was unclear that neural rankers could outperform non-neural rankers on traditional “ad hoc” document and passage ranking tasks when only content-based, textual features are available [18].”
Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval (2019, methodology): “Following Lin (2018) and Yang et al. (2019a), we use BM25 with RM3 query expansion (default parameters), which is a strong baseline, and has already been shown to beat most existing neural models.”
The Benchmark Lottery (2021, background): “[Torralba and Efros, 2011, Armstrong et al., 2009, Machado et al., 2018, Sculley et al., 2018, Lin, 2019, Bowman and Dahl, 2021, Bouthillier et al., 2021, Recht et al., 2019, Lin et al., 2021, Balduzzi et al., 2018, Lipton and Steinhardt, 2018, Bouthillier et al., 2019, Vania et al., 2020, Mishra…”