Are we really making much progress? A worrying analysis of recent neural recommendation approaches

By Maurizio Ferrari Dacrema, Paolo Cremonesi, Dietmar Jannach. 2019

686 citations recorded by the source corpus.

Deep learning techniques have become the method of choice for researchers working on algorithmic aspects of recommender systems. With the strongly increased interest in machine learning in general, it has, as a result, become difficult to keep track of what represents the state-of-the-art at the moment, e.g., for top-n recommendation tasks. At the same time, several recent publications point out problems in today's research practice in applied machine learning, e.g., in terms of the reproducibility of the results or the choice of the baselines when proposing new models. In this work, we report the results of a systematic analysis of algorithmic proposals for top-n recommendation tasks. Specifically, we considered 18 algorithms that were presented at top-level research conferences in the last years. Only 7 of them could be reproduced with reasonable effort. For these methods, it however turned out that 6 of them can often be outperformed with comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques. The remaining one clearly outperformed the baselines but did not consistently outperform a well-tuned non-neural linear ranking method. Overall, our work sheds light on a number of potential problems in today's machine learning scholarship and calls for improved scientific practices in this area.

Subfields: Computer Vision and Pattern Recognition, Information Systems, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Systems

In Management Science and Operations Research

Citations in the corpus

Explainable Recommendation: A Survey and New Perspectives (2018, background): “We acknowledge that whether DL models truly improve the recommendation performance is controversial (Dacrema et al., 2019), but this problem is out of the scope of this survey.”
The neural hype, justified! (2019, background): “A research study with a similar message, but focused on recommender systems, was recognized with the best paper award at RecSys 2019 [4].”
The Computational Limits of Deep Learning (2020, background): “Using established benchmarks to measure progress in these areas is important because it ensures a common baseline for comparison (which is a signiﬁcant problem in parts of the deep learning literature[44]).”
Meta-learning on Heterogeneous Information Networks for Cold-start Recommendation (2020, background): “3403207 1 INTRODUCTION Recommender systems [3, 11, 31] have been widely deployed in various online services, such as E-commerce platforms and news portals, to address the issue of information overload for users.”
The Benchmark Lottery (2021, methodology): “Therefore, it is still quite unclear about which modern RecSys method one should adopt, as model comparisons are difﬁcult to interpret [Dacrema et al., 2019].”
Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5) (2022, background): “Though simple, they have been shown to be more effective than many existing neural models on rating prediction and direct recommendation tasks [8, 9, 46].”
Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems (2019, background): “Embedding-based approaches, such as matrix factorization (MF) [22, 28, 46] or neural collaborative ﬁltering (NCF) [15, 24] are among the most popular and eﬃcient solutions to matrix completion.”
On the Gap between Adoption and Understanding in NLP (2021, background): “Often systematic evaluations are run to compare different state-of-the-art methods and show that the re-sults are sometimes on par with baselines (Dacrema et al., 2019; Errica et al., 2020).”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, background): “3 We also note that whether the usage of deep models improves the recommendation system is an open discussion [42], but this is out of the scope of this survey.”
A Metric Learning Reality Check (2020, methodology): “Papers of this type have been written for machine learning [30], image classification [4], neural network pruning [2], information retrieval [68], recommender systems [10], and generative adversarial networks [33].”
A Fair Comparison of Graph Neural Networks for Graph Classification (2019, background): “Finally, the work of Dacrema et al. (2019) criticizes a large number of neural recommender systems, most of which are not reproducible, showing that only one of them truly improves against a simple baseline.”
DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems (2020, methodology): “In light of recent concerns about poor reproducibility of published results [8, 33, 38], we conducted a fair and comprehensive experimental study with extensive hyper-parameter search to properly tune all the baselines and proposed approaches.”
Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval (2019, result): “Comparisons to weak baselines that inﬂate the merits of an approach are not new problems in information retrieval (Armstrong et al., 2009), and researchers have in fact observed similar issues in the recommender systems literature as well (Ren-dle et al., 2019; Dacrema et al., 2019).”