HARK Side of Deep Learning - From Grad Student Descent to Automated Machine Learning
By Oguzhan Gencoglu, Mark J. van Gils, Esin Guldogan, Chamin Morikawa, Mehmet Süzen, Mathias Gruber, Jussi Leinonen, Heikki Huttunen. 2019
38 citations recorded by the source corpus.
Recent advancements in machine learning research, i.e., deep learning, introduced methods that excel conventional algorithms as well as humans in several complex tasks, ranging from detection of objects in images and speech recognition to playing difficult strategic games. However, the current methodology of machine learning research and consequently, implementations of the real-world applications of such algorithms, seems to have a recurring HARKing (Hypothesizing After the Results are Known) issue. In this work, we elaborate on the algorithmic, economic and social reasons and consequences of this phenomenon. We present examples from current common practices of conducting machine learning research (e.g. avoidance of reporting negative results) and failure of generalization ability of the proposed algorithms and datasets in actual real-life usage. Furthermore, a potential future trajectory of machine learning research and development from the perspective of accountable, unbiased, ethical and privacy-aware algorithmic decision making is discussed. We would like to emphasize that with this discussion we neither claim to provide an exhaustive argumentation nor blame any specific institution or individual on the raised issues. This is simply a discussion put forth by us, insiders of the machine learning field, reflecting on us.
Subfields: Artificial Intelligence, Safety Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Pablo Hernandez-Leal
- Bilal Kartal
- Antoine Yang
- Fabio Maria Carlucci
- Pedro M. Esperança
- Matthew E. Taylor
- Peter Stone
- Mohammad Rostami
In Safety Research
Citations in the corpus
- NAS evaluation is frustratingly hard (2019, background): “Furthermore, the lack of ablation studies in most works makes it harder to pinpoint which components are instrumental to the final performance, which can easily lead to Hypothesizing After the Results are Known (HARKing; Gencoglu et al., 2019).”
- A survey and critique of multiagent deep reinforcement learning (2018, background): “also benefit from reporting negative results [313, 314, 310, 315] for carefully designed hypothesis and experiments.”