The Scientific Method in the Science of Machine Learning
By Jessica Zosa Forde, Michela Paganini. 2019
42 citations recorded by the source corpus.
In the quest to align deep learning with the sciences to address calls for rigor, safety, and interpretability in machine learning systems, this contribution identifies key missing pieces: the stages of hypothesis formulation and testing, as well as statistical and systematic uncertainty estimation -- core tenets of the scientific method. This position paper discusses the ways in which contemporary science is conducted in other domains and identifies potentially useful practices. We present a case study from physics and describe how this field has promoted rigor through specific methodological practices, and provide recommendations on how machine learning researchers can adopt these practices into the research ecosystem. We argue that both domain-driven experiments and application-agnostic questions of the inner workings of fundamental building blocks of machine learning models ought to be examined with the tools of the scientific method, to ensure we not only understand effect, but also begin to understand cause, which is the raison d'\^{e}tre of science.
Subfields: Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Natural Language Processing
- Pablo Hernandez-Leal
- Hanna M. Wallach
- Bilal Kartal
- Matthew E. Taylor
- Abigail Z. Jacobs
- Andrew McCallum
- Charles Sutton
- Mark W. Schmidt
Citations in the corpus
- Measurement and Fairness (2019, background): “Failure to consider reliability hinders the development of meaningful computational systems [21], and failure to report reliability can mask potentially meaningless variation in measurements used in practice.”
- A survey and critique of multiagent deep reinforcement learning (2018, background): “We believe that together with following the advice on how to design experiments and report results [197], the community would also benefit from reporting negative results [100,108,270,286] for carefully designed hypothesis and experiments.”